In response to the requirement of developing competency-based education (CBE) for improving the graduation competency and academic outcomes, CBE using a competency assessment has already been implemented in most domestic medical schools, allowing students to perform in a realistic practice setting.^1-2) However, pharmacy schools have not implemented a systematic approach to prepare all students to achieve their goals or outcomes in Korea. Although the Korean Association of Pharmacy Education (KAPE) has been discussing about reforming the curriculum of pharmacy schools through public hearings on competency-based, outcome-based pharmacy education³⁾ and has established major areas of essential competencies, no assessment tools have been developed for faculty members to measure the outcomes of established competencies and for pharmacy students to easily understand and successfully access the assessment test. However, the Accreditation Council for Pharmacy Education (ACPE) has prepared educational guidelines for United States pharmacy schools to develop a competency-based curriculum and has emphasized the importance of competency-based and outcome-based education in pharmacy education since 2016.4-7) It recommended essential competencies required for pharmacy students after graduation and focused on the importance of curricular experience, alongside a reliable ability assessment for improving the quality of pharmacy education.⁷⁾

When assessing the individual competencies through oral examination, it is important to use a reliable assessment tool that can minimize variability and ensure fairness between individuals to measure the students’ competency effectively.^8-11) Among the assessment tools, Harden’s objective structured clinical assessment (OSCE) was known as the gold standard for competency assessment in Europe and North America in the late 20th century.^12-14) It included a combination of standardized virtual patients and multi-station examination, and this combination made the OSCE a reliable and useful assessment tool.⁸⁾ As a result, OSCE has become widely used in North American medical schools; hence, it is easy for faculty members to set competency goals, design a curriculum, and assess the competency, and the students have become familiar with the OSCE performance.²⁾

Nevertheless, most faculty members in charge have encountered barriers in recruiting a variety of virtual patients to ensure the reliability of assessment due to the lack of individuals with integrated clinical experience to observe the students’ performance successfully. Therefore, implementation of the OSCE may require more time, effort, and costs for selecting and standardizing patients compared with traditional paper examinations.^2,10) Hence, the effect of using faculty assessors must be evaluated to reduce various barriers related to patient selection and standardization. Currently, the influence of faculty assessors on competency assessment has rarely been studied.

Therefore, this study aimed to identify the benefits and limitations of applying a competency assessment tool. It also aimed to explore the influence of faculty assessors’ role as virtual patients on competency assessment. Furthermore, it can serve as a reference to the pharmacy schools and the KAPE, which intends to prepare a competency-based and outcome-based curriculum.

Study design

This study was designed to identify the usefulness of the CBA tool by using the virtual pharmacists, faculty assessors and survey questionnaires. Using OSCE as the CBA assessment tool, students play the role of virtual pharmacists consulting patients. pharmacy faculty members and adjunct faculty members acted as assessors of students’ essential competencies and provided feedback. They assessed the communication skills and styles using a checklist as a reference. A survey of pharmacy students was conducted to assess the benefit of the test, and a survey of the assessors was conducted to establish the value of competency-based assessment (CBA). This study design was approved by the Institutional Review Board (IRB: No. 2019-03-19-05) of Duksung Women’s University with a nonhuman designation, and the study was conducted in accordance with the Declaration of Helsinki.

Study setting and participants

This study was conducted at the pharmacy school of Duksung Women’s University, Korea, in 2019. The pharmacy curriculum for students in the 5^th year included preliminary clinical practice education. This curriculum focused on teaching basic clinical pharmacy skills, including understanding prescriptions and problem solving, applying scientific knowledge when making clinical decisions, and providing drug information to patients. All students were supervised by faculty members who were in charge and in-patient pharmacists during their preliminary clinical practice. At the end of the education session, the students’ basic clinical knowledge and communication skills were assessed using the CBA. Prior to 2018, all 4th year pharmacy students were informed that participation in CBA was mandatory. All pharmacy students played a role of pharmacists during the assessment. To eliminate the problem of variability in the role of the patient as an assessor, this study evaluated whether the faculty members were suitable for the virtual patient role after completion of the examination by assessing the survey questionnaires from both faculty members and students. Six faculty members and 2 adjunct faculty members (all registered pharmacists) participated in the CBA as virtual patients. This study was conducted to provide preliminary data for the development of a competency-based curriculum at the College of Pharmacy. The main goal of the CBA was to ensure students’ acquisition of essential competencies by the end of the 5^th year.

Competency-based assessment process

Three stations were allotted to perform the CBA (two for clinical evaluations and one for demonstrating their clinical skill). Four topics related to the five fields of essential competencies were assessed in each station, which includes 4 topics related to the prescription drugs and disease management and 5 competencies related to the collection of patient information, providing correct and timely responses to requests for drug information with proper eye contact and non-verbal facilitation, communication of the regimen effectively with patients using clear language, assessment of patients’ progress while identifying errors and solving problems, and providing effective regimens. The competency assessment criteria were divided into three levels, such as good, average, and poor, and among them, the good competency was defined as “The student always provided consultation accurately and completely to patients in the corresponding competency.”, and the average competency was defined as “Students occasionally accurately and completely provided prescription drug consultation to patients in the corresponding competency.” The poor competency was defined as “Students most inaccurately or incompletely provided prescription drug consultation to patients in the corresponding competency.”. Assessment of the four topics related to the prescription drugs and disease management in the hospital was performed in stations 1 and 2. Assessment of the three topics related to insulin management according to insulin types was performed in station 3. A total of 28 topics were evaluated during the CBA. Before starting the CBA, the students were informed in detail about the CBA procedures and were provided with a timetable by an assistant. A team of faculty assessors was educated on the role of virtual patients and the objective assessments using case-based patient-consultation videos.

Immediately before entering the examination room, the students had the opportunity to view the test topics and evaluate them for 5 min using the reference material provided. During the test, each student rotated between the three stations with 7 min allocated for each student and 1 minute to move to the next station. To avoid cheating, soundproofing method was applied by installing removable partitions between each station. All the students were provided their CBA scores.

To ensure that the quality of the CBA was met, students provided survey evaluations and feedback regarding the examination process and the attitude of the assessors. Following the completion of the CBA, all faculty assessors also provided an evaluation of the quality of students’ competencies and the value of CBA and discussed the issues that occurred. The preliminary CBA was implemented in 2018, and the CBA process was updated in 2019. Subsequently, the students checked their scores and received feedback.

Evaluation of competencies

All faculty assessors assessed the students’ competencies and communication skills using the CBA checklist, on which they marked and completed where applicable during each observation. Each assessor was assigned to one station and assessed the four types of case-based scenarios using a Likert scale (good, acceptable, and poor). The case-based scenarios were developed based on the clinical knowledge of pharmacotherapy and skills, and these scenarios were revised every year. The Likert scale was improved to better assess the students’ competencies, such as professional knowledge, skills, abilities, behaviors, and attitudes. All assessments were provided by the assessors and survey results from students and assessors were collected after the completion of the CBA.

Data analysis

Quantitative data on students’ competencies were presented as percentages. The essential competencies (by communication skills) were classified into five competency fields, and the competency outcomes were evaluated according to the competency skill types, such as good, acceptable, and poor, using a Likert scale. Satisfaction with CBA value for assessors, satisfaction with the quality of CBA for students, and Satisfaction with the virtual patients for students were assessed using a Likert scale and expressed as percentages.

Identifying the level of competency

In total, 8 assessors, 4 assistants and 96 students participated in the CBA test. The five fields of essential competencies that were evaluated for each topic are shown in Table 1. Table 2 shows the competency scores of all students at each station. At station 1, 70.9% of students showed high levels of competency in collecting patient information, and 66.7%provided drug information while maintaining eye contact. Half of the students communicated effectively with patients using a clear language. 61.5% of students were less competent in assessing patient information, identifying errors, and solving problems. 78.1% of students demonstrated the lowest competency in the field of evaluating regimens and providing effective regimens.

Table 1

Essential competencies required for Pharmacy students and corresponding stations

Station topics•Each topic in stations assessed all 5 essential communication skill competenciesa
Essential competenciesb assessed	Station 1c	Station 2c	Station 3c
•Introduction of role and name of self then check full patient's identity	Blood pressure assessment (Station1_1) •Fungal infection and therapeutic recommendation (Station1_2) •Doxycycline evidence-based counseling for drug-drug interaction (Station1_3) •Diabetes drug management (Station1_4)	DVT management (Staton2_1) •Urinary tract infection and therapeutic recommendation (Station2_2) •Tramadol dosing and drug-drug interaction (Station2_3) •Hyperphosphatemia drug management in Chronic kidney disease (Station2_4)	Rapid acting Insulin syringe/pen dose, injection skill, pen and insulin storage, disposal of sharp needle •Long acting Insulin syringe/ pen dose, injection skill, pen and insulin storage, disposal of sharp needle •Premixed •Insulin pen dose, injection skill, pen and insulin storage, disposal of sharp needle
•Provide correct and timely response to request for drug information completely with engaging eye contact and using non-verbal facilitation
•Communicate regimen effectively with patient using clear language and clarify questions requested from patient
•Assess patient's information and identifying errors and resolving medication-related problems
•Evaluate patient's regimen and provide effective regimens and counseling to patients

^aEssential competencies are goal of pharmacy education in standard 2016 by ACEP.

^bEssential competencies assessed in the OSCE are five

^cEach Station1 and Station 2 contain four topics and Station 3 contains three topics.

Table 2

Pharmacy students’ essential competencies by communication skills and outcome in competency-based assessment (CBA)

	Communication skillsb	Total Scorea(n, %), n=96
	Essential Competencies	Station 1			Station 2			Station 3
		Good	Acceptable	Poor	Good	Acceptable	Poor	Good	Acceptable	Poor
• Introduction of role and name of self then check full patient's identity		68 (70.9)	27 (28.1)	1 (1.0)	88 (91.7)	8 (8.3)	0	88 (91.7)	8 (8.3)	0
• Provide correct, complete, and timely response to request for drug information with engaging eye contact and using non-verbal facilitation cues		64 (66.7)	28 {29.2}	4 (4.1)	72 (75)	22 (22.9)	2 (2.1)	72 (75)	22 (22.9)	2 (2.1)
• Communicate regimen effectively with patient using clear language and clarify patient’s questions		50 (52.1)	39 (40.6)	7 (7.3)	60 (62.5)	33 (34.4)	3 (3.1)	60 (62.5)	33 (34.4)	3 (3.1)
• Assess patient's information and identify errors and resolve medication-related problems		37 (38.5)	41 (42.7)	18 (18.8)	32 (33.3)	43 (44.8)	21 (21.9)	32 (33.3)	43 (44.8)	21 (21.9)
• Evaluate patient's regimen and provide effective regimen and counseling to patients		21 (21.9)	46 (47.9)	29 (30.2)	16 (16.7)	49 (51.0)	31 (32.3)	16 (16.7)	49 (51.0)	31 (32.3)

^aData are presented as percentage

^bThe communication skills score provided here is the percentage of the sum of the performance of each pharmacy student on four subtopics in each station checklist in the CBA.

^bThe communication skills consist of three types according to performance: Good, acceptable, poor.

Abbreviation: competency-based assessment, CBA

At station 2, 66.7% of students showed lower competencies in the assessment of patient information, identifying errors, and solving problems. In total, 83.3% of students showed the lowest competencies in evaluating regimens and providing effective regimens.

At station 3, students demonstrated high competencies in the field of collecting patient information (91.7%) and providing insulin information (75%). Approximately 83.3% of students demonstrated lower competencies in evaluating and providing effective insulin regimens.

Advantage of a CBA tool

Nearly all students (90.5%) either agreed or completely agreed that they understood how to apply their knowledge to patient-centered care. 77.4% were able to effectively communicate with patients. Over 60% of the students agreed or completely agreed that they gained clinical knowledge and improved their essential competencies via the CBA test. Approximately 64.2% of the students either agreed or completely agreed that they were able to solve complex problems (Fig. 1). Eight questionnaires were obtained from faculty assessors (response rate: 100%) in Table 3. All assessors agreed that the CBA tool was important in evaluating the students’ ability to care for patients. They all agreed or completely agreed that CBA was necessary to improve students’ essential competencies in communication skills, evaluation, and problem-solving.

Table 3

Faculty assessors’ satisfaction with the competency-based assessment (CBA) value (n=8)

Items	Evaluation of his/herself as the assessors	Total score (n, %)a, n=8
		agree completely	agree	neutral	disagree	disagree completely
1	I felt that the students were satisfied with the virtual patient’s attitude.	4(50)	4(50)
2	I felt that the students were embarrassed with the virtual patient’s attitude.	1(12.5)	3(37.5)	1(12.5)	2(25)	1(12.5)
3	I felt that real patient’s attitude should be refracted to the CBA via virtual patients.	5(62.5)	1(12.5)	1(12.5)	1(12.5)
4	I think that experience on CBA was helpful for preparing teaching materials.	5(62.5)	1(12.5)	2(25)
5	I knew what the students need to know as CBA assessors.	6(75)	1(12.5)	1(12.5)
6	I think that CBA is important to assess students’ ability to care for patients	8(100)
7	I felt that CBA is important to improve for students’ communication skill, evaluation competency and problem solving.	7(87.5)	1(12.5)
8	I felt that I should be well prepared myself for CBA as a virtual patient.	6(75)	2(25)

^aData are expressed by the number (percentage). Assessors answered the eight number of survey questionnaires regarding value of the CBA after completion of the assessment. Abbreviation: CBA, competency-based assessment

Fig. 1. Students’ satisfaction with the usefulness of CBA. Over 90.5% of students who participated in CBA agreed or completely agreed that they understood how to apply knowledge to patient-centered care.

Usefulness of faculty assessors

Of the total students, 85.4% agreed or completely agreed that the role of faculty assessors as virtual patients was appropriate. Similarly, 73.8% of the students agreed or completely agreed that the virtual patients were well prepared to help the students gain professional experience. Only 11.9%of the students agreed that the assessors should be trained as virtual patients (Fig. 2). The use of faculty assessors minimized the estimated time, effort, and cost required for obtaining consent to perform the task, execution of the task, and personal identification of virtual patients (Table 4).

Table 4

Comparison of estimated time, administrative assistants, and fee required for faculty assessors and Individual recruited from the community for CBA

Estimated time, administrative assistants, and fee	Virtual patients for CBA
	Faculty assessors	Individual recruited from the community
Time for implementation of CBA test Time for patient selection, identification, and consent process	8 hours per day is required No needed	8 hours per day is required 8 hours per day for 7 days.
Assistants for implementation of CBA test Assistants for patient selection, identification, and consent process	4 assistants are required No needed	4 assistants are required Administrative assistants are required
Fee for virtual patients	No needed	Fee for virtual patients is required

Fig. 2. Students’ satisfaction with the role of faculty assessors. Over 80% of students satisfied with the role of faculty assessors as virtual patients.

This study identified the usefulness and limitations of applying a competency assessment tool and faculty assessors in the development of CBE. The study results showed that the competency assessment tool can identify the students’ level of competency and allows assessors to identify the strengths and weaknesses of students’ competencies,^6,13-14) and develop instructional materials based on the assessment scores to improve the students’ graduation competencies. In the test, students demonstrated excellent competency in the field of collecting patient information and providing drug information while maintaining eye contact; however, they showed poor competency in the field of identifying errors, solving problems, and providing effective regimens. It indicates that more effective learning strategies such as team-based discussion and lectures including application of problem solving and providing effective regimens should be implemented in classes to improve these essential competencies.^4,15) In order to develop the competency-based curriculum, it is important to identify the competency levels of learners and develop educational materials and assessment tools based on these competencies.

In this study, the survey results of students on the usefulness of a CBA tool showed that they could better understand how to apply their knowledge to patient-centered care and were able to effectively communicate with patients.¹⁵⁾ The survey results of the assessors showed that the CBA tool could be a useful learning tool to directly assess the ability of students¹⁶⁾ in managing patient care and improving their communication skills. Therefore, these survey results inferred that the CBA tool was useful in improving individual students’ competencies such as problem solving, communication skill, applying clinical knowledge to patient centered care.¹⁷⁾

CBE is the process of designing classes, developing instructional materials for students and instructors, operating curricula, and assessing students’ performance based on the students’ outcomes using an objective structured system.^16,18) Nevertheless, during the process, the faculty members in charge^2,10,19) have encountered various barriers in operating the curriculum, such as selecting and training virtual patients, in ensuring the validity^1,17) and objectivity of the assessment, and in establishing integrated competency assessment^2,16) due to the lack of individuals with integrated clinical experience who can observe and assess the students’ performance. Moreover, faculty members in charge have difficulties in ensuring the education time for developing faculty assessors, time for developing assessment materials, time for teaching students, and time for re-education of students who do not meet the assessment criteria,¹⁸⁾ hence, the implementation of CBA may require more time, effort, and cost compared with traditional paper examinations.^2,10)

To date, the influence of faculty assessors on CBA has rarely been studied. Therefore, this study evaluated the effect of applying faculty assessors with clinical assessment qualifications in CBA to reduce the time, administrative assistants, and effort related to the operating process. This study demonstrates that using faculty assessors can minimize the time, number of assistants, and effort required for selecting and identifying individual patients, providing consent to perform the task, and executing the tasks, and can reduce the fee for virtual patients.

Based on the survey results of this study on the suitability of the virtual patient’s role, 85.4% of the students were satisfied with the performance of the faculty assessors, while 73.8%agreed that the faculty assessors were well prepared to help the students gain professional experience. This result suggests that the faculty assessors with clinical assessment qualifications may be suitable to reduce the time, administrative assistants, and cost spent on operating CBA.

However, the operating curriculum and assessment are completely different from the existing traditional teaching methods; hence, faculty assessors may not be familiar with this new teaching method¹⁾ and might experience difficulties in understanding the new method. Moreover, it is not easy to ensure the reliability and objectivity of the assessment results as it may vary depending on the level of the test, assessor’s individual assessment criteria, and assessor’s response to each student.^8-9)

To overcome these difficulties in using faculty assessors, educational strategies for understanding new teaching methods should be required.^2,17,19) As an educational strategy for CBA, various strategies must be used to educate assessors on how to perform objective assessment, to train them to think and respond consistently to all pharmacy students like real patients, and to encourage faculty assessors to collaborate¹⁷⁾ using case-based consultation videos. This can help them to better understand the role and response of standardized virtual patients and apply their learning to the competency assessment, thus allowing for more interest in and easy access to the role of the standardized virtual patient to reduce the response deviation that may occur between students,^8,20) even if the assessors lack expertise in performing a case-based competency assessment.^17,19)

Other difficulties include re-education and re-assessment of students who do not meet the assessment criteria after conducting a competency assessment, providing required human resources, and providing rewards to the professor in charge who organizes the curriculum.¹⁶⁾

Despite these limitations, this study demonstrated that CBA, which has a beneficial effect on assessing level of student’s competency,²¹⁾ is required to improve the graduation competency²²⁾ of pharmacy students. Faculty assessors are well prepared by patient’s education to meet the quality of standardized patient for students to gain professional experience. Moreover, the present findings suggest that using faculty assessors who are well prepared for patient roles may help to minimize the time, effort, and cost required in implementing a CBA.

The competency assessment tool has a beneficial effect for instructors in identifying the level of students’ competency. Faculty assessors had a beneficial effect in preparing themselves well for patient roles so that students could gain professional experience. Finally, further research is necessary to evaluate the value of faculty assessors and to what extent the CBA is possible to identify the clinical knowledge and competency of students.

I would like to thank the faculty members of Duksung Women’s University and the adjunct professors who participated in the competency assessment.

This research was supported by the Duksung Women’s University Research Grant 2020, Seoul, Republic of Korea (grant number: 3000005207).

The author has no institutional interest or conflicts.