Validation of Questionnaire Designed to Determine Medical Students’ Perspective on the Inexorable Changes in Teaching and Learning Medicine as a Consequence of Covid-19 Pandemic

Introduction

The outbreak of the novel coronavirus disease, COVID-19, with rapid advancement in the disease, has led to a worldwide pandemic, dictating the closure of teaching institutions across the globe. This has resulted in severe impairment of the existing educational curriculum, particularly in medical schools. All face-to-face sessions involving both formal, theory learning and hands-on, such as clinical clerkship and electives, were deeply disrupted, creating many concerns [1-5].

Traditionally, pre-clinical teaching involves little patient interaction, hence the transition did not majorly disrupt learning at our institution. In contrast, the students in the clinical phase lost all opportunities of clinical learning in the hospital and clinic settings and even the face-to-face simulation sessions in clinical skills labs. Clinical exposure is fundamental to medical education thus observation of the practice of medicine in the wards, attending to real patients, practicing critical thinking skills during ward work is crucial to the training of future doctors [1-4].

During the initial outbreak of the coronavirus, both locally and internationally, medical schools were quick to adapt and modify their medical curriculum with regards for the safety of both students and patients alike. Face-to-face sessions were switched to online remote discussions, both synchronous and asynchronous. Clinical postings were all put on hold, with the date of resumption unforeseeable.

Assessment frameworks were adjusted to the pandemic situation, involving mostly online continuous assessments or open book examinations. All face-to-face OSCE examinations were halted. Some universities proceeded with online OSCE via various means, some using online history taking sessions, and some assessing student’s ability to describe findings based on videos [4-6]. This led to reports of feelings of uncertainty and anxiety about personal safety and continuity of learning experience by medical students. There were questions in the student community about how the pandemic would impact their graduation timeline, financial liability, lack competency upon graduation, and housing insecurities [5,6].

During this transition, the main challenges included insufficient electronic devices, lack of technical skills by faculty members to conduct effective online learning sessions and lack of social support leading to academic fallout. During Malaysia’s first Movement Control Order (MCO) on 18 March 2020, many university students returned to their hometowns. This presented its own challenges, mainly unstable internet connection and distractions at home by other family members [5].

All these changes have been a great source of anxiety for many medical students. Whilst there are many articles articulating the perspective of medical schools and faculty members in adapting to this change, few have evaluated the perception of medical students, who are the protagonists in medical education.

The initial aim was to identify the challenges and issues faced by students during this period using a simple survey form. However, the authors’ realized that as the pandemic and the challenges in the medical education were rather novel and available questionnaires did not address current related issues. Hence, we decided to develop and validate a questionnaire that will help our institution and perhaps other medical institutions determine how clinical phase medical students perceive the inexorable changes in medical teaching and learning methods brought by the Covid-19 pandemic and use the information when making changes to curriculum and teaching delivery.

Methodology

Study Design and Research Approach

Questionnaire designing commenced upon identifying the need for a perception questionnaire on changes in teaching and learning during pandemic. Subsequently, literature review search was conducted to identify theories and recent information on the scope of this questionnaire development.

The questionnaire was developed and validated in three phases from 2020- 2021: Phase 1- Needs analysis & planning, Phase 2- Development of questionnaire, Phase 3- Validation and reliability evaluation of the questionnaire. (Table 1) The research was approved by university ethical committee: Project ID: CSc/Sem6(20)2020.

Table 1: Summary of the process in the development and validation of questionnaire.

Phase 1: Needs Analysis and Planning: Phase one involved identification of need for research on students ‘challenges and perception when it was realized that many were facing various challenges learning remotely. A review of the literature focusing on online teaching and learning during a pandemic was carried out. We entered specific terms like online teaching, learning, pandemic, student, and perspective into scholarly databases such as PubMed, Google scholar, Ovid MEDLINE and EbscoHost. Students in the clinical phase who had experienced face to face clinical training prior to the pandemic and had to now switch and adjust to online teaching and learning were included in the study. Hence, pre-clinical medical students who had very minimal clinical exposure have been excluded from the study.

Phase 2: Development of Questionnaire: Information was obtained from students regarding the concerns and anxieties experienced due to the changes imposed on them via email and phone messages. Common themes (psychological, logistics, online learning, and challenges post-MCO) due to the impact on their clinical training were identified and compiled.

A total of four domains with a collective twenty-four items were amassed, titled “Students’ perspectives with the changes from faceto- face to online teaching and learning”, “Perspectives and challenges faced post-MCO face-to-face clinical teaching”, “Perspectives and challenges faced in relation to psychological factors” and “Logistic issues “.

From the identified four domains an initial 24 item questionnaire was designed. Content validity was carried out by contacting 5 experts in the field of academic. Instructions for the entire process were clearly stated. The relevance of each item was judged using a four-point Likert scale:

a) The item is not relevant to the measured domain.

b) The item is somewhat relevant to the measured domain.

c) The item is quite relevant to the measured domain.

d) The item is highly relevant to the measured domain.

The content validity of each item was assessed through the clarity and redundancy of each item’s relevance to the domain. Experts were instructed to write remarks for each item for further clarification. With some suggested minor adjustments, all the 24 items developed were found to be favorable.

Phase 3: Validation and reliability evaluation of the questionnaire:

Settings and participants: The study was conducted online at the International Medical University (IMU), Clinical Campus Seremban. Our sample size was set at a respondent-to-item ratio of 10:1, therefore, a sample size of 240 was planned [7,8].

At the time of the study, there were a total of 363 clinical students in IMU, across 4 different semesters. Out of this student pool, randomization of 60 students per semester was performed, to obtain the targeted sample size of 240 respondents. Students were randomly selected via the RAND function on Microsoft Excel [9]. An email was sent out inviting the selected students to participate in the study.

A total of 213 individuals between the ages of 18-28 years participated in the study, out of which 125 (58.7%) were female and 88 (41.3%) were male. The number of respondents from semester 7 (n = 60, 28.2%) were greater than other semesters (n = 55 ,25.8% semester 8, n = 53, 24.9% semester 9 and n = 45 ,21.1% semester 10). The sample was primarily local Malaysian students (n = 193, 90.6%) with 20 (9.4%) international students. Most respondents (n = 106 ,49.8%) were residing at home with family members whereas 94 individuals (44.1%) were residing with peers, away from home and 13 individuals (6.1%) were tenanting alone.

Data collection and procedure: Data was collected from January to March of 2021. Of the selected respondents of 240, a total of 213 students responded (88.75 %), giving us a respondent-to-item ratio of 8.875:1. Our questionnaire was formatted as a google form. The consent form was included in the e-questionnaires sent to the students’ email. Completing the online questionnaire was taken as consenting to participate in the validation process.

Data analysis: All the raw data was entered into SPSS version 28 for analysis. The reliability was assessed using Cronbach Alpha. Based on external experts’ ratings, the instrument’s content validity was derived by calculating the CVI. The construct validity of the instrument was appraised by using item, factor, and convergent validity analyses. The item analysis involved selecting only those items for which the item–total correlation exceeded 0.3. The Kaiser- Meyer-Olkin (KMO) measure of sampling adequacy and Bartlett’s test of sphericity were used to determine if the correlations were appropriate for factor analysis. Exploratory Factor Analysis (EFA) was then conducted to investigate construct validity by using principal axis factoring with an orthogonal (varimax) rotation to extract factors from the 24 items.

Content Validity Analysis

Content validity was conducted via online by sending an online content validation form to a panel of five experts in which the instructions for the entire process were clearly stated.

The results of content validity are shown below (Table 2). The results showed that item-level content validity index (I-CVI), Scale level content validity index based on Average Method (S-CVI/Ave) and Scale Level Content Validity Index based on Universal Agreement Method (S-CVI/UA) met satisfactory level, hence achieving a sufficient level of content validity.

Table 2: Content Validity Analysis.

Based on the calculations on Table 2, it is concluded that I-CVI, S-CVI/Ave, and S-CVI/UA meet satisfactory levels, and therefore the questionnaire has reached satisfactory degree of content validity except for two S-CVI/UA which was 0.44 and 0.67.

Construct Validity Analysis

Exploratory Factor Analysis

Initial analysis of 24 items revealed 14 items with corrected item-total correlation coefficients greater than 0.3, 4 items had coefficients between 0.2 and 0.3, which were considered to be acceptable for exploratory studies according to Cristobal, et al., and 5 items had coefficients less than 0.2 [10].

After content analysis, the initial 24 items underwent Exploratory Factor Analysis (EFA). Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy generated a middling threshold (.779), indicated that the proportion of variance might cause underlying factor structure, hence an EFA could be conducted. Bartlett’s test of sphericity showed a significant p value (x2=1465.371, p<.001); both these findings ensured that the probability of correlation matrix had significant correlations and was a prerequisite for performing an exploratory factor analysis. As correlation between the factors were not anticipated, thus, orthogonal rotation (namely varimax) was the optimal analysis model, unanimously agreed by the expert panels.

EFA of 24 items resulted in extraction of seven factors from the scree plot (Figure 1) and total variance explained (Table 3), with eigen values between 1.084 and 5.143. The percentage of variance was 21.428 for the first factor whereas the cumulative percentage for all seven factors was 59.510. Factor loading of all 24 items were above 0.3 and Cronbach’s alpha was 0.810. A decision was made by the expert panel to discard one item out of the initial 24 items, due to poor factor loading which was 0.374. The item (A2) was “An online session with a group of less than 10 students makes my online learning better” which had the lowest factor loading (0.374). The team arrived at a unanimous decision to collapse the 4 major themed questionnaire into a unidimensional questionnaire with a single factor. The factor loading for all the 23 items were higher than 0.400. Nine factor loading was above 0.70 indicating well-defined structure (Items A6, A7, B11, C15, C17, C18, C19 D22 and D24). Another 9 items which fell under the practically significant threshold (0.50-0.69) are A4, A5, A8, A9, A12, B10, C16, D20 and D21. Five items fell at the minimum factor loading, i.e., below 0 .40. Overall reliability upon removing A2 is 0.811.

Figure 1: EFA of 24 items resulted in extraction of seven factors from the scree plot.

Table 3: Factor Loading Analysis.

Usability

Usability is also part of developing questionnaires to know how to use and analyze factors obtained from factor analysis as EFA. It depends on the objectives of factor analysis, whether it is univariate or multivariate. Univariate objective is to determine factors related to a construct. To perform bivariate or multivariate analysis will rely on techniques of dependence statistics and the research objectives.

Reliability

Regarding the internal consistency, the value of Cronbach’s alpha for the 24-item questionnaire was calculated to be 0.810. The corrected item-total correlation and squared multiple correlation for all the items are shown in Table 4. All inter-item correlations were moderately correlated. The highest Cronbach Alpha, the itemdeleted value was 0.812, which is approximately 0.810, the overall reliability. Therefore, no items was removed from this analysis.

Table 4: Corrected item-total correlation and squared multiple correlation.

Discussions

The greatest challenge faced in questionnaire development lies in ensuring a valid and reliable tool, with good applicability. Hence, literature on questionnaire development and validation were reviewed [11-14]. Guidelines highlight that there are 3 main phases involved in validating a questionnaire, which are mainly: item development, scale development, and scale evaluation.

In terms of ensuring an appropriate sample size for the questionnaire validation, there are no definite regulations. Guidelines have shown that a good sample size varies from a respondent- to-item ratio of 5:10 to 30:1, but generally the ratio should not be less than 5:10 [7,8].

The questionnaire was designed to gain a better understanding of medical students’ opinions and challenges faced from the changes made to teaching and learning methods during the Covid-19 pandemic. In addition to encountering difficulties in learning activities, the students were also experiencing psychological stress caused by numerous external factors. Medical students, especially those in the clinical phase, had to learn to adapt to new teaching methods. They were introduced to online classes rather than faceto- face sessions that they were used to, albeit for their safety as well as their teachers and peers. It has been found that most higher education students continually improved or changed their learning styles in order to achieve successful outcomes and can adapt their learning preferences to their learning environment [15]. However, continued online sessions however well designed may not address the diverse learning styles and may test the ability of students to adapt. A well-structured questionnaire may be able to identify the mismatch between students’ actual needs and well-intended online curriculum delivery.

Once the MCO was lifted, a limited entry in to wards with strict adherence to standard operating procedures (masks, gloves, a fixed minimal number of students at each time) was allowed. Physical distancing made history taking and physical examination more challenging. There is a need to address the challenges faced by medical students when interacting with patients and the new ‘normal hospital environment’ with many new restrictions. All these changes and the issues with logistics (isolation, food, homesickness, stress of trying to adapt to only online learning and internet glitches) led to mental health disruptions despite mentors being available albeit online and via phone calls. The questionnaire thus was designed to obtain more information in these major areas.

We had response from 213 out of the calculated sample size of 240 students, a response rate of over 80%, therefore the research bias was still acceptable. The study’s reliability was tested using Cronbach’s Alpha. The value was more than 0.80, suggesting good internal consistency.

We specifically included only clinical phase students as they were the ones who were most affected during pandemic and post-pandemic period. Medical students in the pre-clinical phase were thought to be less affected in their learning as most of their learning involved theory though it was problem -based learning. Clinical phase students were required to learn through experience in the wards, clinics and clinical skills and simulation labs. History taking, physical examination, clinical reasoning and management plan of the patients is best learnt and practiced in the clinical setting with the real patients.

Conclusion

The aim of the research was to develop and validate a questionnaire which can be used to assess the perspectives of medical students on the changes of teaching and learning medicine as a consequence of the covid 19 pandemic. The original questionnaire with 24 items was validated to a unidimensional questionnaire, with a total of 23 items. The content validity and reliability indices were established. The questionnaire will enable our institution and others to assess the perception of their adapted curriculum during the covid 19 pandemic and consequently, identify areas for improvement. There is intention for future research including clinical phase students from other universities locally and internationally using this questionnaire.

Limitations

a) The feedback was from only our university, not representing all the other universities in the country.

b) Though there are international students, most are from the Asian continent. There may be differing perceptions in the Western population as culture, learning environment, relationships and support systems are different from Asians.

c) The weightage of online teaching in other universities prior to pandemic may be different and may influence perception and adjustment to the transition from face to face to online.

Statements and Declarations

Conflict of Interest

No potential conflict of interest relevant to this article was reported.

Funding

No funding was received for conducting this study.

Consent to Participate

We used an online questionnaire with consent request that via email and students who consented to participate, completed the questionnaire.

Contribution of authors

TS, YYL, SM, RM, TJY, TJX, SH and SR and were involved in the concept, design, analysis and writing of the script of the study.

This manuscript has been approved by all the authors, that the requirements for authorship have been met, and that each author believes that the manuscript represents honest work.

Availability of Data and Material

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Ethical Approval

The proposal was presented, and approval was obtained from International Medical University Joint-Committee on Research and Ethics. Project ID: CSc/Sem6(20)2020.

The authors declare no conflicts of interest with the conduct of this study.

References

• Ahmed H, Allaf M, Elghazaly H (2020) Covid-19 and medical education. Lancet Infect Dis pp. 777-778.
• Rose S (2020) Medical Student Education in the Time of COVID-19. JAMA 23(21): 2131-2132.
• Rashid AA, Rashid MRA, Yaman MN, Mohamad I (2020) Teaching Medicine Online During the COVID-19 Pandemic: A Malaysian Perspective. Bangladesh J. Medical Sci 19: 77-81.
• Nik-Ahmad-Zuky NL, Baharuddin KA, Rahim AFA (2020) Online Clinical Teaching and Learning for Medical Undergraduates during the COVID-19 Pandemic: The Universiti Sains Malaysia (USM) Experience. Education in Medicine Journal 12(2): 75-80.
• Wong RY (2020) Key lessons learned during COVID-19 at the UBC Faculty of Medicine. BC Med J 62(5): 170-171.
• Gallagher TH, Schleyer AM (2020) We Signed Up for This!- student and trainee responses to the Covid-19 pandemic. N Engl J Med 382: 96.
• Memon MA, Ting H, Cheah JH, Thurasamy R, Chuah F, et al. (2020) Sample size for survey research: review and recommendations. JASEM 4(2): 1-20.
• Nunnally J (1978) Psychometric Theory. New York: McGraw-Hill.
• Asghari-Jafarabadi M, Sadeghi-Bazargani H (2015) Randomization: Techniques and Software-aided Implementation in Medical Studies. J Clin Res Govern 4.
• Cristobal E, Flavián C, Guinalíu M (2007) Perceived e‐Service Quality (PeSQ): Measurement validation and effects on consumer satisfaction and web site loyalty. Managing Service Quality: An International Journal 17(3): 317-340.
• Lee Y, Seomun G (2016) Development, and validation of an instrument to measure nurses' compassion competence. Applied Nursing Research 30: 76-82.
• Boateng GO, Neilands TB, Frongillo EA, Melgar-Quiñonez HR, Young SL (2018) Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer. Front Public Health 6: 149.
• Tsang S, Royse CF, Terkawi AS (2017) Guidelines for developing, translating, and validating a questionnaire in perioperative and pain medicine. Saudi J Anaesth 11(Suppl 1): 80-89.
• Yusoff MS, Arifin WN, Hadie SN (2021) ABC of Questionnaire development and validation for survey research. Education in Medicine Journal 13(1): 97-108.
• Bokhari NM, Zafar M (2019) Learning styles and approaches among medical education participants. J Educ Health Promot 5(1): 181.