A New Approach to Analyzing the Development of Domain-Specific Knowledge Among Undergraduate Medical Students Using Learning Scores

Medicine is one of the domains in higher education with the highest demands [1]. The importance of acquiring domain-specific knowledge to understand the scientific rationales of practical work in medicine is considered vital [2], about the specific responsibilities associated with medical professions. Teachers in medicine are faced with specific instructional challenges, for instance, being required to teach and examine several hundred students with heterogeneous educational backgrounds and study preconditions [3]. Consequently, in medical education practice, Multiple-Choice (MC) tests are often used in examinations because of their high level of efficiency and practicability, and despite their well-known disadvantages such as construct-irrelevant bias and low levels of explanatory power [4-5]. In particular, the extent to which teaching contributes to the development of domain-specific knowledge cannot be measured by simple post-testing with MC tests and analyzing test sum scores only [6-7]. Thus, practical implications for teaching and learning in medicine can only be derived to a very limited extent. In this paper, we present a new approach to analyzing the development of medical knowledge and suggest that a design with preand post-measurements and the consideration of decomposed preand posttest scores, which we define as learning scores, can provide substantial additional information on medical students’ learning over the course of their studies. This practicable approach can help to inform educational practitioners about students’ difficulties in learning and understanding certain medical contents, as well as to uncover possible student misconceptions about medical concepts and models.


Introduction
Usually, the development of domain-specific knowledge is measured by comparing two measurement points or by using a difference score. However, this approach does not sufficiently reflect the learning process and it is difficult to derive implications for teaching practice. In some studies, an alternative approach has been presented, in which knowledge development can be divided into different types of learning: positive (knowledge gain), negative (knowledge loss), retained (stable knowledge) and zero learning (no learning) [8][9]. However, this approach is also limited to domain-specific tasks where distractors cannot be classified as false or correct with absolute certainty. This is often the case in tasks representing specific medical content.
Learning and developmental psychology research suggests that learning processes occur in a non-linear fashion by crossing thresholds [10][11][12], which also includes the acquisition and rejection of misconceptions [13]. Thus, tasks for validly assessing acquired domain-specific knowledge should be more appropriate to learning scores enable the consideration of different learning patterns that go beyond the previously mentioned established approaches.

A New Approach to Learning Scores
The approach presented here offers an innovative complementary method for analyzing pretest and posttest responses from an MC test to assess medical students' learning processes, in this case based on an example from physiology. Using an innovative score decomposition approach, the seven identified student response patterns ( Table 1) provide indicators of four student learning outcomes that, for the purposes of this study, are described as completed learning, progressive learning, incomplete learning and interrupted learning. The decomposed learning scores can be used to analyze how they affect the MC test scores and for investigating factors at the item and student level that are likely to influence students' understanding of concepts in physiology.  Figure 1). The learning patterns can be divided into categories as indicated in Table 1. In our study, we focus on the following two research questions (RQ): I. RQ1: How can the learning process be described regarding the development of domain-specific knowledge between both measurement points-before (t1) and after (t2) attending the physiology lectures when using our decomposed approach?
II. RQ2: Which learning patterns become evident at the individual and item level?
In addition, based on our investigation in the study reported here, we particularly aim to explore which implications for medical educational practice can be drawn from the student learning scores in terms of improving teaching and learning.  To measure the students' use of learning sources, two scales were used at both measurement points to determine the frequency of use of 11 media sources for learning adapted for this study, for an original version, Maurer et al. To evaluate how frequently students use these sources, the participants were asked to respond on a scale from 1 (never) to 6 (multiple times a day) to the two questions "In your last semester at university, how many times a week did you use different types of media to find information about physiology topics/to prepare yourself for lectures and exams in physiology?".

Am J Biomed Sci & Res
They had the option of adding additional sources if applicable.
As control variables, the scale on information overload the scale on need for cognition as well as two further scales were used at t2 to measure the processing of information over the course of studies. Heuristic and systematic information processing were measured with three items each, using a 6-point scale from 0 ("does not apply at all") to 5 ("does fully apply"), validated by Schemer et al. and following the Risk Information Seeking and Processing model [18,19].

Statistical Procedure
After testing the preconditions of one-dimensionality of the domain-specific test in physiology used in this study (by using confirmatory factor analysis) for the two types, learning progression and learning regression, the analysis of the knowledge sum score of t1 and t2 as well as the testing of a mean value difference via a t-test provided first insights into the development of knowledge from t1 to t2. To investigate underlying learning processes, learning patterns were generated (as described above) and then evaluated at the individual and item level.
The development from t1 to t2 was first generated itemspecifically. For the individual-centered analysis, mean scores were then generated for the different learning patterns. Multiple regressions were performed to analyze possible predictors of the learning scores, whereby measured cognitive traits (such as intelligence) were integrated in addition to socio-demographic variables (such as gender). The analyses of the research questions were carried out with Stata Version 15 [20], while MPlus Version 7 [21][22][23] was used to assess the students' preconditions and [24] latent characteristics.

I. RQ1
: How can the learning process be described regarding the development of domain-specific knowledge between both measurement points-before (t1) and after (t2) attending the physiology lectures when using our decomposed approach?
Thanks to the expert rating of the domain-specific test tasks, it became evident that for about half of the tasks, for instance, tasks where knowledge about physiological facts is required, a clear response should be expected, so that the experts recommended adhering to the binary rating-0 (wrong response), 1 (right response). For the remaining test tasks, for instance, with numbers or ranges of values in the distractors, where partial scoring was theoretically possible, the scoring was performed based on the empirical data. In these tasks, the expected learning patterns for the two types, learning progression and learning regression, could be differentiated in this sample [25,26].
II. RQ2: Which learning patterns become evident at the individual and item level?
Significant differences in learning scores could be found at both item and student level. At the item level, for instance, there was one item in which most students evidently had difficulties understanding the underlying physiological concept and where learning scores showed a high value of "interrupted leaning". At the individual level, differences in learning scores about gender were found, indicating more learning patterns in the area of learning progression among female students compared to male students [27,28].

Conclusion
The The limitations of the approach lie in its applicability to only specific medical content, i.e. tasks where a partial scoring is theoretically meaningful. When developing a domain-specific test in medicine, an expert rating would have to be carried out to enable objective partial scoring and to examine the interrater agreement to ensure reliable measurement. In addition, further longitudinal analyses with more than two measurement points are strongly recommended to carry out a more comprehensive validation following the internationally established Standards for Educational and Psychological Testing [24]. The longitudinal data would also allow for the application of more sophisticated statistical analyses such as using cognitive diagnostic modelling. Based on the preliminary results, further work on the theoretical-conceptual analyses such as definition and description of thresholds for the different medical concepts is also required in future studies.