A controlled trial was designed to measure the effectiveness of a group of 48 internet CME activities. Physicians who participated in these activities, matched the target audience for the activity, and completed case vignette self-assessment questions following participation were eligible to participate. A random sample of participants meeting the eligible criteria for each activity was drawn from each overall group. A random sample of non-participant physicians of similar specialties was identified as a control group and was asked to complete the same self-assessment questions. The average evidence-based response rates were calculated for the participant and non-participant samples for each activity, and an effect size was calculated. An overall effect size was calculated, as well as effect sizes for text and case-based activities, and for primary care and specialist participants.
A consistent assessment approach was developed that included 1) using case vignettes to assess clinical practice choices, 2) using a standard hypertext mark-up language programming approach to presenting assessment questions at the end of selected internet activities, 3) applying this assessment approach to specific content reflected in each individual activity, 4) collecting assessment data from CME participants in each individual clinical assessment, 5) collecting assessment data from a comparable group of non-participants in each of the assessments, and 6) analyzing the data to determine the amount of difference between the CME participant and non-participant groups by calculating effect size and the percentage of non-overlap between the two groups. The use of case vignette surveys was reviewed by the Western Institutional Review Board in 2004, prior to initiation of this study; voluntary completion of the survey questions by physicians was considered to constitute consent.
During 2005, a pilot was conducted on three internet CME activities to test a standardized evaluation procedure, and the use of standard hypertext mark-up language (HTML) online forms, for the purpose of systematically gathering clinical case vignette assessment data from physicians following participation in internet CME activities posted on a large medical education site. The pilot was designed to determine the technical feasibility of gathering and transferring large data sets using a standardized evaluation approach; the pilot was not designed to evaluate the effectiveness of the three internet CME activities. The standardized evaluation procedure included the following elements. A standard assessment template consisting of two clinical vignettes and five clinical questions was developed using a multiple choice format; evidence-based responses to the case vignettes were identified from content and references developed by the faculty for each activity. Content for the activities was written and referenced to clinical evidence by the faculty member for each activity. Only content referenced to peer-reviewed publications or guidelines was considered eligible for the development of clinical vignette assessment questions. Case vignettes were written by physicians and were referenced to the content and learning objectives. Content validity of the case vignettes was established by review from medical editors of the online portal; editors represented the appropriate clinical area for each set of case vignettes.
Case vignette evaluations were developed for the three pilot activities according to this procedure. Over 5000 physicians participated in the pilot activities. Data collection and transfer was successful; no technical glitches were identified in data collection using the HTML online forms or in the data transfer. This feasibility pilot established the processes for development and review of case vignette questions, as well as the technical platform for proceeding with the evaluation of the effectiveness of a series of 48 internet CME activities.
During an 18-month period, a group of internet CME activities was identified as eligible for assessment if the activity met the following criteria: 1) designed for physicians, 2) posted during an 18 month period between January 2006 and June 2007 to a large medical education website, 3) certified for CME credit, 4) presented in an on-demand archived format (webcasts and other live activities were not included), and 5) designed in a text-based format for clinical updates or as interactive case-based activities.
Text-based clinical update activities were defined as original review articles on scientific advances related to a particular clinical topic, similar to a written article in an internet journal. Interactive cases were original CME activities presented in a case format with extensive questions and feedback within each activity. Typically, they began with a short explanatory introduction and then presented the content within the context of a patient care scenario with discussion of diagnostic and therapeutic options and outcomes. Questions distributed throughout the activity provided interaction for learners to test their knowledge on either the material that was just presented, or for upcoming content. After submitting a response, the learner was presented with an explanation of the optimal answer, as well as a summary of the responses of past participants. There was no direct learner-instructor or learner-learner interaction in either of these formats.
The case vignette survey template consisted of a set of content-specific, case vignette questions that were delivered to participants at the conclusion of each CME activity. They were also distributed in a survey, by email or fax, to a similar non-participant group. This method was chosen as an adaptation for an online format with automated data transfer of the case vignette assessment method that has been recognized for its value in predicting physician practice patterns; results from recent research demonstrate case vignettes, compared with other processes of care measures such as chart review and standardized patients, are a valid and comprehensive method to measure a physician's processes of care [20, 21].
A sample size of at least 4800 with at least 100 (50 participants and 50 non-participants selected as a desired minimum sample size for individual activities) for each of the CME activities was chosen for the study in order to establish consistency in data collection even though content varied across multiple clinical areas. Participants were eligible for inclusion in the study only if they represented the specialty target audience for the activity, or were providing primary care. Eligible participants were identified for each activity, and a random sample of 50 was drawn from the group of eligible participants. Non-participating physicians were identified from a random sample drawn by specialty from the physician list of the American Medical Association. Participant and non-participant samples were matched on the following characteristics: physician specialty, degree, years in practice, whether or not direct patient care was their primary responsibility, and the average number of patients seen per week.
A statistical analysis software package (SAS 9.1.3) was used in data extraction and transformation, and statistical analyses. Participant and non-participant case vignette responses were scored according to their concordance with the evidence-informed content presented within each activity. Overall mean scores and pooled standard deviations were calculated for both the participant and non-participant groups for each of the activities. These were used to calculate the educational effect size using Cohen's d formula (i.e., the difference in mean divided by the square root of the pooled standard deviation) in order to determine the average amount of difference between participants and non-participants [22]. Effect size representing the difference between the two groups was expressed as a percentage of non-overlap between participants and non-participants. The amount of difference between participants and non-participants in the likelihood of making evidence-based clinical choices in response to clinical case vignettes was expressed using the percentage of non-overlap between participants and non-participants for each activity, and for the overall group of activities.