Evaluation of a training programme to induct medical students in delivering public health talks

INTRODUCTION

Family medicine (FM) is an essential component of the medical education curriculum of medical schools in Singapore. Medical undergraduates are attached to public primary care clinics (i.e. polyclinics) and general practitioner clinics during their clerkships in FM.

In Singapore, family physicians do not only provide medical consultations; they play important roles in health promotion, patient counselling, disease prevention and community health advocacy. For example, family physicians are actively involved in programmes on smoking cessation, vaccinations against endemic infections, and screenings for metabolic syndromes and certain cancers.(1) There are numerous challenges in providing actionable information to the public using conventional media.(2) Barriers include misinterpretation and distortion of the message, and failure to reach out to the target audience. Family physicians can facilitate the process of delivering useful health-related information to the public by holding interactive public talks for the community, which allow the audience’s queries and doubts to be addressed immediately. To hold effective interactive public talks, family physicians need training to equip them with the relevant skills, such as the ability to recognise and select relevant public health topics, as well as the technical expertise needed to organise and deliver talks. However, the subject of organising and delivering public talks is rarely included in the local medical education curriculum, with the exception of the newest medical school in Singapore.(3) Ideally, such training should begin early (e.g. during the undergraduate years), as busy practising family physicians are often pressed for time and unable to personally train students in this area.(4)

SingHealth Polyclinics are among the FM training centres in Singapore certified by the Accreditation Council for Graduate Medical Education.(5) These polyclinics make up one of the two clusters of public primary health care clinics in Singapore. Selected SingHealth Polyclinics provide a platform to train half of the medical undergraduates from the National University of Singapore (NUS) Yong Loo Lin School of Medicine, Singapore. Every academic year, family physicians from SingHealth Polyclinics are appointed as FM faculty staff to precept five cohorts of students during the students’ two-week FM clerkship. Recognising the aforementioned gap in medical education, the FM faculty staff from SingHealth Polyclinics decided to develop a novel training programme to equip medical students with the skills needed to deliver health talks, therefore enabling the students to conduct talks for attending patients and visitors during their FM clerkship.(6) The programme was piloted during the students’ academic years, from June 2012 to March 2013. The objective of the present study was to evaluate the effectiveness of this new training programme, which aims to train medical students to competently organise and deliver public health talks.

METHODS

This study was carried out in four SingHealth Polyclinics. Ethical approval was granted by the SingHealth Centralised Institutional Review Board (reference number: 2013/386/E). The participants consisted of five cohorts of third-year medical students who were posted to any one of the four polyclinics for their two-week FM clerkship between June 2012 and March 2013. Each cohort comprised four groups of 6–7 students. The student cohorts that were not posted to these four polyclinics undertook their FM clerkship in a separate primary care institution.

Each team of 6–7 medical students selected one health-related topic (which was of clinical relevance and importance to the local community) under the guidance and supervision of the FM preceptors. The students reviewed the literature, prepared the content of the talk and delivered the public health talk at the polyclinics. Patients and persons accompanying the patients during their visits to the polyclinic for medical consultations were invited by the students to attend the talk; those who agreed to attend made up the audience for the talk. Each talk lasted approximately 20–30 minutes and included a segment for questions and answers. The FM preceptors were present to answer any queries that the students could not handle and ensure that the students provided the audience with the correct information. Feedback on the talk was obtained from the audience after the session.

The study comprised three segments to evaluate the training programme. The first segment consisted of an anonymous survey of the medical students who were posted to the four designated SingHealth Polyclinics. The survey aimed to determine the students’ level of preparedness, organisation and confidence in delivering public health talks, and the perceived effectiveness and acceptability of the new training programme at the end of the FM clerkship. In the second segment, an anonymous survey of the audience who attended the talks was conducted. The talks were held in education rooms or the patient waiting area outside the consultation rooms in the polyclinic. The audience was surveyed by the medical students using an English-based questionnaire, on a case-encounter basis, after the talks. The survey aimed to determine the audience’s understanding of the talk, their perceived confidence and acceptance of the medical students as speakers, and their willingness to recommend the talks to their acquaintances. In the third segment, video recordings of the talks were reviewed to evaluate the appropriateness of the content, clarity of the delivery, and the speakers’ confidence and their ability to engage the audience and respond to queries. The video recordings were made using mobile phones and evaluated by independent assessors. To ensure consistency, the evaluations were carried out concurrently after the end of the academic year. There were two groups of assessors – one consisted of two medical students who had their FM clerkships in a separate training institution (peers of the cohort who did their FM clerkship in SingHealth Polyclinics), while the other consisted of two FM faculty staff members who were not involved in the coaching of the medical students being assessed. To standardise the evaluation criteria, both groups of assessors were briefed by the lead investigator, albeit separately. Evaluations by the two groups of assessors were conducted independently and the assessors were blinded to each other’s evaluations. Neither the faculty staff nor students were aware of the other assessors’ ratings.

Likert scales ranging from 1 to 10 were used for all the questions in the questionnaires. We regrouped the responses obtained and analysed them in three broad groups – ratings of 1–4 were defined as ‘poor grade’, 5–6 as ‘equivocal grade’, and 7–10 as ‘good grade’. Descriptive statistics were presented for the students’ performance (in terms of preparation and conduct of the talk) and for the audience’s feedback on the relevance of the topic and the speaker’s performance. Shapiro-Wilk test was used to determine whether the data from the videos was parametric or non-parametric. Analyses of the video rating scores by the two medical students, the two FM faculty staff members, and between the medical students and FM faculty staff were performed using Wilcoxon signed-rank test for paired non-parametric data. All statistical analyses were performed using IBM SPSS Statistics version 22.0 (IBM Corp, Armonk, NY, USA). A p-value < 0.05 was considered statistically significant.

RESULTS

The talks conducted by the medical students covered a wide range of topics, from health screening and management of common chronic diseases such as diabetes mellitus and lipid disorders, to cancer screening and home-based exercises. Among the 131 medical students who underwent FM clerkship in the four polyclinics, 120 (91.6%) responded to the questionnaires after their clerkships. Based on those who responded, the majority (80.8%, n = 97) of students felt that they were given adequate time to prepare for the public talks, and almost all of the students felt that the tutor had adequately discussed the selection of the topic with them and provided enough guidance throughout the process (Table I).

Table I

Students’ evaluations of their experience in preparing, organising and delivering a talk (n = 120).

During the preparation process, 97 (80.8%) of the students used well-recognised websites (e.g. the official Health Promotion Board, Singapore, and the Mayo Clinic websites) as their sources of information for the public talks (Table II). Most of the students reported that they had the knowledge (90.3%) and confidence (90.4%) to answer queries from the audience. The vast majority (96.7%) of students perceived that the duration of the talk was appropriate and most of them (95.8%) indicated they had acquired the skill of giving public talks after the session. Overall, most of the students enjoyed the experience of preparing for and delivering the talk. More than nine out of ten students found this modality of training useful in their medical education and would recommend it to future batches of students (92.5% and 90.8%, respectively).

Table II

Top five sources of information students used for the talks (n = 120).

A total of 228 audience participants provided feedback on the talks and student speakers (Table III). Almost half of them (49.1%) found out about the talk from the medical students who were giving them; this finding is expected, as the medical students publicised their own talks. More than half (57.5%) of the audience found that the topic presented during the talk was useful in promoting health to the public. About four out of ten (39.9%) members of the audience perceived that the topic was of relevance to them, and a similar proportion (39.5%) felt that it was a refreshing experience to attend a talk delivered by a medical student. Prior to the talk, only 53.1% of the audience rated their understanding of the topic as good (i.e. a score of 7–10); this percentage rose to 88.5% after the talk. Most members of the audience (82.4%) indicated that the amount of information presented during the talk was adequate, but 20.2% thought the content of the talk needed further improvement. A high proportion (91.2%) of the audience reported that the pace of the talk was appropriate. The audience also perceived the speakers as knowledgeable (53.1%), confident (51.3%) and professional (39.0%). Overall, the audience gave the talks a mean score of 8.13 points out of 10 points. Most of them (87.9%) indicated that they would recommend public talks conducted by medical students to others.

Table III

Audience evaluation of the student-led talks (n= 228).

The investigators received 15 videos from the 18 teams of students (83.3%) for evaluation of the speakers’ performance and delivery of the talks. Table IV shows the results of the evaluation of the video recordings by the independent assessors. No significant difference was found between the ratings of the two student peers, except for the scope of topic (i.e. amount of material covered) and the use of appropriate language. There was no significant difference between the ratings of the two FM faculty staff members, except for the appropriateness of the pace in which the talk was delivered. ‘Relevance of talk content’ received the highest rating from the student peers, while the FM faculty staff rated the speakers highest on ‘clear pronunciation’. Overall, we observed that the FM faculty staff were more likely to give lower ratings for most aspects of the talks as compared to the student peers, except for ‘clear pronunciation’, ‘appropriate pace’ and ‘appropriate language’.

Table IV

Results of the evaluation of the speakers’ performance.

DISCUSSION

The introduction of this novel training module appeared to have had a positive impact on both the students who conducted the talks and the audience who attended the talks. The topics of the talks, which were selected by the students, were largely perceived by the audience to be relevant to community health. This indicates that the students were aware of the major disease burden, public health and primary preventive services in the community. Such talks would help to raise the audience’s awareness of common health issues and common diseases, which could significantly affect their well-being.

In a qualitative study by Hubinette et al, the authors stated that health advocacy is difficult to define, teach, role-model and assess.(1) They suggested the development of a conceptual framework to allow the medical education community to develop novel ways of understanding and engaging in health advocacy.(1) The training programme that was used in the present study has key elements that would enable it to fit into such a conceptual framework. It engages medical students to appreciate the common public health issues in the community and select topics that would have an impact on the target population of their talks. In this way, medical students learn to advocate for health in the community and to use talks as a means of disseminating important and relevant health topics to the public.

Such a training programme also allows medical students to gain direct hands-on experience in organising talks within the spatial and logistical constraints of a busy polyclinic during clinic hours. It increases tutor-student interaction when they discuss initiatives to prepare and publicise the talks using different media, from real-time public announcements to poster design. The majority of the medical students reported satisfaction with their preceptors’ guidance and support in organising the talks. This practical approach appeared to be refreshing and appealing to the medical students, as it is a departure from the usual didactic teaching methods. Based on the survey results, 95.8% of the students reported that they enjoyed the experience.

This programme also allowed the tutors to observe the students’ teamwork and event organisation skills. Teamwork has recently become a focus in healthcare, as it has been shown that teams working together in high-risk and high-intensity work environments, such as in healthcare settings, make fewer mistakes than individuals.(7) Thus, this training programme can also be used as a test bed to assess medical students’ ability to work in teams. Ideally, teamwork assessment should have been incorporated into the present study. However, the science and mechanics of such an assessment for a medical education activity remains uncertain, as most published literature regarding teamwork assessment is based on specific healthcare situations.(8)

In the present study, the students were not required to fill in their names on the questionnaire survey forms to ensure anonymity and encourage truthful feedback. However, as is the case for any research involving surveys and a small number of participants, it may be possible to identify survey respondents from the completed forms and thereby breach anonymity. This possibility was reduced in our study by assigning the task of data entry to designated administrative personnel from the research department of our institution; the administrative personnel did not have direct contact with the medical students.

The audience of the talks also benefited from the programme used in the present study, as attending the talks added value to their polyclinic visits. The results showed that the audience gained additional knowledge after attending the talks and that slightly more than half of the audience felt that the talks were useful in promoting health to the public. However, audience perception of the usefulness of the talks was subjective, as there was a wide variety of topics and they might not have been directly relevant to every member of the audience. Half of the audience also reported that they viewed the students as ‘knowledgeable’ and ‘confident’, although fewer perceived them to be ‘professional’. However, this result was derived from a multiple-response question that contained all three options and some respondents could have only ticked what they thought was sufficient, while other respondents chose all options. Hence, the results could be an inaccurate reflection of the medical students’ qualities.

Student peers were used for the evaluation of the video recordings in the present study, so that we could determine the level of performance assessment expected from the same cohort of students. Such practice is routine in certain academic institutions.(9) The intent was to involve the student assessors in critical reflection and to provide them with better awareness of their own subjectivity and judgement. Wang et al found that that raters inflated their peer scores in selected areas to motivate ratees and tended to accord higher scores to low performers than to high and medium performers.(10) However, Navalta and Lyons showed that performance reviews done by peers were as stringent as those by the teaching faculty.(11) In the present study, although the investigators had concerns about the reliability of peer evaluation, the peers’ appraisal of the delivery of the talks was largely consistent with the faculty’s appraisal. Thus, we opine that peer evaluation should be considered in future studies that involve the evaluation of students’ performance.

To further develop the programme, there are plans for faculties to source for relevant educational media that can serve as references and resource materials for the students in their preparation for the talks (e.g. resources that provide the students with tips on how to enhance their oratorical skills). Such multimedia resources can be viewed at the students’ convenience, after school hours. Following favourable feedback from most of the students in the study, the programme has been adopted as part of the regular curriculum for subsequent cohorts of medical students. New faculty members have been inducted to coach the students on how to organise and deliver health talks, and further research will be conducted to evaluate whether there is long-term retention of the acquired skills.

The present study was not without limitations. We did not have a comparator to benchmark the responses from the audience. Other than that, the ability of the audience to comprehend the talks due to language proficiency and/or educational status was not captured in the questionnaire. The audience’s understanding of the talks would have had direct implications for their perception of the relevance and quality of the talks. In our study, efforts had been made to narrow this gap; some of the student groups delivered the talk in more than one major local language in order to reach out to an audience that consisted of persons of different ethnicities.

The audience’s appreciation of the talks could also have been influenced by the Hawthorne effect, as the students approached the audience face-to-face to gather their feedback.(12) The Hawthorne effect, which is also known as the observer effect, refers to the reactivity of individuals who modify or improve an aspect of their behaviour in response to their awareness of being observed.(11) Nonetheless, the audience had been advised by the medical students to provide truthful responses, as the feedback would help them to improve their performance. The survey was also anonymous; the personal details of the audience (e.g. name, age and education status) were not collected to avoid any source of embarrassment and to minimise the Hawthorne effect.

The video recordings of the talks were done by medical students using handheld mobile phones. As the medical students are not trained videographers, the video recordings may not have captured the speaker-audience interaction in its entirety. This could have restricted the ability of the assessors to carry out a detailed evaluation of the talk.

In conclusion, the present study examined a practical training programme to equip medical students with the skills needed for organising and delivering public talks. Our findings suggest that the programme was effective in helping medical students gain confidence in managing talks and interacting with audiences. The programme has been incorporated as part of routine undergraduate medical education in the NUS Yong Loo Lin School of Medicine. Further studies will be conducted to determine whether the skills gained by medical students during the programme are retained over the long run.