Abstract

INTRODUCTION

The educational environment (EE) reflects the quality of a residency programme and has an association with burnout. Studying the EE allows for interventions to target specific weaknesses. We aimed to measure the EE of an internal medicine residency programme in Singapore, compare the perceptions between genders, residency grades and levels of work experience, and identify specific areas of weaknesses for intervention in hopes of reducing residency burnout rates in Singapore.

METHODS

This study took place between October and December 2017. We adopted a mixed methods approach, quantitatively using the Postgraduate Hospital Educational Environment Measure (PHEEM), and qualitative exploration using semi-structured focus group discussion.

RESULTS

A total of 136 (88.9%) out of 153 residents responded. Our total PHEEM scores (112.23 ± 16.71), along with the scores for all three subscales, were higher than those of institutions in previous studies. There were no differences in overall PHEEM and subscale scores between genders, residency grades or levels of work experience. However, there were differences for individual questions, which were explored in the focus group discussion. Senior residents juggling heavier workloads, responsibilities and examinations appeared to be most prone to burnout. We identified three recurring themes that contributed to a poor EE in our programme: excessive workload, poor faculty relationships and differing unmet needs.

CONCLUSION

Although our programme had a good EE, there were also areas of weaknesses revealed by specific questions, possibly contributing to burnout. We hope to implement interventions to these areas and subsequently assess for longitudinal changes in EE and burnout rates.

 Keywords: burnout educational environment learning climate PHEEM Singapore

INTRODUCTION

The educational environment (EE) is an indication of the quality of a curriculum.(1) Recognition of the importance of EE led the Accreditation Council for Graduate Medical Education (ACGME) to implement the Clinical Learning Environment Review (CLER) programme as part of its next accreditation system.(2) There is evidence that EE influences student career choices(3) and achievements,(1) and likely has a greater impact on postgraduates due to the complex relationships between doctors and the constant tensions between service and training.(4) A positive EE has also been associated with better workplace learning,(5) better examination scores(6) and more career satisfaction.(7,8) EE has been shown to influence burnout rates,(9,10) and this association holds true even after controlling for specialty, level of training, gender and age.(11) This is of interest to us, as a recent study has shown that Singapore residents have higher burnout rates compared to their Western counterparts.(12)

Therefore, the aim of our study was to measure the EE of an internal medicine residency programme in Singapore, compare the perceptions among genders, residency grades and levels of working experience and, in so doing, identify specific areas of weaknesses that may potentially contribute to high burnout rates.

METHODS

Singapore engaged ACGME-International (ACGME-I) to manage the accreditation of its postgraduate training in 2009, and was the first country to successfully receive ACGME-I accreditation in 2010. There are three health clusters in Singapore, National University Health System, National Healthcare Group and Singapore Healthcare Services (SingHealth), each with its own residency programmes. Our programme, SingHealth Internal Medicine Residency Programme, is the largest residency programme, consisting of around 150 residents spread across three years of training. Training takes place at three main sites: Singapore General Hospital, Changi General Hospital and National Neuroscience Institute. The programme admits both undergraduates directly from medical school and postgraduate doctors who have already joined the workforce.

The current study was exempted from formal ethical board review, as no patients were included; there was also no risk to participants and anonymity was maintained. We used a mixed methods approach, including quantitative measurement using the Postgraduate Hospital Educational Environment Measure (PHEEM)(13) and qualitative exploration of PHEEM data using focus group discussions.

The PHEEM(13) consists of 40 items that are divided into three subscales: perceptions of role autonomy; teaching; and social support. It has been administered to different sample groups and various specialties, demonstrating almost similar reliability coefficients. Therefore, it was deemed the most suitable instrument for measuring EE in postgraduate medical education due to its content validity, high reliability and ability to be used in different settings.(14)

Although we made slight modifications to the PHEEM, as some questions were inappropriate, the changes were unlikely to alter the resident’s perspective on these questions. Examples of changes included Question 11 where ‘bleeped’ was replaced with ‘received phone calls’ and Question 17 where ‘new deal’ was replaced with ‘ACGME requirements’. We administered the PHEEM online to 153 active Internal Medicine residents between October and December 2017. We included questions on gender, work experience, residency grade and training sites, and a free-text section where residents could write their narrative feedback. To maximise the response rate, residents received up to three reminders. Participation was voluntary and anonymity was assured.

A semi-structured focus group session was conducted to supplement the PHEEM results. Purposeful sampling of residents (five female residents; two first-year, three second-year, three third-year residents) was done to ensure heterogeneity of participants for different perspectives, and a group size of eight was chosen, as it has been previously suggested to be the optimal number.(15) The PHEEM results were presented to the group and used to influence the question route. The main theme was to understand the factors involved in poor perception of the EE contributing to burnout. The session lasted 75 minutes and was conducted by the first author (Ong AML). Consent was obtained from the participants for audio recording and verbatim transcribing of interviews.

A summary of our analysis was made available to the interviewees to check the accuracy of the summary. This ‘member checking’ did not yield any revisions. Two authors (Ong AML and Fong WWS) used an open coding strategy to reduce the data and uncover the basic concepts. Similar concepts were grouped together to form themes. Differences in concepts were resolved through discussion between the two authors. We adopted an inductive approach, using quantitative data from the PHEEM analysis, open-ended narrative feedback from questionnaires and focus group opinions to triangulate concepts and draw conclusions.

Data was analysed using IBM SPSS Statistics version 23.0 (IBM Corp, Armonk, NY, USA). Data was presented as mean and standard deviation for quantitative data, and proportions for categorical data. Unpaired Student’s t-test was used for parametric quantitative data comparisons and Wilcoxon signed-rank test for nonparametric quantitative data. Analysis of variance with post-hoc Bonferroni corrections was used to analyse comparisons between multiple groups. Differences were considered significant if p < 0.05. Reliability analysis was performed using Cronbach’s alpha coefficient.

RESULTS

A total of 136 (88.9%) out of the 153 residents responded. The characteristics of the respondents are shown in Table I. Cronbach’s alpha was 0.95, and when analysed to exclude each question, no significant improvement in the score was obtained, reflecting high internal reliability and no irrelevant questions.

Table I

Characteristics of the study population (n = 136).

Table II summarises the mean responses to each question. Our mean total PHEEM scores (112.23 ± 16.71), along with the scores for all three subscales, were higher than those of other institutions that have used the PHEEM (Table III).(6,16-22) Based on the recommended scoring,(13) our programme was deemed more positive than negative, with room for improvement. No resident scored the EE as very poor, while 3 (2.2%) residents scored it as having many problems, 97 (71.3%) as more positive than negative, and 36 (26.5%) as excellent.

Table II

Summary of PHEEM results.

Table III

Comparison of PHEEM scores.

The lowest-scoring items were: Question 11, ‘I receive phone calls inappropriately’ (mean 2.10 ± 1.02); Question 17, ‘My working hours conform to ACGME requirements’ (mean 2.13 ± 1.11); and Question 26, ‘There are adequate eating facilities when I am on call’ (mean 2.23 ± 1.09).

There were no differences in overall PHEEM and subscale scores between genders (Appendix, Supplementary Table I). However, female residents perceived that they had fewer opportunities to acquire practical procedures (p = 0.03). There was also a trend for female residents to perceive that they had fewer clinical learning opportunities and to derive less enjoyment from their job, but these were not significant.

In terms of residency grades, there were no differences in overall PHEEM and subscale scores (Appendix, Supplementary Table II). However, there were differences for individual questions. First-year residents (R1) were less clear about clinical protocols (p < 0.01), and perceived that there were fewer relevant educational programmes available (p = 0.01) and less feedback given by seniors (p = 0.01), and that their teachers were less organised (p = 0.02), less able to teach (p = 0.05) and ineffective at utilising learning opportunities (p = 0.03).

Similarly, although no differences were seen in overall PHEEM and subscale scores between the levels of work experience (Table IV), there were differences for individual questions. Residents who were in postgraduate year (PGY) 3–5 perceived their workload to be heavier (p = 0.03), although the very experienced (> PGY5) residents were less likely to have the same perception. PGY5 residents were more likely to perceive: poorer collaboration with other doctors (p = 0.04); their teachers to be less respectful (p = 0.03); and a blame culture (p = 0.01). Again, the very experienced residents (> PGY5) did not share the same perceptions.

Table IV

Comparisons of PHEEM score between levels of work experience.

Based on our analysis of the focus group discussion and free-text narrative feedback, we identified three recurring themes that contributed to poor EE in our programme and illustrate these themes using specific quotations from the residents. First, excessive workload often resulted in a lack of protected educational time for residents and frustration that while they were forced to log educational hours, they were often not able to attend these activities or experienced disruptions when attending them.

  • Resident 7: “Other hospitals have a lot of focused teaching for exams, they have afternoons off with dedicated teaching from 2–5. We don’t have that.”

  • Resident 2: “Even though I physically attend, my mind is not here because I have a lot of work. So I go there just to sign my attendance, or else I get an angry email.”

  • Resident 1: “Daytime hours are difficult no matter how you say it is protected. Nobody respects the protected timing.”

    • Furthermore, the excessive workload and the repetitive nature of the job, for the more senior residents, made them feel jaded and discouraged.

  • Resident 7: “As you work more, you get jaded… you feel whatever you are doing isn’t recognised, not appreciated, and you feel like you are a cog in the system…. Many are just doing the bare minimum to survive, so that they can have a balanced life. So why am I trying so hard?”

  • Resident 5: “Imagine you are PGY 8 or 10, and still doing discharge summaries, same as what house officers are doing.”

    • Another recurring theme was the infrequent contact and poor relationship with residency supervisors and ward consultants, resulting in poor feedback.

  • Resident 6: “The mentors attached to your posting, I hardly talk to any…. The mentor-mentee thing is just for show.”

  • Resident 1: “With every posting, you are supposed to have a supervisor. I’ve only one such supervision…. I only met my residency supervisor halfway into my residency. Some have never met their supervisors.”

    • Residents were often given grades below their expectations, but how the grades were determined was not made clear to them, since there was minimal feedback.

  • Resident 8: “They tell you feedback that is not timely, not specific and no ways to improve yourself.”

  • Resident 6: “It wasn’t as good as I thought it would be…. What did I do wrong?… I’m working my butt off…. If you think I’m just average, at least say what else I can improve on.”

    • Residents with varying levels of experience had different perceptions and expectations of the environment. Junior residents were more critical of teaching programmes, while senior residents were more focused on efficient postgraduate exam preparations.

  • Resident 3: “The house officers probably think they have less access to educational programmes when they compare it to medical school.”

  • Resident 7: “When you are R3, you are concerned about clearing exams. So you want focused and effective teaching… especially in a busy posting and your teaching opportunities don’t help you to clear exams, it is an issue.”

    • A specific concern was the pressure on senior residents as they were receiving unfair workload distributions and responsibilities, along with the pressure of examinations. Therefore, it was highlighted repeatedly that this group was likely to burn out.

  • Resident 5: “There isn’t a culture of having to complete your exams by a certain time point in Australia…. Here, that actually leads to high burnout rates in more senior residents. You have to juggle your work and after work, you stay late practising for exams.”

  • Resident 1: “Junior doctors don’t feel it because if you didn’t know anything, you just escalated it, and you shifted the responsibility from yourself to somebody else…. So they don’t feel the blame or responsibility.”

  • Resident 3: “There are three main things. Expectations, responsibilities and age. The more senior you are, the better your work must be. Your senior expects more from you and it is quite marked. Responsibilities also. There is consent taking, giving sedation, which a PGY1 cannot do. Also, the senior residents also have research going, and the seniors also ask them to do more because they are more familiar with them. And with age, the mentality is that you have done it for so long, you are tired.”

    • DISCUSSION

    As the largest residency programme in Singapore, we found that our EE was perceived by our residents as being more positive than negative, but with room for improvement. Compared to other institutions worldwide, we had the highest overall PHEEM score and the highest scores in all three subscales (Table III). Although overall PHEEM and subscale scores did not differ between gender, work experiences and residency grades, we found significantly different responses to specific questions within these groups. Our qualitative analysis also identified three recurring themes contributing to a poor EE, namely excessive workload, poor relationship with faculty and differing unmet needs. These three themes will likely form the basis of interventions to improve the EE.

    One of the worrying yet unsurprising results of the PHEEM analysis was that two of the lowest-scored items among residents were Question 11, ‘I receive phone calls inappropriately’, and Question 17, ‘My working hours conform to ACGME requirements’. Residents consider balance between work, training and personal needs to be conducive to a positive EE, and that service obligations are detrimental to their learning needs.(23) However, the reality is that residents often face tension between work and learning, and constant disruptions impact the resident’s focus on learning, especially the senior residents who have to juggle examination commitments. This is an impetus for programmes to discuss phone call protocols with nursing staff, work with individual departments to protect educational time and provide examination preparation resources. Other potential solutions include ensuring equal distributions of senior and junior residents among medical teams, or finding creative ways to incorporate educational activities into service obligations.

    Schönrock-Adema et al have highlighted that interaction and collaboration with others was important in determining the EE,(24) while Boor demonstrated that good supervisory strategies contribute to a good EE.(23) In our focus group discussion, a recurring concern was that poor relationships with faculty were resulting in a poor EE, specifically inadequate supervisor contact and minimal feedback given. This problem is not unique to our programme, as many institutions looking into EE also found that residents had limited access to their supervisors.(25) Increasing educational time without increasing time spent on supervisor-resident contact still reduces the measured EE,(26) thus emphasising the importance of the supervisor-resident relationship. A systematic review(27) of EE interventions showed that faculty mentor programmes were highly regarded by students as a method of reducing burnout. These results reinforce the fact that residents yearn for appropriate supervision and feedback, and therefore, clinical educators need to constantly remind themselves about their other roles apart from that of transmitting knowledge.(28,29) Faculty development programmes need to focus on developing the faculty’s pedagogical skills for delivering feedback and effective supervision, as well as considering novel ways of mentoring, such as a small group-based mentoring model(30) or a faculty advisory programme(20) focusing on mentoring support.

    There are conflicting results in the literature of PHEEM differences based on training levels. Some studies(17,18,31) showed that junior trainees have better perceptions of the EE, while others showed that senior trainees had better perceptions.(16,22,32) The conflicting results are perhaps due to the rigidity of programmes that try to deliver the same educational activities to residents of varying levels; this is true of our programme, which admits both undergraduates and postgraduate doctors. Our PHEEM analysis showed that experienced residents were significantly more likely to have these perceptions: a heavier workload; poorer collaboration with other doctors; their teachers being less respectful; and a blame culture. Our focus group discussion affirmed these results and suggested that the more experienced residents are also the most likely to experience burnout due to performance pressures from senior doctors and the need to juggle examinations. The focus group discussion also suggested that different residents may perceive the EE differently due to differing unmet learning needs. A senior resident may be burdened with excessive workload that is boring and repetitive along with heavier responsibilities, while a junior resident is concerned about assimilating medical and practical knowledge, and therefore more critical of teaching quality. This finding of differing unmet needs of residents was also reported in other studies looking at factors that facilitate residents’ learning.(23,33,34) The implication is that we should consider creating flexible training programmes to empower residents to choose learning activities relevant to their stage of training or experience, or offer flexible rotation plans that expose residents to opportunities beyond clinical work (e.g. education or research).

    In terms of gender and training sites, there were some differences in responses to individual questions. Most studies on PHEEM did not show differences between genders,(25) but some revealed poorer scores for questions pertaining to gender discrimination.(18,21) Our focus group discussion suggested that this could be due to personality differences between genders, as women were less assertive in getting opportunities for procedures, but that there was no gender discrimination within the faculty. We found that the main differences among training sites were organisational issues such as poor catering and accommodation, along with receiving calls inappropriately. This was mainly for residents in Singapore General Hospital, the busiest hospital in Singapore. Although each training site has different infrastructure and service obligations, such information should be transmitted to the relevant authorities so that any interventions to improve the EE can be holistic.

    Our response rate of 88.9% was high, providing adequate sampling of the EE in our programme. Quantitative analysis alone would not have identified many of our weaknesses, such as poor relationships with faculty. Using a mixed methods analysis made our findings more robust and gave us more confidence in our conclusions. The qualitative data allowed us to explore ill-defined concepts that are not easily explained by quantitative data. However, there were several limitations to our study. First, we only analysed residents within a single health cluster, which may limit the generalisability of our data. Second, we sampled only eight residents in one focus group due to logistical reasons, and therefore, were unlikely to have achieved data saturation; however, the purpose of our qualitative analysis was to explore findings in the PHEEM rather than draw conclusions solely via qualitative analysis. Third, the study was cross-sectional and hence could not establish causal relationships between factors. Finally, a single researcher who was a programme faculty member conducted the focus group, which could have affected its course; however, this effect was reduced by the use of a directive moderating style.(15)

    Even though our programme had the highest PHEEM scores available in the literature, a local study has reported that the burnout rates and empathy levels of Singapore residents were lower than those of Western residents.(12) A possible reason for this phenomenon is that Singapore residents are generally satisfied with their educational environments despite their struggle to cope at work. Another explanation is that we used a single instrument and thus, the domains measured in the PHEEM questionnaire may not illuminate the complex issue of burnout; this is yet another limitation of the study. However, our own experience is that burnout is a real issue within our programme, as well as a major concern, because of the consequences for patient outcomes and the residents themselves.(11) There is a hypothesis that burnout is rooted in issues that are related to the EE.(11) In support of this theory, a meta-analysis has shown that organisation-directed approaches are more effective in reducing burnout compared to individual interventions, such as mindfulness training.(35)

    In conclusion, the current study identified several weaknesses that contributed to the high burnout rates among our residents: excessive workload; poor relationships with faculty; and differing unmet learning needs. We hope to use this information to implement future interventions and subsequently repeat the measurement of the EE to assess longitudinal changes. In so doing, we endeavour to reduce our residency burnout rates.

    ACKNOWLEDGEMENTS

    The authors wish to express their gratitude to the residents who were involved in the focus group discussion, as well as all the programme executives of the SingHealth Internal Medicine Residency Programme for their administrative efforts in this study.

    Supplementary Material
    SMJ-61–476-Appendix.pdf

    References
    Genn JM.AMEE Medical Education Guide No.23 (Part 1):Curriculum, environment, climate, quality and change in medical education - a unifying perspective. Med Teach. 2001;23:337-44.
    Weiss KB, Wagner R, Nasca TJ.Development, testing, and implementation of the ACGME Clinical Learning Environment Review (CLER) Program. J Grad Med Educ. 2012;4:396-8.
    Mahendran R, Lim HA, Verma S, Kua EH.The impact of the educational environment on career choice and attitudes toward psychiatry. Med Teach. 2015;37:494-7.
    Swanwick T.Postgraduate medical education:the same, but different. Postgrad Med J. 2015;91:179-81.
    Delva MD, Kirby J, Schultz K, Godwin M.Assessing the relationship of learning approaches to workplace climate in clerkship and residency. Acad Med. 2004;79:1120-6.
    Shimizu T, Tsugawa Y, Tanoue Y, et al. The hospital educational environment and performance of residents in the General Medicine In-Training Examination:a multicenter study in Japan. Int J Gen Med. 2013;6:637-40.
    Dyrbye LN, Thomas MR, Harper W, et al. The learning environment and medical student burnout:a multicentre study. Med Educ. 2009;43:274-82.
    Cross V, Hicks C, Parle J, Field S.Perceptions of the learning environment in higher specialist training of doctors:implications for recruitment and retention. Med Educ. 2006;40:121-8.
    Dyrbye L, Shanafelt T.A narrative review on burnout experienced by medical students and residents. Med Educ. 2016;50:132-49.
    Llera J, Durante E.Correlation between the educational environment and burn-out syndrome in residency programs at a university hospital. Arch Argent Pediatr. 2014;112:6-11.
    van Vendeloo SN, Godderis L, Brand PLP, et al. Resident burnout:evaluating the role of the learning environment. BMC Med Educ. 2018;18:54.
    Lee PT, Loh J, Sng G, Tung J, Yeo KK.Empathy and burnout:a study on residents from a Singapore institution. Singapore Med J. 2018;59:50-4.
    Roff S, McAleer S, Skinner A.Development and validation of an instrument to measure the postgraduate clinical learning and teaching educational environment for hospital-based junior doctors in the UK. Med Teach. 2005;27:326-31.
    Soemantri D, Herrera C, Riquelme A.Measang the educational environment in health professions studies:a systematic review. Med Teach. 2010;32:947-52.
    Stalmeijer RE, Mcnaughton N, Van Mook WN.Using focus groups in medical education research:AMEE Guide No.91. Med Teach. 2014;36:923-39.
    Gough J, Bullen M, Donath S.PHEEM 'downunder'. Med Teach. 2010;32:161-3.
    Mahendran R, Broekman B, Wong JC, Lai YM, Kua EH.The educational environment:comparisons of the British and American postgraduate psychiatry training programmes in an Asian setting. Med Teach. 2013;35:959-61.
    Clapham M, Wall D, Batchelor A.Educational environment in intensive care medicine--use of Postgraduate Hospital Educational Environment Measure (PHEEM). Med Teach. 2007;29:e184-91.
    BuAli WH, Khan AS, Al-Qahtani MH, Aldossary S.Evaluation of hospital-learning environment for pediatric residency in eastern region of Saudi Arabia. J Educ Eval Health Prof. 2015;12:14.
    Sastre EA, Burke EE, Silverstein E, et al. Improvements in medical school wellness and career counseling:a comparison of one-on-one advising to an Advisory College Program. Med Teach. 2010;32:e429-35.
    Flaherty GT, Connolly R, O'Brien T.Measurement of the Postgraduate Educational Environment of Junior Doctors Training in Medicine at an Irish University Teaching Hospital. Ir J Med Sci. 2016;185:565-71.
    Khoja AT.Evaluation of the educational environment of the Saudi family medicine residency training program. J Family Community Med. 2015;22:49-56.
    Boor KB. The Clinical Learning Climate (dissertation). 2009;Amsterdam, the Netherlands Vrije Universiteit Amsterdam.
    Schönrock-Adema J, Bouwkamp-Timmer T, van Hell EA, Cohen-Schotanus J.Key elements in assessing the educational environment:where is the theory?. Adv Health Sci Educ Theory Pract. 2012;17:727-42.
    Chan CY, Sum MY, Lim WS, et al. Adoption and correlates of Postgraduate Hospital Educational Environment Measure (PHEEM) in the evaluation of learning environments:a systematic review. Med Teach. 2016;38:1248-55.
    Silkens MEWM, Chahine S, Lombarts KMJMH, Arah OA.From good to excellent:improving clinical departments'learning climate in residency training. Med Teach. 2018;40:237-43.
    Wasson LT, Cusmano A, Meli L, et al. Association between learning environment interventions and medical student well-being:a systematic review. JAMA. 2016;316:2237-52.
    Crosby RMHJ.AMEE Guide No.20:The good teacher is more than a lecturer - the twelve roles of the teacher. Med Teach. 2000;22:334-47.
    Ramani S, Leinster S.AMEE Guide No. 34:Teaching in the clinical environment. Med Teach. 2008;30:347-64.
    Ficklin FL, Hazelwood JD, Carter JE, Shellhamer RH.Evaluation of a small-group support program for first-year medical students. J Med Educ. 1983;58:817-9.
    Al-Shiekh MH, Ismail MH, Al-Khater SA.Validation of the postgraduate hospital educational environment measure at a Saudi university medical school. Saudi Med J. 2014;35:734-8.
    Pinnock R, Reed P, Wright M.The learning environment of paediatric trainees in New Zealand. J Paediatr Child Health. 2009;45:529-34.
    Stok-Koch L, Bolhuis S, Koopmans R.Identifying factors that influence workplace learning in postgraduate medical education. Educ Health (Abingdon). 2007;20:8.
    Thrush CR, Hicks EK, Tariq SG, et al. Optimal learning environments from the perspective of resident physicians and associations with accreditation length. Acad Med. 2007;82:10 SupplS121-5.
    Panagioti M, Panagopoulou E, Bower P, et al. Controlled interventions to reduce burnout in physicians:a systematic review and meta-analysis. JAMA Intern Med. 2017;177:195-205.