A patients’ guide to Modernising Medical Careers and MTAS – Part 3

Pink Floyd - The Wall

You can find my most recent commentaries on the Junior Doctors’ crisis as it unfolds in 2008 by following this link.


You may have heard about the “computer problems” with the junior doctors job system. Calling the problems “computer problems” is like blaming the guillotine for the French Revolution.

(Briefly, as background, you should know that Patricia Hewitt and Tony Blair have decided that 20% of our specialist doctors are surplus to requirements. As of August they will be out on their ears. Most people simply don’t believe this statement, but it is true. To turn the wickedness of this into lunatic farce, doctors cannot apply to specific hospitals or NHS trusts. Instead a young specialist may end up virtually anywhere despite the fact that she probably lives within a handful of miles of a hospital training her peers in just her speciality. This is such a stupid situation that most people don’t believe that either.)

Right. Can’t avoid it any more. Let’s look at MTAS up close and personal.

The doctors have been selected for interview based on a Self-Assessment Competency Questionnaire. They are asked a dozen or so questions and allowed 150 word answers to each. Now I have to say that the questions are sharp and to the point, and do in fact cover stuff that is not explicitly available from CVs or even from references. Most doctors disagree with me and consider them to be a bullshitters’ charter, and an invitation to waffle. Paradoxically, they are right too as we shall see.

I suspect that the civil servants just don’t understand why the doctors see the questions as irrelevant requests for creative writing. This is probably the nub of the problem. I come from an environment where this sort of thing is usual and have considerable respect for the format because I am used to the soul-searching and hand-crafting that’s involved in answering this kind of question. So do the minions in the Department of Health. But the fact that civil servants like it doesn’t make it a good way to evaluate doctors.

You see, if you have not done these things before they are the very devil to answer. I am going to give you a couple of examples, that of Dr Smith and Mr Jones, both from the inimitable blog of Dr Crippen. I’ll restrict myself to comparing their answers to just one question.

B6 What experience of delivering teaching do you have? (150)

I have a broad experience of teaching House officers, SHO’s and medical students. I teach theory, practical skills and etiquette. I thoroughly enjoy teaching and find the challenge a constant spur to my own learning. I am an associate lecturer at the University of Toronto.

B6 What experience of delivering teaching do you have? (150)

I have a keen interest in medical education, and take every opportunity to teach in the clinical environment. Formal medical education training has enabled me to develop the skills, attitudes and practices of a competent educator. I bring innovation and passion to teaching, and have a recognition award for Clinical Teaching beyond the call of duty.

I’ve delivered lecture-based and clinical teaching to [specialty] colleagues, A&E colleagues, and nurse practitioners. I’ve taken responsibility for organising MRCS clinical examinations. As an accredited Basic Surgical Skills tutor, I’ve received excellent feedback from delegates and convenors. I tutor medical students in the Surgical Skills laboratory, and supervise medical students spending electives in [specialty].

I’ve been involved in regular surgical events for schoolchildren since 2002. I find great satisfaction in teaching. My experiences have helped develop my teaching, technical and leadership skills. I hope to continue developing and utilising these skills.

Dr Smith may be no worse at teaching than Mr Jones. He may be no less experienced. But you can see quite clearly that Mr Jones’s answer is by far the better of the two. Dr Smith assumed that his qualifications would speak for themselves. Before I leave them, I am going to link you to their own reflections on their applications. Dr Smith may not have been eloquent on his MTAS form, but he’s most certainly a gent and – from his CV – an excellent doctor. Mr Jones, too, seems a thoroughly decent bloke. I cannot believe we have lost one and may yet lose the other.

The medical profession has put some effort into researching how best to predict which applicants will turn out to be the best doctors. The papers BMJ has published on the subject include one by Fiona Patterson, amongst others. The paper advocates a three tier assessment of competencies based on an application form, a day spent at an assessment centre and structured references. The paper concludes (along with most of the literature on the subject) that for professions such as doctoring, a well planned assessment centre is the way to go. I only mention this specific paper because it is rumoured to be the basis for the Medical Training Assessment System, though if it is, there has been a long and deviant path from paper to process.

The three main problems with the paper are:

  1. It is based on General Practice competencies. Interestingly enough, GPs are being assessed by exam and assessment centre, this time round.
  2. Application forms included 6 self-assessed competency questions and were “rated independently by three assessors (experienced trainers or course organisers). Each assessor attended a four hour training session to enhance reliability”. As opposed to being specialist registrars recruited with no training on how to mark the forms, or – in the most surreal rumour I’ve come across – Police Cadets.
  3. The research questions focused on the outcomes of performance at an assessment centre, not on the effectiveness of the application forms in pre-selecting those to be assessed.

In other words, if the MTAS system was in fact based on this paper, then it’s missed the bloody point.

On top of that, it presupposes several things, none of which have been the case:

  • The applicants trust the system
  • The system is managed competently and fairly
  • The assessors know what the fiddle-dee-dee they are doing
  • Assessment at the assessment centre lasts a whole day, and the assessments comprise “a series of work-related simulation exercises, each lasting between 20-40 minutes” none of them involving origami.

This is an expensive approach. At some point someone decided to put more emphasis on the Self-Assessment Competency Questionnaire and to cut the day at the assessment centre down to a 30 minute interview slot.

Hello?

Is this the same?

The problem is that the circumstances which render an MTAS-style questionnaire invalid are the precisely the circumstances in which it has been introduced. It is seductive for civil servants because it is easy to administrate, simple to score, and – when introduced properly – it is effective at working out which are the suitable candidates. (I am sorry, but it is.) But it really has not been introduced properly.

Brace yourself for some academic jargon.

“A meta-analysis … found predictive validity to be relatively low (r=.29), but that it could be substantially improved (r=.64) by incorporating certain measurement conditions including instructions emphasizing comparison with others, previous experience of self-evaluation, guaranteed anonymity of [candidates] and the expectation that [the applications] will be validated or checked against other measures.”

In other words, this system is rendered ineffective if the candidates don’t trust it or haven’t used a system like it before.

Is this ringing any warning bells? In what conceivable world is it acceptable to use a system which is that fragile to work out who are the “weakest” 20%-25% of candidates in order to kick them out of their profession? But you are not a politician. If you were playing those sorts of games with people’s lives and with the NHS you’d want to make sure you’d kept the cream of the crop, and at least played fair by the rest.

Another problem with this whole Competency-Based Questionnaire debacle is that although – if implemented properly – they may be effective at identifying good doctors it is most definitely not proven that they are a good way at selecting out the bad ones. The people who fill the forms out well are (presumably) good, but I’ve not found evidence that everyone who is good will fill the form out well (remember this is in “ideal” conditions). To test for this, you would have to appoint doctors you believed were going to be bad, and then check after a while to find out if this was in fact the case. Don’t you just love ethics?

The bias is subtle and invidious.

“Harris (1994) … proposed that motivated [candidates] are likely to use thorough and analytic processing strategies (deliberate processing), leading to the retrieval of more information from memory. Those with low motivation are likely to use a quick heuristic-based approach (non-deliberate processing), associated with simplistic decision-making and integration processes (Chaiken, 1980; Petty & Caccioppo, 1986).”

This means that MTAS is actually biased against candidates who consider it to be a flawed system. The difference between “deliberate” and “non-deliberate processing” originates below the level of concious choice. We’ve seen a startling example of this with Dr Smith and Mr Jones. If it was a matter of conscious choice on the candidates’ part, then it would be a way to select out those who are willfully obstructive, but the Dr Smiths are not willfully obstructive. They just see no point in answering these questions when their CVs and academic and professional qualifications should speak for themselves and so they answer the questions quickly and without much analysis. (I had to look “heuristic” up in this context. The compact OED defines it as “…. proceeding to a solution by trial and error or by rules that are only loosely defined”). The thing is, it really is completely irrelevant to their abilities as doctors, so we really are not getting the cream of the crop. We are getting the ones that bought in, I guess, but there is nothing to say that they are better or worse as doctors. This feature of the system is particularly nasty, because it gives the government the opportunity to claim that it is the candidates’ fault for not taking it seriously.

What about the things that show a doctor knows her stuff, you ask. What about their qualifications and Membership exams? What about the things that say if they are a good doctor to work with? What about their references? What about knowing if they’ve worked in similar hospitals and with similar teams? What about their references and their CVs?

What indeed. Their MTAS score has a weighting of 85% in the decision of whether or not a candidate should be interviewed. Everything else, CV, Honours at Med School, research, published papers, higher degrees, doctorates, marks in their Membership exams, portfolios, references, every thing else contributed a maximum of 15%. This might be acceptable if MTAS was flawless, and it wouldn’t matter a damn if there was a shortage of doctors, but it is sheer professional negligence on the part of those implementing the system given what we know about the weaknesses of Self Assessment Competency Based questionnaires. Isn’t it?

I had at least retained some faith in the common-sense of the interview process until I read this which sickened me to my stomach all over again:

The second station [at my interview] involved ‘Communication Skills’. It was awful. First, I had to fold a piece of paper according to verbal instructions. It did not make a crane – perhaps I did it wrong? Then I was given a random series of shapes on a piece of paper and had to describe them to another Consultant for her to draw them. Hmmph. Goodness only knows how I did on this station. I felt stupid and I know that I didn’t show how well I can actually communicate about real things. What I don’t understand is how this is supposed to supply them with reasonable doctors. If I did it all wrong, am I a bad Doctor? If I did it right, should you fast track me to a Consultant’s post?

Bloody weird. – Junior Docspot – Origami Anyone?

The worrying thing is that the much vaunted review of MTAS which has now been promised by the government is going to spend its time checking the ropes of the guillotine, making sure that next time that the blade is sharp and the mechanism is properly oiled and running smoothly. The fact that the site kept on crashing is irrelevant, to be honest. The fact that the papers have been marked in such haste that the scoring is effectively arbitrary is marginally more relevant. The two things that the review will ignore are:

  1. Are Self Assessment Competency Questionnaires an effective way to sort out good doctors from less good ones?
    And the really important one:
  2. Why the f**k are we loosing 20-25% of our specialist trainees?

A note on my sources. I did a fairly quick literature search the other night and much of what I read summarised meta-analyses and other published research studies. Recruitment and candidate assessment isn’t my patch but I do live and work in the neighbouring metaphorical postcode. The paper which was most relevant and which I took my quotes from was:

Ivan T. Robertson and Mike Smith. (2001). Personnel selection. Journal of Occupational and Organizational Psychology 74, 441–472.

The paper in the BMJ was:

Fiona Patterson, Eamonn Ferguson, Tim Norfolk, Pat Lane (2005). A new selection system to recruit general practice registrars: preliminary findings from a validation study. British Medical Journal. 330, 771-714.

Stupidly, I have lost the references for the other papers I read that evening, but interestingly none of them discussed Self-Assessment Competency Questionnaires. Studies, one of them published last year in the BMJ, demonstrate time and again that the most effective predictor of candidate performance is the assessment centre. Ones involving simulated situations, not ones involving origami.

If you do want a list of references then I can find them again. Drop me a comment and I’ll skip through my browser history and dig them out.

A patients’ guide to Modernising Medical Careers Part 1Part 2 and Part 4 consider other aspects of this madness.

Advertisements

11 responses to “A patients’ guide to Modernising Medical Careers and MTAS – Part 3

  1. Pingback: You think you have it bad. « TheYellowDuckPond

  2. Oh god. This is awful. Well, the whole bloody debacle is awful, but this is the bit I find professionally awful.

    I did an exam once which was designed to test competancy in my professional field where I kept doing test papers and failing. Miserably. Eventually someone told me a story about British testing culture which went:

    The thing about British testing culture is that it’s based on someone sitting down to a question like ‘Marriage is good. Discuss.’ The average bod, without further information would probably write about all the good things about marriage and all the bad things and then do a summarising conclusion. And would fail, because what they are looking for is for you to write something like ‘Marriage. What is marriage. Marriange is an institution where… Good. What is good? Good is…’

    So I suddenly realised that while, when I teach exam classes for British exams I spend at least 50 percent of the time teaching students about the exam, I hadn’t applied the same principle to my one. So I did, and got a distinction (top marks).

    I was lucky though as it was a few years into a new version of the exam. Even though it was written by experienced test designers for people who were, by and large, experienced teachers for tests and had just completed a task on test design, they had a terrible time getting people to realise what they wanted them to do.

    Now, I’m with you on the fact that, done properly, this kind of thing can be very good.

    But I’m horrified by the fact that it’s being scored as though it’s an established system, and that… well, everything else as well. Good grief. I had no idea.

    I’m terribly, awfully sorry for any doctors caught up in this.

  3. Actually, apart from the ‘you need to know about it’, I think the main reason why good teachers did badly on my exam (because they did/do) is precisely the point you are making here about trust in the exam. A lot of my colleagues were sceptical about how an exam can test teaching ability.

    Luckily for us you can retake it, and in terms of grades it’s only one componant of three.

  4. It is the reactions from those not directly involved which are so heartening. Thank you both for your comments, your links and your support.

    Aphra

  5. How, how HOW can a self-assessment count for 85%? How? It’s… it’s stupid. Lost for words.

    What I still don’t understand is why the government thinks it needs to lose 20% of specialist doctors. I am currently attending a specialist clinic and hey, my appointments are three months apart because the clinic hasn’t got quite enough specialists. If in a year or so there are 20% less doctors, and I still need to visit the clinic, will my appointments now be four months apart? And that will be good how exactly? I’m going to need the same number of visits whether I get them monthly or annually. Grr.

  6. Hey Aphra,
    Firstly I’d like to say thanks for such a great blog (like everyone else who’s found it!).
    I am yet another junior doctor caught up in the stressful hell that is medicine at the moment, and wanted to add a couple of stories of my own “stomach churning” type.
    1) Friend’s father (consultant trying to score applications) contacted MTAS to state that he could not possibly get through all the applications due to time restraints. MTAS’ reply was “get through what you can and put the others straight into the second round”. So this basically says that your application may not even have been looked at purely because of the rush they have put on this stupid exercise.
    2) I am also aware that this could be a good process. As much as I want to rail and rant against it for as long as anyone will listen, I am perfectly willing to accept change can be a good thing – it just saddens me that all my worst fears before the implementation (ie that it would be disorganised and flawed) have come to pass and there is no other option for us.
    3) I am scared that Australia doesn’t have enough space for all of us that will be unemployed!
    4)The “New Consultant” will be neither properly trained nor competent to hold what that position should mean.
    5) As you so rightly say we need more doctors than there are consultant posts, and we need to change the attitude currently held in medicine that you have failed if you haven’t become a consultant. I suppose in response to Reed – yes we do need the increased numbers, but where does the funding come from to employ us? Deep down I think it is seen as an easy budget cut.
    OK – I’m sounding cynical now. I’ll stop using your space, and finish by saying a huge thank you again for a calm, sane and concise precis of a very emotional subject at the moment.
    Cheers
    Tory

  7. I’ve just applied for my first NHS posts and I must say that the questionnaires are absurd! It would be easy to just shrug my shoulders and mutter something about “brits!” and considering talking the man in my life into moving to my country in stead of vice versa.

    I’ve spent almost thirty years working in the public health service and I’ve been quite convinced that what matters is my skill, professional compentence and experience. I’ve never been much for using a lot of works just for the sake of it, so I guess that my approach as “dr Smith” above won’t pay off…

    …and those absolutely ridicoulus questions about religious beliefs, sexual preferences and ethnicity?!?!? Will it make me a better/worse doctor if I have certain sexual preferences, colour of my eyes or how I worship? I just refuse to answer those kind of questions.

    Here we’re discussing legislation stating anonymous applications. The applicants identity should not be disclosed until the employer decider who s/he wants to interview. The intention is to get the best person at the right place, and not be blinded by ethnicity, gender or other irrelevant issues.

    Just a final comment. In my field of medicine there is a shortage of qualified doctors all over Europe. Approixmately half of my colleagues working in the UK are “overseas”.

  8. The people who fill the forms out well are (presumably) good. I think that’s rather a big assumption. Especially if the definition of “good” includes “honest”, and one of the q’s was specificaly supposed to test probity. In any case, I have known plenty of candidates who are great at writing answers but who have little substance, and fail to support their fine words when probed further at interview.

    At the most I think you can assume that filling out the forms well correlates highly with skill in the English language and a an understanding that there are hoops to jump through, as well as a willingness to do so.

    I don’t think the review will ignore your first question; it’s an issue the doctors have taken up with some enthusiam; see for instance this video clip.

    Reed….uh, because doctors cost money.

    dragonqueen – the questions about ethnicity etc are, I believe, a legal requirement, intended to support monitoring that no groups are disadvantaged (ie you can look afterwards and see whether the proportion of black candidates offered a job was the same as the proportion of black candidates who applied. In the case of MTAS, the answers to those questions are specifically not supposed to be available to interviewers/scorers.

  9. Hi, I was so thrilled to see that you have written about overseas doctors coz I never came across any mention in the plenty of blogs that I have been following. (Please do me know of any such articles in blogs elsewhere, if any).

    I’m looking foward for your Part 4 on the plight of overseas doctors.

  10. Thank you for your third instalment. Another excellent piece of eloquent writing combining accurate facts.
    In a way, I feel even more beaten down than before. Dropping the assessment day is obviously down to money, who cares whether that was the fundamental part of the selection process?
    I agree that competency based self-assessment questions are useful, as PART of a tool, not the only tool!
    Lastly, you are right in that doctors do not trust this system, and we are not used to it. We are used to our qualifications speaking for themselves, and being assessed by fellow docs.
    Never felt so undervalued as a professional by our government

  11. Reed, it’s unfathomable, isn’t it? I don’t understand why the government is saying “we don’t need lots of consultants so let’s reduce the number of training posts” either. They are treating doctors as if they were junior army officers and pushing them out after a few years, instead of using the teachers’ model, as I’ve already mentioned. It’s beyond me.

    Tory. This has been my pleasure. I also consider it to be a duty to get the word out and accessable to as many people as possible. The biggest problem that I can see is that it is SO incredible that most people simply don’t believe it. I, too, worry about “New Labour’s” “New Consultants”. I had better not start on how it shows how intellectually and morally light-weight the labour party are these days or I’ll get political tourettes again and undermine my case with bad language. Anyone is welcome to use my space to provide anecdotes or opinions on this. The more the better.

    Dragonqueen – I would start persuading your man to move to yours. Truly. If I could I would. Potentilla has explained the ethnicity questions to you. They are actually important, because various lobby groups such as Stonewall audit and rank employers’ in terms of whether they are gay-friendly, ethnically balanced and so on.

    Potentilla, the point I was trying to make was about selectivity and specificity of the test but I got myself confused with the terminology and ended up writing what I wrote. My rather big assumptions are that most of our doctors are good and that they are not lying. That is what references are for. (There is a reason why Shipmans in hospital settings, such as Benjamin Geen, get caught.) The MTAS questionaire may well select OUT bad doctors, but it doesn’t necessarily select IN all the good ones, especially given Tory’s evidence on scoring it.

    Vikas, I will do what I can to write about the overseas doctors, but it is something I know less about. But it grieves me that the internal debate has moved on from your situation.

    Thank you Alexandra. Like you, the further I look into this whole thing, the worse I conclude that it is. Thanks for your enthusiasm.

    Thanks all, for reading.

    Aphra.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s