Building on sand

I’ve just come to understand one of the core differences between IT and medicine: most medics know how to arrive at a conclusion based on evidence but most people working in Business and IT don’t.  As a result far too many decisions in Business and IT are based on something that looks like evidence, sounds like evidence, but actually isn’t evidence.  And I don’t know about you, but I find that slightly sickening.  

What’s my evidence for this view? (Ha!) It’s an internal document I’ve just been reading which says:

A conservative estimate of £10m p.a. colleague efficiency savings would be realised through deploying SuperSystem to 25,000 users across the organisation.

It includes no references. There’s no way for me to come to my own conclusion. No links to any validatable and verifiable studies showing what savings have been achieved elsewhere.  Da nada.  Nothing.  As a reader, I have to trust the writer’s judgement.  The older I get, the less I like that.

It’s not just a matter of being given the evidence though.  It’s a matter of being able to evaluate it.  Consider this from a campaigning website:

Dihydrogen Monoxide (DHMO) is a colorless and odorless chemical compound, also referred to by some as Dihydrogen Oxide, Hydrogen Hydroxide, Hydronium Hydroxide, or simply Hydric acid. Its basis is the highly reactive hydroxyl radical, a species shown to mutate DNA, denature proteins, disrupt cell membranes, and chemically alter critical neurotransmitters. The atomic components of DHMO are found in a number of caustic, explosive and poisonous compounds such as Sulfuric Acid, Nitroglycerine and Ethyl Alcohol.

Now if you’ve an iota of common sense you’d want it banned, wouldn’t you?  Ha! Fooled ya (unless you were paying attention in chemistry lessons of course).  It’s water.  The DHMO site is there to show how easy it is to baffle with pseudo-scientific bull.

In Business and IT this is done all the time.   I’ve done it myself.  I’ve found some statistics that seem to support my case and bunged them on to a slide with a footnote saying “Source: Gartner” or wherever.  It’s an empty ritual: no-one makes any pretence of reading the study or critically evaluationg Gartner’s data and conclusions.  We go through the ritual so we can all pretend we’re being objective and rigorous, but in fact we’re not.

This flaccid acceptance of rubbish is widespread.  Some time ago I was in a meeting where the discussion went as follows:

Bloke: Our target is that 80% of balls will be blue
Ben: So that means they’re pie-bald and 80% of the surface of each ball will be blue?
Bloke: Er, 80% of the balls will be blue
Ben: Oh I see – you mean that for every 100 balls, 80 of them will be entirely blue?
Bloke: (Snarling) … It’s just a turn of phrase…

It was all I could do not to snarl back “No it isn’t a turn of phrase – it’s a number” but the other people in the meeting weren’t interested.  Which only goes to prove my point, really.  

Imprecision, innumeracy, lack of scepticism, lack of critical thinking, call it what you will: it lures us in to blarney and over-optimism.    Yes, it’s hard to be rigorous and objective because in Business and IT by and large we don’t have standardised tools like double blind controlled trials to base our decisons on.  There are exceptions of course, like the the rules-based numeracy required of accountants and the occasionally rigorous testing of direct marketing campaigns.  But neither of those apply to the world of IT.  

Wherever we work, we cannot give up the responsibility of using our brains and for seeking out and using the best analysis and measurement tools we can find.  The problem is too many people charged with making decisons lack the habit of critical thinking which you need to cut through this sort of fluff.  It’s a cultural issue too: why was I the only person in the meeting who was outraged that a proverb was presented as a metric?  Because I don’t work for an engineering company, for a start.   Finally, the fact that it’s difficult is no excuse for giving up entirely and relying on “street smarts” “judgement” “instinct” or “gut”.  It was gut instinct that led medics to grind up spiders with dung and rub them in to people’s eyes to cure blindness.


3 responses to “Building on sand

  1. There’s a reason for the paucity of evidence in business. Getting good evidence is HARD. Really hard. So hard, in fact, that, pragmatically, more often than not it’s not worth doing.

    It’s worth doing in medicine, because the alternatives are
    (1) use a drug that doesn’t work and be responsible for KILLING someone
    (2) fail to use a drug that does work, and be responsible for KILLING someone.

    When either of those two things are an option, you better be damn sure you’re dealing with the FACTS. Intuition and experience just don’t cut it.

    But in business, it’s like this:
    (1) use a solution that doesn’t work, and be responsible for… losing some money.
    (2) fail to use a solution that would have worked and be responsible for… losing some money.

    When those are the two worst case scenarios, making the MASSIVE extra effort to be certain, over and above being pretty sure based on your intuition and experience, simply isn’t worth it. It may actually be worse, because the time and effort spent making sure you’re right may be more than the savings you make.

    Where I work, we have two leading lights in R&T. One is an old skool scientist – dots all the i’s, crosses all the t’s. Does everything by the book, backs up all his findings with meticulously documented evidence, has files on his work going back decades, makes all the references, tabulates all the data, draws graphs, you name it. The other is young, smart, and fast. He skips steps, cuts corners, and generally doesn’t bother acting like an academic.

    Both these men have the same objective – to develop new grades of product for our company, and to bring them from an idea, through lab scale trials, to full, and crucially, PROFITABLE, production. They are not paid to write research papers, teach students, gain patents or attend conferences. They are paid to get new products out of the gate, by whatever means necessary. Guess which one of them succeeds best at that?

  2. MMm.

    I agree that it’s stupid to spend £100 to protect a tenner. What bugs me is the sloppy thinking, innumeracy and self-delusion.

    I could respect the honesty if someone said: “we really don’t know, but Joe called ‘Heads’ so heads it is”. But instead you get this pretence of making an informed decision, when in fact it’s just cargo-cult ritual.

    I sound angry, but I’m not. I am exasperated though. In fact of course you are right – it doesn’t actually matter.



  3. The most exasperating module in my degree course was titled “Economic Evaluation of Projects”. In it, we learned about discounted cash flow rate of return, and how to tell whether a project was worth doing or not, from a purely financial perspective. The lecturer had written the textbook, and it was available only from him (good rate of return on that project, presumably…).

    Anyway, the point was this: you had a large, complicated formula, that in the days before a PC on every desk had to be evaluated by hand with a pocket calculator. Into this formula you plugged in the “data”. It concerned the cost of credit, the concept of future value of investments, depreciation, all kinds of stuff. I use the scare quotes around “data” because the “data” consisted of
    (a) the cost of the project
    (b) the ongoing costs, e.g. spares, maintenance, process raw materials etc.
    (c) the amount of money the company would make from selling the product

    Even as a callow student, something seemed amiss to me. I put my hand up and asked where the numbers in (c) came from. With a withering look, the lecturer stated the obvious – they’re ESTIMATES. Or put another way – guesses. Wildly optimistic guesses, probably. Either way, they’re not representing anything real.

    What annoyed me was going through this tedious, complex process to arrive at an answer, KNOWING from before you start that the “data” that is the basis of the answer is garbage.

    What I came to realise over more than a decade as a chemical engineer is that while (c) is of necessity a wild guess, actually in reality so are (a) and (b). For instance, my company uses a raw material which is produced by only two companies in Europe – our competitors, and a small company in Italy. The small company in Italy has just raised the cost of that material by 60% (not a typo, SIXTY percent) as an immediate precursor to announcing that they’re going to stop making it altogether by the end of the year. This stuff is VITAL to almost our entire product line. How do you factor that into your projections?

    I understand that the bean counters have to have some way of telling one project from another that is more quantifiable than choosing the one whose project manager is tallest and has the best hair. However, in my experience, there is a disturbing correlation between height and hairstyle and successful securing of capital for projects.

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s