Does More Data Mean Better Decisions

So this post is partly in response to a comment from Roland Harwood asking about 23andMe, but mostly me thinking out loud about data and risks. While this post is in response to the data generated by 23andMe, it is applicable to almost any isolated set of data.

23andMe, if you haven’t heard of it, is  a DNA analysis service, where they will take a DNA sample from saliva, analyse it, and, based on an automated database, they will essentially tell you your risk factors for succumbing to various medical conditions, and genetic traits that you are carrying (if you are a carrier for a generic condition, or have a genetic indication for an allergy). It is a really interesting proposition, and I have toyed with the idea of sending off for an analysis myself, especially as the price point has fallen to under a hundred dollars.

However, there are a few things that leave me a little disquieted about it, some of which come up frequently in discussions around 23andMe, and others not so much. These are my thoughts as they are now, rather than any conclusive opinions, so they are very much up for comment and debate. None of them are a criticism of the system itself, more an observation that we are only just getting used to dealing with this sort of data.

The first concern is the issue of medical diagnostics being delivered in isolation. If you are going to receive some news that is potentially life changing, then having the right professional help and support on hand is psychologically very important. Perhaps that is there, but I have never heard it mentioned. Data, without the right experts to interpret it, can be a very disturbing thing.

The second is to do with how we process information, and our cognitive biases. I’ve spent much of the last several years studying these biases, trying to design systems for businesses that help to avoid the issues cognitive biases cause. There are a particular set of biases (availability bias, hindsight bias, confirmation bias, …) that we have around assessing risk, which essentially boil down to this: We disproportionately react to perceived risk. If I tell you that there is a 30% chance that you will die if you choose to go to work by your normal route tomorrow, most people would think about changing their route home. But that is a meaningless piece of data. Risks, out of context, aren’t helpful. If I tell you that there is also an 80% chance that you will get killed in a car accident on the alternative route home, then the normal route is actually a safer one (you might actually decide to stay in bed ;) ). It is a meaningless risk unless you balance is against the risks of the alternatives.

Risk is often presented out of context. Responding to risks kicks off a long chain of causality. If I choose to have surgery to correct or mitigate against a genetic defect, then that surgery obviously carries a risk, but down stream from that, I have changed all sorts of other risk factors. It is one of the reasons that John Boyd came up with the OODA loop. Risks (threats) have to be constantly monitored and responded too. It isn’t a one time event. A one off diagnostic can give a false sense of security, as much as it can give a false sense of risk.

The next thing, aside from the issues of assessing probabilities and risks, is that we aren’t good at making judgements about events that are a long way in the future – for example diseases that we might succumb to later in life. There is a whole body of research around risk/reward ratios and timing, which again shows that we don’t deal with this sort of data accurately, at least not unaided. The key here is that while the data may be very scientifically valid and sound, it can cause us to do some unsound things, because it is difficult to process unaided.

The last is that 23and Me is based on science-in-progress. We are still learning about genetics, heritability and what happens when we respond to them. At lot of the outputs that I have seen fall into the ‘well duh!’ category of health advice: eat healthily, do exercises and so on. All the kinds of things that people who take good care of their bodies tell me that I should do more of, leaving me rightly a bit guilty. I don’t need to shell out money for that advice, I can just hang out with some of my healthy friends, and take their advice on the chin.

We have more and more access to data. That doesn’t make us any smarter, and it potentially doesn’t make us any less likely to make good or bad decisions. The issue is about making informed and uninformed decisions. Data can be good, and help us make good decisions, but being misinformed – ie being informed by data that is inaccurate (estimated), or that is misinterpreted or presented out of context  – can be worse than being uninformed.

Data doesn’t always help with making better decisions. It is good to be informed, it is not good to be misinformed, especially if that leads you to take more risky decisions. When looking at information:

  • Keep things in context – back to the journey to work example. What are you balancing risks against?
  • Understand the quality of the data – what is the possibility that it is inaccurate or incomplete?
  • Look for counter indicators – don’t response to single pieces of data.
  • Compare like with like – risks and issues are different things. Don’t compare the past with the future.

If you want to ready more, Noreena Hertz  has written a good piece in the NYT, Why we Make Bad Decisions (which is also a good plug for her book).





  4 comments for “Does More Data Mean Better Decisions

  1. October 22, 2013 at 9:25 pm

    Brilliant post Ben. Thanks so much for sharing and for writing it. One of the things I love about even contemplating getting my DNA sequenced is the fascinating conversations it is prompting like this one. People seem either intruiged to try it or against the idea, with not too many agnostics which is also telling. Anyway re your four valid points my immediate reactions/questions are as follows:

    1. Can’t I just go to my GP to talk about the results and get medical advice?
    2. Take the point but would disagree that it’s not meaningless data, though agree it’s dangerous to take it out of context or without comparisons
    3. The long term mis-judgements point is very interesting. Can you say a bit more about what you mean here (am gonna read the Noora Hertz post in a minute so that might cover it). Obviously the fallacy of youth is that you’ll live forever and the fallacy of middle age is perhaps that you’ll die in peace/happy. But I think you meant something else didn’t you?
    4. Re last point, sure we could listen to our healthy friends but not sure just because it’s science in progress means it’s not worth engaging with. I’ve heard estimates are that DNA testing will be under $1 by the end of the decade and so will become essentially ubiquitous by then. Wouldn’t you rather get a 7 year head start on sorting out whatever issues you may want to tackle.

    The other thing which intrigues me which you don’t comment on is finding out your genetic ancestry which I find as interesting as the health related stuff.

    Anyway, I’ve paid for the kit and am intending to still go ahead with some colleagues though we haven’t done it yet. Will keep you posted.


  2. Benjamin Ellis
    October 23, 2013 at 9:15 am

    Hi Roland!

    Good questions. Taking them one by one:

    1. A GP is just that, a “General Practitioner” – while they are fantastic, they aren’t an oncologist or an expert in DNA sequencing. If products like 23andMe take off and people start visiting their GP as a result, while some of that may be good preventative medicine, it is going to put an increased load on a very limitted NHS service – a cost that the test providers wouldn’t currently have to bear.

    2. Good data can be rendered meaningless by it’s context. If I discover that I have an increased risk of terminal prostate cancer, I am probably going to feel like I am more likely to die. I’d say I wouldn’t be able to help it. But am I more likely to die than I was the day before I got the test results? Of course not. Factor into the fact that I regularly sky dive and go fishing (it’s very dangerous), and I may not live long enough to experience the increased risk. (I don’t actually sky dive or fish, but I do catch the South West Trains service from Woking to London quite frequently, which is pretty much the same thing). Much of the data that DNA testing is based on is correlation data. There are whole books on the issues with that.

    3. I’ll write a post on this one, as it relates to compensation design – I’ll add a comment here when it’s done! If I offered you the choice of £1 right now, £5 the next time I see you, or £5,000 in ten years time, if I am still alive, which would you take? Pick one :). Now build a spreadsheet to justify that choice.

    4. The technology mantra of our age is “look, we can do it, and it is cheap.” – it has been great, but we are starting to get into areas where perhaps technology companies need to tread more carefully. What are the long term consequences of wearing Google Glass, or other forms of AR? We don’t know. What are the downstream impacts of cheap DNA testing, and the gene therapies that will almost certainly follow it? Google pulled out of the medical space, because it didn’t like the regulatory hoops it required them to jump through.

    While I am very excited about the possibilities of “quantified self” tools (I’d put 23andMe in that bucket, roughly), the psychologist in me is cautious about how we use them. Very small shifts in the ‘data’ that we receive can create big changes in behaviour (see ‘Nudge’ and behavioural economics research). That can be a good thing, a bad thing or neutral. Our prevailing view of technology (driven by big company marking spend) is that technology is inherently a good thing. That isn’t the always case. We need to make sure that we hold it too account, in much the same that we hold government to account. There are few formal mechanisms to do that today.

    We need to be selective, and point technology in the right direction, and drive manufacturers to support more research, and more collaboration across the different parks of their ecosystem (e.g. DNA test vendors working with health services and probably insurance vendors and others). There are great things that can be achieved.

    Back to the first point: One day perhaps we will get DNA testing from our GP. It could be great preventative medicine. But, if we did get it from our GP it would probably be a tested, regulated medical product. QS vendors get to side step that process, by selling directly to us. Yes, that probably makes the products cheaper, but there is a cost to it. Current medical approvals processes seem painfully long and costly – test results, documenting side effects, and so on. To me it seems to be an area that should be open to some innovation, and I am sure it has been and is. It is a great shame that Google has, for the moment, decided not to play, as reducing the cost and time-to-market for treatments would have a dramatic effect on world health.

    I think 23andMe is great. Would I do it? Yes, I think I probably would. Would I have surgery or change my life style as a result of it? I’d be very cautious about that.

    I think you nailed it in that last but one paragraph about the genetic ancestry. I think 23andMe can be a great bit of fun, and steer us towards more healthy lifestyles. My concern with people seeking dangerous, life-changing surgery as a result. Yes, the medical experts should keep them from that, but there are a whole other series of systematic biases which mean that isn’t always the case.

    It niggles on a boarder concern that I have: We are still adapting to living in an information-rich world, within a culture that places little value on wisdom, and an increasing emphasis on ‘data’. Data, without context and insight, is meaningless, however good it is.

  3. Benjamin Ellis
    November 25, 2013 at 10:27 pm

    Interestingly, it looks like the FDA have sent another warning to 23andME – FDA Notice – 22nd November 2013

Leave a Reply