Archive | November 2013

The Nudge Unit should publish more of their research – now

The Cabinet Office Behavioural Insights Team (or Nudge Unit) has done lots of interesting work – in particular, around using Randomised Controlled Trials in order to guide public policy. Some of their reported results – for example, when looking at getting people to pay tax more promptly – have been really impressive. I therefore submitted a Freedom of Information Act request to find out more detail about their research – in order to properly interpret research findings, it is important to be able to look at what was done and assess the strengths and weaknesses of the work. Sadly, the request was refused and, when I asked the Cabinet Office to review this response, they upheld this refusal.

The Cabinet Office gave two reasons for refusing my request:

  1. The Cabinet Office stated that some information on Nudge Unit research is already published. I’ve no arguments with this – I’m not asking the Cabinet office to republish what they’ve already published.
  2. The Cabinet Office argue that “It is the intention of the team to continue to publish the findings from its work on its website and, where appropriate, to publish findings in academic journals…academic publication can take some time, and it is important that the publication is timed so that it is released into the public domain as a coherent, consistent and accurate body of work”. I am not happy with this justification, as I’ll discuss in more detail below.

I’m pleased that the Nudge Unit is pro-actively publishing findings from its work on its website and is planning to publish in academic journals (which I entirely understand can take some time). However, I don’t think this is an adequate reason not to publish more information right now because:

  • Nudge Unit plans to publish “findings” in future aren’t sufficient – one needs detail about how the research was done in order to know how to interpret the findings. No research is perfect, so it’s important to be able to look at how the research was done rather than just trusting in the findings.
  • Researchers can publish some details of their trials (for example, trial protocol) prior to publication of the trial in an academic journal.  This is seen as good practice with clinical trials, and there are moves by some social science journals to insist on pre-registration of the trials they publish. I fail to see how publishing detail of trial methodology, hypotheses being tested etc. would prevent subsequent academic publications.
  • It’s not uncommon for policy research to be published in one format (for example, a policy report) and then subsequently written up as journal articles. While academic journals won’t want to run something that is identical to something already published elsewhere, the differing publication formats and audiences mean this often isn’t too hard to work around.
  • The Behavioural Insights Team makes bold claims for what some of its research has found – claims with major policy significance, and which seem to suggest adaptations to Government practice – without providing the information needed to critically assess these claims.  For example, in February 2012 the Nudge Unit argued that “the team has been able to demonstrate effects that, if rolled out, could save hundreds of millions of pounds.” That would be a dramatic achievement, and the paper discussing this research is an interesting read. However, the short summaries of trials given in the paper don’t give enough detail for one to judge how reliable they are. These bold claims were made more than 18 months ago and there is, still, not enough publicly available information for them to be properly assessed. If issues of coherence and accuracy are important to the Cabinet Office and they really feel they can’t publish quickly in more detail, the better approach would be to hold off on making claims until a fuller publication is possible – making the bold claims without a fuller publication is not helpful.

I now need to decide what to do with this Freedom of Information Act Request – I may appeal to the Information Commissioner’s Office. It may be that some Nudge Unit research will be out as academic publications shortly – in which case, it could make sense to wait – but this seems very unlikely to be the case across all their research. I have been disappointed by the lack of a more open response from the Cabinet Office.