The Nudge Unit should publish more of their research – now

The Cabinet Office Behavioural Insights Team (or Nudge Unit) has done lots of interesting work – in particular, around using Randomised Controlled Trials in order to guide public policy. Some of their reported results – for example, when looking at getting people to pay tax more promptly – have been really impressive. I therefore submitted a Freedom of Information Act request to find out more detail about their research – in order to properly interpret research findings, it is important to be able to look at what was done and assess the strengths and weaknesses of the work. Sadly, the request was refused and, when I asked the Cabinet Office to review this response, they upheld this refusal.

The Cabinet Office gave two reasons for refusing my request:

  1. The Cabinet Office stated that some information on Nudge Unit research is already published. I’ve no arguments with this – I’m not asking the Cabinet office to republish what they’ve already published.
  2. The Cabinet Office argue that “It is the intention of the team to continue to publish the findings from its work on its website and, where appropriate, to publish findings in academic journals…academic publication can take some time, and it is important that the publication is timed so that it is released into the public domain as a coherent, consistent and accurate body of work”. I am not happy with this justification, as I’ll discuss in more detail below.

I’m pleased that the Nudge Unit is pro-actively publishing findings from its work on its website and is planning to publish in academic journals (which I entirely understand can take some time). However, I don’t think this is an adequate reason not to publish more information right now because:

  • Nudge Unit plans to publish “findings” in future aren’t sufficient – one needs detail about how the research was done in order to know how to interpret the findings. No research is perfect, so it’s important to be able to look at how the research was done rather than just trusting in the findings.
  • Researchers can publish some details of their trials (for example, trial protocol) prior to publication of the trial in an academic journal.  This is seen as good practice with clinical trials, and there are moves by some social science journals to insist on pre-registration of the trials they publish. I fail to see how publishing detail of trial methodology, hypotheses being tested etc. would prevent subsequent academic publications.
  • It’s not uncommon for policy research to be published in one format (for example, a policy report) and then subsequently written up as journal articles. While academic journals won’t want to run something that is identical to something already published elsewhere, the differing publication formats and audiences mean this often isn’t too hard to work around.
  • The Behavioural Insights Team makes bold claims for what some of its research has found – claims with major policy significance, and which seem to suggest adaptations to Government practice – without providing the information needed to critically assess these claims.  For example, in February 2012 the Nudge Unit argued that “the team has been able to demonstrate effects that, if rolled out, could save hundreds of millions of pounds.” That would be a dramatic achievement, and the paper discussing this research is an interesting read. However, the short summaries of trials given in the paper don’t give enough detail for one to judge how reliable they are. These bold claims were made more than 18 months ago and there is, still, not enough publicly available information for them to be properly assessed. If issues of coherence and accuracy are important to the Cabinet Office and they really feel they can’t publish quickly in more detail, the better approach would be to hold off on making claims until a fuller publication is possible – making the bold claims without a fuller publication is not helpful.

I now need to decide what to do with this Freedom of Information Act Request – I may appeal to the Information Commissioner’s Office. It may be that some Nudge Unit research will be out as academic publications shortly – in which case, it could make sense to wait – but this seems very unlikely to be the case across all their research. I have been disappointed by the lack of a more open response from the Cabinet Office.

Advertisements

2 responses to “The Nudge Unit should publish more of their research – now”

  1. Iris says :

    I would rather they spend their time making public services work better than telling people about what their doing.

    • jonmendel says :

      They already are spending time telling people about what they’re doing – the FOIA response linked above notes plans for future academic publications (and they have also discussed their work and findings on the Cabinet Office website and in other forums). The concern I’d have is that – without publication of more detail on their research – it’s hard to judge whether their work is actually showing how to make public services work better or whether, for example, their striking findings are caused by flaws in how the research was carried out. Publishing more fully would be cheap in the context of the savings they’re claiming (the Nudge Unit is talking about hundreds of millions – the cost of e.g. an extra researcher to help write up their work more promptly would be trivial compared to that).

      No research is perfect, and publishing might for example help bring out some limitations of what has been done (thus letting policymakers think more carefully about applications of Nudge Unit work) or let the Unit find ways to work more effectively to help public services function better. I’d view publication (and the discussion and criticism of one’s work that comes with it) as an important part of the research process, not just an optional bolt-on.

      Thanks for the comment, by the way – the first on this blog 🙂

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: