Opening up the non-personal data in the Home Office

Over the last couple of months I have been working on getting some non-personal data sets published in re-usable form (as recommended in the Power of Information report here). This all a part of the digital engagement strategy, but something close to my heart because of Rewired State and Young Rewired State. I have already seen the huge potential of making this data available: the Rewired State creations page showcases what can be made for next to no money over the course of 12 – 24 hours – imagine the potential?

Not only does it mean that awesome web and mobile apps are created – but it also means that the people who actually need to use this information, can create what they need, in a way that answers the problem they are trying to solve – traditionally this would have taken weeks/months of customer insight, and the solution then developed by industry experts. I really like the approach of looking to the digital community, to find the people facing whichever problem an organisation is trying to solve – and then finding those who also have the ability to create solutions, the geeks, the coders. Young people are an obvious example, and we have proved the success of that; but it can also be applied to say those with long term illness – a group that I know the Scottish government is trying to reach and help.

I thought that it might be useful to explain how we in the Home Office have been approaching responding to the recommendation:  ‘The government should ensure that public information data sets are easy to find and use’, and what we plan to do next.

Taking the simple remit to locate the non-personal data, find the original source and publish that, we began by looking at all of our publications. Most roads led to the Research Development and Statistics unit. So we – in communications and the Office of the Chief Information Office (OCIO) – started to talk to the statisticians about getting access to the raw data. Because we did not really know what we were asking, and RDS did not really know why or what we were asking either, we had a series of telephone conversations, email conversations and finally a good old coffee and a chat. (I tell you what, this is what I love about this work – you get to meet the most incredible people. I had not any idea of the work of the statisticians and I am in awe, and a little bit in love, with what they do). At the end of that, we had a clear understanding of the process of data being analysed and released, the varying degrees of complexity and statistical implications of disclosure (which basically means if we go to too much granular detail, there can be a chance that individuals or locations could be identified <- that’s very bad). Now we all knew what we were asking for, we had an idea of what we wanted to do and so we began to do it.

We have separated data into two high level categories: data that is currently published, and data that is yet to be created.

For data that is currently published we are working closely with the statisticians to get the raw data and we are now publishing it here www.homeoffice.gov.uk/data (published by the rather wonderful Carly Moore in e-comms). (You will see that we also link to PDFs that have data in, just so that you can see what will be coming up). It is working, and we are looking at how we can make this all better: easier to sort and find datasets.

For data that is yet to be created, we are talking to the relevant parts of the Home Office about the data that is required, and will be publishing this on an ongoing basis. We are also preparing guidance to enable officials to produce future data in a format and to standards that will facilitate its reuse. In the longer term we aim to establish a process whereby data is published in reusable form as a matter of course, and is made available promptly, whilst maintaining appropriate controls regarding the security of personal or sensitive data (in accordance with the Hannigan report).

So that’s how we are handling this. Does that help? I hope you will keep an eye on how this progresses. If you do anything with any of the reusable data, do tag it #honpdata and then we will be able to see what you do.

I would seriously love to have a developer session where the statisticians and coders work together – that would be alchemy.

Does this mean we have now clarified the formula for change?

Willingness of public sector + free public data + revised procurement rules + brilliant talent + global sharing = 21st century way of being a part of our community and engaging with our government?

Willingness of public sector

Well, we have just had the brilliant ’09 UKGovBarCamp (I was not there but all of my mates and colleagues were, so I received updates constantly and have seen some of the outcomes). There was the announcement of the Directgov innovations site, pledged support for the Rewired State: National Hack the Government Day and announcement of another event in April around public engagement online, (by the fabulous Mitch Sava from Polywonk).

None of this is done without integral support and co-operation from the public sector, with civil servants and Ministers engaging at every level: essential to make any of this have any point. So I can safely say that this is not just a nod in the right direction: this is a movement.

Free public data

No I don’t mean details about you and I, I mean stats and the like. Facts and figures that are available as APIs and then we can all make our own minds up. Many people have been campaigning to free our data for years, most publicly The Guardian (for some reason I cannot make this link, just Google ‘free our data’). Now a report is due out, here in Beta for your comment for two weeks. Here are the pages most interesting to the data bit: geospatial data and general data.

I have had many conversations recently about data, and how it will ultimately be the tripping point for everything that we all want to happen. The fear seems to really be that nothing will happen because the risk that by freeing the data and people mashing it up will embarrass some and highlight what is happening in certain areas: for us I guess this really means what might it do to the value of our houses? Well… I have no argument for that, except that if that is what is really happening, then we need to know about it and we need to do something about it: government and community alike to change those figures.

Recommendation 9 and recommendation 13 are the ones to watch for this one.

Revised procurement rules

*sigh* anyone who has seen me so far this year will no doubt have been met by my current rage-inducing rant: procurement/HR/head count – driving me nuts. We need to get some stuff done quickly, with the right people at very little, sometimes no, cost. But we can’t – because of procurement. I know that this was discussed at UKGovBarCamp, and boy wonder has set up a google group to start a conversation about this: but nothing else has happened: this MUST be dealt with, and is missed entirely in the Power of Information report.

Growl

Brilliant talent

I don’t need to say anything about this, there are so many superbly talented people, engaging for free and giving as much as they can to help push this on. These people are essential to making this formula work, but stymied by the procurement issue, often, or a perceived ignorance on the part of the public sector to listen to them. This is changing, but needs some proper attention.

H/T to Tom Watson MP for going out of his way to recognise and support this community.

Global sharing

I am delighted to see that over the past year there has been much International engagement and sharing of ideas, concurrent events and the like. Not enough. But this is where the work of futuregov and their competition is so vital. The Obama messiah-like effect has opened up the global awareness of what it means to have transparent governance, from online to the story of Michelle Obama encouraging people to ask anything they liked about her and her husband, their finances, beliefs – everything (never before seen true transparency).

Although I am not so interested in the political opportunities here, but the lessons we can learn – as we are all breaking new ground. The standards have been set high: once again quoting Obama’s inaugural speech:

The question we ask today is not whether our government is too big or too small, but whether it works — whether it helps families find jobs at a decent wage, care they can afford, a retirement that is dignified. Where the answer is yes, we intend to move forward. Where the answer is no, programs will end. And those of us who manage the public’s dollars will be held to account — to spend wisely, reform bad habits, and do our business in the light of day — because only then can we restore the vital trust between a people and their government.

Now a commitment to do everything in the light of day is something that all of our politicians say, but I believe that Obama means it: and the people he has appointed to work with him seem to show that he means it. (Macon Phillips and Katie Jacobs Stanton for example).

Conclusion

I *believe* that this is the formula needed, I cannot think of anything else really. I know that Mitch is running his event on online public consultation which will be great – but I think in order to dot the ‘i’s and cross the ‘t’s we need to do one on procurement rules: this would not be unconference style: this would need to be much more formal. But actually the trend seems to be moving in that direction, the freedom of the barcamp, through the practicality of the Hack Day, through Mitch’s semi-formal event to one aching with bureaucracy (procurement rules) – we need them all to make change happen.