Open data then and now – and er next?

We at Rewired State have been asked to supply evidence to the Public Administration Select Committee. Adam McGreggor is going to go and represent us – as he is very clever at speaking – and answer questions on “IT’s role in government” on the 15th March. Today we have been speaking to the clerk and discussing one particular point: Why has there been no progress in people using public data since 2002?

So I thought in the spirit of my daily blogging we could chat about that here a bit. Firstly, in spite of calling all of our mates and getting hold of none of them, trying the Office of National Statistics and getting nowhere, tweeting and googling – we cannot for the life of us find anything relevant at all from 2002.

The genius we did find came, as ever, from twitter (thanks @harryharrold), when Tim Davies shared this with us from his dissertation:

It is utterly, utterly brilliant – but we are still bewildered about the 2002 thing. We are poring over this this afternoon, it would be ace to have some of your thoughts and ‘interesting points’ on it – and also how you think this line might extend.

I guess the more pertinent question is what happens next? How might this be shaped? It would be seriously interesting to run a Shell future scenarios piece of work on this; now that would be fun. Shall we?

9 responses

  1. ONS Neighbourhood Statistics was from the last 90s (in conception if not full availability), and a load of other ONS public data processes hook into that. There are various people around who were part of that agenda; and while “open data” wasn’t clearly articulated as a concept, it is recognisable as a concept in retrospect – it was just using different words.

    Some of this was from the (O)NS changes post-election in 1997 – it might be worth looking at why those happened; it was part of the “evidence based policy” agenda, and data was seen as a part of that.

  2. David has kindly allowed me to copy here an email that he sent me. I include his details at the end on purpose:

    In addition to web delivery. ONS produced a series of 7 or 8 CD-Rom volumes (of about 1-3 CD each) for the output of stats form the 2002 census. These were primarily to be sent out to LIbraries (Large/county level) and councils. I think the CD could be purchased from TSO at “production cost” e..g not a lot (We diddnt have the contract for the production, just design/build)

    Each volume was focussed around some specific topics which I cant remember, but it was a very comprehensive release I think.

    Detail was down to a very low level of data, e.g. “output area” which is a series of something like 3 or more postcodes. Data was navigable by output area, ward, parish, electoral boundary, authority etc right up to GOR. The presentation of data was in 3 formats, an HTMlL index (for browsing) and CSV for import and a stats package output that enabled 3D Tables. A free viewer was provided for the stats package. The overall interface was HTML that provided links into the CSV/Stats viewer etc. CD would work on a MAC PC etc because they were just an HTML file system, built via some scripts by designed by 2 very clever developers who turned my ideas into reality (as they do).

    Although I did not work on the web side at the time they were trying to do a parallel release on the web, but did have some problems as the website had a very big back end that needed lots of power to crunch the stats before presenting them. Im not sure how open the data was in the output, I think you could download it etc but id doubt if the URL we hackable or RDF etc etc.

    If you go to a larger library they should have the CD (I never kept a copy) and Id be interested to know if they actually work re open data needs today.

    The team at ONS I worked with were really pro doing the right thing, very keen on making it open and not using anything proprietary where ever possible, it was one of those really good project where everyone wants to make it work. The company I worked for was very small <50 employees I think we won it because we had security clearance from the MoD, but also did loads of web stuff for DfES etc. They seemed to like the fact tat the bloke that did the sales pitch (me) would be working on the project, not like a big SI etc.

    Hope this helps

    David Edwards
    Zennor Internet Assurance
    dedwards@zennor.co.uk

  3. Hi Emma,

    I’d take a punt that Census 2001 is the most widely used (UK) data-source when it comes to targeting local services and allocating resources (if you ignore underpinning datasets such as Ordnance Survey geographic stuff). And still widely used, despite being 10 years out of date.

    Also, if you’re looking for commercial applications of open data, census data is one of the biggest successes. Lots of commercial datasets and services are based on Census (e.g. ACORN, MOSAIC segmentations all rely heavily on Census).

    The Census 2001 data was sent out (and still used) much more widely than David (above) may have known about. Unlike previous censuses it was completely free to users (as result of recommendations from the “Better Information” Policy Action Team work in the late 90s) and provided in open csv format. As a result it’s very widely used by local authorities, voluntary and community sector groups, service providers, commercial research groups and so on. The 1991 Census was more limited, as it cost an absolute packet; really only used by universities and big businesses.

    One of our projects is a signposting website http://www.data4nr.net for public datasets (started in 2005 to support local govt researchers, and provided most of the initial data content for data.gov.uk) – from this it is very clear that census is still the best source if you want detailed data on who lives in your local area, what they are like (on things like health levels, skills, (un)employment, occupation, housing, and so on), and how your area compares with other areas across the country.

    So, the question about 2002 might be a good one if you think of it in service delivery rather than IT terms: What progress have we made in using public data since the census 2001 was released (it started coming out in 2002)? If we have made real progress, why are we still paying big bucks to collect Census 2011?

    Cheers, Tom.

    Dr Tom Smith, OCSI
    tom.smith@ocsi.co.uk, @_datasmith, http://www.ocsi.co.uk

  4. This is interesting stuff, but it shows Whitehall trying to catch up with a global trend that was much further advanced in other European countries and the USA. (example: the UK got a Freedom of Information Act in 2000, the USA has had one since 1966.)

    I was working at the UK Hydrographic Office in 1998, where I found MoD trying to cope with bilateral agreements and transnational arrangements on GIS data; it was clear at that stage that the UK’s idea of ‘trading funds’ that sold government data for profit was quite at odds with other countries’ treatment of data as a ‘public good’ that the taxpayers were entitled to have access to.

    I really appreciate your efforts to orient yourself (and the rest of us) in a big picture; I think it helps to consider a lot of what happens in the gov.uk space is about being late to the party in an international arena, not sure whether to copy the others or remain distinctively British.

  5. Pingback: Waiting for Census « DataBridge

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 147 other followers

%d bloggers like this: