Big Data and Government Transparency

shortly before being appointed to the Supreme Court in 1906, Louis Brandeis famously made a quote about corruption and secrecy:

“Publicity is justly commended as a remedy for social and industrial diseases. Sunlight is said to be the best of disinfectants; electric light the most efficient policeman. And publicity has already played an important part in the struggle against the Money Trust.”]

It’s easy to become disillusioned with government. Incompetence, corruption and waste are so baked into our current system that it can feel pointless to even try to make a difference. Unfortunately for us today the “fourth estate” that is the press is doing a piss-poor job of casting sunlight on things that should not be hidden. Our major media outlets are owned by large corporations whose interests lie in protecting profits (this is legally mandated, by the way) rather than doing actual reporting. The two interests are often at odds in fact.

Fortunately, government is an area where data is readily available, and people are pushing hard for more open data in government. As a result the data is just sitting around waiting to be exposed.

Governments have been collecting data for internal use for centuries. The British Royal Statistical Society, of which Charles Babbage was a prominent member, has been collecting interesting data for a long, long time:

Early Big Data and visualizations

But only recently have the technical tools been available to allow governments to expose this data using things like public API’s, data exports, and so on.

The Big Data Revolution is changing our world. Previously we explored how private companies are using open data to produce greater understanding of the underlying trends in society (crimespotting), but now I want to point you towards the possible potential of establishing “the fifth estate”.  The data science estate.

A wealth of government data is available to us today on .gov sites and private sites across the web. If we analyzed this data properly, we could build a rich understanding of how our government works and how it could be improved. But as the big data challenge dictates, the chokepoint is consumption.  

Back in 2004, data scientists from Columbia conducted a thorough exploration of roll call voting data; a small amout of data by today’s standards which is publically available through the Library of Congress online. Using the tools available at the time, they did cluster analysis, pattern recognition, 3D metric mapping, and more to provide a rather deep look into the workings of the US senate. Their primary findings explored the differing clusters within congress, and even showed just how influential certain politicians were. It’s worth checking out.

Those data scientists were comparatively crippled when you consider what we have today. Their available data and tools were laughable compared to our current abilities. So if they built so much with so little what can today’s data scientists do?

Several organizations are trying to find out, they’re gathering massive amounts of data and helping us understand what that data means. By creating tools allowing you to track expenditures, watch your representatives, check voting records, and examine campaign contributions, sites such as  govtrack.us and influenceexplorer.com are peeling back the curtain to reveal the hidden side of politics.

Govtrack lets you see how politicians are voting and where they fit on the political spectrum, and with Influence Explorer you can see where they got their money from. Both sites use large amounts of data to display valuable information about politics, but they fall short because they rely on users to take that information and make it usable.
The dream of Big Data and Government is for all these data sources to be unified and explored, which could truly open our eyes. We could quantify the effect of money on politicians, heck you could give each politician a rating based on what degree they favor their backers over their constituents. This could also help politicians by helping them see how much their affected.  You have probably heard the adage,  “if you can measure it, you can manage it” well big data can measure it.  

Data is huge, and if used well it has the potential to hold the government accountable in amazing ways. This type of reporting and analysis is in its infancy and has been largely unused, but we see the potential, the promised land of government transparency, transparency born from Big Data and Analytics.