You are here

Big Data and Government Transparency

shortly before being appointed to the Supreme Court in 1906, Louis Brandeis famously made a quote about corruption and secrecy:

“Publicity is justly commended as a remedy for social and industrial diseases. Sunlight is said to be the best of disinfectants; electric light the most efficient policeman. And publicity has already played an important part in the struggle against the Money Trust.”]

It’s easy to become disillusioned with government. Incompetence, corruption and waste are so baked into our current system that it can feel pointless to even try to make a difference. Unfortunately for us today the “fourth estate” that is the press is doing a piss-poor job of casting sunlight on things that should not be hidden. Our major media outlets are owned by large corporations whose interests lie in protecting profits (this is legally mandated, by the way) rather than doing actual reporting. The two interests are often at odds in fact.

Fortunately, government is an area where data is readily available, and people are pushing hard for more open data in government. As a result the data is just sitting around waiting to be exposed.

Governments have been collecting data for internal use for centuries. The British Royal Statistical Society, of which Charles Babbage was a prominent member, has been collecting interesting data for a long, long time:


Early Big Data and visualizations

But only recently have the technical tools been available to allow governments to expose this data using things like public API’s, data exports, and so on.

The Big Data Revolution is changing our world. Previously we explored how private companies are using open data to produce greater understanding of the underlying trends in society (crimespotting), but now I want to point you towards the possible potential of establishing “the fifth estate”.  The data science estate.

A wealth of government data is available to us today on .gov sites and private sites across the web. If we analyzed this data properly, we could build a rich understanding of how our government works and how it could be improved. But as the big data challenge dictates, the chokepoint is consumption.  

Back in 2004, data scientists from Columbia conducted a thorough exploration of roll call voting data; a small amout of data by today’s standards which is publically available through the Library of Congress online. Using the tools available at the time, they did cluster analysis, pattern recognition, 3D metric mapping, and more to provide a rather deep look into the workings of the US senate. Their primary findings explored the differing clusters within congress, and even showed just how influential certain politicians were. It’s worth checking out.

Those data scientists were comparatively crippled when you consider what we have today. Their available data and tools were laughable compared to our current abilities. So if they built so much with so little what can today’s data scientists do?

Several organizations are trying to find out, they’re gathering massive amounts of data and helping us understand what that data means. By creating tools allowing you to track expenditures, watch your representatives, check voting records, and examine campaign contributions, sites such as and are peeling back the curtain to reveal the hidden side of politics.

Govtrack lets you see how politicians are voting and where they fit on the political spectrum, and with Influence Explorer you can see where they got their money from. Both sites use large amounts of data to display valuable information about politics, but they fall short because they rely on users to take that information and make it usable.
The dream of Big Data and Government is for all these data sources to be unified and explored, which could truly open our eyes. We could quantify the effect of money on politicians, heck you could give each politician a rating based on what degree they favor their backers over their constituents. This could also help politicians by helping them see how much their affected.  You have probably heard the adage,  “if you can measure it, you can manage it” well big data can measure it.  

Data is huge, and if used well it has the potential to hold the government accountable in amazing ways. This type of reporting and analysis is in its infancy and has been largely unused, but we see the potential, the promised land of government transparency, transparency born from Big Data and Analytics.

Data Stories

How to increase sales by looking at your customer data

Are you a B2C company interested in increasing sales? If you have some form of customer data, you may be in luck.

IBM's Watson and the future of Healthcare Analytics

What would it be like to have a doctor who’s always up on the latest research and has learned about treatments from over 1.5 million previous cases? It would look alot like Watson, IBM’s Jeopardy! playing supercomputer that’s getting ready to roll out with an all new look and a Memorial Sloan Kettering Cancer Center education in oncology.

How Facebook's Graph Search will affect Google, Technology, and Privacy

What has been both feared and expected is finally on its way: Facebook is building a better search; they're opening their vast stores of user data and giving us the ability to discover what’s inside. Lars Rasmussen, the mind that brought us Google Maps, is now hard at work creating Facebook’s new Graph Search, and from the looks of it, it’s going to put unprecedented power in the hands of its users.

How Big Data and Analytics will Change Society.

The mission statement of most police departments includes something like this, “our goal is to increase public safety, prevent crime, and protect human life.” With sufficient records of criminal behavior and analytics tools like Crimespotting, Cities can have the ability to predict when and where crimes are likely to take place and dispatch accordingly.

The Big Data Revolution

Want to learn more about how big data is changing business and how you can take advantage? Pick up our latest book: The Big Data Revolution.

"With everyone talking about Big Data and Data Science, its tough to know whom to listen to and how to make sense of it all. This book cuts through the noise and presents the reader with a roadmap for success. Whether you've been in this space for a while or are just coming up to speed, Secrets of the Big Data Revolution is a must read."
-Chris Crosby, CEO of Inflection Point Global

Trend Watch

What Higher Education Teaches us about Data-Driven Customer Retention

Rio Salado Community College is currently optimizing their student retention through focused testing and they are finding some truly telling results. We can learn a lot about customer retention and segmenting by studying what they've done.

The Growing Collaborative Consumption Market

Remember the Big Data mantra? how Big Data will enable us to better understand everything, reduce waste, and improve efficiency? Well honestly, without concrete examples it fades into the mass of voices shouting about how great the world is going to be. So lets take a moment to talk about collaborative consumption and it’s implications.

Andreas Weigend on the Future of Social Data

Dr. Weigend, Stanford Professor and former Chief Data Scientist at Amazon, tells us about social data and it's implications.

Big Data and Government Transparency

A wealth of government data is available to us today on .gov sites and private sites across the web. If we analyzed this data properly, we could build a rich understanding of how our government works and how it could be improved. But as the big data challenge dictates, the chokepoint is consumption.


Gartner splits the 2014 Business Intelligence Magic Quadrant in two.

In an interesting turn, Gartner decided this year to split the annual Magic Quadrant for Business Intelligence and Analytics Platforms.

New Gartner Magic Quadrant: Advanced Analytics Platforms

The big story this year is how Gartner split the Business

How RoboCharm is using data to optimize customer interactions

Let’s face it, the ultimate goal for any use of data is to drive profits, and more often than not that comes back to learning how to enga