Jun 07 2010
The era of big data is upon us. Over the past 20 years, we have seen a huge increase in the amount of data that the world produces. And it continues to grow. IDC estimates (PDF) that the world will have 988 exabytes (988 billion gigabytes) this year. That’s a 4x increase since 2006, and a forecast of 57% CAGR moving forward.
And that’s just the beginning.
We can’t even yet conceive of the new uses of all of this information. But they are coming, and they will bring both problems and opportunities.
The Problems of Big Data
- Big data requires big resources: 2 Google searches create the same amount of carbon as boiling a kettle of water. And Google routes 90 billion searches every month right now. Data centers are a major consumer of electricity – about 0.5% of all US consumption currently. But, with the amount of data exploding, and the number of computer cycles increasing correspondingly, the amount of energy consumption the world spends on big data is set to go through the roof. That’s part of the reason tech entrepreneurs are moving into clean energy technologies – they know how much the demand for energy will increase.
- Big data requires more advanced analysis: Most business analysis is currently done with Excel. But even with Excel, the process is still quite rudimentary: someone has a hypotheses about how a process is performing; they look for some basic data points within a CSV, maybe throw a few pivot tables together, and make a conclusion. That’s impossible with big data – there are too many variables. So, new services will have to be created.
- Big data relies on interconnectedness: Networks are what make big data possible. Facebook does not have tons of data because it went out and hand-entered a bunch of information into spreadsheets and databases. They created a network, and a bunch of volunteers did that work for them. Google indexes networks moving from link to link automatically. That means that if people, companies or governments opt out of the network, the data can have huge holes.
The Opportunities of Big Data
- Big data creates big insights: There are a number of tools that are coming online to mine information from big data sets. And they will lead to some great insights. Think super-personalized messaging and information delivery. Think more efficient government services, with police and EMTs pre-stationed at the locations that are most likely to need them. Think of fewer failed marketing experiments because the product never had enough interest to be viable. Think custom, real-time, super-fast assembly of complicated machines. Think cheaper and more efficient transportation. Think answers to seemingly intractable problems like finding intelligent life in the universe and disease management.
- Big data creates new layers of meaning: Right now it is difficult to know how many toasters are turned on at any given time. but, as more and more devices are connected to the internet, they will start generating their own data. That data can be aggregated and we can make meaning out of it. Why might we want to know how many toasters are on? Maybe we want to identify ways to green our neighborhood or city (either from a civic standpoint, or as part of demand side management in the power supply). Maybe we want to make better recommendations about what to do with day old bread (the old toast vs. croutons debate may finally be put to rest). Maybe we want to make predictions about the rising or falling of carb consumption in American households. The opportunities are enormous.
- Big data will fuel the next wave of cultural relevance: Culture moves forward by contextualizing new information. Humans assimilate and find value out of technological developments much more slowly than they are created. That means that all of this data will continue exploding for a while. We will feel like we are drowning in information. There will not be efficient ways to manage it all. Then, suddenly, perspectives will shift. Data will be ubiquitous, and therefore, taken for granted. We will not be able to consider making decisions without it.