In the post-mortems from the Presidential election, the biggest news was how President Obama’s team used technology to win and how Nate Silver used technology to analyze. In both cases, the source that created the strategy to success was data, and lots of it. In general terms, it is known as “Big Data.”
The discussion of “Big Data” hit the public consciousness in a very benign way. Grocery chains were signing people up for “affinity accounts” to enable them to deliver discounts automatically on checkout. However, people concerned about information privacy hit the roof, claiming that now consumers’ purchasing habits could be tracked and marketing efforts could be targeted at them.
Most consumers didn’t care, since the benefits (lower prices) outweighed the perceived risks (a loss of privacy). Nevertheless, the nose was under the tent and dossiers on people’s habits were begun. Companies have established huge businesses based on gathering these disparate points of data and presenting them to their clients in ways that enable them to better understand those to whom they are marketing.
Concerns about these companies have grown over the years as their databases have become more and more populated and sophisticated. Privacy advocates cite an incident in 2003 where 1.6 billion customer records were stolen during a customer transaction between one of these companies and a client. During the hacker’s trial in 2006, this incident was called the “largest ever invasion and theft of personal data” ever brought to trial.
What is intriguing is that the discovery of the theft only occurred as part of the investigation of a different incident. Assuming that the companies in the “Big Data” field would be reticent about making public any other breaches, we really don’t know how secure this growing mountain of personal information is.
As with the Presidential campaign, most of the information in these databases was provided with the approval of the person providing it. This is known as data which has been “opted–in.” That means that somewhere along the line, such as signing up for the grocery store membership program, the consumer agreed to this sharing of data.
Now, this is different than a consumer “opting-out” of getting promotional materials. That is the usual option offered to customers. You see it on just about every web site or in every membership promotion. But, this does not generally “opt you out” of data sharing.
Law enforcement has gotten into this information data stream in a number of ways, ranging from license plate recognition to GPS tracking of smartphones. For the most part, this data has been sequestered behind “firewalls” which keep the information separate from other public databases. However, the companies that do the public database collection very often are contracted to law enforcement and do relational analysis, so that a potential hole in that firewall exists.
Most people when confronted with this issue say, “Well, I have nothing to hide.” Perhaps, but the real question is: Who should be responsible for keeping it hidden, and how good is their ability to keep it that way?
