Sometimes the solution is right in front of us but most people fail to find the solution because of perspective. Even the best and the smartest people often fail as well. It is hard to see the solution because the solution is buried inside a sea of data.

What am I talking about? I am talking about business opportunities, cures for diseases, new scientific discoveries, inventions, etc. The success of Google began because there were no real good search engines. Most were mediocre at best. When you did a web search, most results returned were unusable or irrelevant. Google had the “secret sauce”. They were able to take your query and “mine” this vast sea of data called the Internet and return a list of mostly relevant list of search results. Google was able to do what nobody else could do that is pinpoint and uncover data on the Internet. This concept is just the cornerstone of the future.

Most companies do not understand this concept and is constantly looking for answers with the little data they have available. It isn’t the availability but the ability to look outside for answers is the key to having a competitive edge. I have recently done some consulting for a company that built its business on this concept. It is a company that takes vast information of people’s cell phone activities and using their software to look for trends and patterns. If the trends and patterns fall outside of the “norm” (user defined), it is flagged for unusual activity. At some point, these calling patterns can identify possible fraud. This is great stuff, but can you imaging by taking this one step further, I believe if this data is correlated with the justice department it can link unsolved crimes or give our criminal investigators an edge in finding and prosecuting existing crimes. This is just one example of “Big Data”.

Big Data is a term that describes that ability to take vast amounts of data and by using technology (hardware and software) to massage and mine it for new and valuable information. This is “trail-blazing” and discoveries are yet to be uncovered. That is the benefits of investing in a “Big Data” infrastructure. This is all possible because of large storage devices and the HPC (High Performance Computing). This is currently being used in geo-physical and nuclear research, but why limit this concept to those industries? As “Big Data” mining is slowly being adopted, we are just scratching the surface to what can be uncovered.

Believing that there is order in the midst of chaos is where it all begins.