With each passing day, a company creates and captures figures and data. Every purchase, every transaction by your business teams produce information that must be managed. Tech giants like Amazon, eBay and Google have taught us that data can be a source of value creation.
We have had mainframes for a while, which, provided us with sufficient methods to calculate daily business indicators. We then used these BI tools to manage and make the right decisions at the right time.
But, this was earlier, earlier than the coming of the great innovation called the InterWeb. Before you see the power and possibilities of Big Data, what we call the 3 Vs: Large Volume, Value and Great Velocity, let us consider these points: what do you do with this data? Does your information system behave like an open faucet or like an oil well?
The first characteristic of Big Data is of course the volume of your data. What is the cost of storing this information? What do you do with it? What is the data growth factor? Do you know how much information is lost every day in your information system? Let us assume that your technical teams already use dashboards to monitor your machines: CPU, memory, input / output network, technical logs, etc. But, that is of no interest to the Marketing department. Your sales now wish to view real-time traffic to your site. More than technical data, the key is to exploit the volume of business data. This will allow you to predict and better understand your customers. Do you already have these tools? Will they be able to handle three times more data in 6 months?
Quantifying and identifying is what finally helps to define the usability of your data. Whether on the storage format, the tools used, or even your security policy, this volume can be your first source of income in a few months.
We live in a time where it is possible to record everything and store everything, yet we fail to exploit this ability to its fullest. Developers now have access to an ecosystem of technical solutions in order to create the data. Nothing is easier than adding a collector in a Web application and forward purchase or the product of choice through a dedicated system.
Earlier web developers only saw the data warehouse, often a relational database. Today, web developers must also consider storing user behavior of an application to a new system. Yes, these are also Big Data projects. In this vision the developer should be required to create value, prepare and store business information. Technical logs are good. Logs trades are even better. Are your teams adaptable to this evolution? Are simple solutions based on open industry standards being considered?
Pushing a little further, we can imagine two worlds. The first, where the CIO cannot explain to investors why nothing was stored for many years. The second, where the CIO is no longer simply an asset manager. He sits on a volume of important marketing data thus creating a magnificent Headquarters of Defence for the company. When are you insuring your data assets?
The developer has an ace to play here. Using an analogy to the world of finance, where high frequency trading has come into its own, we need to realize the potential of data the size of big data and more. For example, when will a solution be capable of running an arbitration and book an airline ticket for us at the correct time? There are possibilities but when will they be realized? As developers, we now have access to phenomenal power. A search on Redis takes only 10ms, 10 to 15 times less time than a blink of an eye. We can save time by destroying silos and intelligently storing data in a de-structured format thus creating even higher technical format. Why? Because the same memory that cost a pretty penny yesterday is practically free today with the costs having come down drastically. Another thing that makes a big difference is simplicity. And no, Hadoop is not the answer to all Big Data projects. We are still in our infancy and have yet to learn solutions from the world of intelligence, and perhaps a bit of pragmatism.
We will fight tomorrow to find technical analysts. The Hadoop expert will be courted as was his father, the expert BO before him, and his grandfather, the expert Mainframe. Tomorrow, new professions appear in the form of expert analyst, data surgeon, digital actuary data insurer or Conservative digital mortgages. But everything, absolutely everything, first has to pass through the hands of a developer.
Software vendors also have the promise of a new El Dorado. Calculations Solutions, representation, analysis, prediction, and machine learning ... We are certainly at the dawn of a new technical revolution.
Written By: Nicolas Martignole Nicolas Martignole is the Lead Developer at Captaindash. He was previously the Lead Architect at Zaptravel. He’s also the creator and co-organizer of Devoxx France, one of the biggest conferences for Java and web developers in Paris. You can reach him on Twitter on @nmartignole.