Investigating the Mysteries of Big Data

A few days ago, the magazine AutomationWorld published an interesting article about the future jobs that will be generated due to the digital transformation processes in which we are all involved. One of them, which gives the article its name, is that of Data Detective, a position in charge of ‘investigating the mysteries of Big Data, uncovering secrets within the content by examining sensors, devices, fog computing and neural networks.’

From my point of view, I agree that this position, with this name or another, will end up existing, although it is true that I would add a nuance. A good Data Detective should investigate why the results extracted from the data do not apply or coincide with reality.

Currently, all process and infrastructure industries are working on or evaluating data analysis tools, whether they are analytical tools, Business Intelligence, preventive maintenance, resource optimization, etc. But the approach of these integrations, for the most part, is usually done from ‘above’. This type of tool feeds on data stored in databases, and it is from these that they draw their conclusions from large processes and complex algorithms.

For the industrial and infrastructure world, there is a fundamental problem in this. This data stored in databases has not been stored there for the purpose of being exploited by this type of tool, but has been generated and contextualized for another type of application – a SCADA typically -. This differential fact implies that these tools are working on a sampling field that is not properly designed or prepared for it. In the industrial world today there is a maxim: the technological bottleneck for the use of these tools is a correct acquisition of field data.

To give an example, in an automotive parts manufacturing industry it is known that 6% of the parts are defective and are discarded at the time. That information is what reaches the repositories using their SCADA and ERP. However, the most interesting thing would be for these tools to know more about the discarded parts, their manufacturing route and the different elements that were involved, material, supplier, stamping machinery, transport belts, active personnel, etc. But in order to make these correlations, the system needs to meet certain characteristics that it will not have. Faced with this scenario, these tools will either not know how to draw a conclusion or, even worse, they will draw the wrong one.

That is why I believe that we will reach the figure of the Data Detective. I do not venture into whether it will be an internal position of the companies or external consultancies, but I am clear about its mission. Investigate why the contracted exploitation tools are not having the desired ROI, that is, once the tool and its algorithms have been reviewed and it has been verified that everything is correct at that level, where are the contextualization failures of the data sources? In its correct acquisition, in its manipulation, in the sampling frequency, in its temporal stamping, in the volume with which it is worked, in that I need to acquire information from some key elements…?

This Data Detective will most likely have to investigate, coordinate and understand the multiple actors involved in the process and that, most likely, is that in case these tools do not have the expected results, no one will feel responsible for it. And, much to our regret, they will surely all be right. If the data sources have not been properly prepared for this, a preventive maintenance application will be able to work with what it can, but it is evident that it will not be able to extract the most accurate conclusions, for example.

Therefore, to avoid having to need a Data Detective we can propose several initiatives, not mutually exclusive:

• Use a Digital Transformation Master Plan, that is, work on a joint strategy that encompasses all levels of an industry, taking into account the specific needs of each level, but also the transversal ones – cybersecurity, for example -.
• Have trusted partners, either in the tools themselves or in the services that underlie each project. It is evident that the more actors, the more people to call, the more people to coordinate, the more people to justify, etc. Having two or a single partner as Main Director of the projects does not imply that there are more people involved below, but it makes it much easier for the final client to be clear about who to call in case of need.
• Keep in mind that the process is long. Technology and solutions advance at an unaffordable speed to follow, that implies that surely our initial planning should be corrected, modulated and stretched since, surely, more needs will also arrive from the offices. That implies, and it must be aware that there will not be a single large Digital Transformation project within the company, but there will be several of small and medium size. This fact will also serve to test the confidence generated by the partners mentioned in the previous step.

In any case, if we are finally inevitably doomed to the Data Detectives, I only hope that, out of deference to the fans of the noir novel, they wear hats and suspenders… “When I saw that IT manager at my door, I knew I wasn’t going to have a quiet afternoon…”

Fernando Campos, Logitek M2M & IoT Solutions Manager