Posted on 19/05/2017 by
I was fortunate enough to attend Computing’s 6th annual Big Data & IoT summit this past Wednesday. In my opinion, the event was a great success and it was a privilege to see how so many different businesses are pursuing successful Big Data strategies. I’ve compiled a short summary below of some of the main themes discussed and what I took away from the summit.
The changing landscape of Big Data
Stuart Sumner, Editorial Director of Computing, V3 and The Inquirer, opened the conference by commenting on the clear development of the use and understanding of Big Data and how Big Data has been discussed at past conferences. He mentioned how the conversation has moved from what Big Data is, to how to build a business case for it, extracting the data, overcoming storage issues, and now focusses much more on analytics and deriving useful insights from Big Data. This has raised the issue of how to verify large data sets, with Peter Gothard at Incisive Business Enterprise Technology providing an example of how sentiment analysis on Twitter was distorted by bots.
Of course, different organisations are at different stages in the Big Data process. Jamie Shiers, a Data Preservation Project Leader at CERN, explained how CERN has managed enormous quantities of data for decades – from hundreds of terabytes in the early 90s to 50 petabytes every hundred days in 2017. This poses challenges aside from preserving the data, including issues with CPU, storage, documentation and software.
It is a long, arduous journey to implement a truly Data Science enabled Big Data strategy in a scalable, distributed system and only a few organisations have reached that destination so far.
Data Science – business case and democratising access
A number of speakers, panellists and audience members expressed the need to carefully build a business case for Big Data within their organisations. Naturally, companies want insights to be rapidly monetised. Many of the benefits of Big Data and Data Science are realised over a period of several years, so implementations need to be adapted to demonstrate gains more quickly. One way of doing this is by starting small rather than aiming for the moon, so that you can generate profitable insights early on. A number of companies have learned the hard way after implementing Hadoop and building a data lake, that many of the benefits of Big Data would not be realised until they built a Data Science team.
There was also a general consensus on the need to showcase the profitable insights you can glean from Big Data, using pictures or visualisations where necessary, to show the Return on Investment (ROI) and to meet internal expectations. This is directly linked to democratising access to Data Science, making it easy for anyone in the company to generate reports. If anyone can do it, there will be a greater diversity of insights and more people will be convinced of the benefits of Data Science.
Internet of Things – Big Data on steroids
Another clear message I took away from the conference was the vast array of possibilities the Internet of Things has opened up within Big Data. With billions of sensors collecting data in real-time, more data will be processed than ever before, moving at much higher speeds. Dr. Rick Robinson, the Director of Technology at Amey, spoke about the impact this will have with the emergence of smart cities. Traditionally it has been very difficult to gather data in the built environment, as it involves moving away from customer data generated by websites to information generated by tangible materials. The presence of so many sensors and the correlations between them completely changes this idea and it has the potential to transform engineering. Moreover, it can improve public services, public access to technology and social outcomes.
We also heard from Dr Kevin Findlay, Former IT & Digital Board Director at Complete Cover Group, who explained the progress the insurance industry has made in effectively pricing risk and improving road safety with the use of black box technology.
As the Internet of Things becomes more and more a part of daily life, security risks pertaining to IoT will also grow and change quickly. Cyber security will need to be built-in from the start, as developing security features will become increasingly difficult once everything is up and running.
Cyber security – an evolving threat landscape
Peppa Wise of Darktrace gave an interesting presentation on the advancing threat landscape in the world of Big Data. She discussed how computer viruses are now being built to evolve and escape detection, as shown by the recent cyber-attack on the NHS. Darktrace has made substantial advances in this field by bringing machine learning into their cyber defence system. This development in engineering has enabled their system to compare different devices and discover threats that may otherwise have been missed in real-time.
Skills – a growing shortage
The challenge that the current skills shortage poses was mentioned frequently throughout the day. The skills shortage is particularly severe outside of London, motivating some companies to move all or part of their analytics function to London, helping them find & retain talented staff. The industry is addressing this in a number of ways, with many recognising the need to engage with young people about Big Data and creatively attract existing candidates. In a promising development, several companies have fostered partnerships with universities to help students join the industry after graduation. There is also a wide range of excellent consultancies and vendors who provide Big Data expertise.
How is your organisation responding to the opportunities created by Big Data and Data Science? I would love to have a chat with you about how your company has growing your Data Science and Big Data capabilities. Connect with me on LinkedIn and also read the full article here.