While data is probably one of the most traditional tools businesses have used over the decades to compete more effectively, the advent of Big Data, meshed with advanced analytics, is now re-shaping the business environment in ways we have never experienced.

Unquestionably, the driver of this change is technology and it’s not only creating significant opportunities for businesses and consumers, it’s also creating massive career opportunities in virtually every organisation across all industries. LinkedIn voted “statistical analysis and data mining” the top skill that got people hired in 2014.

The increasing volume and detail of information captured by companies, the rise of social media, the use of cookies through to the Internet of Things are all fuelling exponential growth in data and will continue to do so for the foreseeable future.

It’s not only within industry that leveraging data-driven strategies will improve innovation and capture value; harnessing Big Data by professional sports teams and in the public sector also has enormous potential.

>See also: Can Tech City rival Silicon Valley as a tech start-up incubator?

Global management consulting firm McKinsey believes that European governments could save more than €100 billion in operational efficiency improvements alone by using Big Data.

Big data’s elusive nature

But what is Big Data and how can it possibly be so important? Despite being used widely in commercial and academic circles, a definitive description remains conceptually vague. Big Data might more aptly be called gigantic data. It can be defined as huge, complex data sets, that require advanced data analytics and algorithmic tools to enable its management and processing within an acceptable timeframe to generate meaningful outcomes.

Even though Big Data is becoming a crucial factor in companies achieving competitive advantage, many are failing to achieve the results they want. There are a number of reasons for this. First, Big Data has been so hyped that business leaders are expecting it to deliver more than it actually can. The problem isn’t essentially with the data but how the data are interpreted, particularly if they are derived from multiple sources. Furthermore, capitalising on the insights from data analytics may require fundamental changes to the business model that the organisation may not be capable of making.

Secondly, businesses often don’t make the most of the information they already have because they don’t have the skills to manage it, analyse it in a way that enhances their existing knowledge and then make the necessary behavioural changes to leverage the competitive advantages of their new, enhanced insights. Simply investing in high-end data analytic tools won’t achieve the required competencies.

Considering the sheer volume and the different formats of Big Data (both structured and unstructured) that are collected across an entire organisation and the many ways different types of data can be combined, contrasted and analysed to find patterns and other useful business information, Big Data analysis is a challenge for most businesses.

Enhancing customer experience

Herein lies an even bigger challenge. Capturing opportunities to enhance customer experience, improve process efficiency and launch new products and business models using Big Data not only requires an investment in information infrastructure, but also an investment in new talent and training/development programmes.

The starting point to address this challenge is for business leaders to define the business outcomes that they want to achieve from the data and to agree who is going to take responsibility for the analysis of the information provided.

>Related: Emotional intelligence in leadership

Since Big Data spans a wide range of corporate functions from IT, marketing, sales, HR, customer services through to risk and operations, whoever is appointed to take charge of Big Data needs to have multidisciplinary skills and strong leadership experience to influence and inspire appropriate action.

Increasingly, we are finding positions such as Chief Data Officer, Chief Data Architect, VP Global Data Services and Head of Global Analytics appearing in executive search assignments. This “Head of Big Data” – unlike a Chief Information Officer or Chief Digital Officer – will have significant business responsibility for determining what kinds of information the organisation needs to capture, retain and capitalise on and for what purposes.

What does the consummate head of data look like? Such is the importance of this position that business leaders will tend to go out of their way to retain data scientists with significant academic experience and who have also held senior line management positions.

Business background over industry history

Industry is less important than business background and he or she may well come from marketing, IT, sales, engineering, manufacturing or HR. A key requirement is substantial strategic planning experience. In many cases, the position will also report into the C-Suit.But as important as anything else is the ability to bring a clarity of commercial vision to the table. People don’t need to drown in the data but to have the most salient commercial insights extrapolated therefrom.

Do these people exist? Yes they do, but as might be expected the pool of experienced talent is not deep. Some will be found in high-tech companies, pharma, large IT service companies and possibly in a small number of startups, amongst others. One thing the majority will have in common, however, is that they will only work in companies that are innovative, able to provide a real challenge, have really interesting data assets and possess the latest technologies.

Digital gurus may be tempted to join an organisation that is only starting down the Big Data highway, but they will need to be convinced that the entire company is undergoing a cultural shift to become more customer-centric. Additionally, they need to have no doubts that they can be a big part of what the organisation is trying to accomplish.

One characteristic of data scientists is ‘curiosity’, so appealing to this inquisitiveness, convincing them that the opportunities are boundless and standing by that, increases the recruiting success rate – and the “stick rate” thereafter.

And as might also be expected, the competition to recruit this top talent is intense. In the US alone McKinsey has predicted that, by 2018, there will be a shortage of 140,000 to 190,000 people with deep analytic skills, as well as 1.5 million managers and analysts with the know-how to use the analysis of Big Data to make effective decisions.

Compensation is important but it is not the principal factor in recruiting top talent. A compelling position in an innovation-driven organisation with lots of complicated challenges to meet will win out over remuneration every time.

As technology develops, Big Data will help to create new growth opportunities for existing companies and entirely new categories for others. Hiring Big Data specialists will, for the majority, be a challenging process and one that demands significant input from the hiring company. However, those prepared to put in the groundwork, be adaptable and innovative, will reap the benefits.

Further reading: Beyond big data – the evolution of predictive analytics