Data scientist critical to an organisation’s success, says Teradata CTO

A data scientist today was critical for an organisation’s success but data science as a discipline could not be up-skilled to from traditional IT skill sets, and because it was a relatively new discipline, the demand for data scientists far outweighed the supply for now.

Speaking at the Teradata annual summit in Sydney recently, the company's Chief Technology Officer (CTO) Stephen Brobst extolled the criticality of the data scientist to business success. He pointed out that the way business decisions were being taken these days was changing. That was because organisations were being exposed to and had access to more data than ever before. Organisations no longer could only rely on executive opinions to take decisions, and had to rely on the various pieces of information collected from various sources.

Elaborating, Mr Brobst said because of the amount of data available to organisations, they were now able to ask questions that they were not able to do so in the past. The real challenge for most organisations though, was knowing which questions to ask, he added. “This part can be quite tricky. Simply having data does not make one a data-driven organisation, so they have to be willing to experiment on the data to find what information it will yield,” he said.

An organisation has to have some sense of what it was looking to accomplish in order to know which questions to ask, which in itself could be a big challenge. This type of data interrogation and manipulation was not typical of what traditional IT skill sets were capable of, and required a fair degree of creativity and artistry curiosity.

Organisations, said the Teradata CTO, needed to have an appetite to explore their data. They needed to also have good intuition to know which direction let their data take them in, understand the causality and correlation of data, and be able to understand and articulate its value and statistical significance towards the organisation’s expectations.

Mr Brobst stressed the importance of leveraging the existing as well as the new sources of data for business success. Therefore it was critical for organisations to have people who were able to work outside the constraints of corporate protocol to successfully bring the two worlds together. These were the types of people who Mr Brobst said, would not wait for IT department or CFO approval before diving head first into the task. Hence the role of the data scientist had become a critical factor for an organisation’s success, he felt.

There was a growing expectation in the market that technologies would need to evolve to a point where many of the processes of data science would have to be automated. But this kind of machine learning may take as many years to perfect as closing the skills gap, he added.

Mr Brobst felt that though technology did have a part to play in data science, it presently lacked the human intuition required to explore the different directions of data in the context of designing the right questions. Many of the traditional data tools were inadequate for the job. Mr Brobst also acknowledged that the shortage of data scientists was an issue and stressed how important it was not only for the business community, but society in general to encourage data science as a viable career option.

The BigInsights’ Take:

With a plethora of venture backed, Hadoop-based applications emerging in the market, it will be interesting to see how much automation and machine learning will take hold in a technological landscape offering unlimited opportunities. On the one hand, these applications may be seen as the answer to a shortage of available skills. However, we may yet be several years away from the technology that facilitate the explorative intuition that presently come only at the human touch.

About Shayum Rahim: Shayum Rahim is the Research Director at BigInsights and formerly Head of Australian Software Research at IDC. Shayum has previously held senior roles at JD Edwards, Oracle and Microsoft. He has spoken about Big Data at several industry events and has authored several reports and articles on Big Data. He can be reached via shayum@biginsights.co and can be followed @ShayumRahim (Twitter).

About BigInsights: BigInsights is an Australia-based research & advisory firm focused on Big Data Analytics. Its express aim is to help companies with best practices and ROI on using Big Data technologies for customer and operational insights, and to help them track emerging trends in new Big Data technologies.