[#13] The Drive to Become Data-Driven
[#13] The Drive to Become Data-Driven
As companies continue to move towards becoming more data-driven, what are the opportunities and challenges for data professionals?
I recently came across several interesting articles about the drive to become more data-driven among companies from all industries (not just tech companies) and obstacles that are hindering this drive.
Closely connected to this shift towards more data-driven organizations, there has been talk for quite a while about how in-demand data scientists are and how many people are trying to break into a data science career (this is evident by the proliferation of data science courses on Coursera and the numerous other websites and bootcamps devoted to training data scientists).
But, let’s take a step back and see what’s happening now in industry, in general, in terms of the shift towards more data-driven organizations.
The Drive to Become Data-Driven
It is clear that companies, even ones that have been around since long before data science was a hot topic, are trying to become more data-driven. A report in Harvard Business Review on an annual survey of executives from a selection of Fortune 1000 companies illustrates how companies from various industries are prioritizing the shift towards becoming more heavily data-driven.
For example, the number of companies that have hired, or appointed, a dedicated Chief Data Officer has increased from 12% in 2012 to 65% in 2021. Furthermore, 62% of the companies surveyed report data and AI investments of at least $50 million.
The Coronavirus pandemic may even be accelerating this trend, as “big data is now mission-critical to more and more businesses, not just the tech giants.”1
While many businesses eliminated non-essential spending in response to the pandemic, companies continued to invest in B2B software applications that give them more insight into their data such as project management tools, business intelligence, and marketing automation, with 27% reporting that spending had increased in 2020. [1]
However, despite this motivation to invest heavily in data-driven initiatives, these Fortune 1000 companies that were surveyed report that they are still struggling to derive value from these investments.
Often saddled with legacy data environments, business processes, skill sets, and traditional cultures that can be reluctant to change, mainstream companies appear to be confronting greater challenges as demands increase, data volumes grow, and companies seek to mature their data capabilities.2
Surprisingly, the fraction of survey respondents who felt that their organization was “data-driven” decreased to 24% from 38% the previous year.
Based on what I have read, I think the issues impeding many companies’ transition to being truly data-driven can be boiled down to two different, perhaps contradictory, bottlenecks. As data professionals, or aspiring data professionals, it is important to keep our fingers on the pulse of what is a rapidly changing environment and see where opportunities, as well as challenges, lie.
Bottleneck #1: Talent Shortage
The first bottleneck is simply the shortage of experienced data professionals. Obviously, this can be addressed by either training current employees and/or hiring people specifically for data-related roles.
In terms of existing employees, a survey “by Accenture of more than 9,000 employees in a variety of roles found that only 21% were confident in their data literacy skills.”3 Thus, there is clearly a lot more work to be done within companies to build-up current employees’ data skills.
What about hiring new people with data skills? This is a challenge because, according to QuantHub, there was a whopping shortage of 250,000 data science professionals in 2020.
But, what about all the thousands of people enrolled in online data science courses, bootcamps, even data science degrees?
One problem is that there appears to be a disconnect between what many companies are looking for versus what job seekers are training for.
According to a report on Springboard:
As automation takes over the menial parts of a data scientist’s job (80% of a data scientist’s time is spent cleaning data), the bar for landing a data science job is both higher and lower… Companies seek to fill lower-paying data analyst roles that require fewer years of experience, while also hiring highly skilled data scientists with domain expertise who can solve a specific business problem—like determining risk factors for certain diseases or increasing voter turnout.
Therefore, while many aspiring data professionals are training in data science, and seeking data scientist roles, many companies look to people with at least a few years of on-the-job data science experience for those roles.
So, the shortage of experienced data scientists in the job market, combined with the disconnect between the skills job-seekers are trying to acquire and what companies are looking for in entry-level data roles, is creating a huge bottleneck in companies attempts to grow their data workforce.
Bottleneck #2: Culture
Going back to the article from Harvard Business Review:
For the fifth consecutive year, executives report that cultural challenges — not technological ones — represent the biggest impediment around data initiatives.
From the survey of Fortune 1000 companies, 92% of mainstream companies report such cultural challenges “relating to organizational alignment, business processes, change management, communication, people skill sets, and resistance or lack of understanding to enable change”.
So, even when companies actively try to recruit and bring in data professionals to move the company forward on data-driven initiatives, there is still resistance within the organization to fully commit to these changes.
Opportunities and Challenges for Data Professionals
It goes without saying that there are a growing number of job opportunities for people with experience in data analysis, data science, and data engineering. One suggestion from the Springboard report for job-seekers is “to find work as a data analyst but then to offer to help the company tackle machine learning problems to get real-world experience in that domain.”
Even if your job title doesn’t explicitly have “data” in the name, there is clearly a desire to infuse data into many different roles:
[T]he demand for data-literate business professionals is expanding beyond traditional data science jobs. Companies seek candidates with analytics skills, such as data-minded digital marketers (hence the term “growth marketing”), HR professionals, account managers, and financial consultants: people who can query data for business insights, A/B test different approaches, track performance metrics, and show how they’re adding value to the bottom line. [1]
The struggles companies are having to become more data-driven, coupled with the clear desire to make progress in this space, means that there is clearly an opportunity for those of us with data experience to make a meaningful contribution in these organizations that are still figuring out this shift.
It is important, of course, to keep in mind that if you get a job working at a legacy company that is trying to become more data-driven, be aware of how culture might get in the way. There may be some frustration in making progress.
One recommendation, from the Harvard Business Review report, to address this challenge is to go after low-hanging fruit. Find a business problem that can be solved easily by using data in a way that the company has not thought of before, even if it’s not some sophisticated machine learning or AI solution. This could help build confidence across the organization on the path towards becoming more data-driven. Plus, if it’s a solution that people in the organization who are less confident in their data literacy can understand, it might help boost their confidence and really see, in a concrete way, how data-driven initiatives can actually help the company to succeed.
Data Science Resources
If you wanted to see a wide range of data science concepts in a concise, four-page PDF, check out this data science “cheatsheet”.