Reshaping the Data Lake

By: Ori Reshef

The term ‘big data’ has been around since the 1990s and companies have certainly been prioritizing big data investments for almost as long. Still, according to a recent 2021 NewVantage Partners survey of Fortune 1000 executives, enterprises are continuing to struggle to derive value from their big data investments. Only 48.5 percent are driving innovation with data. Just 41.2 percent are competing on analytics. And only 24 percent have created a data-driven organization. 

Over the past decade, enterprise data analytics attention has shifted away from the data warehouse architecture to the data lake architecture. There are various schools of thought on what the modern data lake stack and data lake architecture ought to look like. Many organizations have failed to realize ROI from their data initiatives, largely due to unplanned and unsustainable costs of modeling data and DataOps.

Despite the challenges, there are powerful reasons for organizations to care about achieving big gains in ROI from their data lake investments. Think about pharmaceutical companies looking for the next vaccine candidate. Or consider financial services firms striving to stay ahead of market fluctuations. Media firms want to discover which pieces of content each user is likely to binge next, and security teams want to conduct analytics on the security data lake with greater speed and precision. The common thread between all these strategic initiatives is the ability to analyze as much data as possible, with optimal flexibility and agility. Data users demand to run any query, whenever they need it. In this case, using data warehouse solutions will not deliver the needed agility and flexibility. The ability to quickly transition to a data lake architecture will deliver these benefits as well as a strategic competitive advantage.

Unlocking more powerful insights from data analytics is at the center of the data lake architecture paradigm shift. The ongoing demand for agile, more flexible data analytics to leverage big data investments has fueled the rise of data lakes and distributed SQL query engines like Presto and Trino. The power of data lakes to hold vast amounts of raw data in native formats until needed by the business, combined with the agility and flexibility of distributed engines in querying that data, promises organizations the ability to maximize data-driven growth. 

Although they have bought into the analytics promise of data lake architecture along with its ability to provide cost effectiveness and efficiency, many organizations have yet to unlock the power of data lake architecture. Instead, they are utilizing it as little more than an aggregative storage layer. 

The main value organizations derive from the data lake stack has three aspects:

1. It enables instant ease of access to their wealth of data, regardless of where it resides, with near zero time-to-market (no need for IT or data teams to prepare or move data).
   
   
2. It creates a pervasive, data-driven culture.
   
   
3. It transforms data into the digital intelligence that is a prerequisite for achieving a competitive advantage in today’s data-driven ecosystem.

To create a modern data lake architecture that maximizes ROI, forward-looking data organizations are leveraging new data virtualization, automation and acceleration strategies and reaping the benefits. How can data organizations ensure that their modern data lake stack is analytics-ready? The following are key questions that data organizations need to ask themselves to ensure they are getting the most out of their big data investments.