The Four Pillars of Data Mesh: A Paradigm Shift in Organizational Data Strategy
Key Highlights
- Data mesh decentralizes data ownership, enabling domain teams to control and manage their data life cycle for faster access and improved quality.
- Treating data as a product ensures standardization, discoverability and trustworthiness, with dedicated product managers overseeing data assets.
- A self-service platform provides automated, user-friendly tools that allow teams to independently build, deploy and monitor data products, fostering collaboration.
- Federated governance automates compliance and security, ensuring organizational policies are consistently enforced across all data domains.
- Successful adoption of data mesh requires cultural change, clear communication, executive leadership and strategic use of technology platforms.
Data remains the lifeblood that drives technology innovations. But in the current AI era, enterprises continue to grapple with organizing massive amounts of unclassified data. According to research, 70% of enterprises store petabytes of largely unmanaged and unstructured information, and that has implications for effectively operationalizing AI initiatives.
A data mesh approach offers a methodology for ascribing data ownership and responsibilities to offset that complexity. It also represents a paradigm shift in organizational structure and data management because, instead of IT-controlled, centralized data lakes, it relies on domain-specific business teams to own and manage their data as a product.
It also incorporates federated governance to protect and standardize how data is managed internally, ensuring consistency across the entire organization. In this article, we look at the four pillars used to establish a data mesh as well as common use cases and key steps required for implementation.
What is a data mesh, and why does AI make it more important?
In the past, IT teams were largely responsible for controlling centralized repositories where business intelligence and customer-focused data were stored. They would then process that information and send it on to business teams. However, this approach entails significant drawbacks, from data processing bottlenecks to long wait times for data delivery. These inefficiencies have become particularly acute as enterprises increasingly deploy AI, which requires highly organized, quantified information delivered quickly and consistently.
The data mesh framework was first introduced in 2019 by computer scientist Zhamak Dehghani, who devised a decentralized method for overcoming these deficits. The primary innovation of this approach was to prioritize the producers closest to the business context in which the data originated. Those business teams responsible for the targeted information are best equipped to define data quality. In terms of AI deployments, these domain entities function as the best sources for documentation and establishing lines of responsibility.
“The data mesh approach is mainly aimed at improving the quality and currency of the data — the contextual relevance,” says Brian Remmington, chief software architect for the information management platform M-Files. “This, in turn, improves the basis on which the AI model can reason, and that improves the business outcome the AI agents can achieve,” he adds.
Business domain end users are also in the best position to evaluate service level agreements (SLAs) based on treating that data as a product. In other words, they can measure the reliability and effectiveness of that information from a producer perspective, ensuring that it meets consumers’ needs. Teams can also use targeted KPIs to gauge whether AI deployments are meeting targeted goals.
In general, companies are adopting a data mesh approach to eliminate centralized storage bottlenecks and to scale analytics horizontally across complex business domains. Organizations gain the advantage of improving data quality by making the experts closest to the data responsible for its maintenance. And since different business teams maintain their own analytics infrastructure, they not only achieve greater speed and agility when handling data, they also reinforce company-wide policies for data handling.
Common data mesh use cases across industries
Of course, there’s a diversity of data mesh use cases across industries. For example, in finance, fraud detection engines can instantly query real-time payment data from the processing domain and combine it with separate customer behavior models to calculate risk across domains.
And in manufacturing, instead of funneling all factory sensor streams to a central repository, each assembly line processes its own IoT data (e.g., machine vibrations, temperature, etc.) to make the data instantly discoverable. Finally, in retail, data consumers can use separate marketing and operations data to weave together the customer journey (from a marketing offer to final delivery) and bypass centralized engineering queues.
“The reason most organizations choose to go down the path of a data mesh is to ensure the quality and currency of the data on which the organization bases decisions. The idea is that teams that own the data are the teams that understand the meaning of that data, and so they should own the curation, packaging and publishing of that data,” says Remmington.
The four pillars of a data mesh architecture
Four key principles underlie a data mesh approach, and they function to balance the independence of individual business groups while ensuring widespread interoperability for the use of that data:
1. Decentralized Data Ownership
Decentralized data ownership is a key tenet of a data mesh architecture. Domain-specific data managers retain full responsibility and autonomy to control select information across its life cycle. This proximity to targeted data simplifies and accelerates strategic initiatives because other business teams can quickly access the correct data by directly contacting domain managers for requests, changes and approvals.
2. Data as a Product
Data as a product means that all information assets are standardized, discoverable and reliable, ensuring that data consumers (e.g., data scientists, ML engineers, business leaders, etc.) always receive quality information. Targeted data is required to be discoverable, trustworthy and interoperable. Along with metadata, these assets can include source code, dashboards, features, models and additional components necessary to maintain that data product. Ultimately, data as a product is an operational model with a product manager who treats internal consumers as though they were customers.
3. Self-Service Platform
A self-serve data infrastructure consists of an automated platform to build, run and maintain interoperable data products. Domain teams can access easy-to-use, developer-friendly tools within the framework to independently build, deploy and monitor data products. They can also use the platform to scale the data mesh architecture as data volumes, user bases and business units expand. Moreover, a flexible, multi-use platform enables collaboration between different teams and improves interoperability across all data and AI workloads.
4. Federated Governance
Within a data mesh, federated governance functions to ensure that compliance, security and data quality rules are codified, automated and equally enforced. It not only ensures that a data ecosystem adheres to organizational rules and industry regulations, but also enables domain teams to track and centrally manage compliance using data catalogs, data governance tools and automated policy enforcement. A federated sharing layer means that teams and business leaders can shift governance decisions from time-consuming deliberations to fast execution.
A practical approach to data mesh implementation
For C-suite and business leaders who want to undertake a data mesh, it’s important to keep in mind that adoption entails a people-and-process reorientation to data efficiency. A key aspect of data mesh is decoupling traditional, centralized data management processes and creating a structured way to define, measure and govern data as a product.
“With data mesh, you need to decide what problem you're trying to solve, then review what datasets make sense to bring into the mesh to solve that problem. Avoid bringing too much data into the mesh at the start — use data for one particular use case, then grow from there,” offers Remmington.
Moreover, it’s also clear that foundational practices and approaches to handling data can go a long way towards determining adoption success. These include data management best practices, such as cataloging, lineage, SLAs and federated governance, as well as continuous monitoring of data integrity, adoption goals and business outcomes.
In tandem with that realignment of how data is handled within an organization, vendor platforms can help orchestrate the process. These frameworks function to address key aspects, such as federated data access to disparate storage (e.g., Starburst, Denodo, etc.), domain storage (e.g. Data bricks, Snowflake, etc.), and self-service platforms (e.g., K2View, Callibra, Atlan, etc.).
Why data mesh adoption is often a culture challenge
That said, for many enterprises, technology may not be the biggest challenge. Instead, the business culture shift can pose substantial obstacles to smooth realignment. A change in established processes requires consistent communication with all business teams and clear demonstration of the value of these framework tools. Successful execution depends on strong executive leadership and a well-defined architecture with standardized implementation guidelines.
“Consider which teams in the organization are best set up for success: Which have good data practices already and have the capability to own their data as a product? Which are culturally best aligned to change?” asks Remmington.
About the Author
Kerry Doyle
Contributor
Kerry Doyle focuses primarily on issues relevant to both C-suite and enterprise leaders through technology articles, white papers and analyses. He covers a diverse range of topics, from nanotech to the cloud, open source to AI. Passionate about both the written word and communicating the value of technology, his experience stems from senior editorial positions at PCWeek, PCComputing, ZDNet, and CNet.com. He's a graduate of Boston University with a bachelor's degree in comparative literature.
Resources
Quiz
Your Competitive Edge, DeliveredStay ahead of the curve with weekly insights into emerging technologies, cybersecurity, and digital transformation. TechEDGE brings you expert perspectives, real-world applications, and the innovations driving tomorrow’s breakthroughs, so you’re always equipped to lead the next wave of change.
