Nextdata emerged out of stealth mode this week with a value proposition and technology designed to make the datamesh a reality. Up until today, my point of view on “datamesh” has been mostly sarcastic, laughing at the hype, and doing my best to be truthful with clients and vendors that there is no such thing as a data mesh. In fact, until today, I would say that there wasn’t even a single definition of “datamesh” since every vendor laying a claim to it, had created their own definition. This week, everything changed.
Zhamek Dahghani first developed the concept of the datamesh back in 2018. She released the book, Data Mesh in April of 2022. At its core, her concept was all about the democratization of data, making data available to everyone and everything, everywhere. She announced her new company, Nextdata, on January 17, 2023.
In her blog post to announce the new company, Dahghani says, “The value promised from data hasn’t been realized, primarily because today’s ways of managing data for analytics, AI, and machine learning were conceived in a paradigm where data was treated like oil: as a precious resource to be extracted, pumped via pipelines to a storage facility, processed, and consumed.”
She goes on to explain a series of different challenges created by the old paradigm, describing how fragile data pipelines break, processes are fragmented, and data producers spend their time on low-value processes. While these are problems that are common to the current state of data and analytics, the concept of a data mesh would go a long way towards alleviating the pain felt by most data professionals, if it were to be realized.
Data that is poured into a data lake tends to lack organization and structure; and it therefore requires a specialized skill set to access the data. Data ingested into a data warehouse is more orderly, however, the work involved in creating the data warehouse can be excessive. To address these issues, every major vendor is attempting to unify access data either from a platform or from an access perspective.
From what I can tell, Nextdata is attempting to address this problem from the platform perspective. According to Dahghani, “Nextdata OS is the data mesh toolset designed to decentralize analytical data at a large scale — within and across organizations. We are introducing the data product container, a new unit of data value designed to be responsibly shared and used at scale. We are reimagining a new developer and user experience, native to data mesh and centered around data products.”
Along with her book, Dahghani has done a great job of articulating the principles of the data mesh in her Martin Fowler sponsored blog posts, Data Mesh Principles and Logical Architecture and How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh. However, at this point, there is very little information on how Nextdata will deliver on the promise of the datamesh.
The website defines four areas of the Nextdata technology at a high level: data product containers, analytical data product APIs, embedded computational policies, and data product dynamic discoveries.
- Data product containers package data services or analytical services making them easy to understand, share, and use in the deployment of new applications without additional effort. Data is containerized, much like applications are containerized in Kubernetes.
- Analytical data product APIs make it simple for data consumers to access analytical and machine learning services, and to apply those services to their own unique data or application.
- Embedded computational policies mandates governance for every data service so that every service includes “policy as code” at its core.
- Data product dynamic discoveries make data services easily discoverable and observable based on the automatic generation of semantic, definitive, qualitative, and quantitative data.
There are several areas of the data mesh technology that remain a mystery in the Nextdata announcement, including underlying technologies and architecture. In order to deliver on the promise of the data mesh for modern digital businesses they will need to demonstrate their ability to meet all the requirements of unified analytics, including the ability to handle all types of data, meeting all latency requirements, covering all aspects of analytics, with suitability for all data professionals, and with enterprise-ready elastic scalability.
While the announcement is hopeful and Nextdata is a company to watch, I continue to question the focus on data and data platforms when the real issue is making insight available to decision makers without concern for data and data platforms. Sharing data moves upstream from the legacy focus on storing data. A truly revolutionary advancement will take the focus off of data and move upstream one more level to sharing insight and automating decisions. I will wait and see if the data mesh enables the much needed and long awaited “decision mesh.”
Leave your comments here, or feel free to drop me an email at john@ferraroconsulting.com
© Ferraro Consulting, 2024