A data fabric is an information architecture that unifies data across an organization. According to Techopedia, data fabric describes “a distributed IT architecture in which data is governed the same way whether it is located on premises, in the cloud, or at the edge of a network”.1
Rather than being a single piece of technology, data fabric is a holistic data and AI strategy that helps organizations leverage all existing and future investments within their data estate. The purpose of a unified data fabric is to ensure that an organization's data is always accessible to all authorized parties, regardless of where it is stored.
In the Gartner article “Data Fabric Architecture is Key to Modernizing Data Management and Integration”, author Ashutosh Gupta notes that data fabric “leverages both human and machine capabilities to access data in place or support its consolidation where appropriate [and it] continuously identifies and connects data from disparate applications to discover unique, business-relevant relationships between the available data points”.2
Data Fabric for Dummies author Ed Tittel states that “data fabric is a living, always-evolving collection of capabilities that grows and changes along with whatever organization it serves”.3
Why is data fabric important today?
One of today’s biggest business challenges is figuring out how to close the critical gap between the available data and information, and transforming as much of that data and info into resources like knowledge and insights. These insights are necessary for creating personalized, compelling customer experiences, and cutting edge products and services that enhance a business’ operational efficiency.
As technology becomes more scalable and people are more connected, the complexity and gaps in leveraging actionable insights increase. According to Forrester, “an average of between 60% and 73% of all data within an enterprise goes unused for analytics”.4 Today’s businesses struggle to close that gap.
Here are three key ways that data fabric helps businesses to gain an advantage with the gap between available data and information and by transforming that data and info into powerful resources:
- Allows access to all data sources, types, and domains, which is especially important in a multi-hybrid cloud strategy
- Creates efficiencies throughout the AI ladder, extending from collecting to organizing, analysing, and infusing, and significantly accelerates the processing and utilization of data
- Allows organizations to deploy business-wide enforceable policies regarding data access. When an organization has access to more data and the ability to quickly leverage it, ensuring that the right people are accessing the right data at the right time is critical.
Data fabric is a gateway for businesses looking to move beyond older data management practices and optimize their resources. According to a StrategyR market report, “the market for software products and services that facilitate the creation and management of data fabrics will grow to be $3.7 billion annually by 2026”.5
What are the benefits of data fabric?
Data fabric is often viewed as a gradual offspring of earlier legacy systems with hardware and a network but no overarching network management system for getting data where it needed to go.
Forrester finds that a data fabric “minimizes the complexity [of a data estate] by automating processes, workflows, and pipelines, generating code and streamlining data to accelerate various use cases”.6
In Data Fabric for Dummies, author Ed Tittel declares that “data fabric brings worthwhile change and numerous benefits to those who buy into the vision”.3 Here are several examples of data fabric's benefits:
- Accommodates different environments – Ed Tittel states “a data fabric can accommodate multiple instances of all kinds of execution environments, including on-premises data centers, cloud platforms, and edge systems”.3
- Process data regardless of velocity – According to Tittel, “a data fabric can process and provision data at all velocities from streaming data in real-time to scheduled batch jobs (regularly or infrequently, large and small, fast and slow)”.3
- Reduces time to insights – From the bmc blogs post “Data Fabric Explained: Concepts, Capabilities & Value Props”, author Muhammad Raza states that data fabric helps to “mitigate disruptions from switching between cloud vendors and compute resources to process data stored in disparate locations, [which] reduces time to insights dramatically”.7
- Governance and compliance – Raza notes that data fabric helps unify the overall security and governance process by ensuring that it’s “centralized and consistent across all environments and making it possible for “local management of metadata in compliance with global organizational policies [to] be applied to all data assets”.7
- Data transportation and processing made easy – Tittel notes that data fabric “can move data from platform to platform for ready consumption or storage and archiving [with] no extensive refactoring needed”.3 When it comes to moving processing, data fabric can “[shift] processing from one execution environment to another without extensive recoding”.3
What are the challenges for building a data fabric?
While the benefits of data fabric are multifold, there are many blockers that prevent organizations from successfully adopting a solid data fabric architecture. Challenges to data fabric adoption include, but aren’t limited to, the following:
- The human component
- Requiring agile collaboration
Why does a data fabric require modernization?
Organizations today often own and use their data assets. A data fabric helps to unify all these assets, creating a “consolidated data management environment that extends across an organization’s edge-to-core-to-cloud infrastructure for all platforms and applications”.3
Tittel argues that data fabric implementation usually requires a modernization process of “[taking] existing fragmented and siloed approaches to managing, storing and situating data into the data fabric’s single, consistent, policy-driven, self-service environment, supported by DevOps and DataOps principles”.3
Tittel states that “modernization is a vital job for organizations seeking to deploy a data fabric [and that it] involves nothing less than a review of all data assets relevant to a business case, and all applications and services that consume and produce data [while trying to open] the data fabric’s umbrella to cover everything”.3
How does the human component affect data fabric adoption?
Data fabric, like so many other technology-based elements in business and the world today, has a human factor. Tittel notes that “applications come with stakeholders, users, and developers, all of whom must understand and buy into that fabric [and enterprises] risk fighting “shadow IT” and other end-arounds that favor “quick and dirty” over consistent and coherent”.3
How does the investment factor affect data fabric adoption?
Tittel notes that following the financial crises and the pandemic, there’s increased “demand that any technology investments provide a faster [ROI which] puts added pressure on existing investments [and] it encourages cautious and critical consideration of new [investments]”.3 He argues that the market conditions “[benefit] use cases at the high end of the volume and variety curve where bigger payoffs prevail”.3
Why is Agile methodology important for data fabric?
Tittel notes that for data fabric adoption, modern data fabrics requires the enterprise to leverage DataOps and must use the “CI/CD [continuous integration, continuous delivery] Agile methodology [that] requires data analysts, business stakeholders, and IT personnel to stop working in disparate environments [and] collaborate to ensure [data comprehension and] requires data analysts, business stakeholders, and IT personnel to stop working in disparate environments”.3
Tittel notes that this integration will produce “self-monitoring and self-measurement for the data fabric so that it keeps getting better at managing data through its entire lifecycle”.3 By this logic, the data fabric is in a persistent state of optimization, and is constantly performing better than it had been previously.
How to optimize your data fabric and help it generate value
Data fabric helps you make better decisions through improved compute performance across data channels. Ashutosh Gupta argues that “to deliver business value through data fabric design, D&A leaders should ensure a solid technology base, identify the required core capabilities, and evaluate the existing data management tools”.2
Data fabric collects and analyzes metadata
Gupta notes that “Contextual information lays the foundation of a dynamic data fabric design. There should be a mechanism (like a well-connected pool of metadata) that enables data fabric to identify, connect, and analyze all kinds of metadata such as technical, business, operational, and social”.2
Passive metadata must be converted into active metadata
Gupta argues that for data sharing to work best, enterprises must activate their metadata. For this metadata to become active, their data fabric must do the following:
1. Locate and analyze the key metrics’ and statistics’ metadata
2. Construct a graph model
3. Graphically depict the metadata as it relates to the enterprise’s relationships
4. Use decisive metadata metrics to enable artificial intelligence and machine learning (AI/ML) algorithms
a. These AI/ML algorithms will gradually learn and generate predications for data management, integration, and more2
Knowledge graphs help enrich data with semantics
Gupta notes that the reason data fabric needs to make and curate knowledge graphs, is because these graphs “[enrich] data with semantics [and help] data and analytics leaders to derive business value”.2 He states that the “semantic layer of the knowledge graph makes it more [intuitive and] the analysis easy for D&A leaders”2. Lastly, the semantic layer “adds depth and meaning to the data usage and content graph, allowing AI/ML algorithms to use the information for analytics and other operational use cases”.2
Data fabric requires a robust foundation of data integration
One of the benefits of data fabric adoption is how flexible it is in terms of data delivery. It easily works with multiple data delivery methods, which help its enterprise to accommodate a vast array of data consumers, extending from traditional IT through finance and business.
1. Data Fabric, Techopedia, 28 December 2021.
2. Data Fabric Architecture is Key to Modernizing Data Management and Integration, Gartner, Ashutosh Gupta, 11 May 2021.
3. Data Fabric for Dummies, Ed Tittel, Hitachi Vantara Special Edition, 2021.
4. Hadoop Is Data’s Darling For A Reason, Mike Gualtieri, Forrester, 21 January 2016.
5. Data Fabric World Market Report, StrategyR, February 2022.
6. Enterprise Data Fabric Q2 2020, Noel Yuhanna with Gene Leganza, Robert Perdoni, Christine Turley, Forrester, 10 June 2020.
7. Data Fabric Explained: Concepts, Capabilities & Value Props, Muhammad Raza, bmc - Machine Learning & Big Data Blog, 9 November 2021.