Rise of Data Intelligence

The term Data Intelligence has increasingly caught my attention. Not only because I know the product “SAP Data Intelligence” quite well, but also because I recently got hold of the BARC study “BARC Score Data Intelligence Platforms”.

There were discussions on LinkedIn with an BARC employee about what all is behind the term.

Comment (Mark A. Michel):

“BARC’s definition of “data intelligence platforms” appears to be very different from Gartner: the products in the comparison read like “data catalog” rather than “platform””

Answer (Timm Grosser, BARC):

“Data Catalog in the sense of Search & Discovery (Data Catalog) is in our view one of the core functions of a Data Intelligence Platform. And some of the providers examined actually come from this segment. Beyond that, however, we also see functions for data governance (workflows, data quality, …), business glossaries, lineage functionality, support for data access in the sense of data shopping in DI platforms. The aim is not only to inventory data, but to promote value creation from data. The basis for this is the linking of different metadata (types) in addition to the technical and functional ones. For the integration, preparation, linking, enrichment and analysis of this metadata, we see here concepts, functions for automation as a further functional component. And these are the tools and functions that we have primarily investigated here.”

Upon further research, I found that IDC has been promoting the term as an evolution of Data Catalogs for several years. IDC admits that the term is not used consistently. The IDC definition is:

“Data intelligence leverages business, technical, relational and operational metadata to provide transparency of data profiles, classification, quality, location, lineage and context; Enabling people, processes and technology with trustworthy and reliable data.”

According to IDC, data intelligence helps answer the following questions:

• Who is using What data?
• Where is data, and where did it come from (lineage and provenance)?
• When is data being accessed, and when was it last updated?
• Why do we have data? Why do we need to keep (or discard) data?
• How is data being used, or perhaps more specifically — how should data be used?
• Relationships — what relationships are inherent within data and with data consumers?

BARC summarizes the idea of IDC again compactly: “Activating data knowledge by leveraging metadata”. The study describes data intelligence as follows:

„Data intelligence goes beyond systematic data collection in organizations and aims to generate a better understanding of data assets by leveraging metadata to link and interconnect additional information. Data Intelligence is about integrating, preparing, linking, enriching, and analyzing business, technical, govern-ance and operational metadata, providing data knowledge to empower people and machines to make better data-based decisions on trustworthy, reliable data in an efficient manner.„

BARC, 2022

Various vendors have adopted the term for their products. Alation sees the development towards Data Intelligence as follows:

The point is that data catalogs started out with one audience (analysts) executing one use-case (search and discovery) and have evolved over the last decade to include multiple audiences executing multiple use cases. This is why we say that the data catalog is the platform for data intelligence. Platform means capable of supporting multiple audiences (e.g., analysts, data scientists, compliance, stewards, data engineers, analytics engineers) executing multiple use cases (e.g., search and discovery, governance, privacy, lineage, metadata management).

Alation, 2022

The following is an overview of what lies behind the products. According to IDC, ASG seems to have been one of the first providers to harness the term “data intelligence” since 2016. The features listed are from the vendors tool discription on their website and does probably just give an indication of important aspects of their solution. The list is in alphabetical order.

Vendor: Alation — „A system that deliverer trustworthy, reliable data, while also providing intelligence about said data, or metadata.“

Alation Data Catalog includes the following features:

• Business glossaries and data dictionaries (to store definitions)
• Profiling tools
• Stewardship dashboards
• Data lineage features
• Data cataloging functions, like natural language processing

Vendor: ASG

ASG Data Intelligence includes the following features:

• Automated Data Asset Inventory
• Enterprise Metadata Repository
• Federated Business Glossary
• Automated Data Lineage
• Impact Analysis
• Data Governance / Stewardship Workflows
• Data Privacy Compliance
• Reference Data Management
• Flexible Metamodel

Vendor: BigID — „Discovery is at the core of data intelligence, insight, and analysis — and needs to be both scalable and automated in order to successfully address the volume (and type) of data that organizations collect.“

BigID Data Intelligence includes the following features:

– Catalog
— Classification
— Cluster Analysis
— Correlation

Vendor: Collibra — „Data intelligence maturity can be defined as the ability for your organization to leverage data to make informed business decisions. The degree to which your organization has adopted and implemented the technologies, processes, and policies required to manage your data on demand and at scale.„

Collibra Data Intelligence Cloud includes the following features:

• Data Catalog
• Data Governance
• Data Qualiy & Observability
• Core Services & API
• Data Privacy
• Data Lineage

Vendor: erwin (by Quest)

erwin Data Intelligence - couldn’t finde a good description

Vendor: Informatica — “They’re (customers) harnessing data intelligence with the building blocks of data mastering, data quality, data cataloging, data steward collaboration, and data security and privacy on a unified platform to build trust assurance in the data itself, then feeding that reliable data stream into self-service analytics to refine into data intelligence to transform business through a data marketplace.”

Informatica Enterprise Data Catalog includes the following features:

• Connect and catalog your data assets
• Easily curate and prepare your data
• Automate end-to-end data lineage
• Measure and optimize your data value
• Understand data quality and relationships
• Collaborate on data intelligence