BACK TO ALL POSTS

Data Contracts in DataHub: Combining Verifiability with Holistic Data Management

Data Contract

Shirshanka Das

Sep 19, 2023

Data Contract

676 million

That’s how many results a simple Google search on Data Contracts throws up. As buzzwordy as they might seem, we’ve maintained that they aren’t as different or complicated as they might seem.

And it’s this focus on simplicity that’s guided our approach to building Data Contracts in DataHub. In this blog post, we share how we’ve implemented Data Contracts within DataHub, how you can get started, and how the Data Products functionality can help you get the most out of Data Contracts.

What is a Data Contract?

Here’s what ChatGPT has to say:

A data contract refers to an agreement or specification that defines the structure, format, and semantics of data exchanged between different systems, applications, or components. It serves as a mutual understanding between parties involved in data exchange ensuring that data is transmitted and interpreted correctly.

Not surprisingly, ChatGPT’s take misses a few key nuances, a striking one being the aspect of verifiability.

While contracts encompass agreements, specifications, and various structural aspects, their true value lies in their ability to be validated.
Verifiable elements include schemas, column-level data checks, and operational service level agreements (SLAs that can be programmatically checked and enforced.

For a detailed and nuanced understanding of Data Contracts, check out:

  • The What, Why, and How of Data Contracts, based on an AMA Maggie hosted with Chad Sanderson and me.
  • Data Contracts Wrapped 🎁 2022, that summarizes the main ideas from the most popular writings in the data contracts space.

Here’s a TLDR; version:

A Data Contract is an agreement between a producer and a consumer that clearly defines

  • what data needs to move from a producer/source to a consumer/destination
  • the shape of that data, its schema, and semantics
  • expectations around availability and data quality
  • details about contract violation(s) and enforcement

LEARN MORE: Watch the on-demand webinar:

Data Contracts: A Practitioner's Guide

presented by Gabe Lyons, founding engineer at Acryl Data

Data Contracts in DataHub

In DataHub, Data Contracts are collections of assertions, or verifiable things that can be stated and enforced on individual data assets.

Assertions in Data Contracts revolve around schema-related aspects, service level agreements (SLAs), data freshness, and data quality. DataHub’s dbt and Great Expectations integrations allow you to produce:

  • AssertionInfo aspects (that define the parameters of an assertion)
  • AssertionRunEvents (that provide the evaluation results of assertions)
DataHub already Supports Assertions

As part of DataHub’s Data Contracts implementation, we've added two new kinds of assertions:

  • SLA Assertion (talks about when a dataset should land, etc.)
  • Schema Assertion (what the data should look like here, the fields within it, etc.)

End-to-End Implementation of Data Contracts in DataHub

The implementation of Data Contracts in DataHub is designed so that:

  • Data producers can author data contracts as YAML files and store them in version control systems like Git.
  • These contracts can then be deployed to DataHub, which acts as a repository for contracts and their associated assertions
  • Business users can use DataHub to access and edit/update the Data Contract.
  • Existing data quality tools can evaluate these assertions and report the results.
Data Contracts: End to End

Check out these two videos to see all this in action.

Part 1: Creating and Deploying Data Contracts with DataHub

Part 2: Using DataHub’s UI to access and manage Data Contracts

Data Contracts + Data Products: How DataHub Combines Verifiability with Non-Verifiable Metadata

Going back to the verifiability aspect of Data Contracts, key data elements, such as documentation, ownership, and tags, lack verifiability, but we know how incredibly important they are in the context of the data ecosystem.

What are Verifiable Things

And it’s this focus on both verifiable and non-verifiable metadata that anchors DataHub’s approach to Data Contracts. Data Contracts in DataHub integrate with Data Products for a holistic approach to managing data assets. Here’s how.

Data Products vs Data Contracts

Data Products in DataHub represent collections of assets combined together in a concept for you to manage and maintain. They have owners, tags, glossary terms, and documentation.

Data Products Contain all these important things

But they also provide a way to combine verifiable and descriptive metadata.

Data Contracts

Data Contracts are the verifiable aspects stated and enforced on individual data assets, that cover schema-related aspects, service level agreements (SLAs), data freshness, and data quality.

With DataHub, you can combine the verifiable (via Data Contracts)  and the descriptive, non-verifiable (via Data Contracts) elements to create a curated metadata graph.

Data Products to Data Contract architecture

Data Products and Data Contract Management in DataHub

In the near future, to streamline the management of Data Products and Data Contracts, you can use the same YAML file to define both Data Products and Data Contract specifications – allowing them to be managed as a unified definition. This approach ensures that both documentation and schema assertions can be maintained as code, satisfying the needs of different stakeholders.

Data Product + Contract Management

In the advanced DataHub implementation available via Acryl Cloud, a Data Contract Operator responds to contracts – and starts monitoring, enforcing, and reporting results.

Acryl’s Managed DataHub Implementation for Scalable Data Management

While DataHub serves as the foundation for Data Contracts, Acryl’s managed DataHub version provides the advanced tools and capabilities you need to manage them at scale. This includes:

  • An inference engine to generate proposals for Data Contracts
  • Approval workflows for data producers and consumers, and
  • Enforcement mechanisms for data contracts.

References and Further Reading

Data Products in DataHub

Check out Data Contracts in DataHub

Join Us on Slack! Interested in Learning More? Let's Chat!

Data Contract

NEXT UP

Data Quality Should be Part of the Data Catalog - Introducing Acryl Observe

We didn’t go looking for an excuse to develop a data observability solution. There’s more than enough to keep us occupied in relentlessly improving the best data catalog on the planet! But the more experience we gained in working closely with Acryl customers, the clearer it became that data quality, data discovery, and data governance aren’t just complementary, but mutually reinforce one another. Acryl Observe provides data teams with everything they need to detect data breakages immediately, contain their downstream impact, keep stakeholders in the loop, and resolve issues fast—so that data teams can spend less time reacting and more time preventing.

John Joyce

2024-04-16

Metadathon - Why MetaData Matters

Vaidehi Sridhar, product manager at PayPal, came up with a clever solution: a hackathon—but for documentation and metadata. The goal of Sridhar’s and PayPal’s “Metadatathon” was to crowdsource the labor involved in documenting and adding rich context to the company’s distributed data assets. “Lack of documentation was one of the major problems most of our users called out,” Sridhar explains.

Stephen Swoyer

2024-04-16

Hack Your Way to Data Quality with a "Metadatathon"

PayPal Product Manager Vaidehi Sridhar probably wouldn’t call herself a mastermind, but she did execute on a genius idea to improve data quality across PayPal’s sprawling ecosystem. Even better, Sridhar and her team laid the groundwork for PayPal to federate data governance for its decentralized teams—as well as automate many types of tedious governance tasks. The genius idea? A metadata hackathon, or “Metadatathon.”

Stephen Swoyer

2024-03-19

Get started with Acryl today.
Acryl Data delivers an easy to consume DataHub platform for the enterprise
See it in action
Acryl Data Logo
Acryl DataHub
Acryl Observe
TermsPrivacySecurity
© 2024 Acryl Data