Acryl Logo

Data Contracts in DataHub: Combining Verifiability with Holistic Data Management

Data Contract

Shirshanka Das

Sep 19, 2023

Data Contract

676 million

That’s how many results a simple Google search on Data Contracts throws up. As buzzwordy as they might seem, we’ve maintained that they aren’t as different or complicated as they might seem.

And it’s this focus on simplicity that’s guided our approach to building Data Contracts in DataHub. In this blog post, we share how we’ve implemented Data Contracts within DataHub, how you can get started, and how the Data Products functionality can help you get the most out of Data Contracts.

What is a Data Contract?

Here’s what ChatGPT has to say:

A data contract refers to an agreement or specification that defines the structure, format, and semantics of data exchanged between different systems, applications, or components. It serves as a mutual understanding between parties involved in data exchange ensuring that data is transmitted and interpreted correctly.

Not surprisingly, ChatGPT’s take misses a few key nuances, a striking one being the aspect of verifiability.

While contracts encompass agreements, specifications, and various structural aspects, their true value lies in their ability to be validated.
Verifiable elements include schemas, column-level data checks, and operational service level agreements (SLAs that can be programmatically checked and enforced.

For a detailed and nuanced understanding of Data Contracts, check out:

  • The What, Why, and How of Data Contracts, based on an AMA Maggie hosted with Chad Sanderson and me.
  • Data Contracts Wrapped 🎁 2022, that summarizes the main ideas from the most popular writings in the data contracts space.

Here’s a TLDR; version:

A Data Contract is an agreement between a producer and a consumer that clearly defines

  • what data needs to move from a producer/source to a consumer/destination
  • the shape of that data, its schema, and semantics
  • expectations around availability and data quality
  • details about contract violation(s) and enforcement

Data Contracts in DataHub

In DataHub, Data Contracts are collections of assertions, or verifiable things that can be stated and enforced on individual data assets.

Assertions in Data Contracts revolve around schema-related aspects, service level agreements (SLAs), data freshness, and data quality. DataHub’s dbt and Great Expectations integrations allow you to produce:

  • AssertionInfo aspects (that define the parameters of an assertion)
  • AssertionRunEvents (that provide the evaluation results of assertions)
DataHub already Supports Assertions

As part of DataHub’s Data Contracts implementation, we've added two new kinds of assertions:

  • SLA Assertion (talks about when a dataset should land, etc.)
  • Schema Assertion (what the data should look like here, the fields within it, etc.)

End-to-End Implementation of Data Contracts in DataHub

The implementation of Data Contracts in DataHub is designed so that:

  • Data producers can author data contracts as YAML files and store them in version control systems like Git.
  • These contracts can then be deployed to DataHub, which acts as a repository for contracts and their associated assertions
  • Business users can use DataHub to access and edit/update the Data Contract.
  • Existing data quality tools can evaluate these assertions and report the results.
Data Contracts: End to End

Check out these two videos to see all this in action.

Part 1: Creating and Deploying Data Contracts with DataHub

Part 2: Using DataHub’s UI to access and manage Data Contracts

Data Contracts + Data Products: How DataHub Combines Verifiability with Non-Verifiable Metadata

Going back to the verifiability aspect of Data Contracts, key data elements, such as documentation, ownership, and tags, lack verifiability, but we know how incredibly important they are in the context of the data ecosystem.

What are Verifiable Things

And it’s this focus on both verifiable and non-verifiable metadata that anchors DataHub’s approach to Data Contracts. Data Contracts in DataHub integrate with Data Products for a holistic approach to managing data assets. Here’s how.

Data Products vs Data Contracts

Data Products in DataHub represent collections of assets combined together in a concept for you to manage and maintain. They have owners, tags, glossary terms, and documentation.

Data Products Contain all these important things

But they also provide a way to combine verifiable and descriptive metadata.

Data Contracts

Data Contracts are the verifiable aspects stated and enforced on individual data assets, that cover schema-related aspects, service level agreements (SLAs), data freshness, and data quality.

With DataHub, you can combine the verifiable (via Data Contracts)  and the descriptive, non-verifiable (via Data Contracts) elements to create a curated metadata graph.

Data Products to Data Contract architecture

Data Products and Data Contract Management in DataHub

In the near future, to streamline the management of Data Products and Data Contracts, you can use the same YAML file to define both Data Products and Data Contract specifications – allowing them to be managed as a unified definition. This approach ensures that both documentation and schema assertions can be maintained as code, satisfying the needs of different stakeholders.

Data Product + Contract Management

In the advanced DataHub implementation available via Acryl Cloud, a Data Contract Operator responds to contracts – and starts monitoring, enforcing, and reporting results.

Acryl’s Managed DataHub Implementation for Scalable Data Management

While DataHub serves as the foundation for Data Contracts, Acryl’s managed DataHub version provides the advanced tools and capabilities you need to manage them at scale. This includes:

  • An inference engine to generate proposals for Data Contracts
  • Approval workflows for data producers and consumers, and
  • Enforcement mechanisms for data contracts.

References and Further Reading

Data Products in DataHub

Check out Data Contracts in DataHub

Join Us on Slack! Interested in Learning More? Let's Chat!

Data Contract


Extracting Column-Level Lineage from SQL

We built a SQL lineage parser that's schema-aware and can generate accurate column-level lineage from SQL queries. In our tests, it works significantly better than other open-source, Python-based lineage tools.

Harshal Sheth


Snowflake and Acryl Data: Better Together for Our Users

Some partnering announcements are especially sweet—like this one.

Swaroop Jagadish


Join us for Hacktoberfest 2023: Contribute to DataHub and Win Big!!!

Are you ready to dive into the world of open source and make a meaningful contribution? Hacktoberfest 2023 is here, and we're thrilled to invite you to participate by contributing to the DataHub project.

Maggie Hays


Get started with Acryl today.
Acryl Data delivers an easy to consume DataHub platform for the enterprise
See it in action
Acryl Data Logo
Acryl DataHub
Acryl Observe
© 2023 Acryl Data