BACK TO ALL POSTS

Snowflake and Acryl Data: Better Together for Our Users

Data Engineering

Metadata Management

Data Governance

Swaroop Jagadish

Oct 26, 2023

Data Engineering

Metadata Management

Data Governance

Some partnering announcements are especially sweet—like this one.

We’re pleased to share that Acryl Data has been named a Premier-Tier partner in Snowflake's prestigious Technology Partner Program!

Acryl Data’s flagship offering, Acryl Cloud, is a fully managed service based on the Datahub Project, the leading open-source metadata platform and data catalog. Acryl Cloud provides a central control plane for the decentralized data stack, marrying industry-leading data catalog capabilities with best-in-class features for data governance and observability.

Premier-Tier Partner designation reflects the ongoing collaboration between Acryl Data and Snowflake. Together, we’ve formalized a reference architecture joint customers can implement to ensure reliable, scalable, secure integration between Acryl Cloud and Snowflake.

“Before bringing Acryl on board, MYOB’s data teams would see multiple breaking changes per week. Since integrating Acryl into our workflow about a year ago, even though our overall usage of Snowflake has gone up 4 times, Acryl has helped us significantly reduce the number of breaking changes, to the extent that they are no longer a burden on all teams.”
Asad Naveed, Engineering Manager MYOB

This integration is production-tested. Today, hundreds of companies use DataHub and Snowflake together, including Acryl Cloud customers like Notion, DPG Media , and MYOB . They depend on Acryl Cloud to track, manage, and analyze data lineage; to monitor data quality; to enforce governance rules and notify stakeholders of changes; and to promote collaboration among decentralized teams.

Acryl Cloud helps Snowflake users enforce security and compliance across their distributed data stacks, as well as maintain a high bar for data quality. In addition, it enables them to increase their usage of Snowflake—while keeping their Snowflake subscription costs down.

Here are just a few of the benefits joint customers get from using Acryl Cloud with Snowflake

Unified Metadata Management

Acryl Cloud’s control plane provides a central, unified view of your data ecosystem, including not just Snowflake, but also any other useful source—databases and data lakes, object storage, orchestration engines, files, etc. Specific to Snowflake, Acryl Cloud gives joint customers column-level insight into the provenance and lineage of their data, enabling them to continuously monitor for quality.

Visual Data Lineage + Dependencies

Acryl Cloud’s advanced lineage-tracking capabilities make it easier for teams to visualize and understand dependencies between the data assets distributed up, down, and across their stacks. Acryl Cloud helps Notion, DPG Media, MYOB, and others track and proactively address the impact of breaking changes—especially when changes are proposed in their Snowflake datasets.

Real-time Notifications and Alerting

Organizations can use Acryl Observe, Acryl Cloud’s observability module, to monitor changes to data in real-time, essential for enforcing data quality, data ownership, and other governance standards. If a column in Snowflake gets renamed or if Snowflake starts enforcing new constraints, Acryl Cloud’s Subscriptions and Notifications feature alerts data owners, producers, and downstream consumers. Teams can configure Acryl Cloud to monitor tables, views, files, and other assets for updates, making it easier for them to enforce data freshness SLAs.

Collaborative Data Management

Data consumers can use Acryl Cloud to boost collaboration and communication while working with datasets in Snowflake, giving them at-a-glance insight into who owns and maintains a dataset. Similarly, Acryl Cloud’s rich integration with AIM/SSO services simplifies data sharing. Consumers know whom to approach with any questions, concerns, or requests related to Snowflake datasets, and can also more easily access them, too.

Automated Impact Analysis

Data changes almost always have ripple effects. By using Acryl Cloud to track and understand lineage, and by taking advantage of advanced features like Metadata Tests , teams can determine which data assets are affected by changes to upstream datasets. In addition, they can use Acryl’s Subscriptions and Notifications feature to ensure consumers are informed about and have the opportunity to discuss changes before they're deployed.

Column-level lineage in Snowflake with dbt

It’s still difficult to extract, track, and manage column-level lineage using dbt. Joint customers like MYOB, Australia’s leading business management software platform, depend on Acryl Cloud’s best-in-class lineage capabilities to extract column-level lineage from their dbt transformations in Snowflake. This gives teams a detailed, granular view of their ELT processing.

Understand and reduce costs in the Snowflake cloud

Using Acryl Cloud’s Metadata Tests feature, customers like DPG Media continuously monitor their Snowflake data usage, identifying costly yet underused datasets. With Acryl Cloud’s lineage and impact analysis capabilities, they were able to retire these redundant datasets without any impact on consumers, resulting in a 25% monthly reduction in their Snowflake costs.

“With Acryl, we were able to reduce our Snowflake costs by 25% each month. We used Acryl's Metadata Tests to identify unused or duplicate Snowflake tables across business units. Impact Analysis allowed us to safely manage the clean-up process.”
Mathias Lavaert, PRINCIPAL DATA ENGINEER, DPG MEDIA

I’m not going to lie: being recognized as a Snowflake Premier-Tier Technology Partner is a pretty big deal for Acryl Data! It’s a major milestone in our growing relationship with Snowflake, and it’s also a testament to our success in helping customers navigate both the complexities and the uncertainties of the modern data stack.

These include not just obvious problems—like schema changes, inconsistent metadata definitions, and a wild profusion of data types—but also the gaps that are a feature (not a bug) of the distributed topology of this stack. Premier-Tier recognition is a validation that Acryl Cloud helps Snowflake users address these gaps.

Governing the distributed, decentralized modern data stack

This is about more than just Acryl Data or Snowflake: it’s about meeting the needs of customers. Today, our joint customers are supporting multiple decentralized teams, some of which expect to use additional compute engines or external storage alongside Snowflake.

Many are adopting radically decentralized approaches to structuring their data platform architecture and operations, like Data Mesh. They’re trying to make it easier for teams working autonomously to discover and access data—in Snowflake and elsewhere. Adding to this, customers need to observe and govern what their teams are doing across the entirety of their stacks—not just in Snowflake.

Snowflake’s great features equip joint customers to easily access and use data in the Snowflake environment, while Acryl’s data management, governance, and discovery features integrate Snowflake with the rest of the ecosystem. This certification is a milestone, yes—but it also underscores Acryl Data’s ongoing commitment to helping organizations manage and govern the complexity of their decentralized data ecosystems. This commitment is our raison d'être.

Questions? Feedback? Join us in DataHub Slack and tell us all about it!

Book a demo to see all the other cool stuff we’re building for Acryl DataHub!



Data Engineering

Metadata Management

Data Governance

NEXT UP

Data Quality Should be Part of the Data Catalog - Introducing Acryl Observe

We didn’t go looking for an excuse to develop a data observability solution. There’s more than enough to keep us occupied in relentlessly improving the best data catalog on the planet! But the more experience we gained in working closely with Acryl customers, the clearer it became that data quality, data discovery, and data governance aren’t just complementary, but mutually reinforce one another. Acryl Observe provides data teams with everything they need to detect data breakages immediately, contain their downstream impact, keep stakeholders in the loop, and resolve issues fast—so that data teams can spend less time reacting and more time preventing.

John Joyce

2024-04-16

Metadathon - Why MetaData Matters

Vaidehi Sridhar, product manager at PayPal, came up with a clever solution: a hackathon—but for documentation and metadata. The goal of Sridhar’s and PayPal’s “Metadatathon” was to crowdsource the labor involved in documenting and adding rich context to the company’s distributed data assets. “Lack of documentation was one of the major problems most of our users called out,” Sridhar explains.

Stephen Swoyer

2024-04-16

Hack Your Way to Data Quality with a "Metadatathon"

PayPal Product Manager Vaidehi Sridhar probably wouldn’t call herself a mastermind, but she did execute on a genius idea to improve data quality across PayPal’s sprawling ecosystem. Even better, Sridhar and her team laid the groundwork for PayPal to federate data governance for its decentralized teams—as well as automate many types of tedious governance tasks. The genius idea? A metadata hackathon, or “Metadatathon.”

Stephen Swoyer

2024-03-19

Get started with Acryl today.
Acryl Data delivers an easy to consume DataHub platform for the enterprise
See it in action
Acryl Data Logo
Acryl DataHub
Acryl Observe
TermsPrivacySecurity
© 2024 Acryl Data