BACK TO ALL POSTS

Snowflake and Acryl Data: Better Together for Our Users

Data Engineering

Metadata Management

Data Governance

Swaroop Jagadish

Oct 26, 2023

Data Engineering

Metadata Management

Data Governance

Some partnering announcements are especially sweet—like this one.

We’re pleased to share that Acryl Data has been named a Premier-Tier partner in Snowflake's prestigious Technology Partner Program!

Acryl Data’s flagship offering, Acryl Cloud, is a fully managed service based on the Datahub Project, the leading open-source metadata platform and data catalog. Acryl Cloud provides a central control plane for the decentralized data stack, marrying industry-leading data catalog capabilities with best-in-class features for data governance and observability.

Premier-Tier Partner designation reflects the ongoing collaboration between Acryl Data and Snowflake. Together, we’ve formalized a reference architecture joint customers can implement to ensure reliable, scalable, secure integration between Acryl Cloud and Snowflake.

“Before bringing Acryl on board, MYOB’s data teams would see multiple breaking changes per week. Since integrating Acryl into our workflow about a year ago, even though our overall usage of Snowflake has gone up 4 times, Acryl has helped us significantly reduce the number of breaking changes, to the extent that they are no longer a burden on all teams.”
Asad Naveed, Engineering Manager MYOB

This integration is production-tested. Today, hundreds of companies use DataHub and Snowflake together, including Acryl Cloud customers like Notion, DPG Media , and MYOB . They depend on Acryl Cloud to track, manage, and analyze data lineage; to monitor data quality; to enforce governance rules and notify stakeholders of changes; and to promote collaboration among decentralized teams.

Acryl Cloud helps Snowflake users enforce security and compliance across their distributed data stacks, as well as maintain a high bar for data quality. In addition, it enables them to increase their usage of Snowflake—while keeping their Snowflake subscription costs down.

Here are just a few of the benefits joint customers get from using Acryl Cloud with Snowflake

Unified Metadata Management

Acryl Cloud’s control plane provides a central, unified view of your data ecosystem, including not just Snowflake, but also any other useful source—databases and data lakes, object storage, orchestration engines, files, etc. Specific to Snowflake, Acryl Cloud gives joint customers column-level insight into the provenance and lineage of their data, enabling them to continuously monitor for quality.

Visual Data Lineage + Dependencies

Acryl Cloud’s advanced lineage-tracking capabilities make it easier for teams to visualize and understand dependencies between the data assets distributed up, down, and across their stacks. Acryl Cloud helps Notion, DPG Media, MYOB, and others track and proactively address the impact of breaking changes—especially when changes are proposed in their Snowflake datasets.

Real-time Notifications and Alerting

Organizations can use Acryl Observe, Acryl Cloud’s observability module, to monitor changes to data in real-time, essential for enforcing data quality, data ownership, and other governance standards. If a column in Snowflake gets renamed or if Snowflake starts enforcing new constraints, Acryl Cloud’s Subscriptions and Notifications feature alerts data owners, producers, and downstream consumers. Teams can configure Acryl Cloud to monitor tables, views, files, and other assets for updates, making it easier for them to enforce data freshness SLAs.

Collaborative Data Management

Data consumers can use Acryl Cloud to boost collaboration and communication while working with datasets in Snowflake, giving them at-a-glance insight into who owns and maintains a dataset. Similarly, Acryl Cloud’s rich integration with AIM/SSO services simplifies data sharing. Consumers know whom to approach with any questions, concerns, or requests related to Snowflake datasets, and can also more easily access them, too.

Automated Impact Analysis

Data changes almost always have ripple effects. By using Acryl Cloud to track and understand lineage, and by taking advantage of advanced features like Metadata Tests , teams can determine which data assets are affected by changes to upstream datasets. In addition, they can use Acryl’s Subscriptions and Notifications feature to ensure consumers are informed about and have the opportunity to discuss changes before they're deployed.

Column-level lineage in Snowflake with dbt

It’s still difficult to extract, track, and manage column-level lineage using dbt. Joint customers like MYOB, Australia’s leading business management software platform, depend on Acryl Cloud’s best-in-class lineage capabilities to extract column-level lineage from their dbt transformations in Snowflake. This gives teams a detailed, granular view of their ELT processing.

Understand and reduce costs in the Snowflake cloud

Using Acryl Cloud’s Metadata Tests feature, customers like DPG Media continuously monitor their Snowflake data usage, identifying costly yet underused datasets. With Acryl Cloud’s lineage and impact analysis capabilities, they were able to retire these redundant datasets without any impact on consumers, resulting in a 25% monthly reduction in their Snowflake costs.

“With Acryl, we were able to reduce our Snowflake costs by 25% each month. We used Acryl's Metadata Tests to identify unused or duplicate Snowflake tables across business units. Impact Analysis allowed us to safely manage the clean-up process.”
Mathias Lavaert, PRINCIPAL DATA ENGINEER, DPG MEDIA

I’m not going to lie: being recognized as a Snowflake Premier-Tier Technology Partner is a pretty big deal for Acryl Data! It’s a major milestone in our growing relationship with Snowflake, and it’s also a testament to our success in helping customers navigate both the complexities and the uncertainties of the modern data stack.

These include not just obvious problems—like schema changes, inconsistent metadata definitions, and a wild profusion of data types—but also the gaps that are a feature (not a bug) of the distributed topology of this stack. Premier-Tier recognition is a validation that Acryl Cloud helps Snowflake users address these gaps.

Governing the distributed, decentralized modern data stack

This is about more than just Acryl Data or Snowflake: it’s about meeting the needs of customers. Today, our joint customers are supporting multiple decentralized teams, some of which expect to use additional compute engines or external storage alongside Snowflake.

Many are adopting radically decentralized approaches to structuring their data platform architecture and operations, like Data Mesh. They’re trying to make it easier for teams working autonomously to discover and access data—in Snowflake and elsewhere. Adding to this, customers need to observe and govern what their teams are doing across the entirety of their stacks—not just in Snowflake.

Snowflake’s great features equip joint customers to easily access and use data in the Snowflake environment, while Acryl’s data management, governance, and discovery features integrate Snowflake with the rest of the ecosystem. This certification is a milestone, yes—but it also underscores Acryl Data’s ongoing commitment to helping organizations manage and govern the complexity of their decentralized data ecosystems. This commitment is our raison d'être.

Questions? Feedback? Join us in DataHub Slack and tell us all about it!

Book a demo to see all the other cool stuff we’re building for Acryl DataHub!



Data Engineering

Metadata Management

Data Governance

NEXT UP

Governing the Kafka Firehose

Kafka’s schema registry and data portal are great, but without a way to actually enforce schema standards across all your upstream apps and services, data breakages are still going to happen. Just as important, without insight into who or what depends on this data, you can’t contain the damage. And, as data teams know, Kafka data breakages almost always cascade far and wide downstream—wrecking not just data pipelines, and not just business-critical products and services, but also any reports, dashboards, or operational analytics that depend on upstream Kafka data.

When Data Quality Fires Break Out, You're Always First to Know with Acryl Observe

Acryl Observe is a complete observability solution offered by Acryl Cloud. It helps you detect data quality issues as soon as they happen so you can address them proactively, rather than waiting for them to impact your business’ operations and services. And it integrates seamlessly with all data warehouses—including Snowflake, BigQuery, Redshift, and Databricks. But Acryl Observe is more than just detection. When data breakages do inevitably occur, it gives you everything you need to assess impact, debug, and resolve them fast; notifying all the right people with real-time status updates along the way.

John Joyce

2024-04-23

Five Signs You Need a Unified Data Observability Solution

A data observability tool is like loss-prevention for your data ecosystem, equipping you with the tools you need to proactively identify and extinguish data quality fires before they can erupt into towering infernos. Damage control is key, because upstream failures almost always have cascading downstream effects—breaking KPIs, reports, and dashboards, along with the business products and services these support and enable. When data quality fires become routine, trust is eroded. Stakeholders no longer trust their reports, dashboards, and analytics, jeopardizing the data-driven culture you’ve worked so hard to nurture

John Joyce

2024-04-17

Get started with Acryl today.
Acryl Data delivers an easy to consume DataHub platform for the enterprise
See it in action
Acryl Data Logo
Acryl DataHub
Acryl ObserveCustomer Stories
TermsPrivacySecurity
© 2024 Acryl Data