How Acryl Data Helped DPG Media Save 25% Per Month in Snowflake Costs

Managing Snowflake Costs at DPG Media

Acryl Data provided DPG Media with a detailed overview of their data usage, helping them identify areas of inefficiency and reduce waste.

Step 1 - Setup and Integration

The customer first integrated Acryl’s platform with their Snowflake data warehouse and Looker dashboards and charts. This allowed Acryl to analyze DPG Media’s data usage, lineage and other signals to estimate Snowflake costs.

Step 2 - Implementation of Metadata Tests

The customer utilized Acryl Data's Metadata Tests, which allowed them to monitor their data usage continuously and flag datasets that had low usage but had high costs (determined through proxy cost signals like storage footprint).

Step 3 - Elimination of Redundant Data

DPG Media was able to use the results of Metadata Tests in conjunction with Lineage and Impact Analysis features to confidently retire datasets rapidly with no disruption to the business.

Step 4 - Continuous Monitoring and Optimization

Metadata Tests are built for continuous monitoring of rules on top of Acryl’s metadata graph. By implementing these tests, DPG can ensure that data cleanup efforts are not limited to a one-time occurrence. Instead, future inefficiencies will be proactively identified and addressed to prevent cost overruns.

Acryl provides DPG Media with central governance on top of a decentralized data mesh. By leveraging Acryl's Metadata Tests, they saved 25% per month in Snowflake costs, and also gained valuable insights into their data usage to further ensure the success of their data mesh strategy.

Building a Data Mesh at DPG

DPG initially investigated Acryl Data to implement a data catalog as part of an effort to build a data mesh.

DPG is a large, distributed organization with a lot of different business areas - newspapers, magazines, television broadcasting, advertising. All of these different kinds of businesses produce and consume data, each with their own data teams to manage the process.

The technology stack at DPG is very broad, with business intelligence tools like Tableau, Qlik, and Looker, data warehouses like Snowflake and Redshift, and various AWS accounts and services. The organization uses Kafka for streaming to process behavioral data from websites. As acquisitions are a common occurrence, the data stack continues to grow and evolve in unpredictable ways as well.

“Prior to DataHub and Acryl, we didn’t have an end-to-end view on our data landscape at all. We needed to embrace a distributed environment and a data mesh approach made much more sense,” said Lavaert. “We tried to operate under a large, central team which wasn’t really manageable. In practice, we had already operated much like a data mesh, but without any form of governance.”

With Acryl Data in place, DPG has evolved its focus from simply pushing clean or approved datasets to thinking about the entirety of usable data in the organization, where teams are able to visualize what is available and its dependencies.

Technical groups are able to classify data and gain a better understanding of which data contains personally-identifiable or classified information to limit exposure to a smaller user base. Prior to implementing Acryl, documentation was scattered in Confluence with infrequent updates.

Establishing a Central Data Collaboration Resource

DPG Media has rolled out Acryl Data to a mix of both IT and data analysts and continues to add more and more users from non-technical business teams. With the Acryl browser extension, business users can also see any underlying data problems in the reports they’re viewing and how to contact support resources if needed.

The DPG team worked closely with Acryl during the implementation. As Lavaert described, “It’s been pretty great working with the Acryl team. For any issues that we had in the beginning, we would see fixes within one week. Our work with a lot of other vendors does not have that same level of support.”

Data teams at DPG use Acryl for common support questions. Rather than address each issue ad hoc, they simply forward a link where the information is documented.

“We’re gaining an operational understanding of our data and understanding what goes wrong, where it went wrong, and which components are responsible,” said Lavaert. “There’s real value to having a central place to connect all the information about your data.”

Internally, the tool has been helpful to relay a global view into the work that data teams have undertaken, enabling non-technical teams to understand the full scope of their data landscape. As Lavaert described, “Being able to visualize all of our data in a central place has had tremendous impact. With everything we have going on, Acryl is helpful to communicate to leadership why some problems are more complicated than others.”

Managing Data Incidents

Adding applications and processes like Airflow and Spark jobs to Acryl and integrating Jira has been very helpful.

As DPG has added processes like Airflow and Spark jobs and applications like Jira to their Acryl implementation, they’ve used the platform as a central resource for incident management. Previously, these efforts had been scattered across Jira boards for individual teams, causing a fair amount of confusion as a single team is seldom responsible for an issue. The number of dependencies would cause a series of tickets to be filed from one team to another.

For example, the DPG Media data tracking team produces Clickstream data with many consumers. Another team produces the ‘golden’ Clickstream data sets, another team does profiling on it, and yet another team produces dashboards on the profiling data.

If there were a mistake in a dashboard, the ticket would be filed from a business user with the profiling team - and would make its way back several hops to the root cause. Status quo was a series of redirection efforts to find the right person and track down ownership. Acryl simplified this common process, as Lavaert explained, “With Acryl Data, we can quickly identify what the issue might be, who to contact, and resolve issues faster.”

Why Acryl Data?

“We evaluated eight different catalogs, and Acryl Data was just the perfect fit for our situation,” explained Lavaert.

One of the major differences is how Acryl enables better workflows for technical and non-technical users alike. The DPG team valued how Acryl Data is both push-based, embraces Shift-Left practices, and is architecturally flexible, enabling the data team to run processes both through the UI and APIs.

“We were able to ramp up tremendously because all of the information is taken out of the systems themselves. It’s helpful to have a low barrier to entry - can I ask other developers to document their Snowflake instance - and it will also be documented in the data catalog.”

While most business teams use the UI for lineage, more advanced teams take advantage of the API for incident management. For custom queries that the DPG data team runs often, they use the API to establish an exclusive filter through a script or just filter in Excel.

What’s Next?

As far as future plans go, Lavaert had a few items to share. According to him, “I would like Acryl to be the central tool in our organization. If you’re a business user and you think about data, you look into Acryl, find what or who you’re looking for and what systems data is touching. If you’re an engineer, it can help in day-to-day operations, like integrating into pull request activities with dbt and showing what will potentially break.

The group also is interested in working with Acryl Data to shape its control plane vision, stating, “We’re looking forward to implementing data contracts and showing how powerful they can be for validating and enforcing both large and small SLAs.”