By clicking “Accept”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. View our Privacy Policy for more information.
Jun 24, 2024
Product

End-To-End Data Observability With Declarative Assets and Lineage

Post by
Margot Lepizzera
&

Sifflet’s core mission is to make data observability simpler by providing data stakeholders with the most comprehensive overview of their data pipelines. We are thrilled to announce that declarative assets and lineage are officially live, going one step further in ensuring end-to-end observability for your teams.

Leveraging our declarative framework, you can now programmatically push assets and lineage onto Sifflet to catalog and visualize lineage for your entire data stack. This allows teams to have a more comprehensive data documentation and governance while simplifying the troubleshooting experience thanks to a more complete impact assessment and root cause analysis.

Catalog and Document All Your Assets (And We Mean It, All of Them)

Sifflet comes with a large number of built-in integrations spanning your entire data pipelines’ stack. Built-in integrations automatically collect metadata and lineage information and make it available in the Data Catalog.

In some cases though, you might need to programmatically declare some of your data pipeline assets to ensure a truly end-to-end observability experience. 

  • Thanks to the declarative framework, any custom applications, scripts or APIs can now for instance be added to your Data Catalog. 
  • You can also leverage this declarative framework to reflect any data source (e.g. Salesforce, SAP, HubSpot, etc.) or data consumer asset (e.g. ML models, BI dashboards, etc.) using a technology Sifflet doesn’t yet directly integrate with.
  • You can even use it to reflect orchestrators and transformations jobs.

Example payload declaring a product performance tracker data application

Declared assets show up on your Data Catalog, just like any asset pulled from Sifflet built-in sources. Users can browse them as well as filter and search on them using declared metadata. Ensuring data stakeholders are able to locate the asset they might be interested in, regardless of the asset type and technology is key to avoid asset duplication and improve data self service.

Example Data Catalog featuring the product performance tracker data application as a declared asset

Declared asset pages also surface the same amount of metadata as regular assets: you can include descriptions and tie business terms to the asset to ensure all data stakeholders have the context they need to properly understand the asset. You can also include external links to make sure catalog users can swiftly access the asset itself.

Consolidate Data Governance

As your number of data use cases and assets grows, managing your data becomes increasingly challenging. Declarative assets allow you to ensure your entire data fleet is cataloged, simplifying data governance in turn.

Example declared asset page of the product performance tracker data application

You can assign tags to your declared assets to classify them and ensure any PII data is handled appropriately and in a way that is compliant with regulation. Similarly, you can assign owners to your declared assets in order to drive accountability and simplify collaboration.

Lineage, From the Very Beginning to the Very End

This feature would not be complete without a lineage integration. The great news is that you can connect your declared assets to any other assets, whether they are declared or not, within your data pipelines for true end-to-end lineage.

Example declared asset lineage view of the product performance tracker data application

This end-to-end lineage comes with countless benefits for your team. By reflecting all downstream assets, it ensures accurate and comprehensive impact assessments, preventing any overlooked parties before implementing changes. Similarly, in the event of a data quality issue, having all sources visible upstream of the data warehouses simplifies root cause analysis and avoids the time consuming hustle of piecing together pipeline assets to identify the problematic one.

Start Declaring Assets Now

Want to start declaring assets? Check out the declarative assets documentation page to get started!

If you want to learn more about how Sifflet can help you with data observability, you can check out our documentation or reach out for a demo.

Related content