Absolutely. Containerized environments paired with observability tools like Kubernetes and Prometheus for data enable faster incident detection and response. Features like real-time alerts, dynamic thresholding, and on-call management workflows make it easier to maintain healthy pipelines and reduce downtime.

The MCP (Model Context Protocol) Server is a new interface that lets you interact with Sifflet directly from your development environment. It's designed to make data observability more seamless by allowing you to query assets, review incidents, and trace data lineage without leaving your IDE or notebook. This helps streamline your workflow and gives you real-time visibility into pipeline health and data quality.

When you click on a Fivetran connector node in the lineage, you’ll see key metadata like source and destination, sync frequency, current status, and the timestamp of the latest sync. This complements Sifflet’s existing metadata like owner and last refresh for complete context.

Great question! Data monitoring is like your car's dashboard—it alerts you when something goes wrong, like a failed pipeline or a missing dataset. Data observability, on the other hand, is like being the driver. It gives you a full understanding of how your data behaves, where it comes from, and how issues impact downstream systems. At Sifflet, we believe in going beyond alerts to deliver true data observability across your entire stack.

SQL Table Tracer uses a Monoid-based design to handle complex SQL structures like Common Table Expressions (CTEs) and subqueries. This approach allows it to incrementally and safely compose lineage information, ensuring accurate root cause analysis and data drift detection.

Sifflet helps you identify underused or orphaned data assets through lineage and usage metadata. By analyzing this data, you can make informed decisions about deprecating unused tables or enhancing monitoring for critical pipelines. It's a smart way to improve pipeline resilience and reduce unnecessary costs in your data ecosystem.

Sifflet’s unified data catalog and observability features bring context-rich insights into your data workflows. This integration enhances data lineage tracking and supports stronger data governance by giving teams a holistic view of how data flows and transforms across your systems.

Integrating data observability helps you stay on top of data issues before they impact your users. With real-time metrics, pipeline error alerting, and dynamic thresholding, you can catch problems early and ensure your data meets SLA requirements. This proactive monitoring helps teams maintain trust and deliver consistent, high-quality data services.