Alerting where it matters – Our integration with Datadog | Census
Jeff SloanMarch 03, 2023
Jeff is a Senior Data Community Advocate at Census, previously a Customer Data Architect and a Product Manager. Jeff has strong opinions on LEFT JOINs, data strategy, and the order in which you add onions and garlic to a hot pan. Based in New York City.
Introduction
Today, we’re excited to announce our new Datadog integration to help data teams guarantee the health of their business-critical pipelines.
Your data powers real customer interactions every minute, and deserves the same attention as production software. If anything unexpected happens, you need to know ASAP.
Datadog is the central hub of your monitoring and alerting. With the Datadog + Census integration, you can create customized alerts and anomaly detection for all your reverse ETL syncs. Stay ahead of data surprises and sync with confidence with deeper visibility into all your data activation flows.
In other words, your reverse ETL pipelines create an implicit contract between your data and operational systems that data will continue to be trustworthy, high quality, and available. That’s why Datadog’s data quality monitoring is critical for every team.
While we already provide custom alerting in-app, plus a whole suite of observability features, the new Datadog integration enables users to set up even more granular monitoring customized to their business needs.
Ready to get started?
Simply navigate to your workspace’s Settings > Integrations page, click on the Datadog Integration, and authenticate Census with your Datadog account (more details in our docs). This integration is available on Platform tier plans.
Examples and Use Cases
Keep an eye on your Census workspace’s health, alert on sync failures, and manage incidents in your data pipelines. Census will send metrics and events allowing you to track sync completion statuses and records processed by each ETL workflow. These events and metrics are:
Metrics
census.syncs.sync_completed - the number of syncs that have completed over a time window
census.syncs.rows_processed - the volume of rows processed in your syncs, useful for anomaly detection
Events
census.syncs.completed - this event allows for alerting and incident management flows if the sync fails, and includes the sync's failure reason.
Alongside each metric and event, Census sends a variety of tags to help you quickly identify which sync is experiencing an anomaly and why.
Monitoring
The Census Datadog integration launches with a recommended dashboard to observe the health of your data pipelines. To load this dashboard in your Datadog account, navigate to the Datadog Integration marketplace and “Install” the integration. Using this dashboard or your own creation, monitor for successful and unsuccessful sync runs, as well as records processed to every destination.
Datadog's approved partner integration dashboard for Census observability
Alerting
Create alerts in Datadog to notify your teammates when syncs begin to fail. With Datadog Monitors, you will always be the first to find out about an issue in your syncs, so you can resolve the issue quickly. Use monitor notification settings to loop in specific team members who have the skills and permissions to keep your pipelines healthy.
Incident Management and Troubleshooting
When issues arise, you need to act fast and keep your stakeholders informed. Datadog’s Incident Management feature allows you to rally your teammates and update them on incident progress. All of Census’s metrics and events come readily equipped with tags and metadata to help you quickly diagnose a problem, and navigate to the sync that requires your attention. In the Official Census Integration dashboard, you can even use navigation links from your reports like Failing Syncs to land directly back in the Census application without skipping a beat.
Wrapping up
Data stacks are getting more complex, chaining together several tools like Snowflake, Airflow, dbt, BI Tools, and more. You don’t need yet-another-dashboard in a SaaS product to understand how it’s functioning. You need a single place to identify anomalies, unify alerting, and triage issues. For many data teams today, that’s happening in Datadog.
If you need more detailed debugging information, you’ll find that Census provides a rich suite of additional observability – Sync Logs of every record success and failure, and API Inspector visibility into every API call to your destination services.
