Jeff is a Senior Data Community Advocate at Census. He's a Modern Data Stack native, both as a builder and a practitioner. Ask Jeff for his thoughts on Redshift vs. Snowflake, but clear your calendar for the next 30 minutes.
Today, we’re excited to announce our new Datadog integration to help data teams guarantee the health of their business-critical pipelines.
Your data powers real customer interactions every minute, and deserves the same attention as production software. If anything unexpected happens, you need to know ASAP.
Datadog is the central hub of your monitoring and alerting. With the Datadog + Census integration, you can create customized alerts and anomaly detection for all your reverse ETL syncs. Stay ahead of data surprises and sync with confidence with deeper visibility into all your data activation flows.
In other words, your reverse ETL pipelines create an implicit contract between your data and operational systems that data will continue to be trustworthy, high quality, and available. That’s why Datadog’s data quality monitoring is critical for every team.
While we already provide custom alerting in-app, plus a whole suite of observability features, the new Datadog integration enables users to set up even more granular monitoring customized to their business needs.
Ready to get started?
Simply navigate to your workspace’s Settings > Integrations page, click on the Datadog Integration, and authenticate Census with your Datadog account (more details in our docs). This integration is available on Platform tier plans.
Examples and Use Cases
Keep an eye on your Census workspace’s health, alert on sync failures, and manage incidents in your data pipelines. Census will send metrics and events allowing you to track sync completion statuses and records processed by each ETL workflow. These events and metrics are:
census.syncs.sync_completed - the number of syncs that have completed over a time window
census.syncs.rows_processed - the volume of rows processed in your syncs, useful for anomaly detection
census.syncs.completed - this event allows for alerting and incident management flows if the sync fails, and includes the sync's failure reason.
Alongside each metric and event, Census sends a variety of tags to help you quickly identify which sync is experiencing an anomaly and why.
The Census Datadog integration launches with a recommended dashboard to observe the health of your data pipelines. To load this dashboard in your Datadog account, navigate to the Datadog Integration marketplace and “Install” the integration. Using this dashboard or your own creation, monitor for successful and unsuccessful sync runs, as well as records processed to every destination.
Create alerts in Datadog to notify your teammates when syncs begin to fail. With Datadog Monitors, you will always be the first to find out about an issue in your syncs, so you can resolve the issue quickly. Use monitor notification settings to loop in specific team members who have the skills and permissions to keep your pipelines healthy.
Incident Management and Troubleshooting
When issues arise, you need to act fast and keep your stakeholders informed. Datadog’s Incident Management feature allows you to rally your teammates and update them on incident progress. All of Census’s metrics and events come readily equipped with tags and metadata to help you quickly diagnose a problem, and navigate to the sync that requires your attention. In the Official Census Integration dashboard, you can even use navigation links from your reports like Failing Syncs to land directly back in the Census application without skipping a beat.
Data stacks are getting more complex, chaining together several tools like Snowflake, Airflow, dbt, BI Tools, and more. You don’t need yet-another-dashboard in a SaaS product to understand how it’s functioning. You need a single place to identify anomalies, unify alerting, and triage issues. For many data teams today, that’s happening in Datadog.
If you need more detailed debugging information, you’ll find that Census provides a rich suite of additional observability – Sync Logs of every record success and failure, and API Inspector visibility into every API call to your destination services.
With Entities, data teams can define and expose the most important data in the warehouse as entities, so business teams can sync and segment without code. Now it’s possible to go from data creation to action immediately, without further engineering.
Data Activation is essential for marketing in 2023. 12 thought leaders share their takes on current Data Activation trends and how the data warehouse transforms marketing use cases like Customer 360 profiles, personalization, and automation.
Katy Yuan and
Feb 28, 2023
No items found.
Get the best data & ops content (not just our post!) delivered straight to your inbox