In this article, you'll learn about unexpected mishaps that can occur while operationalizing data, as well as how to avoid them. Hidden roadblocks include:
- API Quotas
- "That's too much data"
- Multiple writers
Syncing data from a data warehouse to operational tools can dramatically improve the efficiency and effectiveness of your business. This is called operationalizing your data. Here are some examples:
- Powering targeted email marketing flows
- Helping customer success teams access the latest data about your users
- Supplying data to digital ads platforms for highly-personalized ad campaigns
Like anything that drives value, operationalizing data can have unexpected roadblocks. It can be time-consuming, error-prone, and costly to troubleshoot these problems when you run into them. In his recent talk at DRE CON, our founder and CEO, Boris Jabes, discussed the various mishaps that can happen on the way to operationalizing data, as well as how to avoid them. I'm here to break down those cautionary tales, but you can see the full presentation here.
If you are helping your company operationalize your data in any way, this article will save you a great deal of time, frustration, and money.
Roadblock one: API quotas
This is the most prevalent disaster. Every operational application, whether it’s a CRM, marketing automation tool, sales tool, etc., comes with API quotas. You can easily overwhelm these applications when you write data into them. Furthermore, the documentation for the API quotas isn't always the easiest to follow.
You can’t just assume “I’m going to hit the data at this rate because the docs say so." Something can still go wrong, no matter how closely you follow the docs, and it can cost you a lot of money. The amount of data you write in a given day likely has both velocity limitations and quantity limitations.
Solution: Spend time to deeply understand the API limitations of each application. Learn how to back off and build incrementally on your system. It’s not easy, but it’s necessary if you want to save money and ensure accurate and up-to-date data.
Roadblock two: "That's too much data"
This may seem almost impossible, but there is, in fact, such a thing as pushing too much data. There are countless applications out there that will happily accept new rows and new columns of data without overwhelming the API quota. However, they don’t make it easy to remove data. If you have one bad JOIN statement (we’ve all been there), you might accidentally push millions of new rows of data that you cannot undo. This can cause countless issues, including:
- Email marketing tools charing you more money
- Duplicate emails going out to prospects (ouch)
- Application slowdowns as they try to process too much data
Solution: Consistently monitor if the tables you are pushing have more data than you expect them to.
Roadblock three: Multiple writers
When you enter the world of operational data, you're not the only writer and you don’t control the application database. There are likely lots of other tools placing data in these applications simultaneously, which can cause your data to be overwritten and duplicated. When this data is inaccurate, the data team is to blame even though they didn't have the knowledge of the other writer(s).
Solution: Develop a hub and spoke model where data flows into a single hub (data warehouse) and out of it via a reverse ETL tool like Census. This lets you control and test all your syncs, including whether or not other syncs are in progress, before pushing data out to a business application. Additionally, you need to align closely with the various departments in your organization so that they don’t freely write data into these applications.
An interim solution is to own a certain number of fields within an application. Example: You align with your customer success team so that they know not two write data into any fields regarding product usage.
Bonus mishaps to look out for:
- The case of disappearing views.
- Supposed “schema-less” applications.
- Date and Time issues. (You experienced this before. 🙃)
- Duplicate entries.
- Orchestrating data syncs rather than scheduling data syncs.
- Basic syntax issues. (Always pay attention to the semantics of applications.)
Don't let all these scary data stories chase you away from your operational analytics dreams. We've created the most reliable (and fastest) reverse ETL tool on the market to take your data worries away (and have a roster of in-house data experts to help you avoid these stumbling blocks). Interested in learning more? Try Census for free today!
The Official Census Blog
