Best Practices

The most important part of your planning | Census

Nicole Mitich
Nicole Mitich June 07, 2022

Nicole Mitich is the content marketing manager @ Census. She's carried a love for reading and writing since childhood, but her particular focus is on streamlining technical communication through writing. She loves seeing (and helping) technical folks share their wisdom. San Diego, California, United States

As the age-old adage says, “The best-laid plans of mice and men often go awry.” And it’s true – building a plan on a massive foundation of data doesn’t make it fail-proof.

Don’t believe us? Just try talking to a few companies whose “Vision 2020” plans were demolished by COVID-19. 😷

“The presence of data gives you the false sense of confidence that you’re going to be right,” explained Barr Moses, Co-founder and CEO at Monte Carlo. “Let’s say you have the best recruiting process. You collect all the data and you follow the perfect process. You still have a one in two chance that the person you hire will be the wrong fit.”

So, if a data-driven process has the same odds of success as a coinflip… Why bother planning at all?

Because the value is in the journey, not the destination. 🧳

“Planning is nothing but planning is everything,” said Boris Jabes, Co-founder and CEO of Census. “You learn; your team learns. There are ancillary benefits.”

The value is in the planning, not the plan

While the chances of everything going to plan are slim, the very act of planning is beneficial. Take the ever-popular “X”-year plan. Whether you’re looking three, five, or ten years into the future, assuming you can predict what will happen in the future is crazy talk.

Of course, you can make guesses based on current trends, but there’s no definitive way to know when an obstacle will rear its ugly head (or what obstacle it will be, for that matter). After all, in 2017, would you have predicted the fallout of a global pandemic? 🤧

Although you can’t foresee the obstacles that might crop up several years in the future, making a plan can uncover the milestones you need to hit to meet your ultimate goals. If you don’t have any initiatives in mind, how do you know what you’re working towards?

For instance, let’s say your three-year plan is to grow your company by 50%. Plotting the course could show you when you will need to hire more people or move to a larger facility. Either way, you’ve not only identified the “what”, but also the “when.”

“The plan itself is always subject to change, and you should revisit it and examine it often,” Barr said. “But the planning process is important because it forces a couple of things.”

Planning forces you to imagine scenarios

Sure, a good plan includes a series of milestones, but it’s so much more than that. It also includes fallbacks for what you will do when obstacles force you to re-examine your roadmap. Detour, anyone?

Barr says Dwight D. Eisenhower summed the entirety of planning up perfectly when he said, “Plans are worthless but planning is everything. When you are planning for an emergency, you must start with this one thing: the very definition of an emergency is that it is unexpected, therefore it will not happen the way you are planning.”

So, we can’t know with certainty what is going to happen in the future, but planning forces us to imagine what might happen, so we can be ready to roll with whatever punches come our way. 👊

Planning helps you to take stock of your resources and rank your priorities

A plan enables prioritization, setting out an organizational goal, so all team members can orient their deliverables around it. Without a broader plan to guide them, teams tend to prioritize what matters to them rather than what matters to the company. With a solid plan in place, you have a clear picture of your available resources and can rank your priorities to match, boosting your confidence to make decisions in the field (not to mention your business value). ✅

“Left to our own devices, there is a default for data teams to set OKRs – like making sure other teams are using our platforms,” Boris said. “The right answer is, ‘If the company as a whole wants to reach this goal, what is our team’s part in that?’”

One of the common pitfalls of OKRs and KPIs is forgetting the primary goal, Barr said. It’s easy for humans to become obsessed with beating a metric just because it’s there, even if it’s a vanity metric that won’t really impact business success. 🎮 While reaching the carrot is a great short-term goal to eventually get you to the end goal, sometimes we’re so focused on the carrot that we forget about the finish line entirely.

“This happens not just in data or analytics teams, but teams in general: Focusing on things whose value is not totally clear,” Barr said. “There’s misalignment on whether this project will have an impact on the company or not.”

Leaving room to explore

Without a plan that sets priorities, a lot of time is wasted. One of the most frustrating examples is when the data team is tasked with answering a time- and labor-intensive question, only to find out it was a low-value metric that stakeholders didn’t really care about anyway. Or worse, they hustled to answer it, but then the question was changed entirely, making the once-valued answer and all endeavors moot.

Businesses could waste far less time if they actually prioritized freeform data exploration as part of their plan. Most of a data team’s time is spent putting out fires or juggling requests of varying importance, Boris said, but their greatest impact comes when they are sifting through the data for the sheer delight of it. When you’re looking for patterns no one requested, you come across the answers to high-valued questions that no one has asked. 💡

“I think it’s very hard to make an incredible impact without prioritizing – and the biggest impacts often come from time intertwined with play,” Barr said. “If the data team has time to explore and examine the data, we may be able to learn things that could change the trajectory of the company. The questions no one knows to ask might be the most interesting part of data.” 🤔

Planning for the next evolution

Planning forces a level of documentation that makes it possible for teams to pass the torch when the time comes. Only a few years ago, when data teams were smaller – and Big Data itself was not so big – documenting as we went felt like a waste of time, so it was thrown on the back burner.

Today, downtime and data issues are frequently caused by systems being handed over to new people without the definitions and documentation to make them manageable. 😧 Documenting code is easy during development – when you’re deep in the trenches – but it gets harder the longer you wait.

“In a weird way, the constraints of something like SLAs, that feel like an extra load, are actually freeing,” Boris said. “You shouldn’t want to own this forever. At some point, you should be able to go and do other things and pass the torch to someone else.”

Handoffs are an inevitable part of software engineering and software development (no one works on everything forever!), yet the same hiccups tend to happen time and time again. As a rule of thumb: when you learn something new that could be useful to others, those docs should be updated – and planning facilitates that.

Plans are living things that will grow and change as information is uncovered and events unfold. But the real value in making a plan isn’t the plan itself, it’s the process you take to get there. 🌄

Want to learn more? You can catch the full conversation between Boris and Barr below, or on your favorite streaming platforms. 🎧

Then head on over to The Operational Analytics Club to join the conversation around this and many other data best practices. ✨

Related articles

Customer Stories
Built With Census Embedded: Labelbox Becomes Data Warehouse-Native
Built With Census Embedded: Labelbox Becomes Data Warehouse-Native

Every business’s best source of truth is in their cloud data warehouse. If you’re a SaaS provider, your customer’s best data is in their cloud data warehouse, too.

Best Practices
Keeping Data Private with the Composable CDP
Keeping Data Private with the Composable CDP

One of the benefits of composing your Customer Data Platform on your data warehouse is enforcing and maintaining strong controls over how, where, and to whom your data is exposed.

Product News
Sync data 100x faster on Snowflake with Census Live Syncs
Sync data 100x faster on Snowflake with Census Live Syncs

For years, working with high-quality data in real time was an elusive goal for data teams. Two hurdles blocked real-time data activation on Snowflake from becoming a reality: Lack of low-latency data flows and transformation pipelines The compute cost of running queries at high frequency in order to provide real-time insights Today, we’re solving both of those challenges by partnering with Snowflake to support our real-time Live Syncs, which can be 100 times faster and 100 times cheaper to operate than traditional Reverse ETL. You can create a Live Sync using any Snowflake table (including Dynamic Tables) as a source, and sync data to over 200 business tools within seconds. We’re proud to offer the fastest Reverse ETL platform on the planet, and the only one capable of real-time activation with Snowflake. 👉 Luke Ambrosetti discusses Live Sync architecture in-depth on Snowflake’s Medium blog here. Real-Time Composable CDP with Snowflake Developed alongside Snowflake’s product team, we’re excited to enable the fastest-ever data activation on Snowflake. Today marks a massive paradigm shift in how quickly companies can leverage their first-party data to stay ahead of their competition. In the past, businesses had to implement their real-time use cases outside their Data Cloud by building a separate fast path, through hosted custom infrastructure and event buses, or piles of if-this-then-that no-code hacks — all with painful limitations such as lack of scalability, data silos, and low adaptability. Census Live Syncs were born to tear down the latency barrier that previously prevented companies from centralizing these integrations with all of their others. Census Live Syncs and Snowflake now combine to offer real-time CDP capabilities without having to abandon the Data Cloud. This Composable CDP approach transforms the Data Cloud infrastructure that companies already have into an engine that drives business growth and revenue, delivering huge cost savings and data-driven decisions without complex engineering. Together we’re enabling marketing and business teams to interact with customers at the moment of intent, deliver the most personalized recommendations, and update AI models with the freshest insights. Doing the Math: 100x Faster and 100x Cheaper There are two primary ways to use Census Live Syncs — through Snowflake Dynamic Tables, or directly through Snowflake Streams. Near real time: Dynamic Tables have a target lag of minimum 1 minute (as of March 2024). Real time: Live Syncs can operate off a Snowflake Stream directly to achieve true real-time activation in single-digit seconds. Using a real-world example, one of our customers was looking for real-time activation to personalize in-app content immediately. They replaced their previous hourly process with Census Live Syncs, achieving an end-to-end latency of <1 minute. They observed that Live Syncs are 144 times cheaper and 150 times faster than their previous Reverse ETL process. It’s rare to offer customers multiple orders of magnitude of improvement as part of a product release, but we did the math. Continuous Syncs (traditional Reverse ETL) Census Live Syncs Improvement Cost 24 hours = 24 Snowflake credits. 24 * $2 * 30 = $1440/month ⅙ of a credit per day. ⅙ * $2 * 30 = $10/month 144x Speed Transformation hourly job + 15 minutes for ETL = 75 minutes on average 30 seconds on average 150x Cost The previous method of lowest latency Reverse ETL, called Continuous Syncs, required a Snowflake compute platform to be live 24/7 in order to continuously detect changes. This was expensive and also wasteful for datasets that don’t change often. Assuming that one Snowflake credit is on average $2, traditional Reverse ETL costs 24 credits * $2 * 30 days = $1440 per month. Using Snowflake’s Streams to detect changes offers a huge saving in credits to detect changes, just 1/6th of a single credit in equivalent cost, lowering the cost to $10 per month. Speed Real-time activation also requires ETL and transformation workflows to be low latency. In this example, our customer needed real-time activation of an event that occurs 10 times per day. First, we reduced their ETL processing time to 1 second with our HTTP Request source. On the activation side, Live Syncs activate data with subsecond latency. 1 second HTTP Live Sync + 1 minute Dynamic Table refresh + 1 second Census Snowflake Live Sync = 1 minute end-to-end latency. This process can be even faster when using Live Syncs with a Snowflake Stream. For this customer, using Census Live Syncs on Snowflake was 144x cheaper and 150x faster than their previous Reverse ETL process How Live Syncs work It’s easy to set up a real-time workflow with Snowflake as a source in three steps: