Growing pains: Moving from DIY to the MDS | Census
Nicole MitichJune 02, 2022
Nicole Mitich is the content marketing manager @ Census. She's carried a love for reading and writing since childhood, but her particular focus is on streamlining technical communication through writing. She loves seeing (and helping) technical folks share their wisdom. San Diego, California, United States
Equipping an early-stage startup with a data infrastructure can feel a little like furnishing a freshman dorm room. You have limited funds to use sparingly, so why would you frivolously spend money on a desk when a board laid across two boxes will do the trick? It doesn’t have to be pretty or durable – you just need a low-cost stopgap for the time being.
The business version of that is hacking together “good enough” infrastructure to solve an immediate problem. When new issues arise – usually as a result of the jerry-rigged solution to the first issue – the business responds by patching on another layer.
This hacky process continues and, before you know it, what started as a minor add-on to your telemetry system has morphed into a full-blown event framework. There are now fully-realized workflows with a foundation made up of a board and two boxes, so the resulting system is brittle, shaky, and prone to collapse.
The group agreed that the Rube Goldberg-esque approach of a homemade data stack is not only inefficient and costly, but also a waste of your data team’s time.
The real cost of DIY data solutions
Like the makeshift desk, DIY data tools are often created to save time and money. As your business starts rapidly scaling, you begin to need more information than they can provide. That’s when these DIY tools start to fail and fall apart, bringing your business down with them.
“[When it comes to homegrown data analytics tools], what we generally see could be described as Python scripts wrapped in duct tape,” Donny Flynn said. Without a proper modern data stack (MDS), this duct-taped substitute leaves your entire infrastructure hanging by a thread, requiring regular patching to keep your business up and running.
Over time, all that solution patching starts to add up – making that “budget solution” more expensive (and riskier to your business operations) than its modern data stack counterpart.
“When I was an analyst, I cobbled together an ETL pipeline and found I spent more time fixing it when it broke, patching things, and rewriting models than I did actually getting insights from the data derived from it,” said Jordan Peck. “Companies have these data engineers who are very experienced and very expensive and they’re using them to maintain homemade legacy analytics platforms. It’s not the best use of their time and it’s not why they were brought into the business.”
When engineers’ primary focus is on shepherding data from Point A to Point B, there’s simply no time for quality analysis. Shifting from homemade, makeshift data tools to a modern data stack allows analysts and engineers to focus on their core job: Drawing insights from the data and using those insights to improve your business.
“The modern analytics engineer is not necessarily a builder, but an architect,” said Scott Breitenother. “When you’re rolling on your own, it feels like you’re always just worrying about keeping things from exploding. It’s cool to not have to worry about the technology; it just works.”
The modern data stack encompasses a suite of data integration tools. By way of natural data flow, that includes:
a fully managed ELT data pipeline
a cloud-based data warehouse or data lake as a destination
a data transformation tool (like dbt)
a business intelligence or data visualization platform
Every company should know what it’s good at and focus its energy and resources there, Scott said. By implementing a modern data stack, internal teams can pivot to the part of the data system that should be unique: The business logic.
Can you trust your data?
Homegrown data stacks are notoriously unreliable. No matter how well-prepared you are in the beginning, you’ll encounter things you never thought of along the way and, more than likely, fall into the dreaded DIY data stack money pit. Even when the system works, analysts can’t fully trust the data that came from it. What if one of the many patches distorted the data quality?
You can’t possibly account for every possible clog in your homemade data pipeline. When it encounters an error (or crashes altogether), you won’t necessarily have easy visibility of the problem or knowledge of how to fix it. These problems are only exacerbated with turnover, meaning the people maintaining the system are not the ones who designed it.
“You can find yourself in what essentially feels like a completely unique situation to you, your architecture, and the problem you’ve got to solve,” Jordan said. “If you’ve got something home-built that’s unique to you, there is no Googling around to find a recognized path to fixing your specific issue.”
Outgrowing the DIY data stack
As your business begins to scale, a modern data platform allows your infrastructure to properly scale with it. Because it’s typically designed for a very specific use case, expanding the parameters of a home-built solution can be both time-consuming and tedious.
Business requirements will change over time – and your data infrastructure must be able to change with them. Admittedly, this evolution is difficult even with reliable tools, since engineers are constantly re-evaluating which metrics to track and what to do with them once they have them. This evolution becomes even more infeasible with unreliable tools. For every new layer your team adds to the DIY system, new opportunities for error are introduced and, as errors surface, you need to make adjustments to correct them.
The more your business tries to grow, the messier the infrastructure becomes until it eventually gets in the way of business operations. Although the homemade infrastructure served you well in the short-term, once you get into the long-term, it’s time to upgrade.
Don’t worry – when it’s time to move out of the dorm room, retire the board and boxes, and buy a real desk, it’s often not as expensive as you’d think. Investing in a modern data stack isn’t even an expense as much as it is a reallocation, so instead of paying engineers and analysts to keep an eye on the pipeline and slap duct tape on the leaks, you’ll pay them for their analysis and insights to drive your business.
“You want to think about what data you want to collect, what analysis you want to do, and the properties and parameters associated with it. You don’t want to think about technical management and overhead,” Jordan said. “When you can trust in the tech, you can start thinking about the business problems.”
Want to learn more? Check out the full webinar with Snowplow and BDC here. Got thoughts and opinions about this topic? Join the conversation around this, and many other data best practices, in The Operational Analytics Club.
