Built by artists, for artists, Big Cartel provides a platform for creators to make money doing what they love. Its e-commerce platform is designed to help artists, makers, and small businesses build and manage their own online stores through a simple and intuitive user experience and customizable templates. Big Cartel’s creative community roots show in their robust “start for free and upgrade as you grow” pricing philosophy, and their commitment to providing resources and support for artists and makers.
● Visibility: “Data all over the place” -> unified data control plane
● Usability: Bespoke and completely unstandardized -> modern composable platform
● Error detection: “Wait until the dashboard breaks and it's discovered eventually by someone" -> Slack-integrated "Data Firehose"
● Data consistency: YBDMMV (Your Business Data Metrics May Vary) → Accessible, consistent, trusted data
Accelerating data reliability and developer productivity
Patrick Trainer describes himself as a “data engineering team of one” hired to turn Big Cartel's patched-together data system into a modern and efficient platform.
This is the story of how a one-person data engineering team transformed Big Cartel’s disconnected and inconsistent data silos into a cohesive, observable platform with Dagster at its core, creating a unified control plane that provides complete visibility across Big Cartel’s entire data ecosystem that:
● Implements Dagster's asset-based approach to achieve reliability by design, eliminating the pain of waiting for dashboards to break and replacing it with proactive monitoring and intelligent alerting.
● Delivers on the promise that data engineers can have nice things too with local testing, branch deployments, and a clean development workflow that minimizes cognitive load.
● Can grow along with the organization and serve as the foundation for building new features like customer-facing data products.
● Uses Dagster as a “Swiss army knife” that efficiently handles current needs while providing a “test bed” for developing new capabilities and features for Big Cartel’s customers
With Dagster, Big Cartel now has a future-proof data platform that scales from internal analytics to customer-facing insights — demonstrating how the right orchestration solution can transform not just your data operations, but your entire business’s approach to data.
Fragmented systems and limited visibility
Hired to build out Big Cartel’s data platform, Patrick found that “data at Big Cartel was in a fairly immature state.” Data was siloed in different applications with no central organization or unified view. Existing processes relied on one-off Python scripts, ad-hoc SQL queries, or random cron jobs — a setup he describes as “‘bespoke’ and completely unstandardized. There was zero observability into data pipeline performance or reliability and no foundation for building advanced analytics capabilities. Worst of all, he says, it meant his Big Cartel colleagues were getting inconsistent answers to business data queries, depending on timing and source.
One of the first things Patrick tackled was Mage, the data orchestration tool in place when he arrived at Big Cartel: “I hashed it out for a little bit and quickly saw it was tough to use."
What Patrick was looking for in a data orchestration platform:
● Flexibility and breadth: Patrick needed a tool that could handle a variety of use cases beyond just pipeline scheduling
● Enhanced observability: After experiencing siloed data with no visibility, Patrick wanted comprehensive monitoring of data flows
● Unified control plane: The ability to connect different systems and provide a single view was essential
● Modern developer experience: As an experienced data engineer, Patrick valued tools with clean development workflows
● Easy deployment: The platform needed to be "easy to stand up" without excessive configuration
● Visualization capabilities: Patrick sought a tool that could clearly illustrate data dependencies to non-technical stakeholders
● Alerting and monitoring: The ability to proactively detect issues rather than waiting for dashboards to break
● Integration capabilities: Support for connecting with their planned tech stack (Airbyte, dbt, Snowflake)
● Future-proof architecture: A foundation that could grow from internal analytics to customer-facing data products
In terms of observability, there really wasn't anything in place. Everything was siloed in different applications and was really more or less bespoke. Not exactly a great foundation to build on. So I ripped that out and put in Dagster.
Why Big Cartel chose Dagster
As a data team of just two people with ambitious goals (“Besides me there’s a data analyst, we’re a scrappy duo”), Big Cartel needed a flexible and comprehensive — but not cumbersome — solution that could grow with them.
Patrick had previous experience with multiple orchestration tools, including Airflow, and none of them felt like the right answer. “I’d run the gamut and basically I knew what tools that I didn't want to use,” he says. He had been hearing good things about Dagster and was intrigued by Dagster’s asset-centric approach to data orchestration: “I liked how workflows are organized around data assets rather than just processes or tasks.”
From the start, he liked how easy Dagster was to stand up. Patrick also appreciated how Dagster offered:
● Modern, composable platform: A more modern orchestration experience with strong support for software engineering practices — and data-centric architecture that aligns with modern data engineering practices.
● Zero complexity: Dagster offered the simple deployment and straightforward local development experience Patrick needed as a team of one.
● Swiss army knife: Big Cartel’s data platform would be rapidly evolving, and Dagster offered both flexibility and a wide breadth of use cases beyond basic orchestration, able to serve as a "Swiss army knife" for a variety of data challenges.
● Democratization of data workflows: Clean, understandable visualization of data flows for all of Big Cartel’s data users, both technical and non-technical.
I liked the flexibility and the breadth of use cases that there are. And with that I saw an opportunity for using Dagster as more than just an orchestrator... Being able to leverage it for all of the roadblocks that I know we will eventually face. And so it's good to have a Swiss army knife of a tool, which is what Dagster is.
Automation, innovation, and lightening the cognitive load
Patrick took a clean-slate approach to Big Cartel’s data platform architecture, designing a modern data platform with Dagster at its core. His goal was creating a robust foundation that could scale with Big Cartel's needs while “keeping data as simple as possible” for stakeholders.
Modern deployment architecture: Big Cartel deployed Dagster on Kubernetes using Helm charts for container orchestration, which included:
● Implementing a branch-based deployment workflow with separate environments: "We deploy on Kubernetes with Helm, a branch deployment running two kind-of concurrent branches: a main, which is prod, and then one for staging."
● Creating feature branches off of the staging branch for development before promoting to production.
● Automating the entire deployment process through Buildkite: "We use Buildkite to build a Docker image of our location."
● Using Dagster’s separate deployment model for distinct instance and code location configurations.
Development workflow: Patrick was heavily focused on automation through declarative configuration and pipelines. “It’s all about lessening the cognitive load. Otherwise I know I’m going to mistype something or forget how to do it," he said. So he:
● Established a streamlined local development process using Dagster Dev to just “spin up local dev right then and there."
● Created a testing environment that could mimic production: "Other times I'll want to test locally running Kubernetes... to try and mimic our production deployment as much as possible"
Big Cartel’s "Data Firehose": Patrick created a specialized Slack channel that centralizes all system events via Dagster. "I call it the data firehose: successful alerts, unsuccessful alerts, pull requests, meta queries, everything goes into one channel." This has many benefits, including:
● Using this consolidated view to develop intuition about system behavior: "When we're reading through that... you start to get a gut feel of the patterns... it allows you to internalize what the beat of your system is"
● Leveraging this approach to detect anomalies: "If you remove all of that prescriptiveness and just put everything in one place, if something eventually goes missing then you'll pick up on that."
● Using the firehose to understand how new tools affect the system: "As you bring in new tools you can see how that affects that beat of that flow... See where that piece fits into everything else."
Tech stack integration: Building out the rest of Big Cartel’s data architecture, Patrick:
● Connected Dagster with Airbyte OSS for data loading.
● Integrated dbt for transformation, mapping dbt models as Dagster assets.
● Incorporated various data sources including Amplitude for product analytics.
● Used Snowflake as the data warehouse, with comprehensive monitoring.
● Created a custom "meta layer" of queries to monitor Snowflake usage: "Being able to understand what is running or what is there in Snowflake, what is being used, what hasn't been used in a long time."
The cognitive load is what I'm trying to minimize. And Dagster's made that simple by allowing you to glue together a lot of different things and push them to where you need them to be pushed.
Accessible, reliable, simplified data.
Within six months of implementation, Big Cartel's Dagster-based data platform has transformed how the company works with data, creating a foundation for data-driven decision making and future growth. At Big Cartel, life with Dagster has meant:
Visualization for stakeholders: One of Patrick’s major goals was making data understandable and immediately useful for everyone else at Big Cartel. He says that Dagster’s asset-based approach, focusing on the data products themselves rather than the processes that create them, has made dependencies more explicit and workflows more maintainable — which has had “huge value” across the company.
● Non-technical users can easily understand data lineage with Dagster’s asset view.
● Made complex data flows understandable, which has been “super valuable" for Big Cartel’s data analyst.
● Dagster’s asset visualizations have replaced lengthy technical explanations: "You can see people's eyes glaze over when you're talking to them trying to connect all of these dots. Pictures just hit a lot nicer for them."
Clarity, both technical and business-related: Enhanced observability across the entire data platform has reduced cognitive load through centralization and automation, with the Data Firehose at the center. Big Cartel now enjoys:
● Proactive issue detection instead of waiting for dashboards to break.
● Consolidated access to information without navigating multiple systems.
● User trust in data, thanks to consistency in business data queries, regardless of timing.
● Better institutionalization of knowledge across teams.
● A clear roadmap for data capabilities that sets appropriate expectations.
● Improved reliability with monitoring for pipeline failures and Snowflake issues.
● A local development environment that streamlines testing and deployment.
Future Dagster features to be implemented: Patrick says that Dagster helped Big Cartel build a strong foundation for future capabilities, like customer-facing analytics as part of the Big Cartel product offering. The roadmap also includes implementing a data quality layer, and a Tableau integration for managing extract schedules: "I'd like to use that to put that last mile piece of the picture into the graph."
Everybody wants to use data and everybody wants to have their questions answered, and that’s exactly what Dagster does. You don't have to go to a bunch of different places to get these things. When you ask somebody to pull some data for you, you don't get a mix of different numbers every single time, depending on the day that you do it. Dagster helps us institutionalize knowledge across teams.
