Most data mistakes don’t look like mistakes. They look like smart, reasonable decisions—but based on numbers that weren’t fully understood.

Teams often think they’re aligned because they’re using the same words:

“Revenue.” “Users.” “Customers.”

But they’re not.

They’re working from different definitions and no one realizes it until something breaks. Then it shows up as rework, wasted time, and wrong turns.

This is a problem of false alignment. It’s common, it’s frustrating, and it’s almost never talked about directly.

Let’s change that.

Here are 7 common offenders that wreak havoc on well-meaning teams, and what you can do about it.

Share

The Sneaky Seven: Metrics that always need a definition

#7: Leads

“Leads” (or prospects) sounds simple, but can have very different meanings across teams. Marketing might count a whitepaper download. Sales may only count someone who requests a meeting.

Depending on the system used to capture them, leads coming from different channels (sales vs. web vs. social, etc.) aren’t always captured or categorized consistently.

Now layer in intent and timing. Are all leads equal? Probably not. And how long is someone considered a lead if they don’t convert into a sale - days, weeks, months?

#6: Traffic

“Traffic” can mean foot traffic at a store or event, or digital activity online. And within each, there are multiple ways to measure it.

At an industry event, traffic could mean total attendees or only the people who stopped at your booth. On a website, it could be page views, sessions, engaged sessions, users, or unique users. Each of these tells a different story and has a different context.

#5: New Product Revenue

“New product revenue” is often used to measure innovation and growth, but there’s usually a lot of debate about what counts as “new.” Timing might be limited to products launched just this year, or, if a product has a long sales cycle, it might include the first few years of a product’s lifecycle.

The type of product also needs definition. Teams differ in whether they count only new standalone or if major new features for existing products are included as well.

Don’t forget to clarify revenue. Some teams use projections. Others use actual revenue. Others use sales booked.

#4: Users and Usage

“Users” and “usage” are highly dependent on your product and distribution model, especially in businesses that deliver services online like banking, software, streaming, or gaming.

A user may be anyone with a login or only someone that is active. But what counts as ‘active’ also varies and needs definition. Perhaps it is someone who accessed the system in the past 30 days, or who took a specific action, or spent a minimum amount of time.

#3: Customers

Counting customers gets surprisingly complicated, and definitions can vary between teams or product lines within the same organization.

In B2B, the first question is usually whether the number is counting companies, contracts, or individual users. One company might have multiple contracts and thousands of users.

In both B2C and B2B, the buyer and the end user may not be the same person - like a team signing the contract while another uses the product, or even a parent buying a toy for their child. This is also an important distinction.

Then there is timing and services. A customer who purchased last year, but not this year, may not be counted by one team, but is included by another. How free subscribers or limited trial / promotional customers are counted may also be a discrepancy between teams.

#2: Retention

Retention (or renewal rate) is the percentage of customers, or revenue, you keep over a period of time.

This metric gets muddy quickly and needs clarification on whether it’s measuring:

• customer retention or revenue retention

• a specific cohort (”customers who signed up in January”) or the whole customer base

• upsells, new customers, or pricing changes

Depending on what is included, retention can exceed 100% and mask churn rates, or not. For this reason, the team might decide for multiple retention metrics based on their objectives.**

#1: Revenue

“Revenue is revenue”… until it isn’t.

Sales vs. revenue. Recognized vs. booked. Gross vs. net.

Revenue is one of the most misunderstood numbers because a few key distinctions get blurred.

Sales and revenue are often used interchangeably, but they’re not the same. Sales reflect what was sold in a given period. Revenue reflects what the company is allowed to recognize as earned.

When looking at revenue, you need to understand how it is “earned” or “recognized.” Depending on the product or service, revenue might be counted when a product ships, when it’s delivered, or as it’s used. Or it might follow a defined schedule - so a 12-month subscription sold today might have revenue that is split over 12 months.

And what counts as revenue isn’t always obvious either. In marketplace models, like Etsy or AirBnB, $100K in transactions might translate to $5K in revenue from fees or $105K, depending on how it’s defined.

Why are these metrics so “sneaky”?

With so much variability, you’d think that we’d always want to clarify the definitions. But we don’t.

Familiar words create a false sense of clarity.

Revenue. Users. Customers.

They sound obvious. So teams assume alignment without confirming what’s actually being measured.

And even when something feels off, people often don’t ask. They feel like they should already know. Or they don’t want to slow the meeting down. Or it’s uncomfortable to challenge a number someone else owns.

So instead, people stay quiet and move on.

When you combine:

• One word that can mean many different things

• A tendency not to pause and clarify

You get false alignment. Nothing breaks right away. It shows up later as confusion, rework, wasted time, and mediocre to poor decisions.

How do you get clarity?

Start here: ask the question. What is included in this metric? How is it defined? Just ask.

Yes, it might feel uncomfortable. But chances are, someone else is wondering the exact same thing and once you say it out loud, you’ll realize the room wasn’t aligned after all.

And when the conversation gets fuzzy, use this playbook:

1. See it for yourself. Go to the source. If it’s a lead, walk through the website or flow. What actually happens when it is captured?

2. Run a quick scenario. Get clear on yes/no cases. “If I download the e-book, am I a lead? What if I request a demo? What if I chat with the bot?”

3. Clarify what’s included. “Does this lead number include only website activity? Or also social, CRM, and sales team inputs?”

4. Tie it back to the report. “So 108 leads in April means 108 people completed web forms or chats, and this excludes phone calls and sales-sourced leads. Correct?”

Your goal: understand it well enough to explain it to someone else simply.

And if you’re being measured on it? Don’t leave the conversation until you know exactly how your actions move that number.

Thanks for reading. Do you have another “sneaky” metric we should add? We’d love to hear it!

Leave a comment

Share

**If you want to dig into more on retention, this article from Churnzero is one of the more helpful explanations we’ve seen. We have no affiliation, we just appreciate their clear explanation of retention and when to use it.

Bonus Reading: Other sneaky metrics (but for different reasons)

Percentages add another layer of confusion because they depend on multiple underlying definitions, especially when it comes to conversion rates and trends. We’ve covered these in more detail in other articles. You can read more below.

We have a lot of new subscribers here this week. Welcome! We are glad you are here. This is a longer post than normal because we detail how we are extracting our Substack data for analysis, where we use AI and where we don’t (and why).

For those of you not on Substack, while the example is focused on a specific platform, the general ideas and findings are similar to what we see when looking at data coming from other platforms like Google Analytics, Shopify, etc. We are in learning mode, so if you have any suggestions on tools or things that worked well, we’d love to hear them!

- Rachel

Leveraging data to accelerate results. That’s what we help managers and entrepreneurs do - by using a disciplined approach defining objectives, selecting supporting metrics, evaluating data inputs, and making sense of the outputs. As we build our company, we turn that tough-love advice on ourselves, especially when it comes to our Substack newsletter.

So this week, we share how we’re handling Substack performance data. While it’s a workable model specific for our Substack goals, it is also adaptable for other creators’ goals.

Initially, we used Substack’s native dashboards. While they’re helpful, we soon found that they didn’t fully answer the questions we care about. So we built a more hands-on process, focused on understanding what drives success for us.

In this post, we will:

• Explain how we use Substack data and where we ran into limits.

• Argue why you should look at the data itself and why AI alone is not enough.

• Detail the steps we followed to unlock our Substack data and what we learned.

Substack metrics we care about (right now)

We identified early on what we were not trying to do: we weren’t focused on paid subscribers or trying to maximize every possible metric.

We focused on building the right audience for us, and understanding what content was resonating. That’s what supports our larger goal, which is helping managers and entrepreneurs build a data feedback loop into their everyday work to accelerate their success.

Along the way, we refined our core metrics to a targeted set:

• Relevant audience growth: subscribers and followers, across Substack and LinkedIn.

• Content engagement, such as shares, restacks, and comments

• Community building: interactions, connections and participation

But all of these metrics are not easily viewed over time and analyzed in the Substack dashboards.

How we use these metrics

We are leveraging the data in two key ways:

1. Ongoing monitoring. We review our target metrics regularly - usually weekly, some monthly - alongside what we actually did in that time period.

2. Analysis. We also wanted to compare different types of posts, notes and other activities. But we needed the data to support looking for patterns between effort and response.

Getting closer to the data

To achieve this, dashboard summaries or just feeding it into AI for analysis were not enough. We needed access to the underlying data so we could explore it directly and see what was being captured (and what wasn’t).

We wanted to build our own understanding of what is happening rather than rely on the conclusions coming from dashboards and AI.

These were good instincts because the dashboards didn’t tell the whole story, and the AI made assumptions and had incomplete conclusions.

So we set up a process and supporting system to extract, store, and explore our Substack data outside the platform.

Here’s how we did it - the quick version

Here’s the overview of what we did. If you want more detail, we have you covered. There is a step-by-step guide on how to do it yourself at the end of this post.

Extract the data
We use StackContacts (from ) to pull Substack data, including subscribers, posts, and engagement, into a local database. This saves us from building API integrations ourselves and gives us access to more than what’s visible in the Substack dashboards. Once configured, the data syncs from Substack to a database, and can be automated to run regularly.

Store the data
The data is stored in a local DuckDB database. That means we own it and can query it directly. We aren’t dependent on a platform interface, and we can preserve our history.

Query the data
We created a handful of SQL queries to pull the data related to our key Substack metrics, such as audience growth and content engagement. As we learn more about the data, we refine the queries.

Connect the data to Excel
Our queries are embedded in an Excel file that we refresh weekly after syncing the data via StackContacts. We used an ODBC connection to directly tap the database from inside Excel.

Use the data to monitor progress and build a feedback loop
The Excel file gives us a consistent view of our key metrics over time, alongside what we actually did that week. It supports a simple, repeatable, review process with familiar tools and a critical subset of our data. Now that this is set up, it takes us only a minute to refresh the data.

Explore the data using AI
Of course, we’re also experimenting with AI tools to analyze the data. This can be helpful for generating queries and surfacing patterns - but only once we have a solid understanding of the underlying data. In our case, the Claude desktop app is accessing the DuckDB database via Model Context Protocol (MCP). This configuration is part of StackContact’s installation process.

Overall, this setup isn’t particularly complex once broken down into its parts, and it gives us something we didn’t have before: direct access to our own data, and the ability to shape how we measure and interpret it.

Want more details? A deeper dive is at the end of this article.

Why we collect more than we need right now

Even though we focus on a small set of metrics, we’re intentionally collecting more data than we actively use. Our early-stage history is valuable - and we don’t know yet what we might want later. Patterns that don’t matter now might matter at scale. Older behavior might become useful for comparison. Additionally, platform definitions can change; we’ve already seen shifts in how metrics like open rates are reported on Substack.

Keeping more complete data now gives us flexibility later.

Why the data was harder than expected

Once we had the data, we ran into something we often experience with clients: the Substack data itself was not cleanly organized.

• There were many tables, with many fields, but no data dictionary with definitions.

• Some fields looked similar, but meant different things.

• Other fields were empty or inconsistently populated.

• Mapping fields back to actual Substack activity took work, and several iterations.

In other words, the data didn’t immediately tell a clear story “out of the box.” We had to look at the actual data to get familiar with it - what the fields meant, which similar sounding field was the right one, how tables related to each other, and what was actually reliable.

For example, we track subscribers and followers weekly. We have been doing this manually, and wanted to pull richer data from Substack in a more automated way. So we developed a SQL query via the AI and produced a tidy data output - except for one problem. It said our followers went back to 2021, which isn’t possible. The issue? The field used in the query was the date our followers created their Substack profiles, not the date they began following us.

These structural issues and lack of data definitions are important to know when considering how you apply AI to analyzing your data.

Where AI helped and where it didn’t

Part of our setup allows AI tools to query and analyze the data. It would be tempting to leave it to AI to learn the structure and quirks of the data itself, but in practice, this can (and did!) backfire.

The detailed data is where you learn what your dashboard really summarizes and when to question the overly confident responses AI gives you.

The first question we tested with the AI was subscriber count. The resulting total subscribers was correct, but all free subscribers were labeled as “unsubscribed.” That was a fundamental misunderstanding of the data - and if that is wrong, everything built on top of it is wrong.

We had given the AI little context or instruction, and it was easy enough to prompt for correction. But even after iterating and guiding the AI through the database, there were still issues.

But if you operate with caution, the AI does bring real value. It helped:

• Explore the data structure: it reliably identified which tables could be safely ignored for our purposes, and teased out relationships much faster than we could have. It was also significantly faster in working through the many fields looking for relevant data. Even when there were errors, the second review generally picked up the right field.

• Generate queries: You don’t really need to know SQL (though a little bit always helps so you know what your query is doing). Rapid development and editing of queries saved hours. We know our strengths, and SQL syntax is not one of them.

• Quickly surface key insights: Simple quick analysis, such as top performing posts by a variety of measures, tended to be reliable. We also tried out topical analysis, having the AI classify our posts and break down performance by theme. The groupings were very close to how we think about our content, which gave us confidence in the approach. More importantly, it highlighted both areas of strength and gaps where we may want to develop new content.

Going forward, given that we now better understand the detailed data, we will continue to use that knowledge to prompt AI to generate additional queries as well as be an analysis partner. We may also use it to run data quality checks when we update and refresh the data.

Summary: what this gives us

This process gives us a more useful set of numbers and a way to work with our data more deliberately.

We’ve chosen a small set of metrics to guide decisions, while keeping a broader record so we can learn over time. We’ve built a simple process we can repeat, refine, and actually use. And we’ve tested where AI can help - and where it can quietly lead us in the wrong direction if we don’t understand the data ourselves.

This is what it looks like, in practice, to build a data feedback loop: define what matters, get close to the data, and use it to inform what you do next.

If you’re interested in how this is set up in more detail, we’ve outlined the technical setup below.

Step-by-step guide: How to do this yourself

We’ll walk step-by-step through the setup. Note that we’re operating on PCs, not Macs. So while you can do all of this on a Mac, it’s going to look different.

In addition, the install manual for StackContacts is extensive, and we will not repeat all of that detail - but it may help you to see some of the setup screens and understand where things went right and wrong for us.

Step 1: Extract your data

1. Purchase StackContacts.

Obtain ’s StackContacts. Note that there are a number of editions to fit your environment: we have just one publication (Practical Data Foundations) and do not need the Gumroad or Kit integrations, so we bought the Hobbyist edition.

Note, this part of the process took us a little while, so set aside a few hours (and possibly more) to do this.

2. Review the install manual and complete prerequisite installations.

Read the install manual! It’s long. It’s dense. But very helpful. There are prerequisites before you can install StackContacts itself, including Node.js, npm, and the MCPB CLI tool. You need to know how to open a command prompt or PowerShell, and how to run those as an administrator. (Don’t know how? Do a search - AI results and lots of helpful shortcuts will pop up). In addition, you’ll need to install either the Claude desktop app or Cursor AI app (we used Claude).

3. Install StackContacts

The install manual will direct you to the product page for the latest installer. Run the installer, follow the prompts and then launch StackContacts. Be patient - it took a couple of minutes for our initial launch of the software.

If you run into difficulty: in our case, the Visual Studio install was accidentally interrupted at some point, which led to failures later in the install process. We went back and installed Visual Studio again, manually. Worst case? Uninstall, then start over.

4. Setup StackContacts with publication and cookie info

Follow the install guide to set up your publication. First, you’ll need to tell StackContacts how to communicate with your Substack publication. Using that login, a cookie is pulled from a specific Substack page, establishing the connection between StackContacts and your Substack stats.

When you click Login, the app will give you the following window:

The automated process described in the manual and what the app defaults to did not work for us, producing login errors. We had to manually pull the cookie values. It’s described in the manual, but here’s what it looks like in Chrome.

5. Sync your data

Once login was set up, the sync process itself was smooth. We’re pulling every possible metric we can from Substack. Our syncs are taking about 30 minutes, so be aware that if you have a lot of data or multiple sites, this may take a while. Our install is running on one local PC; the app is not kept running all the time so we’re not using the automatic syncing (it will fire only when the app is open and your PC is “awake”), but you can set up a date and time for an automatic sync.

Tip: When you sync, you are pulling all available data. What you had in the database before is overwritten - you’re not just appending data that’s new since last time. This has implications; notably that if you want to preserve history as it was, you’ll need to periodically back up your database. For example, say Substack stops making data on DM threads available. If you allow the database to be overwritten, you will lose that particular historical data.

6. Finish AI integrations

From here, you finalize the install by configuring either Claude or Cursor AI to enable AI-powered insights atop the data. For Claude, this meant creating and enabling the appropriate extensions. This process was smooth, aligning with the instructions in the manual.

A step we recommend: once you have completed the actual install, open Claude and instruct it how to find the database. Our initial prompts (e.g. show me my last 10 subscribers) were not successful because Claude had no idea what we were talking about. StackContacts is a “deferred” tool - meaning it didn’t load automatically. Claude could only find it after being told the name, triggering a tool_search to load it. So use a prompt like: “Use tool_search to find any connected database tools, then connect and show me what’s available.”

Step 2: Store your data

As part of the StackContacts setup, you will also set up DuckDB, an open-source SQL database. When Substack data is synced down to your local computer, this is where it will be piped. The database is not proprietary to StackContacts. This is positive because we can’t get locked out of our data even if we uninstall StackContacts. We own it, we manage it, we back it up. And even if you’re a large organization or scaling up, you can operate nicely for quite a while with DuckDB, whether locally or in the cloud.

StackContacts’ data screen gives you quick access to database information and tools, including backup and restore of your database, a summary of records and tables, and a list of all tables from your publication.

Take note of the Database Location (circled) - you’ll need that if you choose to connect to it from Excel.

Want to see your data up close and personal?

Download a tool to browse it. We are using DBeaver - a free, open-source tool for database management. From here you can browse through the database structure, view the actual data, write and run queries and export data.

Step 3: Query your data

One of the things we learned in testing the AI integration was that the data structure and definitions were sufficiently complex that Claude could not do analysis without more significant familiarity on our part with the data to instruct it well. We knew we wanted the underlying data, but those experiments with Claude made it clear that we needed it.

So the next step was to set up standard queries for metrics we wanted to follow regularly. And just like we wanted to avoid writing any API calls, we also wanted to avoid writing the SQL queries - this author is great at defining the logic, but self-admittedly, terrible at syntax. This was a perfect use of Claude.

We started with a set of 6 queries that addressed our key needs: post and notes performance, top content by a variety of measures (engagement, subscriptions, shares), subscription activity over time and by source and a rolling summary of all engagement. (Interested in the queries? Send us a DM). We saved the queries in DuckDB (where they were originally tested), documented them in a text file, and began to build an Excel file that could be refreshed weekly after syncing the data.

(Wondering if you can do this with Google Sheets? There are ways, but they’re clunky - the big constraint is that your DuckDB database is local. If you move it to the cloud the path is easier).

Some tips:

• StackContacts and DBeaver want exclusive access to the database while they’re running. So if you would like to run both at the same time, you will need to have DBeaver work in read-only mode.

• Have one place where you keep your master copy of your SQL queries. You can modify them in DuckDB, in text, and in Excel - and it’s easy to lose track of which is the master. Just choose one, and remember to always update that documentation.

Because we are still refining our queries, we test them in DBeaver, keep a backup in a master text file and copy them into Excel. You’ll see how that works below.

Step 4: Connect the data to Excel

We have long used Excel to manipulate and analyze data. In order to be able to run queries from inside Excel, we had to do a couple of one-time setup tasks that enable the connection:

1. Download and install the DuckDB ODBC driver from duckdb.org (free)

2. Configure the ODBC connection on my computer.

To set up the ODBC connection, click on the Windows key on your keyboard or click the Windows key + S to search. Type in odbc and select ODBC Data Sources (64-bit)

• Click on the SystemDNS, then click Add.

• Choose the DuckDB Driver, then click Finish.

• Type in a Data Source name and the path to your database

Now you’re ready to make the final link between Excel and your data.

3. Connecting to your database from Excel

Open an Excel spreadsheet. From the Data tab, click Get Data | From other sources | From ODBC and select the StackContacts data source.

When you select StackContacts from this list, you can either click on Advanced and paste in a query. Or click OK to navigate the tables, preview and load the data into a worksheet.

Once you establish the connection, there are a number of different ways to explore the data in Excel. Because we had already written our SQL queries, we went to a blank worksheet, connected to the database (following the steps above), pasted in a query and loaded the data in the tab. We did those steps for 6 metrics-focused queries plus one to remind us when the data had been synced.

You can view, edit and refresh all your queries by clicking Queries & Connections from the Data tab.

This queries and connections view is really useful on its own - by right clicking on any item, you have a number of actions you can perform - making Excel a suitable place to explore the data without something like DBeaver as an exploratory tool.

Step 5: Use the data to monitor progress and build a feedback loop

We use this Excel file to focus our discussions when we meet to develop a content calendar, refine our messaging, and decide what we want to test next - whether that’s new topics, formats, or ways of engaging.

While we sync our data weekly, we don’t review every metric on that cadence. Instead, we use the data when it’s most useful. Sometimes that’s weekly, sometimes monthly or higher when patterns take longer to emerge.

What matters is the data feedback loop:

• We look at what we did

• We look at how it performed

• We decide what to adjust

Then we repeat.

Having the data at this level of detail makes that possible. We can go back and isolate the impact of a specific post or outreach effort, compare across time, and test whether changes in what we write or how we share it actually make a difference.

When volumes are small, we aggregate to monthly or even quarterly views to get a clearer signal. Over time, this helps us move from reacting to individual results to recognizing patterns, and using those patterns to guide what we do next.

Step 6: Explore the data using AI

Earlier we described some pros and cons we experienced working with Claude. How you use AI will depend on your objectives, but the real value shows up once you understand your data well enough to ask better questions.

Here are a few ways we’ve used AI to extend beyond hands-on analysis:

Content strategy analysis

AI can help you step back and look at your content more systematically:

• Topic and theme analysis
  Have the AI classify your posts and notes into topics, then compare performance across those themes. Review and refine the categories - s a starting point, not a finished taxonomy.

• Format and channel performance
  Look at how format (short vs. long, structured vs. narrative) and channel (posts vs. notes) influence engagement. Different types of content often perform differently in each format.

• Timing and mix
  Identify periods of stronger performance and examine what you were publishing at the time - topics, formats, and frequency.

The goal here isn’t just to find “what works,” but to understand why it works and where you might want to experiment next.

Audience and subscriber analysis

AI is also useful for understanding how your audience is behaving over time:

• Engagement patterns
  Identify which subscribers are consistently active vs. going quiet, and track how engagement changes over time.

• Growth and re-engagement signals
  Look for moments where activity spikes. Did a particular post or note correlate with new subscribers? Who might need re-engagement?

Perfect attribution is less important than spotting directional signals you can act on.

Tips for getting the best out of an AI for analyzing data

Tell the AI what you already know about your data. As we saw when we experimented with Claude immediately after setting up the StackContacts integration - Claude will make reasonable assumptions, but we need to know the quirks of our own data. Surface those to the AI early and the analysis will immediately get sharper. That can include things like:

• How key fields are defined (e.g., free vs. paid subscribers)

• Which fields or tables to trust or ignore

• How you define engagement or success

• Any transformations you’re applying (e.g., weekly/monthly aggregation)

• The specific question you’re trying to answer

Ask the AI to check the data structure before analyzing it. Asking the AI to look at the columns and a sample of rows first. A prompt like “before you analyze, show me what’s in this table and confirm what each key column means” saves a lot of back-and-forth.

Be specific about what question you’re trying to answer. “Analyze my notes” is not a useful prompt for an AI. Questions such as “Which note topics get the most reactions, and does a short vs. long format change that?” provides a clear target.

Correct the AI when something doesn’t look right. For example, an initial topic analysis of notes was off because the AI was mixing original notes with replies and restacks. Flagging that immediately led to a much more accurate analysis.

We were learning while we were building, and we continue to refine our queries and our analysis. We’ve tried to capture our steps - and our missteps - here. If you have tips and ideas on how to do it differently, or if something isn’t clear, we would love to hear from you. Please comment or send us a message. We reply to them all.

Leave a comment

When a workflow feels clunky, it’s tempting to start looking for new tools - a slick new platform, a better tool, AI, the software equivalent of a fresh start.

But the real fix usually starts somewhere else: understanding what’s actually happening beneath the surface. If you skip that step, you’re likely to treat the symptom, not the cause.

It’s like replacing all your light fixtures when the problem is faulty wiring. Flashy fix, same old flaw.

Before you overhaul anything, pause. Assess what’s actually happening. Understand what you already have and what it really needs.

Step 1: Get clear on the goal

“The process is inefficient!”
“This platform is ancient - it’s time for an upgrade!”

Valid frustrations. But not goals.

A good goal gives you direction. It’s something clear - a specific, actionable outcome - that you can work toward and use as a gauge for whether a change is having an impact.

For example:

“We need to shorten the time between content submission and publishing.”
“We want to eliminate manual data errors.”
“With AI’s capabilities, let’s rethink how we deliver client value on this part of our service.”

Before making any changes, ask: What are we really trying to accomplish?
Speed? Accuracy? Better value? Less frustration?

If no one can give you a clear answer, your job might be to help shape that goal. You can’t fix a process - or choose a tool - if you don’t know what problem you’re solving.

Step 2: Map the people and the process

People are great at describing their pain points. The hard part is figuring out what is actually causing them. That’s where mapping helps.

Why mapping matters:

• It reveals disconnects and redundancies.

• It surfaces silent pain points.

• It gives you clarity - and credibility.

In siloed organizations, one team may offer a detailed view of a single step, but not how that step fits into the broader picture. But without the full context, you can end up solving the wrong problem entirely.

Tips for documenting workflows

If you’re lucky, your teams already have documentation. If not, or if it’s outdated, roll up your sleeves!

Try this:

• Shadow people as they perform common tasks.

• Make it clear you’re observing, not evaluating.

• Ask: “Who does this task? What happens next? Who touches what system, and when?”

• Diagram hand-offs. Make it visual. Review with the people who do the work.

Accuracy matters! These diagrams are not just for clarity. They become the foundation for tool evaluations or change management later on.

(Pro tip: Workflow diagrams are a great way to bring objectivity when operating in a low-trust or “finger-pointing” environment.)

Step 3: Analyze and quantify

Now that you have goals and workflows, dig into what you’ve learned.

Ask:

• Where does time pool up?

• Are there steps we can skip or streamline?

• Where do delays or mistakes happen most?

• Do certain approvals always slow things down?

Look for patterns:

• Bottlenecks

• Points of friction

• Risky or error-prone hand-offs

• Bright spots that could scale or add significant value

Focus on moments in the workflow where a small change could have a big impact. These are your biggest potential opportunities for automation.

Step 4: Pinpoint needs, priorities, and risks

With real data in hand, start identifying:

• What needs to change?

• What’s a process issue vs. a tool issue?

• What are the benefits and risks of each path?

With this information, you can evaluate your real options.

Example:
A research team pushed for a new data platform to fix persistent errors. But when they mapped the workflow, the real issue turned out to be inconsistent formatting in client-submitted spreadsheets. Research analysts and data operations were using different templates.

The solution? Align the templates. The errors disappeared. No new system required.

Takeaway: Sometimes the fix is smaller, cheaper, and right under your nose.

The real cost-benefit calculation

Automation projects require money, time, and people. But most teams only ask one question:

“Is the benefit worth the cost to build?”

That’s too simple.

When you evaluate automation, you need the full picture:

a) Build cost (what everyone sees)

• Development time

• Tools / platforms

• Internal or external resources

b) Implementation cost (what gets missed)

• Training your team or clients

• Rolling out new workflows

• Supporting adoption across the organization

c) Opportunity cost (what no one talks about)

• What are your people not doing while this is happening?

  For example, a sales team learning a new forecasting automation tools isn’t meeting prospects. That’s pipeline you’re trading for process.

Don’t get us wrong - automation can absolutely drive value. But only when you weigh it against everything it displaces.

Too often, teams underestimate effort, overestimate speed to value, and ignore the hidden costs of change.

A better question to ask.

Instead of:

“Should we automate this?”

Ask:

“Is this the highest-value investment, right now?”

Step 5: Get ready to recommend and present

By now, you have done something powerful: not only have you identified a problem, but you have built shared understanding around it.

When you present:

• Don’t go for a dramatic reveal. Show how you arrived at your recommendations.

• Share your findings to test understanding.

• Walk through assumptions and risks.

• Invite feedback. Clarify tradeoffs. Be honest about what is still uncertain.

Frame your presentation as a shared discovery, not a pitch.

What makes you invaluable

You don’t need to be an AI or systems expert to lead this kind of change. Curiosity, persistence, and thoughtful observation go a long way.

The hard part is taking the time to understand what’s really happening. It’s the step many teams skip, yet makes the difference between a failed automation effort and a lasting one.

This kind of attention doesn’t just improve processes. It builds lasting value, clarity, and alignment. And that’s what makes your work truly invaluable.

Leave a comment

More Reading: The “AI at Work” Series

• The Human Factor

• Frame the Problem

• Why Data Inputs Matter

• When to Trust, Adapt, or Toss AI Outputs

• Translating AI Results into Decision and Action

• Using AI in a Lean Marketing Machine

There are moments in almost every career where you realize you’re a little out of your league. You’re expected to have an answer, but you’re not entirely sure what’s actually going on.

Maybe it’s your first role and you want to contribute something meaningful. Or you’re stepping into a new team or business and trying to get your bearings. Or you’re running your own operation and every decision feels high stakes.

Data should help in these moments, providing clarity and direction.

But for many people, it does the opposite. The dashboards look polished and the reports look definitive. Yet there’s a worry: What am I looking at? What does this number represent? What if I’m wrong?

So we rely on analysts. We lean on tools. We increasingly turn to AI.

All of these help, but they don’t solve the core problem. If you’ve only seen the final numbers - and not what they’re built from - it’s hard to feel confident acting on them.

What we’ve found is simple: confidence doesn’t come from better tools or delegating. It comes from spending a little time with the data itself looking at real rows, real customers, real events.

In other words, you have to touch the data.

This feels counterintuitive. With all the technology available, it seems like we should be able to skip this step. But you can’t outsource that familiarity. A polished dashboard or AI output can look convincing, but if you haven’t seen what it’s built on, it’s hard to fully trust or interpret it.

We wanted to understand why this step matters so much. So we looked at how people actually learn. It turns out there are three well-established principles that explain what’s happening.

Share

Why touching the data works: three learning principles

1. People learn through concrete examples, not abstractions

Data is abstract. Summaries and dashboards make it even more so. But learning science shows that people understand concepts better when they start with concrete examples and then move to abstraction - not the other way around.

A marketing professor we know teaches a well-known behavioral economics concept about how people overvalue what they own. For years she explained it through a lecture. Students were skeptical — it just didn’t ring true. So she started running a simple experiment: she gives half the class a Kit Kat, asks them how much they’d sell it for, and asks the other half how much they’d pay. Almost every time, the Kit Kat owners demand more than buyers are willing to pay. The concept sticks because students experience it firsthand.

We are wired to learn this way. We remember what we can see, experience, and interact with.

Looking at the data itself works the same way. When you’re looking at actual customers, transactions, and events, you see the building blocks before it gets rolled up into a summary. Once you’ve seen that, the dashboards start making a lot more sense.

2. Better judgment starts with understanding what’s underneath

Research on expertise shows that people who understand how something is constructed make far better judgments than those who only see the final output. Experts organize knowledge around core concepts and relationships; novices rely on what’s visible on the surface. It’s like the physics student who memorizes formulas versus the one who understands the underlying forces. One will have context that sharpens his or her answer significantly.

The same applies to data. Dashboards summarize. AI spots patterns. Neither tells you about the data below it and that gap matters more than most people realize.

Take a simple example: “Average revenue per customer is $40.” That number could reflect two very different scenarios:

• 1,000 customers all paying between $35–$45 (stable, predictable)

• 1,000 customers ranging from $4–$75 (highly variable)

The dashboard shows the same number either way. Seeing the data itself gives a foundation to better understand what that information means and what it does not.

3. Learning and “transfer” (a.k.a. apply what you’ve learned elsewhere)

Research shows that people apply what they learn to new situations more effectively when they understand the underlying principles and structure, not just the outputs.

This means that when you’ve spent time with the data itself, you understand how it is organized, detailed, and start to recognize familiar patterns. You can then apply this when similar data shows up in different reports and contexts.

Instead of starting from scratch each time, you can get oriented faster because you recognize the building blocks. You have seen before how this kind of data works.

Share

What about analysts and AI?

Analysts, dashboards, and AI tools are genuinely valuable. We use them. But they can’t replace your own grounding in the data.

Analysts bring expertise, but not your day-to-day context. They don’t know what feels off in the business right now. And their analysis is always shaped by their own experience and framing.

AI has a different limitation: it can produce excellent analysis, and it can also miss context or make incorrect assumptions. Sometimes it’s hard to tell which is happening.

If you’re not confident with the data yourself, you’re exposed to both of these limitations. You’re relying on someone else’s interpretation without a reliable way to evaluate it.

Even a small amount of time with the underlying data changes that. It gives you enough familiarity to work with analysts and AI tools more effectively, rather than simply deferring to them.

Why this matters

In a world full of dashboards, reports, and AI-generated answers, it’s easy to skip the step of actually looking at the data. Instead choose to actually understand the data. That small step in looking at real rows, real customers, real activity is what builds skill and confidence.

When you’ve seen the data, you build your understanding to judge when an analysis is overreaching, when something looks off, or when a result doesn’t quite line up with reality.

The three learning principles above explain something we’ve observed repeatedly: people who spend even a little time with the data develop a fundamentally different relationship with it. They are not intimidated or afraid. They are more confident and end up using the data well.

Getting started: building confidence with data

Here’s a practical sequence to work through with your team.

1. Choose one report tied to a real question you care about like revenue, leads, customers, usage, donations, support tickets.

2. Export the data. Look for “Export” or “Download.” A CSV file type is usually the easiest place to start.

3. Open it in Excel or Google Sheets. If it opens as a CSV, save a working copy as an Excel file.

4. Set a 15- or 30-minute timer and get curious. Keep a notepad nearby for observations and questions. The goal is familiarity, not a full analysis.

   • Start by scanning the columns. What does each field appear to represent? What looks complete or incomplete? Are there blanks, duplicates, or labels that seem inconsistent?

   • Sort one column at a time - by highest and lowest values, alphabetically, by date. This is the fastest way to spot ranges, outliers, and patterns that disappear in a summary.

   • Filter the data to ask simple questions. Look at one segment at a time: one month, one region, one product type. Move from “What is this report saying?” to “What is actually happening here?”

   • Do one basic summary. Add up a column, calculate an average, count rows in a category. Even simple calculations connect the row-level detail back to the bigger picture.

   • Build one pivot table. If you haven’t used one before, a quick YouTube tutorial will get you there in minutes. Put one category in the rows, one metric in the values, and see what shifts — product line by revenue, region by customers, month by support tickets.

5. Review your notes. What did you notice? What surprised you? What would you now ask an analyst, colleague, or AI tool?

Leave a comment

Share

Subscribe now

Some references

1. Johnson-Laird, P. N. (1983). Mental models: Towards a cognitive science of language, inference, and consciousness. Harvard University Press.

2. Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5(2), 121–152.

3. Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (2000). How people learn: Brain, mind, experience, and school (Expanded ed.). National Academies Press. https://doi.org/10.17226/59.

More Reading

Most people think of data problems as typos, duplicates, and broken formats. You’ve probably run into it yourself: you open a dataset expecting to run an analysis, and quickly realize you can’t sort it, filter it, or trust the totals.

But not all issues show up that clearly. Ask five people in your organization what “active customer” means, and you’ll get five answers. Maybe six. And every one of them will be right based on the data they’re looking at.

Someone will say it’s anyone who made a purchase in the last two years. Someone else will say it’s anyone who logged in this month. Marketing will count qualified leads. Finance will only count currently paying subscribers. And the person who built the dashboard three years ago? They’re not quite sure anymore, because the logic changed and it was never documented.

Now you have two different problems: data that doesn’t work when you try to use it, and data that seems clear but means different things to different people.

Both show up all the time, and both can quietly derail your analysis. And as more teams rely on AI tools to summarize, transform, and act on data, these issues don’t go away - they get harder to spot and easier to scale.

So what makes data “clean”?

Clean data is data that is consistently understood and reliably usable when it matters.

It doesn’t have to be perfect or exhaustive. It does have to be trustworthy for the job at hand and interpreted the same way by the people using it.

Two things tend to go wrong:

• Meaning problems: what does this actually represent?

• Quality problems: is this usable and consistent?

Both matter. Most teams only focus on one.

The data problems you can actually see

These are the kinds of “messy” data you can spot right away - what most people think of when they hear “dirty data.” You open a customer list expecting to run a quick analysis. Instead, you get this:

At a glance, it looks fine. But look closer:

• Date formats don’t match

• Status values don’t match

• Revenue is stored inconsistently and is undefined

• One record has no customer ID

• One customer appears twice

Teams focus on this because it’s visible, and because it will actively break your analysis if you don’t fix it first.You can’t reliably:

• Sort or group by date: 03/04/2024, 2024-04-03, and April 3rd won’t sort or compare without standardizing first.

• Filter categories: Active, active, and ACTIVE look the same to you. They aren’t the same to a database.

• Sum revenue: mix currency symbols with plain numbers and you’ll get errors or wrong totals.

• Join to your customer database: a missing ID isn’t just blank. It’s a record you can’t link.

• Trust your totals: Customer 1042 appears twice with different data. Which is right?

Even simple questions like “how many active customers do we have?” can give you the wrong answer because the data doesn’t line up.

This kind of mess is frustrating, but it’s also fixable. Most of it comes down to standardizing formats and making sure fields are structured consistently.

The usual suspects: messy data in the wild

Once you know what to look for, these patterns show up everywhere:.

Even if you clean all of this up - standardize formats, fix duplicates, fill in missing fields - you can still end up with the wrong answer if a more basic question hasn’t been answered: what does “active customer” actually mean?

When the same data means different things

This is easier to miss: the data looks clear, but people are using it differently.

We dug into this last week, but you’ve seen it:

• Revenue → gross or net? recognized or booked?

• New customer → signed up or paid?

• Traffic → sessions, visitors, or page views?

• Active user → depends who you ask

None of these are unreasonable. They’re just not defined consistently. So two people look at the same report, both are “right,” and still walk away with different answers.

Quick rule of thumb: if you don’t know the definition, you don’t know what you’re looking at.

Share

Clean vs. messy: five questions to ask yourself

You don’t need to audit every field in every report to catch most data quality issues. But you do need to slow down and ask these questions before acting on the data.

1. Do you know what this field is measuring - and would your colleagues agree? If you had to explain this metric out loud, could you? If the answer is “sort of,” that’s your sign to pause and clarify before you go further.

2. Are there values that look the same but are formatted differently? Scan the unique values in a column. Variations in capitalization, spacing, abbreviations, or symbols can affect filtering and totals. If they should be the same, standardize them.

3. Does the total hold up if you check it a different way? If your report says you have 500 active customers, try counting a different way. Pull raw data, check another system, or ask someone else to run it. If the answers don’t match, take a closer look.

4. Do you know where the data came from and how fresh it is? “I got it from the dashboard” is not the full picture. What is the source? How often is it updated? Who owns it?

5. Are you clear who is included in a category or type? If you filter by “active customers,” could you explain exactly who qualifies? If not, the data isn’t ready to drive a decision.

Subscribe now

Clean data in practice

Clean data isn’t about getting everything perfect. It comes down to two things: shared meaning and data you can actually use. Everyone understands the data in the same way, and when you use it, it supports the analysis or decision you’re trying to make.

Formatting and consistency issues make your data hard or impossible to analyze. You can’t sort it, filter it, join it, or trust the totals. Definition issues are quieter, but just as important. They lead to inconsistent assumptions, miscommunication across teams, and decisions that look right but aren’t grounded in the same reality.

The good news is that both are fixable.

On the technical side, establish standards (consistent date, currency, and number formats), clean up duplicates, and structure fields so data is entered reliably (dropdowns, required fields, validation rules). You can also ensure each field contains a single piece of information, choose a clear system of record, and standardize data before it’s used in reporting.

On the definition side, get clear on what key metrics actually mean and document those choices so they hold over time. In practice, that means defining metrics in plain language, pressure-testing them across teams, and making sure everyone is using the same definition when decisions are being made.

Your data doesn’t have to be perfect, and you don’t have to solve everything at once. But it does require a habit: taking the time to make sure you and your team agree on what you’re looking at - and that the data will support the analysis or decision you’re about to make. As AI becomes part of more workflows, these issues don’t just affect reports - they shape the outputs you rely on.

Leave a comment

Related articles from We Dig Data:

We see numbers all the time at work. We track progress, allocate resources, consume research, and pitch ideas. But how well do we actually understand what we’re looking at?

If you’re making decisions or supporting the people who do, you are accountable for the number and for understanding what is behind it.

Today we’re talking about how to critically shine a light on data you use to make decisions at work. We’ll cover:

• Why to revisit what you think you know about your data

• How to determine data’s strengths and limits by looking at its source

• Uncovering hidden assumptions

• Balancing the decision against the risk, ambiguity, and bias in the data

This isn’t about becoming a data analyst. It’s about strengthening the skills to size up data and then use it to confidently choose a path forward.

Share

Get to know your data

Most misunderstandings begin here. We too often believe that we are clear on familiar metrics…until someone asks us to explain them. That’s when we realize the gaps.

When we don’t fully understand what the data represents, we draw incorrect conclusions. Everyday sources of confusion include:

Familiar terms that are used loosely. Customers and users. Sales and revenue. These sound interchangeable, but aren’t, and the data can show up differently. For example, let’s say you sell a $1200 annual subscription. Sales will show up the month the sale closes, but the business recognizes revenue evenly over the duration of the subscription.

One label, many definitions. Metrics like “traffic” sound intuitive, but have many meanings. “Traffic” can mean page views, sessions, visitors, or unique visitors.

Missing context. A percentage or rate change sounds clear until you ask: from what to what? Statements like: “Lead conversion rates have grown 10%!” have little meaning until clarified into: “Last year, the internal sales team converted 10 out of every 100 website leads into a sale. This year, they are converting 11 out of every 100.”

Vague or inconsistent classifications. A report documents “adverse reactions” for a new drug. But what does that mean? A mild rash or hospitalization? Definitions and classifications vary by industry or even by department in the same organization.

Even if a term seems obvious, confirm that what you think the data represents is accurate. Clarify and define, don’t assume.

The source shapes your interpretation

Where the data came from reveals its strengths and its limits. This context will determine how much trust you put into a number’s accuracy and reliability.

For EXTERNAL data sources, ask:

1. Who published the data? Established, credible sources - like census data or a reputable research firm - are generally more reliable, especially when the organization specializes in data and has a verifiable track record. In contrast, vendors or lobbyists may provide research, but they also have incentives that could bias how results are framed or interpreted.

2. Where do they get their data? How did they collect it? Does the data adequately represent the population being measured? What is the sample size? Strong data sources explain how they collected their data and acknowledge limitations.

3. Is the methodology available? How information is collected, defined, and calculated determines whether it is actually relevant to your situation. Credible sources will usually explain how they collect and analyze their data and what limitations apply. If someone dodges your questions, be cautious.

For INTERNAL data, ask:

1. What is the data source? Some systems cover only part of the business when you are trying to understand the whole. Others may have known quality issues when you need greater accuracy.

2. Is it a recurring report or a one-off request? Ad-hoc reports require more scrutiny. Regular reports have usually been stress-tested and refined over time.

3. Who built the report? Different departments bring different lenses. Finance may be more conservative than Sales. Marketing may define ‘customers’ differently than Operations. Each team has different data access, expertise, and perspectives. These factors influence what gets measured and how it gets presented.

No data is perfect, but solid data can withstand scrutiny.

Uncover assumptions behind the data

Every number is built on choices. Some choices are clear (like a time frame), but many are invisible unless you ask. Let’s start with a few that cause the most trouble:

The sample population. Don’t assume a data point represents the full picture. If a survey on fried chicken sandwiches is conducted with 1,000 college students, it doesn’t represent national preferences - it represents college student preferences. So always ask: Who’s actually included in this data? And who’s missing?

Projections and estimates. These are built on assumptions, and you may not agree with the approach nor the level of risk represented in those choices. For example, a revenue forecast will include assumptions like: average sales price, expected growth or decline, revenue from new clients vs. existing clients or new products vs. existing products. Don’t hesitate to have these discussions. They are usually an invaluable step for refining estimates and projections.

Check the math. You don’t need to audit everything, but a quick spot check often catches errors before you get too far. A 2025 Canva survey reported that 57% of respondents said they make spreadsheet errors that impact their work. With numbers like that, it’s not worth just assuming the math is right.

Data skeptics remember to probe on these areas because they’ve learned (often the hard way) how easy it is to act on a false assumption.

“Good enough” is a real answer

You won’t always have perfect information. The question is whether it’s good enough for the decision you are making.

Match the scrutiny to the stakes. Low-stakes decisions, like an A/B test tweak, may not require bulletproof data. High-impact decisions, like a large investment or launching in a new country, warrant more scrutiny.

Watch for false precision. Specific numbers can signal more certainty than actually exists. For example: “40.2% of the population will go on vacation this summer, up from 39.8% last summer.” But if these are based on two consumer surveys of 100 people each, that extra decimal is indicating a level of certainty that does not exist.

Treat surprises as signals. If something looks off - up, down, or just unexpected - pause. Surprises are your cue to investigate. It might be a real shift. It might be a data issue. Either way, it’s worth understanding.

Ranges can be good enough too. If you have a small client base, it could be good enough to use a small range when analyzing annual client survey data. “Approximately 3%-5% of clients plan to increase their marketing budgets, which is about the same as last year.” Being directionally right is sometimes enough, but you need to know that going in.

Spot the bias

Information is produced and interpreted by people. Everyone has a lens, and sometimes an agenda. This doesn’t make their data wrong, but it does mean you should consider that bias in how you apply that information.

Motivation matters. Incentives shape how data is framed. That doesn’t make it wrong, but it does affect how it’s presented. A pharmaceutical company wants positive results for a new drug or product. A salesperson’s forecasts are influenced by how they are compensated. From students to policymakers, people interpret data through their lens. Know the agenda, and factor it in.

Over-generalizing. When data is missing or hard to find, it’s tempting to apply one piece of information to perceived similar situations. But what is true for one group, market, or channel may not apply more broadly. Instagram users do not necessarily represent all social media users. New York City restaurant trends don’t predict what’s hot in Texas, and the top-selling toys in the U.S. won’t match those in France.

History doesn’t always predict the future. Past patterns can break. A competitor has a breakthrough moment. There’s a technology shift, a viral moment, or a market trend that quietly hits a tipping point. ‘It’s always been this way” is an assumption, not a fact.

Why data skepticism matters

Data is never just numbers. It’s a set of choices about what to measure, how to frame it, and who it represents. Your job isn’t to audit every figure. It’s to stay curious enough to ask the right questions, and confident enough to push back when something doesn’t add up.

Leave a comment

Share

Subscribe now

Additional Reading

Here are two excellent resources from fellow Substack writers if you want to go further on assessing external resources:

• , author of Card Catalog, recently wrote a guide to evaluating information sources in the AI age.

• , author of Slow AI, wrote about how to spot a fabricated source and created a game, Dead Reference, to test your skills.

More related articles from :

You are in a meeting. A dashboard is shared, and a report appears on screen. A number jumps out - maybe it’s higher than expected, or maybe lower. Way lower.

Someone says what everyone else is already thinking: “Those numbers can’t be right.”

And sometimes they aren’t. But just as often, the number itself isn’t the problem. What’s actually being challenged is our expectation of what the number should be.

These moments are signals to slow down and take a closer look.

Despite the internal groan, great data conversations happen when the numbers don’t match the story in our heads.

In this post, we will walk through an example of how one product team handles this situation as well as the process that great teams use for investigating data that doesn’t match expectations.

Subscribe now

When the numbers don’t make sense.

Imagine a product team that has recently introduced a new feature inside their main product that allows users to generate a report, a task previously requiring cumbersome manual work.

Customers have been clamoring for it. The internal team is excited about it. They’ve highlighted the new reporting feature in release notes, mentioned it in onboarding messages, and demoed it internally. Everyone assumes clients will jump in right away.

Two weeks later, after a robust launch campaign, the internal team pulls up a usage dashboard. The usage number is… surprisingly low. Only a tiny percentage of users appear to have engaged with the new reporting tool at all.

Around the table, people start doing the quick mental math. Something about the number doesn’t feel right. What’s going on?

There are a few possibilities. And in practice, it’s often a mix of them.

There is a problem with the data.

Sometimes the issue really is the data.

Our product team’s usage metric depends on analytics signals that fire when someone uses the reporting tool. After digging into the setup, the team discovers those signals were only triggered when users opened the tool from one specific place in the product.

But many people were reaching it through other paths like shortcuts, saved links, or existing workflows. Those sessions simply weren’t being counted.

There are many alternatives that may cause problems with your data, even it is an established report. Data pipelines break. Dashboards miscalculate. Definitions drift over time.

The metric isn’t measuring what you think.

In another scenario, the data itself is fine, but the metric isn’t measuring what the team assumed.

The team thought the metric would count people who opened and used the tool. Instead, it counted only people who successfully finished creating a report.

All those users who explored for a moment or found the data to answer a quick question, and then left? Ignored. The dashboard reflected only users with reports completed, not usage.

Once the definition was clarified, the problem to investigate was no longer, “why isn’t anyone using the tool?” but “why are people starting the process, but not finishing it?”

The underlying assumption is wrong.

Sometimes the number is accurate. Disappointing? Sure. But wrong? No.

When reviewing the data more closely, our product team notices something interesting: the reporting tool is being used only by a few users, but they have logged in multiple times. They identify that only the power users who regularly analyze large datasets are the ones logging in. And most customers only need that kind of reporting occasionally.

The new feature isn’t failing. It’s just for a more specialized audience than the team originally expected.

That realization changes how the team thinks about improving the tool and how they introduce it to customers going forward.

This is often the most difficult situation to accept. In this case, the feature wasn’t as applicable as they’d assumed, so hopefully they learned how to pressure test customer feedback better.

Share

These moments show up everywhere

Mismatches between data and expectations appear across all types of organizations:

• A customer service team runs their quarterly client satisfaction survey and sees scores plummet unexpectedly.

• A small business owner notices a shift in sales patterns that doesn’t match their day-to-day experience.

• A library checks its children’s program dashboard and finds attendance numbers that feel surprisingly low.

In each case, the number doesn’t quite line up with what people thought was happening.

And the best teams investigate…doggedly

Sometimes the explanation is technical like a data pipeline problem or a change in how the metric is calculated. Other times the number is pointing to something real that simply hasn’t been noticed yet.

Strong teams resist the urge to immediately defend or dismiss these situations. Instead, they treat the moment as a serious investigation. When pursuing an explanation, they typically follow this path:

1. Check the source
Is the data pulling from the right system? Are records missing, duplicated, or only partially captured? Has the calculation changed? Has the report been refreshed?

This requires investigative effort for you and your technical partners. If you aren’t familiar with how that report is created, now is a good time to start learning about the data pipeline.

The good news? In this situation, once the issue is corrected, the numbers change significantly and tend to return to “normal.”

2. Confirm the definition
What exactly does this metric measure? Is it counting the thing people assume it is?

Data’s intangible nature means that we often are not speaking the same language. In this case, a picture is indeed worth a thousand words. If you suspect your metric is not measuring what you think it is measuring, meet the report builder and sketch or show the behavior that you think you are measuring.

Even if you both are using the same words, don’t assume that their definition of ‘customer’, ‘sales’, ‘conversion’, or ‘traffic’ is your definition.

3. Revisit assumptions
If the number holds up so far, then it’s time to re-assess your assumptions. What else might explain the results? Are we expecting something that isn’t actually happening?

This is when you find your assumptions need adjusting. Sometimes, you even discover that a fundamental belief or ‘truth’ about your business or industry simply is not accurate.

These perspective-altering pieces of information are incredibly valuable, but can be difficult for you, or your organization, to accept. If this is your situation, triple check the data and look for explanations that might help you interpret what you are seeing.

Subscribe now

Why this matters

Handled this way, the moment of skepticism becomes productive. It helps the team surface the real issue and focus on the problem that actually needs solving. Teams clarify what their metrics actually mean, surface hidden data quality issues, and notice real patterns that might otherwise go unnoticed.

This effort strengthens the team’s relationship with their data and with this familiarity, they also gain more confidence in the numbers they rely on. Just as important, the culture around data shifts. Instead of defending assumptions or dismissing surprising results, teams get more comfortable investigating what’s really happening.

So when someone looks at a report and says that number can’t be right, they might be correct. But even when they aren’t, the reaction is still valuable.

It’s an invitation to pause and ask a few better questions: Where did this number come from? What does it actually measure? And do our assumptions still hold?

Have you ever had a moment where the numbers didn’t match what you expected?

Leave a comment

Related articles

If you’re an entrepreneur, product manager, or basically anyone who builds stuff, you’ve probably felt that tension that surfaces when engineering pushes back on your desired launch timeline. Or the data team challenges the data sources behind your projection. Or IT wants to review security and integration before approving the new tool that is going to save you a ton of time and effort.

It can feel like resistance. But it might actually be a sign of a healthy system.

Technical teams optimize for different priorities, and those differences can be genuinely valuable to the organizational mission. While you might be optimizing for speed to market, customer impact, or feature delivery, your technical partners may be focused on system reliability, scalability, data quality, and long-term maintainability.

While your priorities and their priorities often don’t naturally align, both are necessary. And when you harness that tension, it produces better decisions. You’ll even find that great products or results often emerge from the space between those two sets of perspectives.

Vision vs. Feasibility
Someone pushes the idea forward. Someone challenges how it will work.

Speed vs. Durability
Someone wants to launch now. Someone is thinking about after launch and the ongoing user experience.

Business Outcomes vs. System Health
Someone focuses on growth metrics. Someone fortifies the system that growth depends on.

Healthy organizations don’t eliminate this tension. They encourage learning how to leverage it to their benefit.

When perspectives collide and you work through those constraints, stronger solutions emerge. When you don’t, you often end up with brittle or broken systems, unrealistic roadmaps, or solutions that look good, but arrive too late.

The goal of working well with your engineering, data and IT/ops partners isn’t to eliminate friction. The goal is to make that friction productive, to navigate the different, seemingly competing, priorities and to treat pushback as an opportunity rather than as a roadblock.

The best partnerships with these roles rarely feel effortless. More often, they feel like two smart people pulling the same problem from different directions until the right answer emerges. That creative tension, when managed well, can develop into one of the most valuable, and rewarding, working relationships you have.

Why It Matters

This post may sound a bit like a love song to partnerships between business and technical teams. In some ways, it is. Over the years, we’ve watched difficult working relationships evolve into genuine partnerships, and we’ve coached dozens of product, business, and technical professionals through that transition. When those partnerships take hold, something remarkable happens: products move faster, solutions improve, and problems get solved at a level that wouldn’t have been possible otherwise.

So, if you are looking to improve your relationship with a technical partner, here are three mindset shifts that can accelerate the journey.

Lean teams don’t measure everything, and they don’t wait until they’re bigger to start. They decide what matters most right now and measure that.

Growth rarely moves in a straight line. It happens through experiments, plateaus, and jumps. Data provides the feedback loop and without it, you’re guessing.

And yet, many teams avoid the data they already have. They don’t ignore data because they don’t value it. They ignore it because it’s overwhelming.

They have too much data, and that information is spread across different systems. The definitions are often unclear, and measurements may not tie together. Plus when numbers are small, it is hard to know what’s meaningful.

How can you leverage data to accelerate your progress? And how can you integrate this into your operations without feeling like you are just adding one more thing to your “to-do”?

Going from overwhelm to focus

Instead of tracking everything (or the wrong things), you intentionally narrow the field. You choose a very targeted subset of your data.

Which data? The data that measures your current top priority(-ies).

Priority —> Drivers —> Data

What’s your priority?

What are the drivers?

What data do you need to measure progress?

1 | Start with one priority.

• “Grow referral business.”

• “Improve checkout rates.”

• “Build an engaged email list.”

• “Increase net subscription enrollment.”

• “Increase on-time payments.”

2 | Next, identify the drivers: the actions that will, or could, create movement and momentum in that priority.

If your goal is “improving checkout rates” on your online shop, drivers might include:

• “Increasing traffic to product pages”

• “Improving ‘add to cart’ conversions”

• “Sending cart reminder emails”

• “Testing time-bound promotions”

3 | Narrow the list to 1-3 drivers, and for each one, choose at least one clear way to measure progress.

Start with drivers you are already focused on or plan to start working on in the near-term.

For example: “Increase traffic to key parts of our website as measured by “weekly product page views.”

Now your data conversation has focus.

Example: How We Are Doing This

One of our priorities is “building an engaged, relevant audience” not simply increasing raw subscriber counts.

Once we clarified that, we asked ourselves which actions would actually attract that audience. Knowing we couldn’t do everything at once, we then narrowed our focus to just a few of those drivers:

• Grow our email list

• Increase meaningful engagement with posts

• Expand awareness on LinkedIn and Substack

For each of those drivers, we chose simple measurements we could review each week:

• Monthly net email subscriber growth

• Engagement per post (comments, restacks, shares)

• Follower and subscriber trends across platforms

We didn’t decide to track everything Substack or LinkedIn offered. We chose a handful of signals tied directly to what we are trying to create. Reviewing those consistently helps us decide where to invest our time - and where not to.

Use data to go further, faster

Using data to fuel your progress requires focus. And that is what you’ve done here. So far you’ve identified a few metrics that measure the drivers for a top priority. Integrating data into your regular operations is simple in comparison, but it does require some set up and consistency.

What gets measured gets done. BUT those measurements need to be kept visible. To implement a data feedback loop:

1. Assign an owner for each metric to pull and review it.

2. Schedule regular review time on a weekly or monthly basis.

3. Discuss what changed and why, allowing time to problem solve, brainstorm, and hypothesize.

4. Test and experiment - try something new.

5. Review next time. What worked, what didn’t? Why or why not?

Make this a habit and you will not only see accelerated progress against your priority, you will also begin to expand this method to other priorities and data.

Want help applying these concepts? We Dig Data is bringing our workshops to a group format. These workshops help individuals and lean teams use data better to accelerate their impact. Learn more here or send us a note.

What if your numbers are still very small?

When you are starting out or have a specialized mission, data volume is often low making trends are harder to read.

Two approaches to try:

• Aggregate the data over longer time periods (example: monthly instead of weekly)

• Group similar activities together (emails categories, not individual campaigns)

They may not be statistically perfect, but you are looking for hypotheses and signals you can test.

Even when numbers are small, record them consistently. Add notes on activities that might have impacted that metric. Over time, you will see patterns emerge.

Summary

You don’t need perfect data. You need aligned data. When your metrics connect directly to a clear priority, the noise fades and the next step becomes obvious. Start small. Choose what matters now. Measure the drivers that move it. Review consistently. Over time, the habit of focused measurement won’t just help you grow faster, it will help you grow on purpose.

Tell us how you are measuring your key priorities or the roadblocks you run into!

Leave a comment

Share

Subscribe now

We work with a lot of people in small businesses or lean teams that are part of larger businesses or non-profit organizations. They are smart, passionate and resourceful. They’re juggling a lot as they pursue meaningful objectives.

Data? It’s rarely at the top of the list.

We hear:

“We are still too small. The data isn’t useful yet.”

“I have bigger priorities right now.”

“Reviewing my data is just one more thing on my list of urgent tasks. And I’d have to figure out the tools. And I don’t even like data!”

We get it. We’re also a startup. We’re less than a year old. We’re strapped for time. We’re small in numbers. Really small. We’ve had a handful of client engagements and fewer than 1,000 subscribers across Substack and LinkedIn, our primary platforms.

Given that, you might think that this is an absurd time to be thinking deeply about analytics. And yet - here we are.

Here is why, despite being small, we review our (very minimal) data regularly - and why lean teams should too.

Reason #1: A Few Select Metrics Create Focus When Resources Are Tight

When there’s too much to do and not enough time, you have to focus somewhere. If you don’t, you end up spread too thinly. And when you’re building toward something larger, the signals you pay attention to matter - even if the signals are still small.

For us, those signals currently live on Substack.

Substack is not our product. It’s a platform where we’re cultivating community ❤️, testing ideas, earning trust, and attracting the kind of curious, builder-oriented audience that might eventually want to go deeper with courses and workshops.

Because of that, we intentionally invest time there to become a valuable part of the community.

Reason #2: Growth Isn’t Linear. Data Gives You a Feedback Loop.

Building anything involves experimentation. Data gives you something to react to.

It helps you adjust more quickly, even when the numbers are modest. No matter how much we might love a feature or a “brilliant” idea, it may not resonate, or it may need refining.

Over the past few months in Substack, we’ve:

• Experimented with format, features, and cadence.

• Written more Notes.

• Tried Recommendations.

• Shared workshop concepts to gauge interest.

• Promoted differently on LinkedIn.

Each week, we document a small set of Substack metrics, review Google Analytics and track LinkedIn engagement. The numbers are modest, but we’re starting to see small pockets of traction. We’re learning what’s resonating and with whom. That clarity guides our efforts and shapes where we invest our limited time. Without that feedback loop, we’d be guessing.

Reason #3: Data Helps You See Progress When It Feels Slow

There are stretches in any builder’s journey where you hit setbacks, or momentum feels invisible. It’s easy to compare yourself to another lean team that seems to be skyrocketing. It’s easy to question whether the effort is worth it.

This is where tracking a small set of meaningful metrics over time becomes grounding. When you look back at your numbers - subscriber growth, engagement patterns, steady increases - you often see forward movement that didn’t feel obvious at the time.

Data helps you see how far you’ve actually come.

How to Get the Biggest Return on Your Data (With Minimal Effort)

Start with priorities, not dashboards. Not all movement is progress- start by deciding what kind of movement you’re actually trying to create.

For us, the priority right now is building a relevant audience, not just a larger one. We want to know whether our ideas resonate meaningfully. And we want to steadily grow an email list we can responsibly market to when we bring our client-specific methods to a broader audience through courses and workshops.

Once priorities are clear, the data conversation begins to gain focus.

We typically see lean teams will engage with data in two ways: analysis projects and indicator metrics.

• Analysis projects are deeper efforts to understand patterns of behavior. Mapping how someone moves through a website. Establishing a baseline before changing strategy. Answering a strategic question that can’t be resolved with a single dashboard metric.

• Indicator metrics are the signals that our platforms generate to tell us what is happening. Traffic trending up. More people commenting. Subscriber adds per post increasing.

At We Dig Data, we’re focused on indicator metrics right now.

Analysis matters - especially baselines - but our current priority is directional clarity: Are the experiments we’re running creating movement in the areas that matter?

So we track a small number of Substack metrics tied directly to our priorities. That doesn’t mean other metrics aren’t important. It means they aren’t the ones guiding our weekly decisions.

Next Up

Next week, we’ll share more specifics about challenges you may encounter and how to navigate them. We’ll also walk through the exact metrics we track and why we’re intentionally ignoring the rest.

In the meantime, if you’re looking for step-by-step guidance on setting up Google Analytics in Substack or understanding the Substack dashboard, several strong walkthroughs already exist. Two good starting points:

• breakdown of Substack analytics

• Google Analytics setup guides by

What’s one metric you pay attention to right now - and why? Leave a comment. We’d love to compare notes.

Leave a comment

You are responsible for the data you use to make decisions. But there’s more data, more claims, and more “insights” than any reasonable person can thoroughly vet. A key skill today is deciding what deserves scrutiny and how much. But if you burn out fact-checking the data source for every stat that crosses your screen, you’ll end up trusting nothing - or worse, trusting everything because you’re too tired to care anymore.

So how do you evaluate a source in a world of information overload?

Ignore Unsourced Claims

If someone makes a data claim without citing where it came from, skip it. If the author didn’t take the time to source their data, then it’s not worth your mental energy to consider it as proof. Following this rule alone will filter about 80%* of what comes your way.

(*See what we did there? That 80% isn’t sourced because we made it up. The right move was to question it or ignore it!)

Evaluating Data, and an Example

With so much data, it is relatively easy to find a piece of data that supports a point you want to make. And someone making a claim using data is usually trying to convince you of something. That something may be true, but as with anyone trying to influence you or inform you, it’s worth asking:

Where did the data come from?

Let’s walk through a real example from 2025. This is a Canva press release citing data about data usage (yes, it’s meta, but it’s a good example).

Consider Who Published the Data

Data from established, credible sources - like census data or reputable research firms - is generally more reliable, especially when the organization specializes in data and has a verifiable track record.

Be more cautious with research from vendors, industry organizations, or think tanks. It may be solid information, but it usually aligns with their goals. It doesn’t mean the data is wrong, just that the data publisher has incentives that could bias how the results are framed or interpreted.

In our example, Canva - a company that makes online design tools - launched a 2025 product focused on packaging data visually. That gives them an incentive to highlight findings that make their product seem essential. We don’t automatically reject the data, but we do read it with that incentive in mind.

How Do They Get Their Data?

Context shapes how you interpret the data and how much trust you put into its accuracy and reliability.

Understanding who the data represents, how they were selected, and how much of that audience is actually captured is key. A small, well-defined sample can still be meaningful. A large, skewed one can be misleading. You need to ask whether the data appropriately represents the population behind the claim.

Who got surveyed? Too often, a data claim is attributed to a broader audience than it supports. In our example, Canva’s headline claimed “89% of professionals work with data weekly.” However, the Canva survey only targeted marketing and sales professionals and not operations, IT, finance, or product. That’s a significant gap.

Understanding who the data represents, how they were selected, and how much of that audience is actually captured is key.

Methodology Matters

All sources require a critical lens, including government agencies or official statistics. How a metric is defined, calculated and scoped determines whether it is actually relevant to your situation.

For example: Did you know that measures like GDP and inflation are calculated differently across countries? In the US, multiple official measures of inflation exist, and the most commonly cited one excludes food prices. If your industry depends on food costs, that number may not tell the story you need it to.

Transparency

Strong data sources explain how they collected their data and acknowledge limitations. If methodology is unclear or hard to find, that’s a signal to dig deeper. No data is perfect, but credible data can withstand scrutiny.

In our example, Canva shared methodology details and worked with an outside survey company to conduct the research. That level of transparency is meaningful even if there are still limitations to consider.

The Reality: Most Data Won’t Be This Transparent

The Canva example is somewhat unusual in that it clearly states who published the data, where it came from, and how it was collected.

More often, you’ll see something more like this from The Economist:

No source for the estimate. No footnote. No context. Just a number presented as fact. At that point you either accept it as face value or go hunting for the original source yourself. (We did. Here’s the link.)

“This Is a Lot of Work. Can’t I Just Use AI?”

Yes, with some caveats. Despite AI’s well-earned reputation for occasionally making things up, it can be surprisingly useful for the legwork of validating sources.

AI can usually help you:

• Find and summarize the original source

• Interrogate the methodology and limitations

• Search for alternative data from credible sources that support (and contradict) what you are reading

• Flag potential conflicts of interest based on available context.

For a structured approach to research using credible sources, this article from offers a helpful framework and case study using Google’s NotebookLM.

AI should not replace your judgment on whether to trust a piece of data. It can however reduce friction and mental load of evaluating the data that crosses your screen.

More Reality: Everyone Has an Agenda

Everyone has goals, and those goals shape how data gets framed. A pharmaceutical company wants to show positive results to the FDA. A salesperson might tweak forecasts based on how they’re compensated. From students to policymakers, people interpret and feature data through their own lens.

Motivation doesn’t make data invalid, but it absolutely shapes presentation, emphasis, and interpretation. Factor in their agenda to interpret data responsibly.

Canva’s Study: What’s the Verdict?

Overall, the Canva study is directionally useful - with limits. We wouldn’t make high-stakes decisions based on it alone, but it’s reasonable to say that a large share of marketing and sales (not all) professionals are using data weekly and many of them feel anxious about it or would like more support.

We also keep in mind the context: Canva wants these findings to support their product launch, and the survey reflects marketing and sales professionals - not all professionals. Even within that group, results likely vary by industry.

One unexpected insight stood out more than the headline statistic:

That finding is less about Canva’s product than a universal risk; if nothing else, it’s a reminder to spot check a spreadsheet before acting on that data.

Why All of This Matters

Your decisions are only as strong as the data behind them. In a noisy world saturated with numbers and claims, your advantage is knowing how to discern what data deserves your confidence and how much.

That said, scrutiny has its limits. You can’t fully investigate every statistic you encounter. The goal isn’t perfect; it’s the discipline you can sustain.

Filter out unsourced claims. Prioritize transparent research. Leverage AI to help shoulder the investigative work. The result isn’t just better data consumption, but clearer conclusions and stronger decisions.

Share

Leave a comment

Subscribe now

Subscribe to Practical Data Foundations

You start a new role and inherit a set of dashboards you didn’t build. Or you’re running a small business that’s finally growing, and you realize gut instinct isn’t enough to make the next decision. Or perhaps you’re looking at new systems, sitting through demos and trying to imagine what the data would look like once it’s actually yours. Different situations, same feeling: suddenly there’s a lot of data on the table, and an unspoken expectation that you should be able to make sense of it.

In those moments, the data itself isn’t the hardest part. It’s the space between knowing the business and knowing the systems: recognizing familiar metrics, but not trusting the numbers or being clear on what they stand for; feeling like you should have some answers while still trying to understand what you’re even looking at. And particularly in small businesses, it’s a space you’re in without the staff or time to reconcile everything. It’s where many people get stuck.

And when you’re there, there can be a temptation to start analyzing just to have something to say. Any answer can feel better than no answer. But that’s like moving spontaneously through unfamiliar terrain just because you have a map in your hand. Before you head off in any direction, you stop and orient yourself. You figure out where you actually are, what landmarks you can trust, and which tools will help you make sense of the surroundings. Working with data is no different. You need to get your bearings.

When you have data but not a clear answer

Consider an established HVAC service business serving a core geographic area. Demand is steady, crews are close to capacity, and leadership is starting to ask whether it makes sense to expand service into neighboring ZIP codes or towns. On the surface, it sounds like a simple data question: Do the numbers support expanding our service crews?

The business has data, but it’s scattered across systems built for different purposes. Scheduling and dispatch data documents where crews are already going and how long jobs take. Billing data shows revenue by job and customer. Customer records reflect repeat service and maintenance plans. Lead intake data captures calls and requests, including inquiries from areas they don’t currently serve. Website data may hint at interest beyond the core territory.

Some of these systems will overlap. The same customer, job, or dollar amount may show up in more than one place. That’s not a flaw. It’s a clue. Overlap usually means the data is being used to answer different questions, from different angles.

Taken together, these datasets don’t add up to a single answer. Before modeling or projecting anything, the business needs to get oriented. Which data speaks to demand? Which reflects capacity? Which helps frame financial risk?

At this point, the goal isn’t to find the answer yet. It’s to understand what each dataset is actually good for. After that, you can move forward without guessing.

Build your data map

When you’re surrounded by data from multiple systems, the most useful thing you can do is stop treating it as one big pile. Instead, create a simple map that connects different types of data, usually contained in different systems, to the kinds of questions they’re designed to answer. This isn’t documentation or cleanup. You’re just trying to make the landscape visible.

Start by listing the systems and the data from those systems that you have. For each one, answer three basic questions:

• What kinds of questions was this system built to answer?

• What does it do particularly well?

• What should it not be used for?

For our growing HVAC service business, that might look something like this:

• Operational systems (scheduling, dispatch, job tracking)
  Show capacity and how crews spend their time, not unmet demand.

• Financial systems (billing, accounting, invoicing)
  Show revenue and margins, not why demand exists or where it’s coming from.

• Customer systems (service history, maintenance plans)
  Show repeat behavior over time, not early interest or acquisition.

• Lead intake systems (calls, forms, service requests)
  Show demand signals, not completed work or guaranteed revenue.

• Digital behavior data (website activity, service pages)
  Show awareness and interest, not operational or financial reality.

One metric, multiple locations (and definitions)

Sometimes the same activity shows up in more than one place. A service call appears in the scheduling system, in the billing records, and in the customer history, each time with slightly different context. A lead might show up as a missed call, a website request, or a note in a customer record, depending on where it entered the system. The overlap isn’t accidental. Each system is capturing the same moment from a different angle.

The goal here isn’t to decide which system is “right.” It’s to understand what role each one plays. Once you do that, overlapping data stops feeling like a contradiction and becomes context: different views of the same business, answering different questions.

Create a small shared vocabulary

Once you’ve outlined your data sources and mapped those to the kinds of questions they answer, it helps to pause and agree on how you’ll describe what you’re seeing. With data coming from multiple systems, alignment with your team matters.

Pick a short list of terms that keep coming up and agree on what you mean by them. For example:

• What counts as a customer?

• What do you mean when you say demand?

• What does revenue include? What’s not included?

• When you talk about a job, lead, or request, where does it show up first?

These don’t need to be perfect or permanent. They just need to be clear enough to keep conversations moving without constant clarification.

Once you’re oriented, take the next step

For our HVAC business, being oriented means they know what data they have in their landscape and can make intentional choices about which systems to rely on in different situations.

When considering expansion of their service crew, they know which systems they look to for capacity and scheduling lead times, which ones they use to understand demand, and which ones they trust when evaluating financial risk.

They understand what the data means, and what it doesn’t. So when lead counts don’t match completed jobs, that difference isn’t treated as an error to fix. It’s expected. Lead data is used to gauge interest and potential demand; completed jobs are used to understand what the operation can actually deliver. Each number has a role.

One last note: once you have your bearings, the next step isn’t to answer everything at once. It’s to choose where to focus; to choose your next destination.

Understanding the landscape gives you a way to be deliberate. You can decide which question matters most right now, and which view of the data is most appropriate for that question. And instead of reacting to every number on the page, you prioritize the signals that fit the question and let the rest stay in the background.

Then, when you are ready, you choose the next question or destination, and map how you will get there. Orienting with your data soon turns into forward motion step by step.

You get an alert saying the weekly dashboard is ready. You click the link and start scanning the numbers. Some are up. Some are down. You close the dashboard and move on to the next thing on your to‑do list.

Either that dashboard shouldn’t exist, or you’ve fallen into a common, but very understandable, trap: stopping at review instead of turning what you see into a conclusion and then a next action.

We’re not picking on you. Turning information into a conclusion and next action is genuinely hard work. It requires practice and repetition, plus the confidence to know which numbers matter and say what they imply.

Go Beyond Reporting

Reporting is the starting line, not the finish. To move from dashboards to decisions, we hold ourselves accountable by using this template when reviewing a metric, analysis, or report at work: Say – Mean – Do.

Start with a regular dashboard that reflects your team’s goals or priorities. Then pick one or two changes and ask yourself: What does the data say, and what does it mean? Numbers tell you what happened, but insight comes from explaining why. Context matters because data rarely speaks for itself. Next, comes the harder question: What am I going to do? This is where insight earns its keep, by turning into action. Insight without a next step is just trivia.

This is easier than you think.

Instead of: “Attendance was down 10%.” [Say]

Try: “Attendance was down 10%, likely because of last week’s snowstorm.” [Say + Mean]

And then identify a step like: “Next time, let’s plan a few programs that work better for families during bad weather.” [Do]

Or sometimes, no action at all. “Snowstorms are rare. Let’s make sure attendance rebounds next week.” [Do]

That’s still a decision.

The Say–Mean–Do Template

This is what you practiced above in a format you can reuse.

[What does the data say] because [why / what it means].

My next step(s) is [what will you do].

That’s it. Say. Mean. Do.

Using Say-Mean-Do in Meetings

You don’t need to formally introduce this framework for it to be useful. Used in a team setting, these questions shift the conversation from reporting to thinking together. They’re not about challenging authority or catching mistakes. They’re an invitation to think together, opening the door to curiosity, problem-solving, and shared judgment.

• “Let’s slow down. What do we see in the data?” [Say]

• “Okay, what does that mean?” [Mean]

• “So what’s the move?” [Do]

This kind of discussion enlivens the process of turning data into action. Over time, it weaves data naturally into your day-to-day operations and workflows, strengthening decisions and improving outcomes (without anyone ever having to say “Say, Mean, Do”).

Design Dashboards for Action

A big roadblock in data analysis is going from insight to action. Why? Because it will require time, energy, knowledge, confidence, navigating uncertainty - or any combination of these factors.

To reduce this friction, design your dashboards for decisions. We are not saying that your dashboard should tell you what to do and when. That type of design is development intensive and usually ends up being too rigid.

Instead, integrate the “Do” portion of Say-Mean-Do by defining up front what decisions that dashboard is meant to support. Draft a short statement clarifying why the dashboard exists, who it’s for, and, most importantly, what decisions it’s meant to inform.

Doing this work streamlines the data analysis piece of your workflow, and prevents dashboards from growing bloated - which also saves time on report production and review.

Template for Your Dashboard “Do” Statement

This dashboard exists to help [who uses it] decide [types of decisions it should support]. It’s reviewed [how often].

When a metric changes, the expected actions are [what we would do differently]. —> Spend your energy defining this portion, and refining as needed over time.

If a metric would not change a decision or action, it doesn’t belong here.

Share

Here are some examples with different contexts, but the same principle: decisions first, metrics second.

For a Marketing Team

This dashboard exists to help the marketing team decide where to invest next quarter’s budget. It’s reviewed monthly by the VP of Marketing and channel owners.

If a metric moves meaningfully, the expected action is to reallocate spend, pause underperforming campaigns, or double down on high‑performing ones. [DO]

If none of these decisions would change based on a piece of data, it doesn’t belong in the dashboard.

For a Yoga Studio

This dashboard exists to help the studio owner decide which classes to offer and where to focus outreach. It’s reviewed monthly.

When attendance or engagement changes significantly, the expected action is to adjust class times, class formats, or marketing efforts.

If a metric wouldn’t change those actions, it doesn’t belong here.

Leave a comment

Summary

Information is powerful, but it’s also easy to get lost in it. When there’s too much to review, you can lose sight of why you opened the dashboard in the first place and what decision you were trying to make.

With practice, you can build the habit of turning information into impact. Name what you see, say what it means, and decide one possible move. You don’t need perfect certainty. You just need to move from looking at data to using it.

When I first began using data as a way to communicate and support decisions, I was very close to the work. We were making choices about where and how to invest in our website, which drove leads, supported sales with existing clients, and delivered our products.

My job was the analysis: traffic, conversions, leads, performance. My audience was small and familiar: my team, my manager. If people had questions, we went deeper.

As my responsibilities expanded, I wasn’t just close to the work anymore - I was coordinating work.

There was still a strong foundation of data, and our team still worked through the details. But now my role included bringing that work up to other managers so we could prioritize and allocate resources. These were people accountable for timelines, budgets, and risk across areas they didn’t personally touch every day.

This is where something changed. People skimmed instead of reading. The “right” level of detail varied widely. Questions shifted from ‘how were these numbers calculated?’ to ‘so what?’ or ‘should I be worried?’ Accuracy alone no longer guaranteed understanding or support.

Eventually, I found myself supporting decisions even further from the work: helping set direction. I was presenting to senior leaders who weren’t trying to understand the analysis in detail. They were trying to make decisions under uncertainty.

The question was no longer ‘Is this analysis solid?’. It was ‘What path does this data support, and what happens if we’re wrong?’

They were weighing trade-offs. Assessing risk. Deciding where to commit or pull back resources.

I didn’t move smoothly through these shifts. For a while, I kept presenting data the way I always had, assuming the work would speak for itself. Sometimes it did. More often, it didn’t. The result wasn’t just confusion - it was delay. Work stalled because the case for what we needed wasn’t persuasive.

It took a few misfires to realize that accuracy and detail alone weren’t enough. I needed to rethink how I was presenting information, not just what I was presenting.

What changed wasn’t the data.

What changed was the decisions being made with the data, and my responsibility for how the information shaped those decisions.

A New Part of the Job: Translation

What I eventually realized is that my audience’s needs had to drive how I expressed the data: from which charts or analyses I led with, to how I talked about implications and uncertainty.

I needed to take the data my team and I understood intimately and present it in a way that helped others make decisions and secure the people, funding, and direction the work needed to move forward.

That was a shift: from delivering analysis to translating it for decision-making.

Earlier in my work, providing all the evidence was essential for credibility. And that rigor didn’t go away; I still needed to have the detail ready if someone wanted to go there. But now, the goal wasn’t to show everything I knew. It was to support a decision and make a clear case for the path I believed we should take.

Translation was about taking work my team and I understood deeply and making it usable in the context of a real decision.

Translation is choosing what to lead with, what to summarize, and what to hold in reserve, so the data support the decision at hand and the work has a chance to move forward.

Same data. Different questions. Different decisions.

To make this real, let’s look at a familiar example and see how the way we talk about the data changes as decision responsibility changes.

I’m using a website here, but the same pattern shows up anywhere people interact with your work, whether that’s product pages, public portals, library databases, donation inflows or enrollment forms. Wherever people engage, the data describing that engagement becomes an input to decisions about where to spend time, money, and attention.

When you’re close to the work, website data helps answer: What’s happening? Is this working? Traffic patterns, drop-offs, and usage help diagnose issues and test improvements. The dollar impact is indirect; your work supports better execution.

When you’re coordinating work, the same data helps answer: What should we focus on? Which areas deserve more attention? Where should staff time, design effort, or marketing support go, and where should it be pulled back? At this level, the data starts to support recommendations and trade-offs that map directly to budgets and capacity.

When you’re setting direction, the question sharpens further: Is this where we want to place our bets? Leaders are deciding whether the signals are strong enough to justify investment, what the risk is if they’re wrong, and what they’re choosing not to fund as a result.

The data hasn’t changed. The decision and the consequences have. That means the way you frame the data has to change too.

How the Same Data Gets Used Differently

Here’s how that shift shows up in practice: what you emphasize, what you summarize, and what you keep in reserve.

Where Translation Starts: Your Message

When data doesn’t have the effect you expect, it’s rarely because people are anti-data. People don’t resist data. They resist irrelevance.

To avoid this situation, consider these questions before you start polishing slides or pulling screenshots:

• Who is this for, and what kind of decision are they responsible for?
  Are they close to the work, coordinating it, or setting direction?

• What decision is being made - and what outcome am I trying to support?
  People, budget, direction, or permission to proceed?

• What does this audience already know, and what don’t they?
  What context can you skip, and what must be explicit?

• What would be easy to misread without framing?
  Where could someone draw the wrong conclusion?

Once you’re clear on those answers, then build the charts.

A few guardrails help keep the focus where it belongs:

• Don’t lead with detail when the decision is directional.
  Keep depth available, but don’t make it the entry point.

• Don’t hide uncertainty to make the case cleaner.
  Decision-makers need to understand risk, not be shielded from it.

• Don’t present data as neutral when you have a recommendation.
  Translation often means stating your point of view — and standing behind it.

And resist the urge to re-explain the chart, add more slides, or defend methodology when that isn’t what’s actually being questioned.

Done well, this preparation keeps you from drowning people in detail or oversimplifying and losing trust - and it lets you lead the conversation instead of waiting to be asked.

Data Translation is a Leadership Skill

As your responsibility grows, the work stops being just about producing accurate analysis. It becomes about helping good decisions happen: advocating for the resources, direction, and support the work actually needs, while making the trade-offs visible and the risks explicit.

You’ll encounter these shifts more than once. Roles and contexts change. Decisions move further from the hands-on analysis. Each time, the data may stay the same, but how you position it has to adapt.

That ability to adapt - to translate with judgment - is using data as a leadership tool.

Choose well, and metrics create focus, clarity in decision-making, and momentum. Choose poorly, and metrics will distract, demotivate, or push effort in the wrong direction.

Great leaders use metrics deliberately: to clarify what matters, align daily work to meaningful outcomes, and tell an honest story about progress or challenges. Like any leadership skill, this one takes practice.

This article covers how to choose metrics more thoughtfully and leverage them as tools for learning, focus, and impact - both for yourself and for your team.

Share

Recognizing Intentions vs. Goals

We’ve all heard statements like, “Improve client contact this year,” or told ourselves, “This year, I’m going to get healthy.” They sound motivating, but they’re not actually goals.

Why? Because they’re vague. When success isn’t clearly defined, there is nothing to measure, and nothing concrete to improve.

When fuzzy goals show up, it’s a cue that more clarity is needed.

Take “improved client contact.” That could mean talking to clients more frequently, or having higher-quality conversations, or seeing clients use your product more. Each interpretation points to a different measure of success and motivates very different actions.

The same is true for personal goals. “Getting healthy” might mean walking three times a week, sleeping seven hours a night, or eating more vegetables. Until you define what “healthy” means for you at this moment, you won’t make progress because you haven’t defined it (and if you haven’t defined the “what,” you certainly cannot track it).

The Art of the Metric - Lesson 1:

Turn intentions into goals.

When you replace vague resolutions with measurable definitions, teams gain focus, momentum, and a shared understanding of success.

Choosing the Right Metric for the Job

Not all metrics serve the same purposes.

Broadly, metrics fall into two categories: leading and lagging indicators.

Lagging indicators confirm what has already happened. Financial results like revenue, subscriber growth, or total clients served. They are important performance indicators, but they arrive after the fact.

Leading indicators act as predictors of future performance. In an established business, incoming leads may predict future sales. In content-driven environments (like right here on Substack!), consistent publishing habits may predict audience growth.

So how do you choose? It depends. Are you still learning what drives results, or are you focused on better executing those behaviors that you know influence performance?

If you’re figuring out what works, lagging indicators give you freedom to test and learn how to best achieve the goal.

For example, subscriber growth on Substack. When you’re experimenting with content or distribution, tracking subscriber growth keeps the focus on learning and creative problem solving without locking into one tactic too soon.

But once you know what reliably drives results, switch your focus. An outcome like “Reach 1,000 subscribers” becomes “Write Substack Notes five days a week.”

You can’t control who subscribes, but you can control showing up consistently with useful content.

The Art of the Metric — Lesson 2:

Loose metrics for learning. Tight metrics for execution.

When you’re unsure what drives results, measure to learn. When you’re sure, measure to build habits and get better every day.

Unintended Consequences and Motivational Tools

When pay, bonuses, or performance reviews are tied to a number, people get very good at hitting that number.

Because once a metric matters, behavior changes.

The risk is that people optimize for the metric itself and not the outcome you actually care about.

Measure a salesperson only on revenue, and they’ll sell what’s easiest to close, even if the company needs to push a new strategic product. Measure product managers on the number of client meetings, and you’ll get more meetings - not necessarily better ones.

Metrics shape behavior whether you intend them to or not. Without guardrails, they can quietly derail the very behaviors you need.

Before rolling out a metric, avoid unintended consequences by considering: If someone focused only on this number, what behavior would it encourage?

If the answer isn’t what you want, refine the metric’s definition and guardrails. Game the system on your own metrics or others will do it for you.

Metrics as motivation - people commit to numbers when two things are clear:

• How their work influences the metric

• Why the metric actually matters

Revenue growth isn’t just a finance goal - it funds salaries, programs, and investments. Meals served isn’t just a count; it represents families being supported. Even personal metrics work this way: more sleep makes it easier to show up well.

This is where metrics become a tool for alignment and momentum. When people understand both how they influence a metric and why it matters, motivation follows.

Art of the Metric - Lesson 3:

Metrics shape behavior. Design them with intention.

Before rolling out a metric, ask what behavior it rewards. Combine intentional guardrails with a clear why, and metrics become tools for alignment—not control.

When You and Others Share a Key Metric

You may hear: “Why am I being measured on something I don’t actually control?”

Totally fair.

Revenue is a classic example. People often say, “I’m not in sales—why is revenue in my goals?” While you might not be signing contracts with new clients, you likely contribute to that goal in meaningful ways.

Marketing fuels revenue through lead generation. Product teams contribute by launching or improving offerings. A finance analyst surfaces insights around new opportunities.

This tension shows up with many metrics: customer satisfaction, engagement, retention, foot traffic.

This is where your leadership judgment comes in. Strong leaders don’t reject big outcome-based goals just because they aren’t perfect.

They translate them into team or person specific metrics or behaviors their teams can influence.

They use metrics to connect everyday work to meaningful broader impact on the organization.

Metrics as Focus vs. Distraction

Metrics are meant to create focus, but too many will do the opposite.

Most people can realistically focus on one to three priorities at a time. That might look like:

• One major initiative to launch

• One core metric to maintain

• One metric you’re actively trying to improve

Additional data still matters, but it shouldn’t compete with priorities.

For example, if your goal is to increase marketing leads, email click-through rates are useful context, not the headline. If the click-through rate dips, and you don’t know that it meaningfully drives leads, it should not suddenly become an additional priority.

More metrics don’t create more progress. Focus does. Choose the few that matter, and let the rest stay in the background.

When a Metric No Longer Serves

Sometimes a metric isn’t working. That certainly happens, but before changing course, assess: Is this truly the wrong metric or is progress slower than we hoped?

If a metric drives the wrong behavior, no longer reflects what you care about, or isn’t teaching you anything useful, it’s time to adjust.

But if the metric is sound and the work is simply hard, staying the course may be the right leadership call.

Metrics as a Leadership Practice

Metrics shape focus, behavior, and motivation. And the metrics you choose send a signal about what matters most.

Great leaders use metrics to translate strategic goals into something their teams can influence. They create meaning from numbers, so people understand not just what is being measured, but why it matters

Where to start? Pick one metric you have today and ask yourself: What behavior does this metric encourage? Does it reflect where I am putting my focus and energy?

If not, make one small tweak. And then another.

That’s how you start to turn metrics into a practical leadership tool, not a reporting obligation.

As data has become central to how organizations operate and make decisions, the number of roles and titles for “data people” has grown and the boundaries between them have blurred. Analyst. Engineer. Scientist. Steward.

These titles are meant to hint at different kinds of work, but in practice they overlap far more - and shift more often - than most job descriptions suggest.

Titles are a weak proxy for how someone can help you

Data roles evolve. Tools change, teams grow, and work shifts from exploratory to operational. Titles often stay the same because changing them feels like overhead - even when the day-to-day work looks very different from what the title implies.

In other cases, the mismatch starts from the outset. A title is chosen based on reasonable, but incomplete, assumptions about what a “data analyst,” “data scientist,” or “data engineer” does, often borrowed from another organization or a job description. The title signals intent or aspiration, not a precise account of the work.

In larger organizations, the issue is often structural. A small set of standardized data titles spans many teams and responsibilities. These titles work for HR systems and career paths, but they flatten important differences in how people actually spend their time.

The takeaway? It’s simple: titles are a weak proxy for how someone can help you. Whether you’re coordinating data work or contributing to it, assuming fit based on a title alone will create friction with your data partner. Understanding what people actually do - where they add value and how they work with ambiguity - matters far more than what their role is called.

Common Situation #1: One title, many jobs

Data analysts are everywhere. Most teams have at least one, regardless of size or industry. In my own experience on both small teams and in larger organizations, the role consistently shows up, but what it involves can vary widely.

While the title suggests analytic work (examination, interpretation, decision support), in practice these roles are often heavily operational. Maintaining pipelines and reports, and keeping data reliable and consistent, make up much of the day-to-day work. Analysis still happens, but it’s typically incremental and bounded by existing data structures and processes.

Common Situation #2: Same title, different work

I once managed a small team that included two people with the title “data scientist.” One had a robust academic background in advanced math and machine learning theory, and excelled at model selection - but only once the work was clearly defined. His grasp of the business context was limited, and his approach tended to be more literal than creative when deciding how to apply data science techniques.

The other data scientist did not come from a rigorous academic training environment and was less familiar with more complex models. This person was exceptionally strong at shaping the work itself - developing creative approaches, having an instinctive feel for what the data could and couldn’t support, and anticipating the end-to-end impact on the resulting product.

The title alone would never have told you who was best suited to define the work and who was strongest at executing it. That clarity is what allowed the team and its partners to move faster and work more effectively.

Finding the right data partner in practice

Finding the right data partner isn’t a one-time decision. It depends on where the work is and what it needs next.

If the work is still fuzzy, you may need someone who can help shape it, even if they aren’t the most technically specialized person on the team. If the work is well defined, execution strength matters more. And if the work needs to run over time, operational skills are critical, regardless of title.

Titles can help you find people. They can’t tell you how the work will unfold. Expect variation within roles, expect to involve more than one partner, and expect your needs to change as the work evolves.

Translation Guide: from need to data language to titles

Once you can translate your need into language data teams recognize, titles become useful not as answers, but as navigation aids.

This isn’t about getting the terminology exactly right. It’s about giving your data partners something concrete to react to.

The same title may appear in multiple rows - that’s expected. What matters is the work being asked for, not the label.

Note that some organizations use the title applied scientist to distinguish method-heavy, machine learning focused work from broader data science. Others don’t. The work shows up either way.

When you let the work guide you, titles stop being a source of friction and start becoming useful signals.

Whether you’re planning a job search or just want to be ready, it’s a great time of year for a resume refresh. So today we cover how data plays a key role on your resume by showing results, not just activity.

Strong formatting, action verbs, and keywords matter, but your quantified experience shows impact and makes your resume unmistakably yours.

This does take a bit of upfront thinking and effort. But once you’ve done that work, you have those accomplishments to use forever.

WOW with Trend Numbers

Trend data are especially powerful on resumes. They grab the reader’s attention quickly, and there is a clear before‑and‑after: a starting point, an action, and a measurable change. Hiring managers notice that clarity and impact.

Express trends as percentages (XX%), factors (2X, 3X), or strong action verbs (“doubled,” “tripled”). We generally favor percentages, but a thoughtful mix keeps your resume easier, and more interesting, to scan.

Examples of increased trends that work well on resumes include: revenue, sales, clients, leads, margin, products or features shipped, market share, average account size, and volume-based outputs like events or campaigns.

• Increased revenue 3x to $10M, while improving contribution margin from 5% to 18%.

• Doubled qualified leads in 7 months.

• Expanded average account size by 45% and improved client satisfaction scores 40%.

• Processed 55% more invoices with the same headcount by implementing process and technology improvements.

In rapid-growth situations, increases can reach the hundreds - or even thousands - of percent. In those cases, use factors instead (for example, 10x instead of 1000%) for faster, easier scanning.

Quick math reminder: if something grew from 10 to 12, that’s a 20% increase, not 120%. If you doubled a result, that’s 100% growth, not 200%.

Decreasing trends can be just as impressive. These usually show up where you improved efficiency, reduced errors, or saved time or money. For example:

• Reduced time-to-market 50%, from 6 months to 3 months.

• Decreased accounts receivable by 23%.

• Cut an average of 5 hours from monthly operations report production, a 33% time savings.

• Slashed error rates by 5X in 4 months.

If you don’t know the exact percentage, it’s okay to estimate as long as it’s a realistic reflection of your results. For example, if something increased, but you’re unsure whether it was 95% or 105%, use “2X,” “doubled,” or “approximately 100%.” If something decreased by roughly 30–35%, you can reasonably say “reduced by one-third.”

Share

Add Flair with Scope and Ranking Metrics

Not all strong metrics are trends. Quantifying the scope of your responsibilities or your rank / standing adds context and credibility, especially when growth percentages aren’t the whole story.

Here are several effective ways to do this:

1. Scope of responsibility (team, budget, portfolio, footprint)

• Managed facilities for 80 buildings totaling 1.5M sq ft.

• Responsible for a $50M client portfolio.

2. Output measured by units or counts

• Launched 5 new products in 6 months.

• Built and scaled a team of 100+ developers.

3. Rankings, ratings, or relative performance

• Top 3 ranked salesperson for 5 consecutive years.

• Maintained employee satisfaction ratings of 96%+ annually.

• Moved ABC program from #7 to #1 in XYZ Industry Rankings over two years.

4. Company or industry awards

• Winner of the company’s annual Rocket Award (1 of 2,500 employees).

• Winner of 3 Webby Awards for Best User Experience.

5. Scope of a problem solved (often expressed as a percentage)

• Reduced unresolved customer complaint backlog by 50% in 2 months.
  (This is more informative than “Resolved 10 outstanding customer complaints,” which lacks context for how big the problem was to start.)

• Built an invoice collections process and recovered 43% of outstanding revenue.
  (In this case, the $430K recovered did not sound like a lot, but it was on a $1M revenue product making the impact meaningful.)

Unsure Where to Begin? Start Here

Staring at your resume and drawing a blank?

Begin with metrics commonly associated with your role or the role you want. In marketing, that might be engagement or leads. In product, time‑to‑market or product revenue. In sales, revenue growth or average contract value.

It also helps to prioritize metrics tied to company goals, such as: revenue, market share, or customer and employee satisfaction. Wherever possible, show how your work contributed to those outcomes. “Grew my product revenue 54% in 2 years.”

Finally, showcase metrics you’re genuinely proud of. You’ll sound more confident telling the story behind them. And yes, interviewers will ask about metrics on your resume so be ready to walk through each metric using the simple structure:

• What was the situation?

• What did you do?

• What were the results? (your metric)

Still struggling to find solid numbers? Revisit past performance reviews. That’s often where your strongest, most resume‑ready metrics may be hiding. And don’t forget to look at relevant job postings for ideas too.

Share

My Role is Hard to Measure

You can quantify much more than you think. Here are some common situations and how to express them with data.

• In high‑throughput responsibilities like in accounting, operations, or copy editing, sometimes the focus is on getting the work done rather than measuring it. Look back and quantify aspects like on‑time delivery, accuracy, volume, or turnaround speed.

  • Delivered monthly reporting 100% on-time and correct.

  • Reduced average copy edit turnaround from 2 days to 4 hours.

• Perhaps you were part of a team that delivered a single, major outcome. Be explicit about your role and the team’s achievement.

  • Served as Operations Lead on a small team that launched a product that grew to 20% of company revenue in its first year.

• Creative roles can be especially tricky. Identify aspects of your role that make you proud or where others often compliment you.

  • For example, if you’re a graphic designer and effective listening is important. Maybe you consistently deliver work that requires only one small round of revisions while others need two or three. That’s measurable.

If you’re still unsure, ask a colleague or someone you trust to help brainstorm. Sometimes you just need an outside perspective.

Leave a comment

You’ve Got This.

Resume writing is not likely high on your want‑to‑do list, but identifying your measurable achievements can be surprisingly gratifying.

One word of caution: as we look back on what we’ve accomplished, we are often too hard on ourselves. We compare ourselves to others.

“I grew my product revenue only 10% per year. That person grew 50% per year.”

“I managed a team of 8 people, but they managed 80.”

Don’t. Do. This. Avoid this common trap, and put your accomplishments in the right context. Context matters and hiring managers will recognize that.

Final example: A teen in my life recently tried out for a club sport and felt disappointed because he “only” made the third‑ranked team. When we looked closer, he’d competed against 140 athletes - many with years more experience - and finished in the top 35. Same result, very different perspective. Context matters.

Don’t let the absence of obvious metrics diminish your confidence in your contributions. Impact comes in many forms, and most of them can be measured with a little reflection and a little creativity.

Still struggling to frame a metric? Share it in the comments or send us a DM. We read and respond to every message.

Years ago, a senior leader asked my data team to prove that our dataset, combined with a machine learning model we’d been experimenting with, could support a more granular customer product than anything the company had offered before.

It was a new, exciting methodology. The buzz spread quickly. People unfamiliar with the technology began making bold promises and sketching out roadmaps. The enthusiasm was contagious, and the pressure to deliver something impressive mounted fast.

If you’ve worked in data, product, or analytics, you’ve probably seen this dynamic before. Organizational excitement can take on a life of its own, creating momentum long before the viability of an approach has been tested.

When Data Hits Its Limits

What happened next made this task difficult. But not because the model failed. It didn’t.

We delivered. We developed the model, ran it, iterated, and hit the deadline.

What made it difficult was realizing that the outputs were asking us to pretend.

The model relied upon the assumption that at more granular levels, the company’s data would hold up - there would be sufficient, reliable information to support more detailed analysis.

But the moment we pushed in that direction, the foundation cracked.

At finer levels of detail, the data was not robust nor reliable enough to support the claims the product would need to make. Incorporating those results wouldn’t just risk being wrong. It would create the appearance of insight without the substance to back it up.

Why? You can run a model perfectly and still end up with nonsense, because the model can’t invent reality that isn’t in the data. Past a certain point, you’re no longer analyzing. You’re interpreting smoke.

My team and I demonstrated exactly where the data stopped reflecting reality. And just as importantly, we showed that a fix wasn’t a matter of “cleaning the data” or “tuning the model.” The dataset, as it existed, could not responsibly support the outcome the organization was hoping for.

When Data Work Stops Being Analytical

At that point, the real question wasn’t whether we could keep trying - running more iterations, plugging holes. It was whether it made sense to. We had reached the point where the data work stopped being analytical and started becoming performative.

If we’d kept going, the model outcome wouldn’t have been neutral. We would have created something polished - and wrong, where the output no longer reflected reality.

That’s a dangerous place to be. Not because the math is hard or the model is complex - but because the work starts to serve the momentum rather than the reality.

Knowing When to Stop

Whether you are leading a data-related project or you are on a data team, there’s a strong pull in data and analytics work to keep going. If the results don’t look right, try a different model. Explain away gaps. Refine what data “counts” until the results look cleaner.

All of that effort feels productive and looks rigorous. And when advanced methods or AI are involved, it signals sophistication. But there’s a point where continuing doesn’t get you closer to the truth. It just gets you closer to a story you want to tell.

Leadership shows up in recognizing the facts of your current situation and making the call to stop.

Leadership is knowing when to say: this is as far as this can go.

Slow the Room Down

Calling a halt is easier said than done.

There is pressure at every level to keep an exciting project moving forward - whether personal visibility, career momentum, team justification, market competition, or the expectations of leadership above you.

Resisting that pressure when the situation suggests a pause requires both data and leadership skill. It’s also uncomfortable, politically delicate and often risky.

When teams - leaders, product managers, data scientists - get excited about a new method, anchoring the conversation in reality creates space for better decisions.

A few well-chosen questions can slow the momentum and bring clarity back into the room:

• What is the objective?
  If the real goal is “prove this hypothesis could be ‘right,’” you’re already at risk of chasing a story instead of insight.

• What would have to be true in the data for this experiment to work?
  In our case, it would have required the data classification process to be consistently accurate at a much finer level of detail - something it was never designed to support.

• Where is this data doing its job and where does it stop?
  The data was fit for its original purpose and trusted for that reason. The problem wasn’t quality; it was overreach.

• What’s the risk if we’re wrong, and who pays for it?
  If a polished output persuades customers or executives to act, the consequences become reputational, financial, or operational.

And a final gut check:

• If we didn’t already want this to work, would we still pursue it?
  If the answer is no, that’s usually your signal to stop.

What Comes After Stopping

Knowing when to stop isn’t just an internal judgment call. It’s also a communication challenge.

In this case, stopping meant clearly showing where the data stopped reflecting reality and translating the technical limits into terms that stakeholders could understand and trust. No jargon or drama. Just a focus on facts, boundaries, and consequences.

Just stopping isn’t enough.

Your stakeholders will naturally ask “Now what?” Leadership means being prepared for that question. If this experiment can’t responsibly deliver results, then what can? Take advantage of this opportunity to propose alternatives.

Anyone can push forward.

Leadership is knowing when to say: this is as far as this can go - and then helping the organization move somewhere better.

Share your examples! Have you had a data project turn into an exercise in leadership? Tell us about it!

Leave a comment

Understanding the journey of your data gives you real leverage. You ask sharper questions. You avoid rework. You communicate better with technical partners. And you start using data not just as information, but as a leadership tool.

The Plain‑English Definition

The data lifecycle is how the raw material turns into something you can rely on and put to work.

From the moment data is created or collected to the moment it’s used, and eventually archived or deleted.

While technical teams often use 5–8‑step models, you typically don’t need that level of detail. More useful is a mental model to quickly parse out where your data comes from, how it changes, and what it means when you finally see it.

A Useful Way to Think About the Data Lifecycle

Forget the traditional 5+ stage “data lifecycle wheel.” Think of data like a product moving through a production workflow.

Let’s say you are building a car. There are:

1. Inputs: raw materials

2. A manufacturing process: where inputs are shaped, refined, and tested to become something usable

3. Outputs: the final product that you drive

Data works the same way. The below workflow will help you anchor technical concepts and jargon in everyday language that make it easier to communicate with your technical partners, clients, and others.

#1 - Inputs: Collecting or Creating the Data

Your data inputs are the raw materials. Data can be generated automatically or entered manually, and each path introduces different opportunities and risks.

Common data input sources include:

• Website or app activity

• Customer transactions or updates

• Third‑party datasets

• Forums, surveys, or manual data entry

For example -

Email newsletter: You collect subscribers’ names and emails plus behavioral data like open rates or click paths.

Local breakfast taco guide: You manually enter restaurant names, locations, and your personal ratings.

Some teams separate “data generation” (the moment data is created by a system or event) from “data collection” (the moment an organization chooses to capture and store it).

For our purposes, data is data and what matters is:

Where does the data come from, and how trustworthy is it?

#2 - The “Manufacturing” or Data Production Phase

Processing & Transforming — Making the Data Usable

These activities are invisible to most users, but essential to producing reliable and useful outputs.

Common processing activities include:

• Cleaning: fixing typos, removing duplicates, addressing missing data

• Standardization: aligning formats (dates, phone numbers, categories)

• Validation: checking for impossible values or inconsistencies; applying business rules

• Enrichment: adding calculated fields; applying external attributes like demographic information or industry classifications; applying internal attributes like communication preferences

• Aggregation & transformation: shaping, modeling, summarizing data for analytics and reporting.

Going back to our two examples -

Email newsletter: Remove duplicates, validate email formats, merge subscriber information with internal customer records where available.

Local breakfast taco guide: Enforce 1–5 rating scales, add business hours or Instagram handles.

Managing and Maintaining Data

When data is collected and processed, it needs a home and a set of rules to maintain it. This is where the messy middle gets housed and maintained.

Management of data includes:

• Data organization and storage (databases, warehouses, lakes)

• Access and permissions (view, edit, download)

• Governance (quality rules, privacy and compliance)

• Documentation (definitions, refresh frequency, lineage)

Back to our examples -

Email newsletter: Where is the list stored? Who owns it? How often do you remove bounced addresses? What privacy rules apply?

Local breakfast taco guide: Is the list saved on your laptop or in the cloud? Can others view or edit it? Who updates closures or ratings changes?

These may feel technical at first, but at their core they’re just practical ways to keep your data organized and trustworthy.

• Storage choices affect cost and refresh frequency.

• Weak governance leads to multiple versions of the truth.

• Clear documentation helps prevent misinterpretation.

#3 - Outputs: Sharing and Publishing the Data

This is where the processed data becomes usable and actionable to the organization. Common outputs include:

• Dashboards and reports

• Downloadable datasets

• APIs for other systems

• Customer‑facing features, services, or products

Email newsletter: Marketing downloads datasets to segments subscribers into targeted audiences. The Analytics team tracks open and click-through rates using a dashboard.

Local breakfast taco guide: The dataset might become a public map for others looking for that perfect taco.

Strong documentation is important throughout the data lifecycle. Here, it sets critical context for understanding well the data will be used to make decisions to support the organization’s activities. For example: what each field means, what assumptions were made upstream, and what limitations or caveats apply.

Without this context, even good data can lead to poor decisions.

Share

Here’s Where YOU Step In and Shine

You have your data output in the formats, time periods, and granularity that you need. You understand what the data is, where it comes from, how it was cleaned and augmented. You have reporting and analytics that surface trends and insights.

This begins your most important role in the data lifecycle - turning data into decisions, communication, and action.

This is hard work.

And where your leadership shows up:

• Operationalizing information into daily decisions and activity, like maximizing social media messaging and timing.

• Communicating data-driven stories to stakeholders, like where to invest time and resources to better drive progress against the organizational goals.

• Choosing metrics that align with team or organizational goals.

• Using data to inform and shape organizational strategies and allocate resources.

This takes judgment, not technical skills. And the faster you can move from Output → Analysis → Activation, the more impact you create.

Guardrails That Keep You & Your Data Safe

These topics span multiple stages and influence how data is handled across its entire lifecycle.

Data Security & Privacy

Security and privacy obligations vary by industry and data type. Regulations like GDPR and CCPA set rules for consent, deletion, and usage of personal data in certain geographies.

Strong security and privacy practices build customer trust while also protecting high‑value data assets from cyber threats.

Data Retirement

Not all data should live forever.

• Archiving stores older data at lower cost.

• Purging removes data that’s no longer needed.

• Retirement policies reduce risk and prevent unnecessary storage costs.

These practices matter for cost, compliance, and operational clarity.

The Real Reason the Data Lifecycle Matters

Understanding the data lifecycle isn’t about memorizing steps or learning technical jargon. It’s about giving yourself the clarity to lead with confidence.

When you know where your data comes from, how it’s shaped, and what assumptions sit underneath your dashboards and reports, you make better decisions — faster and with far less friction.

Here’s what that clarity unlocks:

• You stop treating data as a black box.

• You make sharper decisions that save time, money, and energy.

• You spot issues earlier — before they derail a project or launch.

• You communicate more effectively with technical partners.

• You build trust by bringing context, not just numbers, into conversations.

Most importantly, understanding the lifecycle helps you use data as a leadership tool — one that strengthens communication, sharpens judgment, and amplifies impact.

You don’t have to be technical. You just need the right lens.

And now, you have a straightforward framework to do just that.

You’ve got this!

Bloom & Nest Home Goods (a fictionalized example inspired by real teams) is a small online brand selling seasonal home and fragrance products, including pumpkin spice diffusion oils, hand-blown glass ornaments, and a winter hearth candle trio.

They were sending frequent email campaigns, but couldn’t answer a foundational question:

Which email themes reliably drive engagement and sales?

This case study shows how Bloom & Nest applied the core skills we’ve been teaching to get clarity: framing the problem, preparing structured inputs, evaluating AI results with human judgment, and adding small governance habits.

There’s a quick start guide at the end with some steps to get you started incorporating AI into marketing analysis on your team, regardless of team size.

Situation: Life As a Lean Marketing Team

Bloom & Nest’s Marketing team’s environment will be familiar to anyone who’s worked in a small, fast-moving business:

• Campaign names varied widely (“Fall Drop,” “Cozy Update,” “wk3/Oct”).

• Some emails were labeled with their theme (Launch, Seasonal, Care Tips); others had no labels at all.

• Tracking links (so the team could see what happened after someone clicked) were inconsistent.

• Older campaigns came from multiple platforms and didn’t follow the same naming or tagging conventions.

• Drafts lived in a mix of Google Drive, Gmail, and the email/SMS platform.

• Product descriptions differed in tone, structure, and detail.

This made it hard for the team to see patterns or explain why some emails worked better than others. The founder believed that seasonal content was the big driver. The marketing manager wasn’t convinced. The way their content and data were organized made it nearly impossible to answer the question with confidence.

Most small teams have a version of this: scattered inputs, inconsistent labels, and no single place that shows the whole story.

Challenge: Use AI to Uplevel Email Performance

When Bloom & Nest first tried using AI to “analyze our marketing,” the outputs looked polished, but they weren’t reliable:

• AI misinterpreted product names (e.g., flagged “Morning Matcha” as wellness content).

• It confidently referenced a “Black Friday 2021 sale” that never happened.

• Some performance numbers didn’t match the team’s own metrics.

• Recommendations leaned on generic industry advice instead of the brand’s real behavior.

Nothing was “wrong” with the AI itself. The lack of useful insight came from how the team’s own content and data were organized.

Before they could get trustworthy analysis, Bloom & Nest needed to give the AI clarity and structure: a focused question and well-prepared inputs.

Approach: Getting AI to Add Real Value

The team approached this project using the practical framework from our AI at Work series: start with a clear question, organize the inputs, then review the results with human judgment before refining the process.

Step 1: Frame the Question Clearly

Instead of asking AI to “analyze all our marketing,” the team narrowed the problem to something specific and answerable:

“Summarize our recent email campaigns and group them into consistent themes we define, so we can identify which types of content drive engagement and conversions.”

A well-defined question set boundaries around what AI should do (summaries, classifications, basic analysis) and what remained human responsibility (strategy, interpretation, and decisions).

It also established the right success criteria:

• Accurate summaries

• Correct classification into the chosen themes

• Insights that reflect actual performance, not AI’s assumptions about marketing

This reframing alone made the AI process more predictable and easier to evaluate.

Step 2: Prepare Better Inputs

Next, the team improved the information they were feeding into the AI. This wasn’t a big cleanup project, just a focused, lightweight effort to make the inputs clearer and more consistent.

Create Consistent Campaign Names

The team renamed the last 18 months of emails using a simple format: date + category + topic.

This gave the AI (and the team) a predictable foundation: at a glance they could see when an email was sent and what it was about.

Define a Small Set of Content Themes

Bloom & Nest agreed on six clear, durable categories: Launch, Restock, Care Tips, Behind the Scenes, Seasonal, Promotion.

Short, one-sentence definitions ensured everyone - including AI - knew what each category meant.

Clean Up Missing or Inconsistent Metadata

They filled in missing dates, corrected a few major mislabels, and retired duplicate tags (e.g., “fall,” “autumn,” “fall-promo” all became “Seasonal”).

Perfection wasn’t the goal - clarity was. These steps made the inputs understandable to both the AI and the team.

Give Product and Brand Context

To prevent AI from drifting into generic marketing language, the team provided information about the company and a short description of Bloom & Nest’s target audience.

Product descriptions were scattered, so the team pulled them into one file. They used AI to create first-pass summaries of each product’s scent, tone, and purpose, and then manually corrected the descriptions for accuracy. They loaded the AI with the final product catalogue.

Both documents reduced noise in later analysis; the team decided to maintain this documentation going forward.

Prepare the Right Metrics

Content alone wasn’t enough. The AI also needed structured performance data, so the team pulled together a small set of useful metrics that reflected what they cared most about:

• Open rate (baseline engagement)

• Click-through rate (actual interest in content)

• Click-to-purchase rate or sales per send (conversion signal)

• List growth or unsubscribes (signal of fatigue or mismatch)

They organized these in a simple, consistent structure with one row per campaign with the same fields each time. Below is an example row; the full table followed this format for every campaign:

How They Told AI to Use the Metrics

AI does better when you tell it exactly how the numbers matter. Bloom & Nest used prompts like:

“Here are the performance metrics we track. When looking for patterns, prioritize click-through rate and purchases. Use open rate as context, not the main signal.”

“Compare categories using averages, ranges, and notable outliers. Do not generalize beyond the data provided.”

This kept the model grounded in what the team valued instead of falling back on generic marketing logic.

How to Ask AI to Analyze the Data

Once the table was prepared, they used prompts such as:

“Using the table provided, identify which content categories perform best across opens, clicks, and purchases. Show your reasoning step-by-step, including which metrics influenced your conclusion.”

And:

“Highlight any campaigns that deviate significantly from their category’s typical performance and propose possible reasons, based only on the content summaries.”

Giving AI structured metrics and explaining how to use them reduced misinterpretations and made the performance analysis reliable. It also helped the team push past vague impressions (“seasonal content does well”) toward evidence they could act on.

Step 3: Evaluate Outputs - Trust, Adapt, or Toss

With clearer inputs, the team tested how well the AI followed their instructions and whether the outputs could be used in real analysis. They used three prompts:

Prompt 1: Summaries and classification

“Here are 20 past Bloom & Nest newsletters with their dates and topics. Summarize each in one sentence and classify it using the six categories provided.”

This tested comprehension. If the AI couldn’t summarize or classify correctly, deeper analysis would not be reliable.

Prompt 2: Identify performance patterns

“Using these summaries and the performance metrics (opens, clicks, purchases), identify which content themes perform best. Show your reasoning step-by-step.”

The “show your reasoning” instruction exposed the logic behind each conclusion, making review faster and more transparent.

Prompt 3: Flag unsupported claims

“Point out any conclusions that cannot be directly traced to the provided data.”

This limited AI’s tendency to generalize or rely on general marketing patterns.

How the Team Labeled the Outputs

Once the inputs and metrics were ready, the team could finally test how well AI understood the task. Each AI response was labeled as Trust, Adapt, or Toss, creating a shared vocabulary for evaluating what the AI produced.

Trust (accurate and usable)

AI correctly identified that Care Tips emails had roughly twice the click-through rate of new-scent launches. The numbers matched internal analytics, and the explanation of the AI’s reasoning process was sound.

Adapt (directionally correct but needing edits)

AI sometimes misclassified campaigns; for example, it treated a restock announcement as a new product launch because of the language used. The team clarified the difference between the two categories, added clearer examples, and re-ran the prompt.The classification corrected on the next run.

Toss (unsupported or incorrect)

Some outputs were confidently wrong, like referencing a “Black Friday 2021” sale that never happened. Since this was a testing phase, the team excluded these results from the process. If an output couldn’t be traced back to real data, it wasn’t considered further.

The team added a brief note explaining why the output failed (AI filling in a pattern that wasn’t there) and updated their instructions for the AI. On the next test run, the error didn’t reappear. That gave them confidence these issues wouldn’t slip into real analysis later.

Step 4: Maintain a Simple Improvement Loop

To keep the workflow stable and transparent, the team documented:

• The prompts they used

• The inputs supplied

• Trust/Adapt/Toss decisions

• Notes on misunderstandings

• Adjustments to examples or definitions

They also saved a small set of representative emails (10–12 campaigns that produced clean results) as a reference set for re-running prompts and making sure the AI stayed consistent over time.

This wasn’t heavy governance; it was just enough structure to keep the work consistent.

Subscribe now

Multi-faceted Results

Identified Business-Driving Performance Patterns

With a clear question, better inputs, and a consistent review process, the insights became easy to see and easier to trust.

• “Care Tips” and “Behind the Scenes” emails consistently performed strongest in terms of open rates and clicks.

• Restock announcements converted to a sale better than brand-new product launches.

• Seasonal content mattered, but was not the primary driver the Founder assumed.

Cut Planning Time

• Quarterly planning time dropped by more than half because the team no longer had to manually sift through months of campaigns for content or performance.

• Understanding performance by content type also allowed them to set a more predictable content calendar.

Clearer Team Focus

The team shifted away from seasonal storytelling and focused on the content types that actually drove engagement.

Stronger Metadata and Analysis

Simple naming conventions and ongoing product documentation now support every future analysis, whether human or AI.

These results mattered, but the more durable change was in how the team approached the work itself.

The Learning Went Beyond ‘How to Use AI’

As they moved through the project, the team sharpened habits that support good decisions in any organization. Clarifying the question, organizing the inputs, reviewing results carefully, and building a simple workflow strengthened the practices that lead to clearer, more confident decisions.

Here’s what their experience shows:

Clarity moves work forward.

Bloom & Nest made real progress the moment they shifted from “analyze our marketing” to a specific, answerable question. That clarity simplified every step that followed.

Clean inputs create shared understanding.

Organizing information isn’t administrative noise. It’s what lets teams see the same picture.Clear names, shared categories, and consolidated notes reduced confusion and made the insights possible.

People’s judgment is still required.

Tools can summarize and sort, but they can’t decide which findings matter. The Trust/Adapt/Toss decisions helped the team practice making thoughtful, evidence-based choices.

A little structure goes a long way.

They didn’t need heavy processes but just enough structure to make the work predictable and repeatable: simple conventions for naming campaigns, defining themes, and reusing examples where they fit.

Better structure leads to better choices.

By organizing their information and reviewing outputs deliberately, Bloom & Nest finally saw what was truly working and could make decisions with more confidence.

Habits matter more than tools.

Whether a team uses Klaviyo or Mailchimp, Shopify or Squarespace, the underlying practices stay the same: ask a good question, prepare clear inputs, and review outputs with care. Good habits travel with you, no matter the platform.

Start Now: How to Do This With Your Team

If you want to put these ideas into practice, you don’t need a full audit or a big project. A short, focused exercise can help you see the same kinds of patterns Bloom & Nest uncovered and build the habits that make analysis easier over time.

Here’s a simple way to get started:

1. Pull 10 - 15 recent emails or recurring updates.
   Don’t overthink which ones - recent, representative samples are enough. Don’t have marketing campaigns? Use email newsletters or product updates.

2. Create 4 - 7 clear content categories.
   Keep them simple and mutually distinct.

3. Standardize names (date + category + topic).
   This alone will clarify patterns you couldn’t see before.

4. Ask AI to summarize and classify them.
   Use short, direct instructions.

5. Apply Trust/Adapt/Toss.
   Identify what you can use as-is, what needs refinement, and what to ignore.

6. Record insights and update your categories for the next round.
   This turns one exercise into a habit.

This exercise gives you a sharper view of the patterns behind your content: what you send, how often you send it, and which themes resonate.

Do you have a case study to share? Tell us about it!