We are measuring AI productivity too early. I’ve done it myself. I’ve spent a lot of time thinking about Return on Inference: how we measure the value created when AI becomes part of the work. Productivity. Speed. Quality. Cost. New capabilities. New revenue.
All of it matters. But at this stage of AI adoption, we’re overcomplicating the measurement problem. The argument arrives a stage too late.
Before we measure where the value is coming from, we need to measure whether people are spending enough serious time with the technology to develop fluency.
The most important metric right now is simple:
How many hours are you using AI?
And the follow-up question that actually matters:
Is that number going up?
That’s where we start.
Not time with a chatbot window open. Time spent using AI against real problems — providing context, testing outputs, critiquing them, trying another route, deciding what to keep. We’re still in the capability-building phase, and we haven’t fully appreciated what that means yet.
My Substack - Some Assembly Required - has always been where I work things out in public. It’s where the ideas are longest, sharpest, and least edited for palatability.
The Hours Are Doing Something
Every serious hour you spend working with AI is doing more than producing whatever happens to be on the screen in front of you.
It’s building familiarity. It’s building muscle memory. It’s building habit. It’s teaching you how to ask better questions, how to provide better context, how to recognise a mediocre answer, how to push the machine further, how to abandon an approach that isn’t working and how to recognise possibilities you wouldn’t have seen before.
Usage → Productive exposure → Familiarity → Fluency → Better questions → Better outputs → Better judgement → Higher-value use → ROI
And eventually, somewhere down that chain, you get the thing everyone is desperately trying to measure:
ROI.
We keep trying to jump to the end. We should be measuring the beginning.
AI Fluency Is More Like Fitness Than Software Adoption
This is where I think we’ve made a category error. We tend to treat AI adoption like software adoption. Buy the software. Deploy the licenses. Train the employees. Measure productivity.
Done.
But AI fluency behaves much more like fitness than software adoption.
You don’t become fit because your company bought you a gym membership. You don’t become fit because you attended a two-hour seminar explaining how the equipment works. You don’t become fit because you watched someone else exercise. And you certainly don’t become fit because the CEO announced that fitness is now one of the company’s strategic priorities.
You get fit by doing the reps.
Again. And again. And again.
The same is true here. You don’t become fluent in AI by having access to AI. You become fluent by using it seriously, repeatedly, and with increasing difficulty. Some sessions are productive. Some aren’t. Sometimes you try something that fails completely. Sometimes you realize you’ve been approaching something incorrectly for months.
But every serious use gives you another chance to learn. And eventually something interesting happens. You stop thinking so much about how to use the tool. You start thinking through the tool.
As Often As I Can
I use ChatAndBuild as the foundation of my “AI Brain”. Recently, in its “Inspiration” thread on WhatsApp, someone asked me how many hours I spend using AI. My answer is:
As often as I can.
Thinking about it afterward, that might actually be the best answer I could have given. Because I don’t only use AI for the big things. I use it for small things. Sometimes ridiculously small things.
Questions I could answer myself. Connections between ideas I could probably make myself. Things I could Google. Things I could sit with for ten minutes and eventually figure out.
That’s how you get good at this. The stakes don’t always need to be high. Sometimes the value of the interaction isn’t the answer. It’s the rep.
Every time I use AI seriously, I’m learning a little more about how to use it. I’m learning how much context matters. I’m learning when to push. I’m learning when the first answer is bullshit. I’m learning when another model might be better. I’m learning when I need a prompt, when I need a system, when I need an agent, and, increasingly, when I need no AI at all.
I’m learning how to get from one idea to the next without losing the thread.
Most importantly, I’m building the habit of considering AI as part of the way I approach a problem in the first place. And those hours compound.
The Tool Starts Changing the Operator
Eventually something even more important happens. AI stops simply helping you answer questions. It starts changing the questions you can ask.
That’s the part that gets lost when we reduce AI productivity to time saved. The real transformation isn’t that AI allows you to do the old thing 30% faster. It’s that familiarity with AI changes what you imagine doing in the first place.
At first you ask it to summarise the meeting. Later you ask whether the meeting needs to exist. You ask it to write the presentation. Later you ask it to challenge the strategy behind the presentation. You ask it to speed up the workflow. Eventually you ask why the workflow exists at all.
But you don’t get there by reading about AI alone. You don’t get there because your company bought you Claude. And you don’t get there by attending an afternoon workshop. Courses and workshops can shorten the learning curve. They can give you useful models, language, and starting points. But they cannot substitute for the time spent with the technology.
You get there by using it.
Hours Aren’t Enough
Now, there is an obvious problem with my argument. You can use AI for thousands of hours and still use it badly. Someone who spends three hours a day asking ChatGPT to rewrite emails isn’t necessarily becoming an AI master.
Doing something badly for longer doesn’t magically make you good at it. Just because you have been laid a lot doesn’t mean you are good in bed.
So hours can’t be the only metric. Training matters. Experimentation matters. Difficulty matters. Reflection matters. You have to push yourself into increasingly sophisticated uses. And that’s exactly how fitness works too. Going to the gym every day and sitting on the bench looking at your phone isn’t going to make you particularly fit. The hours matter because of what you’re doing inside those hours.
What I’m really interested in is productive exposure. Real problems. Real experimentation. Real failures. Better prompts. Better instructions. Different models. New workflows.
Systems. Agents. Critiquing outputs.
Trying again.
I would not turn AI hours into a quota. That would confuse practice with productivity. The point isn’t to reward people for just spending more time with the technology. The point is to create enough repeated, serious exposure for capability to develop.
The hours aren’t the capability. The hours create the conditions in which capability develops.
So What Should Companies Measure?
This brings us back to ROI. Eventually Finance is going to want the numbers. How much faster are we? How much did we save? How much more did we produce? Did quality improve? Did revenue increase?
Those are completely legitimate questions. But they’re lagging indicators. Right now, I would be interested in a much simpler leading indicator. If I were running an organization trying to understand whether its AI transformation was actually happening, one of the first things I’d want to know is:
How many hours did our people spend working with AI last week?
Not how many licenses we bought. Not how many employees attended AI training. Not how many people logged into ChatGPT once. How many people actually spent serious time working with AI?
And then:
Is that number going up?
If usage is flat across the people expected to change how they work, I would be skeptical of any claim that a broad transformation is underway. Some local ROI may exist. But the capability has not yet spread. I’d also want to know how widely those hours were distributed. Ten power users can make the total look healthy without changing how the organization works.
The verbs matter too. “Logged in” tells you almost nothing. “Tried, critiqued, redirected, built and questioned” tells you much more about whether capability is developing.
You can have an AI strategy. You can have enterprise licenses, governance frameworks, an AI Center of Excellence, workshops, innovation days, internal champions, and a beautifully designed transformation roadmap. But if people aren’t actually spending time with the technology, none of that adds up to transformation.
They add up to infrastructure.
Training Should Change the Number
This also changes how I think about AI training.
The purpose of an AI workshop shouldn’t simply be to create trained employees. It should create active users. That means the most interesting measure of an AI training program may not be the satisfaction score at the end. It’s what happens to usage afterward.
How many hours were people using AI before the training? How many hours are they using it a month later?
Did the number go up? Did they try more difficult things? Did they use AI against real work? Did they come back with better questions? Did the behavior stick?
Because a workshop can teach you the movement. Only the reps build the muscle.
The Metric Before ROI
I’m not suggesting companies should maximize AI hours forever. That would be ridiculous. In fact, if we get good enough at this, the opposite may eventually happen.
Tasks that once required an hour of active AI collaboration may take ten minutes. Agents may perform work without continuous human involvement. AI may become so deeply embedded in our tools that separating “AI time” from “work time” becomes impossible.
That’s fine.
The metric’s meaning changes over time. In the capability-building phase, rising usage can be a useful leading indicator. Later, declining usage may be evidence of fluency, automation or better system design.
Hours are a metric for this moment.
Because right now we’re not just deploying a technology. We’re developing a capability. And capabilities require practice. So before we obsess over Return on Inference, we should ask a much simpler question:
How many hours are you using AI?
And then the one that really matters:
Is that number going up?
Because capability isn’t installed. It isn’t licensed. It isn’t announced. It is practiced. AI fluency is more like fitness than software adoption.
Do the reps.
Christopher Smith is a strategist, writer and filmmaker exploring AI, creativity and culture. His work examines how storytelling changes when the tools change, why human presence becomes more valuable in a synthetic world, and what brands and organizations continue to misunderstand about people over 55.
More at christophersmith.sg.