TL;DR
- I reviewed our traffic-decline-analysis skill's and found four real problems: a traffic number from the wrong source, a real drop mislabeled "stable," no way to trace where a number came from, and no long-term trend view
- I gave that feedback in a 9-minute recording and an agent picked up my feedback and fixed it the same day, and it shipped to everyone at Seer.
- I found another problem the skill was picking which checks to run instead of running all of them, every time
- An offhand idea from that same review became a new feature days later. The tool now runs off a simple spreadsheet export instead of needing live client data
Here’s how it all happened…
What I mean by “skill”...
When we use the word, “skill” at Seer, we’re referring to written instructions that tell our AI exactly how to do one specific piece of the job, the way you'd write up a process for a new hire.
We've got one skill for client's traffic drops, one for competitive research, dozens of these across the company, and more getting built every week.
The important part is where our skills live (Github).
The ones I’ve created have one shared library everyone at Seer pulls from. This means when somebody makes a skill better, it doesn't just get better for them.
We ran our traffic decline analysis skill
Our traffic decline analysis skill looks at why a client’s numbers are down and runs through all the possible reasons why. All you have to do to build this yourself is point it at a client’s traffic history, tell it to explain why the numbers moved instead of just reporting what they did.
Ours breaks down the decline into phases, so that you can see if things are getting worse, holding or leveling off.
It also checks every other channel to see what might be offsetting the loss. And it checks whether the traffic that disappears was ever even converting in the first place.
I ran our traffic decline analysis skill against our site, here is what happened...
It got a lot right. Broke the decline into three phases instead of one flat number.
While reviewing it, I was thinking: I love this, the three phases it gave. You can see below, how in Github, my transcript got parsed into things we can improve in the skill.
Then it went looking for where else we were growing to offset the loss: direct traffic had doubled, referral had tripled, email was up.
I’m thinking, yeah, here we motherfucking go. I love the offsets.
Reading more I thought: The silver lining, conversions are up... the traffic you're losing wasn't converting anyway. Love it…
I spent 9 minutes on feedback (an eval) Found four problems, the agent fixed them
Here’s what I found wrong with the output:
The traffic decline analysis skill pulled a number that made it look like our social traffic had freaking vanished. It dropped from around 2,000 visits a month to around 150, practically nothing. A mistake was made.
The skill found a sharp real drop in the channel and it literally said:
“Stable to me, that’s way too broad of a range.”
If I had 1,300 last month and 2,600 the next month, that’s not stable, that’s growing. SO that also seems like something I would correct.
See below how we use Claude to create structured github issues, so when any employee does an eval our agent knows how to structure their ramblings into a set of issues to potentially fix.
So the next step was the fix the gaps
Step 1: recorded my feedback on the skill’s output,
Step 2: acted on instantly and
Step 3: scaled to the company instantly
Jordan took my transcript and pointed an agent at it with one instruction:
"Please review this issue, run all necessary skills to implement changes, and write a reply and PR for Wil and I to review. Go."
That was all it took to hand it off.
The agent worked the issue itself, start to finish, and came back with a pull request and a reply before either of us touched a keyboard.
Jordan pointed and the agent moved. We still decided what shipped, and every note became a real change. Humans still in the loop.
Each note that I'd flagged for the skill mapped to a real, specific change immediately updated for the whole company.
The table's the clean version. Once someone approved the fix, it took the best of the feedback and automatically updated the skill for everyone in the company.
Jordan even confirmed that this happens almost immediately: Git commits sync to our Claude plugins company-wide.
This is what change management looks like in an company trying to push AI, IMHO
Every fix linked back to something I said, reviewed and merged by a human before it went live. And the moment it merged, it wasn't just my copy of the skill that got better, as soon as the change is committed to Github, it propagates to Claude for the whole team to use, improving it for every client in the company, same day. My brain + 9 minutes of feedback, scaled to every team member and every client in a day. No trainings, no scheduling, just better outputs.
Round #2: I tested our skill again with a client’s data, and most of it held up.
I ran the improved skill again, against a different account's traffic this time, not seerinteractive.com, to see if the fixes held up on data it hadn't seen before.
This is the work in an AI world, evals baby. Over and over.
Mostly, they did. The phased framing survived, and so did the rolling 12-month view we'd just added, the AI Overview cross-reference, and the root-cause attribution.
It even correctly caught something new: December and January showing up as seasonal dips instead of real decline, and it held up the Twyman's Law check (we'd added the rule that says any number that looks too good or too bad to be true probably is, so verify it before you trust it).
But round two surfaced something round one never could have. The skill wasn't running the same checks every time. It was deciding, case by case, which ones applied, so most of the time it got it right, and every so often, it quietly skipped one.
That's a different kind of bug than a wrong number.
A wrong number is embarrassing. A model that decides for itself which checks matter is a trust problem waiting to compound.
We found a bigger problem.
So the fix here was a change to how the skill operates entirely. A simple change in wording wouldn’t cut it.
Before, the skill decided case-by-case which checks applied to a given account.
Sometimes it ran the Twyman's Law gate flagging any number that looks too clean to be true.
Sometimes it didn't.
Sometimes it cross-referenced AI Overview data before drawing a conclusion.
Sometimes it skipped straight to one.
That’s no good.
If a check can't run because data's missing, it says so explicitly instead of moving on quietly. And it only reports the checks it actually completed and never one it skipped.
We couldn't continue to run with a tool that implied things that it wasn't doing and we need to close that gap.
Could this help us in BD?
How often do I sit across from a prospect and ask, "is X metric up or down?" without either of us having a clean way to actually check quickly?
A few days later, there it was. The same analysis now runs from a simple export a prospect sends over with no live client data required.
It now works for:
- Active clients with live data
- Prospects who send over a traffic export
- Quick, on-the-spot reads in the middle of a pitch
The infrastructure built to allow everyone to have instant access to this update is what's really cool.
This same loop applies to any skill anyone at Seer builds, and any feedback anyone has on anything one of those skills produces.
Jordan said, "we push that skill out to the whole team through our skill plugins." The next person who runs that skill gets the upgrade, whether or not they were the one who gave the feedback.
Everything in this post so far happened in one thread, in public, in one repo.
This was an issue, a merged PR, a second review that found a deeper pattern, another merged PR, a bonus capability that shipped through the exact same mechanism a few days later.
But the loop also connects innovators at Seer who never talked to each other
Jordan told me about a skill Marianna had built to surface organic growth opportunities.
It had a gap and she wanted a way to help identify upsell angles, but that part wasn't built yet.
So Jordan dropped the skill Marianna built into GitHub as an issue and pointed an agent at it.
While it worked, it looked at the rest of our codebase and found an sales skill I'd built for my own use, sitting in my personal plugin, that I'd never shown Marianna and she'd never asked me about.
Fixing this only took 4 days, all done in one slack thread.
Look at the actual dates: Issue #9 opened on May 11th1, and three days after that, an idea from the same review became a real feature, merged as PR #32 on May 15th2.
What's awesome is how little time now sits between someone noticing something could be better than it was, and everyone downstream of that skill benefiting from it being fixed.
.png)
