I hope you invest less in AI projects after the MIT report...cause I won't be
The MIT report claiming 95% of AI projects have zero return has been making waves, and I've been noodling on it. Let talk failure in innovative efforts and the ROI that doesn't show up on your P&L statement.
The $10 Million Dollar Failure at Bloomberg
Measuring AI experiments by traditional ROI metrics will likely show that you undervalue learning in your organization. No better example than Bloomberg's GPT, a custom built LLM that cost 10 Million bux & was launched on March 30, 2023 to fanfare unfortunately March 14th (2 weeks earlier) GPT 4 launched, and by October someone got around to testing the custom Bloomberg model against the publicly available GPT 4 model, and GPT 4 and most LLMs in that class beat Bloombergs.
When I invest in projects that fail I am investing in my team "skinning their knees" that is education. That is money well spent. We'll be smarter for it later.
As much as I love the trainings people take with Marketing AI Institute at Seer, last I checked I think we've had over 200 people go through it. I can see it already the same people who saw this report and were like "told ya, it's hype" are laughing at Bloomberg too, but here's my perspective: I see thousands of hours of experimentation and learning happening - I see a team of lucky employees who got to learn all kinds of intricacies around AI, making them better users of every model that comes after. That bloomberg failure is a hungry employees heaven.
People who point out hallucinations or "6 fingers" in AI conversations, learn nothing from that. I try to stay as far away from them as possible. They don't teach me shit.
Sure is it funny to show, does it get you laughs, yup but what did you LEARN?
Hire the person who sees "six fingers" and can show you all their failed prompts to try to fix it. I want the person who tests it over and over, so they'll be the first to know when it is better while others aren't using it b/c 6 months ago it had 6 fingers in an image.
The 20k waste of time, or was it?
Most of the things I built using AI have failed. I once spent 16 hours trying to get a bunch of n8n workflows done — they never worked. At my billable rate, that's $20k with zero ROI to show for it.
But that 16 hours was my springboard to hire people who knew more than me and let them build out the solutions.
That "failed" experiment unlocked learning that made me better, gave me the knowledge to ask the right questions, and gave me confidence. I learned more about n8n - what's easy, what's hard, and how to better instruct a future team building on the platform.
What is the ROI of investing in my literacy and my confidence?
If you looked on the P&L statement where does that show up? I will tell you it doesn't.
You know what rolling my sleeves up and torching 20k of my time does? It prevents me from falling as easily for AI smoke and mirrors. How many people outsource learning to others and some day make a MILLION dollar decision without knowing how to sort through the HYPE?
It's 1995 all over again
This reminds me of evaluating the internet in 1995 based on whether corporate websites were generating direct sales. Companies that spent money "failing" to figure out web strategies in the 90s were the ones positioned to dominate e-commerce later.
"I have not failed. I've just found 10,000 ways that won't work."
Leaders who want to slow down AI investment after seeing this report could be underinvesting in their team gaining knowledge and training at a pivotal time.
I want to double down on my team testing, making small mistakes sol we don't make them in the futyure when the big dollars are on the line. How many businesses have been sold snake oil in SEO? Tons, why because the company sounded good, said all the right things, etc.
I want MORE knowledgeable teammates who have gone deep and felt the pain and the scars and have the bruises from being in the arena right now, not the ones sitting in the stands, waiting for it to be easier.
Yeah, but...
The opportunity cost is real - Yes, you could invest in proven technologies with higher likelihood of return. But my fear is that gets you short-term gains while you put your head in the sand to what AI can do. I've seen this with CEO friends who run agencies I've consulted with - someone will say their margins are great on current work, but their total addressable market shrinks when clients start expecting AI solutions and they don't have them. At some point you will need to decide how important those margins are for the next few years.
There's no user manual for ChatGPT or Claude. Without an instruction manual on how these things are supposed to work, we're all figuring it out through experimentation. The pace of change is so fast that new models come out constantly. You can wait for adoption when everybody's adopted it or you can learn and lead.
The companies running AI pilots now are building institutional knowledge. The ones cutting budgets after this report are ensuring they'll be dependent on expensive consultants or poaching talent from competitors who invested in learning. Just look at Meta - they were a bit late to the AI game, LLama fell way short of expectations (for whatever reason), and now Mark Zuckerberg is paying literal "athlete-level" salaries to play catch up. Reports of $200 million and $300 million compensation packages are now a thing? Yup.
Now imagine your business: do you want to lose money in your experimentation and R&D while building the talent in-house, or wait until the dust settles and have to pull your version of a Zuck? Will you even have the budget to do that? This is your call, but something to think on - the cost of failed experiments against the cost of a poorly trained staff and the cost of having to pay through the nose to get the people you need?
Is Underinvesting in AI Choosing a Slow Death?
Here's what really drives my thinking: if your company's okay dying a slow death, you could be right and it's more hype than substance. I worked at Amazon in 1998-99 and all I remember is being there in warehouses, being like this is the future and my parents being like... you sure people are going to put their credit cards in... on the internet?
Sears had over 3,500 stores and 355,000 employees in 2006. By 2018, they filed for bankruptcy. Bed Bath & Beyond lagged behind competitors like Target, Amazon, and Walmart in building e-commerce presence and couldn't adapt fast enough. Barrons ran this story...see how that turned out.
I would rather overinvest in making mistakes than wake up one day and be the sears execs reading Barrons articles saying "see I told ya". Or as I told my exec team this year, we're going out swinging, not cowering.
Seer in an AI world with all this naysaying is becoming a close your eyes, vaseline your face, and start throwing them hands kinda agency.
A lot of wasteful money was spent in the early ecommerce days, I had clients like pets.com and webvan.com, they flamed out, so there are learnings there and there are failures, but with these AI projects, I see most of them not as companies betting the farm, but running tests and experimenting with pilots, that is a good investment in your people and institutional learning.
Here's the real costly pain of being undereducated with AI testing (using a different example).
Let's keep it 100. I am going through a database migration right now, and I know so little that I have had to spend HOURS of my time, and my team's time and thankfully people in my network's time trying to figure out the right direction, all that time is also opportunity cost, time wasted, etc all because I never went through the "knee skinning" moments.
Investing in AI failures now is investing in your institutional knowledge
But here's what this whole debate misses - this isn't really about the back and forth on ROI calculations or failure rates.
If you're winding down AI projects because many are going to fail, you bought into those projects for the wrong reason. You actually thought you were going to get an ROI you could calculate immediately. What is the ROI of having people ready & educated for what is coming, with hands on knowledge? I'll wait. :)
At some point, you're going to need people in your company who understand how to deploy AI, how to know what's real versus fake, what's hype versus reality. You don't learn that in trainings - you learn it by wrestling with prompt engineering, dealing with hallucinations, figuring out integration challenges, running the same dang prompt in every new model 10x to understand what was once too expensive can now be done on a local machine with an NVIDIA RTX 4090 graphics card to handle the AI load, and developing the judgment to spot when AI is confidently wrong.
The companies spending money on "failed" AI pilots today are building the institutional knowledge that will be invaluable tomorrow. The ones waiting for guaranteed ROI are ensuring they'll be buying that knowledge from consultants at a premium prices.
.png)
