What Happens When the Machines Get Smarter, But Our Organizations Don't?

You ever have a moment that forms your thinking for the rest of your life?

For me, it was nine-year-old me sitting in a movie theater with my childhood best friend, Jared Cooper, watching Terminator 2—and with his mom, because it was rated R.

IMHO, one of the few sequels to actually outdo the original.

The scene I remember most is Sarah Connor's nightmare about the nuclear apocalypse. She's standing behind a chain-link fence, screaming at people who are completely oblivious to what's coming. Kids are playing. Parents are going about their lives. Nobody knows that in a matter of seconds, everything is about to be gone.

Then the blast hits.

People are burned to ash. Buildings disappear. The world as we know it is destroyed.

But weirdly, it's the sound of that chain-link fence that stuck with me.

Sarah rattling it. Pounding on it. Desperately trying to warn a bunch of blissfully unaware people that something terrible is coming.

Thirty-five years later, I can still hear it.

The culprit, of course, was Skynet.

Skynet became self-aware.

A computer system designed to help humanity got smart enough to decide humanity was actually the problem. Which, to be fair, after spending five minutes on Facebook, you can at least understand how it arrived at the conclusion.

But in 1991, the whole premise seemed ridiculous.

Computers becoming sentient? Machines thinking for themselves? Artificial intelligence becoming smarter than humans and eventually turning against us?

Pure science fiction.

My nine-year-old brain couldn't begin to comprehend the technological advancements I would see in my lifetime.

And yet, here we are.

Artificial intelligence can write our emails, create our images, analyze our businesses, write software, make videos, imitate voices, develop strategies and carry on a conversation that can occasionally make you forget there isn't actually a person on the other side.

And we're just getting started.

As Elon Musk once put it when talking about our approach to artificial intelligence:

“We're like children in a playground… We're not paying attention.”

— Elon Musk

Which is reassuring.

So, what are the actual odds that AI becomes self-aware and kills us all?

Well. Not likely. But also not zero. And this is where things get a little uncomfortable.

A recent study asked 582 AI researchers about the possibility of AI developing what researchers call “subjective experience”—essentially, some form of conscious experience.

The median estimate? A 25% chance by 2034. By 2100, that number jumps to 70%.

Now, before you start digging a bunker and stockpiling canned goods, sentient AI does not automatically mean murderous AI. That’s a critical distinction. Becoming self-aware and deciding to exterminate humanity are two very different things.

So researchers have been asked about that too.

In the latest large survey of AI researchers, the average estimate that future advances in AI could cause human extinction—or similarly permanent and severe human disempowerment—was 18%. Eighteen percent. For perspective, that's roughly the same odds as getting food poisoning sometime this year. So apparently AI ending civilization and gas-station sushi ruining your weekend are now living in roughly the same statistical neighborhood. I'm not a statistician, but that feels considerably higher than the number I would prefer.

But here's the weird part: while we're all worried about AI destroying humanity, business owners are looking at the exact same technology and seeing dollar signs.

The buzzword of the day is automation, and I can hear Oprah's voice chanting:

“YOU get a bot! YOU get a bot! EVERYONE gets a bot!”

Suddenly, CEOs and middle management everywhere are salivating over the idea that they can have fewer employees. Fewer employees means lower payroll. Lower payroll means more profit.

More output.

More leads.

More content.

More customers.

More. More. More!

And THEN we can maximize shareholder value! Remember as a kid when all you wanted was to one day grow up and maximize shareholder value? Me too.

And I get it. The possibilities are intoxicating. Elon Musk has predicted a future of “universal high income” where AI and robotics create an abundance of goods and services. If you're running a business, it's hard not to hear that and start doing the math.

But I worry that somewhere in all that possibility, business owners' eyes are starting to haze over green. We see the dollar signs, the efficiency, the speed and the scale—and we don't stop long enough to ask what exactly we're optimizing.

Because AI doesn't just have the ability to optimize our businesses.

It has the ability to optimize our devastation.

Let me paint this picture for you.

Frank has been in the pizza business for 52 years. Same shop. Same recipe. But lately, declining profits. Costs are high and the margins are thin.

Suddenly he's competing with some Michelin-star chef down the street who uses fermented sourdough for the crust, hand-crushes his own tomatoes and somehow gets people to pay $40 for a large pie.

So here's Frank, tucked away in the back office, clanking on the keyboard, cranking out his new marketing strategy courtesy of your friendly neighborhood AI agent.

AI generates the menu.

AI writes the descriptions.

AI creates the food photography.

AI builds the social posts.

AI creates the ads.

And Frank is thrilled.

“Do you know how much money I just saved?”

There's only one problem.

The pizza looks like a Teenage Mutant Ninja Turtle's fever dream.

The food descriptions sound like they were written by someone who's never actually eaten food.

And the smiling chef on the menu has six fingers.

But hey.

He optimized!

Except Frank's problem was never his marketing.

Frank saw declining profits and assumed he had a marketing problem. So he found the fastest, cheapest way to make more marketing. But nobody stopped to ask why his business was declining in the first place.

His service is terrible. Customers wait forever. His staff is frustrated. Orders come out wrong. Nobody answers the phone. And Frank's been doing things the same way for 52 years because, well, that's the way Frank has always done them.

AI didn't solve Frank's problem. It just helped more people find out about it.

Frank didn't optimize his business. He optimized his devastation.

You see, AI doesn't actually fix what it touches.

It amplifies it.

It multiplies it.

For better or worse.

This means a great process can become even faster—and a broken process can break faster. Great customer service is scalable, and a terrible customer experience can be scaled just as fast. Suddenly, strategy becomes activity and proliferation replaces prioritization.

But before you automate a process, ask whether the process deserves to survive.

Don't use AI to make the wrong things faster.

You see, the question isn't just, “What can AI do?” It's, “What problem are we actually trying to solve?”

Maybe nine-year-old me was worried about the wrong thing. I'm still not pumped about the possibility of computer-induced Armageddon. But the threat I'm most worried about today is that people and organizations have been handed extraordinarily powerful technology—and haven't stopped long enough to figure out what problem they're actually trying to solve.

Maybe Skynet isn't the biggest threat after all.

Maybe it's what happens when the machines get smarter, but our organizations don't.