Social media was built to connect people. The early platforms were genuine in that intent: stay in touch with friends, share moments, find your people across distance. Facebook started out as a networking tool for US college campuses, and grew from there.
Then somewhere along the line, something shifted.
After social media’s glory days, the platforms stopped being about real connection and became something else: platforms of user-retention. If there’s one AI governance lesson worth learning before AI repeats the pattern, it’s this: We need to stay vigilant; the shift didn’t happen through a single decision anyone could point to and stop.
How the Shift Happened Without Anyone Noticing
The shift happened in a way it always does with commercial platforms chasing growth: an algorithm change here, a product update there, together with a long series of A/B tests, until eventually the original purpose of social media got so buried that most users couldn’t tell you when it disappeared.
The transition from “connecting you with people you know” to “connecting you with content that keeps you on the platform” was remarkably seamless. That seamlessness was not accidental.
Social media’s flywheel is built on the most powerful engine in human psychology: emotion.
- Outrage drives shares
- Validation drives return visits
- Anxiety drives compulsive checking
- Comparison drives aspiration and insecurity in equal measure.
These were not bugs in the system. They were levers that were identified, refined, and deployed at scale, orchestrated with talented engineers working alongside best-in-field behavioural psychologists.
Yet many of us hardly took notice, or at least hardly took issue with the changes, because people “liked” what they were seeing, even as the platforms changed underneath them.
All the while, platforms shipped updates to help fulfill their commercial objectives without ever publishing a guide for users on responsible social media use. Education was not part of the product, because education was not good for their business.
The Regulatory Response We Got
Australia moved in 2025 with a social media ban for users under 16.
Whether the ban is the right mechanism is a separate debate — I still believe that education around responsible social media use is more sensible than a national wide, government enforced, blanket ban — but in this instance, time for debate had passed before most people even got the chance to have it.
Nevertheless, the intent behind this ban shows something: the harmful effects of social media are now considered sufficiently established that a government felt compelled to legislate it. Other countries have since followed Australia’s stance.
AI at the Same Inflection Point
Now we’re watching AI’s deployment at scale, and the question worth asking is whether we’ve absorbed any AI governance lessons from the social media era, or whether we’re sleepwalking into the same pattern at a far wider scale.
The signs are already visible. AI apps start out as genuinely useful tools; AI companion apps then optimise for emotional engagement, with interaction design that rewards continued use over genuine benefit. The early warning signs are not hidden: they are present, documented, and largely unremarked upon in mainstream conversation.
The difference between AI and social media isn’t that AI is more dangerous in kind. It’s that the surface area of potential harm is significantly wider.
The Governance Challenges Ahead
Job displacement and the decoupling of human labour from enterprise value
As I have covered in my job displacement article, Goldman Sachs estimates AI substitution is now a net drag of roughly 11,000–16,000 jobs a month on US payroll growth, concentrated in entry-level and early-career roles. These changes are something that the workforce notices first before reaching the headlines.
At the same time, many businesses that are starting out are hiring people much later than what was required in the past, because AI is now doing the heavy lifting for tasks that businesses previously had to hire for. In a sense, this is progress from a commercial and opportunity stand point, and as AI consultants and transformation partners, something that we would like to see taken up in a responsible way.
But there is also a concern that cannot be ignored: that lower employment leads to less customers long term. While the immediate benefits of AI automation are more immediate for businesses, at a societal scale and eventually, businesses will face less growth because there will eventually be less people that can afford to buy.
The decoupling of humans from companionship and connection
When AI can simulate empathy and positive regard on demand, without reciprocal effort, what happens to the tolerance and skill genuine human connection requires? Companion apps are already being framed as mental health tools.
We didn’t ask this question soon enough with social media, where users kept their networks at arm’s length, while avoiding real contact.
With AI companionship apps, where users can develop interactive and seemingly deep connections with essentially an algorithm, the possible consequences will be felt even more. Fortunately, China has noticed and enacted policy around AI companionship apps. The questions is whether this will be brushed off as China being China, or see other countries take up a similar policy position.
Cognitive and skill atrophy from over-reliance on AI
Less discussed than job losses is what happens to human capability when AI handles the reasoning and problem-solving that used to build it.
- A junior lawyer who has never had to construct a legal argument from first principles — because AI drafts it — is not developing the judgment that makes a senior lawyer valuable.
- A student who outsources their essay to a language model is not building the capacity to think through a complex problem under pressure.
This isn’t just an emotional or social cost: it’s cognitive, and rectification is not easy once the AI tool is wrong in ways the user can’t detect.
The narrowing of human agency and freedom of choice
As AI increasingly mediates decisions — who gets a loan, what medical pathway is recommended, who drives — the space for individual discretion narrows. Self-driving cars are the clearest emerging case: the shift away from manual driving is unlikely to arrive as a ban. It’s more likely to arrive through insurance premiums and infrastructure that make manual driving impractical long before it’s prohibited. That pattern — efficiency framed as objectively correct, and discretion quietly priced out — isn’t limited to transport.
This erosion of human agency and discretion leads to implications on our basic human freedoms and autonomy, and can also lead down more darker lanes. What if one day, self-driving cars are mandated to have overrides to support law enforcement? When this gets introduced, this will potentially be dressed up by our politicians as “protecting our communities” — because on the surface that’s true, though not the whole story — but creates even more avenues for government and large institutions to tighten the screws on individual freedoms.
The automation of cybercrime
AI has lowered the cost and raised the sophistication of malicious activity.
- Phishing campaigns that previously required human effort to craft convincingly can now be generated and personalised at scale.
- Vulnerability scanning, social engineering, and intrusion attempts are being automated in ways that outpace the defensive tools most organisations have in place.
Smaller organisations without dedicated security resources are disproportionately exposed.
This was chiefly why Anthropic conceived Project Glasswing: buying time for defensive patching to catch up with AI’s ability to find and exploit weaknesses, with the long-term aim of using those same capabilities to fix weaknesses before software releases.
Deepfakes, impersonation, and the weaponisation of identity
Identity fraud is nothing new, but AI has made it significantly cheaper and more convincing. Synthetic video and audio of real people can now be generated with modest resources and deployed for fraud, manipulation, or blackmail.
Sextortion targeting mainly young men has risen sharply, with AI used to build convincing fake personas. Deepfakes of the actual target can still be constructed off an innocent enough conversation, even when the scam doesn’t land.
Legal frameworks haven’t caught up, and unlike most fraud, the damage from a convincing impersonation can be irreversible.
The Education Gap, Repeated
The same gap that defined the social media era — between the pace of deployment and the pace of public understanding — is present in AI adoption. Platforms and developers are moving fast during this AI gold rush. The public education that should accompany that pace isn’t keeping up.
Are we educating people well enough on what it means:
- when people implicitly trust AI output?
- when a business automates a hiring decision?
- when a model is trained on data reflecting historical bias?
- to depend on a system whose incentives may not align with your interests?
The pace of AI innovation isn’t the problem by itself — innovation creates real value in productivity, access, and the democratisation of capability. The problem is innovation running ahead of accountability, transparency, and the public understanding that makes informed consent possible.
The AI Governance Lessons That Still Need Answers
The social media story did not need to unfold the way it did. The choices that led to the harm were made by people, in boardrooms, one feature at a time. They were also allowed by the rest of us: regulators, institutions, and everyday users who did not yet have the acute awareness, the language or the frameworks to push back soon enough.
Responsible AI governance requires defining, before the paradigm shifts in ways we can’t walk back, what role AI should play in society — not just at the product level or the enterprise level, but at the philosophical, moral, and ethical level. What should AI never be optimised to do? What safeguards should be non-negotiable? What does it mean to adopt AI responsibly, not just in a firm, but as a society?
These are the questions I want to see get serious, sustained attention — from organisations, from policymakers, and from the broader public — much sooner than we let social media go unfettered.
In future posts, I’ll go deeper into what responsible AI adoption looks like for each of the challenges raised here, and more.
Stay tuned.
