You open YouTube to watch one video about how to fix a dripping tap. An hour later you are watching someone restore a 1970s radio, with no clear memory of how you got there. You linger for a second on a friend’s holiday photos, and the next day your feed is full of beach resorts. You search for running shoes once, and they follow you around the internet for a week.
Of course, none of this is coincidence. Every time a recommendation system surfaces a video, a post, or a piece of news, it is making a small bet about what you will do next. Behind that bet sits an AI model, trained on the behaviour of billions of people, including yours. If it is right, it learns. If it is wrong, it also learns. Every pause, scroll and skip counts as signal, and so does the video you watched to the end. Across billions of these moments every day, the system is continuously updated by the behaviour it is simultaneously shaping. You are training it and, in turn, it is training you. This is a clear example of a feedback loop.
Cybernetics has been describing systems like this since the 1940s. What it offers is a precise vocabulary for understanding why they behave the way they do, and what would need to change for them to behave differently.
The ecosystem around the algorithm
Before we look at the machine itself, it helps to map the world it lives in. A recommendation algorithm never works alone. It sits inside a network of people and organisations whose interests sometimes line up and sometimes pull against each other.
Laid out like this, the network shows something that is hard to see in prose. Regulators and civil society do have influence, but every path they have to the algorithm runs through the platform, and the platform listens most closely to its advertisers. Here Ashby’s Law of Requisite Variety earns its keep. A regulator can only control what it can model, and the algorithm changes faster than any regulator’s model of it.
The feedback architecture
Inside the ecosystem, two amplifying loops drive the system’s behaviour, and neither is matched by a stabilising loop of comparable strength.
The first is the engagement loop. Your behaviour generates signal, the signal trains the model, the model refines its recommendations, and the recommendations shape your behaviour. The second is the supply loop. Recommendations create demand for a certain kind of content, creators make more of it, you meet it more often, and your behaviour drifts further in that direction.
What is largely missing is what cybernetics calls a stabilising loop: a feedback path that damps the swings and pulls the system back towards balance. Platforms do moderate and demote content, but those measures act on the outputs after the fact. They leave the optimisation target itself untouched. So the system is working exactly as it was designed to work, and the real question is whether the design objective is a good one.
Stafford Beer liked to say that the purpose of a system is what it does. These systems describe their purpose as connecting people with content they love. What they actually do is maximise the time you spend on the platform, as measured by engagement signals. Those two aims overlap, but they are far from identical.
The technical control architecture
To see why intervening is so hard, we need to look at how the system is built. A recommendation pipeline has many layers, and the critical control path runs through components that work at machine speed, far below the level where a human editor could step in.
The diagram makes plain something that platform transparency reports tend to blur. The safety filter sits between the ranker and the user interface, which means it catches some outputs after they have been scored. It has no say over what the model learns to value. That job belongs to the label store, the implicit record of what counts as a good outcome, which sits at the very base of the system, upstream of everything else. Whoever decides what goes into the label store decides the system’s values. That makes it a question of governance, though it is rarely discussed as one.
The adversarial surface
A system built this way, learning continuously from behaviour and optimising for engagement, has weak points that are easy to predict. Anyone who can generate coordinated, inauthentic engagement at scale can, in principle, bend what the system learns.
This diagram rewards a slow reading, so let us take it one piece at a time.
On the left, inside the dashed red line, is the attacker. We call it a coordinated actor because it is a single organisation, though it is designed to look, from the platform’s side, like thousands of unrelated people. On the right is the platform’s defence, which you can think of as its immune system. Each red arrow is a line of attack, and the two solid red nodes are the components the attack manages to turn.
The account farm and the content classifier. The account farm is a large stock of fake or hijacked accounts, ready to post on command. Its target is the content classifier, the automated filter that inspects each post and decides whether it breaks the rules. Classifiers are good at clear cases and much less sure at the edges, so the farm posts large quantities of borderline material, each piece slightly different from the last. Some of it is caught. Much of it slips through, and by the time the classifier, or the people behind it, catch up, the content has already travelled.
The amplifier network and the ranker. The amplifier network is a second set of accounts whose only job is to react: to like, share, comment and watch to the end. Its target is the ranker, the component we met earlier that decides what appears in your feed. Remember that the ranker reads engagement as a sign of value. Fed a stream of manufactured enthusiasm, it does exactly what it was built to do and promotes the content to real people. The ranker has simply been lied to, and it believed what it was told.
The narrative hub and the users. The narrative hub is the editorial centre of the operation, where the message itself is crafted: the slogans, the images, the talking points. Its arrow points at the users, because they are the true target, and everything else is a means of reaching them. Once real people begin sharing the content of their own accord, the operation’s work is largely done. From that point the engagement is genuine, and no detector can tell it apart from the real thing, because it is the real thing.
The operator and the appeals process. The operator runs the whole campaign. Its arrow goes to the appeals process, the route by which users contest removals. When content is taken down, the operator files appeal after appeal. Each one must be reviewed, the moderation queue grows, and while it grows, removals are delayed and the content keeps gaining reach. The appeals process exists for a good reason, to protect honest users from mistakes, and here that very fairness is turned into a tool for wearing the system down.
The two quiet defences. Finally, notice the two nodes at the bottom right with no arrows pointing at them. The behaviour detector looks for accounts acting in suspiciously similar ways, and graph integrity looks for unnatural clusters in the web of who follows and interacts with whom. These are the defences best placed to spot coordination, so a well-run operation works hard to stay beneath them, staggering its timing, varying its behaviour and spreading its activity across many loosely connected accounts. Their quiet corner of the diagram is the attacker’s real achievement: the components that could see the pattern find nothing unusual to look at.
Step back, and you can see what makes this a cybernetic attack. Every component keeps working exactly as intended. What has been corrupted is the information they rely on: the posts the classifier inspects, the signals the ranker trusts, the queue the moderators work through. Shannon and Wiener both treated noise as the great enemy of communication, something that creeps into a channel by accident. Here the noise is put there on purpose and shaped to look like signal, which makes it far harder to filter out. The weakness is as old as feedback itself: a system that steers by what it senses can be steered by whoever controls what it senses.
The same loop, now in conversation
Everything we have described so far applies to the newest generation of AI models too. AI agents, and the chat assistants many of us now use every day, are built on models tuned using human feedback: people rate their answers, and the model learns to give answers that earn high ratings. Here we meet the problem of the proxy once more. A rating measures whether an answer pleased someone, which is close to, though never quite the same as, whether it was true or helpful. Left unchecked, a system trained this way drifts towards telling us what we like to hear, much as a feed drifts towards what keeps us scrolling.
So who are the people doing the rating? Many of them are freelancers: large numbers of contractors around the world, some with specialist expertise, working through online platforms and paid by the task. They compare pairs of answers, rank them, and write better ones, following lengthy guidelines drafted by the AI companies. It is careful, demanding work, and a good deal of a model’s character is quietly shaped by it. Notice, though, where these people sit in the loop. The rater is the system’s sensor, and a sensor working against the clock will favour what can be judged quickly: answers that sound confident, read smoothly and agree with the person asking. Their judgement, made under those conditions, becomes the proxy, and the model learns to please it.
And the loop keeps turning. Text written with AI flows back onto the web, where it becomes training data for the next generation of models. The feed taught us to notice what we click; the assistant may be teaching us how to write and think. Wiener would have recognised the pattern at once.
What would a well-regulated system look like?
A cybernetic reading will not hand us a ready-made answer, but it does help us ask the right question. Feedback itself is fine: every adaptive system relies on it. The trouble lies in what the loop is closed around. Here it is a single variable, engagement, and engagement is a poor stand-in for the things we actually care about, such as wellbeing, informed public debate and a healthy range of views.
Imagine instead a system regulated at the level of its optimisation target. It would have several feedback signals competing with one another, some measuring engagement and some measuring other values, and a governance layer deciding how much weight each one gets. That governance layer would have to be open to inspection, open to challenge and accountable, in a way that a private product roadmap never is.
This is simply what requisite variety asks of us, described in engineering terms. The system we are trying to regulate is extraordinarily complex, and a regulatory apparatus to match it does not yet exist.
Building one is among the great systems challenges of our time.