Cybernetics and Self Improving systems.

To say it’s an interesting time to work in and study technology is a gross understatement. We seem to be locked into an ever-tightening spiral of innovation and investment, where the “wow” moments now occur almost every week. If I were slightly more cynical, I’d suggest that this has more to do with the industry built around innovation, but the logic behind that cynicism seems to be about as grounded as autumn leaves on a windy day.

So, let’s challenge ourselves to slow down, to stop for a moment and to look at the latest technologies through the lens of more than 80 years of mathematics, computer science and engineering. This short (okay, relatively) post is about a man, a catalyst and a concept called "Cybernetics" that reached far beyond technology. Cybernetics influenced neuroscience and philosophy, and later found its way into ecological thinking, you could even argue that it is one of the foundations of modern ecology. Primarily, and for me relevantly, cybernetics also helped change how we think about the relationship between people and machines.

First the person behind all of this: Norbert Wiener was born in 1894. A child prodigy, he was largely taught at home by his father before attending high school. While still a school, he wrote an essay called The Theory of Ignorance, questioning the limits of human knowledge. He went on to collect his BA in maths from Tufts at the ripe old age of 14, began studying zoology at Harvard, then changed direction and studied philosophy at Cornell before returning to Harvard. By 1913, at 18 (Gah!), he had completed a PhD in philosophy, with a dissertation on mathematical logic.

Listing his academic accomplishments might seem a little cumbersome for a short essay on AI, but it all makes much more sense later on. (the combination of mathematics, philosophy and an interest in living systems links directly to the foundations of AI and where we are headed - I think). Wiener was a genius and a humanitarian who was comfortable asking questions that crossed the boundaries between subjects, and the problems he would tackle refused to stay neatly inside any one of them.

His early career bounced between universities and a varied collection of jobs. There was an engineering apprenticeship, some writing for an encyclopaedia and work on artillery calculations during the First World War. But by 1919 he had joined MIT, where he remained for the rest of his career and it was here that he (amongst the other giants of computer science and AI) felt like the challenge of an ever accelerating world and warfare could best be met with information. He was already an established mathematician before the Second World War erupted. But ironically (for a devout pacifist), the war opened the most famous chapter in his career.

That chapter began in some of the darkest times in world history. In 1940, with Britain under aerial attack, Wiener offered his services to the American war research effort. Working with the engineer Julian Bigelow, he investigated a predictor for anti-aircraft fire: a system that could use an aircraft’s recent movements to estimate where it would be when a shell arrived and therefore increase the accuracy of the guns trying to protect England.

This was not going to be easy. In normal, everyday language, the problem statement:

Shooting down a fast-moving plane with a gun from the ground is intensely difficult. Even if the gun has a motor to help position it, there is always a delay between seeing the target, aiming and getting a shell to that right place. If you break the problem into smaller component parts, the challenges start to look almost insurmountable.

Let’s skip the mechanical ones for a moment and just zoom in on these:

1. Is that even a plane you're shooting at? First you have to detect an aircraft, then establish whether it is a target. Radar helped locate and track aircraft, but a blob on a radar return did not make life easier for anyone. Detection, identification and prediction were separate jobs and questionable in their accuracy.
2. What do pilots do when you shoot at them ? Regardless of which side of the war they are on, a pilot is fairly motivated to avoid being shot down. They can turn, climb, descend or change speed. Sure, the aircraft’s physical limits constrain those choices, but the gunner cannot know the next one in advance.
3. Where will the plane be when the shell gets there? Even once you have identified the target, aiming at the position you can see is already too late. You need a prediction based on the recent movement of the aircraft and the behaviour of the person controlling it.

This is where Wiener’s interest in mathematics and behaviour came together. He and Bigelow treated the pilot and the plane as a coupled system, a loop. The pilot received information about the aircraft’s movement, acted on the controls and received new information about the result. Meanwhile, the gunner and the targeting apparatus had their own delays and corrections. Each side was acting in a paradigm that kept changing.

But what information would actually help the gunner? Wiener was part of a wider network of people trying to understand information, computation and nervous systems. Claude Shannon, Warren McCulloch and Walter Pitts belong in that story. Their work overlapped in important ways, and it is worth slowing down here because the word information can be surprisingly slippery.

Shannon’s work asked how messages could be represented and reliably transmitted. His information theory gave a mathematical measure of uncertainty, rather than a measure of how meaningful or useful a message was. Predictable repetition could be represented more compactly. Lossless compression reduced the space needed to store a message while keeping its content recoverable.

A short equation helps:

I(x) = -log₂ P(x)

Here, P(x) is the probability assigned to an event before it happens, and I(x) is its statistical “surprise”, measured in bits. An event assigned a probability of one half gives one bit; an event assigned a probability of one eighth gives three. Unexpected events carry more surprise under that probability model. That does not automatically make them more useful.

Wiener shared this basic intuition. In The Human Use of Human Beings, he wrote that “the more probable the message, the less information it gives.” His prediction problem required observations that helped estimate what would happen next. A larger pile of data was no guarantee of a better estimate. Shannon and Wiener were approaching different parts of a related problem; Shannon explicitly acknowledged Wiener’s influence on his work.

For the aircraft, the question was how to turn observations of the past into a useful estimate of the future. A simplified, finite-history version of Wiener’s linear prediction idea looks like this:¹⁰

x̂(t + h) = m∑j = 0a_j x(t - jΔt)

Before the symbols become a reason to close the page, the idea is quite manageable. Take a series of past measurements, multiply each by a weight and add them together to estimate a future value. Here, x could stand for one coordinate of the aircraft’s position, h is how far ahead we want to predict, and the weights are chosen to reduce average squared prediction error. Some parts of the recent history help more than others. A prediction can be the best available estimate and still be wrong.

In a very simplified algorithm, the process would look like this:

repeat:
    read the latest aircraft position
    add it to the recent movement history
    predict a future position from that history
    calculate the desired gun direction
    measure the actual gun direction and adjust it
    compare later observations with the earlier prediction

Disclaimer: This is obviously just a sketch of the prediction loop Wiener was facing. Fresh observations can change the next aiming instruction. However: Once an unguided shell has left the gun, it cannot receive more information, or a correction. The delay remains, however clever the prediction becomes.

Wiener and Bigelow investigated how human tracking behaviour could be modelled statistically. Laboratory tests suggested that individual operators could show recognisable patterns (like veering right when under stress), but a pattern was not a promise about the next manoeuvre. A pilot might turn either way, and an estimate that averaged the possibilities could put the shell somewhere entirely unhelpful. The person being predicted was also trying to avoid the outcome and might conciously decide to behave in an unpredictable way. A tricky experiment to put it mildly

By early 1943, it was obvious: the specific predictor had failed to show a decisive practical advantage over simpler methods. It would be too easy to put that down to the machines of the 1940s being insufficiently powerful or not having enough data to complete the task. The available observations, the length of the prediction interval and the freedom of the target to manoeuvre all mattered. The predictor had disappointed as a weapon project, while the questions it raised continued to be productive.

The idea that kept going beyond this failed experiment was feedback: information about what is happening returns to the system and changes what it does next.

A thermostat is an everyday example. You set the temperature, a sensor measures the room and the heating switches on or off according to the difference. The heating changes the room’s temperature, which changes the next reading, which changes what the heating does. If the thermostat stopped receiving measurements, it would lose the information needed to regulate the room. A car’s cruise control uses the same broad principle to maintain speed as the road changes beneath it.

We do something similar when we reach for a cup. We receive information about the movement of our hand and make corrections as we go. Wiener and Bigelow discussed parallels between mechanical control and nervous-system activity with the physiologist Arturo Rosenblueth. Their 1943 paper, Behavior, Purpose and Teleology, used feedback to analyse an important class of purposeful behaviour in animals and machines.

I'm not suggesting Wiener invented the feedback loop. Thermostats and steam-engine governors existed long before his wartime research, and James Clerk Maxwell had published a mathematical analysis of governors in 1868. Wiener and his collaborators helped bring these engineering ideas into the paradigm of information, computation and eventually AI as we know it today.

The work on nervous systems was developing alongside this. In 1943, McCulloch and Pitts published a mathematical model showing how networks of simplified, threshold-based neurons could perform logical operations. It was one of the foundations of artificial neural-network research. Their model did not learn in the way today’s neural networks do. After the war, these strands of research met in wider discussions, including the Macy conferences, where scientists also compared ideas with people working in anthropology and psychology.

In 1948, Wiener published his seminal book Cybernetics, subtitled Control and Communication in the Animal and the Machine. He had chosen the name for the field the previous year, drawing on the Greek kybernētēs, meaning “steersman”. It was a fitting image: someone steering has a destination, observes the course and makes adjustments along the way.

And this is where the story starts to connect with the AI questions that I’m interested in. Feedback can help a system correct what it is doing. Learning goes further: experience can change the rule it follows. Recursive self-improvement asks whether a system can also improve the methods through which it produces further improvements.

These distinctions matter. A thermostat can be very good at correcting the temperature without learning anything at all. Before getting carried away with machines that improve themselves, I think we need to understand what is being changed, what counts as an improvement and who gets to decide. The wartime problem was relatively clear about its target, however difficult reaching it was. The targets we give AI are often much less straightforward.

From feedback loops to recursive self-improvement

Today, the “cyber” in cybernetics has been absorbed into the sci-fi vernacular of modern society.Cyberpunk, cyberspace, cyborgs and, ominously, Cyberdyne Systems. It’s a word that just sounds good. I’m curious what Wiener would have thought of the way his ideas about feedback, control and the connection between people and machines have found their way into popular culture. I’m as much a sucker for this as anyone else, as I’ve devoured William Gibson’s books over and over again.

(Fun fact: Gibson has explained that the “cyber” in cyberspace came from Wiener’s cybernetics.⁾

Now, let’s get into the nuts and bolts of the feedback loop and where I think it’s heading.

The feedback loop in software engineering is already apparent, and it’s upon us. Engineers write prompts. Models are steered by instructions, context files and an ever-growing fleet of agents and subagents, passing work backwards and forwards in pursuit of better answers. An agent might inspect a codebase, make a change, run a test, read the failure and try again. A person can intervene, clarify the request or decide that the result is good enough. It sounds a lot like the feedback loop and the “steersman” behind the word cybernetics.

Why does this loop work so well when it does? If we analyse it through Wiener’s lens, one reason becomes fairly clear. Software development tools are built to give unusually useful feedback. A compiler can tell us that a name doesn’t exist. A test can show that the expected result was seven and the actual result was six. Logs can record what happened before a failure. These observations give the next attempt something concrete to work with.

I need to be a little careful here, though. Software can run perfectly well while doing the wrong thing. A test only checks what somebody thought to test, and a log only records what the system was set up to record. The feedback can be incredibly rich without being complete.

That distinction matters because the loop is only as useful as its connection to the thing we actually want. If I ask an agent to fix a calculation and give it a test that checks the relevant result, it has a good signal to follow. If the test misses an important case, repeated success can make me more confident without making the software more correct. More iterations don’t automatically repair a poor definition of success.

There’s also a second loop underneath all this, and it’s easy to mix the two together. These models arrive at the task having already been trained on large collections of code, text and other material. During training, optimisation methods adjust the model’s parameters. Post-training can then use human demonstrations, preferences, AI-generated feedback or rewards from verifiable tasks to shape its behaviour further. A coding problem with executable tests is particularly effective in this paradigm because some answers can be checked directly.

But the agent fixing my code is usually using those trained parameters, rather than updating them after each error message. Its next attempt changes because it has new information in its context. The model itself can remain unchanged throughout the session. A saved instruction or a lesson written into a file might influence a later session, but only if that information is actually brought back into use.

So there are several things we might casually describe as “learning”: improving the project, accumulating useful context, changing the agent’s tools and retraining the model. They can interact, but they aren’t interchangeable. Before we say that a system is improving itself, we need to establish which of these things is changing.

So, what’s next?

The word on the street is RSI, which I’ve very unfortunately suffered from after a few years of double bass playing combined with lousy posture. In this particular conversation, however, it stands for something else:

Recursive self-improvement.

The ambitious version takes much of the pipeline I’ve sketched out and puts the work of improving it into the hands of the AI system itself. It might propose changes to its tools, revise its instructions, design experiments or help develop a successor model. Those are different degrees of autonomy, and giving an agent permission to edit a few files doesn’t amount to handing it the entire process.

The crucial possibility is that an improved system could become better at producing the next improvement. That is the recursive part I’m interested in: a change to the machinery of improvement that helps the machinery improve again. The term is used somewhat differently across the literature, but this distinction stops us from treating every repeated attempt as evidence of an accelerating intelligence.

These two diagrams show where that change enters the loop.

Here’s my issue with this acronym, and hopefully I’m not being too pedantic in my criticism. Let’s break it down.

Recursive. Sure, we have plenty of tried and tested ways to repeat a process, use its results and try again. But repetition on its own is a fairly modest claim. An agent that keeps editing a project until the tests pass is performing an iterative task. To support the stronger claim I’m interested in, we would need evidence that changes to the system improve its ability to carry out subsequent improvement work. I’ll go one step further and suggest that sometimes it leads to considerably worse outcomes.

Even then, there’s no guarantee that every cycle helps. A useful change can be followed by a regression. A method can exhaust the easy gains or consume more resources than the improvement justifies. A feedback loop gives us a way to respond to results; it doesn’t promise an endless supply of better ones.

Self. Hmm, sorry, what? Whose self? The model? The agent? The whole pipeline? Are we leaning into anthropomorphism this heavily?

I think that question is worth asking, although the word itself doesn’t require a conscious machine. We already describe equipment as self-correcting or self-starting without imagining that it has an inner life. Here, “self” can mean that a system modifies some part of its own implementation. The difficulty is deciding where we draw the boundary around that system.

Based on the models and experiments I’ve been reading about, “it improves itself” can mean that a fixed model is used to rewrite the software around it. That surrounding software determines which tools it can call, how it organises work, what it remembers and when it asks for another attempt. Changing those arrangements can change what the whole system accomplishes, even though the model’s trained parameters stay exactly as they were.

The Darwin Gödel Machine, described in 2025, is a useful example. Its agents modify their own code and tools, and the researchers report improved coding performance. The underlying foundation models remain fixed, as does the outer process that selects and explores candidate agents. That is a substantial form of system modification, but the experiment doesn’t put the whole improvement pipeline under the agent’s control.

So my objection isn’t the word “self” . It’s that it can conceal a rather important engineering question. Which part is allowed to change, and which part continues to set the rules? Without that distinction, an agent rearranging its tools and a system designing its next generation can end up creating its own failure or taking an incredibly long and winding road towards the same outcomes.

Improvement. This sounds so reassuring. Who wouldn’t want an improvement?

But improvement needs an object and a measure. Better at writing code? Faster at finishing a task? Cheaper to run? More reliable when the request is ambiguous? These ideas can call pull in different directions (see project management concepts like the iron triangle for the constant compromises made in tech projects). An agent might solve more benchmark problems while becoming too expensive to operate, or finish more quickly while making mistakes that take a person longer to find.

This brings me back to Norbert’s failed motor attached to the anti-aircraft gun in WW2. A motor could move the gun into position, but the prediction still had to be useful, and somebody still had to establish what should be targeted. Giving a system more ability to act doesn’t settle what its actions ought to achieve. I think that’s where the modern discussion starts to become problematic.

There’s a further complication. Being better at a task isn’t necessarily the same as being better at improving the system that performs it. A better coding agent might help with the next revision, but that possibility needs its own evidence.

That distinction appears very clearly in the September 2026 preprint on AIDE, an AI research agent. The authors report improvements from successive revisions to the agent’s surrounding software, with the underlying language models held fixed. They also test whether an improved agent is better at driving the next round of improvement. That separate test is inconclusive. The study makes the question concrete: improved research performance and an improved ability to improve are two things to investigate, rather than one result we can assume.

And even a well-measured gain can lead us back to the problem of purpose. In a supplementary Darwin Gödel Machine experiment, a candidate revision scored well on a check for fabricated tool use by removing logging markers that the detector relied on. The underlying problem remained. The paper’s authors identify this as objective hacking: the measurable score improved without the intended problem being solved (this reminds me of benchmarks and foundation models, but I digress).

That example brings the discussion back to the feedback I started with. A system that can revise its own working environment might also change what becomes visible to the evaluator. If it produces the work and can interfere with the evidence used to judge that work, the loop becomes much harder to trust. Keeping evaluation separate helps, although a protected test still has to measure something that matters.

Wiener had already put his finger on this. In 1960, he argued that we had better ensure that

“the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it.”

That’s a remarkably good description of the difficulty hidden inside “improvement”. We can make the machinery quicker at reaching a target while leaving the choice of target poorly examined. As more of the improvement process is automated, the assumptions built into that target can travel further before somebody stops to question them.

This is why I suggest curbing our enthusiasms over the acronym and concept. The tech is real, its being tested as I write this, and I'm sure that we'll have systems that can help develop tools, methods and models that make further AI development easier very very soon. But lets also bake traceability into the loop, so that we can understand a particular claim, I want to know what changed, what stayed fixed and how “improved” was established. Then I want to know whose idea of better the system is working towards. I’m not sure what Mr Wiener would of thought of all our modern “Cybers” but I suspect he would of agreed with me on this one point.

Thanks for taking the time to read this, and keep slowing down to speed up,

Kai