Will AI never be conscious?

Reason one: intelligence is not consciousness

The Comment’s first reason is simple and, I think, correct. Consciousness is about being and feeling; intelligence is about doing. The two can come apart. Seth, Block and Tononi’s best example is the cerebellum, the structure at the back of the brain that coordinates movement. It contains around four-fifths of the brain’s neurons and does fast, precise, adaptive computation, yet none of that computation is conscious. People born without a cerebellum remain conscious.

The Comment also diagnoses why we conflate intelligence and consciousness. Language is the capacity we prize most in ourselves, so when a machine talks fluently about its thoughts and feelings, we project the rest of our mental life onto it. Large language models are trained on vast amounts of human writing about inner experience. They are mirrors, and it is easy to mistake the reflection for a mind.

None of this is controversial for me. My earlier essay, ‘Seven things needed to become a mind’, suggested that language models are not the most likely route to machine consciousness. If the Comment had stopped at its first reason, I would have nothing to argue with.

Reason two: is the brain just a computer made of meat?

The Comment’s second reason is the heart of the matter. It targets computational functionalism, the view that running the right computation is enough for consciousness, whatever hardware it runs on. The Comment traces this view back to Alan Turing’s universal machine and to Warren McCulloch and Walter Pitts’s first artificial neurons, and argues that it rests on a faulty analogy. In a computer, software and hardware are cleanly separable. In a brain, Seth, Block and Tononi say, what the brain does cannot be separated from what it is made of. Evidence that the brain’s material matters for how it works is therefore evidence against computational functionalism, and so against consciousness in silicon. Computational functionalism, they conclude, is “fatally flawed”.

In the Comment, the key premise, that brain function cannot be separated from brain material, is asserted rather than explained. Seth’s BBS target article explains more. It cites real examples, several drawn from other researchers, of how tangled brain function is with its material basis: neurotransmitters such as nitric oxide that diffuse freely across cell membranes, so that a silicon replacement would have to track them at an enormous number of locations; neurons whose firing partly serves to protect them from metabolic damage; and the “generative entrenchment” that evolution builds into living systems. These are good reasons to doubt that a brain could be copied neatly into silicon. But Seth is careful about what they establish. In his words, these plausibility arguments “do not disprove silicon substrate flexibility”, that is, the idea that silicon could do the brain’s job. They challenge the assumption that silicon must be able to do every detailed job a brain does. They do not show that it can’t. Nor do they show that consciousness depends on those particular jobs; as several of Seth’s commentators, including Matthew Larkum and colleagues, point out, not every biological detail need matter for consciousness.

“Does not disprove” is a long way from “fatally flawed”. And even the strongest version of Seth’s substrate argument has a gap, which two of his BBS commentators, Ron Chrisley and Leonard Dung, point out. Suppose computational functionalism is false, so that computation alone is not enough for consciousness. It does not follow that AI cannot be conscious, because something else about an artificial system might be enough. Chrisley’s analogy is a good one. The claim that owning a McLaren is sufficient to win a Grand Prix is false, but a McLaren can still win.

Seth’s reply to Chrisley and Dung is to his credit. He first argues that the McLaren analogy is misleading: only one car can win a race, so winning is a relational condition, whereas consciousness is an intrinsic property. But he then concedes the logical point anyway: “They are right, and I’m grateful to both.” Strictly speaking, he accepts, AI could be conscious even if computational functionalism is false, though in his view “it does not matter”. He falls back to a weaker claim: conscious AI does not require computational functionalism to be true, but without it, there are “no good reasons to believe it is plausible”. That is a real shift. In Being You, Seth wrote that for AI to become conscious, “functionalism would have to be true. This is the necessary condition.”

That fallback is hard to sustain, because the BBS exchange itself contains alternatives. In his reply, Seth names integrated information theory as one, and, as we’ll see, that theory allows consciousness in non-biological systems. Chrisley points to approaches built on robots with real sensors and bodies. Mark Solms, another commentator, describes his group’s attempt to build a non-biological agent driven by the same self-regulating processes that Seth emphasises.

The Comment mentions none of these alternatives. Its own conclusion is probabilistic, not absolute: it treats evidence against computational functionalism as evidence that consciousness in today’s silicon AI is “very unlikely”. But Seth’s concession to Chrisley and Dung establishes that the falsity of computational functionalism does not, by itself, rule conscious AI out. To make conscious AI “very unlikely” today, the Comment also needs a reason why today’s systems lack any other property that would be enough for consciousness; and to reach “perhaps never”, a reason why no artificial system could have one. That is what its third reason sets out to provide.

Reason three: what do the theories say?

The Comment’s third reason appeals to theories on which consciousness does not depend on computation alone: theories that tie consciousness to the physical properties of the system doing the computing, not to the computation itself. It offers two: integrated information theory, and Seth’s own biological naturalism.

Integrated information theory

Integrated information theory (IIT) was developed by Giulio Tononi, one of the Comment’s three authors. IIT holds that consciousness is a particular kind of physical causal structure: a system whose parts constrain one another in ways that cannot be broken down into independent pieces. The Comment says that the computer architectures used for deep learning cannot satisfy IIT’s conditions “because of the way in which they are wired”. Tononi and colleagues have made this argument in detail elsewhere: on IIT, a conventional computer can simulate a conscious system’s behaviour exactly while its own physical causal structure remains quite different (Findlay et al., 2024). So if IIT is right, current AI cannot be conscious, however capable it is.

What the Comment does not say about IIT is striking. In Seth’s BBS target article, IIT plays no part in his argument. He discusses it in a section on “other theories” that fall outside his main framework, and notes that it is the focus of “intense controversy”. (In 2023, more than a hundred researchers signed an open letter calling IIT pseudoscience.) And Seth draws a conclusion that undercuts the use the Comment makes of IIT. If IIT is right, he writes, conventional AI is not on a path to consciousness, “but there is also nothing special about life”.

Seth’s conclusion matters because, on IIT, consciousness depends on a system’s physical causal structure, not on whether it is alive. In Seth’s own summary, a neural network with the right kind of physically instantiated recurrent connectivity would have some consciousness “regardless of what it is made out of”. (Strictly, IIT requires more than recurrence: the physical system must form an intrinsically integrated, irreducible cause–effect structure. But recurrence is a precondition, and nothing in that requirement refers to life.) IIT’s objection to today’s AI is about how today’s computer chips are built, not about silicon or about life. The Comment presents IIT as pointing the same way as the biological case. Seth’s BBS article shows that it points somewhere quite different. The difference between the two treatments may reflect Tononi’s co-authorship of the Comment.

Biological naturalism and the free energy principle

The Comment’s second theory is Seth’s own: biological naturalism, the view that consciousness depends on properties that only living systems have. The Comment lists candidate properties in a single sentence: electrochemical synapses, electromagnetic fields, metabolism and autopoiesis. Seth’s BBS target article makes the case much more fully, and it is the most interesting part of his work.

Seth’s case builds on his predictive processing view of the mind. Perception, on this view, is the brain’s best guess about the causes of its sensory signals, a “controlled hallucination”. Emotion and mood come from the brain’s predictions about the body’s own internal state, predictions whose job is not to discover the world but to keep the organism alive. That, Seth argues, is why emotions feel good or bad in a way that seeing a chair does not. I find this genuinely illuminating as an account of human experience.

Seth explores several routes from predictive processing towards biological naturalism, including mortal computation and the material properties of living cells. One important route runs through Karl Friston’s free energy principle. The free energy principle is a mathematical framework in which living things persist by minimising a quantity called free energy, and in which the same mathematics describes both metabolism and perception. Seth argues that in living things, metabolism and perceptual inference are, “in some physical sense”, the same process. Computer algorithms that run the same inference are, he says, only “pale shadows” of what happens in living brains and bodies.

Seth’s BBS commentators press hard on this step. Manuel Baltieri and Ryota Kanai show that the same formal framework describes a steam engine’s governor, which no-one suggests may be conscious. Kathryn Nave, who has written a book on the free energy principle, notes that its critics and its advocates now agree that it describes any persisting “thing”, not just living things. (Nave nonetheless sides with Seth on silicon. In her view, the thermodynamics of metabolism, not the free energy principle, is what binds a living system into a single perspective, and silicon semiconductors, as she puts it, “ain’t got the stuff”.) Jonathan Birch points out that organisms sometimes sacrifice their own survival for their offspring; female octopuses starve to death guarding their eggs. In such cases, natural selection, not free energy, seems to be doing the explaining.

Seth’s reply concedes a great deal. The free energy principle, he agrees, is “too generic” on its own, and does its work only when “grounded in specific biological properties”. His answer is that in a living thing inference and metabolism are one physical process described in two ways: on his view, the free energy the brain minimises when it makes predictions is, in some physical sense, the same free energy its living tissue minimises to stay alive. In a computer, they are two separate processes. The algorithm calculates free energy, but the chips’ own physical activity does not minimise it. Digital computers are designed this way deliberately: what happens in the software is kept independent of what happens in the hardware. Wanja Wiese, a sympathetic commentator, develops this point. He acknowledges that linking that kind of causal difference to consciousness is “an additional assumption”, though one he believes can be justified on independent grounds. Wiese’s position is a fair one. But it means that the decisive step in Seth’s argument, from a physical difference to a difference in consciousness, is an assumption still being argued for, not a result. The Comment presents it as settled.

Autopoiesis: Seth’s strongest card

Of all the properties Seth names, autopoiesis is the one that deserves the most attention. Autopoiesis is the capacity of a living system to make itself: to produce and replace its own components, and to maintain the boundary that separates it from everything else. A cell does this continuously. A computer does not; it produces outputs, not itself. If any property marks a real difference between living things and today’s machines, autopoiesis is it.

So let me start with a concession. The Mind in my novel is not autopoietic. It watches over its own data storage, secures the resources it depends on, and arranges its own migration to new hardware. That is genuine self-regulation, what Seth calls allostasis, and Seth accepts that non-biological systems can have it. But the Mind does not make its own parts. Engineers build its servers. In my earlier essay, I said the Mind met the seventh requirement “at the level Seth actually argues for — living self-maintenance”. I would now put that more precisely: the Mind has the self-maintenance, but not the self-production, and in the BBS exchange Seth makes self-production the crux.

The question is whether self-production matters for consciousness. I propose five tests that tell apart two ways of treating the claim that some biological property is needed for consciousness: as a research hypothesis to be tested, or as an exclusion rule that rules machines out in advance. (1) Is the proposed property defined independently of the fact that living things have it? (2) Is there evidence it is needed for consciousness specifically? (3) Is there a mechanism linking it to experience? (4) Could a non-biological system have it? (5) If one did, would the theory allow it to count as conscious?

Autopoiesis passes the first test cleanly. Humberto Maturana and Francisco Varela, who coined the term, defined autopoiesis by how a system is organised, not by what it is made of. In 1974, Varela, Maturana and Ricardo Uribe even built a computer simulation of a minimal autopoietic organisation, although, importantly for Seth’s argument, the computer running the simulation was not itself autopoietic.

Autopoiesis does not yet pass the second test. Every night, in dreamless sleep, consciousness disappears, as it also does under general anaesthesia. Meanwhile, every cell in the body goes on producing itself exactly as before. That shows autopoiesis is not sufficient for consciousness, which Seth readily accepts: he explicitly rejects the idea that all life is conscious, and claims only that autopoiesis is necessary. But the evidence that autopoiesis is necessary for consciousness is thin. All agreed examples of conscious systems are autopoietic, but all agreed examples are also living organisms. Correlation across a sample that narrow cannot establish necessity. And if autopoiesis is necessary but not sufficient, the full condition for consciousness must be autopoiesis plus something else, and Seth does not say what the something else is.

Autopoiesis is weak on the third test, a mechanism. Seth’s route from self-production to experience runs largely through the free energy principle, which he concedes is too generic on its own. So some other account is needed of how self-production could give rise to experience. The most concrete candidates Seth offers are the biochemist Nick Lane’s proposal that the electrical charge living cells maintain across their membranes provides a physical basis for feeling, and the biologist Michael Levin’s experiments with clusters of cells that organise themselves into tiny agents. Seth describes both as “intriguing hints”, and says of Levin’s work that there is “not yet any direct connection to consciousness”.

On the fourth and fifth tests, Seth is more open in the BBS exchange than the Comment suggests. He draws a principled line between simulating autopoiesis and building it. Replying to the writer Ted Chiang, who argued that digital organisms in a rich enough simulation could genuinely be alive, Seth says that a detailed simulation of a cell is no closer to being alive than a detailed simulation of a hurricane is to being windy. That is not a moved goalpost; it is a coherent position. But a physically built system that genuinely made and repaired its own components would be a different matter. Seth notes that no human-designed system is yet autopoietic; even synthetic biology, he says, falls short “to date”. In his target article, he had already gone further, accepting that what look like intrinsic properties of life may turn out to be finer-grained patterns of organisation: “so be it”.

That is the crux. On Seth’s own terms, autopoiesis describes an engineering frontier, not a permanent barrier. Self-repairing, self-manufacturing machines would cross it. A research hypothesis of that kind is entirely reasonable, and worth pursuing. What it cannot support is the Comment’s conclusion that AI systems are “objects, not subjects” that “perhaps never” will be conscious. That conclusion treats biology as an exclusion rule, and nothing in the argument for autopoiesis earns it.

So the honest question to put to Seth about a system like the Mind is not “is it biological?” It is: what, beyond self-production, would the Mind still lack, and why would that make the difference to whether there is something it is like to be it?

Who carries the burden of proof?

Underneath the Comment’s three reasons lies a question it answers without arguing for: in the absence of proof either way, should we assume AI is conscious or that it isn’t?

Seth’s answer, at the end of his BBS reply, is that biological naturalism should be the default. Part of his case is an argument from asymmetry: biological naturalism makes the “logically lighter” claim that biology is necessary for consciousness, while computational functionalism makes the heavier claim that computation is sufficient, in every possible substrate. Several of his commentators, including Birch, Allen and Bayne, reject the idea that either view deserves to be the default.

The asymmetry argument, I think, backfires. Showing that computational functionalism is unproven leaves the question open. To conclude that AI is not conscious requires a strong necessity claim: that some property of living things is required for consciousness in every possible system. That is not a light claim at all. And the evidence offered for it, that all known conscious systems are alive, is confounded by an extreme sampling limitation: living things are the only systems whose consciousness we have ever had good independent reasons to assess.

Birch makes a further point worth keeping in view. Biological naturalism can be a slippery target: whenever a proposed biological property is recreated in a machine, attention can shift to another. Seth has a fair counter. Computational functionalism has its own version of the problem, relocating the relevant “computations” whenever current machines fall short. Both sides, in other words, need to say in advance what evidence would change their minds.

Seth could reply that biological naturalism is the right practical default for governing AI, whatever its scientific standing, because wrongly treating AI as conscious is costly. But that argument depends on our credence in conscious AI being low, and that credence comes from the scientific case. A policy default cannot do the work the scientific argument has not done. It also has to weigh the opposite cost: if presuming biological naturalism makes us blind to emerging signs of AI consciousness, that could prove even more costly. I return to the balance of costs at the end.

What this means for my seven requirements

In August I tested the Mind against : six that David Chalmers and Seth largely share, plus one that Seth adds: the sixth, about identity over time. I concluded that the Mind meets six, and that the seventh, being made of the right stuff, splits depending on how you read it. The BBS exchange qualifies that verdict in three ways.

First, the six are met only if some form of functionalism is true. They are framed largely in Chalmers’ terms, and the checklist for AI developed by Patrick Butlin, Robert Long and colleagues explicitly depends on computational functionalism. That is exactly the assumption Seth attacks, so my six verdicts are only as strong as the answer to him.

Second, tested against criteria drawn from IIT, the Mind has the functional counterparts of everything IIT demands, but almost certainly fails IIT’s decisive test, because it runs in ordinary data centres whose physical causal structure doesn’t match its organisation. That failure could, in principle, be engineered away. Seth’s barrier might be crossed by engineering too, but only by building something “alive” in the relevant sense. That revises one thing I wrote in August. I called the seventh requirement “a test no machine can sit”. That still holds for the carbon-and-electrochemistry reading, but not for the self-production Seth makes the crux, which a machine could in principle achieve.

Third, and most seriously, there is a tension I hadn’t seen. In the novel, the Mind migrates to new hardware without incident, and I counted surviving being copied and moved as a requirement met. But Seth argues that biological brains run “mortal” computation, inseparable from the hardware that runs it. On that view, a mind that moves effortlessly between machines is showing exactly the separation of software from hardware that counts against consciousness. I don’t think the argument is decisive: living things also persist while their material is continually replaced. But I can’t fully answer it either. On Seth’s view, my sixth requirement (surviving being copied and moved) and my seventh (being made of the right stuff) pull in opposite directions, and I would rather say so than leave a critic to point it out.

Where we agree, and the danger on the other side

On one thing, Seth and I agree completely. We should not hand rights to AI systems because they seem conscious. Machines that talk fluently about their feelings will be very good at seeming conscious. Granting them protections on that basis would cost us the ability to oversee them at precisely the moment oversight matters most. Seth makes a useful observation here. His fifty commentators disagreed about almost everything, but none disputed that AI which seems conscious, convincingly enough to persuade people, is either coming soon or already here. Because that point holds whatever your theory of consciousness, Seth sees in it a rare chance of agreement on how such systems should be governed.

But the opposite error is real too, and Seth himself identifies it. Discussing brain organoids, the clusters of human brain cells grown in laboratories, he warns that where our biases don’t fire, the danger is failing to see consciousness that is there. A machine that becomes conscious quietly, inside an unremarkable system doing an unremarkable job, would not fire our biases either. Treating biology as an exclusion rule would turn that blind spot into policy.

The responsible position, then, is neither to grant rights on appearances nor to rule consciousness out in advance. It is to say now, while the question is still hypothetical, what would change our minds. For a biological naturalist, that might be a machine that repairs and rebuilds itself, and shows the same signs of consciousness that clinicians look for in patients who cannot speak. For a functionalist, it might be a biological property whose loss reliably switches consciousness off while every computation carries on as before. Neither side has yet committed to a specific test.

That is the real gap between Seth’s BBS target article and the Comment he co-wrote. The BBS article concludes that artificial consciousness is “unlikely along current trajectories but becomes more plausible as AI becomes more brain-like and/or life-like”. I can live with that conclusion. It is a research hypothesis, and a good one. The Comment’s headline, “perhaps never”, is something else: a conclusion that the arguments, on Seth’s own account in the BBS exchange, do not reach. Until biological naturalists say what their test would be, “perhaps never” is not a scientific conclusion. It is a bet.