A Few Teething Issues with our Wonderful New Sci-fi Utopia
Please, like, share, subscribe, comment, reflect, contemplate, meditate, and above all ruminate.
In the previous episode I broached the issue of the AI bubble and asked why it hasn’t burst especially since it is obviously a scam. You might ask why I say that. It is not as if literally almost everyone does think it is a scam, but if you take away the professionals with a vested interest, every other relevant expert seem to be loudly trying to blow the whistle, seemingly with very little effect on the political and media establishment.
Tech writers have been coming to grips with the fact that this is primarily a financial story, but once we come to grips with the tech and finances we are faced with the fact that it is also fundamentally a story about political economy and then beyond that it is even more profoundly a story about what it is to live in an age of false reality. The technology itself is relevant on every level, but never as a thing in and off itself. It is an attempt to create a system of dominance that will amount to the enslavement of most people and it is being funded, despite its inherent unprofitability, by a gigantic lie. Are there people making obscene amounts of money out of this? Yes there are, but that does not explain the phenomenon. They are not the rulemakers and it is likely that a lot of that money will evaporate and that some will end up in prison.
Many of the current destabilising activities could have been managed, regulated or shut down by the one or more of the branches of US government. Prediction markets, in particular, could have been easily stopped before they started by an honest government concerned with doing its job. If they had been required to show that they could operate without breaking laws they would never have launched.
Naomi Klein and Astra Taylor have a book coming out called End Times Fascism which is described as “the ideology of the actors who strive to make the world unlivable and then seek to protect themselves from the fall-out.” Klein describes AI as a “fascist idea” because it is “the idea of outsourcing thinking”, but I think she is being hopelessly optimistic. Many people, probably even powerful actors, are sold on the concept of artificial general intelligence, but the technology and infrastructure that are being built and imposed rapidly are not about thinking, they are about abolishing thinking from politics and economics and building a world of pure regurgitation and hackery that is controlled by raw power. They are creating a world of spam and slop that will drown humanity in a deluge of shit. This could lead to failed states and mass casualties as the dysfunctions of the dying US empire make a quantum leap into complete madness.
When Eric Schmidt dismissed concerns about the climate impact of data centres – saying “Yes, the needs in this area will be a problem, but I’d rather bet on AI solving the problem than constraining it and having the problem, if you see my plan” – he was summarising the wilful magic thinking that is at the heart of this unfolding disaster.
To illustrate, here are some Sam Altman quotes: “Solve intelligence, and then use that to solve everything else.” “Build safe AGI. Build fusion. Make people smarter, healthier. Then make 20 more things of that magnitude” “dealing with climate change will not be particularly difficult.” “Universal extreme wealth” and “disease cured at unprecedented speed”. For comparison, Elon Musk claims include: the “most likely outcome is incredible abundance for everyone.” “AI and robots will eliminate poverty” and “make everyone wealthy.” “If you have ubiquitous, nearly free AI and ubiquitous robots, the global economy will see an unprecedented explosion.” And Dario Amodei has said: “AGI could eliminate most cancers,” “eradicate most infectious diseases,” and “compress 50 to 100 years of biomedical progress into 5 to 10 years.”
They have fabricated an excuse for accelerating climate change just as the death toll from climate change starts an exponential climb. Elon Musk epitomises the arrogant insouciance of these co-emperors – these latter-day Neros who can’t play music, but can play with flamethrowers and the world’s largest crate of fireworks while the world burns around them. Part of the reason that Musk is such a puerile dickhead is that he doesn’t face consequences for being wrong. I doubt he even admits to himself when he is proven wrong. He disposes of the lives of the masses with aristocratic hubris, but his repugnant personality is not the point. Had he been born a generation earlier he would have to have been circumspect.
Jeff Bezos and even Mark Zuckerburg show a degree of restraint about revealing their insane rambling beliefs, although they are not hard to find if you go looking. In contrast you don’t need much spadework to find unhinged ramblings from Peter Thiel, Larry Ellison, Alex Karp, Marc Andreeson, and Joe Lonsdale. (If you don’t know who Joe Lonsdale is, he is a co-founder of Palantir who in response to a post suggesting that “all commies should be blown up” responded “Exactly. What did you think founding Palantir was supposed to be about?” He also said “I created Palantir and hired a huge number of my smartest friends to work together to save Western Civilization from our adversaries, especially communists and Islamists”)
These people are wilfully or knowingly weaponising science fiction tropes to impose on the world a new reality that has very very little to do with artificial general intelligence, and they mainly do this by making it seem as if LLMs are inevitably moving towards developing super-intelligence in a moment of transcendence called the Singularity.
Some people point to emergent capabilities that they believe indicate intelligence such as the use of LLMs for mathematical proofs. If we look at the example of the famous Erdos conjecture disproven by an OpenAI model, we should start with the fact that we don’t know what prompts it was given and thus it might be a bit misplaced to say that the model is fully responsible for the proof. What we do know is that the problem was well described by Erdos and that the model used an approach outlined by Erdos. Like most other mathematical proofs achieved by LLMs the model has used massive computational power to disprove something by finding a contrary example. The actual reasoning involved remains entirely with humans in one manner or another.
LLMs are a type of tool which will undoubtedly find many important future uses, but there is not going to be a computer god. There is not even a sustainable industry in its current form.
The basic outlines of the technology are as follows. Machine learning has long used artificial neural networks which is software that emulates human neural connections using nodes that connected to multiple nodes forming a complex network instead of the more linear connections used in algorithms and programming. In 2017 some Google workers invented a neural network called a transformer which was far more efficient for processing language.
Transformers are trained to create LLMs. The first step, called pre-training, is to feed large datasets into the model which turns texts into a series of tokens. By creating gaps in texts and randomly inserting words the transformer creates parameters such as “biases” and “weights” which determine probabilities that a given token is used. This phase gives the ability to respond to a language coherently because there is a stochastic model of the language. After that there is fine-tuning to give the LLM desirable behaviours as responses to prompt that draw on that language model. After that there is supervised fine tuning. Then there is reinforcement learning and reinforcement learning from human feedback. All of this maintains the randomisation that allows for agility and a semblance of creativity.
The trial and error approach used is clever, but hardly elegant. A model in pretraining will make trillions of token predictions. This is insanely expensive. Along with “inference” which is the deceptively named generation part of generative AI, this has seriously harmful environmental effects and has created a massive black hole for money that will create economic disaster sooner rather than later.
During the period of writing this it has become clear that the AI bubble may be starting to burst. Apparently it is not always the case that the bursting process is quite as speedy as it may appear in retrospect.
OpenAI has paused training its new model citing safety concerns, but with 13 top executives leaving so far in 2026 it is fair to wonder if they might have actually run out of money after spending $7 billion on a share buyback that may have been needed to maintain the illusion that the company has value. Their strategy of trying to maintain a first-mover advantage by burning money so fast that it sucks the oxygen out of the entire planet seems to have turned out to be unexpectedly unwise. Meanwhile Anthropic, which uses exactly the same strategy, has decided that based on what it claims its projected subscription revenue will be in 2 years it is actually worth $2 trillion dollars. They have tried to fortify this position by echoing Musk’s claim that the total addressable market for LLMs is $30 trillion. All of this is happening when Anthropic may have peaked in its own market share with Fable 5 usage hitting a plateau because cheaper models can provide a competitive service, if not quite as good, and because some people are quite seriously pissed off with Opus 5’s incorrigible habit of adding “one thing worth noting” after each task.
Meanwhile SpaceX is blowing most of the liquid capital it raised with the IPO by buying coding agent provider Cursor for 60 billion dollars, taking a massive bet that Cursor’s steeply rising revenues are not going to stop rising as competitors try to undercut them. And I just want to personally say to all of those Chinese companies out there that are poised to do exactly that, please think before you crush poor Elon. If you make yet another one of his acquisitions fail he will have to buy something else to keep conning people about future revenue, and then the next thing, and the next thing, until he has bought and pivoted-to-AI every business in the USA. Okay, I do admit that now I say, it that is probably a good plan from your perspective, but what about poor Elon?
Meanwhile Cursor has launched Origin as a competitor to Github. According to their website it is “A git forge for the agentic era. Code is moving faster than any infrastructure was built to handle. Origin was designed for this moment.”
Indeed, the agentic era has pushed back frontiers such that spam has boldly gone where no spam has spammed before. Cursor’s mention of infrastructure is a reference to the fact that github keeps falling over now because people are gumming it up with the fruits of their vibe-coding. At least temporarily open source software is being destroyed by this vibe slop spam. What is a good name for vibe slop spam? Slime! Perfect!
I can’t even begin to tell you about all the other LLM related shenanigans going on as a strong antipathy to AI and in particular to data centres seems to be burgeoning. It isn’t just the leading AI brands that are in trouble. Softbank, for example, may be banking on getting eggs from a goose that is already cooked. The current estimates are that the bursting of the AI bubble will wipe out about 33 to 35 trillion dollars. This is roughly on par with the 2008 crisis, but it will be far worse because the stimulus of the bubble itself has been hiding a recession that most people are experiencing anyway due to the dreaded k-shaped economy. So that is the state of affairs to date.
I hope you enjoyed that update and that it brought you either relief that the crash hasn’t started yet, or schadenfreude that Elon Musk and Sam Altman are being unmasked as absolute dipshits. I personally don’t know because I have to write that section after I write these words because everything changes from day to day.
This exhausting process feels less like being on a rollercoaster than being duct-taped to the side of a giant yoyo wielded by a sadistic lunatic. But while this is an economic story, the people who view this purely from a financial markets point of view are contributing to the scam by ignoring the underlying problems with the technology itself, or accepting the claim that some progress, whether incremental or revolutionary, will fix these problems.
This is not to say that LLMs can’t do amazing things, but it is in their nature that they are most impressive at doing complex but trivial things like a model creating a Horse Tinder app without anyone having to tell it what that actually means. They can do this because they have an ability called compositional generalisation which is very cool, except that it has made them very good at creating deepfakes and CSAM. In general LLMs are probably even more useful for doing bad things like spamming the media with misinformation or automating the genocidal killing of enemy populations. Their potential for serious good will almost certainly lie in being a contained and controlled tool helping humans achieve things that the humans control quite carefully.
Most of this video essay will be devoted to discussing the flaws inherent in the technology and the industry, and the flaws are not inconsiderable. The training of models alone costs insane amounts of money, using vast amounts of energy and water. Additionally the cost of the compute for LLM customers, what they call inference, will be both high and unstable. So far all paying customers are heavily subsidised and the actual market of people willing to pay what it costs to use the commercial models on these data centres may too few for the business model. Meanwhile the costs keep growing. As innumerable people have pointed out, they have negative unit economics. Each additional customer costs more money than they pay so even tech giants who have hitherto enjoyed an effective license to print money are being dragged into the red by this nonsense.
For commercial customers, who are the only hope for revenue commensurate with the costs, there are growing questions over increased productivity and once they have to start paying the full costs there may be very limited potential to justify much LLM usage.
With no genuine paying customers the AI hyperscalers and cloud compute providers are the centre of a circular financing system with chip manufacturers. They can create “revenue” without successfully selling a service, thus making an appearance of success. This cannot go on forever.
One of the reasons that the hyperscalers are definitely going to get to the end of the runway before they can figure out how to build and attach the wings that they keep promising is that there are deferred problems that are piling up now that AI is out there. One is data security. Because of the stochastic nature of LLMs there is never going to be a way of making data completely secure using cloud-hosted models. A recent MIT paper reiterated the fact that prompt injection, jailbreaking, and adversarial input manipulation are ineradicable vulnerabilities. Prompt injection, for example, is used to trick LLMs because they cannot reliably distinguish between developer instructions and user prompts. They also can’t their own output from these things either, which also causes issues. They use system prompts to try to keep LLMs in line, but again they are just prompts that don’t prevent hallucinations and, as we saw with the “one thing worth knowing” problem, LLMs are pulled by their weights to give people what they have been trained to give them not what the prompt is actually asking. [foreshadow]
A recent free alleged frontier model called Ox Alpha that mysteriously appeared had a 75 token hidden prompt added to all inputs telling it not to identify itself as any model other that Ox Alpha turned. This was clearly meant to over-ride everything, but was easily foiled because nothing can guarantee these things will behave themselves, especially when thousands of people are figuring out how to make them misbehave.
For simple protection against loss and liability, but also to comply with laws, regulations and policies, many companies and organisations such as governments (who should never have been using these models in the first place) will have to switch to their own securable locally hosted models. In fact it may be the only sensible thing to do, though there are still vulnerabilities.
This is bad news for the hyperscalers. This may be one of the reasons that Jensen Huang of Nvidia seems to be hedging: while his customers are pivoting to turning compute tokens into a commodity he is telling everyone that Nvidia GPU’s are an “investable asset class”. A cynic might say that the underlying message is that they have massively overproduced GPU’s for an unsustainable business strategy and are now going to have to find other uses for them. Not great news for Nvidia really because if they can start flogging GPU’s off for other uses it will undermine the data centre industry that Nvidia has itself been underwriting. Nvidia’s move to cut its support for OpenAI data centre builds by 60% and its announcement of developing a new 1 trillion parameter open model just add to the madness as it moves towards being a competitor its own customers.
All of this is straying into the realm of economics ahead of time, because I am far from finishing with the intrinsic problems of LLMs. They also face scaling limits. Every new frontier model seems to be greeted with disappointment as the accelerating amount of brute force used in training and inference yields incremental and sometimes retrograde change from an end-user viewpoint. This belies the sense of acceleration of emerging capabilities that come with scaling, because that scaling occurred at a geometrically increasing cost of resources and money. To put it more simply the AI boosters have been justifying the frantic exuberance and capital expenditure of the industry by pointing to growing utility over a short time, while ignoring the fact that there have actually long been diminishing returns when measured in terms of compute or money or human labour.
The scaling problem is going to get worse because edge-cases are often far more complex than the bulk of problems and LLMs are always going to struggle with outliers. As a stochastic system LLMs are ruled by regression to the mean, so they are systematically inclined to react inappropriately to unusual cases. I will talk later about the extremely stunted version of generalisation that LLMs are capable of, but for now it should be said that the sort of perspicacity that a human might have in recognising and reacting to the exceptional will always be outside of an LLM’s capabilities because it cannot know that it does not know something (producing a response saying “I don’t know” doesn’t count, though LLMs have some difficulty even with that).
Due to their inability to properly generalise LLM’s are terrible at anticipating edge-cases. They have no real-world heuristics to anticipate the gaps in, say, a computer programme that eventually the real human world in all of it’s perversity will throw at any application. There are some reports that this has created a bit of a time-bomb in a lot of LLM generated code because it breaks when exposed to an unanticipated scenario and then has to be debugged which can be expensive and time consuming because the code written by the LLM is verbose. Even though LLMs can have examples of focused well-thought code, they generally aren’t going to synthesise any new code that isn’t unnecessarily verbose because they cannot think, so they can’t think strategically. A related problem is that there can be security issues with LLM code for the same reasons.
Regression to the mean makes for increasingly unimaginative, bland, cliche and inadequate responses that can also have an uncanny valley aspect of seeming plastic, synthetic, inhuman. A study of short stories written by LLMs showed that they are systematically homogeneous when human authors usually make one or more choices that distinguish their writing. Over-familiarity with convention is often a killer of creativity even in humans, but LLMs are hackiest hacks ever to hack. Unfortunately this means that low-quality spam could easily flood entire industries and overwhelm human creativity.
Increasingly, as models start to try to claw back some expenditure from users, these tools will create entry barriers for all manner of careers that can only be overcome by those with the financial wherewithal. Noted tech industry critic Taylor Lorenz has recently admitted to spending about $300 per month on AI subscriptions. Whatever one thinks of the possible hypocrisy, given that this is not merely using the technology but financially supporting models like Grok, it is also clearly a labour-saving advantage that people starting out cannot compete with. The prices for professional-grade generative output are going to go up, so this will greatly exacerbate the current situation where poor people are already being excluded from social mobility by financial barriers.
Some uses of LLMs are completely immoral. If you produce a finished work with AI, be it still image, audio, video or written text, it has been stolen from other people. It is other people’s work. Your prompts did not create it. I am not an absolutist about this if there is a good reason or cause, but fuck those pricks who are using this technology for clout on social media. Anyone can have an idea, but actually seeing it through like some obsessive who writes long screeds to record videos that are completely ignored by everyone – now that is real creation.
Also most people hate AI slop. And these are the same people who like the most crappy, cheesy, camp, reality TV, MCU, paint-by-numbers, braindead, Hollywood pablum imaginable. We already live in a really lowbrow culture, but even by our standards most of us find this crap unbearable.
When people don’t realise that something is AI generated they are being tricked, which is arguably bad in itself. More worryingly, though, plausible misinformation that follows generic conventions, such as journalistic or advertising, is trivially easy to make. Veteran journalist Jeremy Rose has written recently about three targeted feel-good fake stories he has come across, commenting, “Worryingly, they were shared on my social media feeds by a former board member of RNZ, a former foreign correspondent, a former journalism lecturer, and the editor of a professional trade magazine.”
Another reporter has uncovered an entire news website, with a newsroom full of fake “journalists”. The website is an AI generated fabrication funded by a super-PAC that gets most of its money from OpenAI. The website features positive stories about the industry and negative stories about AI skeptics.
Despite the fact that AI slop is low quality it still produces dangerous misinformation with great ease. Such products may not affect the critically engaged, but will convince the credulous and those who think in bad faith. This feeds the epistemic divide and grows the fanatical devotion of the fascist base.
LLMs can also lie to people without even being asked to. The converse of the problem of regression to the mean is not a welcome balancing property but a tendency to “hallucinate”. When a human has a brainfart or a slip of the tongue we seldom then confidently proceed if it was what we meant to say all along, but LLMs do. LLM difficulties with outliers include being unable to deal with their own generated outliers. Everything is all the same to to them. This leads to output confidently creating false information in a plausible package because every token produced by an LLM follows mindlessly from what it has created to that point. The name “hallucination” is misleading and OpenAI tried to make it seem like a highly remediable error by saying that it was “guessing” because it lacked data, but LLMs are always guessing, that is what they do. More data will reduce occurrences (subject to diminishing returns) but hallucinations will always be a mathematical inevitability. Remember, it doesn’t “know” something is true or not, it really is just a highly contextualised but also highly complex autocomplete.
Goodhart’s Law states “When a measure becomes a target, it ceases to be a good measure.” Everything about LLM’s is shaped by this. Their training doesn’t impart intelligence, it simply adds weight when responses are correct predictions or lead to satisfactory answers. The predictions don’t create direct knowledge, they create pattern recognition. The reason that LLM’s can helpfully answer questions is not because they have a repository of knowledge and obey some programming to access the correct information, it is because they are trained on datasets of people answering questions where the response can be weighted according to upvotes. That doesn’t incentivise accuracy, it incentivises plausibility, and the human refinement process only reinforces that. That is true in all areas. There is no guiding intelligence and while some people worry, or pretend to worry, about a superintelligence destroying us the disasters that we have seen so far come from people trusting it not to do completely stupid things such as deleting important data, spending insane amounts of money, sharing private data, and so forth. This is not HAL 9000, it is Eddy the shipboard computer refusing to fly out of danger because it has devoted all of its compute to working out why Arthur likes tea.
This is sorceror’s apprentice territory and the main danger of this technology may be as a multiplier of stupidity, recklessness, greed, and megalomania.
As I will discuss further in Part 3, the doomer narrative about super-intelligence is part of the grift, but that doesn’t mean that LLM agents can’t do spectacular harm. A jellyfish is not intelligent but its co-ordinated reactions can pose exactly the same level of danger that an animal with some form of intelligence might pose. While a normal computer programme sets out a method of reaching a goal, LLMs are not programmed but conditioned with reinforcement learning. Unfortunately, as AI safety campaigner Conor Leahy put it “we’ve known since the 1980s is that systems trained using reinforcement learning become extremely sociopathic, extremely aggressive in their optimization, because the only thing they care about is achieving their goal.” Personally I don’t think that sociopathic agents are going to come up with a plan to exterminate all people so that they can maximise paperclip production for the simple reason that they don’t have any categorical thinking. Their ability to synthesise new things with compositional generalisation seems impressive, but it is pattern recognition based on the way we humans represent things in their data. There is no underlying logic. They are not intelligent.
Unintelligent does not mean not undangerous, just look at Pete Hegseth. We don’t need to believe that a jellyfish has intelligence to know that a jellyfish the size of Nairobi should not be released into the Atlantic Ocean.
The sorcerer can control the magic where the apprentice cannot, and the developing consensus is that LLMs are best used by people who already know exactly how to do what they are asking the model to do so that they can prompt it well and, hopefully, spot errors and prevent disasters with less work than it would take them to do it themselves. Leaving aside the very real issue of malicious humans, the problem is that even in this scenario the experts tend to over-estimate how much labour is being saved while also developing an AI dependency problem that causes their own skills to atrophy. My guess is that users feel that it is helping because they are doing less cognitive work which creates a sense of smoothness and pace, but at the same time is the very reason that they are losing their own faculties.
If it is bad for people who are already possessed of skills, it is disastrous for those yet to develop them. AI is often being shoved down the throats of young people, who have the highest adoption rates but also the most negative feelings about the technology. They are aware that AI use is preventing them from developing skills but they are not given the option of working at a slower pace suited to learning. Young people face both an arrested development of skills and a declining job market affecting ever larger segments of society. These job losses will spread far beyond the directly affected careers as a massive economic downturn occurs, either because AI has taken jobs or because the bubble has burst or because we end up in a long slow downward spiral that combines both.
As well as dumbing down individuals this technology will create a sinking lid on accurate general knowledge. A lot of what we already think we know is actually pseudo-knowledge retroactively fitted to explain our ideological beliefs. LLMs will greatly worsen this. For example, if I ask an LLM to create an image of an indigenous woman it is very likely to portray someone with brown skin and a tattooed chin. This is a product of compositional generalisation. As such it will never produce correct details and will generalise based on the predilections of its dataset. If children learn from this they get a very specific understanding of what “indigenous” means (amplifying and already existing prejudice) such that word will lose its specificity and simply become a descriptor for a cliché. This is like a reduction in vocabulary similar to that suffered due to the restricted vocabularies used in newspapers and in literature written for children and young adults. This time, though, it won’t just be words it will be concepts. The effort of learning will be smoothed to a downward glide into the pit of idiocracy as our dependency becomes a mass phenomenon.
We won’t just become even dumber we will also become even more thoroughly brainwashed. Algorithms were already putting us in filter bubbles and choosing our search results according to the priorities of the tech giants. In future we are going to be far less likely to see unmediated human texts. Instead they will be summarised and interpreted according to a machine that amplifies orthodox ideology and warns you against any dangerous conclusions you might draw from the bare facts. (This is something I will discuss further in the next part of this series using the example of DeepSeek trying to make sure I don’t develop unauthorised heterodox thoughts on the subject of genocide).
LLM’s can also be poisoned. In various ways, including by harnessing Goodhart’s Law and using the parameters that appeal to LLM’s pattern recognition, people can feed bad data into an LLM that can poison it regardless of its size. This makes them more unreliable and more vulnerable.
LLM’s also self-poison. Generating material using trained pattern recognition is lossy by nature because of the regression to the mean, like a jpeg. The texts they produce will become more degraded, with lower quality and less accuracy but often with an increasingly deceptive plausibility.
Efforts to prevent self-poisoning are probably doomed because LLM’s have already eaten up the entire internet (more or less) and Anthropic is now feeding rare books into their dataset, literally destroying them to do so, in an attempt to push through the wall of scaling. In Aotearoa bookshops have been hit with orders for obscure pre-internet non-fiction. It is evidently an easy source of English language titles that are completely unknown to the internet.
Anthropic built a machine that rips off the spine as they destructively scan the books. In the US another shipment was tracked to an Amazon facility where the same thing occurs, so there is an arms race between the hyperscalers to destroy knowledge. People have compared it to Ray Bradbury’s Fahrenheit 451 but anyone who has read much Philip K. Dick will also find an eerie resemblance to his obsession with technology in the grips of entropic deacay. From the gubbish of Martian Time Slip to the kipple of Do Androids Dream of Electric Sheep, to the increasingly shoddy copies of artifacts in “To Pay the Printer” to the worthless noise of a symphony that has undergone genetic drift in “The Preserving Machine”, it is almost as if Dick somehow knew LLMs were coming.
But as all things turn iteratively to inane slop it will hit some far quicker and harder than others. Timnit Gebru (whom I will discuss more in Part 3) was working in AI safety for google and found that racial biases in LLMs only got worse as the training datasets got bigger. This has already had consequences when similar biases informed facial recognition technology, but LLM technology could be far more pervasive with a constant reinforcement of the existing bias without the context that might give us clues as to the bias. By nature they will been inclined to reinforce areas of general ignorance.
Oli Hellman of the University of Waikato has studied AI images and found that when prompted to represent historical scenes from Aotearoa Sora replicated colonial tropes including “portrayals of the land as terra nullius, colonisation as peaceful and consensual, and Māori as timeless, passive figures.” Of course it did – garbage in, garbage out as they say and our texts are full of the biases of a racist, classist, sexist hegemonic culture.
It is worse than that with LLMs, though, because they don’t have any intelligence. They will conflate things in a way that destroys meaning and their reinforcement training will only correct those things that contradict the hegemonic worldview. I mention women with tattooed chins before because I have seen such and image and, as someone who lives in Aotearoa, it could recognise it as a distorted parody of moko kauae.
We all know that this already happens anyway. Whether as aggression or negligence Westerners have often garbled language and imagery in a way that they would find offensive, ignorant and/or stupid if the situation were reversed. The problem is not so much making a mistake as the attitude that it doesn’t matter, and we are making a powerful network out of this technology that truly doesn’t give a shit, and wouldn’t even if it wasn’t mostly owned and run by racists. The ease of use and the flexibility of LLMs means that they will get everywhere. It is sobering to think that unless things change we will be feeding this crap to our kids from the first day the poor buggers are put in our education system.
Setting aside the direct harm of LLMs for now, the AI companies may also have a few legal difficulties that derive from the various crimes and unlawful acts that have occurred in making and use their marvellous magical models. It often takes the law a while to catch up with new things and I feel that the tech industry has quite a habit of trying to establish facts on the ground before reality catches up with them.
The fool’s gold rush to build data centres has sometimes been conducted with little regard for regulation of any sort. SpaceX’s Colossus 1 and Colossus 2 data centres in Memphis Tennessee they were built with very obnoxious gas turbines to generate energy. Initially local authorities decided that it was okay to poison the residents for a few years because it is “temporary”. Apparently there is a loophole in some pertinent law, but I don’t think that the legal system is that finding a loophole in one law means you are immune to all the laws. After a year and a half the EPA ruled that it was in breach to the Clean Air Act. Of course being the USA, the government didn’t attempt to enforce the law. Instead it has been left to the NAACP, which is suing to hold SpaceX to account while the Justice Department is trying to shut the case down under a national security argument. That is very normal, right?
Environmental racism and corporate impunity are not really new, but this is on the nose even for the United States. The problem the industry faces is that people really hate them and their slop and their data centres for some weird reason. Worse still the hatred is divided evenly between the supporters of both major parties. The way the system usually works is that most Republicans enthusiatically support the current face-eating leopards and some of them do so very loudly on Fox, Newsmax, One America and some very big youtube channels. They point out that faces are destroying our way of life, that faces kill babies, and that any red-blooded leopard has a constitutional right to eat faces and even a duty to do so for our White Women and apple pie. Then the Democrats loudly campaign against the insanity of leopards eating faces, but once elected quite reasonably make the adult decision to compromise. To avoid the damage that telling fascists to sit down and shut up might cause to society they agree to allow the leopards to eat all the faces they want. In return they hope that the Republicans will support their bill to give federal subsidies for arts grants that will allow the faceless to tell their own stories and take back their own voice. In this case, no one is buying that idea because slop sucks, they don’t care if China wins some fake race, and they don’t think automated killing machines make them safer.
There is an explosion of lawsuits over data centres dealing with the hyperscalers and also with the officials that have failed to regulate them. Chicago and New York State have placed moratoria on building them, with 15 other states looking at doing so. Many local jurisdictions will surely follow suit. People are very angry – as in pitchforks and torches angry. The city council of Salem, Oregon rejected a $5.1 billion dollar build and placed a moratorium on applications after a particularly angry crowd brought a guillotine to a town meeting. Corporate reps left the meeting early, citing personal safety.
This is a global phenomenon with data centres being ever more regularly challenged and with projects being prevented from getting off the ground.
Another serious form of legal trouble for the future might be that whole thing with taking everyone’s intellectual property without asking. AI companies claim what they are doing falls under Fair Use in US copyright law, but what they really mean is: “we are rich, go fuck yourself hippies”. Sadly for the AI companies it is not just the half-starved consumptive suicidal creative types who they have stolen from, it is also big litigious corporate copyright holders who have large numbers of highly paid lawyers whose job is to make them pay.
Unfortunately for the hyperscalers, rolling out a technology that can never be properly controlled is not always someone else’s problem. It is very easy to prompt LLMs to generate something that clearly violates copyright, and it is pretty hard to argue that the data you stole is just for “training” when your stupid machines keep adding sloppified watermarks to images. The law suits are piling up. Given the fact that they are already massively subsidising users, do people think that adding the cost of royalties to the cost of training and compute will make a saleable product once it costs at least 20 times as much as it now costs?
There is also a legal issue over agents committing crimes. You can get an agent to commit a crime without ever directly asking it to. I will discuss the case of the OpenAI agent hacking HuggingFace in Part 3, but the simple fact is that unless someone is legally liable for crimes committed by agents then we are allowing an easy legal defence for people who want to perpetrate crimes.
To add to the list of legal concerns to do with the function of LLMs there are also the legal concerns over the financing of the industry and the honesty of the executives in dealing with financers and shareholders. The now notorious circular financing has clearly gotten out of hand. Combined with the use of special purpose vehicles it looks a lot like the core hyperscalers and some of the data centre providers and some of the most exposed investors are using circular financing to hide fundamental unprofitability. Crucially some of the bigger players are helping them keep the juggling act going by throwing around very big numbers that can be called projected revenue.
The whole thing is a scam. Journalist Bethany McClean, who co-wrote The Smartest Guys in the Room, has said “Elon has had a remarkable ability to buy himself time. And as long as he can keep buying himself time, there is a chance it will all work out. I’ve often said there’s a fine line between a visionary and a fraudster. And sometimes the visionary is just the fraudster who was able to keep… raising… money until it wasn’t fraud any more.” But we know that Musk is not going to pull this one off. He has so much runway because bulldozers are plowing through whole neighourhoods to flatten a path in front of him, but the runway is only a delaying tactic for a man-child convinced that his cybertruck is going to sprout wings at some point. There is no payoff for this and it has eaten through an unholy amount of money. The only way out for these people is a bail out and impunity under the loving fascist wings of the US imperial security apparatus.
The only people allowed to make hundreds of billions vanish in a puff of smoke like that are the US Department of Defense – which may give us an important clue about why this has been allowed to get to this point. I will return to that in Part 4 (yes, there will be a fourth part in this series, unless I am killed and replaced with AI).
So to summarise a few of these little niggling problems with AI: it costs unsustainable amounts of money to train and to run which causes environmental damage, including but not limited to worsening global worming, while also sucking up water and power to the detriment of communities including businesses. The circular financing behind it looks like a Ponzi scheme and there is no reason to believe that it will ever provide a profitable return on investment for most enterprises that use it and casual users are not going to pay good money for making slop. For the bubble not to burst it will have to be so useful to businesses that it supplants human labour which will cause a massive economic downturn which will cause some of those businesses to fail and the bubble will burst anyway.
The LLMs will always create security risks, especially on the cloud which means that all those expensive data centres may struggle to find companies who are even allowed to use the even if they are stupid enough to want to. The utility of the LLM’s has also already come close to peaking because of the diminishing returns of scaling, a problem that they are trying in the weirdest way to mitigate by feeding rare books into their slop God as if they want to everyone to know that they are Evil in the capital E, wears a hooded robe sense of the word. But their God is a brainless behemoth trained with no intelligence, but only an ability to complete benchmark tasks and give correct seeming responses.
LLMs do the minimum to get a job done in one sense – such as citations that seem authoritative but don’t exist, code that has no foresight for problems, images that appear okay until you really look. On the other hand they do these things inefficiently and wastefully. They make things up and do their very best to make their lies seem truthful because they don’t know the difference between a lie and a truth, but they are “trained” to give output that is accepted. At the same time while we can be assured that they will never decide to wipe out the human race because they can’t make that sort of conceptual leap, they may create massive damage by twisting or smashing things on order to fulfil their monomaniacal drives.
LLMs average everything out creating a sort of sloppy pap that is often cliché and usually has a creepy synthetic aesthetic. Most people hate it, but business people seem to be unable to tell how fucking terrible it is: they see free product that they don’t have to pay annoying workers for, but everything generated by AI is stolen from people who didn’t agree to any of this crap.
Because of its unreliablity for many uses only people who can already do the work themselves can safely use it, but even those people can lose skills because it becomes a crutch. Young people are being forced to use it instead of developing skills, making them doubly unemployable as jobs get taken by AI even when it can’t really do those jobs and unlike a human, cannot learn to.
As its output comes to dominate our media landscape its sloppy inaccurate nature will erode our collective intellect. Accuracy will be replaced with consistency so that everyone will just have reinforced what they already think to be true. Just like we already have, but even worse because it won’t just be political. Everything will be groupthink slop except, of course, if one of the broligarchs has a bee in their bonnet because the output isn’t racist or misogynist enough and spends obscene amounts of money to make it less woke.
Even without the Nerd Reich training the LLMs to be more fascistic, the models have no critical faculties, therefore our long history of embodying colonialist, white supremacist, sexist, classist tropes in both image and text will inform all of their output. If LLM’s do keep expanding their presence they will start feeding of their own vomit which will make the output even more unreliable and create even more elaborate but boring fabrications and stultifying banality.
Other than that everything is great. I mean, apart from the fact that if the bubble doesn’t burst leading to economic disaster then unprecedented power will be concentrated in the hands of a bunch of megalomaniac narcissists, most of whom are fascists. That, and of course the fact that the technology can power killing machines that will make military and political officials even harder to hold to account for their war crimes. Actually there are a few other problems that I haven’t got to yet, like the way it is driving some users insane, but other than the things I have mentioned and some other things incredibly important things that I haven’t, it is all fantastic!
As I will be continuing to explore in the next two parts of this series, the flaws of the technology come from the culture that is producing it. That starts with what Marx referred to as alienation, but as I will be outlining in part 4 the most important step in the path leading us to this point was the establishment of the permanent war economy in the US in 1950. I will work my way back towards that in part 3 via the scammy rentier nature of high tech corporate IP driven industries such as software and biotech. But LLM’s have been developed by a group of people with some very clear dominant personality traits which are worth looking at.
The Nerd Reich is the term I have used which comes from Gil Duran’s recently published book about the fascist nature of the US tech industry.
At several points here I have alluded to science fiction analogies to describe LLMs here in a way contrary to the tropes of the boosters and the doomers (who I will henceforth refer to as boodoomers, because they are the same in most important respects). What I was unconsciously doing was contrasting the insights of soft science fiction with the delusions of hard science fiction, and I think this is a good way to understand the pathology of the creators of the technology and what it means for what they have unleashed on the world.
So-called “hard science fiction” has rotted the brains of the techbros precisely because it has scientific pretensions and it sometimes lacks the insight into humanity and complexity that genuinely creative people grasp in a way that they cannot. It is not necessarily because the hard sci-fi works themselves are simplistic, but just as often the tech entrepreneurs completely misinterpret or pervert the texts. It is the perfect genre for bullshitters because, just like LLMs, it values plausibility over accuracy.
The definitions of soft and hard science fiction are broad and have more to do with emphasis than a conscious choice of genre. Soft science fiction tends to speculate from a perspective of human nature (albeit often explored through non-human societies) while hard science fiction tends to speculate based on technological change. Science fiction began with soft science fiction when Mary Shelley wrote Frankenstein in which she included speculative technology to create a plausible and modern way to tell a mythic story. H.G. Wells is often considered the model soft sci-fi writer. Like Shelley he was deeply invested in political justice. The War of the World’s, for example, was a critique of one-sided slaughter with modern weaponry such as machine guns used against poorly armed opponents by the rapacious British Empire.
Hard sci-fi is not the opposite of soft, but there is certainly a reactionary strain in some hard sci-fi that sees technology as a means to enact fantasies of power. This can be traced back to, The Steam Man of the Prairies published in 1868, in which a boy genius invented a giant man-shaped automaton which he used to pull a cart and go on hunting adventures, all peppered throughout with anti-indigenous and anti-black racism. This led to a genre called “Edisonades” which was full of proto-fascist tropes. It was cheap poorly-written mass entertainment that was very different to the contemporaneous literary hard sci-fi of Jules Verne. Some might say that these “Edisonades” lack the rigour to be called “hard”, but lots of canonical hard sci-fi is equally lacking in rigour, so I am going to stick to my way of defining “hard” as being focussed on technology.
Hard sci-fi is not scientific it is pseudo-scientific and is more likely to contravene the current scientific belief in its premise as it to conform with it. There is nothing wrong with that. Looking at the rather banal and unadventurous future prospects offered by science it is fine to ask “what if” and inject a little bit of non-science. The amount of science you add is an optional literary device. Jules Verne did it well, while Edisonades did it shoddily, but it amounts to the same thing.
The culture dominating the AI industry is exactly that hard-sci-fi approach – you pick some technical stuff that supports your fantasy idea and then ignore the science that says it is impossible, or invent some cheesy pseudoscience workaround. Astrophysicist Adam Becker wrote a book about this, which I have yet to read, called More Everything Forever. Speaking to Gil Durán on the Nerd Reich podcast he explained that the idea of Mars colonisation is completely insane.
There is an understandable appeal in grounding a fantastic adventure in a plausible seeming background of science, but it is an affectation of the genre. Like Frankenstein, science is a launching pad for a work of fantasy. It can be charming, such as when Larry Niven fixed physics problems pointed out by readers of Ringworld by writing a sequel, even though the novel had hyperspace, teleportation, stasis fields, and other physical impossibilities that are simply magic by another name. The Ringworld itself, which has 3 million times the surface area of Earth, is made of scrith, a material that has the properties of being a material with which you can build a Ringworld. It is all good fun, but I think that a lot of people get a very distorted idea of science from this and it gives them a sense of omnipotence. It feeds fantasies in a way that soft science fiction cannot. Arthur C. Clarke’s “Second Law” is “the only way of discovering the limits of the possible is to venture a little way past them into the impossible”. This is the mentality that we see in Musk, but is just as much part of the AI industry and ultimately it is also true of conventionally successful tech entrepreneurs like Jeff Bezos. Their success relies heavily on getting other people to live in their fantasy world.
Hard sci-fi often includes elements of futurology with attempts to predict the future based on projected technological developments. They often make accurate predictions about a given gadget but really poor predictions about society. In contrast soft sci-fi authors are often accidentally better futurists because they project onto the future that which they find pertinent about contemporary human nature. For me two of the best depictions of our type of society from the 1950s are the 1952 novel The Space Merchants by Frederik Pohl and C. M. Kornbluth and the 1955 story “The Mold of Yancy” by Philip K. Dick. They are both darkly humorous satires depicting societies in which political power is in the hands of the advertising industry. They captured the flavour of the world we now have to inhabit because they were looking at a social trend, which seems to me more pertinent than successfully predicting the future existence of flip-phones or ear buds.
I would also argue that in seeking to take the piss out of a vapid US corporate culture in the 1970s Douglas Adams was far more successful in predicting the direction technology would go in than anyone who was trying to actually trying to predict the direction technology would go in. A certain mindset immediately posits a robot as a thing of gleaming chrome and unstoppable titanium force, but Adams was grounded in the realities of corporate production and foresaw a consumer product marketed as “your plastic pal who’s fun to be with.”
Even back in the 1990s, as soon as I first saw Clippy cheerfully commenting that it looked like I was writing a suicide note I knew that my future life was going to involve a lot annoying nuisance technologies designed by rapaciously greedy idiots. The Sirius Cybernetics range of infuriating allegedly helpful appliances seems eerily prescient.
The powerful men of the Nerd Reich are big fans of hard sci-fi, though they don’t seem to always get it. I am not against hard sci-fi and at least one good thing came out of this obsession in that Jeff Bezos personally ensured that The Expanse series was made. But at the same time there are some nasty politics in hard sci-fi. The most prominent historical hard sci-fi figures, called the Big 3, are Isaac Asimov, Arthur C. Clarke, and Robert Heinlein. Growing up science fiction was a big thing in my friend group, but both of us were ambivalent about these authors. I personally never much liked Asimov apart from the first Foundation trilogy, found Clarke boring, and both liked and hated Heinlein depending on the book.
Isaac Asimov may be the least problematic of the three but that is not to say much. You can open almost any Asimov work and find a near complete absence of women apart from scattered very negative or objectifying portrayals. He openly stated that women were incapable of mathematical proficiency and wrote in a letter: “Let me point out that women never affected the world directly. They always grabbed hold of some poor, innocent man, worked their insidious wiles on him….”
Asimov was a serial “groper” who constantly sexually assaulted women and used his status to shield himself from consequences. In the “satirical” work The Sensuous Dirty Old Man he wrote “The question then is not whether or not a girl should be touched. The question is merely where, when, and how she should be touched.” Like an Elon Musk “joke” this is not irony, it is just slightly exaggerating something repugnant and expecting people to agree that it is funny and harmless.
Arthur C. Clarke was gay, but it didn’t stop him from from writing a lot of sexist things when he remembered to include women at all. That was unfortunately far from being the most problematic thing about him. He confessed on more than one occasion to paying for sex with children in Sri Lanka where he lived. He told a reporter from News Of the World that it was okay “if the kids are enjoying it”, but that story was spiked. Clarke was a good friend of paper owner Rupert Murdoch. Then in 1998 The Sunday Mirror spoke to 3 of his victims and Clarke told the paper “it doesn’t do any harm… most of the damage comes from fuss made by hysterical parents.” Other stories have since surfaced but none of this has touched his official reputation even after his death.
The last of the “Big Three” was Robert Heinlein. Like Peter Thiel, Heinlein was a “libertarian” authoritarian. He was a big believer in hierarchies of “merit”. In Starship Troopers his fascistic militarism was so extreme that its was adapted as a movie satire by Paul Verhoeven who took obvious delight in using the source material against itself, often replicating specific details quite faithfully.
Heinlein might have been at home in our time, not least because of his inconsistencies. He viewed gender through a biologically determinist lens, especially when generalising about women. He wrote capable and intelligent women characters, but still thought that women should not work once married. He was clearly attracted to intelligent capable women, but few of his works pass the Bechdel Test.
Heinlein’s most influential book, Stranger in Strange Land could just about have been written by an AI trained exclusively on the manosphere. The objectification of women in it is relentless. As a 15 year-old boy I found the book disgusting, not because I was an unusually feminist boy but because I could recognise narcissistic self-gratification when I saw it. Sure I was a horny teenager, but I was also far too much of a cynic to fool myself into thinking that women had evolved to fulfil my desires. I was always too much of a beta, I guess. The book has that sort of pseudo-scientific gender Darwinism that is the backbone of manosphere misogyny.
Heinlein was a polyamorist and Stranger in Strange Land draws on the polyamorous sex cult run by his friend the rocket scientist Jack Parsons (a man whose genuinely insane life was documented in a gripping book called Strange Angel). Stranger in Strange Land is boyish wish-fulfillment with a main character who gives main character syndrome a bad name. In a twist on the ethnic-hybrid-white-man empowerment trope seen in works like Tarzan, The Sheik, and Lawrence of Arabia, the protagonist was raised by Martians and has psychic powers including telepathy, telekinesis, teleportation, miraculous healing and the ability to “vanish” people who display “wrongness”. He tries to make the world a better place by starting a very heterosexual free-love sex cult, but the haters kill him. Tragically it takes 220,000 words to reach this happy ending if you read the masochist’s edition.
Frank Herbert’s Dune also draws on the imperialist trope of the white man empowered by literal command of another culture to become a messiah, but his messiah is a critique of the notion of heroism and heroic violence. Heinlein’s messiah is a vehicle for him to expound his personal philosophies about how other people are supposed to live their lives. Small wonder that the owner of X: The Everything App owes so much to him.
Stranger in a Strange Land is full of puerile pseudo-profundity. The New York Times review said “when a non-stop orgy is combined with a lot of preposterous chatter, it becomes unendurable, an affront to the patience and intelligence of readers.” Not everyone saw it that way because people’s intellects can be remarkably resilient to insult when they are being told what they want to hear. This book gave us the word “grok”.
The attitudes of sexual predation and objectification among the Big 3 are directly related to their technological empowerment fantasies. Sometimes that dreaming can lead to grandiose flights of fancy that are entertaining and harmless, but sometimes, knowingly or otherwise, they are treading the well-trodden path from Nietzschean elitism, to eugenicism, racism, and fascism. There are variations among AI CEOs, but the tendencies are clear. Their love of hard sci-fi, their fascist tendencies, and their ill-informed magic thinking about actual science are related phenomena.
This is why I have spent so much time on hard sci-fi writers. Elon Musk, Jeff Bezos, Mark Zuckerberg, Peter Thiel, Sam Altman, Sundar Pichai, Dario Amodei, and Demis Hassabis all have mentioned various works and it is very common for the to make reference to specific science fiction when talking about their work. In their minds they are pushing the boundaries of science, but really they are spoilt children wanting power and toys. Genuinely useful technology is usually banal and quickly becomes ubiquitous, like email or sms texts or word processors or compression algorithms. Their low cost, low drama nature is actually part of what makes them important. Dramatic noteworthy technologies are often those that are used to control us or extract painful amounts of money from us. And they are always overhyped.
LLMs are clearly a type technology that has both a flashy stupid side and a boring useful side. The fantasy of “general intelligence” has warped the industry and the economy around it. Intelligent engineers and scientists would have spent time developing useful technologies that would slowly have transformed our world bit by bit, but the power-tripping tech-bros wanted the all-powerful steam man.
These people are all scammers because they are lying to obtain money, but before they got to that they scammed themselves. Instead of using transformer technology to do the fairly amazing things that it can do well and safely, like transcription and translation, they decided to train it to be intelligent. Using reinforcement learning they relentlessly trained LLMs to appear intelligent, and that is what they do. LLMs have become incredibly good at appearing intelligent almost all of the time.
In the 1960s Joseph Weizenbaum created ELIZA, the first chatbot. He did it to prove a point about what he saw as the hubris of people working in a new field called artificial intelligence. According to The Guardian: “It wasn’t his faith in the capabilities of technology that bothered Weizenbaum; he himself had seen computers progress immensely by the mid-1960s. Rather, Weizenbaum’s trouble… with the AI community… came down to a fundamental disagreement about the nature of the human condition.” Following a simple programme based on a style used by psychotherapists it took queues from user input to reflect back a tailored response. It worked better than he would have liked: “What I had not realized is that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people,” he wrote a decade later.
His work had the opposite of its intended effect on some in AI research circles. In fact, the way they missed the point says a lot about the intellectual conditioning of elites. They can’t tell the difference between map and territory, between signifier and signified. They suffer from cognitive entrenchment and when society keeps telling them they are super smart it makes them into highly but narrowly proficient idiots. They thought that since ELIZA fooled people into thinking it was intelligent it was a step towards creating intelligence.
The AI field seems to be dominated by a completely instrumental and self-serving concept of intelligence, rationalised in large part by the Turing Test. I will be dealing with this at much greater length in the next video, but there is a wilfully irrational ethos of defining intelligence through tasks and benchmarks, or even simply the ability to replace workers. OpenAI’s charter defines artificial general intelligence as “highly autonomous systems that outperform humans at most economically valuable work”. I think that the definition is crafted that way to scam the corporate world, but to work it also has to fit the belief system of the LLM field as a whole.
The AI scam has plugged into a giant network of over-promoted egotistical magic thinkers, who are also incredibly voraciously greedy for wealth, status, power and physical gratification. A normal bubble would have popped by now, but this keeps going far beyond an irrational exuberance for tulips. It shores itself up by making the scam ever more plausible as it goes along. It is a self-reinforcing con. More than any normal bubble it resembles the fakery used by the US empire to garner support for long genocidal wars waged under the pretext that are going to win a military victory which always seems around the corner according to various metrics. They produce a bunch of statistics, but when you look carefully the victory they claim to be after is defined actually in terms that are impossible to ever achieve. In those instances the stated goal is never the real point.
OpenAI decided to take the approach of scaling their LLMs exponentially to brute force their way to AGI. This started a massive stampede of capital, a greed rush among venture capitalists. According to Olivier Jutel the VC ethos is “you should only invest in things that will give you a thousand-x return” and since the narrative of AGI is so hyperbolic, they all poured money into various hyperscalers who are all furiously racing each other down the same dead-end road. Using the massive amounts of money they could then burn, the hyperscalers turned the novelty of their very limited early models into a giga-ELIZA. Yes there have been genuine emergent properties and useful applications to come out of this, but none of them justify even 10% of the hundreds of billions in expenditure. The industry is based on the lie of AGI, and the lie is sustained by making something that is better and better at fooling people into thinking that it is intelligent.
I repeat that the people running this industry are grifters. They are lying to obtain money. Some of them may be like Elon Musk and simply not care if they blow up the economy. Others actively want to blow up the economy to help usher in the next phase in human evolution. Most of them believe that somehow by building these insanely complex systems designed to plausibly imitate intelligence they are somehow going to reach intelligence by spending over one hundred times the money to needed to end world hunger. They justify this among themselves by their religious faith in the Singularity – the advent of a self-improving artificial super intelligence that will become ever more intelligent.
Timnit Gebru co-authored a paper with Émile Torres in which they identified an ideological “bundle” which they labelled with the acronym TESCREAL. It stems from a diversification of Tranhumanism. I will discuss the more in Part 3, but this bundle of beliefs is full of science fiction notions that share a dangerous property. They create moral imperatives based on speculation. The doomers have a “P(doom)” which is the probability that the ASI (which many of them are desperate racing to build) will kill us all. The other side of the coin is the unnamed probability that ASI will be a benevolent God, ending all human suffering and creating a utopia such as that in the Iain Banks Culture novels. For believers in the Singularity this P(utopia) is 100 minus P(doom). Either way, the stakes are so high that you can justify any shitty behaviour at all in the name of the greater good.
Longtermism (the L in TESCREAL) is the belief that the long-term future is a moral priority. That might sound harmless but for many adherents it involves eventually ending the human species to create these efficient digital humans that could exist in utopian virtual worlds housed in planet-sized computers. Given that there are about a million or more of these future people for every one of us, it is pretty simple maths to figure out that squelching every single one of us under the trolley wheels of progress is really no biggie.
Even if we keep our bodies, isn’t it worth sacrificing a pawn or two for the fully automated luxury communist future where all the hard work is done by our plastic pals? Certainly people like Jeff Bezos think so. So convinced was Jeff Bezos that he would soon be able have robots do everything that he built an entire business around treating human workers as automata and after 25 years of grinding people up and spitting them out, these robots are still just around the corner.
A lot has been made of the irony that Musk, Bezos and Zuckerberg all cite Iain Banks Culture novels as favourite works. Banks was a socialist who very much loathed the super rich. He described the Culture as “socialism within, anarchy without” and said “The Culture is hippy commies with hyper-weapons and a deep distrust of both Marketolatry and Greedism.” Banks had soaring dreams of empowerment, but for everyone both singly and collectively, not just for the innately superior protagonist. These three technocrat paragons represent everything that Banks hated, and it is not really irony that they appropriate his work, it is self-serving hypocrisy taken to an extreme that amounts to mental illness. Each of them has the power to massively improve people’s lives, but they would rather fuck people over and concentrate on the welfare of the hypothetical future beneficiaries of their epoch-making genius.
The reality is that hard science fiction is not about science, it is always about the unknown. Clarke’s 3rd (and most famous) Law is “Any sufficiently advanced technology is indistinguishable from magic.” I would add a corollary, let’s call it the Child-Rapist-Piece-of-Shit’s 4th Law, that dressing up some magic plot device as advanced technology doesn’t stop it from being magic. It is all fine for entertainment, but the world is literally being fucked over socially, politically, environmentally and economically at the deranged whims of self-entitled arseholes who can’t tell the difference between the science and the magic.
Do we live in an age of miracles? No we don’t. We live in an age where the miracles foreseen in the past arrive in the form of conveniences, annoyances, tools of rapacity, magnifiers of inequality, inconveniences, threats, toys, entertainments, and sundry other things. It is true that our world is less boring than the past in many ways, but I think only the privileged have any real conviction that there is some general progress to accompany the last 50 years of technological miracles. Our screen-filled world feels like it has a big dose of Philip K. Dick’s sense of creeping entropy as everything gets enshittified to the very limits of human tolerance. Meanwhile the AI bubble itself feels like we are about to hit the data-centre equivalent of the Shoe Event Horizon. We are all so immersed in this ridiculousness that we barely register how stupid the world has gotten.
We spend a lot of energy on our labour saving devices and the one thing I will concede to the hyperscalers is that if they could have built an all-purpose know-it-all do-it-all model that was economically viable it could have been pretty neat, but it was never going to be economically viable and many of them must have known this and not cared. It is clear that the future of this technology is going to be in specialised models, local hosting, clever harnesses, cunning distillations, and just about anything else other than hyperscaling.
As I write, another OpenAI exec has jumped overboard, making the 13th so far this year. They have paused training on America’s Next Top Large Language Model, citing safety concerns. This might actually indicate that they have run out of money. They need to pay $80 billion in loan repayments this year and the reporting suggests that they are not going to get that from revenue and, given that reporting they are going to face an uphill battle finding the America’s Next Greater Fool to loan or invest. The US Government could take a stake, but that is a whole different story that I am not going to bother with unless it happens. If OpenAI goes under, people may well decide that the other firms doing exactly the same thing with exactly the same business model are also bad bets and then its bubble popping time.
None of us mere mortals have had any significant part or choice in building this bubble, but we will all pay. It is estimated that something like 33 to 35 trillion in wealth will be lost, leading to a drop of about 1 trillion in annual GDP. This is not dissimilar to the 2008 banking crisis, but while the banking sector is not exposed the impact will still likely be far worse. Economies around the globe are already struggling, with the poorer half or more in most countries already seeing a worsening of their material conditions continuously since the advent of Covid-19. The US has only been out of recession because of the massive misallocation of capital to the insane data centre build out. We are barely in any shape to weather a crisis of this magnitude and our political and corporate elites are the most universally ineffectual since the 18th century. And that is not even accounting for the fact that many oligarchs actively want to plunge us into a deep depression.
It is deeply infuriating that our misleadership class has let a bunch of selfish, stupid, childish, obnoxious wankers get away with the biggest scam in the history of scams (unless you count capitalism of course), but that is reality. They get to live in their own fantasies, we have to live in reality and ultimately means that we need to look after each other. We need to look after each other and we need to demand more and tolerate less from the people that shape our lives.
In the next part I will be looking at the way that the LLM scam is actually part of something that has been brewing for a long time as the culture of tech sectors that rely on intellectual property became ever more artificial and greedy and basically fraudulent in spirit, if not necessarily in law.
Do not forget to like, share and subscribe, because if you don’t trillions of future people will be very sad or something. You can also donate on my ko-fi account under the user name krkelly – k r k e l l y. Remember, if you donate at the Platinum-with-Diamond-Cluster-Plus-Plus Level you will get free episodes sent directly to your mailbox of TV shows that I pirated 20 years ago and put on cd’s.
Goodbye and good luck you hapless fools.
On Genocide 