Erik McClure

For basically my entire software engineering career, I have argued that programmers should strive to write as little code as possible. I have long been a proponent of using type systems to eliminate as many classes of bugs as possible, and a side-effect of strong type systems is that it makes fuzzing and static analysis easier, which can further reduce bugs. I have a vivid memory of arguing with a web developer at a convention nearly a decade ago about this, who was annoyed whenever the compiler “got in their way” and shockingly admitted they didn’t even understand the point of type systems at all.

There is a certain kind of person who, instead of thinking about the best way to solve a problem, will simply Executive Function their way through it because they have infinite spoons. These kinds of people are often celebrated for their “productivity” despite their actual output being extremely questionable, especially in open-source, where convincing people to do work for free is extremely difficult. So, we ended up with open-source projects selecting for people who had lots of spoons but little judgement, and could ship an impressive number of features that had an equally impressive amount of security flaws, like whatever the GPG devs are doing.

I have noticed there are two key types of people:

1. Gosh, this software is stupid, why would you have to do this same thing 20 times *proceeds to do it 20 times*
2. Wow, do I really have to do this 20 times? Surely not. *finds out solution* ah, right that makes more sense

- Orchestructive

Some people keep asking why all the open-source projects are full of fascists, without realizing that the structural incentives of open-source virtually guarantee this result. Open-source has no way to pay anyone, so the currency in open-source is how productive you are. This worship of executive function has trapped open-source projects into deals with the devil, like how the entire Rust ecosystem is propped up by David Tolnay’s batshit insane metaprogramming macro system that reimplements the entire rust compiler’s internal syntax parsing outside the compiler. When they tried to do something about this, the attempt to replace the macro-based metaprogramming with something more sensible was sabotaged by David Tolnay, which he even indirectly admitted to in an extremely weird non-apology.

The reason this keeps happening is because open-source projects are competing for one extraordinarily scarce resource: a programmer’s spoons (time and energy). They are forced to compete because people only use software that fixes their problems. They don’t care how, only that it does. Thus, whoever ships something wins, no matter how shitty it is. In academia, you publish or perish - in open-source, you ship or sink. This pressure forces open-source projects to rely on anyone willing to help, and heavily incentivizes turning a blind eye to any questionable beliefs held by those people. This pressure is immense, and those who have not attempted to maintain an open-source project likely aren’t aware of just how bad it is.

Nazis, bullies, and sociopaths are extremely aware of this, which is why a catastrophic supply chain attack nearly succeeded by simply offering to help one overworked guy on a project nobody cared about that happened to be shipped with every copy of Linux. This will keep happening, forever, because people only care if you ship something, and people with infinite spoons can ship features and fix bugs faster than an engineer trying to avoid writing bugs in the first place. A nazi has a near infinite fountain of important but neglected random libraries that nobody is helping. Politely asking companies to contribute back is never going to work, because many of the projects with a nazi problem aren’t even commercial products, they’re completely free!

So, any prospective nazi simply needs to find a useful-but-neglected library, and start fixing bugs. The project, desperate for any kind of help, will overlook any red flags (if there even are any), and then the new contributor starts inviting their friends to help contribute too, and now the open-source project is a nazi bar. The structural incentives make it extremely hard to prevent this from happening, because you have to either catch the first nazi (which can be very difficult, and involves turning away much needed help), or after you find out, you have to evict a significant chunk of your most productive contributors all at once, which is very hard, and they might just fork the library and win over the userbase, because time and time again, users have shown they really only care if it works, not who makes it. Sometimes, even if a user does care, they don’t have a choice, because there is only one library that fixes their problem, and they’re too busy fixing a different problem to invest time in maintaining a fork.

AI is a way to get infinite spoons. It has basically no judgement, but also never gets tired, or bored, or distracted. It is a way to turn compute into executive function. If you attempt to use it without giving it very strict guidelines, this goes about as well as telling a JavaScript dev who doesn’t like type systems to write a kernel module in C. Unfortunately, open-source projects are so desperate for executive function that they have huge structural incentives to accept AI-enabled contributions, for the same reason they’ve been accepting contributions from nazis. Of course Debian is accepting AI contributions now, the whole ecosystem is being pushed in that direction. If they banned AI contributions, then the most productive people would just be ones secretly using AI and not telling anyone, because open-source cares about perceived productivity.

The clear and unavoidable result is simple: Open-source projects will either accept AI contributions and piss everyone off, or deny AI contributions and be dominated by people who lie about not using AI and then secretly use it anyway. I’m not saying AI usage is inevitable - I’m saying the structural incentives of open-source make it inevitable. In order to stop this from happening, you must restructure the incentives around open-source projects away from the raw spoon-count of contributors. In a way, AI helps with this, because now we actually have a baseline to compare a contribution against. If an infinite slop machine with no judgement could produce the same results as someone’s PR, even if they wrote it themselves, those contributors should not be celebrated more than contributors that help the project by saying no to bad design that avoid creating those problems in the first place.

The immensely frustrating part of this is that people keep insisting on having their cake and eating it too: they want nice things, but absolutely refuse to put into place structural incentives to make the nice things happen. They want high quality open-source software, but refuse to pay the huge team of software developers necessary to make it actually happen, then get mad because devs rightly refuse to spend their free time fixing garbage code that has to fight a bunch of other poorly maintained libraries and OS APIs just to exist. A Nazi, on the other hand, will be happy to deal with lots of dumb nonsense, either for the purpose of trying to take over the project, or because they are actively being supported by far-right interests. They understand that being a thankless maintainer of a sufficient number of boring-but-important code can itself be weaponized against a particular ecosystem.

Volunteer run projects often make very poor engineering decisions because they’re volunteers, and it does actually require some kind of monetary support to convince people to go through all the effort necessary to do actual good engineering, and build tests, and think about things that aren’t on the golden path, and deal with hardware edge-cases, and all that. Despite this, almost no open-source project can survive on individual donations (with only a few exceptions) and this has been the case for years. The ones that are self-sufficient almost always either accept corporate sponsorship, or are a corporation that sells something, and then people act surprised when NixOS gets sponsored by Anduril. What were you expecting? The structural incentives force any open-source project to essentially take any money it can possibly find that won’t cause an outright rebellion among its users! It would be great if more governments had something like nlnet that funded open-source directly, but that largely doesn’t exist outside the EU.

I’m so tired of everyone’s response to this being “well, people should just be willing to stand up for what they believe in and starve instead of accept dirty money!” Almost nobody actually does this, almost everyone will take the job at Evil Corp Inc. to avoid being homeless. Those that are willing to go to such extremes often refuse to engage with any power structure that might give them the ability to actually change anything. An alarming amount of so-called leftist thought seems designed to disempower progressives by convincing them that acquiring the power to change anything is evil, and given the current state of things, it appears to have been quite effective.

Someone I follow on mastodon, who wishes to remain anonymous, pointed at this video discussing capitalism in America. The video was about the general act of creation, but as they pointed out, it applies to open-source as well: “the solution is that we go back to creating stupid, imperfect, small programs and sites, slowly, at a reasonable pace, and meet people one at a time who find this stuff useful”. The incentives have to change. Stop trying to solve systemic issues by demanding everyone suddenly becomes more individually responsible for no reason. Give people an incentive to change their behavior. Push them in the right direction.

There is a crisis in the open-source community right now. Several blog posts have been discussing the blatant exploitation of open-source developers by corporations, especially AI scrapers, and how open-source work might be enabling warcrimes. There has been a rising bubble of discontent with the current open-source model, and especially the nature of open-source licenses that are OSI-approved.

Developers, however, have been upset with the status-quo for years. Nearly a decade ago, I made a repository full of licenses written as jokes, alongside real licenses so poorly written they seemed like jokes. Eventually it expanded to include licenses that weren’t just badly written but had incredibly awful consequences.

To this day, many developers don’t seem to understand the legal consequences of what they put in their licenses, likely because they are not lawyers, and keep trying to write licenses that say “you cannot use this software for evil”. Recently, there has been an increasing number of developers who are deliberately using legally unsound licenses for their open-source software because they don’t want corporations using it.

when I was at google, their class on licenses basically said "we can't use any license that's funny" so I suggest everyone license their software as funny as possible

[image or embed]

— Foone (@foone.bsky.social) Nov 17, 2025 at 1:53 PM

NEVER DO THIS, unless your goal is to ensure nobody uses your code in anything. For things that aren’t libraries (like stand-alone applications) this is probably fine, but if you don’t want anyone using your code, please just say that alongside the joke license, it’s already hard enough to communicate with other people in a world filled with AI-slop, we don’t need to make it even harder.

I always use nonsense licenses, but my favorite has been something along the lines of "absolutely nobody is allowed to use this software" My thought was that it'd scare off anyone who either values or is bound by intellectual property, and anyone else would happily ignore the clearly silly license - razorgirl.diy

If your goal is to write a license that is so farcical that a court would rule it meaningless, you will have succeeded in releasing your code without a license, which means nobody actually has permission to use it. The copyright defaults to you, and you have given it to nobody. It doesn’t go into the public domain, there is no default open-source license GitHub grants you, it is still legally your code and nobody else can use it unless it falls under fair use. If you really hate the concept of intellectual property, the correct thing to do is to explicitly release it into the public domain - you can use the unlicense, which also has a section handling countries that do not recognize the public domain, ensuring everyone can truly use your software anywhere in the world.

Remember, everyone is bound by intellectual property. The State will happily drag you into court on behalf of someone suing you for damages regardless of your personal opinions about intellectual property. This means that I can only use code that I don’t actually have a license to if I trust that the person who wrote it will never change their mind, ever. Even if I were to somehow vet every developer of every dependency now, sometimes people just go crazy. In 2016 it was MAGA. In 2020 it was Covid. In 2025 it’s AI psychosis. Every 5 years a new group of people go insane. A license exists to protect people from your future self, providing legal assurance that the values you claim to uphold will be respected in perpetuity.

But I’m A Pacifist

Sometimes, instead of writing incoherent licenses, people attempt to write licenses that make real, honest attempts at defining acceptable moral use cases for the licenses, and it is always nightmarishly complex. Let’s take a look at one example that was particularly notable because it tried very hard to clearly define what “evil” is, yet created a license no sane person would ever agree to: For Good Eyes Only License 0.2

This license has some fascinating terms in it, like:

the penalty for Permanent Termination violations is donating 4% of the total assets (if more than €1 million) / the total annual income (if more than €500,000) or of the annual turnover (if more than €1 million per month) to a charity

To a charity? any charity? Gee, sure thing buddy, let me donate 4% of my total assets to my charitable organization that happens to be entirely and wholly owned by me, which is dedicated to scientific research into how best to replace all artists with AI. The license actually tries to deal with this later on by saying “the Licensee is obliged to make a punitive donation to one or more non-profit organisations or other charitable associations of his/her choice, which are officially recognised as such, are not illegitimate organisations and are primarily dedicated to…”, but fails to define what an “illegitimate organization” is, and includes “general scientific research” as an acceptable purpose. Trust me, you can get the IRS to recognize some wild stuff as a “legitimate charity” doing “scientific research”.

This Licence shall be interpreted on a common-sense basis and as a coherent, non-absurd and non-contradictory flow of ideas, thoughts and moral points of view.

Ah yes, the ultimate defense against people misinterpreting what you say: just tell them to not misinterpret you! Amazing!!! Clearly all legal jargon is completely pointless and can be replaced with this one neat trick of just telling people to magically figure out what you intended to say, based on your personal notion of what “common-sense” means.

The Licensee is not allowed to: behave in unethical ways or have ever been guilty of any inexcusable offences

Now, this is normally where I’d ask “and what exactly is an inexcusable offence?” but this license goes to surprising lengths to define exactly what it means. It starts by listing “intentionally or negligently supporting, committing or preparing genocide” and then goes on for several pages. At one point it defines “Ostracised weapons and methods of warfare” as any subject to the Hague Conventions, the Geneva Protocol, or “any other comparable international conventions”, which is definitely the kind of super-not-vague language I want in my software license that is attempting to define what a war crime is. It also tries to define an “illegitimate regime” with several potential qualifiers, including one that is “the state’s score in the Reporters Without Borders “Press Freedom Index” is worse than 40 (based on the 2022 scale)”. I love checking the Press Freedom Index to see if I might accidentally be supporting some random country I’ve never heard of by allowing them to download my software!!! What even happens if the Press Freedom Index stops being reliable? What if it goes away? WHY IS THIS IN MY LICENSE FILE???

Putting any of this inside a software license is simply a terrible idea. Morality is just too complicated to be captured inside LICENSE.md - it’s why we have a gigantic, sprawling legal system trying to capture thousands upon thousands of nuances and edge-cases. Even if some actions are conceptually easy to define as evil, it is almost impossible to capture that conceptual idea with words on a page, and such debates do not belong in your license file. If you don’t want evil people using your software, don’t make it open-source. Evil people were just going to ignore your license and steal the code anyway, so what’s the point?

I believe the best way to think about this is the same way we think about basic rights - even assholes should be able to feed themselves and get shelter and use free software, because otherwise someone will change their definition of asshole to mean YOU.

Post by @Xibanya@mastodon.xyz

View on Mastodon

Years ago, I realized that trying to become a “programming expert” was both an ill-defined goal and did not actually correlate to being paid well. Throughout my career, I noticed that things software developers considered “amazing feats of engineering” often amounted to dealing with a shitty tool nobody else wanted to deal with, being able to contort a shitty tool into doing unholy things, or papering over a shitty tool with a very thin abstraction layer. Companies that were built on bewildering towers of duct tape and prayers always paid whoever was willing to duct tape more crap to the tower, never the ones suggesting how to fix it. Your pay at a large tech company mostly correlates to how much bullshit you are willing to put up with, and how much tribal knowledge you’ve accumulated.

Even in open-source communities, nobody wants to replace shitty tools with better tools. Instead, programmers often worship the shitty tools, taking pride in knowing how to properly utilize a particularly annoying and unnecessarily complex tool, lording their knowledge over people who might point out that the tool is needlessly complex and actually just bad. This behavior was rampant in the early 2000s when Linux fanboys would say things like “everyone should just learn how to use a terminal”. Nowadays they just complain about people using rust, presumably because it takes away their inalienable right to push shitty, insecure garbage.

Nowadays, it feels like it’s impossible to complain about anything without someone insisting that I’m just “holding it wrong”. It’s as if I’m being handed a hammer full of rusty nails, and when I insist on trying to build a hammer without rusty nails instead of learning how to hold their shitty hammer full of rusty nails in just the right way, I’m told that I’m either being “lazy” or “unrealistic” or that I’ll “fragment the ecosystem”. Open-source advocates absolutely love using XKCD #927 as a sledgehammer against any attempts at fixing anything. Contributors may only pull rusty nails out of the hammer, one by one, agonizingly slowly, and if anyone even hints at replacing the entire hammer, they’ll be accused of “fragmenting the ecosystem”.

Once you get old enough, this is recognizable as a very obvious, very transparent attempt to rationalize an emotional need to preserve social status. Being a major contributor on a very important package grants you social status. If the only impressive thing you’ve ever done in your entire life is being a major contributor to some random package relied on by half the ecosystem, you will treat anything that threatens to obsolete that package as an existential threat to your life, because humans are hardwired to treat loss of social status as life-threatening. To a lesser degree, many people may simply have an emotional attachment to their contributions.

Seeing this kind of behavior crop up so often is infuriating, because it’s what you expect from insecure college kids who don’t have any hobbies, but the industry is overrun by professionals who do this, and worse, are often paid large sums of money to deal with whatever cursed bullshit a particular company has built its entire product on top of. Then these people get into dick-measuring contests and immediately affirm the consequent - they believe that because they’re paid a lot of money, they must be good programmers. This makes everything worse, because now attempting to replace whatever nonsense they’re maintaining is an actual threat to their livelihood.

This juxtaposition of misaligned incentives has turned the entire field of software engineering into a shit-eating contest, where programmers compete not on their ability to create useful tools, but instead on who can deal with the most bullshit. Eating shit yields the best short-term rewards, so the best shit-eaters quickly accumulate wealth and influence, which they immediately use to try to convince everyone to use the same bullshit they’re paid to deal with.

This is also why the entire industry is panicking over AI tools that are objectively terrible at high-level design, or even moderately complex tasks, because the AIs are superhumanly good at dealing with bullshit. If you’ve ever had to deal with an enterprise codebase, modern AI, despite being unable to handle anything beyond simple logic, is incredibly good at figuring out weird C macros and whacky build rules and weird in-house utility functions that have been built-up over the years, because an AI agent can read literally all of your documentation and code comments in a matter of hours. If your job security was based on absorbing all this tribal knowledge and wielding it to solve strange bugs and problems that your coworkers had, you’re screwed - they can just ask the AI now instead of having to ask you. The AI is probably also nicer to talk to.

AI can eat shit better than any human, which is also why they are superhumanly good at finding certain classes of bugs, because AIs can find any needle in any haystack. What they lack is judgement, common sense, and the ability to do high-level design. It turns out a whole lot of human coders also couldn’t do high-level design, they were just good at eating shit, and they’re the ones who think AI is going to destroy the world, because to those people, AI is already destroying the only world they knew - one ruled by enterprise bullshit.

Three months ago, the idea of using AIs to help debug code sounded like complete nonsense to me, given that they couldn’t even write code well. In my experience, the AI models I’ve tried still can’t write code very well, but it turns out this is a completely different skill from finding bugs. In reality, AIs are already superhumanly good at finding logic errors, and while Anthropic’s Mythos is usually what comes to mind, much weaker models can actually find the same security flaws if given specific instructions in highly constrained environments.

FreeBSD detection (a straightforward buffer overflow) is commoditized: every model gets it, including a 3.6B-parameter model costing $0.11/M tokens. You don’t need limited access-only Mythos at multiple-times the price of Opus 4.6 to see it. The OpenBSD SACK bug (requiring mathematical reasoning about signed integer overflow) is much harder and separates models sharply, but a 5.1B-active model still gets the full chain.

What’s going on here? How is anyone getting useful outputs out of the infinite slop machine? Keep in mind that frontier models being tested in labs or provided to high-profile tech companies are vastly more powerful than what most people have access to. Many people who try to use weaker, publicly accessible AIs to do basic research, or ask questions, or perform tasks often get subtly wrong results or outright nonsense. Currently, the models are also quite easy to poison because companies don’t bother sanitizing data inputs, preferring to just scrape the entire internet as fast as possible, so you get AIs telling people to go kill themselves.

I would like to propose that these are two different modes of operation. On one hand, we have neural networks capable of universal plausible content generation, either using diffusion models for images and audio, or LLMs for text. Because LLMs use neural networks an incomprehensible pile of linear algebra to search latent space for concepts that are similar to the ones in the input prompt, this has accidentally created the most effective search tool in history. The problem is that it can only search for abstract concepts that it was trained on in the dataset (unless provided with an external search tool), and this is inherently an inadvertent consequence of generating plausible textual outputs, which means the only way it can respond is by generating a plausible-sounding answer based on what it found in latent space. We’ve invented a universal needle in a haystack finder that hallucinates needles, attached to an infinite hay generator.

On the surface, this might sound useless (and also funny), but to understand how some people are getting useful results out of hallucinating AIs in weirdly specific domains but not others, you need to understand the NP computational complexity class. Basically, if a problem lives in $$ NP $$, then any proposed solution to the problem can be quickly verified (where quickly means “in polynomial time”, but that’s not important here). However, actually finding a correct solution may be incredibly difficult. A simple example is Sudoku. An $$ N \times N $$ Sudoku board might be incredibly hard to actually solve, but a solution is trivial to verify: simply make sure that every digit from $$ 1 $$ to $$ N $$ exists in each row, each column, and each subgrid.

This applies to hallucinating AIs. If you use them to find something easily verifiable (for some definition of “easy”), they will save you time. If you use them to find something you can’t easily verify, you are playing with fire. If you ask the AI to find a very specific sentence that you remember reading in a book once, and it says it found the book, you can just press ctrl-F and try to find the sentence yourself, or just read the paragraph it links to. If the AI says it found a bugfix, you can try the fix and see if it works. On the other hand, if you ask the AI to find an appropriate substitution for an ingredient in a recipe, unless you have the cooking experience to already know what a reasonable substitution would be, you have absolutely no way to know whether or not what the AI responds with actually makes sense, which is how someone ended up in the hospital with Bromism.

Basically, AIs are good at solving strong-link problems but incredibly dangerous to use on weak-link problems unless you take steps to mitigate the risk. “Weak-link problems” are any problems where the overall quality is determined by the worst outliers, like food safety, which is why bad things keep happening when people use AI for food related things, because AI output can be incredibly bad. Safely using AIs in any weak-link situation always involves minimizing the consequences of the worst possible outputs by building safeguards that ensure it isn’t a big deal if the AI hallucinates subtly wrong answers, like having AIs write provably correct code instead of python, so any subtle mistakes turn into compiler errors instead of a catastrophic production incident.

What makes AIs so powerful is that the “needles” they can look for don’t resemble any kind of needle a human expects, and as the AIs get more powerful, they can turn increasingly abstract concepts into “needles” that they can find. It turns out that security flaws are one such abstract needle in a haystack that the AIs can find, and are easily verifiable. AIs are also great at certain kinds of glue code, because it’s a needle in a haystack problem where the abstract needle is “the example code for this obscure API used by an old HDR monitor that was published on this company website 10 years ago”, which can be easily verified by checking if it crashes or not. However, anything that looks like an engineering problem shifts the problem domain back into plausible text generation, where weaker models are still very bad at writing nontrivial, maintainable code. More powerful AIs are better at code generation largely because they can understand more complex abstract needles, which allows them to avoid the plausible text generation domain by utilizing reasoning loops. This is how many harnesses work, by prompting the AIs to look at their own output and saying “find logical errors”, which shifts the problem domain back into finding a needle in a haystack.

Because AIs can search for a needle that is an abstract concept, they have managed to reach human-level performance in certain constrained scenarios by chaining these concepts together in a reasoning loop. It is entirely possible that in a few years, AIs will be able write certain kinds of code at superhuman levels if given appropriate guardrails and architecture limitations. However, even if this happens, current AIs are still very spikey intelligences unable to duplicate human judgement and intuition, because intuition requires general common sense, which would require AGI. Thus, humans will still need to guide the AIs, designing the high level architecture, building guardrails and double-checking the AIs work. This would be a death knell to the current AI bubble, which is driven by the promise of being able to fire all the humans, which you can’t do if the AIs can’t actually duplicate human judgement, even if they are superhumanly good at many different tasks in isolation.

However, even if the AI bubble pops, current AIs still have profound implications for research because a huge amount of science involves trying to find needles in a haystack. Protein folding? You betcha. Finding candidate drugs that don’t have horrible side-effects? In progress. Spotting cancer? Actively being generalized. Think about how much time scientists have to spend trying to find tiny, subtle signals in a sea of data. Modern AI is the solution to our massive overabundance of data that we struggle to analyze. A universal needle in a haystack finder is the key to unlocking personalized medicine by allowing an automated system to sift through terabytes of data in hours to help flag potential problems to doctors. The only problems are that current AIs take an enormous amount of computational power to train, can only find needles they have been trained to find, and are built on top of an infinite hay generator that occasionally hallucinates needles.

If everything feels insane right now, that’s because we are in the midst of the most ironic technological development in recent history - a needle in a haystack finder that only works when attached to an infinite hay generator that makes it harder to find needles and also hallucinates them. If you want nothing to do with this, that is completely reasonable, but I would strongly discourage trying to shame others into not using AI, as behavioral science shows this doesn’t work. This post is not an endorsement of AI, it is simply a reference in case work is forcing you to use AI, or a friend is using AI for something incredibly dumb.

Research shows that people are less likely to rely on AI the more they understand it, so I hope this has at least demystified what AI might theoretically be useful for in ways that don’t involve generating slop or being told to eat bromine. However, if your main concerns about AI are environmental, I would caution that a lot of the water-usage claims are extremely dubious or outright lies. This is extremely frustrating, because AI has plenty of real issues, like being used to mass-manufacture scams and misinformation on an industrial scale never seen before, bot crawlers destroying the entire open internet, AI psychosis, and data centers putting enormous strain on local power grids that aren’t given enough time to adapt (which is driving the spike in CO2 emissions).

Trying to figure out if AI is fundamentally good or bad, however, is beyond the scope of this post - I simply hope you have a better idea of what the current technology is capable of and why it behaves the way it does. Hopefully this can inform discussions about whether or not we should even be using this technology.

My relationship with AI is getting increasingly strange. Generalist AIs are still mostly useless, but narrow AIs continue to produce very impressive results. We have plenty of AIs that are better than any human at specific tasks like spotting cancer, but no AIs that can exercise common sense. We can synthesize terrifyingly realistic recreations of almost anyone’s voice, but they must be handheld by humans to produce consistent emotional inflection. We have self-driving cars that work fine at noon on a clear day with no construction, but an errant traffic cone makes them panic.

This is called “spiky intelligence”, and it is why ChatGPT can solve incredibly difficult math olympiad questions but struggle to push a button on a webpage. It seems to me that all these smart people with PhDs saying that AI will take over the workforce are convinced that, if AIs can continually get better at tackling difficult problems, eventually they’ll be able to train AIs that can also handle “easy” problems.

This is the exact same error that resulted in the second AI winter of the 90s - when researchers built expert systems could outperform humans in narrow situations, they simply assumed they would soon be able to outperform all humans in all situations. This, obviously, didn’t happen, but task-specific engines did emerge from this, and now it is a well-known fact that your phone has enough processing power to effortlessly destroy every single human chess grandmaster that has ever lived. We still play chess, though.

What worries me is that this kind of spiky intelligence, despite lacking general common sense, will still radically upend the economy in ways that are simply impossible for a human brain to anticipate, because these AIs are, by definition, alien intelligences that defy all human intuition. The coming AI revolution is dangerous not because it’s going to destroy the whole world if we get it wrong, but because it is almost impossible to anticipate in any meaningful way. No human is capable of accurately guessing what weirdly specific task an AI might find easy or extraordinarily difficult. It’s XKCD #1425 but randomized for every single task on the entire planet:

AI enthusiasts often like to talk about The Singularity, a point in time when technological progress accelerates beyond human understanding and thus the future beyond it becomes unknowable. To me, this is not a very useful thing to think about. After all, we’re already incapable of predicting what society will look like 10 years from now. What’s concerning is that we’re used to being able to prepare for the next 2-3 years (ignoring black swan events), and I anticipate that AI will cause economic chaos in ways we cannot predict, precisely because it will rapidly automate entire categories of human employment out of existence, randomly. What will happen when we start automating entire industries faster than we can retrain people? What happens when someone tries to migrate to another career only to have that career automated away the moment they graduate?

We already struggle to keep up with the rapid pace of change, and AI is about to automate everything even faster, in extremely unpredictable ways. There may be a moment when it becomes impossible to anticipate the trajectory of your career three months from now, without AGI ever happening. We don’t need a superintelligent godlike AI to fuck everything up, the extraordinarily powerful narrow AIs we’re working on right now can fuck up the whole global economy by themselves. This moment is the only “Singularity” that I care about - a sort of Technological Tsunami, when entire economic sectors are swept away by rapid automation so quickly that workers can’t course correct to a new field fast enough.

We have options, since we know that it will fuck up the economy, we just don’t know how. The easiest and most pragmatic solution is UBI, but this seems difficult to make happen in a society run by rich people who are largely rewarded by how evil they are. There are plenty of political groups that are pushing for these kinds of solutions, but global policymakers appear to have been captured by AI money, which is only concerned about the dangers of a mythological AGI superintelligence instead of the impending economic catastrophe that is already beginning to develop. Because of this, I think there is a real question over whether or not human society will survive the coming technological tsunami. Again, we don’t need to invent AGI to destroy ourselves. We didn’t need AI to build nukes.

With that said, some people seem to deny that AGI will ever happen, which is also clearly wrong, at least to me. There are many things that will eventually happen, based on our current understanding of physics (and assuming we don’t blow ourselves up). Eventually we’ll cure cancer. Eventually we’ll reverse aging. Eventually we’ll have cybernetic implants and androids. Eventually we’ll be able to upload human minds to a computer. Eventually we’ll build a general artificial intelligence capable of improving itself. It might take 10 years or 100 years or 1000 years, but these are all things that will almost certainly happen given enough time and effort, we just don’t know when. At the very least, if you build as much computational power as the entire combined brainpower of the human race, you’ll be able to brute-force a superintelligence of some kind, and we’d have better solved the alignment problem by then, or augmented ourselves enough to handle it.

At the same time, AI companies continue making wild extrapolations about the capabilities of AIs that simply don’t line up with real world performance. You cannot assume that an AI that scores better than all humans at every test will actually be good at anything other than taking tests, even if humans who score highly on those tests sometimes accomplish amazing things. I have a friend who was placed in Mensa at a very young age after scoring high on an IQ test. They complain that the only thing this group of very smart people do is argue about how to run the organization and what the latest cool puzzles are.

If you know you are actually much more intelligent than the statistical average, increase your humility. It is too easy to believe your own judgements, to get stuck in your own bullshit. Being smart does not make you wise. Wisdom comes from constantly doubting yourself, and questioning your own thoughts and beliefs. Never think, even for a moment, that you have 'settled' anything completely. It's okay to know you are bright, it is not okay to think that gives you any certainty or authority of understanding. — Chatoyance

The world’s smartest people are struggling to extrapolate the capabilities of an extremely spiky and utterly alien narrow intelligence, because it defies basic human intuition. Assuming an AI will be good at performing arbitrary tasks because it scored well on a test is the same kind of attribution error that happens with experts in a specific field - people will trust the expert’s opinion on something the expert has no experience with, like the economy, even though this almost never works out. This is such a persistent problem because highly intelligent people can invent plausible sounding arguments to support almost any position, and it can be exceedingly difficult to find the logical error in them. We are lucky that our current LLMs usually make egregious errors that are obviously wrong, instead of extremely subtle errors that would be almost impossible to detect.

We are in the middle of an AI revolution that will create new, extraordinarily powerful tools whose effects are almost impossible to predict. Instead of doing anything about the impending economic catastrophe, we are chasing AI safety hysteria and telling AGI superintelligence ghost stories that will likely not happen for decades, if not centuries. Otherwise intelligent people are convincing themselves that there’s no point worrying about the economy crashing if AGI makes humans irrelevant. We’re so busy trying to avoid flying too close to the sun we haven’t noticed a technological tsunami rising up beneath us, and if we continue ignoring it, we’ll drown before we even become airborne.