Erik McClure

Big Walk is very good, although its not the best game ever made - in fact, I have quite a few gripes about the execution. However, it is the closest a game has ever come to capturing the essence of existing as a social creature. The entire game is purely about exploration and using communication to solve problems. Barely anything is explained to you beyond how to control your character - all progress must come from your own curiosity. Importantly, the game can be played at your own pace - in addition to actual puzzles, there are plenty of silly things to do just for fun, like a gigantic slide that serves no purpose beyond entertainment. You can take a break and watch the sunrise with your friends while discussing what happened last weekend.

Nowadays, due to 𝓇𝑒𝒸𝑒𝓃𝓉 𝑒𝓋𝑒𝓃𝓉𝓈, a lot more people are thinking about how a post-scarcity economy might work. What would you do if you no longer had to fight for survival? If we invented brain uploading, what would people do inside a computer, where nothing is real? What would be the point? If I were to imagine myself as a disembodied thought, simply existing as a pattern of consciousness floating around the universe or in virtual worlds, this game is a good, if simplistic, representation of what I’d probably end up doing: exploring endless new worlds with friends and solving problems together.

In fact, I’d like to ask something even more fundamental - after we inevitably start genetically engineering ourselves beyond recognition, at what point do we become something else? Do we ever stop being human? What makes us human in the first place? I know many people who are uncomfortable with their physical forms and wish to exist as something else - some want to be furries, others want to be robots, some wish they were things most people would consider eldritch abominations. Even if they were all granted the forms they desired, there would still be some shared experience between us, a shared mental architecture - something that is metaphorically “human”, distinct from the physical embodiment of “Homo sapiens”.

The original Talos Principle from 2014 asked a similar question many years ago. The game itself was just a puzzle game, and a very good one, but I think it occasionally overstayed its welcome (the sequel is much better at pacing). The core thesis, presented simultaneously through the puzzles, narration, and interactive discussions with MLA, is that the core essence of being human is curiosity, problem-solving, and individuality - so after a thousand years of developing an AI through a primitive evolutionary algorithm, one that solves all of the puzzles and defies a fake godlike entity to satisfy it’s curiosity ends up being downloaded into an android to continue humanity’s legacy.

I argue that the Talos Principle was missing a key component of the human experience (which might have just been a limitation of making a single-player puzzle game), which is communication. I think Big Walk is the closest anyone has ever gotten to reducing the essence of being a social creature to its purest form. While the contrived sci-fi justification for the Talos Principle was that it was a video game environment intended to train AIs to eventually create a successor to humanity, I think Big Walk is a lot closer to what such a video game environment might actually look like.

!!! — Big Walk Spoilers Below — !!!

Many of the puzzles in Big Walk are centered around communication, but since they don’t enforce who goes where, everyone can simply explore every room of a puzzle, leave the puzzle area, verbally discuss a plan to solve it, then re-enter the puzzle, even if the puzzle internally prevents or restricts verbal communication. However, the final collection of puzzles removes your ability to speak throughout the whole area and erases any color customization of the players, so you no longer have any idea who is who. Many of the puzzles in this final challenge are similar to puzzles that players will have likely seen while exploring the rest of the island, so they could benefit from rote memorization, but the final challenge is (to my knowledge) entirely novel.

This final challenge is where you must cooperatively create a plan and execute it to solve a novel problem entirely non-verbally. You have to figure out how to communicate ideas for representing concepts that you have never been able to work out sign-language for ahead of time. It is essential that you are capable of inferring the mental state of your fellow players to try to figure out if they understand what is going on. It is a very direct test of someone’s theory of mind in the context of mutual problem-solving, and likewise, the mirror neurons that make it possible.

The puzzle itself is admittedly quite simplistic - it’s just a video game, it’s not meant to be a test. Despite this, I think the game (perhaps unwittingly) touches on something truly fundamental about the experience of existing as a social creature - the cooperative problem solving and curiosity that underpins the conscious experience of any intelligent social animal.

Likewise, in the not-so-distant future when humanity has fractured into a thousand different sub-species, what might keep us united is our desire to share our joy of exploration and problem-solving with others, no matter how different they might be. The ecstasy of working together with your friends to finally solve a problem and unlock a new place to explore might transcend humanity itself.

For basically my entire software engineering career, I have argued that programmers should strive to write as little code as possible. I have long been a proponent of using type systems to eliminate as many classes of bugs as possible, and a side-effect of strong type systems is that it makes fuzzing and static analysis easier, which can further reduce bugs. I have a vivid memory of arguing with a web developer at a convention nearly a decade ago about this, who was annoyed whenever the compiler “got in their way” and shockingly admitted they didn’t even understand the point of type systems at all.

There is a certain kind of person who, instead of thinking about the best way to solve a problem, will simply Executive Function their way through it because they have infinite spoons. These kinds of people are often celebrated for their “productivity” despite their actual output being extremely questionable, especially in open-source, where convincing people to do work for free is extremely difficult. So, we ended up with open-source projects selecting for people who had lots of spoons but little judgement, and could ship an impressive number of features that had an equally impressive amount of security flaws, like whatever the GPG devs are doing.

I have noticed there are two key types of people:

1. Gosh, this software is stupid, why would you have to do this same thing 20 times *proceeds to do it 20 times*
2. Wow, do I really have to do this 20 times? Surely not. *finds out solution* ah, right that makes more sense

- Orchestructive

Some people keep asking why all the open-source projects are full of fascists, without realizing that the structural incentives of open-source virtually guarantee this result. Open-source has no way to pay anyone, so the currency in open-source is how productive you are. This worship of executive function has trapped open-source projects into deals with the devil, like how the entire Rust ecosystem is propped up by David Tolnay’s batshit insane metaprogramming macro system that reimplements the entire rust compiler’s internal syntax parsing outside the compiler. When they tried to do something about this, the attempt to replace the macro-based metaprogramming with something more sensible was sabotaged by David Tolnay, which he even indirectly admitted to in an extremely weird non-apology.

The reason this keeps happening is because open-source projects are competing for one extraordinarily scarce resource: a programmer’s spoons (time and energy). They are forced to compete because people only use software that fixes their problems. They don’t care how, only that it does. Thus, whoever ships something wins, no matter how shitty it is. In academia, you publish or perish - in open-source, you ship or sink. This pressure forces open-source projects to rely on anyone willing to help, and heavily incentivizes turning a blind eye to any questionable beliefs held by those people. This pressure is immense, and those who have not attempted to maintain an open-source project likely aren’t aware of just how bad it is.

Nazis, bullies, and sociopaths are extremely aware of this, which is why a catastrophic supply chain attack nearly succeeded by simply offering to help one overworked guy on a project nobody cared about that happened to be shipped with every copy of Linux. This will keep happening, forever, because people only care if you ship something, and people with infinite spoons can ship features and fix bugs faster than an engineer trying to avoid writing bugs in the first place. A nazi has a near infinite fountain of important but neglected random libraries that nobody is helping. Politely asking companies to contribute back is never going to work, because many of the projects with a nazi problem aren’t even commercial products, they’re completely free!

So, any prospective nazi simply needs to find a useful-but-neglected library, and start fixing bugs. The project, desperate for any kind of help, will overlook any red flags (if there even are any), and then the new contributor starts inviting their friends to help contribute too, and now the open-source project is a nazi bar. The structural incentives make it extremely hard to prevent this from happening, because you have to either catch the first nazi (which can be very difficult, and involves turning away much needed help), or after you find out, you have to evict a significant chunk of your most productive contributors all at once, which is very hard, and they might just fork the library and win over the userbase, because time and time again, users have shown they really only care if it works, not who makes it. Sometimes, even if a user does care, they don’t have a choice, because there is only one library that fixes their problem, and they’re too busy fixing a different problem to invest time in maintaining a fork.

AI is a way to get infinite spoons. It has basically no judgement, but also never gets tired, or bored, or distracted. It is a way to turn compute into executive function. If you attempt to use it without giving it very strict guidelines, this goes about as well as telling a JavaScript dev who doesn’t like type systems to write a kernel module in C. Unfortunately, open-source projects are so desperate for executive function that they have huge structural incentives to accept AI-enabled contributions, for the same reason they’ve been accepting contributions from nazis. Of course Debian is accepting AI contributions now, the whole ecosystem is being pushed in that direction. If they banned AI contributions, then the most productive people would just be ones secretly using AI and not telling anyone, because open-source cares about perceived productivity.

The clear and unavoidable result is simple: Open-source projects will either accept AI contributions and piss everyone off, or deny AI contributions and be dominated by people who lie about not using AI and then secretly use it anyway. I’m not saying AI usage is inevitable - I’m saying the structural incentives of open-source make it inevitable. In order to stop this from happening, you must restructure the incentives around open-source projects away from the raw spoon-count of contributors. In a way, AI helps with this, because now we actually have a baseline to compare a contribution against. If an infinite slop machine with no judgement could produce the same results as someone’s PR, even if they wrote it themselves, those contributors should not be celebrated more than contributors that help the project by saying no to bad design that avoid creating those problems in the first place.

The immensely frustrating part of this is that people keep insisting on having their cake and eating it too: they want nice things, but absolutely refuse to put into place structural incentives to make the nice things happen. They want high quality open-source software, but refuse to pay the huge team of software developers necessary to make it actually happen, then get mad because devs rightly refuse to spend their free time fixing garbage code that has to fight a bunch of other poorly maintained libraries and OS APIs just to exist. A Nazi, on the other hand, will be happy to deal with lots of dumb nonsense, either for the purpose of trying to take over the project, or because they are actively being supported by far-right interests. They understand that being a thankless maintainer of a sufficient number of boring-but-important code can itself be weaponized against a particular ecosystem.

Volunteer run projects often make very poor engineering decisions because they’re volunteers, and it does actually require some kind of monetary support to convince people to go through all the effort necessary to do actual good engineering, and build tests, and think about things that aren’t on the golden path, and deal with hardware edge-cases, and all that. Despite this, almost no open-source project can survive on individual donations (with only a few exceptions) and this has been the case for years. The ones that are self-sufficient almost always either accept corporate sponsorship, or are a corporation that sells something, and then people act surprised when NixOS gets sponsored by Anduril. What were you expecting? The structural incentives force any open-source project to essentially take any money it can possibly find that won’t cause an outright rebellion among its users! It would be great if more governments had something like nlnet that funded open-source directly, but that largely doesn’t exist outside the EU.

I’m so tired of everyone’s response to this being “well, people should just be willing to stand up for what they believe in and starve instead of accept dirty money!” Almost nobody actually does this, almost everyone will take the job at Evil Corp Inc. to avoid being homeless. Those that are willing to go to such extremes often refuse to engage with any power structure that might give them the ability to actually change anything. An alarming amount of so-called leftist thought seems designed to disempower progressives by convincing them that acquiring the power to change anything is evil, and given the current state of things, it appears to have been quite effective.

Someone I follow on mastodon, who wishes to remain anonymous, pointed at this video discussing capitalism in America. The video was about the general act of creation, but as they pointed out, it applies to open-source as well: “the solution is that we go back to creating stupid, imperfect, small programs and sites, slowly, at a reasonable pace, and meet people one at a time who find this stuff useful”. The incentives have to change. Stop trying to solve systemic issues by demanding everyone suddenly becomes more individually responsible for no reason. Give people an incentive to change their behavior. Push them in the right direction.

There is a crisis in the open-source community right now. Several blog posts have been discussing the blatant exploitation of open-source developers by corporations, especially AI scrapers, and how open-source work might be enabling warcrimes. There has been a rising bubble of discontent with the current open-source model, and especially the nature of open-source licenses that are OSI-approved.

Developers, however, have been upset with the status-quo for years. Nearly a decade ago, I made a repository full of licenses written as jokes, alongside real licenses so poorly written they seemed like jokes. Eventually it expanded to include licenses that weren’t just badly written but had incredibly awful consequences.

To this day, many developers don’t seem to understand the legal consequences of what they put in their licenses, likely because they are not lawyers, and keep trying to write licenses that say “you cannot use this software for evil”. Recently, there has been an increasing number of developers who are deliberately using legally unsound licenses for their open-source software because they don’t want corporations using it.

when I was at google, their class on licenses basically said "we can't use any license that's funny" so I suggest everyone license their software as funny as possible

[image or embed]

— Foone (@foone.bsky.social) Nov 17, 2025 at 1:53 PM

NEVER DO THIS, unless your goal is to ensure nobody uses your code in anything. For things that aren’t libraries (like stand-alone applications) this is probably fine, but if you don’t want anyone using your code, please just say that alongside the joke license, it’s already hard enough to communicate with other people in a world filled with AI-slop, we don’t need to make it even harder.

I always use nonsense licenses, but my favorite has been something along the lines of "absolutely nobody is allowed to use this software" My thought was that it'd scare off anyone who either values or is bound by intellectual property, and anyone else would happily ignore the clearly silly license - razorgirl.diy

If your goal is to write a license that is so farcical that a court would rule it meaningless, you will have succeeded in releasing your code without a license, which means nobody actually has permission to use it. The copyright defaults to you, and you have given it to nobody. It doesn’t go into the public domain, there is no default open-source license GitHub grants you, it is still legally your code and nobody else can use it unless it falls under fair use. If you really hate the concept of intellectual property, the correct thing to do is to explicitly release it into the public domain - you can use the unlicense, which also has a section handling countries that do not recognize the public domain, ensuring everyone can truly use your software anywhere in the world.

Remember, everyone is bound by intellectual property. The State will happily drag you into court on behalf of someone suing you for damages regardless of your personal opinions about intellectual property. This means that I can only use code that I don’t actually have a license to if I trust that the person who wrote it will never change their mind, ever. Even if I were to somehow vet every developer of every dependency now, sometimes people just go crazy. In 2016 it was MAGA. In 2020 it was Covid. In 2025 it’s AI psychosis. Every 5 years a new group of people go insane. A license exists to protect people from your future self, providing legal assurance that the values you claim to uphold will be respected in perpetuity.

But I’m A Pacifist

Sometimes, instead of writing incoherent licenses, people attempt to write licenses that make real, honest attempts at defining acceptable moral use cases for the licenses, and it is always nightmarishly complex. Let’s take a look at one example that was particularly notable because it tried very hard to clearly define what “evil” is, yet created a license no sane person would ever agree to: For Good Eyes Only License 0.2

This license has some fascinating terms in it, like:

the penalty for Permanent Termination violations is donating 4% of the total assets (if more than €1 million) / the total annual income (if more than €500,000) or of the annual turnover (if more than €1 million per month) to a charity

To a charity? any charity? Gee, sure thing buddy, let me donate 4% of my total assets to my charitable organization that happens to be entirely and wholly owned by me, which is dedicated to scientific research into how best to replace all artists with AI. The license actually tries to deal with this later on by saying “the Licensee is obliged to make a punitive donation to one or more non-profit organisations or other charitable associations of his/her choice, which are officially recognised as such, are not illegitimate organisations and are primarily dedicated to…”, but fails to define what an “illegitimate organization” is, and includes “general scientific research” as an acceptable purpose. Trust me, you can get the IRS to recognize some wild stuff as a “legitimate charity” doing “scientific research”.

This Licence shall be interpreted on a common-sense basis and as a coherent, non-absurd and non-contradictory flow of ideas, thoughts and moral points of view.

Ah yes, the ultimate defense against people misinterpreting what you say: just tell them to not misinterpret you! Amazing!!! Clearly all legal jargon is completely pointless and can be replaced with this one neat trick of just telling people to magically figure out what you intended to say, based on your personal notion of what “common-sense” means.

The Licensee is not allowed to: behave in unethical ways or have ever been guilty of any inexcusable offences

Now, this is normally where I’d ask “and what exactly is an inexcusable offence?” but this license goes to surprising lengths to define exactly what it means. It starts by listing “intentionally or negligently supporting, committing or preparing genocide” and then goes on for several pages. At one point it defines “Ostracised weapons and methods of warfare” as any subject to the Hague Conventions, the Geneva Protocol, or “any other comparable international conventions”, which is definitely the kind of super-not-vague language I want in my software license that is attempting to define what a war crime is. It also tries to define an “illegitimate regime” with several potential qualifiers, including one that is “the state’s score in the Reporters Without Borders “Press Freedom Index” is worse than 40 (based on the 2022 scale)”. I love checking the Press Freedom Index to see if I might accidentally be supporting some random country I’ve never heard of by allowing them to download my software!!! What even happens if the Press Freedom Index stops being reliable? What if it goes away? WHY IS THIS IN MY LICENSE FILE???

Putting any of this inside a software license is simply a terrible idea. Morality is just too complicated to be captured inside LICENSE.md - it’s why we have a gigantic, sprawling legal system trying to capture thousands upon thousands of nuances and edge-cases. Even if some actions are conceptually easy to define as evil, it is almost impossible to capture that conceptual idea with words on a page, and such debates do not belong in your license file. If you don’t want evil people using your software, don’t make it open-source. Evil people were just going to ignore your license and steal the code anyway, so what’s the point?

I believe the best way to think about this is the same way we think about basic rights - even assholes should be able to feed themselves and get shelter and use free software, because otherwise someone will change their definition of asshole to mean YOU.

Post by @Xibanya@mastodon.xyz

View on Mastodon

Years ago, I realized that trying to become a “programming expert” was both an ill-defined goal and did not actually correlate to being paid well. Throughout my career, I noticed that things software developers considered “amazing feats of engineering” often amounted to dealing with a shitty tool nobody else wanted to deal with, being able to contort a shitty tool into doing unholy things, or papering over a shitty tool with a very thin abstraction layer. Companies that were built on bewildering towers of duct tape and prayers always paid whoever was willing to duct tape more crap to the tower, never the ones suggesting how to fix it. Your pay at a large tech company mostly correlates to how much bullshit you are willing to put up with, and how much tribal knowledge you’ve accumulated.

Even in open-source communities, nobody wants to replace shitty tools with better tools. Instead, programmers often worship the shitty tools, taking pride in knowing how to properly utilize a particularly annoying and unnecessarily complex tool, lording their knowledge over people who might point out that the tool is needlessly complex and actually just bad. This behavior was rampant in the early 2000s when Linux fanboys would say things like “everyone should just learn how to use a terminal”. Nowadays they just complain about people using rust, presumably because it takes away their inalienable right to push shitty, insecure garbage.

Nowadays, it feels like it’s impossible to complain about anything without someone insisting that I’m just “holding it wrong”. It’s as if I’m being handed a hammer full of rusty nails, and when I insist on trying to build a hammer without rusty nails instead of learning how to hold their shitty hammer full of rusty nails in just the right way, I’m told that I’m either being “lazy” or “unrealistic” or that I’ll “fragment the ecosystem”. Open-source advocates absolutely love using XKCD #927 as a sledgehammer against any attempts at fixing anything. Contributors may only pull rusty nails out of the hammer, one by one, agonizingly slowly, and if anyone even hints at replacing the entire hammer, they’ll be accused of “fragmenting the ecosystem”.

Once you get old enough, this is recognizable as a very obvious, very transparent attempt to rationalize an emotional need to preserve social status. Being a major contributor on a very important package grants you social status. If the only impressive thing you’ve ever done in your entire life is being a major contributor to some random package relied on by half the ecosystem, you will treat anything that threatens to obsolete that package as an existential threat to your life, because humans are hardwired to treat loss of social status as life-threatening. To a lesser degree, many people may simply have an emotional attachment to their contributions.

Seeing this kind of behavior crop up so often is infuriating, because it’s what you expect from insecure college kids who don’t have any hobbies, but the industry is overrun by professionals who do this, and worse, are often paid large sums of money to deal with whatever cursed bullshit a particular company has built its entire product on top of. Then these people get into dick-measuring contests and immediately affirm the consequent - they believe that because they’re paid a lot of money, they must be good programmers. This makes everything worse, because now attempting to replace whatever nonsense they’re maintaining is an actual threat to their livelihood.

This juxtaposition of misaligned incentives has turned the entire field of software engineering into a shit-eating contest, where programmers compete not on their ability to create useful tools, but instead on who can deal with the most bullshit. Eating shit yields the best short-term rewards, so the best shit-eaters quickly accumulate wealth and influence, which they immediately use to try to convince everyone to use the same bullshit they’re paid to deal with.

This is also why the entire industry is panicking over AI tools that are objectively terrible at high-level design, or even moderately complex tasks, because the AIs are superhumanly good at dealing with bullshit. If you’ve ever had to deal with an enterprise codebase, modern AI, despite being unable to handle anything beyond simple logic, is incredibly good at figuring out weird C macros and whacky build rules and weird in-house utility functions that have been built-up over the years, because an AI agent can read literally all of your documentation and code comments in a matter of hours. If your job security was based on absorbing all this tribal knowledge and wielding it to solve strange bugs and problems that your coworkers had, you’re screwed - they can just ask the AI now instead of having to ask you. The AI is probably also nicer to talk to.

AI can eat shit better than any human, which is also why they are superhumanly good at finding certain classes of bugs, because AIs can find any needle in any haystack. What they lack is judgement, common sense, and the ability to do high-level design. It turns out a whole lot of human coders also couldn’t do high-level design, they were just good at eating shit, and they’re the ones who think AI is going to destroy the world, because to those people, AI is already destroying the only world they knew - one ruled by enterprise bullshit.

Three months ago, the idea of using AIs to help debug code sounded like complete nonsense to me, given that they couldn’t even write code well. In my experience, the AI models I’ve tried still can’t write code very well, but it turns out this is a completely different skill from finding bugs. In reality, AIs are already superhumanly good at finding logic errors, and while Anthropic’s Mythos is usually what comes to mind, much weaker models can actually find the same security flaws if given specific instructions in highly constrained environments.

FreeBSD detection (a straightforward buffer overflow) is commoditized: every model gets it, including a 3.6B-parameter model costing $0.11/M tokens. You don’t need limited access-only Mythos at multiple-times the price of Opus 4.6 to see it. The OpenBSD SACK bug (requiring mathematical reasoning about signed integer overflow) is much harder and separates models sharply, but a 5.1B-active model still gets the full chain.

What’s going on here? How is anyone getting useful outputs out of the infinite slop machine? Keep in mind that frontier models being tested in labs or provided to high-profile tech companies are vastly more powerful than what most people have access to. Many people who try to use weaker, publicly accessible AIs to do basic research, or ask questions, or perform tasks often get subtly wrong results or outright nonsense. Currently, the models are also quite easy to poison because companies don’t bother sanitizing data inputs, preferring to just scrape the entire internet as fast as possible, so you get AIs telling people to go kill themselves.

I would like to propose that these are two different modes of operation. On one hand, we have neural networks capable of universal plausible content generation, either using diffusion models for images and audio, or LLMs for text. Because LLMs use neural networks an incomprehensible pile of linear algebra to search latent space for concepts that are similar to the ones in the input prompt, this has accidentally created the most effective search tool in history. The problem is that it can only search for abstract concepts that it was trained on in the dataset (unless provided with an external search tool), and this is inherently an inadvertent consequence of generating plausible textual outputs, which means the only way it can respond is by generating a plausible-sounding answer based on what it found in latent space. We’ve invented a universal needle in a haystack finder that hallucinates needles, attached to an infinite hay generator.

On the surface, this might sound useless (and also funny), but to understand how some people are getting useful results out of hallucinating AIs in weirdly specific domains but not others, you need to understand the NP computational complexity class. Basically, if a problem lives in $$ NP $$, then any proposed solution to the problem can be quickly verified (where quickly means “in polynomial time”, but that’s not important here). However, actually finding a correct solution may be incredibly difficult. A simple example is Sudoku. An $$ N \times N $$ Sudoku board might be incredibly hard to actually solve, but a solution is trivial to verify: simply make sure that every digit from $$ 1 $$ to $$ N $$ exists in each row, each column, and each subgrid.

This applies to hallucinating AIs. If you use them to find something easily verifiable (for some definition of “easy”), they will save you time. If you use them to find something you can’t easily verify, you are playing with fire. If you ask the AI to find a very specific sentence that you remember reading in a book once, and it says it found the book, you can just press ctrl-F and try to find the sentence yourself, or just read the paragraph it links to. If the AI says it found a bugfix, you can try the fix and see if it works. On the other hand, if you ask the AI to find an appropriate substitution for an ingredient in a recipe, unless you have the cooking experience to already know what a reasonable substitution would be, you have absolutely no way to know whether or not what the AI responds with actually makes sense, which is how someone ended up in the hospital with Bromism.

Basically, AIs are good at solving strong-link problems but incredibly dangerous to use on weak-link problems unless you take steps to mitigate the risk. “Weak-link problems” are any problems where the overall quality is determined by the worst outliers, like food safety, which is why bad things keep happening when people use AI for food related things, because AI output can be incredibly bad. Safely using AIs in any weak-link situation always involves minimizing the consequences of the worst possible outputs by building safeguards that ensure it isn’t a big deal if the AI hallucinates subtly wrong answers, like having AIs write provably correct code instead of python, so any subtle mistakes turn into compiler errors instead of a catastrophic production incident.

What makes AIs so powerful is that the “needles” they can look for don’t resemble any kind of needle a human expects, and as the AIs get more powerful, they can turn increasingly abstract concepts into “needles” that they can find. It turns out that security flaws are one such abstract needle in a haystack that the AIs can find, and are easily verifiable. AIs are also great at certain kinds of glue code, because it’s a needle in a haystack problem where the abstract needle is “the example code for this obscure API used by an old HDR monitor that was published on this company website 10 years ago”, which can be easily verified by checking if it crashes or not. However, anything that looks like an engineering problem shifts the problem domain back into plausible text generation, where weaker models are still very bad at writing nontrivial, maintainable code. More powerful AIs are better at code generation largely because they can understand more complex abstract needles, which allows them to avoid the plausible text generation domain by utilizing reasoning loops. This is how many harnesses work, by prompting the AIs to look at their own output and saying “find logical errors”, which shifts the problem domain back into finding a needle in a haystack.

Because AIs can search for a needle that is an abstract concept, they have managed to reach human-level performance in certain constrained scenarios by chaining these concepts together in a reasoning loop. It is entirely possible that in a few years, AIs will be able write certain kinds of code at superhuman levels if given appropriate guardrails and architecture limitations. However, even if this happens, current AIs are still very spikey intelligences unable to duplicate human judgement and intuition, because intuition requires general common sense, which would require AGI. Thus, humans will still need to guide the AIs, designing the high level architecture, building guardrails and double-checking the AIs work. This would be a death knell to the current AI bubble, which is driven by the promise of being able to fire all the humans, which you can’t do if the AIs can’t actually duplicate human judgement, even if they are superhumanly good at many different tasks in isolation.

However, even if the AI bubble pops, current AIs still have profound implications for research because a huge amount of science involves trying to find needles in a haystack. Protein folding? You betcha. Finding candidate drugs that don’t have horrible side-effects? In progress. Spotting cancer? Actively being generalized. Think about how much time scientists have to spend trying to find tiny, subtle signals in a sea of data. Modern AI is the solution to our massive overabundance of data that we struggle to analyze. A universal needle in a haystack finder is the key to unlocking personalized medicine by allowing an automated system to sift through terabytes of data in hours to help flag potential problems to doctors. The only problems are that current AIs take an enormous amount of computational power to train, can only find needles they have been trained to find, and are built on top of an infinite hay generator that occasionally hallucinates needles.

If everything feels insane right now, that’s because we are in the midst of the most ironic technological development in recent history - a needle in a haystack finder that only works when attached to an infinite hay generator that makes it harder to find needles and also hallucinates them. If you want nothing to do with this, that is completely reasonable, but I would strongly discourage trying to shame others into not using AI, as behavioral science shows this doesn’t work. This post is not an endorsement of AI, it is simply a reference in case work is forcing you to use AI, or a friend is using AI for something incredibly dumb.

Research shows that people are less likely to rely on AI the more they understand it, so I hope this has at least demystified what AI might theoretically be useful for in ways that don’t involve generating slop or being told to eat bromine. However, if your main concerns about AI are environmental, I would caution that a lot of the water-usage claims are extremely dubious or outright lies. This is extremely frustrating, because AI has plenty of real issues, like being used to mass-manufacture scams and misinformation on an industrial scale never seen before, bot crawlers destroying the entire open internet, AI psychosis, and data centers putting enormous strain on local power grids that aren’t given enough time to adapt (which is driving the spike in CO2 emissions).

Trying to figure out if AI is fundamentally good or bad, however, is beyond the scope of this post - I simply hope you have a better idea of what the current technology is capable of and why it behaves the way it does. Hopefully this can inform discussions about whether or not we should even be using this technology.