Appliances And AI In Software Development
I wanted to wait longer to write this, because AI still isn’t very good at coding except in narrow circumstances when used by experts. This makes it difficult to explain that in the near future, AI will likely be better than every human on earth at most mundane coding tasks, and programmers will have to grapple with when it’s appropriate to use, versus when the sanctity of code as an art form should be preserved. Unfortunately, it’s becoming clear that we need to have this conversation a lot sooner than I’d like.
Despite the overblown hype on Project Glasswing, random open-weight models running in people’s basements are currently superhumanly effective at finding critical security vulnerabilities in software. While most people are still making fun of vibe-coded garbage websites, security professionals and IT admins currently live in a constant waking nightmare of new catastrophic remote code execution attacks on random software packages that nobody was expecting, like the merge operation in Git, or Windows Paint! Luckily, the models are not nearly as good at writing exploits as they are at finding them, for now, but it’s not clear how long this will last.
Given how incredibly bad AI currently is at maintaining any kind of non-trivial codebase without frequent corrections from human supervisors, I probably sound completely insane to many people for even suggesting that AI could soon become better at humans at any part of software engineering. Unfortunately, I personally laughed off the possibility that AI would be able to find security issues just a year ago, and I have been proven extremely wrong. This is a reality that we’ll have to confront in just a few years, or possibly sooner. If this post seems a bit too far-fetched for you right now, come back in 12 months and it’ll make a lot more sense, but current events have forced my hand.
Naturally, I’m talking about the Big Walk problem. To summarize, Big Walk had a serious performance issue with shadow cascades in Unity. They asked the Unity devs to help them, and the Unity devs said they couldn’t fix it, and nobody else could either, so they asked an LLM to help fix it, and the LLM found the problem in unity and gave them a 10-line patch to fix it. So technically, 10 lines of code in Big Walk were written by AI, which ultimately got them disqualified from the Indie Game Awards. I have previously pointed out that this seems a little silly given that the fact they used Unity to make their game, which definitely uses AI, yet somehow using Unity is perfectly acceptable, but the game devs fixing a problem in Unity with AI isn’t?
Big Walk is in trouble for other reasons too, but the 10 lines of code is what a lot of debates were initially about, and I want to focus on why nobody could agree on what to do about it: It’s actually a disagreement over appliances. An appliance is simply a inconsequential tool that is used to solve a problem or automate a particular kind of workload, like a washing machine. It isn’t expected to be the best possible solution, because that would be a high-quality work of craftsmanship, it is only expected to be an affordable, good enough solution for the average person. For most people, their computer is an appliance, as is the operating system that runs it. Unity is also an appliance - it’s a game engine that works well enough for most game devs. Appliances exist because most people don’t want to build an entire custom game engine for every single game.
However, what one person might consider an appliance is a labor of love to another. Anyone using Gentoo Linux as their main driver cannot possibly consider their computer to be an appliance, because it requires active maintenance and tinkering to keep it running smoothly. Some people enjoy fiddling around with their Linux installations for hours upon hours every day, trying new things, seeing what works and what doesn’t. The polar opposite of this are Macs, which are designed to be appliances and get very angry if you try to use them as anything other than an appliance. Hence, when we look at what someone is spending their time on, we need to keep in mind which things they are treating as appliances, and which things they are actively working on. This is, unfortunately, entirely subjective.
That means the line that determines “where is it okay to use AI” is also subjective, because it depends on what you consider an appliance. Game devs are almost expected to use an off-the-shelf game engine, and for good reason - I built my own game engine and never actually made a game with it. However, not writing your own render pipeline in something like the demoscene would be considered extremely distasteful - re-using large parts of someone else’s code would be downright unthinkable, because the entire point of a 64k demo is your team’s unique technical approach to artistic expression under extreme space constraints (64000 bytes). The context determines what tools, appliances, or pre-built components are acceptable to use.
In this way, I think software engineering is starting to look a lot like musical genres, where a particular technique considered routine in one musical genre would be considered outright fraud in another, as discussed by Adam Neely. To summarize: Musical genres are largely defined by their unique ethical code, and the ethical code for one genre often seems like completely nonsense when applied to another. In Jazz, the quality of your improvisation is paramount - attempting to play a solo you wrote ahead of time while ignoring the other performers is heresy. However, in Rock, the quality of your performance is paramount - the entire point is to practice a solo you wrote ahead of time over and over and over until you can play it flawlessly, live on stage. The context in which a work of art is presented matters, because it determines what techniques, tools, or appliances are acceptable to use in that context.
The reality is that the future of software development will be rent in twain, if it hasn’t already. I agree with fellow programmers who argue that coding is a form self-expression, but at the same time I don’t really want someone expressing themselves in the firmware for my washing machine - it’s an appliance, I just want it to work. With that said, even in a world filled with AI that can write provably correct firmware and boilerplate, I still think human expression has a place. Humans still play chess even though the phone in your pocket can defeat every chess grandmaster that has ever lived. Engines are obviously banned when playing against someone, but grandmasters use those chess engines to explore new lines and practice new techniques.
There are some obvious parallels to game development here - Game Jams will almost certainly completely ban all AI usage and provide whitelists of acceptable pre-made components, because the entire point of the jam is to see what a human can accomplish, not a machine. However, when we step outside of competitions, things start getting blurry - is it okay to use a game engine written by AI? If not, is it okay to use a game engine written in an IDE written by AI? Is it okay if the autocomplete is powered by AI? Is it okay to write it on an OS written by AI? Is it okay to write it on hardware whose firmware is written by AI? What if the hardware was designed by AI? The line has to be drawn somewhere, and what someone considers an appliance is going to heavily influence where they think the line should be.
Game developers may end up developing human-maintained “silos” isolated from AI influence in order to preserve the sanctity of their craft, possibly even limiting development to approved, human-designed hardware specifications. However, given historical trends, it seems likely that we will end up with many different silos, each choosing a different layer where they isolate themselves from AI. Some may allow AI in carefully controlled components. Some may only allow certain types of AI, or approved AI models running on known hardware. We’ll have to navigate a frightening new world with consequences we don’t understand, but I find it extremely unlikely that AI will be completely expelled from the entire hardware stack, because we tend to explore all the possible permutations of how to use new technologies. Right now, we simply don’t have well-defined rules, or even social frameworks, for handling this, and we need to fix this as soon as possible - should AI progress continue unabated, AI becoming capable of writing robust code is almost inevitable.
However, AI itself is not inevitable, and if the current AI labs continue their wildly short-sighted diesel fueled idiocy instead of riding on the exponential solar boom, this ai bubble might end in riots burning down the data centers, rendering all my pontificating moot. Sometimes though, I feel like we’re all so traumatized by modern society that we treat anyone who has different values as evil, instead of just someone with different priorities. I don’t know what’s going to happen - I personally think “removing all AI from everything” is extremely unlikely unless people actually pull off a butlerian jihad, and if that doesn’t happen soon, we’re going to live in a world where AIs can write better firmware than any human.
Basically, if you’re creatively expressing yourself, you should never allow an AI to touch any part of the creative process - but not everyone thinks writing glue code for the OpenGL C headers is “creative expression”, and that’s where most disagreements seem to originate from.