Erik McClure

The most surreal experience I ever had on discord was when someone PMed me to complain that my anti-spam bot wasn’t working against a 200+ bot raid. I pointed out that it was never designed for large-scale attacks, and that discord’s own rate-limiting would likely make it useless. He revealed he was selling spambot accounts at a rate of about $1 for 100 unique accounts and that he was being attacked by a rival spammer. My anti-spam bot had been dragged into a turf war between two spambot networks. We discussed possible mitigation strategies for worst-case scenarios, but agreed that most of them would involve false-positives and that discord showed no interest in fixing how exploitable their API was. I hoped that I would never have to implement such extreme measures into my bot.

Yesterday, our server was attacked by over 40 spambots, and after discord’s astonishingly useless “customer service” response, I was forced to do exactly that.

A Brief History of Discord Bots

Discord is built on a REST API, which was reverse engineered by late 2015 and used to make unofficial bots. To test out their bots, they would hunt for servers to “raid”, invite their bots to the server, then spam so many messages it would softlock the client, because discord still didn’t have any rate limiting. Naturally, as the designated punching bags of the internet, furries/bronies/Twilight fans/slash fiction writers/etc. were among the first targets. The attack on our server was so severe it took us almost 5 minutes of wrestling with an unresponsive client to ban them. Ironically, a few of the more popular bots today, such as “BooBot”, are banned as a result of that attack, because the first thing the bot creator did was use it to raid our server.

I immediately went to work building an anti-spam bot that muted anyone sending more than 4 messages per second. Building a program in a hostile environment like this is much different from writing a desktop app or a game, because the bot had to be bulletproof - it had to rate-limit itself and could not be allowed to crash, ever. Any bug that allowed a user to crash the bot was treated as P0, because it could be used by an attacker to cripple the server. Despite using a very simplistic spam detection algorithm, this turned out to be highly effective. Of course, back then, discord didn’t have rate limiting, or verification, or role hierarchies, or searching chat logs, or even a way to look up where your last ping was, so most spammers were probably not accustomed to having to deal with any kind of anti-spam system.

I added raid detection, autosilence, an isolation channel, and join alerts, but eventually we were targeted by a group from 4chan’s /pol/ board. Because this was a sustained attack, they began crafting spam attacks timed just below the anti-spam threshold. This forced me to implement a much more sophisticated anti-spam system, using a heat algorithm with a linear decay rate, which is still in use today. This improved anti-spam system eventually made the /pol/ group give up entirely. I’m honestly amazed the simplistic “X messages in Y seconds” approach worked as long as it did.

Of course, none of this can defend against a large scale attack. As I learned by my chance encounter with an actual spammer, it was getting easier and easier to amass an army of spambots to assault a channel instead of just using one or two.

Anatomy Of A Modern Spambot Attack

At peak times (usually during summer break), our server gets raided 1-2 times per day. These minor raids are often just 2-3 tweens who either attempt to troll the chat, or use a basic user script to spam an offensive message. Roughly 60-70% of these raids are either painfully obvious or immediately trigger the anti-spam bot. About 20% of the raids involve slightly intelligent attempts to troll the chat by being annoying without breaking the rules, which usually take about 5-10 minutes to be “exposed”. About 5-10% of the raids are large, involving 8 or more people, but they are also very obvious and can be easily confined to an isolation channel. Problems arise, however, with large spambot raids. Below is a timeline of the recent spambot attack on our server:

This was a botched raid, but the bots that actually worked started spamming within 5 seconds of joining, giving the moderators a very narrow window to respond. The real problem, however, is that so many of them joined, the bot’s API calls to add a role to silence them were rate-limited. They also sent messages once every 0.9 seconds, which is designed to get around Discord’s rate limiting. This amounted to 33 messages sent every second, but it was difficult for the anti-spam to detect. Had the spambots reduced their spam cadence to 3 seconds or more, this attack could have bypassed the anti-spam detection entirely. My bot now instigates a lockdown by raising the verification level when a raid is detected, but it simply can’t silence users fast enough to deal with hundreds of spambots, so at some point the moderators must use a mass ban function. Of course, banning is restricted by the global rate limit, because Discord has no mass ban API endpoint, but luckily the global rate limit is something like 50 requests per second, so if you’re only banning people, you’re probably okay.

However, a hostile attacker could sneak bots in one-by-one every 10 minutes or so, avoiding setting off the raid alarm, and then activate them all at once. 500 bots sending randomized messages chosen from an English dictionary once every 5 seconds after sneaking them in over a 48 hour period is the ultimate attack, and one that is almost impossible to defend against, because this also bypasses the 10-minute verification level. As a weapon of last resort, I added a command that immediately bans all users that sent their first message within the past two minutes, but, again, banning is subject to the global rate limit! In fact, the rate limits can change at any time, and while message deletion has a higher rate limit for bots, bans don’t.

The only other option is to disable the @everyone role from being able to speak on any channel, but you have to do this on a per channel basis, because Discord ignores you if you attempt to globally disable sending message permissions for @everyone. Even then, creating an “approved” role doesn’t work because any automated assignment could be defeated by adding bots one by one. The only defense a small Discord server has is to require moderator approval for every single new user, which isn’t a solution - you’ve just given up having a public Discord server. It’s only a matter of time until any angry 13-year-old can buy a sophisticated attack with a week’s allowance. What will happen to public Discord servers then? Do we simply throw up our hands and admit that humanity is so awful we can’t even have public communities anymore?

The Discord API Hates You

The rate-limits imposed on Discord API endpoints are exacerbated by temporary failures, and that’s excluding network issues. Thus, if I attempt to set a silence role on a spammer that just joined, the API will repeatedly claim they do not exist. In fact, 3 separate API endpoints consistently fail to operate properly during a raid: A “member joined” event won’t show up for several seconds, but if I fall back to calling GetMember(), this also claims the member doesn’t exist, which means adding the role also fails! So I have to attempt to silence the user with every message they send until Discord actually adds the role, even though the API failures are also counted against the rate limit! This gets completely absurd once someone assaults your server with 1000 spambots, because this triggers all sorts of bottlenecks that normally aren’t a problem. The alert telling you a user has joined? Rate limited. It’ll take your bot 5-10 minutes to get through just telling you such a gigantic spambot army joined, unless you include code specifically designed to detect these situations and reduce the number of alerts. Because of this, a single user can trigger something like 5-6 API requests, all of which are counted against your global rate limit and can severely cripple a bot.

The general advice that is usually given here is “just ban them”, which is terrible advice because Discord’s own awful message handling makes it incredibly easy to trigger a false positive. If a message fails to send, the client simply sends a completely new message, with it’s own ID, and will continue re-sending the message until an Ack is received, at which point the user has probably send 3 or 4 copies of the same message, each of which have the same content, but completely unique IDs and timestamps, which looks completely identical to a spam attack.

Technically speaking, this is done because Discord assigns snowflake IDs server-side, so each message attempt sent by the client must have a unique snowflake assigned after it is sent. However, it can also be trivially fixed by adding an optional “client ID” field to the message, with a client-generated ID that stays the same if the message is resent due to a network failure. That way, the server (or the other clients) can simply drop any duplicate messages with identical client IDs while still ensuring all messages have unique IDs across their distributed cluster. This would single-handedly fix all duplicate messages across the entire platform, and eliminate almost every single false-positive I’ve seen in my anti-spam bot.

Discord Doesn’t Care

Sadly, Discord doesn’t seem to care. The general advice in response to “how do I defend against a large scale spam attack” is “just report them to us”, so we did exactly that, and then got what has to be one of the dumbest customer service e-mails I’ve ever seen in my life:

Excuse me, WHAT?! Sorry about somebody spamming your service with horrifying gore images, but please don’t delete them! What happens if the spammers just delete the messages themselves? What happens if they send child porn? “Sorry guys, please ignore the images that are literally illegal to even look at, but we can’t delete them because Discord is fucking stupid.” Does Discord understand the concept of marking messages for deletion so they are viewable for a short time as evidence for law enforcement?! My anti-spam bot’s database currently has more information than Discord’s own servers! If this had involved child porn, the FBI would have had to ask me for my records because Discord would have deleted them all!

Obviously, we’re not going to leave 500+ gore messages sitting in the chatroom while Discord’s ass-backwards abuse team analyzes them. I just have to hope my own nuclear option can ban them quickly enough, or simply give up the entire concept of having a public Discord server.

The problem is that the armies of spambots that were once reserved for the big servers are now so easy and so trivial to make that they’re beginning to target smaller servers, servers that don’t have the resources or the means to deal with that kind of large scale DDoS attack. So instead, I have to fight the growing swarm alone, armed with only a crippled, rate-limited bot of my own, and hope the dragons flying overhead don’t notice.

What the fuck, Discord.

One of the most common things I hear from people is that companies should be “apolitical”. The most formal way this concept is expressed is that a company should make decisions based on what maximizes profits and not political opinions. Unfortunately, the statement “companies should only care about maximizing profits” is, itself, a political statement (and one I happen to disagree with). Thus, it is fundamentally impossible for a company to be truly apolitical, for the very act of attempting to be apolitical is a political statement.

How much a company can avoid politics generally depends on both the type and size of the company. Once your company becomes large enough, it will influence politics simply by virtue of its enormous size, and eventually becomes an integral part of political debates whether or wants to or not. Large corporations must take into account the political climate when making business decisions, because simply attempting to blindly maximize profit may turn the public against them and destroy their revenue sources—thus, politics themselves become part of the profit equation, and cannot be ignored. Certain types of businesses embody political statements simply by existing. Grindr, for example, is a dating app for gay men. It’s entire business model is dependent on enabling an activity that certain fundamentalists consider inherently immoral.

You could, theoretically, try to solve part of this quandary by saying that companies should also be amoral, insofar that the free market should decide moral values. The fundamentalists would then protest the companies existence by not using it (but then, they never would have used it in the first place). However, the problem is that, once again, this very statement is itself political in nature. Thus, by either trying to be amoral or moral, a company is making a political statement.

The issue at play here is that literally everything is political. When most everyone agrees on basic moral principles, it’s easier to pretend that politics is really just about economic policy and lawyers, but our current political divisions have demonstrated that this is a fantasy. Politics are the fundamental morals that society has decided on. It’s just a lot easier to argue about minor differences in economic policy instead of fundamental differences in basic morality.

Of course, how companies participate in politics is also important to consider. Right now, a lot of companies participate in politics by spending exorbitant amounts of money on lobbyists. This is a symptom of money in general, and should be solved not by removing corporate money from politics, but removing all money, because treating spending money as a form of speech gives more speech to the rich, which inherently discriminates against the poor and violates the constitutional assertion that all men are created equal (but no one really seems to be paying attention to that line anyway).

Instead of using money, corporations should do things that uphold whatever political values they believe in. As the saying goes, actions speak louder than words (or money, in this case). You could support civil rights activism by being more inclusive with your hiring and promoting a diverse work environment. Or, if you live in the Philippines, you could create an app that helps death squads hunt down drug users so they can be brutally executed. What’s interesting is that most people consider the latter to be a moral issue as opposed to a political one, which seems to derive from the fact that once you agree on most fundamental morals, we humans simply make up a bunch of pointless rules to satisfy our insatiable desire to tell other humans they’re wrong.

We’ve lived in a civilized world for so long, we’ve forgotten the true roots of politics: a clash between our fundamental moral beliefs, not about how much parking fines should be. Your company will make a political statement whether you like it or not, so you’d better make sure it’s the one you want.

It’s been known for a while that windows has a bad habit of eating your exceptions if you’re inside a WinProc callback function. This behavior can cause all sorts of mayhem, like your program just vanishing into thin air without any error messages due to a stack overflow that terminated the program without actually throwing an exception. What I didn’t realize is that it also eats assert(), which makes debugging hell, because the assertion would throw, the entire user callback would immediately terminate without any stack unwinding, and then windows would just… keep going, even though the program is now in a laughably corrupt state, because only half the function executed.

While trying to find a way to fix this, I discovered that there are no less than 4 different ways windows can choose to eat exceptions from your program. I had already told the kernel to stop eating my exceptions using the following code:

HMODULE kernel32 = LoadLibraryA("kernel32.dll");   
assert(kernel32 != 0);   
tGetPolicy pGetPolicy = (tGetPolicy)GetProcAddress(kernel32, "GetProcessUserModeExceptionPolicy");  
tSetPolicy pSetPolicy = (tSetPolicy)GetProcAddress(kernel32, "SetProcessUserModeExceptionPolicy");   
if(pGetPolicy && pSetPolicy && pGetPolicy(&dwFlags))  
  pSetPolicy(dwFlags & \~EXCEPTION_SWALLOWING); // Turn off the filter

CoInitialize(NULL); // do this first   
if(SUCCEEDED(CoInitializeSecurity(NULL, -1, NULL, NULL, RPC_C_AUTHN_L_VEL_PKT_PRIVACY, RPC_C_IMP_LEV_L_IMPERSONATE, NULL, EOAC_DYNAMIC_CLOAKING, NULL)))
{
  IGlobalOptions *pGlobalOptions;
  hr = CoCreateInstance(CLSID_GlobalOptions, NULL, CLSCTX_INPROC_SERVER, IID_PPV_ARGS(&pGlobalOptions));
  if(SUCCEEDED(hr))
  {
    hr = pGlobalOptions->Set(COMGLB_EXCEPTION_HANDLING, COMGLB_EXCEPTION_DONOT_HANDLE);
    pGlobalOptions->Release();
  }
}

Assertion failed!

Program: ...udio 2015\\Projects\\feathergui\\bin\\fgDirect2D_d.dll
File: fgEffectBase.cpp
Line: 20

Expression: sizeof(_constants) == sizeof(float)_(4_4 + 2)

Any good developer should be using assertions to verify their assumptions, so having assertions silently fail and then corrupt the program is even worse than ignoring they exist! Now you could have an assertion in your code that’s firing, terminating that callback, leaving your program in a broken state, and then the next message that’s processed blows up for strange and bizarre reasons that make no sense because they’re impossible.

I have a hard enough time getting my programs to work, I didn’t think it’d be this hard to make them crash.

About two months ago, I got a new laptop and proceeded to load all my projects on it. Despite compiling everything fine, my graphics engine that used DirectX mysteriously crashed upon running. I immediately suspected either a configuration issue or a driver issue, but this seemed weird because my laptop had a newer graphics card than my desktop. Why was it crashing on newer hardware? Things got even more bizarre once I narrowed down the issue - it was in my shader assignment code, which hadn’t been touched in almost 2 years. While I initially suspected a shader compilation issue, there was no such error in the logs. All the shaders compiled fine, and then… didn’t work.

Now, if this error had also been happening on my desktop, I would have immediately started digging through my constant assignments, followed by the vertex buffers assigned to the shader, but again, all of this ran perfectly fine on my desktop. I was completely baffled as to why things weren’t working properly. I had eliminated all possible errors I could think of that would have resulted from moving the project from my desktop to my laptop: none of the media files were missing, all the shaders compiled, all the relative paths were correct, I was using the exact same compiler as before with all the appropriate updates. I even updated drivers on both computers, but it stubbornly refused to work on the laptop while running fine on the desktop.

Then I found something that nearly made me shit my pants.

if(profile <= VERTEX_SHADER_5_0 && _lastVS != shader) {
  //...
} else if(profile <= PIXEL_SHADER_5_0 && _lastPS != shader) { 
  //...
} else if(profile <= GEOMETRY_SHADER_5_0 && _lastGS != shader) {
  //...
}

For almost 2 years, I had been feeding DirectX total bullshit, and had even tested it on multiple other computers, and it had never given me a single warning, error, crash, or any indication whatsoever that my code was completely fucking broken, in either debug mode or release mode. Somehow, deep in the depths of nVidia’s insane DirectX driver, it had managed to detect that I had just tried to assign a vertex shader to a pixel shader, and either ignored it completely, or silently fixed my catastrophic fuckup. However, my laptop had the mobile drivers, which for some reason did not include this failsafe, and so it actually crashed like it was supposed to.

While this was an incredibly stupid bug that I must have written while sleep deprived or drunk (which is impressive, because I don’t actually drink), it was simply impossible for me to catch because it produced zero errors or warnings. As a result, this bug has the honor of being both the dumbest and the longest-living bug of mine, ever. I’ve checked every location I know of for any indication that anything was wrong, including hooking into the debug log of directX and dumping all it’s messages. Nothing. Nada. Zilch. Zero.

I’ve heard stories about the insane bullshit nVidia’s drivers do, but this is fucking terrifying.

Alas, there is more. I had been experimenting with direct2D as an alternative because, well, it’s a 2D engine, right? After getting text rendering working, a change in string caching suddenly broke the entire program. It broke in a particularly bizarre way, because it seemed to just stop rendering halfway through the scene. It took almost an hour of debugging for me to finally confirm that the moment I was rendering a particular text string, the direct2D driver just stopped. No errors were thrown. No warnings could be found. Direct2D’s failure state was apparently to simply make every single function call silently fail with no indication that it was failing in the first place. It didn’t even tell me that the device was missing or that I needed to recreate it. The text render call was made and then every single subsequent call was ignored and the backbuffer was forever frozen to that half-drawn frame.

The error itself didn’t seem to make any more sense, either. I was passing a perfectly valid string to Direct2D, but because that string originated in a different DLL, it apparently made Direct2D completely shit itself. Copying the string onto the stack, however, worked (which itself could only work if the original string was valid).

The cherry on top of all this is when I discovered that Direct2D’s matrix rotation constructor takes degrees, not radians, like every single other mathematical function in the standard library. Even DirectX takes radians!

WHO WRITES THESE APIs?!

Did you know that CSS3 does all its linear gradients and color interpolation completely wrong? All color values in CSS3 are in the sRGB color space, because that’s the color space that gets displayed on our monitor. However, the problem is that the sRGB color space looks like this:

Trying to do a linear interpolation along a nonlinear curve doesn’t work very well. Instead, you’re supposed to linearize your color values, transforming the sRGB curve to the linear RGB curve before doing your operation, and then transforming it back into the sRGB curve. This is gamma correction. Here are comparisons between gradients, transitions, alpha blending, and image resizing done directly in sRGB space (assuming your browser complies with the W3C spec) versus in linear RGB:

sRGB	Linear

The amazing thing here is that the W3C is entirely aware of how wrong CSS3 linear gradients are, but did it anyway to be consistent with everything else that does them wrong. It’s interesting that while SVG is wrong by default, it does provide a way to fix this, via color-interpolation. Of course, CSS doesn’t have this property yet, so literally all gradients and transitions on the web are wrong because there is no other choice. Even if CSS provided a way to fix this, it would still have to default to being wrong.

It seems we have reached a point where, after years of doing sRGB interpolation incorrectly, we continue to do it wrong not because we don’t know better, but because everyone else is doing it wrong. So everyone’s doing it wrong because everyone else is doing it wrong. A single bad choice done long ago has locked us into compatibility hell. We got it wrong the first time so now we have to keep getting it wrong because everyone expects the wrong result.

It doesn’t help that we don’t always necessarily want the correct interpolation. The reason direct interpolation in sRGB is wrong is because it changes the perceived luminosity. Notice that when going from red to green, the sRGB gradient gets darker in the middle of the transition, while the gamma-corrected one has constant perceived luminosity. However, an artist may prefer the sRGB curve to the linearized one because it puts more emphasis on red and green. This problem gets worse when artists use toolchains inside sRGB and unknowingly compensate for any gamma errors such that the result is roughly what one would expect. This is a particular issue in games, because games really do need gamma-correct lighting pipelines, but the GUIs were often built using incorrect sRGB interpolation, so doing gamma-correct alpha blending gives you the wrong result because the alpha values were already chosen to compensate for incorrect blending.

In short, this is quite a mess we’ve gotten ourselves into, but I think the most important thing we can do is give people the option of having a gamma correct pipeline. It is not difficult to selectively blend things with proper gamma correction. We need to have things like SVG’s color-interpolation property in CSS, and other software needs to provide optional gamma correct pipelines (I’m looking at you, photoshop).

Maybe, eventually, we can dig ourselves out of this sRGB hell we’ve gotten ourselves into.