Erik McClure

Why You Can't Use Prebuilt LLVM 10.0 with C++17


C++17 introduced an alignment argument to ::operator new(). It's important to note that if you allocate something using aligned new, you absolutely must deallocate it using aligned delete, or the behavior is undefined. LLVM 10.x takes advantage of this alignment parameter, if the compiler supports it. That means if you are compiling on Windows with MSVC set to C++14, __cpp_aligned_new is not defined and the extra argument isn't passed. Otherwise, if it's compiled with MSVC set to C++17, __cpp_aligned_new is defined and the extra argument is passed.

There's just one teeny tiny little problem - the check was in a header file.

Now, if this header file were completely internal and LLVM itself only ever called ::operator new() from inside it's libraries, this wouldn't be a problem. As you might have guessed, this was not the case. LLVM was calling allocate_buffer() from inside header files. The corresponding deallocate_buffer call was inside the same header file, but sadly, it was called from a destructor, and that destructor had been called from inside one of LLVM's libraries, which meant it didn't know about aligned allocations… Oops!

This means, if your program uses C++17, but LLVM was compiled with C++14, your program will happily pass LLVM aligned memory, which LLVM will then pass into the wrong delete operator, because it's calling the unaligned delete function from C++14 instead of the aligned delete function from C++17. This results in strange and bizarre heap corruption errors because of the mismatched ::operator new and ::operator delete. Arguably, the real bug here is LLVM calling any allocation function from a header file in the first place, as this is just begging for ABI problems.

Of course, one might ask why a C++17 application would be linking against LLVM compiled with C++14. Well, you see, the prebuilt LLVM binaries were compiled against C++14… OOPS! Suddenly if you wanted to avoid compiling LLVM yourself, you couldn't use C++17 anymore!

Luckily this has now been fixed after a bunch of people complained about the extremely unintuitive errors resulting from this mismatch. Unfortunately, as of this writing, LLVM still hasn't provided a new release, which means you will still encounter this problem using the latest precompiled binaries of LLVM. Personally, I recompile LLVM using C++17 for my own projects, just to be safe, since LLVM does not gaurantee ABI compatibility in these situations.

Still, this is a particularly nasty case of an unintentional ABI break between C++ versions, which is easy to miss because most people assume C++ versions are backwards compatible. Be careful, and stop allocating memory in header files that might be deallocated inside your library!


Someone Is Stealing Tracker Songs And Selling Them


Today, in response to a tweet talking about old untitled song ideas, I mentioned that I had a strange file called “t.mp3” sitting in my downloads folder that had been there for years and have no attached metadata or hint as to where it came from. It appeared to be a complete recording of a chiptune song, and it sounded very nice, but I had no way of knowing the original source. To my surprise, someone else had heard the song and hunted down a source on youtube: S0 C1os3 - H4X0r.

But something was wrong. That video was uploaded in 2019. The file I had was last modified in January 2010, over 10 years ago, and for that matter it had been sitting in my downloads folder for almost as long. Perhaps the artist simply hadn't uploaded it to YouTube until recently? I decided to scroll through the comments, and discovered that someone posted the real name of the track: “So Close” by Floppi. I was even able to find the original XM file, and sure enough, it had been uploaded way back in 2003.

Now, some random person uploading an old song on youtube and trying to take credit isn't really that surprising, but this Youtube channel is special - it's a Topic channel, which is autogenerated from the Google Play Store. It even says in the description, “Provided to YouTube by DistroKid”. DistroKid is a music distribution service that automatically registers and uploads your album to all the major online music stores. I've used a similar service for my own albums, and do in fact have my own autogenerated Topic channel on Youtube, which is… kind of creepy, but that's besides the point.

The point is that someone has stolen 11 classic demoscene tracker songs and is selling them on every major online music store. The album is called “H4x0r R007z” and it consists entirely of blatantly stolen tracker songs lazily renamed using 1337 text, which usually means finding the original song is pretty easy. Because it was published via DistroKid, you can find it on Spotify, Google Play, iTunes, Amazon, Deezer, iHeartRadio and it's been registered into the automatically populated copyrighted songs databases! That means the artist “H4x0r” could legally issue copyright takedowns for every other legitimate upload of the actual songs, or abuse Google's automatic fingerprinting system to force monetization on all of the videos and take all the income.

The songs that were stolen were often used in keygen cracking software that was popular in the 2000s to pirate software. Many people have noted that this was often their first introduction to tracker music, and this is likely the source of the name “H4x0r R007z”, even though the songs themselves usually had nothing to do with the hacker groups. The original tracker files are still available in the scene archives, and other Youtube channels have uploaded MP3 recordings of the songs with proper attribution. I was able to source 9 out of the 11 songs from the album, with the real name and artist, plus a Youtube or Soundcloud recording (if you'd like to play the original files yourself, you can use XMPlay).

  1. 0 Pr1m4ry
    Real Name: “primary” by Ramosa
    Original file: ramosa-primary.mod
    Only exists on the H4x0r channel
  2. 5murf-E5Que
    Real Name: “Smurf-Esque ‘98” by AGAiN
    Author Info: Made with FastTracker 2 by Three Little Elks (3LE): Android (Jonas Kapla) Ant (Anton Halldin), Coma (Daniel Johansson), Nude (Sten Ulfsson), Tabasco (Niklas Soerensson)
    Original file: smurf-es.mod
    Youtube
  3. Vib35 F20m Pa57
    Real Name: “Vibes From Past” by Fegolhuzz (Figge Wasberger)
    Original file: fegolhuzz_-_vibes_from_past.xm
    SoundCloud
  4. Bra1n Wound
    Real Name: “Brain Wound” by Ghidorah
    Original file: bw.xm
    Youtube
  5. Hy8r1d 50n9
    Real Name: “Funky Stars (Hybrid Song)” by Quazar (Axel Christofer Hedfors)
    Author Info: Quazar is an old alias of Axwell, one of the members of the Swedish House Mafia.
    Original file: external.xm
    Youtube
  6. D0d3ch3dr0n
    I was unable to source this song, possibly because the original song name was a mispelling of Dodecahedron
  7. 51ne-15te2
    Real Name: “Sine-ister” by dreamfish
    Original file: sineiste.mod
    Youtube (Some recordings seem to have more clarity than XMPlay provides, but it's not clear if this was intended)
  8. Au70ma71k
    Real Name: “Automatik” by Rez
    Original file: rez-automatik.mod
    Youtube
  9. S3rV1l
    I was also unable to source this song, possibly due to spelling errors.
  10. S0 C1os3
    Real Name: “So Close” by Floppi (Tero Laihanen)
    Original file: intro13.xm
    Youtube
  11. 5or3 Po1nt
    Real Name: “Sore Point”
    Author Information: The original author of this song is unknown, although it was converted to .xm format for Infinitode 2. It was featured in several keygens made by AiR.
    Original file: Unknown. Various conversions to other formats are available.
    Youtube

Pressure Based Anti-Spam for Discord Bots


Back when Discord was a wee little chat platform with no rate limiting whatsoever, it's API had already been reverse-engineered by a bunch of bot developers who then went around spamming random servers with so many messages it would crash the client. My friends and I were determined to keep our discord server public, so I went about creating the first anti-spam bot for Discord.

I am no longer maintaining this bot beyond simple bugfixes because I have better things to do with my time, and I'm tired of people trying to use it because “it's the best anti-spam bot” even after I deprecated it. This post is an effort to educate all the other anti-spam bots on how to ascend beyond a simple “mute someone if they send more than N messages in M seconds” filter and hopefully make better anti-spam bots so I don't have to.

Architecture

There are disagreements over exactly how an anti-spam bot should work, with a wide-range of preferred behaviors that differ between servers, depending on the community, size of the server, rate of growth of the server, and the administrators personal preferences. Many anti-spam solutions lock down the server by forcing new members to go through an airlock channel, but I consider this overkill and the result of poor anti-spam bots that are bad at actually stopping trolls or responding to raids.

My moderation architecture goes like this:

-- Channels --  
#Moderation Channel  
#Raid Containment  
#Silence Containment  
#Log Channel

-- Roles --  
@Member  
@Silence  
@New

When first-time setup is run on a server, the bot queries and stores all the permissions given to the @everyone role. It adds all these permissions to the @Member role, then goes through all existing users and adds the @Member role to them. It then removes all permissions from @everyone, but adds an override for #Raid Containment so that anyone without the @Member role can only speak in #Raid Containment.

Then, it creates the @Silence role by adding permission overrides to every single channel on the server that prevent you from sending messages on that channel, except for the #Silence Containment channel. This ensures that, no matter what new roles might be added in the future, @Silence will always prevent you from sending messages on any channel other than #Silence Containment. The admins of the server can configure the visibility of the containment channels. Most servers make it so #Raid Containment is only visible to anyone without the @Member role, ensure #Silence Containment is only visible to anyone with the @Silence role, and hide #Log Channel from everyone except admins.

This architecture was chosen to allow the bot to survive a mega-raid, so that even if 500+ bot accounts storm the server all at once, if the bot goes down or is rate-limited, they can't do anything because they haven't been given the @Member role, and so the server is protected by default. This is essentially an automated airlock that only engages if the bot detects a raid, instead of forcing all new members to go through the airlock. Some admins do not like this approach because they prefer to personally audit every new member, but this is not viable for larger servers.

The @Member role being added can also short-circuit Discord's own verification rules, which is an unfortunate consequence, but in practice it usually isn't a problem. However, to help compensate for this, the bot can also be configured to add a temporary @New role to newer users that expires after a configurable amount of time. What this role actually does is up to the administrators - usually it simply disables sharing images or embeds.

Operation

The bot has two distinct modes of operation. Under normal operation, when a new member joins the server, they are automatically given @Member (and @New if it's been enabled) and are immediately allowed to speak. If they trigger the anti-spam, or a moderator uses the !silence command, they will have the @Silence role added, any messages sent in the last 5 seconds (by default) will be deleted, and they'll be restricted to speaking in #Silence Containment. By default, silences that happen automatically are never rescinded until a moderator manually unsilences someone. However, some server admins are lazy, so an automatic expiration can be added. A manual silence can also be configured to expire after a set period of time. If a user triggers the anti-spam filter again while they are in the containment channel, then the bot will automatically ban them.

It's important to note that when silenced, users have both the @Member and the @Silence role. The reason is because many servers have opt-in channels that are only accessible if you add a role to yourself. Simply removing @Member would not suffice to keep them from talking in these channels, but @Silence can override these permissions and ensure they can't speak in any channel without having to mess up anyone's assigned roles.

The bot detects a raid if N joins happen within M seconds. It's usually configured to be something like 3 joins in a 90 second interval for an active server. When this happens, the server enters raid mode, which automatically removes @Member from all the users that just triggered the raid alert. It sends a message pinging the moderators in #Moderator Channel and stops adding @Member to anyone who joins while it is still in raid mode. It also changes the server verification level, which actually does have an effect for the newly joined members, because they haven't had any roles assigned to them yet. Moderators can either cancel the raid alert, which automatically adds @Member to everyone who was detected as part of the raid alert, or they can individually add @Member to people they have vetted. The raid mode automatically ends after M*2 seconds, which would be 180 in this example. A moderator can use a command to ban everyone who joined during the raid alert, or they can selectively add @Member to users who should be let in.

This method of dealing with raids ensures that the bot cannot be overwhelmed by the sheer number of joins. If it crashes or is taken down, the server stays in raid mode until the bot can recover, and the potential raiders will not be allowed in.

Pressure System

All the anti-spam logic in the bot is done by triggering an automatic silence, which gives someone the @Silence role. The bot determines when to silence someone by analyzing all the messages being sent using a pressure system. In essence, the pressure system is analogous to the gravity function used for calculating the “hotness” of a given post on a site like Hacker News.

Each message a user sends is assigned a “pressure score”. This is calculated from the message length, contents, whether it matches a filter, how many newlines it has, etc. This “pressure” is designed to simulate how disruptive the message is to the current chat. A wall of text is extremely disruptive, whereas saying “hi” probably isn't. Attaching 3 images to your message is very disruptive, but embedding a single link isn't that bad. Sending a blank message with 2000 newlines is incredibly disruptive, whereas sending a message with two paragraphs is probably fine. In addition, all messages are assigned a base pressure, the minimum amount of pressure that any message will have, even if it's only a single letter.

My bot looks at the following values:

  • Max Pressure: The default maximum pressure is 60.
  • Base Pressure: All messages generate 10 pressure by default, so at most 6 messages can be sent at once.
  • Embed Pressure: Each image, link, or attachment in a message generates 8.3 additional pressure, which limits users to 5 images or links in a single message.
  • Length Pressure: Each individual character in a message generates 0.00625 additional pressure, which limits you to sending 2 maximum length messages at once.
  • Line Pressure: Each newline in a message generates 0.714 additional pressure, which limits you to 70 newlines in a single message.
  • Ping Pressure: Every single unique ping in a message generates 2.5 additional pressure, which limits users to 20 pings in a single message.
  • Repeat Pressure: If the message being sent is the exact same as the previous message sent by this user (copy+pasted), it generates 10 additional pressure, effectively doubling the base pressure cost for the message. This means a user copy and pasting the same message more than twice in rapid succession will be silenced.
  • Filter Pressure: Any filter set by the admins can add an arbitrary amount of additional pressure if a message triggers that filter regex. The amount is user-configurable.

And here a simplified version of my implementation. Note that I am adding the pressure piecewise, so that I can alert the moderators and tell them exactly which pressure trigger broke the limit, which helps moderators tell if someone just posted too many pictures at once, or was spamming the same message over and over:

if w.AddPressure(info, m, track, info.Config.Spam.BasePressure) {
	return true
}
if w.AddPressure(info, m, track, info.Config.Spam.EmbedPressure*float32(len(m.Attachments))) {
	return true
}
if w.AddPressure(info, m, track, info.Config.Spam.EmbedPressure*float32(len(m.Embeds))) {
	return true
}
if w.AddPressure(info, m, track, info.Config.Spam.LengthPressure*float32(len(m.Content))) {
	return true
}
if w.AddPressure(info, m, track, info.Config.Spam.LinePressure*float32(strings.Count(m.Content, "\n"))) {
	return true
}
if w.AddPressure(info, m, track, info.Config.Spam.PingPressure*float32(len(m.Mentions))) {
	return true
}
if len(m.Content) > 0 && m.Content == track.lastcache {
	if w.AddPressure(info, m, track, info.Config.Spam.RepeatPressure) {
		return true
	}
}
Once we've calculated how disruptive a given message is, we can add it to the user's total pressure score, which is a measure of how disruptive a user is currently being. However, we need to recognize that sending a wall of text every 10 minutes probably isn't an issue, but sending 10 short messages saying “ROFLCOPTER” in 10 seconds is definitely spamming. So, before we add the message pressure to the user's pressure, we check how long it's been since that user last sent a message, and decrease their pressure accordingly. If it's only been a second, the pressure shouldn't decrease very much, but if it's been 3 minutes or so, any pressure they used to have should probably be gone by now. Most heat algorithms do this non-linearly, but my experiments showed that a linear drop-off tends to produce better results (and is simpler to implement).

My bot implements this by simply decreasing the amount of pressure a user has by a set amount for every N seconds. So if it's been 2 seconds, they will lose 4 pressure, until it reaches zero. Here is the implementation for my bot, which is done before any pressure is added:

timestamp := bot.GetTimestamp(m)
track := w.TrackUser(author, timestamp)
last := track.lastmessage
track.lastmessage = timestamp.Unix()*1000 + int64(timestamp.Nanosecond()/1000000)
if track.lastmessage < last { // This can happen because discord has a bad habit of re-sending timestamps if anything so much as touches a message
	track.lastmessage = last
	return false // An invalid timestamp is never spam
}
interval := track.lastmessage - last

track.pressure -= info.Config.Spam.BasePressure * (float32(interval) / (info.Config.Spam.PressureDecay * 1000.0))
if track.pressure < 0 {
	track.pressure = 0
}
In essence, this entire system is a superset of the more simplistic “N messages in M seconds”. If you only use base pressure and maximum pressure, then these determine the absolute upper limit of how many messages can be send in a given time period, regardless of their content. You then tweak the rest of the pressure values to more quickly catch obvious instances of spamming. The maximum pressure can be altered on a per-channel basis, which allows meme channels to spam messages without triggering anti-spam. Here's what a user's pressure value looks like over time:

Pressure Graph

Because this whole pressure system is integrated into the Regex filtering module, servers can essentially create their own bad-word filters by assigning a huge pressure value to a certain match, which instantly creates a silence. These regexes can be used to look for other things that the bot doesn't currently track, like all-caps messages, or for links to specific websites. The pressure system allows integrating all of these checks in a unified way that represents the total “disruptiveness” of an individual user's behavior.

Worst Case Scenario

A special mention should go to the uncommon but possible worst-case scenario for spamming. This happens when a dedicated attacker really, really wants to fuck up your server for some reason. They can do this by creating 1000+ fake accounts with randomized names, and wait until they've all been on discord for long enough that the “new to discord” message doesn't show up. Then, they have the bots join the server extremely slowly, at the pace of maybe 1 per hour. Then they wait another week or so, before they unleash a spam attack that has each account send exactly 1 message, followed by a different account sending another message.

If they do this fast enough, you can detect it simply by the sheer volume of messages being sent in the channel, and automatically put the channel in slow mode. However, in principle, it is completely impossible to automatically deal with this attack. Banning everyone who happens to be sending messages during the spam event will ban innocent bystanders, and if the individual accounts aren't sending messages that fast (maybe one per second), this is indistinguishable from a normal rapid-fire conversation.

My bot has an emergency method for dealing with this where it tracks when a given user sent their first message in the server. You can then use a command to ban everyone who sent their first message in the past N seconds. This is very effective when it works, but it is trivial for spammers to get past once they realize what's going on - all they need to do is have their fake accounts say “hi” once while they're trickling in. Of course, a bunch of accounts all saying “hi” and then nothing else may raise enough suspicion to catch the attack before it happens.

Conclusion

Hopefully this article demonstrates how to make a reliable, extensible anti-spam bot that can deal with pretty much any imaginable spam attack. The code for my deprecated spam bot is available here for reference, but the code is terrible and most of it was a bad idea. You cannot add the bot itself to a server anymore, and the support channel is no longer accessible.

My hope is that someone can use these guidelines to build a much more effective anti-spam bot, and release me from my torment. I hate web development and I'd rather spend my time building a native webassembly compiler instead of worrying about weird intermittent cloudflare errors that temporarily break everything, or badly designed lock systems that deadlock under rare edge cases. Godspeed, bot developers.


Debugging Through WebAssembly Is Impossible


I recently added some very primitive debugging support to inNative to allow you to step through the original source code during execution via sourcemaps. My client, however, has expressed the need for proper variable inspection in the original C++ code. This is distinct from current debugging methods available in Chrome and Firefox that let you examine the current WebAssembly locals, not the original language. They are using WebAssembly as a plugin system, which makes sense, and they want to have a toolchain that allows debugging the compiled WebAssembly as if one was stepping through the original C++ source code.

It turns out this is basically impossible without modifying LLVM, and making it work with PDB files might actually be impossible given the format's constraints. None of this is because of a lack of debugging information, at least in debug mode (LLVM 10 is necessary for using the new DWARF webassembly standard to debug optimized webassembly). All of the type information is included. All of the stack offsets are there. All the variable names and source files and line information is all there with all the information you need.

The problem is that no debugger understands a webassembly pointer. WebAssembly uses linear memory sections and encodes load/store operations as 32-bit zero-based offsets from the beginning of the memory section, which could be anywhere. This means that there are two values required to calculate the actual location in memory:

base_memory + offset

Now, unlike LLVM[1], DWARF is actually very well documented. The problem is that the documentation makes it clear that you get exactly one “custom” value to work with - whatever is pushed on top of the DWARF location expression. Any other values must exist in hardware registers (which are architecture and ABI-dependent), as built-in values, or as constants. LLVM makes it extremely hard to actually gaurantee than a specific hardware register get assigned anything, and it's even harder to make sure it doesn't get clobbered, so this is wildly impractical, even if we were only targeting x86-64. We still have built-in values like DW_AT_FRAME_BASE that might help, but we have a problem.

We can't just redirect DW_AT_frame_base somewhere else becuase it's currently pointing to the very real function framebase of our compiled function, which has it's own entirely valid stack that we still need to be able to find. We can't use DW_AT_static_link because it's coupled to the frame base, which itself still needs to exist. We would need some value at a specific, hardcoded address which could be accessed via DW_OP_addr in order to acquire the memory base pointer inside the expression, but that's totally incompatible with relocatable binaries. There is simply no way to have access to both an offset and a global base memory pointer in a DWARF expression without loading the global base memory pointer into a register.

Okay, fine, what if we nail our base memory location to a single hardcoded location? Since we're compiling the WebAssembly binary on the user's machine, we can dynamically pick a location in memory we know is free and hardcode it into the binary. Now we only need one custom value, the offset itself. The rest is just convincing the debugger to add X bytes to the memory location.

DWARF provides several ways of doing this. The cleanest would likely be to use DW_AT_data_location, which is intended for just this situation, but the problem is that it can't be applied to raw pointer types. We could try to use DW_TAG_ptr_to_member_type, but this is trying to perform the wrong operation. Okay, fine, how about we convert all our “pointers” to struct types that contain the type that is being pointed to? Then we can either use DW_AT_data_location on the struct type itself to derive the correct location, or we can use DW_AT_data_member_location on the contained type.

Unfortunately for us, LLVM can't emit DW_AT_data_location, and it hardcodes DW_AT_data_member_location as a simple byte offset from the start of a structure. In fact almost every useful data location parameter that we might be able to encode a location expression in isn't actually supported by LLVM, or is hardcoded to a specific operation. LLVM doesn't even consider DW_OP_addr to be a valid operator (probably because of binary relocation). Given these constraints, we'll have to get creative.

What if we tell the debugger our offset is a pointer, let it dereference the “pointer” to an obviously invalid memory location near the beginning of memory, but then offset the type contained inside the struct by the base memory location? The end result would be the contained “struct member” that just so happened to be offset by 2 billion bytes lands on the correct part in memory, and this would work with PDB files!

Unfortunately, while the bit offset of struct members is stored as a 64-bit value in both DWARF and PDB files, Visual Studio's debugger appears to use a signed 32-bit integer during the offset calculation. As a result, this technique only works if all memory addresses fit inside 31 bits.

Normally, when doing a hardcoded offset like this, one would reserve 4 gigabytes of memory (the maximum amount a 32-bit wasm binary can utilize) for every single module. This would all work out fine because most of them wouldn't actually use this much memory, but their layout in virtual address space ensures that they won't run into each other. Being limited to 31-bits means that you can only load a single module that has no maximum memory size, or you have to require maximum memory sizes for all loaded modules so they can be tightly packed.

The second problem is that, even though DWARF lets you specify the size of the pointer you are loading, Visual Studio doesn't care and will always assume a pointer is 64-bits on a 64-bit architecture. As a result, in order to make this technique work, you have to compile in the nonstandard wasm64 mode so that all offsets are stored in 64-bit integers. Technically, this does work, but the result is horrid:

WASM debugging prototype

Now, if we consider the constraints we already have, there are some much more farfetched ideas that come to mind, but unfortunately none of them are viable. One idea would be to allocate the lowest addressable page in Windows using VirtualAlloc, which is 0x10000, and then convince clang to compile an executable whose low address boundary is 65536 instead of 1024. Then the allocate memory command could allocate memory starting from 0x10000 but actually return 0 as the offset, which then turns all the webassembly offsets into true addressable pointers.

At minimum, this idea requires modifying clang, because there is no parameter to do this (or it's hidden). However, it also doesn't solve the function pointer problem. In WebAssembly, functions also are referenced by offset, but this offset is relative to a completely seperate function table. You can't have both of these at address 0, so one of them would need to be offset past the end of the other. At the moment, the function table can't grow, so sticking it in front of the actual linear memory region might work, but this immediately falls apart when reference types are added. You would also have to then convince clang to skip 16384 function indices when generating webassembly.

Another option would be to take advantage of how C++ programs are compiled into WebAssembly. Clang maintains a global variable that stores the current stack frame of the program. If you had a constant offset, you could add that to the initial value of the stack frame and have the custom malloc() implementation return offsets that include the base memory offset, then simply change the memory.load wasm operation to not add the base memory offset. This has the same effect of turning the webassembly offsets into true pointers (but still requires compiling as Wasm64). Unfortunately, the C++ model also stores globals that are referred to by hardcoded values in the linear memory space, which would be extremely difficult to differentiate from normal integer constants.

All of these ideas are extremely fragile and will immediately break upon trying to support multiple memory spaces and extensible function tables. The simple fact is, even if you were to modify LLVM to support all these DWARF operations, at minimum you would have to nail your memory locations to hardcoded addresses. In addition, any such solution would only work with GDB, forcing you to use Visual Studio's remote gdb debugger on Windows. The suggested WebAssembly DWARF extensions won't even be sufficient to deal with multiple linear memories or tables, because you need to encode which linear memory or table a given value is supposed to belong to.

I will be discussing these issues at the next WebAssembly meeting, but I'm posting this article now in the vain hope that I am actually completely wrong, and that by invoking Cunningham's Law, perhaps someone in a comment section somewhere will miraculously provide a solution that I have overlooked.

Until that happens, debugging a language through WebAssembly is impossible.


[1] In order to convince LLVM to generate PDBs in the first place, you must add a module flag with the magic string “CodeView”. This string does not exist as a constant anywhere outside of the LLVM function that checks for it's existence. There is no documentation anywhere that explains how to do this. I found it by digging through clang's source code. This is probably the only page on the entire internet that tells you how to do this.

For bonus points, LLVM also mentions that llvm.dbg.declare() is deprecated, but it won't use llvm.dbg.addr() unless you set a hidden command line parameter - the function directly checks this command line parameter, making it almost impossible to set programmatically.


Name Shadowing Should Be An Operator


I recently discovered that in Rust, this is a relatively common operation:

let foo = String::from("foo");
// stuff that needs ownership
let foo = &foo;  
Or this:
let mut vec = Vec::new();
vec.push("a");
vec.push("b");
let vec = vec; /* vec is immutable now */  
This is a particularly permissive form of name-shadowing, which allows you to re-declare a variable in an inner scope that shadows the name of a variable in an outer scope, making the outer variable inaccessible. Almost every single programming language in common use allows you to do this in some form or another. Rust goes a step further and lets you re-declare a variable inside the same scope as another variable.

This, to me, is pretty terrifying, because name-shadowing itself is often a dangerous operation. 90% of the time, name-shadowing causes problems with either temporary or index variables, such as i. These are all cases where almost you never want to name-shadow anything and doing so is probably a mistake.

for(int i = 0; i < width; ++i)
{
  for(int j = 0; j < height; ++j)
  {
    //
    // lots of unrelated code
    //
    
    float things[4] = {1,2,3,4};
    for(int i = 0; i < 4; ++i)
    	box[i][j] += things[i] // oops!
    //      ^ that is the wrong variable!
  }
}
These errors almost always crop up in complex for loop scenarios, which can obviously be avoided with iterators, but this isn't always an option, espiecally in a lower-level systems programming language like Rust. Even newer languages like Go make it alarmingly easy to name-shadow, although they make it a lot harder to accidentally shoot yourself because unused variables are a compiler error:
foo, err := func()
if err != nil {
	err := bar(foo) // error: err is not used
}
println(err)
Unfortunately, Rust doesn't have this error, only an unused-variables linter warning. If you want, you can add #![deny(clippy::shadow_unrelated)] to be warned about name-shadowing. However, a lot of Rust idioms depend on the ability to name-shadow, because Rust does a lot of type-reassignment, where the contents of a variable don't change, but in a particular scope, the known type of the variable has changed to something more specific.
let foo = Some(5);
match foo {
    Some(foo) => println!("{}", foo),
    None => {},
}
Normally, in C++, I avoid name-shadowing at all costs, but this is partially because C++ shadowing rules are unpredictable or counter-intuitive. Rust, however, seems to have legitimate use cases for name-shadowing, which would be reasonable if name-shadowing could be made more explicit.

I doubt Rust would want to change it's syntax, but if I were to work on a new language, I would define an explicit name-shadowing operator, either as a keyword or as an operator. This operator would redefine a variable as a new type and throw a compiler error if there is no existing variable to redefine. Attempting to redefine a variable without this operator would also throw a compiler error, so you'd have something like:

let foo = String::from("foo");
// stuff that needs ownership
foo := &foo;
Or alternatively:
let mut vec = Vec::new();
vec.push("a");
vec.push("b");
shadow vec = vec; /* vec is immutable now */
While the := operator is cleaner, the shadow keyword would be easier to embed in other constructions:
let foo = Some(5);
match foo {
    Some(shadow foo) => println!("{}", foo),
    None => {},
}
Which method would depend on the idiomatic constructions in the language syntax, but by making name-shadowing an explicit, rather than an implicit action, this allows you to get the benefits of name-shadowing while eliminating most of the dangerous situations it can create.

Unfortunately, most modern language design seems hostile to any feature that even slightly inconveniences a developer for the sake of code safety and reliability. Perhaps a new language in the future will take these lessons to heart, but in the meantime, people will continue complaining about unstable software, at least until we put the “engineer” back in “software engineering”.


Avatar

Archive

  1. 2020
  2. 2019
  3. 2018
  4. 2017
  5. 2016
  6. 2015
  7. 2014
  8. 2013
  9. 2012
  10. 2011
  11. 2010
  12. 2009