Low-level is easy

My previous entry has the amazingly interesting title “Blogging is hard”. Gee, what a wuss, says the occasional passer-by. Gotta fix my image, fast. Think, think, think! OK, here’s what I’m going to tell you: low-level programming is easy. Compared to higher-level programming, that is. I’m serious.

For starters, here’s a story about an amazing developer, who shall rename nameless for the moment. See, I know this amazing developer. Works for Google now. Has about 5 years of client-side web programming under his belt. And the word “nevermore” tattooed all over his body (in the metaphorical sense; I think; I haven’t really checked). One time, I decided that I have to understand this nevermore business. “Amazing Developer”, I said, “why have you quit the exciting world of web front-ends?” “I don’t like it”, says the Amazing Developer. “But, but! What about The Second Dot Com Bubble? VC funds will beg you to take their $1M to develop the next Arsebook or what-not. Don’t you wanna be rich?” “I really don’t like web front-ends very much”, he kept insisting. Isn’t that strange? How do you explain it? I just kept asking.

Now that I think of it, he probably was a little bit irritated at this point. “Look, pal”, he said (in the metaphorical sense; he didn’t actually say it like that; he’s very polite). “I have a license to drive a 5-ton truck. But I don’t want a career in truck driving. Hope this clarifies things”. He also said something about your average truck driver being more something than your typical web developer, but I don’t remember the exact insult.

Now, I’ve been working with bare metal hardware for the last 4 years. No OS, no drivers, nothing. In harsh environments in terms of hardware, by comparison. For example, in multi-core desktops, when you modify a cached variable, the cache of the other processor sees the update. In our chips? Forget it. Automatic hardware-level maintenance of memory coherency is pretty fresh news in these markets. And it sucks when you change a variable and then the other processor decides to write back its outdated cache line, overwriting your update. It totally sucks.

A related data point: I’m a whiner, by nature. Whine whine whine. You’ve probably found this blog through the C++ FQA, so you already know all about my whining. And it’s not like I haven’t been burnt by low-level bugs. Oh, I had that. Right before deadlines. Weekends of fun. So how come I don’t run away from this stuff screaming and shouting? Heck, I don’t mind dealing with bare metal machines for the rest of my career. Well, trying out other stuff instead can be somewhat more interesting, but bare metal beats truck driving, I can tell you that. To be fair, I can’t really be sure about that last point - I can’t drive. At all. Maybe that explains everything?

I’ll tell you how I really explain all of this. No, not right in this next sentence; there’s a truckload of reasons (about 5 tons), so it might take some paragraphs. Fasten your seatbelts.

What does “high level” basically mean? The higher your level, the more levels you have below you. This isn’t supposed to matter in the slightest: at your level, you are given a set of ways to manipulate the existing lower-level environment, and you build your stuff on top of that. Who cares about the number of levels below? The important thing is, how easily can I build my new stuff? If I mess with volatile pointers and hardware registers and overwrite half the universe upon the slightest error, it sucks. If I can pop up a window using a single function, that’s the way I like it. Right? Well, it is roughly so, but there are problems.

Problem 1: the stuff below you is huge at higher levels. In my humble opinion, HTML, CSS, JavaScript, XML and DOM are a lot of things. Lots of spec pages. A CPU is somewhat smaller. You have the assembly commands needed to run C (add/multiply, load/store, branch). You have the assembly commands needed to run some sort of OS (move to/from system co-processor register; I dunno, tens of flavors per mid-range RISC core these days?). And you have the interrupt handling rules (put the reset handler here, put the data abort handler here, the address which caused the exception can be obtained thusly). That’s all.

I keep what feels like most of ARM946E-S in my brain; stuff that’s still outside of my brain probably never got in my way. It’s not a particularly impressive result; for example, the fellow sitting next to me can beat me (”physically and intellectually”, as the quote by Muhammad Ali goes; in his case, the latter was a serious exaggeration - he was found too dumb for the US army, but I digress). Anyway, this guy next to me has a MIPS 34Kf under his skull, and he looks like he’s having fun. That’s somewhat more complicated than ARM9. And I worked on quite some MFC-based GUIs (ewww) back in the Rich Client days; at no point I felt like keeping most of MFC or COM or Win32 in my head. I doubt it would fit.

Problem 2: the stuff below you is broken. I’ve seen hardware bugs; 1 per 100 low-level software bugs and per 10000 high-level software bugs. I think. I feel. I didn’t count. But you probably trust me on this one, anyway. How many problems did you have with hardware compared to OS compared to end-user apps? According to most evidence I got, JavaScript does whatever the hell it wants at each browser. Hardware is not like that. CPUs from the same breed will run the user-level instructions identically or get off the market. Memory-mapped devices following a hardware protocol for talking to the bus will actually follow it, damn it, or get off the market.

Low-level things are likely to work correctly since there’s tremendous pressure for them to do so. Because otherwise, all the higher-level stuff will collapse, and everybody will go “AAAAAAAAAA!!” Higher-level things carry less weight. OK, so five web apps are broken by this browser update (or an update to a system library used by the browser or any other part of the pyramid). If your web app broke, your best bet is to fix it, not to wait until the problem is resolved at the level below. The higher your level, the loner you become. Not only do you depend on more stuff that can break, there are less people who care in each particular case.

Problem 3: you can rarely fix the broken things below your level. Frequently you don’t have the source. Oh, the browser is open source? Happy, happy, joy, joy! You can actually dive into the huge code base (working below your normal level of abstraction where you at least know the vocabulary), fix the bug and… And hope everyone upgrades soon enough. Is this always a smart bet? You can have open source hardware, you know. Hardware is written in symbolic languages, with structured flow and everything. The only trouble is, people at home can’t recompile their CPUs. Life cycle. It’s all about life cycle. Your higher-level thingie wants to be out there in the wild now, and all those other bits and pieces live according to their own schedules. You end up working around the bug at your end. Sometimes preventing the lower-level component author from fixing the bug, since that would break yours and everybody else’s workaround. Whoopsie.

How complicated is your workaround going to be? The complexity raises together with your level of abstraction, too. That’s because higher-level components process more complicated inputs. Good workarounds are essentially ways to avoid inputs which break the buggy implementation. Bad workarounds are ways to feed inputs which shouldn’t result in the right behavior, but do lead to it with the buggy implementation. Good workarounds are better than bad workarounds because bad workarounds break when the implementation is fixed. But either way, you have to constrain or transform the inputs. Primitive sets of inputs are easier to constrain or transform than complicated sets of inputs. Therefore, low-level bugs are easier to work around. QED.

Low-level: “Don’t write that register twice in a row; issue a read between the writes”. *Grump* stupid hardware. OK, done. Next.

High-level: “Don’t do something, um, shit, I don’t know what exactly, well, something to interactive OLE rectangle trackers; they will repaint funny”. I once worked on an app for editing forms, much like the Visual Studio 6 resource editor. In my final version, the RectTracker would repaint funny, exactly the way it would in Visual Studio 6 in similar cases. I think I understood the exact circumstances back then, but haven’t figured out a reasonable way to avoid them. Apparently the people working at that abstraction level at Microsoft couldn’t figure it out, either. What’s that? Microsoft software is always crap? You’re a moron who thinks everything is absolutely trivial to get right because you’ve never done anything worthwhile in your entire life. Next.

Problem 4: at higher levels, you can’t even understand what’s going on. With bare metal machines, you just stop the processor, physically (nothing runs), and then read every bit of memory you want. All the data is managed by a single program, so you can display every variable and see the source code of every function. The ultimate example of the fun that is higher-level debugging is a big, slow, hairy shell script. “rm: No match.” Who the hell said that, and how am I supposed to find out? It could be ten sub-shells below. Does it even matter? So what if I couldn’t remove some files? Wait, but why were they missing - someone thought they should be there? Probably based on the assumption that a program should have generated them previously, so that program is broken. Which program? AAARGH!!

OK, so shell scripts aren’t the best example of high-level languages. Or maybe you think they are; have fun. I don’t care. I had my share of language wars. This isn’t about languages. I want to move on to the next example. No shell scripts. You have JavaScript (language 1) running inside HTML/CSS (languages 2 & 3) under Firefox (written in language 4) under Windows (no source code), talking to a server written in PHP (language 5, one good one) under Linux (yes source code, but no way to do symbolic debugging of the kernel nonetheless). I think it somewhat complicates the debugging process; surely no single debugger will ever be able to make sense of that.

Problem 5: as you climb higher, the amount of options grows exponentially. A tree has one root, a few thick branches above it, and thousands of leaves at the top. Bend the root and the tree falls. But leaves, those can grow in whichever direction they like.

Linkers are so low-level that they’re practically portable, and they’re all alike. What can you come up with when you swim that low? Your output is a memory map. A bunch of segments. Base, size, bits, base, size, bits. Kinda limits your creativity. GUI toolkits? The next one is of course easier to master than the first one, but they are really different. What format do you use to specify layouts, which part is data-driven and which is spelled as code? How do you handle the case where the GUI is running on a remote machine? Which UI components are built-in? Do you have a table control with cell joining and stuff or just a list control? Does your edit box check spelling? How? I want to use my own dictionary! Which parts of the behavior of existing controls can be overridden and how? Do you use native widgets on each host, surprising users who switch platforms, or roll your own widgets, surprising the users who don’t?

HTML and Qt are both essentially UI platforms. Counts as “different enough” for me. Inevitably, both suck in different ways which you find out after choosing the wrong one (well, it may be obvious with those two from the very beginning; Qt and gtk are probably a better example). Porting across them? Ha.

The fundamental issue is, for many lower-level problems there’s The Right Answer (IEEE floating point). Occasionally The Wrong Answer gains some market share and you have to live with that (big endian; lost some market share recently). With higher-level things, it’s virtually impossible to define which answer is right. This interacts badly with the ease of hacking up your own incompatible higher-level nightmare. Which brings us to…

Problem 6: everybody thinks high-level is easy, on the grounds that it’s visibly faster. You sure can develop more high-level functionality in a given time slot compared to the lower-level kind. So what? You can drive faster than you can walk. But driving isn’t easier; everybody can walk, but to drive, you need a license. Perhaps that was the thing mentioned by the Awesome (Ex-Web) Developer: at least truck drivers have licenses. But I’m not sure that’s what he said. I’ll tell you what I do know for sure: every second WordPress theme I tried was broken out of the box, in one of three ways: (1) PHP error, (2) SQL error and (3) a link pointing to a missing page. WordPress themes are written in CSS and PHP. Every moron can pick up CSS and PHP; apparently, every moron did pick them up. Couldn’t they keep the secret at least from some of them? Whaaaam! The speedy 5-ton truck goes right into the tree. Pretty high-level leaves fall off, covering the driver’s bleeding corpse under the tender rays of sunset. And don’t get me started about the WordPress entry editing window.

Now, while every moron tries his luck with higher-level coding, it’s not like everyone doing high-level coding is… you get the idea. The other claim is not true. In fact, this entry is all about how the other claim isn’t true. There are lots of brilliant people working on high-level stuff. The problem is, they are not alone. The higher your abstraction level, the lower the quality of the average code snippet you bump into. Because it’s easy to hack up by the copy-and-paste method, it sorta works, and if it doesn’t, it seems to do, on average, and if it broke your stuff, it quite likely your problem, remember?

Problem 7: it’s not just the developers who think it’s oh-so-easy. Each and every end user thinks he knows exactly what features you need. Each and every manager thinks so, too. Sometimes they disagree, and no, the manager doesn’t always think that “the customer is always right”. But that’s another matter. The point here is that when you do something “easy”, too many people will tell you how it sucks, and you have to just live with that (of course having 100 million users can comfort you, but that is still another matter, and there are times when you can’t count on that).

I maintain low-level code, and people are sort of happy with it. Sometimes I think it has sucky bits, which get in your way. In these cases, I actually have to convince people that these things should be changed, because everybody is afraid to break something at that level. Hell, even bug fixes are treated like something nice you’ve done, as if you weren’t supposed to fix your goddamn bugs. Low-level is intimidating. BITS! REGISTERS! HEXADECIMAL! HELP!!

Some people abuse their craft and actively intimidate others. I’m not saying you should do that; in fact, this entry is all about how you shouldn’t do that. The people who do it are bastards. I’ve known such a developer; I call him The Bastard. I might publish the adventures of The Bastard some day, but I need to carefully consider this. I’m pretty sure that the Awesome Developer won’t mind if someone recognizes him in a publicly available page, but I’m not so sure about The Bastard for some reason or other.

What I’m saying is, maintaining high-level code is extremely hard. Making a change is easy; making it right without breaking anything isn’t. You can drive into a tree in no time. High-level code has a zillion of requirements, and as time goes by, the chance that many of them are implicit and undocumented and nobody even remembers them grows. People don’t get it. It’s a big social problem. As a low-level programmer, you have to convince people not to be afraid when you give them something. As a high-level programmer, you have to convince them that you can’t just give them more and more and MORE. Guess which is easier. It’s like paying, which is always less hassle than getting paid. Even if you deal with a Large, Respectful Organization. Swallowing is easier than spitting, even for respectful organizations. Oops, there’s an unintended connotation in there. Fuck that. I’m not editing this out. I want to be through with this. Let’s push forward.

The most hilarious myth is that “software is easy to fix”; of course it refers to application software, not “system” software. Ever got an e-mail with a “>From” at the beginning of a line? I keep getting those once in a while. Originally, the line said “From” and then it got quoted by sendmail or a descendant. The bug has been around for decades. The original hardware running sendmail is dead. And that hardware had no bugs. The current hardware running sendmail has no bugs, either. Those bugs were fixed somewhere during the testing phase. Application software is never tested like hardware. I know, because I’ve spent about 9 months, the better part of 2007, writing hardware tests. Almost no features; testing, exclusively. And I was just one of the few people doing testing. You see, you can’t fix a hardware bug; it will cost you $1M, at least. The result is that you test the hardware model before manufacturing, and you do fix the bug. But with software, you can always change it later, so you don’t do testing. In hardware terms, the testing most good software undergoes would be called “no testing”. And then there’s already a large installed base, plus quick-and-dirty mailbox-parsing scripts people wrote, and all those mailboxes lying around, and no way to make any sense of them without maintaining bugward compatibility (the term belongs to a colleague of mine, who - guess what - maintains a high-level code base). So you never fix the bug. And most higher-level code is portable; its bugs can live forever.

And the deadlines. The amount of versions of software you can release. 1.0, 1.1, 1.1.7, 1.1.7.3… The higher your abstraction level, the more changes they want, the more intermediate versions and branches you’ll wind up with. And then you have to support all of them. Maybe they have to be able to read each other’s data files. Maybe they need to load each other’s modules. And they are high-level. Lots of stuff below each of them, lots of functionality in them. Lots of modules and data files. Lots of developers, some of whom mindlessly added features which grew important and must be supported. Damn…

I bet that you’re convinced that “lower-level” correlates with “easier” by now. Unless you got tired and moved elsewhere, in which case I’m not even talking to you. QED.

Stay tuned for The Adventures of The Bastard.