Paul Graham was once asked the following RAQ (rarely asked question):
How can I avoid turning into a pointy-haired boss?
His answer:
The pointy-haired boss is a manager who doesn't program. So the surest way to avoid becoming him is to stay a programmer. What tempts programmers to become managers are companies where the only way to advance is to go into management. So avoid such companies and work for (or start) startups.
Why be a manager when you could be a founder or early employee at a startup?
Why?! Oh wow. I could fill a book explaining why. But many of my reasons are my own, and aren't relevant to you unless you're much like me. So I'll focus on the general answer to the question implied by Paul Graham: Why do large firms exist?
The question was addressed by the economist Ronald Coase in his article "The Nature of the Firm". This article, together with his work on externalities (the Coase Theorem), earned him a Nobel Prize in economics. This is one evidence that the question is interesting and far from trivial.
Suppose there are no good answers to Paul Graham's rhetorical question. That is, it's always objectively better to start or join a small firm than to be a manager in a large one. You'll always get more work done, or will be more satisfied, or both. Well, if so, competition should eventually drive large firms out of business. So why are they still around?
For starters, clearly there are problems best solved by small groups of people armed with off-the-shelf tools. For instance, two iconic YC startups funded by Paul Graham, Reddit and Dropbox, each solve a problem with the help of a few programmers and a bunch of commodity servers running a commodity software stack. A larger company could hardly improve on what they do.
Note that off-the-shelf products are key to being small (or at least starting small). Reddit or Dropbox could never build those servers from scratch. A small group of people can not erect a $5G chip fabrication facility. Building and operating a fab - or a search engine - requires lots of custom development, so you need a lot of people.
Or do you?
Of course the total number of people involved has to be very large. But it doesn't follow that they should be organized as big companies. Instead, the work could be done by many small organizations, each contracting out most of the work to others.
You're big because you hire. Why hire if you can buy, contract out - and stay small?
Indeed, this seems to make perfect sense. To quote Wikipedia's summary of The Nature of the Firm (1937):
The traditional economic theory of the time suggested that, because the market is "efficient" (that is, those who are best at providing each good or service most cheaply are already doing so), it should always be cheaper to contract out than to hire.
Then why do most people prefer employment to self-employment, as evidenced by their actions (and an economist never trusts anything but actions as a tool to reveal someone's preferences)? Why do I hate the idea of running a small firm?
Either the "traditional economic theory" is right - one should run a small firm, and I'm a freak of nature destined to extinction due to economic evolutionary pressure, together with much of the population - or the theory is lacking, and there should be a concept formalizing my aversion to self-employment.
And in fact, at this point, Coase introduces the term - transaction costs:
Coase noted, however, that there are a number of transaction costs to using the market; the cost of obtaining a good or service via the market is actually more than just the price of the good.
Oh, yeah - MUCH more if you ask me.
Other costs, including search and information costs, bargaining costs, keeping trade secrets, and policing and enforcement costs, can all potentially add to the cost of procuring something via the market.
YES! Here's a Nobel Prize-winning economist from the notoriously "pro-free market" Chicago school that UNDERSTANDS ME. He knows why I hate markets. ("Pro-market" doesn't mean you love markets, just that you think governments are even worse.)
This suggests that firms will arise when they can arrange to produce what they need internally and somehow avoid these costs.
Avoiding these costs can enable work that just can't happen outside the context of a big company.
For instance, I work on chips for embedded computer vision, at a company that's now fairly large. This is an example where a lot of people need to cooperate in a custom development effort (as opposed to fewer people using off-the-shelf products).
In theory, I could start a computer vision hardware startup instead of it being an internal project. In practice, it wouldn't work, because:
I wouldn't know what to build. Hardware accelerates algorithms - what algorithms? I only know because I'm in the same company with developers of very effective unpublished algorithms. Without that knowledge, what could I build - an OpenCV accelerator? Good luck selling that.
I couldn't build it nearly as efficiently. A great source of efficiency is fitting hardware to the specific workload. But if we were not a part of the company but a vendor, the company would make sure there are competing vendors to keep prices low. This means that we, no longer having a guaranteed customer, would have to support as many different workloads as possible, to increase the pool of potential customers. As a rule, more generic hardware is less efficient.
I couldn't explain how to program it. Once you gave away your programming model to the customer - as you have to if you want them to, well, program you processors - only very strong patents can prevent them from cloning your hardware (possibly with the help of your competitor). A big company that, among other things, designs its own hardware doesn't have to explain it to the outside world. And even if its hardware ends up cloned - it's just one part of the secret knowledge behind the product. But if you're a small company only making hardware and it's cloned, you're busted. You shouldn't even start before making sure your ideas are "sufficiently patentable" - which you don't know before you developed those ideas.
Of course, the number one real reason I couldn't run a hardware startup is that I'm no businessman. But the problems above are also very real, and frequently insurmountable for people who can do business. Not all custom development is impossible to successfully outsource, but much is. The problems result from economic fundamentals.
In econ-speak, such problems are collectively known as "search and information costs, bargaining costs, keeping trade secrets, and policing and enforcement costs". Indeed, all these problems were featured in my example. In plain English, a simple way to sum up all those problems is trust - or more precisely, the lack thereof:
A company can't trust a vendor, so a vendor can't know its algorithms.
A company can't trust a vendor to keep qaulity high and prices low if it guarantees to remain its customer…
…So a vendor can't trust a company to remain its customer, so it can't invest too much in a solution just to that company's specific needs.
A vendor can't trust a company to keep buying from it if enough knowledge is given away so that the product can be cloned instead - so some products are not worth building.
When you work for a big company, you deal with coworkers, and you're all playing for the same team. The smaller the company, the more you deal with customers and vendors, which means playing against them. There's no such word as "co-customer" or "co-vendor" for a good reason.
At least that's how things are framed by the rules. The rules say that all employees are agents acting towards a common goal, "to promote the company's interests" - whereas different companies have different bottom lines and different interests.
Of course, reality is never like the rules - in reality, everyone in the company plays by their own rules, attempting to promote the interest of any of the following - or a combination:
Shareholders
Customers
Employees
His team
His manager
His friends
Himself
So in reality, of course there's a lot of chaos in a big company. And it doesn't help that the bigger it is, the harder it is to make sense of what's going on:
…There is a natural limit to what can be produced internally, however. Coase notices "decreasing returns to the entrepreneur function", including increasing overhead costs and increasing propensity for an overwhelmed manager to make mistakes in resource allocation. This is a countervailing cost to the use of the firm.
…Which explains why we aren't all employed by a single all-encompassing huge company.
But at least the rules of a large company frame things right - as cooperation more than competition. (Competition generally isn't an end - it's a means to ultimately force people to cooperate, and, as Coase points out, it only gets you this far.)
Of course, corporate rules also create competition - employees compete for raises, etc. But in practice, overall most would agree that it's much safer to trust co-workers than customers or vendors.
Why be a manager when you could be a founder or early employee at a startup? Here's the part of my answer that is based on economic fundamentals.
I specialize in areas requiring custom development by many people. Many people can only tightly cooperate under rules implying trust. Therefore they must not be customers and vendors, but coworkers, which leads to large firms. Such is The Nature of the Firm.
Of course there are problems that can be solved by a small group of people with mutual trust, without tightly-coupled, joint development with others - for example, the problems solved by Reddit and Dropbox. One reason I personally never looked that way is my aversion to business. Such is my own nature.
It just so happens that the nature of the firm suits my nature nicely - because there are situations where big companies are a good idea. When you can't buy and have to build, trust is fundamental to getting the job done.
UPDATE (December 9, 2011): just found an interesting analogy between company size and program size. Doing many things in one big program can be easier than using many small programs because of "transaction costs" - the cost of exchanging data between the programs.
When chased by a bear, engineers want to run faster than the bear, managers want to run faster than you. This is known as "the best vs the good enough", and is a very common theme.
For instance, company A releases a good enough technology, company B releases the best technology on the market. B fails and A succeeds, because A releases earlier, or because A's technology is more compatible with the status quo, etc. Engineers will commonly feel sympathy for B, managers will applaud the shrewdness of A.
It's a common story and an interesting angle, but the "best vs good enough" formulation misses something. It sounds as if there's a road towards "the best" - towards the 100%. Engineers want to keep going until they actually reach 100%. And managers force them to quit at 70%:
There comes a time in the life of every project where the right thing to do is shoot the engineers and ship the fucker.
However, frequently the road towards "the best" looks completely different from the road to "the good enough" from the very beginning. The different goals of engineers and managers make their thinking work in different directions. A simple example will illustrate this difference.
Suppose there's a bunch of computers where people can run stuff. Some system is needed to decide who runs what, when and where. What to do?
An engineer will want to keep as many computers occupied at every moment as possible - otherwise they're wasted.
A manager will want to give each team as few computers as absolutely necessary - otherwise they're wasted.
These directions aren't just opposite - "as many as possible" vs "as few as necessary". They focus on different things. The engineer imagines idle machines longing for work, and he wants to feed them with tasks. The manager thinks of irate users longing for machines, and he wants to feed them with enough machines to be quiet. Their definitions of "success" are barely related, as are their definitions of "waste".
An engineer's solution is to have everybody submit their jobs to a queue managed by a central server. The Condor software is an example implementation; there are many others, with many subtle issues and differences - or you can roll your own. The upshot is that once a machine becomes idle, it can immediately yank a next job from the server's queue. As long as there's anything left to do, no machine is idle. This is the best possible situation.
A manager's solution is to give the ASIC team 12 servers: asic01, asic02, … asic12. QA gets 20 machines, qa01 … qa20, and so on. If you want to work on a machine, you ssh to it and you work. If you're an ASIC engineer and you want to use a QA machine, you can't log in. If users fight over their team's machines, they can go to their manager. If their manager decides the team needs more machines, he goes to upper management. Good enough.
The "good enough" is not 70% of "the best" - it's not even in the same direction. In fact, it's more like -20%: once the "good enough" solution is deployed, the road towards "the best" gets harder. You restrict access to machines, and you get people used to the ssh session interface, which "the best" solution will not provide.
Which solution is actually better? Tough question.
The manager's solution requires no programming or installation, and trivial administration.
The manager's solution requires no changes of habits (ssh is the standard).
The engineer's solution yields ~100% utilization, the manager's 50%, 30%, or 10%, depending.
The manager's solution will cause less headache to the managers, on average. Once in a while, you buy a team some more machines, and they shut up. The engineer's solution requires to set priorities at times of overload, and people will constantly argue about priority with managers.
The engineer's solution doesn't run things on machines that are already busy anyway. Users armed with ssh will tend to do this, possibly with horrible slowdowns due to swapping, etc.
The engineer's solution can provide the lowest latencies.
The engineer's solution makes it trivial to utilize new machines - no need to set up sessions, just submit jobs as previously. This is good - and bad: people will demand new machines instead of reducing the load.
So there are many conflicting considerations. Their importance varies between cases, and is very hard to measure.
"The best" solution is not necessarily the best - rather, it's what the mindset of looking for the best yields. Likewise, the "good enough" solution is not necessarily good enough - it's just what you come up with when you look for a good enough approach.
Obviously, portraying "engineers" and "managers" this way is a gross oversimplification, and most real people can look at things from both angles. I like this oversimplification for two reasons:
It does help to understand and predict many common arguments and reactions.
A person's perspective frequently does coincide with the title: engineers aim at the best, managers look for the good enough. Just the title is thus a good basis for prediction.
How does title affect judgement? There are arguments to the effect of "aiming at the good enough is wiser, and managers are in a better position to achieve wisdom", or vice versa. However, I don't believe either viewpoint is more correct than the other. Rather, they are biases. People in different positions acquire different biases, because they have different incentives and constraints.
The difference in incentives is that engineers are rewarded for success, while managers are rewarded for non-failure.
An exceptional engineer is unique and valuable, like a great chef - being an OK engineer is more like cooking for McDonald's. An engineer needs unusual success to be noticed. One reason such success is achievable is because he works within his area of expertise, on things that he understands. There's rarely a formal quality metric, but his deep understanding creates in him a sense of beauty that serves as his metric.
A manager is fine as long as the project doesn't fail. Most projects fail - if yours didn't, it's quite an achievement. One reason they fail is that there are a lot of ways to fail. Forget just one item in a lengthy checklist, and it won't matter how well the other 99 items are handled. People may not buy a car because there's no convenient place for a resting arm. A manager looks after a whole lot of items, usually without being able to understand most of them in any depth. He's inclined to look for simple pass/fail criteria.
If your success function is a continuous metric, you'll aim at the best. If it's a long list of Boolean values - V or X - you'll want "good enough" (V) at every entry. "The best" is just a costlier way to get a V, at a higher risk to end up with an X. The manager's outlook leads to binarization.
Another difference is that engineers and managers control different things - "One man's constant is another man's variable". In our example, the engineer doesn't decide how many machines to purchase or how to allocate them. If the number of machines is constant, what remains is to make the most of them. The manager, on the other hand, doesn't directly control utilization, and frequently doesn't know how to improve it. If utilization is constant, what remains is to properly ration it.
More generally, engineers tend optimize within constraints set by management, in part because they can't change those constraints. As to managers, they tend to settle for "good enough", in part because they can't optimize.
Which biases lead to more sensible solutions depends on the situation. However, there's a subtle reason to like the engineer's devotion to excellence, despite the manager's pressure for mediocrity. It's precisely when the manager is right in that excellence will not contribute to sales when the engineer contributes the mostto users.
If an engineer's job is done so poorly as to make the product non-marketable, the product will fail and won't be used. When the quality along all dimensions passes the adoption threshold, every improvement along any dimension is effectively a gift to the users - who'd be using the product anyway. This is false where improvements contribute significantly to popularity, but true where they don't.
The improvements are inconsequential for business, but consequential economically. Economically, anything that helps people is good. For business, anything that creates profit is good. Often these definitions of "good" coincide, but sometimes they don't. In these cases, people who aim at excellence for excellence's sake help bridge the gap.
The manager's checklist approach - the business angle - ensures that the product is marketable, which is great because otherwise it won't get used. But the engineer's urge to optimize is closer to the really important goal: making the best of what we've got; this is the economics angle. So while it can lead to problems just like any other bias, it's likeable that way.
Schrödinger used to have this quantum cat which was alive and dead at the same time as long as nobody opened the box, and it was the very act of looking at the cat that made it either alive or dead. Now, I'm not sure about this quantum stuff, but if you ask me you'd always find a dead cat upon opening the box, killed by the act of notlooking. In fact, if you open any random box nobody was looking into, chances are you'll find a dead cat there. Let me give an example.
I recently chatted with a former employee of a late company I'll call Excellence (the real name was even worse). Excellence was a company with offices right across the street that kept shrinking until the recent financial crisis. It then had to simultaneously fire almost all of its remaining employees, carefully selected as their best during the previous years when other employees were continuously fired at a lower rate. Giving us a whole lot of great hires, including MV, the co-worker in this story (though he was among those who guessed where things were going and crossed the street a bit earlier).
Rumors tell that to the extent possible, Excellence used to live up to expectations created by its name. In particular, without being encouraged or forced by customers to comply to any Software Development Methodology such as the mighty and dreadful CMM, they had (as CMM puts it) not only Established, but rigorously Maintained an elaborate design, documentation and review process which preceded every coding effort. Other than praise, MV had little to say about the process, except perhaps that occasionally someone would invent something awfully complicated that made code incomprehensible, having gone just fine through the review process because of sounding too smart for anyone to object.
Now, in our latest conversation about how things were at Excellence, MV told how he once had to debug a problem in a core module of theirs, to which no changes had been made in years. There, he stumbled upon a loop searching for a value. He noticed that when the value was found, the loop wouldn't terminate - a continue instead of a break kind of thing. Since the right value tended to be found pretty early through the loop, and because it was at such a strategic place, test cases everyone was running took minutes instead of seconds to finish. Here's a dead cat in a gold-plated box for ya, and one buried quite deeply.
My own professional evolution shaped my mind in such a way that it didn't surprise me in the slightest that this slipped past the reviewer(s). What surprised me was how this slipped past the jittery bodybuilder. You see, we have this Integration Manager, one of his hobbies being bodybuilding (a detail unrelated, though not entirely, to his success in his work), and one thing he does after integrating changes is he looks at the frame rate. When the frame rate seems low, he pops up a window with the execution profile, where the program is split into about 1000 parts. If your part looks heavier than usual, or if it's something new that looks heavy compared to the rest, it'll set him off.
So I asked MV how come that the cat, long before it got dead and buried, didn't set off the jittery bodybuilder. He said they didn't have one for it to set off. They were translating between the formats of different programs. Not that performance didn't matter - they worked on quite large data sets. But to the end user, automatic translation taking hours had about the same value as automatic translation taking seconds - the alternative was manual translation taking weeks or months. So they took large-scale performance implications of their decisions into account during design reviews. Then once the code was done and tested, it was done right, so if it took N cycles to run, it was because it took N cycles to do whatever it did right.
And really, what is the chance that the code does everything right according to a detailed spec it is tested against, but there's a silly bug causing it to do it awfully slowly? If you ask me - the chance is very high, and more generally:
Though not looking at performance followed from a reasonable assessment of the situation,
Performance was bad, and bad enough to become an issue (though an unacknowledged one), when it wasn't looked at,
Although the system in general was certainly "looked at", apparently from more eyes and angles than "an average system", but it didn't help,
So either you have a jittery bodybuilder specifically and continuously eyeballing something, or that something sucks.
Of course you can save effort using jittery automated test programs. For example, we've been running a regression testing system for about a year. I recently decided to look at what's running through it, beyond the stuff it reports as errors that ought to be fixed (in this system we try to avoid false positives to the point of tolerating some of the false negatives, so it doesn't complain loudly about every possible problem). I found that:
It frequently ran out of disk space. It was OK for it to run out of disk space at times, but it did it way too often now. That's because its way of finding out the free space on the various partitions was obsoleted by the move of the relevant directories to network-attached storage.
At some of the machines, it failed to get licenses to one of the compilers it needed - perhaps because the env vars were set to good values with most users but not all, perhaps because of a compiler upgrade it didn't take into account. [It was OK for it to occasionally fail to get a license (those are scarce) - then it should have retried, and at the worst case report a license error. However, the compiler error messages it got were new to it, so it thought something just didn't compile. It then ignored the problem on otherwise good grounds.]
Its way of figuring out file names from module names failed for one module which was partially renamed recently (don't ask). Through an elaborate path this resulted in tolerating false negatives it really shouldn't.
…
And I'm not through with this thing yet, which to me exemplifies the sad truth that while you can have a cat looking at other cats to prevent them from dying, a human still has to look at that supervisor cat, or else it dies, leading to the eventual demise of the others.
If you don't look at a program, it rots. If you open a box, there's a dead cat in it. And if a tree falls in a forest and no one is around to hear it, it sucks.
As a part of my continuous moral degradation and the resulting increasing alignment with the forces of Evil, I'm sharing an apartment with a gal who used to work in HR assessment. She recently got me acquainted to a friend of hers, BC, who works as a business consultant (names have been changed to protect the guilty).
BC's primary educational background is in applied mathematics. Having put the math they teach in CS departments to relatively few uses as a working programmer, I asked her about the uses of applied mathematics in business consulting. BC cited the following two examples.
The first example involves compensation and its dependence on key performance indicators, affectionately known as KPI and estimated by HR assessors. One way of looking at this dependence is to consider how it affects compensation over time as an employee's competence increases.
A psychological discussion is then possible on the relative merits of the different graphs plotting the compensation functions f(KPI). If f is linear (has a constant derivative), we make people struggle equally hard at every step. If f's derivative increases over time (for instance, when f is exponential), we make elevation hard at first and then increasingly easy. If f's derivative decreases over time (for example, if f is logarithmic), we make the last mile the hardest. And so on.
Through a psychological discussion of this sort, someone in the consulting company decided that what was really needed in some case or other was an S-shaped curve. The problem was that you couldn't just draw an S-shaped curve - the plotting had to be done in Excel according to a formula; an S-shaped curve which just blithely goes through arbitrary points doesn't cut it when you deliver a Compensation Model. But how do you make a formula to go up slowly, than fast, than slowly again? Exponents don't work. Logarithms don't work. A sine does look like an S, but it's a wrong kind of S, somehow. What to do?
Enter BC with 6 years of math studies under her belt. A compact yet impressive formula is spelled out, and - presto! - Excel renders an S-shaped curve. (I guess she used the sigmoid function but I didn't check.) The formula brought delight to management and fame to BC, and compensation payments issued according to its verdict keep adding up to scary numbers (BC's agency works with some really big companies).
The second example involves the compensation of managers. Naturally, a good manager near the bottom is worth less to the firm than a bad manager near the top, and therefore the compensation function should now depend on the manager's level in the hierarchy as well as his KPI (or better). Equally naturally, the numbers coming out of the compensation spreadsheet will under no circumstances arise through an externally conducted study of their psychological implications or any similarly unpredictable device. The numbers will result from nothing but the deep understanding of the organization possessed by the top management.
The development process of the managerial compensation function is thus complementary to that of the employee compensation function. Instead of producing numbers from a beautiful spreadsheet, what is needed here is to produce a beautiful spreadsheet from the numbers specified by the top management. The spreadsheet then promptly generates back these exact numbers from the input parameters.
The purpose of the spreadsheet is to relieve the top managers from the need to justify the numbers to their underlings. In order to guarantee that they are relieved from this need, the formula should not contain terms such as 1/(level^2), which could raise questions such as why not use 1/level, why not use 1/log(level) and other questions along these lines. Rather, the formula should contain terms which could raise no questions at all simply due to their size and shape.
BC faced this problem at an early stage of her career, and despite the natural stress, came up with an interesting Compensation Model, its key term being e raised to the power of something unspeakably grand, combining the trademark Gaussian look and feel with an obvious ability to deter the skeptics from asking questions. The only problem with that term was the very source of its utility, namely, the fact that it evaluated to 0 for all values of KPI and hierarchy level.
The deadline being close, BC told the manager of the consulting project in question about the status of her work and expressed her doubts regarding the delivery of the Computational Model. The manager told her that she just doesn't get it, does she, it's great, the right numbers come out and that's all there is to it and we should send it right away. And so they did, to everyone's complete satisfaction.
Her command of applied mathematics aside, BC is generally quite powerful.
For instance, she once got invited to consult some government agency about a project of theirs, while being on vacation and without it being explained to her that she was about to attend a formal meeting with the whole team. In order to maintain the reputation of the guy who somewhat clumsily brought her in, she had to improvise.
The project, worthy of a government agency, was related to some sort of war on corruption, the unusual thing being that they wanted to fight the corruption of other governments. Their weapon of choice was the training of representatives of another government, financed by the other government, in their supposedly superior methods of governance. While the general concept was impressive on many levels, the details were unclear.
BC had to speak, and she spoke based on a principle appearing in a book by some McKinsey alumni (she didn't tell its name nor generally recommended it): whatever you tell people, it should contain 3 main points. Possibly 4. But preferably 3. More is overwhelming and less is boring. So she said: "At its core, your project is about teaching people. It is therefore essential to clearly understand three things:
Whom you're teaching,
What you're teaching them,
And how you're teaching it."
And they started writing it down.
I asked BC whether there was some way to unleash her on the company employing me so that she grinds a few bullet points into them (a handsome Compensation Model being as good a place to start as any). She said something to the effect of "it only works on the weak-minded"; it was apparent, she said, that the government agency in question had little previous exposure to consulting.
BC says she (still) believes that business consulting is meaningful and valuable, which sounds paradoxically at this point. But, looked at from another angle, it really isn't. Don't view her stories as ones undermining the credibility of business consulting but rather as ones building her own credibility as a person aware of the actual meaning of things and willing to sincerely share that understanding (how many people would instead say that they Developed Cutting-Edge Compensation Models?) If she says there's meaning to it, perhaps there is.
A few days ago a coworker stepped into the room of our Integration Manager, with his iPhone in his hand and his face glowing with happiness, telling he has this great new game on his phone. He asked for a permission to take a picture of the Integration Manager, and did so to the sound of dramatic, though somewhat repetitive, music. He then marked the areas around the eyes and the mouth of his subject.
Then the game started. The guy pressed some buttons, delivering virtual blows to the photographed subject, who couldn't really fight back according to the rules of the "game". The face on the photograph got covered with blood and wounds (especially the eyes and the mouth), and the speaker yelled out something designed to convey pain. And the guy stood there and kept punching and laughed his ass off.
Then he talked about how iPhone was this great thing and how iPhone app developers got rich. And I told him in a grim voice to go ahead and get rich off iPhone apps and he sensed disrespect, to him or to the iPhone or both. But mainly it wasn't disrespect, it was desperation. I told him to go ahead and do these apps because he probably can, and I certainly can't, so I have no choice but leave all this wealth for him to collect.
Take this punching app. Where do I fit in?
For instance, I know enough computer vision to eliminate the need to manually select the eyes and the mouth. You find the eyes using circular Hough transform, and then you get a good idea where to look for the mouth and then it can't be too hard. How much value would this automation add? The happy user I observed didn't seem bothered by the need to select, worse, I suspect having to do it made him happier - this way he actually worked to deliver his punches.
And then what I couldn't do, and I couldn't do this for the life of me, is to invent this app in the first place. It's just something that I don't think I'll ever understand, something worse than, I dunno, partial differential equations which you can keep digging into, something more like the ability to distinguish between colors, which is either there or not.
Could someone PLEASE explain just how can such a profoundly idiotic activity yield any positive emotions in a human soul?
Perhaps it's because our Integration Manager is a bodybuilder, with bulging biceps and protein milkshakes and stuff, so there's no other way to beat him up for that guy? Well, he could probably beat me up, and he still enjoyed going through the process with me. So this just shows how childish my attempt to dissect the psychology of the phenomenon is - I'm not even close. How could I ever think of something I can't even begin understanding after I've already seen it?
The serious, or should I say sad, side of this is that to develop something valuable to users, you need empathy towards these users, and the way empathy works is through identifying with someone, and so the best chance by far to develop something valuable is to develop something for yourself. But someone like me, who bought his first PC in 2007 and uses a cell phone made in 2004, with no camera, no Internet connection, no nothing, simply can't develop something for himself because he clearly just doesn't need software.
The only chance for a programmer who uses little software except for development tools is to develop development tools. Take Spolsky, for instance - here is a programmer who is, by the programmers' standard, extremely user-aware and user-centric, as evident in his great UI book, among other things, and yet he eventually converged to developing software for programmers, having much less success with end-user products.
The trouble with development tools is that the market is of course saturated, with so many developers with otherwise entrepreneurial mindsets being limited by their lack of empathy to this narrow segment of software. This is why so many programs for programmers have the price of zero - not because copying software has a cost near zero; developing software has a cost far from zero so there aren't that many free face punching apps, but with software for developers, there's unusually fierce empathy-driven competition.
So every time I see an inexplicably moronic hit app, I sink into sad reflections about my destiny of forever developing for developers, being unable to produce what I can't consume and watching those with deeper understanding of the human nature reaping all the benefits of modern distribution channels.
P.S. To the fellow programmer who, as it turns out, uses FarmVille - you broke my heart.
Suppose you have a sparse RAM API, something along the lines of:
add_range(base, size)
write_ram(base, bytes)
read_ram(base, size)
People use this API for things like running a simulated CPU:
define the accessible memory with add_range()
pass the initial state to the simulator with write_ram()
run the simulation, get the final state with read_ram()
Suppose this API becomes a runaway success, with a whopping 10 programmers using it (very little irony here, >95% of the APIs in this world are used exclusively by their designer). Then chances are that 9 of the 10 programmers are API users, and 1 of them is an API wrapper. Here's what they do.
API users
The first thing the first API user does is call you. "How do I use this sparse thing of yours?" You point him to the short tutorial with the sample code. He says "Uhmm. Errm…", which is userish for "Come on, I know you know that I'm lazy, and you know I know that docs lie. Come over here and type the code for me." And you insist that it's actually properly documented, but you will still come over, just because it's him, and you personally copy the sample code into a source file of his:
add_range(0x100000, 6) # input range
add_range(0x200000, 6) # output range
write_ram(0x100000, "abcdef")
# run a program converting the input to uppercase
print read_ram(0x200000, 6) # should print "ABCDEF"
It runs. You use the opportunity to point out how your documentation is better than what he's perhaps used to assume (though you totally understand his frustration with the state of documentation in this department, this company and this planet). Anyway, if he has any sort of problem or inconvenience with this thing, he can call you any time.
The next 8 API users copy your sample code themselves, some of them without you being aware that they use or even need this API. Congratulations! Your high personal quality standards and your user-centric approach have won you a near-monopoly position in the rapidly expanding local sparse RAM API market.
Then some time later you stumble upon the following code:
You knew the API was a bit too low-level for the quite common case where you need to allocate a whole lot of objects, doesn't matter where. In that case, something like base=allocate_range(size) would be better than add_range(base,size) - that way users don't have to invent addresses they don't care about. But it wasn't immediately obvious how this should work (Nth call to allocate_range() appends a range to the last allocated address, but where should the first call to allocate_range() put things? What about mixing add_range() and allocate_range()? etc.)
So you figured you'd have add_range(), and then whoever needed to allocate lots of objects, doesn't matter where, could just write a 5-line allocate_range() function good enough for him, though not good enough for a public API.
But none of them did. Why? Isn't it trivial to write such a function? Isn't it ugly to hard-code arbitrary addresses? Doesn't it feel silly to invent arbitrary addresses? Isn't it actually hard to invent constant addresses when you put variable-sized data there, having to think about possible overlaps between ranges? Perhaps they don't understand what a sparse RAM is? Very unlikely, that, considering their education and experience.
Somehow, something makes it very easy for them to copy sample code, but very hard to stray from that sample code in any syntactically substantial way. To them, it isn't a sparse RAM you add ranges to. Rather, they think of it as a bunch of add_range() calls with hexadecimal parameters.
And add_range() with hex params they promptly will, just as it's done in the sample. And they'll complain about how this API is a bit awkward, with all these hex values and what-not.
API wrappers
If there's someone who can see right through syntax deep into semantics, it's the tenth user of your API, or more accurately, its first wrapper. The wrapper never actually uses an API directly in his "application code" as implied by the abbreviation, standing for "Application Programming Interface". Rather, he wraps it with another (massive) layer of code, and has his application code use that layer.
The wrapper first comes to talk to you, either being forced to use your API because everybody else already does, or because he doesn't like to touch something as low-level as "RAM" so if there's already some API above it he prefers to go through that.
In your conversation, or more accurately, his monologue, he points out some admittedly interesting, though hardly pressing issues:
It's important to be able to trick a program using the sparse RAM API into allocating its data in specific address ranges, so that the resulting memory map is usable on certain hardware configurations and not just in simulations.
In particular, it is important to be able to extract the memory map from the section headers of executables in the ELF and COFF format.
Since add_range() calls are costly, and memory map formats such as the S-Record effectively specify a lot of small, adjacent ranges, there is a need for a layer joining many such ranges.
An extensible API for the parsers of the various memory map formats is needed.
…
When you manage to terminate the monologuish conversation, he walks off to implement his sparse RAM API on top of yours. He calls it SParser (layer lovers, having to invent many names, frequently deteriorate into amateur copywriters).
When he's done (which is never; let's say "when he has something out there"), nobody uses SParser but him, though he markets it heavily. Users won't rely on the author who cares about The Right Thing but not about their problems. Other wrappers never use his extra layers because they write their own extra layers.
However, even with one person using it, SParser is your biggest headache in the sparse RAM department.
For example, your original implementation used a list of ranges you (slowly) scanned through to find the range containing a given address. Now you want to replace this with a page table, so that, given an address, you simply index into a page array with its high bits and either find a page with the data or report a bad address error.
But this precludes "shadowing", where you have overlapping segments, one hiding the other's data. You thought of that as a bug in the user code your original implementation didn't detect. The wrapper thought it was a feature, and SParser uses it all over to have data used at some point and then "hidden" later in the program.
So you can't deploy your new implementation, speeding up the code of innocent users, without breaking the code of this wrapper.
What to do
Add an allocate_range() API ASAP, update the tutorial, walk over to your users to help replace their hex constants with allocate_range() calls. Deploy the implementation with the page table, and send the complaining wrapper to complain upwards along the chain of command.
Why
Your users will switch to allocate_range() and be happy, more so when they get a speed-up from the switch to page tables. The wrapper, constituting the unhappy 10% of the stakeholders, will have no choice but fix his code.
Ivan drank half a bottle of vodka and woke up with a headache. Boris drank a full bottle of vodka and woke up with a headache. Why drink less?
Users are many, they follow a predictable path (copy sample code) and are easily satisfied (just make it convenient for them to follow that path). Wrappers are few, they never fail to surprise (you wouldn't guess what and especially why their layers do), and always fail to be satisfied (they never use APIs and always wrap them). Why worry about the few?
The only reason this point is worth discussing at all is that users offend programmers while wrappers sweet-talk them, thus obscuring the obvious. It is natural to feel outrage when you give someone an add_range() function and a silly sample with hex in it, and not only do they mindlessly multiply hex numbers in their code, but they blame you for the inconvenience of "your API with all the hex in it". It is equally natural to be flattered when someone spends time to discuss your work with you, at a level of true understanding ("sparse RAM") rather than superficial syntactic pattern matching ("add_range(hex)").
He who sees through this optical illusion will focus on the satisfaction of the happy many who couldn't care less, securing the option to ignore the miserable few who think too much.
Ever noticed how academic asses are analog and industrial asses are digital? It's legitimate to not know whether P equals NP, or to not know what x is if x*2=y but we don't know y, for that matter. But it isn't legitimate to not know how many cycles, megabytes or - the king of them all - man-months it will take, so numbers have to be pulled out of one's ass.
The interesting thing is that the ass adapts, that the numbers pulled out of this unconventional digital device aren't pure noise. Is it because digital asses know to synchronize? Your off-by-2-months estimation is fine as long as other estimations are off by 5. But it's not just that, there must be something else, a mystery waiting to be discovered. We need a theory of computational proctology.
Ever noticed how painful the act of anal estimation is for the untrained, um, mind, but then eventually people actually get addicted to it? Much like managers who learn that problems can be made to go away by means such as saying a firm "No", without the much harder process of understanding the problem, not to mention solving it? Anal prophecy is to the technical "expert" the same raw enjoyment that the triumph of power over knowledge is to the manager. "Your powers are nothing compared to mine!"
There once was a company called ArsDigita (I warmly recommend the founder's blog and have his Tenth Rule tattooed all over my psyche), a name I tend to misread as "ArseDigital" - a tribute to an important method of numerical analysis and estimation in the computing industry.
I never managed a group larger than 5 people, luckily for the people in the group (perhaps more so for those remaining outside). Good managers are hard to find, which is the basis of my self-motivating motto: "This job could have been done worse". Such is the background for the hereby presented pearls of wisdom assortment. As to "The Virtue of a Manager" title, it's a ripoff of Paul Krugman's exquisite title "The Conscience of a Liberal". "The Private Part of a Self-Important Self-Description" is a great template.
***
A prime virtue of a manager is the ability to take pride in someone else's work.
No, seriously. We've recently deployed a debugger internally and an algorithm developer had a look at it. I knew it was good, but it's used to debug the sort of thing algo devs hate: code with an anal-retentive performance focus. So the last thing I expected was praise, but praise it the guy did.
Now, I had previously known proud moments from having done things myself, and here I had this proud moment with 90% of the work done by someone else. And I'm telling you, it was just like the real thing.
***
The defining trait of a manager is the distinctly wide gap between responsibility and understanding.
By far the funniest spot to have a gap at, hence the easiest target for a low blow: try to make jokes about a gap between one's teeth and you'll soon be exhausted, but this here is gold. This is mean-spirited though. Imagine living with a gap between your responsibility and your understanding and everybody laughing at you - how would that make you feel? Show compassion.
***
One can have the title of a manager or nominal reports for any of a number of reasons:
An HR system with per-title wage ceilings: can't give someone a raise without faking a title.
A diametrically opposed case: some forms of brain damage cause people to accept lower paychecks given more impressive titles, larger rooms, etc.
Someone is too senior to report to a team leader but doesn't want a team to report to him, either.
…
In a roomful of managers, how do you find the real ones among this variety - not "real" as opposed to incompetent or unimportant, but "real" as opposed to fake?
There are several cues, for example, only real managers can have other managers report to them. But the perfect, if-and-only-if discriminator is that real managers don't write code. (The precise rule is that they can spend up to 2% of their time on a favorite piece of code without getting disqualified.)
***
The principal function of a manager is being the responsible adult.
Some managers occasionally point this out in frustration, both mourning their technical skills which dry up during their current gig where they only get to exercise adulthood, and because being the adult means getting tired of the annoying kids. A gal who both managed and met literally hundreds of managers during her career in some consulting agency said "Now I really understand management" when she got to babysit.
This is why I have hard time believing management can be taught - you can't teach adulthood, it can only result from people growing up by themselves. I'm not sure if this feeling is fully aligned with reality, but quite some very successful managers never went to a management school (at least one of those is somewhat critical of MBAs), and some of those who went say it was worthless in terms of useful things learned.
The opposite is also true: childishness is fitting for a programmer. We were two fake code-writing managers in a meeting with one real one, and at one point the real one said: "Let's not be childish about this". The technically correct reply to her would have been "I'M NOT CHILDISH ABOUT THIS, HE IS!", but I suppressed it for tactical reasons. Some time later I told her: "You don't want us to stop being childish about this, not as long as you're interested in our output as programmers. Recall: the reason you aren't still programming is because of not being childish enough to truly enjoy this sort of game."
And in fact since she started managing 20 programmers, she's been talking about her work all the time, which she didn't when she was programming. Well, some people like to play and some prefer to babysit. (I'm not sure where this leaves the quasi-managers who write code; presumably some are the elder and most responsible kid while others are the most restless who invent games for the gang.)
***
I've recently got a driving license. One thing I learned was that someone pushing his (presumably broken) car along the road is a "driver" as far as the law is concerned. I find this counter-intuitive, probably because pushing a car is not categorized in my head as "driving experience", but, at least in Israel, that's the law.
Likewise, doing the work of three people is not what most of us associate with "managerial responsibility". However, if you're given two reports without a drive of their own to work, that's what your responsibility will be.
***
A manager will have favorite words. For example: acute (critical), priorities, agenda, rationale, integrity (shoot this manager first), responsibility (ownership), stakeholder.
Keep laughing at them. Once you become a manager, you'll have favorite words whether you want it or not - it is useless to resist the dynamics inherent to your situation. My favorite word is "dynamics". Its connotations are deep and its applicability wide - heartily recommended.
***
Managers get to do a lot of knowledge-free decision making, which necessarily drives them insane. Here's how the manager's bipolar disorder works.
During maniacal periods, the manager is the only one who can do anything around here. This frequently happens when the manager is under external pressure, and he feels that control is slipping out of his hands. He's trying to compensate for his lack of knowledge by immense concentration and willpower. (Managers always have ample emergency supplies of both.) "Concentration" translates to an ability to derive general and far-reaching conclusions from insignificant details, then "willpower" translates to aggression.
Then depression follows: "Don't bother me with details". This results partly from exhaustion quickly arrived at during the mania (especially if reports were wise enough to not argue with the manager, letting his efforts defeat their own purpose.) The manager has delivered his trademark concentration and willpower, so he no longer feels guilty on that front. However, he's overwhelmed by information and (rightly) feels that he doesn't know what's going on. He decides it is none of his business and concentrates on the Big Picture (does nothing). Usually, the cycle repeats upon a new wave of external pressure.
Awareness of the management cycle on behalf of the manager himself can help soften the cycle but not eliminate it. It is up to reports to apply counter-cycle measures by scheduling most work into depression periods when it is least disrupted. Special attention must be given to long-term projects, frequently characterized by a prolonged depressive apathy period at the beginning followed by a period of maniacal frenzy lasting until the end.
***
There's a naive brain model in the spirit of "the brain has a reptilian part, a mammal part and a human part". For example, if a student fails to answer a question in an oral exam with his human brain, the mammal brain feels bad about it and complains to the reptilian brain. The reptilian brain then cheerfully replies, "Who's causing the trouble? Oh, that little guy behind the table? Not to worry - I'll kill him". The higher brains then supposedly suppress this - "What do you think this is, reptile - Jurassic Park?", and the tension is translated into sweating.
The manager is the team's reptilian brain; he doesn't know enough to do real thinking, but he's good at "taking responsibility", bargaining, fighting, socializing, etc. A manager doesn't know how to implement the feature, except for suspecting, based on experience, that it will conflict with a couple other features and it will take a week or three for the whole thing to stabilize (with him taking the heat when things break during those weeks). Therefore, instead of technical advice (which he might be otherwise qualified to give), he'll propose something which solves the problem at his favorite social plane:
Prioritize the feature away, delaying the implementation until forever
Negotiate the feature away, by talking to whoever wants it out of it for something in return
Redefine the feature away, by reducing the scope to the few scenarios which absolutely can't be ignored
…
Do not drag management into anything you actually want solved. Presented with a question, the manager will answer it by killing the little guy behind the table, so only go to him if you really want that. And once awakened, he might take a lot of sweat to suppress. (If he's really a programmer posing as a quasi-manager, the chances for an actual solution can actually be worse: he's more likely to feel guilty about his managerial ability and use the opportunity to exercise and develop that ability, instead of using his technical ability to think about the issue.)
***
There's this quote from The Mythical Man-Month, supposedly by a pessimistic manager:
All programmers are optimists. Perhaps this modern sorcery especially attracts those who believe in happy endings and fairy god-mothers. Perhaps the hundreds of nitty frustrations drive away all but those who habitually focus on the end goal. Perhaps it is merely that computers are young, programmers are younger, and the young are always optimists. But however the selection process works, the result is indisputable: "This time it will surely run," or "I just found the last bug."
This is backwards. In reality, programmers are the more pessimistic people. Perhaps it's because experience teaches programmers that programs always have bugs while teaching managers that programs always ship. Perhaps it's because the programmer is the one with the actual knowledge, and the ignorant are always optimists. But however the selection process works, how many programmers have you seen saying "it will never work" and how many managers?
A programmer might be more optimistic locally, hoping in vain to have fixed this one piece of code where he has the illusion of complete understanding. However, it is invariably the manager who believes that everything will work out. A programmer can't really believe that because there are so many things nobody even understands that are yet to be faced.
But the manager is used to knowing little and understanding less, and thus has learned to translate uncertainty to optimism. In fact a programmer can learn it, too, in the areas which are of little interest to him. I know a programmer who doesn't care about optimization and who consequently describes others' efforts to fit a program into a given performance budget as doomed to success: "It runs at the word of command" - a programmer's expression of the managerial worldview worthy of a seasoned manager.
***
We don't know how to test for programming ability. The best tech companies spend 5 to 10 interviews to solidly confirm that the candidate knows what is taught during the first 1.5 years of an undergraduate CS curriculum. Other processes measure less accurately by asking less relevant questions; the inaccuracy is somewhat ameliorated by the lack of precision - the non-uniform quirks of interviewers and general randomness of the process eliminate biases, causing all kinds of good candidates to sneak through the gates.
It is well known that we can't find out during the interview what we inevitably find out once someone gets the job, but what are the corollaries? Here's one I've heard a lot: trust recommendations more than interviews. Here's another I haven't: let others interview and get the new hires, then steal the best.
(Objection: the first recommendation is good for the company while the other is only good for the manager following it. Well, "competition between managers over team members isn't a zero-sum game - it improves teamwork across the company", this one we weasel out of in a snap.)
***
We have a VIP club at work called Bottleneck, its principal activity being the collective purchasing and consumption of alcoholic beverages. The club operates during work hours (regular meetings held on Thursdays, emergency meetings scheduled upon arrival of packages from abroad). Our room being the headquarters, I'm naturally a member. By now the club has shifted to high-end liquors at prices causing the consumption to contract to a sip per cup of coffee, but originally it was affordable to actually drink.
I noticed that minor alcoholic intoxication has a notable impact on my programming ability. I can still lay my hands on the right variable, but by the time I do I forget what you do next with these things. There's that handy member somewhere in it, dot something, but dot what?
However, managerial ability is not affected. Things I can do just fine following a meeting of the Bottleneck club include progress monitoring, planning, risk assessment, general technical advice, and requirement negotiation. Now that I think of it, perhaps the managerial functions are affected for the better.
There's this broad metaphor I have, with no conclusions attached - just an attempt to describe dynamics. I've recently shared it with the commie ex-VP school teacher and he liked it, so I thought it could fit well with the other borderline stuff I host at Proper Fixation under the "wetware" category.
So. One recurring theme in the history of civilization is the conflict between nomadic and settled people. Nomads think that land is for feeding cattle and you move elsewhere once there's nothing left to graze. Villagers figure that land is for growing food, so you settle on it and fertilize it and irrigate it and stuff. Initially, nomads typically dominate the landscape, periodically attacking the settled villagers and taking their crops. However, the settled people eventually accumulate enough surplus to support cities, nation states and standing armies, extending their control to more and more lands and eventually exterminating the nomadic lifestyle altogether.
The way I painted this picture, I tend to side with the hard-working settled folk, the nomads being the parasitic losers I've depicted, and I think most of us civilized humans share similar sentiments. However, in my metaphor I side with the nomadic programmer, at least to a large extent, and I do so because of the meaning my metaphor assigns to "land".
The thing I find analogous to land in programming is problems, because that's where programmers live. Programmers live on (in?) problems in the sense of dealing with broken things most of the time - once something starts working, you move on to something that doesn't. In another sense, large problems or problem areas a programmer deals with define that programmer's territory. The programmer is in immediate demand to the extent that solutions to "his" problems are in demand; problems feed programmers. Strong programmers seek, in one way or another, to expand their responsibility to encompass more problems, and to preserve their existing responsibilities. And so on.
Now if we restate the respective worldviews of nomads and settlers in the terms of this metaphor, we'll get this. Nomads think that problems exist for solving them and you move elsewhere once there's nothing left to graze. Settlers think that problems exist for growing them, so they settle on them and fertilize them and irrigate them and stuff.
And now you can see why I'm inclined to sympathize with the nomadic programmer. Two other things fueling this sympathy are issues of personality to be discussed soon, and the fate of the nomad to be discussed immediately. And while the nomad is no longer the parasite, rest assured that he's still, in the long run, the loser.
Initially - in a young and small organization - nomadic programmers tend to dominate the landscape. There are more problems than people around. The nomadic programmer travels from one urgent problem to another, grazing through them as fast as he can. Occasionally he stumbles upon a settler who has settled on a problem near the nomad's territory and grown crops of code there. Well, if the problem occupied by the settler becomes urgent, or if the crops stand in the way of solving the nomad's adjacent urgent problem, the nomad will go ahead and brutally solve the settler's problem, wiping out his crops. The politics of the invasion will be trivial - a promise to deliver by the nomad carries lots of weight at this stage and the settler will not issue a counter-promise (to deliver in his own way) because he's a peaceful code-growing villager who isn't into stress which necessarily comes with delivering quickly.
However, the time goes by and sure enough, the settled people accumulate quite some surplus. What you grow on land is surplus wheat; what you grow on problems is surplus code. Code that wouldn't naturally grow on a problem - but now that the problem was fertilized by the original settlers, they've grown enough code on it to support whole cities, a nation state, and a standing army of programmers, all making a living by fiddling with this code.
The nomad starts running out of pasture. Sure enough, there are lots of problems just like there used to be. But you can no longer solve them because (1) now it's the majority and not a minority of problems that are already owned by someone (growing them rather than solving them) and (2) in most cases invasion is no longer an option. Now that the problem is owned by a nation state, responsible for lots of code and with lots of people working on that code, the nomad's promise to deliver quickly carries very little weight compared to the danger of irritating the sovereign. While it is quite likely still true that a nomad will probably deliver more quickly than the whole nation state team, the nomad will not be able to take over the entire responsibility of the team. (It is possible that the single reason for the latter is the problems grown by the team itself and that a few nomads could in fact handle the original problem. But it is irrelevant since problems that could have been avoided are no less real than problems that couldn't.)
So if the organization, by some decision making mechanism, lets the nomad invade the territory of the settled team and solve the stupid problem, and then the offended team, by some decision making mechanism, fights back by effectively going on strike, there is nothing the nomad will be able to offer the organization at this point. Of course it doesn't have to come to this, just like political conflicts don't have to come to full-scale wars, or personal conflicts to fist fights or court hearings. It's enough for the worst case scenario to be likely to work out in favor of A rather than B to shift the balance decisively in favor of A. Even if neither A nor B nor anyone making decisions affecting A and B actually think in terms of this scenario, things tend to evolve and adapt such that decisions are made in favor of A. And in our case, the nomadic programmer is B.
Solving problems just isn't the big thing in this organization anymore, just like the quality of life experienced by the inhabitants of some territory isn't the main theme in international politics. Perhaps there are ways to improve the quality of life in Siberia, however this is not nearly as important politically as the fact that there's already a guy exclusively responsible for the quality of life in Siberia. Perhaps Socialism with Chinese Characteristics could yield improvements in the lives of Siberians that Managed Democracy could not, however, if the Chinese try to act on this assumption, there will be a nuclear war. If what remains of a nomadic tribe somewhere in the region makes a similar attempt, then it will remain no more.
The disgruntled nomadic programmer reduces his ambition to merely being left alone to wander the remaining wilderness. However, this option is no more real for him now than the option of being left alone was available to the settler in the old days. Back then, the settlers were never safe since a nomad could always bump into them in an attempt to solve a related problem, and if their stuff got in the way, he'd rewrite or delete/disable their stuff. Now it is the nomad who is never safe since the nation states keep expanding their responsibilities into neighboring problems - having enough people to have some of them free for that some of the time.
(Actually having even partially idle workers on a team leaves few satisfying alternatives to an attempt at expanding the team's responsibilities since other teams are always happy to seize an idle worker. Likewise, back in the old days the nomadic programmer had few satisfying alternatives to invading and solving others' problems since otherwise he couldn't keep his promises to deliver. It's not (just) the intentions that fuel wars, it's (also) the situation.)
The nation states seeking to expand won't fight each other since the nomad is a much easier target, not having resources (time and reports) to look over his entire territory. Once a nation state team managed to take over some of that ever-shrinking territory, the nomad will never gain it back. Increasingly, the nomad has to reach compromises with neighboring nation states whenever his work is related to their work. Then it turns out that in order to be able to work on what he wants at all, he has to do it the way a chief commander or an officer of a nation state team wants him to do it - and then that in order to work on anything at all, he has to report to such a manager.
At this point the nomadic programmer can use his reputation and seniority to get pseudo-promoted to a non-productive position. Alternatively, he can actually become a report of a nation state team manager with whom the relationship is likely already strained - and his seniority, reputation and ambitions won't make the transition into this particular position of a report any smoother. Alternatively he can quit. His failure is now complete.
(It may sound like a natural thing for a nomad to change jobs fairly frequently - part of a lifestyle rather than the failure of that lifestyle. However, nomadic programmers are those who like to travel from problem to problem - not from job to job; some like the latter but some don't. A new job at a new place means a temporary, but possibly significant loss of confidence and efficiency. An African nomad won't necessarily welcome a relocation to Alaska.)
As I've said above, I have reasons having to do with my personality to side with the nomadic programmer, especially at the stage of mounting pressure from nation state teams. The people I tend to relate to most easily seem to be those who prefer freedom to power. A talented freedom-seeker with a strong sense of responsibility will accumulate, well, responsibilities much more quickly than reports - a lot of territory to wander, and no standing army to protect it. (The problem with reports is that you take their freedom by telling them what to do and they take your freedom through your responsibility for their actions; who wants reports?) Since many freedom-lovers disdain politics, they won't respect international borders - a problem should be solved, dammit; hence they're likely to initiate invasions.
However, while this means that I personally will tend to find myself sympathizing with particular nomadic programmers, this does not mean that theirs is the right way or something. For example, it is unclear which share of programming problems out there can really be "solved" - grazed through and left alone - and which problems actually require continuous care and gardening that a true nomad is not likely to supply. Also, whether there's a "solution" you only need to "maintain" or an "infrastructure" you want to "extend", the code needs a permanent owner. I don't believe in collective code ownership any more than in collective ownership of anything else - what it usually means is that everybody collectively fights over something. Therefore I think that ownership should generally be respected, and so a compromise which is, from a technical viewpoint, quite moronic, can otherwise be a great thing - a belief outside the nomad's way.
So while I know where my sympathies lie, I don't know which camp I'm in and this is why this metaphor doesn't come with any conclusions, just the dynamics. In fact I'd rather leave it without conclusions but I wouldn't mind expanding more on the dynamics. For example, some - but not all - settled civilizations were actually started by nomads enslaving argicultural villagers and settling among them. Apparently a similar distinction can be made between nation state teams of programmers; it is then interesting whether differences in their behavior can be traced to their different origins. Perhaps a person more entertained than appalled by the sort of perspective on the adventurous lives of programmers here presented is also the kind of person more entertained than appaled by the history of mankind in general and so could help develop this line of thought based on his knowledge of history. Could be fun.
Update (2009-08-18) - Chuck Moore: "I’ve met too many people who want to make a career out of a project instead of completing it" - the nomad's view of the settlers. Nomadism is apparent in other writing by Chuck Moore - his disdain for "complexity" (which implies dependency on large teams of people you ought to manage, annoying constraints imposed by systems made by someone else and other things nomads don't like), his firm opinion that distinct projects should have distinct code bases (customizability and "reuse" imply complexity and otherwise reduce the chances to "hermetically close" and truly complete a project), etc.
Proper Fixation always had more unfinished drafts than posts, but recently it's getting ridiculous. I do have a couple of drafts I seriously intend to finish (usually the drafts which don't make it to posthood during the first 4 hours or so go to the eternal drafthood land.) Until I'm able to think this stuff out to the point where I can share the results of my thinking, I figured I could share the far less scarce resource of Wisdom with ya.
***
Since I've violated the Golden Rule of Helping Friends with their PC Problems and attempted to help a friend with his PC problem, expectedly wiping out his hard drive in vain, I had many opportunities to explain the Programmer Paradox: how can a programmer fail to make a computer do as he wishes? While the difficulty of debugging a program without the source proved hard to explain to laymen, I think I've found a metaphor that does a good job. A programmer is to the blue screen of death what Mikhail Kalashnikov is to a loaded AK-47: just as helpless a victim as any other mortal, except for having a profound understanding of the mechanisms of his execution.
***
I would like to get some statistics on file encryption. For example, of all the files on the planet, X% are encrypted. Of all those files, Y% will never be read by someone due to encryption. Of all those files, Z% will never be read by malicious intruders. If I could lay my hands on the value of just one of these unknowns, I'd pick Z, because at least 100-Z% of the files will never be read by their owners. I would bet on Z lying somewhere between 0 and 1.
***
One of the key traits of good code is the ease at which it can be modified. One of the key traits of bad code is the high cost of modifying it. So good code is likely to deteriorate until it's bad enough to become hard to change, and bad code is likely to stay bad. In short, code has a strong tendency to end up bad.
This can sound worthlessly pessimistic, similarly, for example, to "It is easier to break a leg than it is to cure it, therefore, most legs end up broken." However, I think it's more analogous to aging - the accumulation of changes in an organism, observably causing most animals to end up dead. Similarly, code that is used will be changed, code that is changed will degrade, and code that degrades beyond a certain point will die.
***
Health tends to be simpler than disease. For example, everybody can brush their teeth but few people can treat cavities. Similarly, it's not very hard to maintain a sane development environment, but pretty hard to deal with the tide of bugs and of long-living branches resulting from a failure to do so. However, I'm generally optimistic about the chances of such cavities to be treated, and as usual, the optimism is based on the pain they cause - a strong incentive to seek and reward treatment.
***
There's this evolution vs Intelligent Design debate. Well, I don't know about life on Earth, but I sure have hard time believing in Intelligent Design in software. Code has to repeatedly survive exposure to users upon whom its fate depends. Yes, "users" can be a set containing just the author, but only if it's honest-to-God USAGE, that is, the author has to pay a price when the program is hard to use - like not getting important things done properly. Show me a program that someone finds useful and that wasn't subject to such evolutionary pressure, but rather was Intelligently Designed as useful.
I think that my intense hatred of the word "design" has to do with its prominent place in the speech of software creationists. These people are likely to constantly complain about not having enough resources to do The Right Thing in the ugly real world. They are also likely to give you software that you hate enough to wish to kill them, and be articulate enough to convince you that the problem is at your end, and fail to notice how this latter ability quadruples your desire to slash their body into square millimeter pieces.
***
I'll conclude with an off-topic request: if you know a good text advocating a collectivistic or other kind of heterodox approach to economics, I'd be very grateful for a reference. By "advocacy", I mean a text for laymen expressing support for a certain set of policies (as opposed to merely criticizing the effects of existing policies) - like Milton Friedman's "Capitalism and Freedom", for example.
