10x more selective

There's this common notion of "10x programmers" who are 10x more productive than the average programmer. We can't quantify productivity so we don't know if it's true. But definitely, enough people appear unusually productive to sustain the "10x programmer" notion.

How do they do it?

People often assume that 10x more productivity results from 10x more aptitude or 10x more knowledge. I don't think so. Now I'm not saying aptitude and knowledge don't help. But what I've noticed over the years is that the number one factor is 10x more selectivity. The trick is to consistently avoid shit work.

And by shit work, I don't necessarily mean "intellectually unrewarding". Rather, the definition of shit work is that its output goes down the toilet.

I've done quite a lot of shit work myself, especially when I was inexperienced and gullible. (One of the big advantages of experience is that one becomes less gullible that way - which more than compensates for much of the school knowledge having faded from memory.)

Let me supply you with a textbook example of hard, stimulating, down-the-toilet-going work: my decade-old adventures with fixed point.

You know what "fixed point arithmetic" is? I'll tell you. It's when you work with integers and pretend they're fractions, by implicitly assuming that your integer x actually represents x/2^N for some value of N.

So to add two numbers, you just do x+y. To multiply, you need to do x*y>>N, because plain x*y would represent x*y/2^2N, right? You also need to be careful so that this shit doesn't overflow, deal with different Ns in the same expression, etc.

Now in the early noughties, I was porting software to an in-house chip which was under development. It wasn't supposed to have hardware floating point units - "we'll do everything in fixed point".

Here's a selection of things that I did:

  • There was a half-assed C++ template class called InteliFixed<N> (there still is; I kid you not). I put a lot of effort into making it, erm, full-assed (what's the opposite of half-assed?) This included things like making operator+ commutative when it gets two fixed point numbers of different types (what's the type of the result?); making sure the dreadful inline assembly implementing 64-bit intermediate multiplications inlines well; etc. etc.
  • My boss told me to keep two versions of the code - one using floating point, for the noble algorithm developers, and one using fixed point, for us grunt workers fiddling with production code. So I manually kept the two in sync.
  • My boss also told me to think of a way to run some of the code in float, some not, to help find precision bugs. So I wrote a heuristic C++ parser that automatically merged the two versions into one. It took some functions from the "float" version and others from the "fixed" version, based on a header-file-like input telling it what should come from which version.
  • Of course this merged shit would not run or even compile just like that, would it? So I implemented macros where you'd pass to functions, instead of vector<float>&, a REFERENCE(vector<float>), and a horrendous bulk of code making this work at runtime when you actually passed a vector<InteliFixed> (which the code inside the function then tried to treat as a vector<float>.)
  • And apart from all that meta-programming, there was the programming itself of course. For example, solving 5×5 equation systems to fit polynomials to noisy data points, in fixed point. I managed to get this to work using hideous normalization tricks and assembly code using something like 96 bits of integer precision. My code even worked better than single-precision floating point without normalization! Yay!

For months and months, I worked as hard as ever, cranking out as much complicated, working code as ever.

And here's what I should have done:

  • Convince management to put the damned hardware floating point unit into the damned chip. It didn't cost that many square millimeters of silicon - I should have insisted on finding out how many. (FPUs were added in the next chip generation.)
  • Failing that, lay my hands on the chip simulator, measure the cost of floating point emulation, and use it wherever it was affordable. (This is what we ended up doing in many places.)
  • Tell my boss that maintaining two versions in sync like he wanted isn't going to work - they're going to diverge completely, so that no tool in hell will be able to partially merge them and run the result. (Of course this is exactly what happened.)

Why did this end up in many months of shit work instead of doing the right thing? Because I didn't know what's what, because I didn't think I could argue with my management, and because the work was challenging and interesting. It then promptly went down the toilet.

The hardest part of "managing" these 10x folks - people widely known as extremely productive - is actually convincing them to work on something. (The rest of managing them tends to be easy - they know what's what; once they decide to do something, it's done.)

You'd expect the opposite, kind of, right? I mean if you're so productive, why do you care? You work quickly; the worst thing that happens is, nothing comes out of it - then you'll just do the next thing quickly, right? I mean it's the slow, less productive folks that ought to be picky - they're slower and so get less shots at new stuff to work on to begin with, right?

But that's the optical illusion at work: the more productive folks aren't that much quicker - not 10x quicker. The reason they appear 10x quicker is that almost nothing they do is thrown away - unlike a whole lot of stuff that other people do.

And you don't count that thrown-away stuff as productivity. You think of a person as "the guy who did X" where X was famously useful - and forget all the Ys which weren't that useful, despite the effort and talent going into those Ys. Even if something else was "at fault", like the manager, or the timing, or whatever.

To pick famous examples, you remember Ken Thompson for C and Unix - but not for Plan 9, not really, and not for Go, not yet - on the contrary, Go gets your attention because it's a language by those Unix guys. You remember Linus Torvalds even though Linux is a Unix clone and git is a BitKeeper clone - in fact because they're clones of successful products which therefore had great chances to succeed due to good timing.

The first thing you care about is not how original something is or how hard it was to write or how good it is along any dimension: you care about its uses.

The 10x programmer will typically fight very hard to not work on something that is likely enough to not get used.

One of these wise guys asked me the other day about checkedthreads which I've just finished, "so is anyone using that?" with that trademark irony. I said I didn't know; there was a comment on HN saying that maybe someone will give it a try.

I mean it's a great thing; it's going to find all of your threading bugs, basically. But it's not a drop-in replacement for pthreads or such, you need to write the code using its interfaces - nice, simple interfaces, but not the ones you're already using. So there's a good chance few people will bother; whereas Helgrind or the thread sanitizer, which have tons of false negatives and false positives, at least work with the interfaces that people use today.

Why did I bother then? Because the first version took an afternoon to write (that was before I decided I want to have parallel nested loops and stuff), and I figured I had a chance because I'd blog about it (as I do, for example, right now). If I wrote a few posts explaining how you could actually hunt down bugs in old-school shared-memory parallel C code even easier than with Rust/Go/Erlang, maybe people would notice.

But there's already too much chances of a flop here for most of the 10x crowd I personally know to bother trying. Even though we use something like checkedthreads internally and it's a runaway success. In fact the ironic question came from the guy who put a lot of work in that internal version - because internally, it was very likely to be used.

See? Not working on potential flops - that's productivity.

How to pick what to work on? There are a lot of things one can look at:

  • Is there an alternative already available? How bad is it? If it's passable, then don't do it - it's hard to improve on a good thing and even harder to convince that improvements are worth the switch.
  • How "optional" is this thing? Will nothing work without it, or is it a bell/whistle type of thing that can easily go unnoticed?
  • How much work do users need to put in to get benefits? Does it work with their existing code or data? Do they need to learn new tricks or can they keep working as usual?
  • How many people must know about the thing for it to get distributed, let alone used? Will users mostly run the code unknowingly because it gets bundled together with code already distributed to them, or do they need to actively install something? (Getting the feature automatically and then having to learn things in order to use it is often better than having to install something and then working as usual. Think of how many people end up using a new Excel feature vs how many people use software running backups in the background.)
  • How much code to deliver how much value? Optimizing the hell out of a small kernel doing mpeg decompression sounds better than going over a million lines of code to get a 1.2x overall speed-up (even though the latter may be worth it by itself; it just necessarily requires 10x the programmers, not one "10x programmer").
  • Does it have teeth?If users do something wrong (or "wrong"), does it silently become useless to them (like static code analysis when it no longer understands a program), or does it halt their progress until they fix the error (like a bounds-checked array)?

You could easily expand this list; the basic underlying question is, what are the chances of me finishing this thing and then it being actually used? This applies recursively to every feature, sub-feature and line of code: does it contribute to the larger thing being used? And is there something else I could do with the time that would contribute more?

Of course it's more complicated than that; some useful things are held in higher regard than others for various reasons. Which is where Richard Stallman enters and requires us to call Linux "GNU/Linux" because GNU provided much of the original userspace stuff. And while I'm not going to call it "Gah-noo Lee-nux", there's sadly some merit to the argument, in the sense that yeah, unfortunately some hard, important work is less noticed than other hard, important work.

But how fair things are is beside the point. After all, it's not like 10x the perceived productivity is very likely to give you 10x the compensation. So there's not a whole lot of reasons to "cheat" and appear more productive than you are. The main reason to be productive is because there's fire raging up one's arse, more than any tangible benefit.

The point I do want to make is, to get more done, you don't need to succeed more quickly (although that helps) as much as you need to fail less often. And not all failures are due to lack of knowledge or skill; most of them are due to quitting before something is actually usable - or due to there being few chances for it to be used in the first place.

So I believe, having authored a lot of code that went down the toilet, that you don't get productive by working as much as by not working - not on stuff that is likely to get thrown away.

26 comments ↓

#1 GD on 04.19.13 at 10:27 am

Were you referring to yours truly in the article above? I guess you did. I just want to debate some of the thesis above. Yes, being selective/avoiding failures is important, but also having the courage to dare and fail in something is necessary. And yes, I think the 10x is certainly also an aptitude/knowledge and mind-set thing. You need to avoid wasted effort, that is for sure. Indeed I guess the 10x man will just write at the first try what an 'average' one will take 10 versions to achieve if at all …

I am not sure the 10x is also the correct metric, as you said, productivity can not be measured. The more important contributions are sometimes just things that would not have existed/thought of at all by the 1x / average person.

I am also not sure your personal example above was in place, as I do think you are in the 10x category, and apparently you did produce quite a lot of non trivial code/work during that time too. With hindsight you know it was wasted/doomed to fail effort, but that's because you've become more experienced over the years, like all of us …

#2 Yossi Kreinin on 04.19.13 at 10:46 am

Well if I accept your compliment then my example was surely in place, because avoiding wasted effort is much of the difference between then and now. More generally nobody can change innate aptitude (to the extent that aptitude is innate…), but you can change your attitude.

So yeah, there's more to it and I admitted that; I focus on this one thing because it contributes a lot and it can be changed. What kind of things you come up with might be innate or very "person-specific" somehow - I don't know; I know for sure that there are people with seemingly similar aptitudes, working on roughly equally hard stuff, except that one's stuff gets used and the other's much less.

Regarding courage to fail - sure, when it's hard and one can fail through the "fault" of one's own; but less so when circumstantially the thing looks like having poor chances even if you do a perfect job. In particular, I very easily see you taking the first type of risk; the second type, much less likely, not?

#3 Karl McFiff on 04.19.13 at 7:08 pm

That is why the most productive programmers use template based code generators. The programmer just has to click the boxes to specify the features needed and then finally click the Generate Code button and the generator creates the code for you.

All you need to do is connect the output from one code generator to the input from another.

#4 Yossi Kreinin on 04.19.13 at 11:21 pm

You ought to be kidding.

#5 John M on 04.19.13 at 11:26 pm

From my experience - a lot of coders end up writing 10x more code to do the same work than the 10x coders.

Often because they don't seem to step back and understand things and just throw more code at it until it "works"

#6 Michael Moser on 04.19.13 at 11:29 pm

Your notion of success/measure of selectivity will vary depending on the context of your work and on your position within an organization. I see the following contexts for programming work:
 
1) employee in a shop that works on 'product'
2) employee in a shop that works on 'site/service' / 'customer project'
3) independent consultant
4) work on open source project

working on 'products' values more such things as 'completeness' at the expense of 'efficiency'.
Working on 'service' your work must reasonably fulfill the customers use cases; most often at the expense of 'completeness'; As a consultant you might try to minimize the effort that has to be put into a task, while  completing the customers spec.
 
So what ever you do, you can be as selective as the realities around you allow you to be. Most freedom
you have with your own pet projects, that is; but how to achieve some wider impact so far has been beyond me ;-)
 
 

#7 Yossi Kreinin on 04.20.13 at 12:31 am

@John M: a part of it is seeing code as a good thing, so that more must be better; let's add this and let's add that. A special case of gladly doing needless work under the assumption that hard work is always a good thing.

@Michael Moser: I think you always want to minimize the effort you put into a task (including not doing it altogether because it can be avoided). It's not different for consultants and full-time employees, except if the employee has a manager that insists on getting needless work done, which is perhaps more likely than someone hiring a consultant and paying his own money insisting on getting needless work done.

#8 Assaf Lavie on 04.20.13 at 2:10 am

An interesting, thought provoking post.

I think this has to do with real programming craftsmanship being more like an art than a science. A seasoned programmer will know, almost intuitively, what efforts are worth pursuing. What doesn't need to be reinvented. What can be tolerated by "it works.." and what has to be avoided at all costs. The best artist isn't the one who paints at the greatest speed or with the highest accuracy - he's the one that does the most meaningful work.

Being 10x productive, I agree, has less to do with the speed at which a given task can be accomplished, and more with _which_ tasks get accomplished.
This, btw, is exactly the reason why every single software project does't finish on time. People love to focus on how _fast_ things get done (estimates, performance) and they neglect the fact that _what_ actually gets done is the tricky part. All that stuff you didn't realize you were going to do as part of the project, and all the stuff you ended up doing that ultimately went to the trash bin. That's why being truly productive often just means being more careful, considering things a little longer, planning ahead a bit more before diving in.

#9 Yossi Kreinin on 04.20.13 at 3:07 am

I think "project" is a red flag, by itself; I can tell as someone who underwent project management training by the Project Management Institute, qualifying me to attempt to pass an exam and become a certified Project Management Professional - PMP ™…

A project is basically something one-off with a rather precise ROI estimation, and that ROI is not a whole lot of different from what you'd get by lending money to someone else. So with "projects" if you're a bit late then you're probably losing money.

The decent thing to work on is products, where there's no "final" delivery but rather things are incrementally improved, and where the thing might be a big-enough hit to more than cover very large development expenses.

#10 Marcel Popescu on 04.22.13 at 6:22 am

I have to agree with John. In one of the projects I'm working on, to use a single example, I have a 68 line udf in the database that does the equivalent of this:

SELECT * FROM [table]

Seriously. It has comments, with nice stars and well aligned fields that nobody will ever care about. It has beautifully aligned field names. It has… stuff.

(And no, we don't have a DBA. We programmers have full access to everything we want in the project - we can redesign both the code and the database at will. For some reason, this programmer thought a 68-line UDF was just what was needed for this.)

#11 Marcel Popescu on 04.22.13 at 6:36 am

I just remembered another example - I can't even decide if it's worse than the previous one. I've seen a programmer write a 3-page switch to convert a string to its Enum equivalent (I think it was a list of country codes). I can understand not knowing about Enum.Parse, but I can't for the life of me understand *not asking*. After you write the same thing for the 10th time, shouldn't you figure out that there has to be a better way to do this?

#12 Yossi Kreinin on 04.22.13 at 12:07 pm

I saw a look-up table where table[i] gave you 1<

Some people don't think of verbosity as a problem; on the contrary, work is being done. Hence the non-asking.

BTW, it's still much less waste than my story above there, and I certainly wouldn't have write boilerplate like in these smaller examples back then or ever. This goes to show that the magnitude of the ultimate failure isn't a simple function of what we think of as "raw aptitude/competence".

#13 Yossi Kreinin on 04.22.13 at 12:08 pm

…table[i] gave you 1<<i, I meant to say but WordPress wouldn't let me get away so easily.

#14 Bob on 04.24.13 at 5:43 am

I do think selectivity and negotiation have a huge impact in productivity. Being able to spot a rabbit hole before you start. Offer alternatives and give the customer the 80% solution that still solves their problem. But that usually comes with good logic skills and falling in the rabbit holes enough to know when they're coming.

#15 Albert1 on 04.26.13 at 1:27 am

Oh, don't mind me: I was "finally" studying the new C++11 features when I suddenly felt the need to scream but I didn't want to alert other people - then, after an epiphany(?), I found myself here…

#16 Yossi Kreinin on 04.26.13 at 1:58 am

I actually suggest to everyone to upgrade to C++11 from C++98/03; it sure has a ton of horrible stuff in it, but a few things making it worth the trouble - notably auto, "smart for", lambdas and initializer lists.

#17 Yuriy Zhilovets on 05.01.13 at 3:22 am

Russian translation: http://habrahabr.ru/post/178553/

#18 Yossi Kreinin on 05.01.13 at 11:45 am

I'd gladly write a Russian version myself if you asked for it, at least as a first draft :) While my Russian is admittedly not very well developed when it comes to programming terminology/slang (I say that programs "run", while Russians say that they "walk"/"go", etc.), I think it could come out nicely if edited by a true Russian programmer from Russia.

#19 Jacob on 05.02.13 at 9:47 am

@gd Failure and "failing fast" to iterate toward success are not quite the same thing. The later might better be called 'testing', 'iteration' or my favorite Succession. See: https://www.facebook.com/notes/facebook-engineering/software-design-glossary/10150309412413920

I've found some of the most successful engineers I've known to employ both your suggestions and Yossi's. Part of being good and not doing shit work - is figuring out what is shit work. This is hard, and beyond experience, the best way to do that is paradoxically doing shit work and failing fast. But the key here is to fail fast and and to use data to guide decisions. Being smart about choosing to do the easy things that you have some evidence of having the most impact and then iterating or switching manages the risk (and cost) of failure which means you may just end up achieving success.

#20 Yuriy Zhilovets on 05.06.13 at 2:28 am

You have such a colorful English that I cannot resist temptation to translate it myself.
For the next articles, I will gladly edit your drafts in Russian trying as much as I can to preserve your style

#21 Michael Moser on 05.19.13 at 11:05 pm

Hi Yossi,

Did you implement your ideas in practice? What where the results? I long for a sequel to this article.

#22 Yossi Kreinin on 05.20.13 at 12:48 am

Well, I don't know - do you expect me to say that I'm now 10x more productive than before or than someone else?.. Yeah, I think I got less gullible over the years, I think it helped… But others should be the judges…

#23 Unproductive Programmer on 06.18.13 at 9:48 am

Although I am still inexperienced, I thought that I should add my two cents. Interning at a company at the moment, I feel that I am definitely doing "crap work", (but difficult much like your example).

However, reading this article reminded me of a fellow programmer who never re-invents the wheel and gets a lot done. The only problem is that he has the creativity of a plank.

I think that solving difficult problems, and solving problems which have yet to be uncovered (these are usually revolutionary) requires a lot of useless work before an epiphany.

#24 Yossi Kreinin on 06.19.13 at 12:18 am

Well, it depends on context; an experienced programmer easily tempted into doing shit work or not protesting when ordered to do it has a real weakness, someone just starting out is a bit different.

There is in fact some tension between efficiency and creativity, but efficiency can also be a force multiplier, to some extent, if it makes you focus on those hard things which actually matter.

#25 joseph on 09.18.13 at 12:07 am

I'm a 1x, and I think it's at least in part due to the fact that my main motivation is to have fun rather than to get shit done. It's fun to learn a new language, or write a new tool, or read this blog, and it can kind of be considered productive, so it gets rationalized. I'm hoping that at some point I'll reach a level of maturity where my priorities will change.

#26 Yossi Kreinin on 09.18.13 at 7:29 am

Well, I think at some point getting shit done becomes the fun; as in, the first several languages arouse more curiosity than the next, etc., and at some point it's not as much fun to learn a new language as it is to get this particular piece of shit done already, and in fact people start to seek ways to learn as little new info as necessary just because so much of the info they learned in the past turned out perfectly useless either immediately or in the long run.

Leave a Comment