The internal free market

This is going to be a bit atypical, because I'm going to talk, like, about organizing large teams of programmers. Which I rarely do, for the simple reason that it's not my problem. I'm not a manager, I don't think I'm likely to do a particularly good job as a manager in the near future, and I don't want to be a manager. As far as I'm concerned - if your problem is organizing lots of people, you brought it upon yourself. So this "internal free market" thing, which tends to work well according to my observations, is an exception to my general rule of not making or thinking much about "organizational" observations.

So, free markets. Basically a way to create incentives (because you have to compete) at the cost of redundancy (because of duplicated efforts by many competitors). A redundancy vs dependencies issue, if you like - several competitors means less dependence on each - and since I generally think redundancy is a fair price to pay for removing dependencies, you can guess that I'm leaning towards free market fundamentalism.

At this stage I'm skipping the detours where I'm dragging exciting pseudo-related stuff into this, like the subprime mortgage crisis and the enigmatically overwhelming support for Barack Obama by top programmers and tech bloggers. I'm skipping that to get to the simple point of there being no good way for an employer to create incentives for programmers with money.

How exactly this trick fails to work, and what kind of LOCs you get when you pay by the LOC is out of my scope. If you're sincerely surprised, there's lots of material for you to browse - Econ 101 Management being as good a place to start as any. The single thing I have to say about the "financial incentive" method here is that its failure isn't at all surprising to the free market fundamentalist.

In a free market, people solve their own problems, and (pay to) use the output of other people only when it helps them solve their problems, in a completely distributed way. Setting prices for things and subsidizing them is the trademark of a centralized, controlled economy. Now, does an employer paying by the LOC or by 1/#defects or whatever create an "internal free market", or an "internal government bureaucracy"? Without looking at soft stuff, like a hypothetical offense to sacred engineering values carried by the act of creating incentives, we can see that subsidizing LOCs will create surplus LOCs, just the way it works with agricultural surplus and everywhere else.

What would we do if we wanted a real internal free market? "Free market" means that we want to have people solving their problems, and let them "pay" each other in some way to solve them, without us controlling the latter process. In our specific context:

  1. "People" means "strong programmers" (or at least decent - or else why did "we", the employer, hire them, dammit?) Folks who, at the very least, like to be productive and have their stuff used. Maybe they also "like to solve puzzles", but not all do. For example, I hate puzzles, and have a strong preference for Alexandrian solutions. But you should still hire me.
  2. For those people, "solving their problems" means delivering user-visible features. This is the basic responsibility of developers towards the organization, this is what the organization is capable of judging (it better be), and this is what links the whole operation to reality through the external market forces.

The only question is, what does it mean for a developer to "pay" another developer? Paying with money makes no sense, not with our definition of "people". People who're into those transactions tend to be self-employed. However, developers do have their own currency, karma points or whatever term you prefer to use for it (they are all irksome; economics is to life what proctology is to anatomy - it's ugly because it's true). I know two kinds:

  1. Time. You can "pay" to developers, teams and the whole organization by volunteering to do particularly unsatisfying but important/urgent work.
  2. Code. When someone uses your code, they are paying you (I repeat, it's not you who are giving something to them; do not make this error or they'll stop using your code.)

Time and code are not unlike gold and printed money, because you can't make more time but you can make more code. However, proceeding with this analogy and trying to scale it to include inflation and such will expose my economical illiteracy and general lunacy to an extent making me want to stop now.

What we'll do now is examine how "trading" time and code works in programming, and how it creates incentives to invest efforts into the most needed things similarly to the way prices do in free market economies. We'll start with "trading code" - the less intuitive but the more fundamental kind of transaction.

You have to deliver something user-visible. The user couldn't care less about the guts making up your program - how you parse things, what your communication protocols are, which optimized libraries you use for processing bottlenecks, etc. However, you do care about these things, because they are really needed for things to work. With all these things, you could reuse "infrastructure" others work on (for example, me), or you could roll your own (let's ignore "international trade" where you use a third-party library for the moment).

To you, depending on me is a risk - who knows how many bugs I have, how well my stuff maps to your needs, etc. To me, on the other hand, having you reuse my code is the best thing that can happen to me during work hours. After work, better things can happen, in particular, those having to do with spending the money earned during work hours. But at work, the best thing that I can do is be productive and have others use my code. Lots of users is the workplace fortune equivalent to being rich in the real world. Do you see who pays who here?

What can an organization do to manage these infrastructure transactions?

  1. The "economical", "capitalist" solution: leave them alone except for securing them. "Leaving them alone" means not controlling them - not mandating the development and reuse of infrastructure and not assigning workforce to it. This means that by making my modules reusable, I'm only trying to please my internal users, so I'm likely to (try to) invest most effort into what they find important and helpful for doing their ultimate job. "Securing transactions" means something similar to the way public companies are forced to expose their accounting. If something becomes reusable code, it ought to have proper documentation, versioning, etc., and the organization must make sure it does.
  2. The "political", "socialist" solution: assign the task of developing a parser, an optimized library, etc. to a person/team - subsidize the parser (the price to a user is now lower - even if the parser is all buggy, a person is officially assigned to fix the bugs, and the responsibility for failures moves to that person and not to the one who decided to reuse the code). This means that the parser will be created even if in a "free company" nobody would want to develop and maintain it, knowing that most people wouldn't take the risk of using it for the benefits it provides. Leading to surplus crops of parsers.
  3. A further improvement on 2, the "communist" solution: force everyone to use The Parser. This means there are no "economical" means to punish the author whatsoever - where "punishing" means "not paying" and "not paying" means "stop using the code". However, there's still hope: you have political means to punish the author. For example, poke fun at the goddamn nightmare infrastructure, yell at the author, yell at his manager, ask your manager to yell at his manager's manager - a whole slew of counter-infrastructure measures. Victims of infrastructural communism use them all the time.

So this is how "trading code" is (more accurately, "can be") a better way of evolving reusable "infrastructure" than centralized planning. In general, the only thing I'm discussing is the reusable stuff - that's what organizations can optimize (or pessimize, creating useless "reusable" modules and not creating the actually needed ones). Nothing can be done about things which aren't reusable by definition, belonging to a single "feature"/"project" - those will have to be written once and only once no matter what.

What's wrong with this picture? Could be many things, but one thing I'll talk about (because I have a good answer for it) is the problem of "instant gratification"/"disruptive changes"/"local optimums"/etc. There are grand things that just can't be done by small incremental changes, the by-products of work on "specific features". You really need a person/team assigned to these things. This is somewhat similar to economies of scale which can be achieved by purchasing expensive machinery. How are many small farmers/shoe makers/etc. going to raise money for that machinery without central planning, if they're all busy with their small short-term profits?

This is where entrepreneurs come into play. Entrepreneurs are people with fire up their asses. Normal people want enough money to get by, enough money to not worry about money, or enough money to not have to work for money. Entrepreneurs want more money than they can sensibly spend during the decades of their lifetime. And they want it because they desperately need that money to feed the fire raging up their asses. When they see a potential for making truckloads of money, many of them are willing to put their own savings on the line to chase that chance.

This psychological profile is a speculation of mine - my best attempt to comprehend the inexplicable behavior of making efforts and burning nerve endings to make more money than you could possibly need. However, I do have motivation which is quite similar in the context of our "economics of programming" analogy. I'm a "programming entrepreneur", or at least I have, um, the same trademark proctological fireworks. I'm thrilled by opportunities to make stuff that, like, everybody will use, everything will depend on, …and everyone will want a piece of me when it breaks - so? It's still worth it.

I can't make such stuff as a by-product of working on something reasonably user-visible. I need to be assigned to it. What are the savings that I can put on the line? Time invested into doing unsatisfying, but important work. I call my own way of making these deals "buying development with debugging". I'm usually willing to debug the weirder of the urgent problems, although it's not much fun by itself, because it translates to a lot of karma points. I can then spend those karma points by working on what I want 80% of the time, 20% being the continuous urgent debugging tax.

Again, there's more than one way for that kind of "entrepreneur" to start a programming venture:

  1. The "economical" way - spend my own time implementing my ideas. Like a "real" entrepreneur putting his savings on the line and forced to look at his company bleeding that money if it doesn't take off, I will want to stop as soon as possible when I realize that I'm failing. Those so-called "organizational karma points" you gain in the trenches have better uses than wasting on the development of worthless stuff nobody will use.
  2. The "political" way - convincing "the government" (a manager) that my idea is worth implementing, and have someone assigned to it. Now nobody wants to admit the failure early on. I'm not losing anything when someone else struggles with the implementation - "I could do it better". The person working on the thing isn't really held responsible for the failure, either - not his idea, so why not keep trying to make it work? Everybody wants to make his stuff work and be used, after all. And the manager won't want to admit the failure because of all people, he'll get most of the blame. Therefore, the worthless effort will not be stopped for a lot of time.

Free market supporters are sometimes blamed for disrespecting people and reducing human nature to primitive egoism. Well, the only thing I can say is that I sure am a Good Person (how could it be different?), I respect myself lots, I successfully "launched" more than one "programming venture" both ways - "economical" (DIY) and "political" (persuasion), and of each of these two kinds, some succeeded and some failed.

And believe you me, deep down I refuse to take responsibility for the failing "politically launched" projects even now when we talk about it. On the other hand, the "economically launched" failures are - seriously - the best thing that happened to me in my professional life. I attribute most of my occasional successes - or, more accurately, non-failures - to lessons learned from the DIY failures, which I had no choice but admit responsibility for. (Damn, that was painful. To the extent that wasn't on my job description.)

Now, I'm not an "internal free market fundamentalist", simply because I know much more about programming than I do about economics, and obnoxious/oversimplified opinions usually correlate with ignorance. However, my experience seems to show that "internal free markets" are healthy enough to sustain continuous improvements on many scales, and eventually punish both "greedy" "instant gratification" techniques of pleasing managers/customers and architectural masturbation, promoting solid work.

And if you're not a manager (I mostly care about non-managers, guys and gals like me, you know), I think this quasi-economical angle can contribute to your ability to look at some young initiative around you and say "Hm, this might work out" and conversely "Epic fail on the way, I'm not going to touch this with a laser pointer, man". So, FYI.

17 comments ↓

#1 Zung! on 11.25.08 at 12:34 am

This is strange. I couldn't help thinking about the external free market, while reading your definitions of solution types:

1. The “economical”, “capitalist” solution - this description seems to match free software…
2. The “political”, “socialist” solution - a software product for sale
3. The “communist” solution: a monopoly

#2 Harr on 11.25.08 at 1:18 pm

Rewrite it; cut it down to half or so, then maybe it'll be focused and clear enough for me to understand.

#3 Harr on 11.25.08 at 1:26 pm

… sorry, I don't know what got into me for the tone there.

#4 TNO on 11.25.08 at 10:13 pm

Hi,
I think many things doesn't work the way you've described
(which is a nice way to tell that you're entirely wrong ;-)
Let's look at the article: first few paragraphs tell us that
the people want to have a high paid jobs and to do the work they like.
Well, not everyone can acheve this obvious goal for various reasons:
they're not competent enough, their lovely job is underpaid or something else.
Now, you're confining your observation to the small group of people - highly proficient
workers, that love their job and not afraid to be fired. So, within this small group of people
free market exist, you say. But it's not true: what you've describing is not a free market.
Frankly enough, it resembles communism - not the soviet one, but communism as it should be
(if such a thing exists), i.e.people zealously giving their skill and time on
the work they like, and instead of money (capitalism) they're give some abstract idea,
or nothing, but satisafction that their job is done well(communism).
Time and code is not money (you're wrong at this point too). Partly, beacuse it is a basic
resource that people have and it can't be replenished.
The other reason is that there's ouside world DO exist and you can't buy a bread and butter
by giving some LOC to the teller.
So, basically, I may conclude that communism may be achieved by a small group of talented people,
which are not afraid of the regime (or regime favours them). But this is not new : this happened even in
Soviet Union. Take for example Arzamas-16, Dubna or other cities which had brilliant scientists live in.
I don't think that the fear of the regime is their main motive for launching people
into the space or exploring a nucleus. They were in favor of the goverment and did job they like.
Idon't think that they did the job for the sake of the Lenin's Prize alone.
Just like decent programmer doesn't work for a money alone (as you've mentioned).
I could say more but I have to run to my high-paid job which I like because I'm a decent programmer.
And people's respect to me based on what I am and how well I'm doing my job, instead
of amount of money I have.
Have a nice day for now.

#5 Adam on 11.26.08 at 12:20 am

This is a great post. I was recently wondering why most great programmers and even great managers embrace the external free market, as an earlier commenter put it, but most have not tried internalizing this kind of self-organized system.

Great work.

#6 dshack on 11.26.08 at 10:09 am

Reminds me a little of internal prediction markets, using monetary incentives to crowdsource actual sentiments about projects and company direction. Google and Intel have both been doing it for a few years now.

#7 Yossi Kreinin on 11.26.08 at 12:42 pm

@Adam: thanks!

@dshack: there's a similarity, I guess. However, I don't see a good way to provide monetary incentives in the area of internal code as opposed to user-visible projects, because it's unclear how to rate the success of "infrastructure" (even the importance of individual user-visible features of end-user projects is hard to estimate numerically), and without reliable measurement, a "real" price system is hard to set up. Which is why I think there's room for a non-monetary "free market" within the developers that isn't controlled by the organization, only "moderated".

@TNO: I don't quite understand what you're saying, but: (1) yes, I said explicitly that I only talk about the stronger and more motivated programmers, and I think they are the, um, more interesting group - but the vast majority of the weaker and even awful programmers also care lots about having their stuff used, so it's a universal incentive; look at people pitching their code to their colleagues some time. (2) I didn't say time and code "were" money, I said they were somewhat analogous to it in the context I set up. (3) I didn't talk about convertibility of your reputation in the "internal free market" to actual money; at some places it happens and at others it doesn't. However, the basic fact of an employed programmer's life remains the same - you spend X hours at work, and get paid more or less the same money no matter what (at least at the short term), so the game is not and can not be money; I claim that the game is getting your code used. This works even in organizations that actually punish productivity, just the way "real" (monetary) economical forces work everywhere including regimes punishing traders, etc.

#8 Frank Hirsch on 11.26.08 at 4:44 pm

Hi!
Like your posting, especially the style. Actually it could be much cleaner, but I tend to ramble in much the same kind of way. /me is amused… :)

#9 links for 2008-11-26 « My Weblog on 11.26.08 at 8:04 pm

[...] The internal free market (tags: economics psychology) [...]

#10 anon on 12.09.08 at 4:14 am

tl;dr

#11 ray on 01.05.09 at 3:53 am

I really love your description of entrepreneurs as having "fire up their asses".

#12 Yossi Kreinin on 01.05.09 at 11:13 am

@ray: guess it's because none of us is one of them…

Come on, baby, light my fire!

#13 Steve on 10.07.09 at 9:35 am

I generally agree, but one thing irked me while reading: it's only sort of true that someone is paying you when they use your code. It actually goes both ways: when you provide code for other people to use, you're also implicitly giving them a promise that you will maintain it. This is not a small thing! (If you don't think that's true, you will find out it is when people start using your code and then start asking for changes.)

So, having things go both ways—you give someone something by providing code and maintenance for that code, you are rewarded by having people use your code—breaks down the analogy with money to a certain extent.

As a programmer, I've found that the better a job I do, the more interested people are to use my work, which is great– but the hidden corollary is that I am now expected to keep supporting that work, in effect, forever, as long as no one steps up to take over. If I want people to continue using it (and generally I do), it becomes a Sissifian task. Not only that, but if at some point you decide to stop cold turkey, and people are still using your code, you sometimes get the negative reward of being considered a bit of an asshole. ("Common, it's just a small feature/bug/update, it'll only take an hour, can't you just take care of it over the weekend?")

Now, this problem can and needs to be nipped in the bud—the first time someone asks for a "small feature", you have to say, "well, give me a patch and I'll look it over." This can help a lot, and avoid the snowball effect, but you'll still find yourself in a position of responsibility that you may not have expected from a one-off project you decided to pursue when you had a bit of free time.

Any thoughts on how this analogy with the free market might relate to this problem? I suppose in the real world, regulations say that a product must be guaranteed for 30 days (or something like that), and it's usually possible to purchase longer guarantees optionally. Perhaps this idea can be translated to code maintenance, but I'm not sure. (How would they "purchase" their maintenance guarantee, for example?)

#14 Yossi Kreinin on 10.08.09 at 1:01 pm

Generally it was more important to me to make some points which I consider sufficiently counter-intuitive to a notable share of people - for example, that you can have what amounts to a "market" without either money or businessy talk/attitudes, or that giving your code to someone isn't doing them a favor (or at least isn't just - than it was to make the analogy complete.

Now, regarding the maintenance issue.

What I think usually happens when one is saturated by supporting one's code is that people are tasked to help with the support; frequently the saturated person thus becomes a manager (heuristically considered an upward motion though in fact it can be more of a PITA), and/or gets a larger paycheck matching the larger responsibility, thus finally connecting the internal code market with the external monetary market. Alternatively one can refuse to support some of the code but give the responsibility away to whoever wishes to take over; if the code is useful enough, someone usually does wish to take over because of the tendency mentioned above. Alternatively one can announce that he's no longer responsible for the code but not really relinquish the responsibility, getting in the way of whoever is now nominally in charge; this will give at least one of the two a bad reputation, hopefully only to the first one unless he's much more skilled politically.

In terms of the original analogy - when people use your code, they actually pay continuously - one large payment the first time they started using it, binding themselves to you (the way it happens when they buy a car), and then many small payments each time they need features (the way it happens when they buy replacement parts for the car or upgrade the sound system). If they are burnt by the code right after starting using it, they can stop doing so cheaply the way you can return a defective car and get a refund; later they can't stop using the code cheaply (too much code for them to change and functionality to get somewhere else) so they're locked in, the way one can't easily get rid of a defective car once they've used it for some time - selling it would amount to a monetary loss and so will investing into fixing it.

The reason both the initial decision to use your code and the continuous requests for support pay you is that it increases your responsibility, which, in a healthy organization, is linked pretty directly with the official currency of the state where it operates. For them it's better to implement everything themselves to get the most responsibility just like you or me would happily assemble our own car or build our own house if it were possible for us to be good enough at that - why spend money? We spend money, and they use your code, because it's ultimately cheaper and they're thus better off - but still you give money when you get a car, and you give away responsibility plus get in the risk of depending on someone for your success when you use someone's code.

Of course if the organization is dysfunctional enough to not reward people for their increasing responsibility, this is no longer that simple and I think this is why I didn't mention the whole business of such rewards in the text. The thing is, even in an apathetic or even actively moronic organization punishing initiative from below, there's still a psychic gain from having your stuff used - this is why you'll still tend to fight the organization and keep maintaining your code used by others, sometimes on your time - I've seen it happen. I think this is an internal black market - the code market is punished by the employer paying the real money, somewhat similarly to the way a state can punish some voluntary transactions with real money. But the black market is very rarely completely exterminated that way, it just doesn't operate as well as in the cases when nobody gets in its way.

Sounds like a fuller analogy this time?

#15 Steve on 10.12.09 at 1:33 pm

Thanks, actually that is a great explanation. I'll continue to think on it. :)

#16 Vlad Patryshev on 01.09.11 at 10:07 pm

Amazingly good reading. Thanks a lot! A very deep insight.

(Otoh, ever tried to apply this to family life? :)

#17 Yossi Kreinin on 01.10.11 at 1:22 am

Very glad you liked it!

As to family life - I guess Becker did, with a bunch of other so-called economic imperialists… Usually this stuff exceeds my tolerance for creepiness, with the exception of David Friedman's writing.

Leave a Comment