August 7, 2000

12:26 PM

Ok, here comes the firestorm. Joel on Software has some very good things to say -- though, like most user-interface-design mavens, I think about 50% of the time that he hasn't comprehended what the problem really is... but in this piece, he's wrong.
posted by baylink (17 comments total)

The topic is whether it's *ever* acceptable to do a ground-up rewrite. "Like software *rusts*, or something", he scoffs.

Sorry, Joel; it does.

There comes a point where you no longer gain anything by patching over bugs. Certainly there's lots of institutional knowledge and memory tied up in those bug fixes -- but keeping unmaintainable, inextensible code to preserve it is not the solution. Using a revision control system that allows commentary on the problem solved and the theory behind the solution is the answer.

What? You didn't do that the first time? Bummer.

Especially at today's cusp, the quality of the programmer support tools available is a couple orders of magnitude better than it was when your code was written... and since you're in trouble already or you wouldn't be thinking about a rewrite, you need all the help you can get.

Plus, you have all that accumulated knowledge extending your understand of the problem domain to take advantage of, so when you write your design document (you *did* make one of those, right?), it will more closely match reality.

As far as *I* am concerned, unless you *do* make changes not too less extensive than a total rewrite, you aren't even entitled to go from 1.7 to 2.0.

x.y.z

patches go in z. new features (or clusters thereof) go in y. x doesn't change unless the program changes so massively that it's effectively a new program -- major new pieces of problem domain or a rewrite are really the only acceptable reasons, to me.

This used to be a standard. Version numbers are almost useless these days.

I used to be able to say "except on free software"... until sendmail 8 shipped. <sigh>

Nope, Joel; I'm sorry. I've *done* both. And the rewrites are cheaper, faster, and more reliable.
posted by baylink at 12:36 PM on August 7, 2000

Err... did you rewrite every library that your program called? every underlying API? If you did, and your rewrite still worked out OK, then I am very impressed.

However, I do agree with the article: massive rewrites on large pieces of engineering work (that being software, cars or planes) are a bad, bad idea. Witness the two browsers: essentially they started off from the same code-base: Marc Andreesen left NCSA with: a) either an extremely good knowledge of the Mosaic code, as he himself admits, or b) with big chunks of the codebase itself and started Netscape. Spyglass later licensed Mosaic off NCSA and later licensed the codebase to MS, which eventually became IE 3.3.

In other words: Mozilla is now a complete rewrite of the NCSA-based Navigator. IE 5 is essentially a complete rewite of the NCSA-based Spyglass. The difference is, MS did the rewrite the HARD way, i.e. one small part at a time, cleaning up code and improving performance here and there, whereas Mozilla.org wimped out and went the EASY way of the total rewrite. Which one is the better browser? IE 5 or NS 4.74? IE 5 or Mozilla M17(?)/Nightly?

Now, I know that me saying Mozilla wimped out will be interpreted as flamebait, but it really isnt. When faced with 4 year old code that is basically layer upon layer of cruft, the easy way to do it is to sweep the table clean and start again, learning from your past design mistakes and/or insights. The hard way is to incrementally improve the code, one krufty function at a time and change the design from the bottom up, rather than from the top down...

Witness the other side of the Web wars: Apache is (now) a complete rewrite of NCSA httpd; but it was an *incremental* rewrite of that beast (hence the "A patchy server" wordplay), whereas IIS was a new codebase. Apache is understandably winning that war (and so is Netscape/iPlanet too).

I like Mozilla and I respect their work, and if their browser ever becomes competitive (I am sure Gecko will survive, but I am not so sure of the rest of the code) I might even use it, but as an engineering effort --i.e. one that has timelines and deliverables to produce-- they have failed.

posted by costas at 1:55 PM on August 7, 2000

You are evaluating the project by your goals, Costas, not the *project's* goals. Those goals are publicly posted, and the project team isn't at all unhappy, I understand, with where they stand relative to them.

You're welcome to say "They're therefore not relevant *to me*".

You're even welcome to say "I don't think they're relevant in general anymore", though there are those of us who will disagree with you (when MS tanks in 5 years, *I* don't want to be the guy who bet my company on their browser).

It is not your place, however, to say they have *failed*.

And no, in the rewrites I've mentioned, I didn't rewrite the OS, or the DBMS, or the 4GL. Just the application. It had already been patched to death.

Yeah, maybe it's the HARD way on the *programmers* to rewrite it all at once. But the other way is HARD on the *paying customers*. Maybe I'm missing something. But I thought that was why we paid the programmers the big bucks.
posted by baylink at 2:01 PM on August 7, 2000

OK, you're right on the failure part: I am outside of the project, I have no contract with Mozilla/AOL and thus I cannot call Mozilla a failure. At least I can say that they have failed me as an ex-Netscape user and former anti-MS bigot.

However, I do not agree that an incremental rewrite is hard on the paying customers. Actually, most of the time it is the customers that force you into an incremental rewrite --and most of the time they are right (I work for an ERP firm, so I talk to our customers all the time).

posted by costas at 2:22 PM on August 7, 2000

baylink: if the Mozilla project's goals include posing a viable alternative to MSIE, they are going about it the wrong way. If the Mozilla project's goal is merely to create a cool browser the authors and a few tens of thousands of their friends can play with, they're succeeding - but that's not what they SAY they're doing.

Have they failed? Well, they sure have failed *me*. This note is composed in MSIE 5.0; after years of swearing off Microsoft products, I find that their offering is the only real choice. As MSIE has something like 75% market share now, I don't think I'm the only one who feels this way.

And you got it backwards: the hard way, for programmers, is to rewrite it one step at a time. Throwing the whole thing away and rewriting from scratch is the lazy person's way out - anybody can do that. Rewriting in place, without breaking the system, without taking five years between versions - that's hard. It takes skill, determination, careful planning, and a strong stomach (to put up with all the crap you can't fix just yet).

-Mars
posted by Mars Saxman at 2:33 PM on August 7, 2000

(Well this is written from Opera 4.12. IE makes Windows unstable so I can't be bothered using it no more; I was having six month reinstalls to keep the thing ticking until I jumped ship).

Netscape 4 seemed very inconsistent in the way it handled DHTML. CSS would occasionally break it (N4 could move text but not underlined text - ok, not that, but something equally stupid).

Mozilla is turning out quite nice. There are a few gecko embeders around that get the speed back now.

I'm not glad Mozilla decided to ignore native UIs, though.
posted by holloway at 11:05 PM on August 7, 2000

It is not your place, however, to say they have *failed*.

Mozilla advocates are spending all of their time these days redefining the meaning of the terms "succcess" and "failure."

Do we need any more proof that the project is a collosal disaster?
posted by rcade at 12:00 AM on August 8, 2000

baylink, what Joel says seems absolutely correct to me. As the leader of a programming team I have quite a range of programmers working for me, the more junior programmers want to trash code they're working on and get on with re-writing it from the ground up. Why? Because a)They think the code is cumbersome, but don't realise that the reason it's so cumbersome is that it's been tried, tested and fixed over several years of real-world use. b)They are too eager to get on and code, instead of sitting back and looking at what they already have, the little nuances and tricks used by the original programmers to circumvent potential problems.
In my experience, it's a pride thing - less experienced programmers, fresh out of college thinkk that they have been taught all the latest techniques (I was just as guilty myself, ten years ago) and can re-write to a higher standard. More experienced programmers have learned (usually the hard way) that *most* code has evolved and has a good reason for looking like it does, often all it requires is a bit of rationalisation - amalgamating duplicate code into a library routine for instance.
What Joel appears to be saying is that there will be cases where some code requires re-writing, but you'd better be damn sure you've covered all your bases before you dump what you've got to start again.
posted by Markb at 2:10 AM on August 8, 2000

I don't have any problem, Mark, with your appraisal of the situation, and I'm not encouraging *gratuitous* rewriting.

But Joel is overruling *ever* rewriting *anything*, from what I can see, and I'm sorry, but the real world just doesn't work that way. There are instances when the target just is *not* attainable from the currently available pieces, no matter *how* long you allocate for patching them.

And it really doesn't matter to me whether you rewrite a piece at a time, or all at once... though in some cases, you haven't the former luxury.

If you rewrite the whole thing, you rewrote the whole thing, no? FWIW, Fred Brooks and Jon Bentley both seem to agree that there comes a time when your patching work actually has a negative investment in functionality, not just functionality vs time. And for that matter, both of you who take issue seem only to be taking issue with trying to do it all at once.

No, it may well not be a good idea to do it all at once. But there are other considerations. Training is one, deployment speed another. The browser replacement cycle *now* is said to be almost 2 years, for non-geeks. That pokes a bit of a hole in the argument, too, no?

An interesting insight on the topic is also the 18,100 hits on Google on the phrase "complete rewrite". I don't offer any opinion on whose case it makes, though. :-)
posted by baylink at 7:06 AM on August 8, 2000

I don't disagree with your point that after a certain point patching becomes counter-productive, however I don't think that's what Joel meant. I think he was saying that when the time comes to produce version 2.0 or 3.0 or whatever, it's not a good thing - especially when you're up against internet timescales - to ditch what you've got and re-write from the ground up.
I get the impression that what he means is that although it would initially appear as though a total re-write would be the best route, by the time you've ironed out all the bugs your testers can find, your competitors have not only beaten you to the market, but also have a more thoroughly tested offering.
We could both use the 2-year argument, mine would be that if you were to use the code you have, when your deadline approaches you still have a finished product (maybe not what the original brief called for, but hey plenty of companies have done very well by delivering half of what they originally promised...) not a pile of unfinished, bug-ridden procedures waiting to be slapped together with band-aids.

Joel is probably taking a purists line on it, and in the real world there are few companies with the time or resources to *never* re-write anything. But he is correct in saying that there are huge benefits in taking the time to look at code you already have and question why it looks like it does - as he says, it's been out there and has been tested in real-life situations.
posted by Markb at 7:40 AM on August 8, 2000

Well, you did start with "Here comes the firestorm...":-)
posted by Markb at 7:54 AM on August 8, 2000

I did indeed.

Ask around, though. I'm sure you can find a sufficient number of non-partisans who would agree that NS4.7 is just about as far as you can stretch that piece of taffy.

I'm one of them, actually.

I don't use IE because *I don't trust Microsoft*. I'm sure you can't think of any compelling reasons why I should. There are certainly enough documented instances of MS software doing things behind your back.

It's noteworthy here, probably, that I don't use SmartDownload, either.

OTOH, I've heard the (justified) bitching about the poor quality of NS6PR2. For that matter, I'm not yet all that impressed with M17. But I can afford to be patient.

I'm amazed no one's yet taken up the gauntlet on "in 5 years, when Microsoft's out of business"... :-)
posted by baylink at 4:37 PM on August 8, 2000

bay, I read Joel (back in April, when this came out) as saying not so much that one can't ever do a re-write... as much as, every time it's been done to a large commercial package before, it's killed it in the marketplace, and hence it's a bad idea.

Can you name a time when a complete re-write worked, commercially?

Because I go back in my own head to Borland, and Paradox. When Borland bought Ashton-Tate, they owned PC databases. And I'm with Phil Greenspun on this one, databases are the foundation of just about all business apps. For a brief time there, Phillipe Kahn was the most influential man in PC computing. If he had decided to go with OS/2, instead of Win3.0, it would've been all over. Developing Paradox for Win3.0 legitimated Windows for a lot of business users.

Borland convinced A-T to surrender mostly on the strength of Paradox for Windows, which was going to blow everything else at that time away. I saw a preview of that product, at a users' group in Pasadena.

A funny thing happened, though. The product kept being delayed. The reason? While riffling through Ashton-Tate's bag of goodies post-acquisition, Phillipe found a thing called Interbase, which was a database engine that ran on everything from DOS PCs to UNIX mainframes. Phillipe ordered a -- you guessed it -- complete re-write to Paradox to use Interbase.

They lost two years to that.

Two years in which Microsoft not only brought out Access and acquired Fox, but also started the whole Office thing.

Had Phillipe released what he had, and then incrementally implemented Interbase, he'd probably still have a company. Well, that and his lack of ability to delegate... It's a shame, he was Gates' one competent adversary that I've seen over the years.

AOL/Netscape/Mozilla just played out the whole thing, all over again. As did Lotus/Samna/IBM. Or Novell/WordPerfect/Corel.

As to your amazment on the MS out of business in five years thing... You might be able to sell me on the idea that MS will be irrelevant. But IBM, Corel, Borland/Imprise, Novell, etc. etc. seem to indicate that one can keep on chugging along in a business environment, long after a company's relevance fades. Put it to you this way: Have you shorted MSFT stock yet? :)

But tell us why you think MS will be out of business in five years... before you explode by holding it in. :)

posted by aurelian at 1:32 AM on August 9, 2000

Can you name a time when a complete re-write worked, commercially?

Yes. Acrobat 1.0 -> Acrobat 2.0

Was it a 100% rewrite? No, but about 80% of the code was redone and for the better.
posted by plinth at 7:21 AM on August 9, 2000

Thanks, Plinth.

You're right, Hal. Major, commercial packages don't often survive the implementations of complete rewrites by their vendors... but note that I'm singling out the *implementation* there; I don't think it's the rewrite itself that's at fault. There are several major free-ware packages that have been essentially completely rewritten between releases; sendmail coming immediately to mind.

Perhaps we're merely seeing more cracks in the foundation of the current commercial software-production model; cf. SourceXchange, etc.

As far as my amazement, it was merely that I didn't get any takers on that purposely loaded comment. As to why I think they're going to be irrelevant, examine Linux and WINE in the context of Neil Stephenson's (absolutely excellent, and *very* funny) In The Beginning Was The Command Line.

Short version: Microsoft has been an operating system company, and we're right at the back end of when you can make a living on that. And they won't port Office to Linux except at gunpoint. So as soon as Linux can practically run Windows applications (which stage I don't really think it's *quite* at yet, though it's getting damned close), a large portion of Corporate America, who are tired of crashes and maintenance, will likely switch.

And MS will be in a *world* of hurt.

And it *will* happen.
posted by baylink at 11:55 AM on August 9, 2000

Plinth, as far as I can tell, Acrobat is a gussied up printer driver. I mean, useful, sure... But not the kind of product I think we're talking about here. Not only that, but it has no competition, to my knowledge.

Bay, I don't think separating out the implementation from the act is useful. "It's not that anti-gravity can't be done, it's that no one's implemented it the right way..." See the problem? It's basically chicken and egg. Joel and I are blaming the egg, you're blaming the chicken.

As to Microsoft... A quick glance at MS' annual report shows that apps have made more moeny for them than OSes for at least the last three years (1997-99), and probably far longer than that... I seem to recall apps took the lead from OSes around the time of Win3.0 So they're really in the business of selling you the OS so that you'll buy the apps, not the other way around. Hell, Windows was developed to leverage MS' existing app dominance on the Mac platform to Intel. Apps always sell the OS, not the other way around, which is something Amiga, NeXT, IBM (via OS/2), and Linux have never really understood, and Apple only just barely.

I think a Linux port of Office is unlikely... But only because I don't see Linux ever getting to the point where a port of Office would sell enough copies to be profitable. Until such time as productivity app reviews outnumber programming tool reviews in Linux magazines, I don't see that changing. Should Linux ever get that much traction, though, a port of Office is not just likely, but inevitable, especially after MS lets DoJ win and break up the company, never to darken their doorstep again.

Even given the premise that MS generates more "crashes and maintenance" than the alternatives, frankly, "Corporate America" just doesn't care. Remember that most IS staff are in support, after all, and private sector bureaucracies are just as resistant to headcount and budget reductions as public sector bureaucracies... One of the under-reported stories of our time.

But the real problem with your hypothesis can be summed up in three letters: IBM. By your argument, IBM should have gone under years ago. Instead, not only are revenues at record levels, they continue to outpace Microsoft and Intel combined.

The reason can be summed up fairly quickly, in O'Brien's Law: The box is always the cheapest part of IS.

Recently, my employer bought an upgrade to their AS/400. The AS/400 exists pretty much because there was a very specific niche: IS guys too old to move to PCs, but too young to retire. It has gone forth and sold like hotcakes. Why? Because the box is cheap, compared to the labor costs. The real investment is not in the box, but in the salaries of the staff. The box cost maybe $50K. The staff costs $200K-$250K/yr. And who recommends to senior management the box to bought? Ahhhh... :)

This is probably the biggest obstacle to Linux: The cost of labor. We're already in a tight labor market, particularly for IS, most especially for Linux. That means moving to Linux is not only expensive in terms of salaries, it also means completely retraining all of the in-house end users. It means never finding anyone with previous Linux end-user experience. It is, truthfully, an H/R nightmare.

Like you said at the beginning: "...like most user-interface-design mavens, I think about 50% of the time that he hasn't comprehended what the problem really is..." And from a business point-of-view, it doesn't seem you do, either. You appear to think this will be settled on the basis of technical chops. This business has never behaved like that... If it did, the great installed-base religious war would be between Amiga and NeXT.

I'm not saying it's impossible that MS or MS-derived companies will go under within five years. Anything's possible.

But to go under the way you describe is really unlikely.

posted by aurelian at 12:51 AM on August 10, 2000

All I can say on that topic, Hal, is that IT deserves what it gets if it manages an engineering function on terms other than engineering. In IT, and especially internetworking, I've found that cliff to be even steeper than in most forms of endeavour in which you can hide bad engineering under making money.

Corporate America probably doesn't care that it's support employees are overstressed. And they won't, until 20 of them come into work with Uzi's on the same day.

Short version: it *will*, eventually, be settled on the issue of technical competence to do the work, on the part of the programs involved.

How many people and Companies will get Fucked on the way, is another matter.
posted by baylink at 7:01 AM on August 10, 2000