From time to time, the not entirely insane tale of FogBugz and Wasabi resurfaces, sparking off another round of misinformed speculation. Perhaps sparked by a Coding Horror classic from yesterdecade. Now unlike at least 95% of the people commenting, I worked at Fog Creek, on FogBugz, in Wasabi, which I believe makes me twenty times more authoritative than average. There’s not much point in correcting people who are determined to be willfully misinformed, but maybe I can set the record straight for the more neutral minded. Besides, it’s an interesting story (at times).

All of Joel’s posts about Wasabi are quite old, from long before the final chapter. He doesn’t say much about what working with Wasabi was actually like. From time to time, you might see an update in a random forum comment, but no real retrospective. I had the unique experience of getting hired in the middle of the Wasabi .NET transition and stepping into the thick of it right from my first day. Other hires came later, after things had calmed down some, or were there to start and can remember a less crazy time.

In some ways, I imagine this sounds like an apology or defense. True. No such thing as a positive misunderstanding on the internet, so no need to set those people straight.

versions

There are two or three versions of Wasabi, depending on how you count. The first version was a simple ASP to PHP compiler, called Thistle. The input language was not customized. Then it became Wasabi, gained a few more features, and accepted an input language more like ASP++, but still output classic ASP and PHP. Then came Wasabi .NET with a whole new backend. That’s when I joined. Not a whole lot is written about this version except when Joel called it supersonic. Wasabi .NET is the craziest of the variants, bearing the least resemblance to any existing language, but still remarkably like ASP. The point of the exercise, after all, was to avoid changing FogBugz as much as possible.

technical debt

Working on FogBugz changed my perspective on technical debt . I used to believe, as I suspect many do, that it was strictly a bad thing. You took a shortcut because you’re lazy, and then it comes back later to bite you. Count how many times that Wikipedia article uses the word “lack”. As the originator of the term Ward Cunningham explains, that’s off the mark. Buying something with credit doesn’t automatically imply you’re not going to pay your bills.

Instead of thinking of technical debt as yesterday’s work that I failed to do, I think of it as tomorrow’s feature I can have today. You have to pay interest, but in the mean time you’re shipping a product, have a roof over your head, and are keeping the lights on. A much hipper programmer might say something like “you ain’t gonna need it.”

In one sense, Wasabi was a rather substantial payment on the debt we had accumulated. FogBugz was written in a now dead language. Building a compiler extended the life of the product, though of course now we had to pay for the compiler, too. So maybe it was more like a bridge loan, or refinancing. The financial wellbeing of Fog Creek at the time depended on FogBugz, so a total rewrite would have been a terribly risky investment. Even if it costs more in the long run, spreading those payments out over time gets you a lot of stability.

tacking into the wind

This isn’t really about tacking into the wind. Nor about beating to windward, to be more accurate, but it sounds cooler than going where you can go.

Imagine you’re in Savannah, Georgia and you want to go to London, England. You don’t have a map, just a vague sense of direction. Maybe GPS coordinates or a compass. You can’t go in a straight line, at least not without building a boat, because there’s an ocean in the way. But there’s a nice beach leading northeast, which is in the direction you want to go. Off you go. Time passes. You notice you’re not heading directly to your destination, but with each step you take, you are getting closer.

Somewhere around Boston, or maybe Nova Scotia, you finally stop and think about your choices. Maybe this isn’t the way to London. From high above in the peanut gallery, you can hear the cackles. “Hahaha, look at these retards. Can’t tell the difference between England and New England. Get these fools a map.” But that’s just the thing; you didn’t have a map. Maps are made by people who, almost by definition, don’t know where they’re going.

How does this silly story relate to Wasabi? A journey of a thousand miles starts with a single step. From the outside, you only see the finished product, not all the little decisions that led there. Nobody sat down and thought it’d be a bright idea to write a brand new compiler just so we could write a bug tracker with it. (Some people seem rather deliberately confused about the chronology here. The bug tracker came first, by a long shot.)

In the beginning, FogBugz was written in classic ASP because that’s what Joel was familiar with. Because he worked at Microsoft. Because that’s where he got a job after college. Maybe he should have learned a different language, or worked at a different company, or gone to a different college to get a different job offer. The second second second guessing game can go on ad nauseam. But let’s begin our story at the point where FogBugz exists, it’s written in ASP, it’s making money, and Fog Creek wants to make more money by creating a unix version.

The super obvious answer is to tell unix customers to install Chilisoft ASP. Boom. Done. Except it costs more than FogBugz does. Enter Thistle, the Wasabi precursor. This all happened before my time, but it’s all true as far as I know. What’s funny is this article was published a year and a half before the one sentence that exploded Atwood’s head. Wasabi wasn’t some secret bomb that Joel dropped out of nowhere.

From there, one thing led to another. In order to support conditional compilation, some additional syntax had to be added and an ASP backend created. Once that happened, the frontend was no longer technically coupled to ASP and any number of improvements was possible. At the time it wasn’t obvious that FogBugz was hitching its wagon to some bizarro mutant Wasoxen. Every change was moving in the direction of making the product better.

Looking back, maybe Fog Creek should have built itself a boat instead of walking up the beach. Except rule number one was never, ever rewrite code. Was Fog Creek a little too devoted to the no rewrite policy? Arguable, but if your policy is “don’t rewrite, except when you do,” that’s not much of a policy. Thistle was almost a no brainer, in context. It’s what came later that dug the hole deeper and wider. It’s worth repeating that what’s generally overlooked is that each decision was made by the team and a variety of alternatives were considered. Moving forward with Wasabi .NET was only done after considering a lot of alternatives, including porting to python.

What was the point of no return, the Rubicon moment? I’d say adding a javascript backend. Javascript was the new hotness and everybody was doing it, but at the time it simply wasn’t plausible to depend on customers enabling javascript. Everything done client side needed to have a server side fallback. In order to keep the logic consistent, it was decided to write it once in ASP/Wasabi and have that translated to javascript. The downside is that now, even if Fog Creek decides to do a one shot translation of FogBugz to python or ruby or whatever, the client side javascript will have to be redeveloped or it will immediately decay.

Perhaps the FogBugz problem was actually a failure to tack. Sailing too long at a diagonal. Whatever. FogBugz predates radar; there was no way to see over the horizon and plan that far ahead.

wasabi

What is Wasabi actually like? How hard is it to learn? If you read Joel’s articles and got the impression that Wasabi is some entirely new alien language, you suck at reading comprehension. If you got that impression from somebody’s forum comment, they suck at reading comprehension and you suck at source verification. “Wasabi is 100% backwards-compatible with VBScript.” Admittedly, the number of new college grads proficient in VBScript is somewhat diminished these days, but it’s not unlearnable. I did all my editing in vim, with the filetype set to ASP. Syntax highlighting just worked.

For one thing, you learn Wasabi by editing and fixing bugs in a large project. You’re not starting from scratch. Add an if here, tweak a loop there, pick up the syntax and conventions as you go. Second, it’s a language that looks like a lot of others. If blah Then boo Else far Endif Wasabi, VBScript, ruby, lua, javascript. You know one, you know them all. I picked it up almost instantly despite having no ASP experience. Either I’m a genius, or it’s just not that hard. Fog Creek has never given any particular weight to years of experience with any particular language for new hires. Instead, they hire programmers, not XYZ programmers, so a custom language shrinks the applicable talent pool by exactly zero. (Programmers who insist on working solely in one language aren’t likely to be hired, even if Fog Creek uses that language. Needs change, people move around.)

Wasabi is statically typed, but owing to its dynamically typed roots, the compiler does most of the work via type inference. It’s global type inference, which means it occasionally goes off into the weeds if you call the wrong function with the wrong type, but even so, the compiler errors are more tractable than what C++ gives you. Low bar, I know. It’s fully debuggable in Visual Studio. Breakpoints, variable inspection, single stepping, the whole enchilada. It’s just .NET under the covers.

As far as programming languages go, the learning curve was rather agreeable. There was no bottom to the rabbit hole, but you could work productively at the surface without blindly leaping into the deep end. Most of the time you need not worry whether the code you were modifying would run on the server, on the client, or in the compiler. What, in the compiler? Wasabi also included a complete interpreter which you could activate during compilation. More like lisp macros than anything, you could inspect a partially compiled class and touch it up, adding a method here or there as needed.

There were also some features which an outside observer would no doubt qualify as horrendous hacks. Wasabi knew more about FogBugz than any generic compiler should ever know about its input. Then again, Wasabi was never a generic compiler. It automated and mandated a number of things that would have been tedious and error prone to do by hand on which FogBugz depended. It was over a year after I started that I wondered how listbox I had just created managed to implement special behaviors despite the fact I did nothing special. Compiler magic. The default input and output target names were tuned for FogBugz, stuff like that.

There are some quirks; it’s not all rainbows and unicorns. The way .NET interop and type annotations were bolted onto the syntax means some types (e.g., generic containers) are undeclarable. You can only create them in C# code. Yeah, it was frustrating, but the one minute a day I lost to that misfeature added up to less than twenty hours during my tenure, far less time than would have been spent rewriting everything. Just one of those technical debt interest payments. Designing your own language isn’t crazy, but it is hard. When it was just ASP, Wasabi was fairly cohesive (not to mention externally designed, which spares a lot of brain cells). As the feeping creature grew, it started developing a kind of language lupus.

Did interns hate it? I think the truth is closer to interns aren’t into working with hoary old code, regardless of the language said hoary old code is written in. For what it’s worth, new FogBugz code is more often than not written in C#, and Wasabi was never used for any other product.

The Wasabi compiler itself was all C#, evolving somehow out of the Thistle compiler written in Java. Maybe that should count as a rewrite, but porting Java to C# isn’t much more complicated than renaming the file. Fun fact: Wasabi has more unit tests than FogBugz. (Compilers are unusually adept at accumulating unit tests. I suspect gcc may have more unit tests than all other open source projects combined.)

monobugz

I spent a great deal of my time at Fog Creek getting the unix (Linux and OS X) port working after the switch from PHP to .NET. In the beginning, the team thought mono was sufficient to run FogBugz. Even the PHP version, starting from FogBugz 6, used mono to provide the search service. So we had some experience shipping code that depended on mono. Just not enough.

I fixed a crazy number of silly bugs in mono’s class libraries. To be fair, implementing all of .NET is a herculean task, and most of the fixes were easy enough to make. The result of these fixes meant that we had to ship a patched, custom version of mono, because vendors weren’t going to put out updated rpm or deb packages nearly fast enough. On the bright side, we controlled the entire stack. You provide a kernel, we provide nearly everything else. It was a lot of work wrestling this into shape, but in the end, we had far less trouble than dealing with incompatible PHP versions.

If I would have done anything differently, I’d have targeted java instead of .NET. The JRE is a known quantity on Windows and unix. Of course, I arrived a few months late for that decision and as the new guy wouldn’t have argued too loud. And nobody at the time knew what we know now. Anyway, Wasabi JRE wouldn’t have changed the central narrative of FogBugz is written in Wasabi.

impact

How much of what’s wrong with FogBugz (whatever you think that is) is attributable to using Wasabi? Not much. Day to day, it didn’t slow development. Aside from mono, I mostly dealt with issues like email parsing and figuring out why MySQL X would only accept a parenthesized query while MySQL Y would only accept the query without parentheses. Or any amount of UTF-8 dickery and the infinite ways MySQL text encoding can be misconfigured. The butt ugly end of the 80/20 split where it doesn’t matter how awesome and polished the 80% is.

There was a very long delay getting FogBugz 7 out the door. This could be attributed to the .NET conversion, but cutting bait and starting over would have meant a bigger delay. A planned 6.2 release, which would have been just like 6.1 but running on .NET, was cancelled when we realized that “exactly the same, just different” wasn’t much of a selling point. FogBugz 7 was a particularly ambitious release, considering we got a late start on it, but that’s not Wasabi’s fault. 6.2 was more or less ready to ship at the time of the August memo.

soy sauce

As I was winding down at Fog Creek, I worked on something called Project Soy Sauce. The anti-Wasabi. We already had the ability to generate C# code, but it wasn’t pretty. For one thing, due to the way for loop bounds were implemented, the generated code had temporaries galore. (The end condition is evaluated once: in For i = 1 To list.Count, list.Count is saved to a temp.) The lambda implementation preceded .NET, so it does things the old fashioned way by generating a new object to store the closed over variables. Enter Soy Sauce, which would unlower the generated code back to something that resembled the original. As mentioned before, the big issue here is the javascript code that would be orphaned by transitioning away. As far as I know, Soy Sauce is still sitting on the shelf somewhere, waiting for its moment to shine.

As you have undoubtedly guessed, Soy Sauce was written in perl.

lessons learned

Those who don’t learn from history are doomed to repeat it. Any day now, I suspect some fools are going to want to write web apps, but they won’t want to use raw javascript and they’ll create some ridiculous custom language to javascript compiler. I can only hope they have the good sense not to tell anybody about it.

This post is dedicated to all those interns who loved and lost. Whatever that means. :)