Q & A: Eric Anderton

This month’s Q&A victim is Eric Anderton. Around the D community he is most commonly known as ‘Pragma’. His most well-known project is DDL (D Dynamic Libraries), but he is also the maintainer of Enki, a frontend parser generator. Another of his projects, DSP (D Server Pages), is awaiting the completion of DDL. Something many probably don’t know is that he also works on the TracForums project for DSource. And with all of this going on, he still finds the time to post frequently in the D newsgroups.

When I recently asked Eric on very short notice to take some time out for a Q&A, he graciously accepted. As usual, my questions are in bold and his answers in normal type.

Your DSource profile says you are a web developer. How long have you been doing that? Which technologies do you usually work with? What are your favorites?

I’ve been doing web development and design for almost eight years now. It all began by working for a few start-ups during the dot-com boom, and then gradually moving on to consulting and then contracting; all in that arena. I’ve done other things besides web development (intern work, C++, tech support), but those avenues gave way to what became the most lucrative; a man’s gotta eat.

Technology-wise, I’m all over the map. I’ve done some personal stuff with PHP, some consulting work using ASP, and have used .NET, Java and ColdFusion professionally. At one point, even VB and Access were on the menu. Right now, ColdFusion combined with AJAX (dojo and YUI mostly) pays the bills. There have been plenty of other hobby languages, but all but D have failed to hold my interest.

My favorite web technology has to be ColdFusion, despite how much I’ve bashed it in the past. Ever since MX7 came out, it’s demonstrated itself to be very mature, well-documented and flexible. For all its quirks, it’s very even-handed at both RAD and highly-scalable designs. This translates into less time spent in “hacker land” trying to get trivial stuff done. I’ve also seen it work wonders for helping teams, with a broad range of proficiencies, work together as a team. I’m also very excited about what Adobe has done for CF8 and Flex.

Browsing through the DDL project pages, it seems obvious that you have a strong interest in things like linkers, grammars and such. What is your background with all of that? Is it just a hobby or have you done anything professionally?

Mostly, it’s just a hobby. I’ve had minimal classroom instruction in language theory and compiler mechanics, so I’m not formally trained in any of this. I’ve never composed a compiler, wrote my own linker, or even composed a code generator professionally – although I’d love to be paid for that kind of work. Thankfully, I find this kind of hacking on my own time very rewarding.

I just read what I can, and experiment even more. I’m one of these guys that can’t stand not knowing how something works, so I was always taking stuff apart as a kid; I always thought dinosaurs looked much more impressive as skeletons since you could see how they “worked” on the inside. So you could say that when undertaking DDL, I had a strong internal motivation to take all this “meta education” on. I certainly didn’t know as much about all this three years ago.

How and when did you first learn of D? Wat was it about the language that hooked you?

At the time I was working on something resembling a MUD/MMOG framework in .NET, largely due to the platform’s strengths: garbage collection, reflection and remoting. I found the performance to be a real drawback and the locked-in feeling of being tied to the .NET platform itself even worse. Java really didn’t feel much better. So I went back to investigating how C++ could be enhanced with macros and such to give it some of these capabilities without these problems.

Then I learned about D on slashdot.org, gave it a glance and promptly forgot about it. D just didn’t seem ready yet. I came back to it about six months later and decided to give it a chance after I realized it had everything I was looking for: no lock-in agenda, native compilation, and garbage collection. I also saw a golden opportunity to jump into a fledgling technology and help make it something really great. In then end, I think that’s really what has kept me interested: the D community and the raw potential of D.

If you could add one new feature to D right now, what would it be? What other features do you think would be useful to have?

I think the language, at the 1.0 mark, is adequate for most of what anyone could ask for. Someone is always going to come along and say “well ‘Python’ has this”, “Forth has that”, or “someone else broke LISP again.” That’s not going to change. Where D is weak, and has been in need of improvement for some time, is in how it’s conveyed to the world online.

People are clamoring for something that does what D does, and gives project managers warm-fuzzy-feelings by being a complete product. Honestly, having better documentation and a completely fleshed-out ABI is a must. And it doesn’t need a miracle to happen either: if the DigitalMars.com D Programming Language web were opened up as a wiki, I think we’d see an overnight improvement in this area. After that, the top support projects, a good IDE and GUI Debugger need to find their way into a DMD/GDC installer; a DigitalMars branded installer would be even better.

It seems to me that DDL must have been quite challenging to put together. What led you to undertake such a challenge?

The short version: I’ve learned to turn moments of naivete into an asset. Sometimes, the conventional approach exists for reasons that may not apply to what you are trying to accomplish.

Back in high-school, a friend of mine was showing me how to use C for the first time. When it was explained to me what an .obj file was (symbolic information, unlinked code and data), my first reaction was: “Can we load that into memory from our program and run it?” My friend then explained that it wasn’t possible, and that’s what .dll files were for. I took that for the plain truth and moved on to getting a for() loop to work correctly.

Fast-forward 10 or so years later and I find myself trying to turn D into a web-programming language (the DSP project). I was dismayed at how huge a .dsp page was when compiled into a .dll file, not to mention the technical drawbacks due to how partitioned a .dll is from its host. I saw how tiny my .obj files were and thought: “If only I could load those instead.” I then realized that I had the correct insight to solving this very problem a decade ago, but merely lacked the skill to do it then. Scratching that old itch was all the motivation I needed.

What has been the biggest difficulty during the development of DDL? Was there ever a time when a feature you wanted to implement had you stumped?

OMF parsing, OMF parsing and OMF parsing. This file format was the bane of my very existence for well over a year of working on DDL, and caused me to be stumped on more than one occasion.

The documentation for this .obj file format is almost more of a reference than a specification to code against, for reasons that completely escape me. There were so many times that little gotchas in the docs allowed me to code down a blind alley for weeks before a bug would manifest itself. Then, finding where I strayed from the specification took even more effort. This was all made worse by the fact that runtime linking has proven to be very hard to debug in and of itself. Thankfully, DMD’s handling of OMF seems to cut extremely close to the specification – so at least I knew that I could trust the data coming into my test harness as valid.

Once I had my head around most of the idiosyncrasies of this arcane format, it all started to come together. Things improved even more when I got some help from the community at large, which was great since it helped both my morale and the project itself by leaps and bounds. Now my only major problem is having the time to work on it!

What is the future direction for DDL? What sort of additional features do you have on the drawing board?

At the moment, I’m porting DDL to Tango, and removing the old Mango-parser scheme. So this is a win-win for most folks as back-porting the new DDL to Phobos should prove pretty easy. However, I have no explicit plans to support both libraries in the near future.

The other planned features for DDL really haven’t changed and are all on the project roadmap: ELF support, COFF support, better reflection, module unloading and better performance via lazy loading. I’ve also heard that there are some performance issues with large libraries (big symbol tables and such), that should be addressed.

How did Enki come about? What are your plans for it?

In retrospect, I guess Enki came about because D doesn’t have AOP.

When composing parsers for various tasks, I found myself repeating the same idioms, over and over again, for function after function. What was worse was that I learned that I wasn’t really very good at converting a grammar decomposition (typically BNF) to code, since the result was typically rife with assumptions and shortcuts that led to bugs and bad behavior. So I also wound up fixing the same bugs, over and over again.

So I couldn’t find that magic balance point between compactness, readability and performance that I wanted. Either I had too much code to maintain, too much overhead to be efficient, or not enough code and/or overhead to be correct. AOP could, in theory at least, allow me to decorate my code with the needed checks, preambles and epilogs without sacrificing clarity or performance. But since this wasn’t an option, I had to look elsewhere.

Next I started looking at The Language Machine as a possible alternative to hand-coding a parser. While it is a brilliant piece of software, I found the syntax to be a little too hard to get my head around. I also found it to be too generalized. All I wanted was something like LEX/YACC for D, without imposing so much on the developer. Just “a front-end generator,” I thought, “with a minimal runtime so you can maintain the rest in D.” When I realized that such a tool could effectively generate it’s own frontend, I couldn’t resist the urge to code one up.

Enki’s future looks very bright. I’m in the process of re-vamping the project to use Tango. I’ve also refactored the engine so it uses a visitor pattern for code generation, which will make composing alternate backends for Enki much easier. As Enki loves to eat it’s own dog food, this technique is now used to generate both the BNF output and D code output. The consequence of this has been a vastly improved code generator, and a redesigned parser backend that will now accept a distinct lexer/tokenizer, improved error handling, and something I call “repeated predicate optimization.” There’s even some new EBNF constructs I concocted to support repetition ranges and delimiters, plus a new attribute syntax that will allow coders to target multiple backends from a single .bnf file.

I’ve also toyed around with making a compile-time parser generator with some limited success. I’ve held off on developing it further until Walter comes up with some better meta-programming support.

Aside from your own projects, which D libraries and/or tools do you regularly make use of?

I would have to say that Tango, Mango and DBBI have had the most impact on my D projects. The first two are obvious as they feed directly into DDL and Enki. I made great use of DBBI when composing a PHPBB to TracForums translator for the TracForums project. We haven’t used it on dsource.org yet, but I anticipate it coming in very handy once TracForums has matured.

Are there any specific, and currently unimplemented, libraries or tools you would like to see developed in the future? Any you’re thinking of implementing?

On the unfinished project front, my pet R&D project DSP comes to mind. I’m also still working on TracForums, which currently enjoys a very nice life as a functional beta on the Tango site.

I’ve also talked with the Tango team about drafting an “updater tool” (think: “apt-get update”), possibly based on DSSS. I’ve held off on that only because the D front-end is still maintained in C++. Of course, this would be a grand opportunity for Enki, or even for anyone willing to just translate the DMD front-end to D since D 1.0 is now frozen. As every translation effort in the past has been crushed by DMD’s Beta schedule, there’s not as much (aside from bugfixes) from keeping such a project from flourishing now.

DBBI needs some serious TLC, since it seems to languish for huge stretches at a time. I’d love for someone to put up a “call for patches” on the forum, merge everybody’s improvements in, and then fold in a solid Variant type (like the one that popped up on the Tango forums recently) for good measure.

I’d also like to play a bit more with Rodin, since I bugged Gregor to try that concept out in the first place. I think D needs better reflection and supporting that via a pre-processor is probably the most elegant way to go short of using DDL. Not to mention, that those two approaches are highly compatible, and are just bristling with possibility.

Thanks again to Eric for taking the time to do this.

Technorati Tags: D Programming Language, DDL, Enki, DSP, Trac