High Leverage Development

During our third month of porting SftpDrive to OS X, it became clear that creating and maintaining a cross-platform codebase of high performance network and filesystem code would be far more effort than we had hoped. It wasn’t that the project was impossible, or even absurdly difficult. It just wasn’t any fun. Every time I looked at a #ifdef WIN32, it was even more clear the code was becoming much more tedious to debug and maintain.

It seems our work on Slingshot spoiled us. A few months working with Ruby, Objective-C and C# left us feeling happy and optimistic about programming—anything was possible! Needless to say, the tens of thousands of lines of procedural C in SftpDrive for Mac no longer brought about the same feeling of joy. It seemed unfair that hotshotwebdevelopers, with their pretty MacBooks, got all the attention, and they got to use fun high-level languages like Ruby or Python. We were developing a truly useful piece of technology, but were stuck on Windows and spending more than 50% of our time dealing with “pedestrian” details like pointers, memory management, IRQLs, and IRPs.

Still, we fancied ourselves hardcore and kept at it even though it was hard (and sometimes boring). When it was time to write an auto updater for SftpDrive we spent hours upon hours searching on Google and MSDN trying to find a clean implementation that would work on a vanilla Windows 2000 installation. One option, WinInet was ridiculously ugly and verbose. Another option, WinHTTP, didn’t work on Windows 2000 GM. We ended up using libCURL. It was a ridiculous and frustrating waste of time.

We wanted to import httplib, and then just start making things happen. XKCD hits the nail on the head:

XKCD python

We couldn’t afford to keep spending time and energy writing software this way. Even if we could afford it, we didn’t want to spend our time this way. Web applications were being developed at an astounding pace in part because of centralized management and deployment (they never have to maintain different versions for Macs and PCs), but also because they were using modern interpreted languages. Web developers also used community-developed open-source projects when they needed some help on a routine problem. They didn’t have to reinvent the wheel at every turn, but instead focused on the core of their product. With high-level languages and good libraries, small teams can create great products at a rapid pace. We realized that we could write applications for the desktop in the exact same way.

We rewrote SftpDrive from top to bottom in Python, with a GUI in Objective-C. It’s called ExpanDrive, and it took 1/3rd the time that SftpDrive took to develop. We greatly leverage Python and and many open source projects—just like a web-developer. To minimize conflicts and to have the necessary control over the runtime environment, our build process extracts only the necessary bits from the full python distribution and packages it into the .app. We trim Python from 5000+ files to a few more than 400. Like many OS X apps, we use Sparkle.Framework to automatically distribute and install updates. We’re pushing out weekly updates which include more than just bug fixes. ExpanDrive has been a breeze to maintain and extend and the core remains perfectly cross platform.

Desktop applications aren’t dead, they’re just about to really get going.

  • rick

    Awesome story. I’m sure the web and the desktop will converge at some point. Let’s just hope it isn’t with silvercrap or air (adobe).

  • Smithe

    Do you use FUSE?

  • Jeedee

    The only major problem with interpreted language is that it makes it really easier to steal the source of an app since it is not compiled.

  • http://bottomdwellersmusic.com Mark Eagleton

    This is indeed very much like web development. Our GUI’s are written in the same language (CSS) for each platform (browser), but require different very different rules. The desktop applications may be more ahead than we realize.

  • http://neoglam.com the daniel

    Jeedee: Not exactly – with Python as well as many others, you can compile and ship bytecode instead of source. It can still be decompiled, but shrug so can Java.

  • curios in chicago

    I totally dig the story, but one thing isn’t that clear to me.

    Are you saying that now that you’ve:

    (a) rewritten the core of the program in Python, and

    (b) now it is a better cross paltform app?

    So, are you planning on now taking the ExpandDrive codebase and porting it back to Windows using a different layer for the gui?

    Just wondering. Good job on the program, though!

  • http://cyth.net/~ray/ Ray Lai

    It usually takes less time to rewrite an app than to develop it for the first time.

  • Ciaran

    How much of a downside is having to also ship the language environment with the app?

  • http://ramin.firoozye.com ramin

    Outstanding! Am a happy ExpandDrive user. Good to hear Python getting used for system level stuff.

    The desktop app I’ve been working on uses a slightly different architecture — Python, Django, Twisted, and SQLite, all rolled into a ‘microserver’ that runs in the background. The GUI can be anything you want as long as it talks HTTP. Cross-platform as all get out and the response is pretty decent. As you say, a joy to develop on.

    BTW, that very XKCD cartoon is taped up on my wall :-)

  • http://www.macresearch.org charles

    “To minimize conflicts and to have the necessary control over the runtime environment, our build process extracts only the necessary bits from the full python distribution and packages it into the .app. We trim Python from 5000+ files to a few more than 400.”

    Thanks very much for this very interesting post. Now, the above sentence could use a whole (geeky) other post, or even some separate source code sharing… you know, this whole “giving back to the community” ;-) No pressure, of course!

  • komac

    yeah,recently I have ported an MFC app to Mac OS X,and it leads me to realize how important it is to write cross platform codes.

  • Emanuel

    Great post and an outstanding work!
    I’d also be interested in knowing how you made the trimmed down version of Python!


  • Pingback: Stranger Than Fiction » Computer Science Degree in a Blog Post()

  • http://blog.magnetk.com jonshea

    @Mark: I’m not at all jealous of people who spent their days working on cross browser compatibility. Ugh. Still, desktop applications

    @curios: That’s certainly in interesting idea, isn’t it?

    @Charles, Emanuel: We’re going to do followup blog posts about some of the tools and techniques we’ve used, including how to create a dependancy extracted subset of python.

  • http://www.wooji-juice.com/blog/high-leverage-why-yes.html Canis/Wooji Juice

    We’re doing something similar over at Wooji Juice, using Cocoa and Python to make commercial Mac apps. I’ve written a post exploring our reasons in more detail, over at our website. The short version? Impedance mismatch — or the lack of it.

  • Thomas

    I couldn’t agree more. People can be very scathing of higher-level languages but to me it’s about the enjoyment of programming, if I didn’t care about that I would try to write everything in pure assembly.

  • Pingback: » Blog Archive » MacBook Review()

  • Pingback: Random Non Sequitur » Blog Archive » ExpanDrive for president()

  • Pingback: ExpanDrivel » on C++()

  • http://none JoelB

    I too am interested in learning about the part of your build process that strips down Python from 5000+ files to 400. I would imagine that it mainly involves removing unused library modules … but it would be cool to see exactly how you guys do it.