Redmine/Rails email through Gmail SMTP

Here is a bit of help getting [Redmine](http://www.redmine.org/), or any Rails app, to send email over SMTP through an existing [Gmail](http://www.gmail.com) account.
It’s quite easy to set up and you gain the usual perks of having everything centralized in a Gmail account. Note: this works with [Google Apps for your domain](http://www.google.com/apps/intl/en/business/index.html), too.

First you’ll need to install the smtp_tls plugin into your application’s lib directory so you can correctly authenticate. For posterity, I’ve uploaded a copy [smtp_tls.rb](http://blog.expandrive.com/wp-content/uploads/2008/12/smtp_tls.rb) along with this post.

www@host:~/redmine$ cd lib
www@host:~/redmine/lib$ curl -O http://blog.expandrive.com/wp-content/uploads/2008/12/smtp_tls.rb

Redmine utilizes email.yml to configure [ActionMailer](http://wiki.rubyonrails.org/rails/pages/ActionMailer). Here’s what ours looks like:

production:
    delivery_method: :smtp
    smtp_settings:
      address: smtp.gmail.com
      port: 587
      domain: gmail.com
      authentication: :login
      user_name: [email protected]
      password: password

Next, in production.rb makre sure to

require 'smtp_tls'

That’s it, you’re good to go.

If you want to set up any old Rails app [using Action Mailer] to send mail via Gmail, add this to production.rb

require 'smtp_tls' 

ActionMailer::Base.delivery_method = :smtp
ActionMailer::Base.perform_deliveries = true
ActionMailer::Base.raise_delivery_errors = true
ActionMailer::Base.default_charset = "utf-8"

ActionMailer::Base.smtp_settings = {
:address => "smtp.gmail.com",
:port => 587,
:domain => "gmail.com",
:authentication => :login,
:user_name => "[email protected]",
:password => "password",
}

git-fu: long term fork

Here is a [great post](http://programblings.com/2008/07/21/setting-up-a-long-term-fork-with-git/) by [Mathieu Martin](http://programblings.com/) about how to use Git to maintain and extend a long term fork of a project.

Recently at GiraffeSoft we started a new project, based on another existing project we already had going. We could call this a long term fork.

I’m sure you’ve done this before. You were probably using Subversion, and your project is probably 2 years behind HEAD of the original project.

Your invariants probably looked something like this:

  • * These projects will both keep being actively developed in the future;
  • * They will have some fundamental differences that will not be reconciled;
  • * They will however keep many similarities and we expect that they will benefit from the exchange of some specific patches, as development on both moves forward.

This happens all the time, especially in web development. You find a fantastic open source project to base your project on, but you need to customize and extend it. You also want to easily incorporate future features and bug fixes from the original project as they’re developed.

In days past, this problem could have been solved reasonably well by cloning the central repository and then exchanging patches and applying them manually.

I’ve tried this before. Remember that scene in the Dark Knight where the Joker makes the [pencil disappear](http://www.youtube.com/results?search_query=dark+knight+pencil+trick&search_type=&aq=f)? It’s kinda like that.

As you’ve guessed already, we’ve decided to try using Git to help manage this long term relationship.

It’s still not going to be easy unless you control both projects, but using a setup like Martin describes will be much easier than checking the man page for [patch](http://www.rt.com/man/patch.1.html) as you try to merge in new features by hand.

All aboard the Git train

I’m sure you’ve heard about Git. Chances are you’re still using subversion. You scratch your head and ask: “Why switch away from subversion, it does everything I need?” Or maybe: “Why on earth would I want a distributed version control system? Screw you fanboy.”

For a long time I felt the same way. All the hype seemed weird; it’s just version control. I felt: “Git, probably pretty neat—but I don’t really care.” Is it going to make my day any better? Probably not. Is it going to be better than perforce, which is ridiculously expensive for a small business? Probably not.

It turns out I was wrong.

When the Rails project migrated away from subversion to Git, I thought I’d give it another look. Our nightly subversion backup had grown to 400+ megabytes, mostly due to an infuriating 7-year old subversion bug/limitation/feature that won’t let you fully obliterate a file from the repository. It sucks to inadvertently check in a 10 megabyte DMG knowing it’ll be in the repository FOREVER.

After the release of ExpanDrive, we were at a point in our development cycle where we had a little room to re-evaluate our development toolchain and see if our current process could be improved. At Jon’s urging, we gave it a shot. Using git-svn [awesome], we imported our entire source tree and history into a hosted GitHub repository. We’ve been using it full time for about a bit over a month are loving it. GitHub does a great job and provides a wonderfully simple interface. It could be better, but that’s another post. In the next few days we’ll be posting a series of articles on the highs and lows of our first month with Git.

bash-completion

Are you using [bash-completion](http://www.caliban.org/bash/index.shtml#completion)? I bet you think you are. I bet you’re not. It’s a horrible name (up there with “The Wire” and “Battlestar Galactica”), because people hear it and think that they know what it is, but they’re wrong. Don’t prejudge the `bash-completion`. It’s much, much more.

Everyone knows about `[tab]` completion. It’s great. We love it. But, it leaves a lot to be desired. How many times have you done something like this:

Continuity:~$ cd m[tab]

and then seen this:

magnetk/ makay_tex_source.tex mathjob.pdf
methods of theoretical physics.pdf monterpp.pdf

Three pdfs, a tex source file, and one directory. Do you know how many times I’ve wanted to `cd` into a pdf file? Zero. I’ve never wanted to `cd` anything but a directory. So why does `[tab]` completion show me all those files that I’m obviously not interested in?

There’s a better way. In fact, `bash` already has a robust [programmable completion](http://www.faqs.org/docs/bashman/bashref_104.html#SEC111) system managed through the builtin commands `compgen` and `complete`. While `bash` is distributed with “programmable completion”, it isn’t distributed with “programmed completion”.

That’s where `bash-completion` comes in. It’s more than 9000 lines of pre-programmed completion artificial intelligence. It knows that you only `cd` into directories. It knows `passwd` only works on users, `groupmod` only works on groups, `unalias` only works aliases, and `which` only works on commands. And that’s just in the first 150 lines. The current version even includes a whole `svn` subprogram, so that it can figure out valid targets even for the various `svn` commands.

You can install `bash-completion` by Macports (`sudo port install bash-completion`), fink (`fink install bash-completion`), or apt-get (`apt-get install bash-completion`). You can also install it by hand. This involves downloading the file, unzipping it, and putting it somewhere.

Regardless of how you install it, you need to make sure it gets sourced when you start a new shell. Sometimes Macports et alias will do this for you, and sometimes they won’t. If you’re new to this, then `source` just reads a file and executes it line by line. Add a line to `.profile` (or another file that gets loaded when you open a new shell) that says `source /path/to/bash_completion`. For those of you with Twitter-eqsue character constraints on your dot-config files, you can also say `. /path/to/bash_completion`, where the `.` operator is a synonym for `source` that saves 5 characters at the cost of being infinitely less readable, and infinitely easier to misinterpret.

Then you’re done. Load up a new shell and enjoy. If you’re used to stupid file completion, then prepare to be amazed. You’ll find yourself groping around with `[tab]` in places you never would have imagined before, and you won’t even realize you’re doing it.

Packing It All In: Distributing Python With an App

Python has lovely built-in distribution tools. They’re great to use if you need a nice, repeatable, easy way to distribute your source code and have it install cleanly on a platform that has its `$PATH` set up correctly. However, if you want to distribute Python as part of a commercial software package, to platforms that may not even have Python installed, the procedure is not as clean or clear-cut. We devised a way to do it that mostly works, though we have to tweak it somewhat for each release. I’ll show you here our method for doing just that, using the [Snakefood](http://furius.ca/snakefood/) program for dependency extraction and a custom script to fill in the gaps that Snakefood can’t quite bridge.

Python is an interpreted language, which means, very basically, that it will not compile down to something that will run natively on any platform. The standard way to get Python to operate is to use the CPython interpreter, a program written in C that reads Python code performs the actions it describes (called “interpreting” it). There are other options, too, like [Jython](http://www.jython.org/) and [IronPython](http://www.codeplex.com/Wiki/View.aspx?ProjectName=IronPython), which do basically the same thing as CPython except that they translate the Python code to Java and .NET, respectively. We stick with C. After all, the whole reason we’re doing any of this is that we can’t count on Python being installed. We certainly can’t count on Java of .NET being installed.

As a very basic step one, we need to bundle the CPython interpreter with our app. It’s only about 15MB and is highly compressible, so we can easily include the interpreter, but the standard libraries in Python make for a fairly large installation: the estimated size of Python 2.5.2 is about 180MB. Even if we compress that, it’s still a huge download and a not-so-inconsequential amount of hard drive space. The good news is that we don’t use all of the standard libraries. The even better news is that there’s a pretty simple way of extracting only the files you do need and packaging them into a much smaller distribution. The trick up our sleeve is a small program written in Python called Snakefood. It’s not perfect, but I’ll show ways to get the most out of it.

The first step, of course, is getting Snakefood and installing it. If Python is in your $PATH, just extract the source, then run:

`% python setup.py install`

from the Snakefood directory, which will install Snakefood to wherever your current Python installation is. You can then run it with:

`% python sfood `

from any directory. The target file is the main script of your program. With just that command, it will pull the dependencies from the ‘import’ statements in your main script. That’s probably not good enough, so use the option `–follow`, which follows all the `import` statements in each of the imported modules to their leaves. That gets most of what you need.

The output of running Snakefood on a target is not entirely intuitive. It is a list of tuples like the following:

`((,), (, ))`

But sometimes the entry looks like this:

`((,), (None, None))`

It may be tempting, but you *can’t* skip these lines.

The format of the dependencies tells you that `` depends on ``, so you need to preserve it in your pared-down distribution. For us, this is as simple as making a new directory called `dist/`, and copying the file at path `os.path.join(, )` into it. You can make a list of these files directly from the Snakefood output (piped from `stdin`) with the following script:

import sys
import os
files = set()
for dep in map(eval, sys.stdin):
if dep[1][0] is not None:
path = os.path.join(dep[1][0], dep[1][1])
files.add(path)
else:
path = os.path.join(dep[0][0], dep[0][1])
files.add(path)

Now take this set of files and copy them into your new directory. Preserving the directory hierarchy is nontrivial, but not that hard. Hopefully, you have already created a custom Python installation so that all of the relevant files are in one place anyway. From there, you must find the root of the dependency tree. My custom Python installation is at `/Users/matthewmoskwa/ExpanDrive/python`, so on each path in the file set, I split on `’python’` and copy the new path into the `dist/` directory (making sure to create new directory nodes first):

import shutil
for fi in files:
distPath = os.path.join(‘dist’, fi.split(“python”)[1])
if not os.path.exists(os.path.dirname(distPath)):
os.makedirs(os.path.dirname(distPath))
shutil.copy(fi, distPath)

At this point, the writer of Snakefood claims 99% accuracy. I haven’t measured that claim, but I have found a major drawback: Snakefood misses all `__init__.py` files, and therefore any `import` statements in those files. Rather than being smart about it, I just use `os.walk()` to find all the `__init__.py` files and copy them into `dist/`. I then ru my code from `dist`/ and look for `ImportError`s. When I see one, I modify my script to manually copy the missing file to `dist/`. Not perfect, but it works, and it’s still much faster than doing the whole thing by hand.

The final step is to compile all of the files down to `.pyo` and remove all the `.py` and `.pyc` files. We use a Python script called `compileall.py`, located in the standard library, to compile, and then

`% find . -type f -name ‘*.pyc’ -print0 | xargs -0 rm -rdf`

to remove the files. Make sure to run `compileall.py` with the `-OO` option to get rid of docstrings and other unnecessary stuff.

Until someone writes an OS in Python or all OSes are guaranteed to have Python installed, this is a pretty good way to distribute Python code to the masses. The next step, actually getting it to run like an application, is up to you, though [py2app](http://pypi.python.org/pypi/py2app/) and [py2exe](http://www.py2exe.org/) can certainly help.

Finessing international characters out of Python

Whilst we whittled our filesystem problems down to a remaining few and sent our first Release Candidate out into the wild, we discovered we had another specter on the horizon to deal with: International Filename Support. Python generally handles this pretty well: it defaults to the web standard, UTF-8, so if you received a UTF-8 string, python will print the correct representation upon your call to “print”. No other work is necessary. This does not go so smoothly if the string you get is not encoded in UTF-8 (or ascii, since it is a true subset of UTF-8). We learned this limitation, and how to overcome it, over the course of two frustating days.

In our testing, we used another commercial SFTP Client to put some files with international characters in their names onto our test server (to wit: the files were called Québécois and Dvořàk). Unbeknownst to us, the client we used defaulted to Latin-1, aka ISO-8859-1 encoding. However, at this point, we also did not know about encoding in python, so we just output the strings as we received them. What we saw was Qu?b?cois and Dvo??k from the Terminal, and even worse in Finder, Qu? and Dvo? (more on why this was so later).

Python does not auto-detect encodings. You can get some third-party modules to get Python to try and do this.

We knew we had international characters, and we also knew that Mac OS X likes its characters to be encoded as UTF-8 (sort of).

So we tried this:

`output_string = input_string.encode(‘utf-8′)`

Exception!

`UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xc3 in position 0: ordinal not in range(128)`

It looks like python is guessing the string is ASCII. We think it’s UTF-8, so let’s try it again:

`ouptut_string = input_string.decode(‘utf-8′).encode(‘utf-8′)`

Exception!

`UnicodeDecodeError: ‘utf8′ codec can’t decode
bytes in position 2-4: invalid data`

Oh dear. At this point, I insisted the client we were using was definitely not encoding filenames as UTF-8 data, but Jeff insisted that it had to be (it’s the standard, after all). Then we had an argument about the semantics of decoding vs. encoding. On a whim, I tried decoding the string using ‘latin-1′ as an argument. Ta da! No more Unicode exception! We came to the following conclusion about python encoding/decoding: python always stores strings in an internal, canonical representation. Therefore strings are *always* implicitly decoded from ASCII to this form.

In short, python does this with every incoming string:

`canonical_string = decode(input_string, ‘ascii’)`

`output_string = encode(canonical_string, ‘ascii’)`

If the incoming strings are not ASCII-encoded, you must explicitly call decode() on them with the appropriate codec as an argument. Our codec in this case is Latin-1 (aka ISO-8859-1); so far so good.

Now that we have our string object, we must call encode() on it with ‘utf-8′ as an argument, since UTF-8 is almost what Mac OS X expects. I say “almost” because there are two possibilities for UTF-8 encoding: “Canonical From” and “Decomposed Form”. The difference is in how characters with diacritics, like à or é, are transmitted. Mac OS X uses decomposed form, which simply means that à is transmitted as two characters, \` and a, which are then combined. Python defaults to canonical form, so before we re-encode the strings as UTF-8, we’ve got to make this switch.

import unicodedata
decomposed_string = unicodedata.normalize(‘NFD’, \
input_string.decode(‘latin-1′))

Now we can finish up the task.

`output_string = decomposed_string.encode(‘utf-8′)`

Hooray! We’re done.

But wait… what happens if some other client uses a different encoding? Well, of course the characters will display incorrectly. We need some sort of default encoding that will work. We saw above that using UTF-8 as a default will not work, since there are encodings of characters in latin-1 (and probably other codecs) that are invalid in utf-8. We settled on defaulting to ASCII. This is acceptable in all cases because of a basic truth about text encoding: every single character is transmitted as at least one byte of data. ASCII has a printable representation of every possible byte. So while the character à does not have an encoding in ASCII, its byte sequence, `\xc3\xa0`, does, though it will usually just print as `??` since both those numbers are greater than `0x7F` and ASCII is not standardized above `0x7F`.

Putting it all together, this is basically the function we use to handle these strings.

import unicodedata

def re_encode(input_string, decoder = ‘utf-8′, encoder = ‘utf=8′):
try:
output_string = unicodedata.normalize(‘NFD’,\
input_string.decode(decoder)).encode(encoder)

except UnicodeError:
output_string = unicodedata.normalize(‘NFD’,\
input_string.decode(‘ascii’, ‘replace’)).encode(encoder)
return output_string

And that’s really all there is to it. Python wins the game. By defaulting to ASCII encoding, you won’t get any unhandled exceptions, and you’ll also know pretty quickly that something is wrong (just look for the `???????`s). For a much lengthier discussion of what Unicode is and does, see Joel Spolsky’s verbose take on the matter.

Twisted Software Foundation

Magnetk is a proud sponsor of Twisted, an open-source framework which helps make up the core of ExpanDrive. Twisted is a cross-platform event-driven networking engine, written in Python, that is developed by a smart & dedicated team across the world. Consider making a donation to the Twisted Software Foundation and help out this extraordinary open-source project. If you donate at their bronze level or higher, your banner will be displayed on their front page for the rest of 2008. Donate before May 15th and you’re a founding sponsor. Your banner will have a permanent home on their founding sponsors page.

High Leverage Development

During our third month of porting SftpDrive to OS X, it became clear that creating and maintaining a cross-platform codebase of high performance network and filesystem code would be far more effort than we had hoped. It wasn’t that the project was impossible, or even absurdly difficult. It just wasn’t any fun. Every time I looked at a #ifdef WIN32, it was even more clear the code was becoming much more tedious to debug and maintain.

It seems our work on Slingshot spoiled us. A few months working with Ruby, Objective-C and C# left us feeling happy and optimistic about programming—anything was possible! Needless to say, the tens of thousands of lines of procedural C in SftpDrive for Mac no longer brought about the same feeling of joy. It seemed unfair that hotshotwebdevelopers, with their pretty MacBooks, got all the attention, and they got to use fun high-level languages like Ruby or Python. We were developing a truly useful piece of technology, but were stuck on Windows and spending more than 50% of our time dealing with “pedestrian” details like pointers, memory management, IRQLs, and IRPs.

Still, we fancied ourselves hardcore and kept at it even though it was hard (and sometimes boring). When it was time to write an auto updater for SftpDrive we spent hours upon hours searching on Google and MSDN trying to find a clean implementation that would work on a vanilla Windows 2000 installation. One option, WinInet was ridiculously ugly and verbose. Another option, WinHTTP, didn’t work on Windows 2000 GM. We ended up using libCURL. It was a ridiculous and frustrating waste of time.

We wanted to import httplib, and then just start making things happen. XKCD hits the nail on the head:

XKCD python

We couldn’t afford to keep spending time and energy writing software this way. Even if we could afford it, we didn’t want to spend our time this way. Web applications were being developed at an astounding pace in part because of centralized management and deployment (they never have to maintain different versions for Macs and PCs), but also because they were using modern interpreted languages. Web developers also used community-developed open-source projects when they needed some help on a routine problem. They didn’t have to reinvent the wheel at every turn, but instead focused on the core of their product. With high-level languages and good libraries, small teams can create great products at a rapid pace. We realized that we could write applications for the desktop in the exact same way.

We rewrote SftpDrive from top to bottom in Python, with a GUI in Objective-C. It’s called ExpanDrive, and it took 1/3rd the time that SftpDrive took to develop. We greatly leverage Python and and many open source projects—just like a web-developer. To minimize conflicts and to have the necessary control over the runtime environment, our build process extracts only the necessary bits from the full python distribution and packages it into the .app. We trim Python from 5000+ files to a few more than 400. Like many OS X apps, we use Sparkle.Framework to automatically distribute and install updates. We’re pushing out weekly updates which include more than just bug fixes. ExpanDrive has been a breeze to maintain and extend and the core remains perfectly cross platform.

Desktop applications aren’t dead, they’re just about to really get going.

Sharing a Virtual Machine between VMWare Workstation and Fusion

Here is how to share a VM between Windows-based VMWare Workstation and Mac-based Fusion:

  1. Create a large FAT32 partition. You can either carve up your primary hard drive using something like Partition Magic – or do something more sane like buy external Firewire drive [USB drive performance on OS X is abysmal]. I own this Firelite drive which is powered over Firewire and also this Firewire 800 G-Tech drive. Let me re-emphasize: get a Firewire drive, USB is painfully slow.
  2. Format the drive using Disk Utility with the ‘MS-DOS’ filesystem. Windows, for no apparent reason, refuses to format a FAT32 volume larger than 32GB – so the format must be done in Disk Utility.
  3. FAT32 is limited to 4 GB files, so you’ll need to make sure your virtual disk is split in to into 2GB segments. It’s easy to specify this option during VM creation or you can convert an existing VM with the command line VMWare disk utilities. I recommend Robert Petruska’s DiskManager GUI, which makes things much easier. I recommend copying the virtual disk to a local drive first, it’ll save a lot of time.
  4. Modify the VM configuration to point at the split disk you just converted, and you’re good to go!

The only real drawback is that Fusion cannot do much [anything] with the tree of snapshots in created in workstation.

Great Bugs are like magic tricks

From Steven Frank, via Daring Fireball

A good bug, I mean a really good, pound-your-head-on-the-desk-for-a-week bug,
is exactly like a magic trick in that something impossible appears to be happening.

Isn’t that the truth?
We have seen more than our fair share of magic tricks. Stuff that you just can’t believe is happening. Impossible stuff.

Our mantra while debugging: “The best assumptions are wrong assumptions.”

Older Posts

Page 1 of 2