Moving to Google Apps with imapsync

For 2.5 yrs work has made use of a Kolab server I set up, but recently it has been taking up too much of my time to maintain. Google Apps was chosen as the replacement, so the problem then became how to get 39GB of data uploaded to Google over a fairly slow ADSL link, whilst keeping everyone up and running as much as possible.

imapsync seemed to be the tool of choice for a lot of people doing migrations of IMAP users, and it has done a very good job here too. I used version 1.311 from source rather than the older package provided with Ubuntu at the time as it contained various fixes for Google’s IMAP servers.

Moving standard folders over to Google Apps using imapsync is pretty straightforward and can be done without any special consideration:

 imapsync --host1 imap.oldmail --user1 $user1 --passfile1 pass1.txt --authmech1 PLAIN \
    --host2 imap.gmail.com --user2 $user2 --passfile2 pass2.txt --authmech2 LOGIN --port2 993 --ssl2

Just running the command as above doesn’t account for any naming differences between the old imap server and Gmail, this is easily taken care of with the –regextrans2 option which allows regular expressions to be applied to folder names as they are copied.

The old sever kept it’s sent mail in a folder called “Sent Items”, so a first attempt at renaming was:

 imapsync --host1 imap.oldmail --user1 $user1 --passfile1 pass1.txt --authmech1 PLAIN \
    --host2 imap.gmail.com --user2 $user2 --passfile2 pass2.txt --authmech2 LOGIN --port2 993 --ssl2 \
    --regextrans2 's/Inbox\/Sent$/Sent Mail/'

This unfortunately led to a label being created in the Gmail web interface called “Imap/Sent Mail” – not the desired result. The mail should end up in the actual “Sent Mail” folder.

Sent Mail not Imap/Sent Mail

Having a look at the gmail account in Thunderbird, the standard gmail folders show up under a parent folder called [Google Mail]. Armed with this information a quick google search reveals a few posts about using the –prefix2 option to get the mail to sync into the correct place – it basically boils down to putting the existing sent mail into a folder called “[Google Mail]/Sent Mail”. Because of needing to use the –prefix option the sync of the mail was split into two passes.

Two pass email syncing

• Sync over all the folders in your current imap account, but make sure you exclude your sent, draft, junk and deleted items
• Sync over the sent mail, drafts and if needed deleted items (you may have somebody who uses their deleted items folder as a filing system – beware!)

So the two pass solution becomes:

 #transfer the standard folders, skipping sent, drafts, deleted and spam
 imapsync --host1 imap.oldmail --user1 $user1 --passfile1 pass1.txt --authmech1 PLAIN \
    --host2 imap.gmail.com --user2 $user2 --passfile2 pass2.txt --authmech2 LOGIN --port2 993 --ssl2 \
    --exclude "Sent Items|Deleted Items|Drafts|Spam" \
    --skipsize

 #transfer the [Google Mail] folders
 imapsync --host1 imap.oldmail --user1 $user1 --passfile1 pass1.txt --authmech1 PLAIN \
    --host2 imap.gmail.com --user2 $user2 --passfile2 pass2.txt --authmech2 LOGIN --port2 993 --ssl2 \
    --folder "Inbox/Sent Items" --prefix2 '[Google Mail]/' --regextrans2 's/Inbox\/Sent\ Items$/Sent Mail/' \
    --folder "Inbox/Deleted Items" --prefix2 '[Google Mail]/' --regextrans2 's/Inbox\/Deleted\ Items$/Bin/' \
    --folder "Inbox/Drafts" --prefix2 '[Google Mail]/' --regextrans2 's/Inbox\/Drafts$/Drafts/' \
    --skipsize

 #add the following to the lines above if you want to keep your spam!
     --folder "Inbox/Spam" --prefix2 '[Google Mail]/' --regextrans2 's/Inbox\/Spam$/Spam/' \

A localisation gotcha

One word of caution – the “[Google Mail]” part as well as the actual folder names will change depending on the language set for the account. With the language set to US English (the default) rather than British English the folder prefix was [GMail] and some of the folder names were different e.g. Trash rather than Bin.

Incremental Syncing

As the sync took the best part of two weeks, it was run multiple times. One by one users were switched over to Google Apps. imapsync really comes into it’s own at this point as it can be run multiple times and will not re-transfer email. The –skipsize option is needed if you re-run imapsync otherwise due to the way the Google imap works the messages will not be regarded as the same and will be re-transferred.

The little details

The final version of the sync script ended up being a bit more complicated as a result of having to deal with the way the Outlook plugin a few people were using worked.

Skipping the Outlook folders was easily done with:

 --exclude "Tasks|RSS Feeds|Outbox|Junk E-mail|Junk|Junk Email" \
 --exclude "Notes|Journal|Calendar|Contacts|Sent Items|Trash|Drafts|Sent$|Deleted Items|Deleted Messages|sent-mail|Sent Messages" \
 --exclude "Sent Items Imap|Drafts Imap|AVG Virus Vault" \

The Kolab plugin was storing the email inside of the mail IMAP INBOX in a folder called Inbox i.e. INBOX/Inbox. This meant that for the users who had been using Outlook the command line to transfer the mail boxes change slightly to account for the extra Inbox folder:

 --folder "Inbox/Sent" --prefix2 '[Google Mail]/' --regextrans2 's/Inbox\/Sent$/Sent Mail/' \

Syncing over the main Inbox to the correct place has to be done in a separate pass:

 --include "INBOX/Inbox" --regextrans2 's/Inbox$/INBOX/' \

Conclusions

All in all imapsync got 12 users and 39GB of data migrated in a very painless fashion and allowed people to be moved over to Google Apps one at a time as and when their mail was synced. So far nobody has missed anything!