msg upload to gmail ruby class (imap)
December 4th, 2007
… moving to Google Hosted (Google Apps), there was the need to upload some thousands of e-mail messages. Since they were (mt) MediaTemple Berkeley mbox stores, i scp’d them locally (backup) and them imported them into Evolution local folders.
The problem was the copy/move to the google hosted imap mail server. Evolution couldn’t finish the task on severall tries. Mozilla Iceape (Seamonkey) had a similar behaviour, suffering from server disconnects, thus stopping the operation. Because i had severall folders, the task needed lot’s of attention, and i wanted the machine to do it all by itself…
Solution: let’s code !! What response is the server sending… ?
Shugo Maeda’s Ruby net/imap.rb class (stdlib) is simple and trusty.
Not wasting lots of time in it, i made a new class with the following usage, to send specified mbox file mail messages to a gmail (google hosted) account:
$ ./2gmail.rb MBOXFILE GMAILFOLDER [STARTINGMESSAGE]
If GMAILFOLDER doesn’t exist, gets created (i prefered messages in temporary folders) upthere.
If it breaks, you can:
$ tail 2Gmail.log
to see the last message sent, and restart the process in that message. I could have threaded the whole thing, but this is done just once, so lets keep in focus…
Usual problems during the tests, msg headers/body separation, timestamps, etc…
Class Mail in mailread.rb does a great job creating an Hash for the Message Headers. Sweet…
Everything is logged, so that it’s easy to ‘grep’ for errors in the 2Gmail.log file.
I had one only problem on a message with a 25MB attachment (gmail didn’t accept it).
All the messages are now up there, nobody knows for certain where (EU ? US ? both ?…)
The code needs one or two more Exception Handling, but it works very well. I had 6 xterm’s uploading at the same time. And yes, I could have openned 6 tabs in Gnome Terminal since we have them for ages, Mr. Leopard ones! Hey, we can even detach Tabs to independent Terminal windows via drag-n-drop (like in Epiphany, way way long ago… way…)
There are some thousands more messages to be uploaded, from ancient backups… maybe next week.
Operation screenshot: at Flickr as usual
Wanna try ? Get it here (ready for Debian & Ubuntu. For other distros, check line 1 or just prepend ruby interpreter on call)
$ chmod u+x 2gmail.rb
$ ./2gmail.rb MBOXFILE GMAILFOLDER [STARTINGMESSAGE]
Rúben Fonseca said:
Excellent work Pedro, and a nice demonstration on how to bind simple concepts with the beauty glue of Ruby!
Love the screenshots and the Vi theme. Are you using Debian testing?
Andrew said:
Oddly, this mashes the body of most messages when run under OS X leopard — perhaps it has to do with the /r/n replacement. When I run this under Windows I had to make a change to explicitly use a “Time” type for the date in the imap.append call:
@imap.append(@folder, wrap_msg, nil, Time.parse(@msg.header[”Date”]))
Otherwise, the date is stored properly (from the date in the body) but the IMAP create date (or whatever it’s called) isn’t set properly and thus the far right column in Gmail shows the IMPORT date instead of the “original” date.
Many thanks for the script! I’m happily using it to import my almost 1GB of messages.
pedro said:
@ Rúben: Hi, thank you. Yes, Debian Testing, Lenny, stable enough for me
@Andrew: yes, the problem is for sure in the \r\n. I had those problems too until i got it right. Thank you and I hope the 1GB goes up well
I have now only 2GB left to go… The most recent 1GB messages are there already. Thank you for the timestamp patch.
pedro said:
[POST UPDATE]
i had some emails asking about the creation of the mbox file.
Since I had older emails inside a complex hierarchical tree, a quick workaround to upload them all into the Gmail archive is to create a http://www.gnome.org/projects/evolution/features.shtml Search Folder in Evolution setting a rule for what you pretend. In my case:
All the messages with Date < 1-1-2050
Apply the filter, and then all the messages get into that Search Folder scope.
Now, select them all and copy them into a new Inbox sub-folder, say, a 2upload folder (fortunatelly, Evolution didn’t fire up the filter while doing the copy…). This will create a 2upload file:
$HOME/.evolution/mail/local/…/Inbox.sbd/2upload
The bigger sized is the only one you need, since Evolution will create indexes and other extension 2upload files. Now, depending on what Label settings you want in Gmail, have the mbox files created as you wish. As for maildir, yes, doable too, but i didn’t need it.
priit said:
Hey, Pedro, thanks for that script. Works well
Ion said:
Hey!
How can I solve this please?
bash: ./2gmail.rb: /usr/bin/ruby: bad interpreter: No such file or directory
I’m using ubuntu 8.04!
Thanks!
pedro said:
Hi,
please, remove the first line on the script.
Then, run:
ruby 2gmail.rb MBOXFILE GMAILFOLDER [STARTINGMESSAGE]
or with all the paths:
/path/to/your/ruby /path/to/2gmail.rb /path/to/MBOXFILE /path/to/GMAILFOLDER [STARTINGMESSAGE]