Ruby imap sync script error with attachments

I have found a ruby imap script online that i’m having some trouble
with, its originally from
http://blog.wherenow.org/blog/2008/01/10/transferring-email-via-imap/
but I was hoping to get some help from the group with it.

I’m primarily a sysadmin who scripts in shell and some perl, i’m kinda
lost in ruby.

Long story short i run a Scalix server and have moved mail hosts over
to google apps. I have a bunch of imap accounts that I would like to
sync to google apps (gmail for domains).

Unfortunately Scalix (the source IMAP server) adds the non-standard
flag \X-Has-Attach as well as \X-Forwarded, contrary to RFC 3501.

I’ve seen in another forum where they add the following regex to
imapsync to get around this

-regexflag ‘s/\X-*//g’

But, I have about 250 mailboxes to migrate, imapsync and needing to
specify all folders is out of the question.

Is there any way that you can help me out? Follows is the script and
the error I get when it runs;

======
#!/usr/bin/env ruby
require ‘net/imap’

Source server connection info.

SOURCE_HOST = ‘192.168.1.150’
SOURCE_PORT = 143
SOURCE_SSL = false
SOURCE_USER = ‘[email protected]
SOURCE_PASS = ‘password’

Destination server connection info.

DEST_HOST = ‘imap.gmail.com
DEST_PORT = 993
DEST_SSL = true
DEST_USER = ‘[email protected]
DEST_PASS = ‘password’

Mapping of source folders to destination folders. The key is the

name of the

folder on the source server, the value is the name on the

destination server.

Any folder not specified here will be ignored. If a destination

folder does

not exist, it will be created.

SFOLDERS = {}
DFOLDERS = {}

Utility methods.

def dd(message)
puts “[DEST: #{DEST_HOST}] #{message}”
end

def ds(message)
puts “[SOURCE: #{SOURCE_HOST}] #{message}”
end

Initialization

puts “[BACKUP] from ‘#{SOURCE_USER}’ to ‘#{DEST_USER}’”

Connect and log into both servers.

ds ‘connecting…’
source = Net::IMAP.new(SOURCE_HOST, SOURCE_PORT, SOURCE_SSL)

ds ‘logging in…’
source.login(SOURCE_USER, SOURCE_PASS)

dd ‘connecting…’
dest = Net::IMAP.new(DEST_HOST, DEST_PORT, DEST_SSL)

dd ‘logging in…’
dest.login(DEST_USER, DEST_PASS)

ds ‘enumerating folders…’
sfolderslist = source.list(’’, '’)
dfolderslist = dest.list(’’, '’)

ds ‘populating folders lists…’

List all source folders

i=0
while sfolderslist[i]!=nil
SFOLDERS[sfolderslist[i].name] = sfolderslist[i].name
i=i+1
end

List all destination folders no more present on the source

i=0
while dfolderslist[i]!=nil
if SFOLDERS[dfolderslist[i].name] != dfolderslist[i].name
DFOLDERS[dfolderslist[i].name] = dfolderslist[i].name
end
i=i+1
end

Loop through destination folders and remove unused ones

#DFOLDERS.each do |source_folder, dest_folder|

Open destination folder in read-write mode

#begin
#dd “removing folder ‘#{dest_folder}’”
#dest.select(dest_folder)
#dest.delete(dest_folder)
#end
#end

Loop through source folders and copy messages.

SFOLDERS.each do |source_folder, dest_folder|

Open source folder in read-only mode.

begin
ds “selecting folder ‘#{source_folder}’…”
source.examine(source_folder)
rescue => e
ds “error: select failed: #{e}”
next
end

Open (or create) destination folder in read-write mode.

begin
dd “selecting folder ‘#{dest_folder}’…”
dest.select(dest_folder)
rescue => e
begin
dd “folder not found; creating…”
dest.create(dest_folder)
dest.select(dest_folder)
rescue => ee
dd “error: could not create folder: #{e}”
next
end
end

Build a lookup hash of all message ids present in source and in the

destination folder.
source_info = {}
dest_info = {}

dd ‘analyzing destination existing messages…’
uids = dest.uid_search([‘ALL’])
if uids.length > 0
dest.uid_fetch(uids, [‘ENVELOPE’]).each do |data|
dest_info[data.attr[‘ENVELOPE’].message_id] = true
end
end

ds ‘analyzing source existing messages…’
uidd = source.uid_search([‘ALL’])
if uidd.length > 0
source.uid_fetch(uidd, [‘ENVELOPE’]).each do |data|
source_info[data.attr[‘ENVELOPE’].message_id] = true
end
end

Loop through all messages in the destination folder.

uida = dest.uid_search([‘ALL’])
if uida.length > 0
dest.uid_fetch(uida, [‘ENVELOPE’]).each do |data|

mid2 = data.attr[‘ENVELOPE’].message_id

If this message is included in both folders, skip it.

next if source_info[mid2] && dest_info[mid2]

Else delete the message from the destination folder.

dd “removing message #{mid2}…”
dest.uid_store(data.attr[‘UID’], ‘+FLAGS’, [:Deleted])
dest.expunge
end

end

Loop through all messages in the source folder.

uids = source.uid_search([‘ALL’])
if uids.length > 0
source.uid_fetch(uids, [‘ENVELOPE’]).each do |data|
mid = data.attr[‘ENVELOPE’].message_id

If this message is already in the destination folder, skip it.

next if dest_info[mid]

Download the full message body from the source folder.

ds “downloading message #{mid}…”
msg = source.uid_fetch(data.attr[‘UID’], [‘RFC822’, ‘FLAGS’,
‘INTERNALDATE’]).first

Append the message to the destination folder, preserving flags and

internal timestamp.

dd “storing message #{mid}…”
dest.append(dest_folder, msg.attr[‘RFC822’], msg.attr[‘FLAGS’],
msg.attr[‘INTERNALDATE’])
end
end

source.close
dest.close
end

puts ‘[BACKUP] done’

error

=====================

[[email protected] ruby]# ./is2.rb
[BACKUP] from ‘[email protected]’ to ‘[email protected]
[SOURCE: 192.168.1.150] connecting…
[SOURCE: 192.168.1.150] logging in…
[DEST: imap.gmail.com] connecting…
[DEST: imap.gmail.com] logging in…
[SOURCE: 192.168.1.150] enumerating folders…
[SOURCE: 192.168.1.150] populating folders lists…
[SOURCE: 192.168.1.150] selecting folder ‘Drafts’…
[DEST: imap.gmail.com] selecting folder ‘Drafts’…
[DEST: imap.gmail.com] analyzing destination existing messages…
[SOURCE: 192.168.1.150] analyzing source existing messages…
[SOURCE: 192.168.1.150] downloading message
[email protected]MHS
[DEST: imap.gmail.com] storing message
[email protected]MHS
/usr/lib/ruby/1.8/net/imap.rb:952:in get_tagged_response': Invalid Arguments: Unable to parse flag \X-has-attach (Net::IMAP::BadResponseError) from /usr/lib/ruby/1.8/net/imap.rb:998:insend_command’
from /usr/lib/ruby/1.8/net/imap.rb:986:in synchronize' from /usr/lib/ruby/1.8/net/imap.rb:1004:insend_command’
from /usr/lib/ruby/1.8/net/imap.rb:598:in append' from ./is2.rb:162 from ./is2.rb:148:ineach’
from ./is2.rb:148
from ./is2.rb:82:in `each’
from ./is2.rb:82

==============================================================================

On Apr 5, 2008, at 10:00 , [email protected] wrote:

I’ve seen in another forum where they add the following regex to
imapsync to get around this

-regexflag ‘s/\X-*//g’

But, I have about 250 mailboxes to migrate, imapsync and needing to
specify all folders is out of the question.

I’d still use imapsync. It is the only thing out there I’ve found that
does a good job, esp considering it is a real sync (you can rerun to
sync anything new). Use ruby (or whatever) to generate a script that
invokes all the imapsync’s you need. It should be a fairly simple
matter to iterate through /etc/passwd’s users and generate an imapsync
for all of them (not for each and every folder). Here is what I used a
while back:

#!/bin/bash

user1=$1; shift
user2=$1; shift

imapsync --host1 localhost --user1 $user1 --host2 xxx --user2 $user2 –
ssl1 --ssl2


I didn’t have 250 users, otherwise I’d have written what I suggest
above and given it to you. :slight_smile:

On Apr 5, 4:22 pm, Ryan D. [email protected] wrote:

On Apr 5, 2008, at 10:00 , [email protected] wrote:

I’d still use imapsync. It is the only thing out there I’ve found that
does a good job, esp considering it is a real sync (you can rerun to
sync anything new). Use ruby (or whatever) to generate a script that
invokes all the imapsync’s you need. It should be a fairly simple
matter to iterate through /etc/passwd’s users and generate an imapsync
for all of them (not for each and every folder). Here is what I used a
while back:

Fair enough, thought I could get away with an easy loop ;-), i’ll look
into imapsync some more.

Thanks for your reply.

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs