Here’s the short-story on the current situation with our mailing list
to Usenet gateway:
- Our Usenet host rejects multipart/alternative messages
because they are technically illegal Usenet posts - This means that some emails do not reach comp.lang.ruby
(several messages each day according to the logs) - We don’t like this
To solve this, we want to enhance the gateway to convert multipart/
alternative messages into something we can legally post to Usenet. I
have two thoughts on this strategy:
- If possible, we should gather all text/plain portions of an email
and post those with a content-type of text/plain - If that fails, we can just post the original body but force the
content-type to text/plain for maximum compatibility
Now I need all of you email and Usenet experts to tell me if that’s a
sane strategy. If another approach would be better, please clue me in.
I’ve pretty much made it this far. The code at the bottom of this
message is the mail_to_news.rb script used by the gateway rewritten
using this strategy.
If you aren’t familiar with the gateway code, you can get details
from the articles at:
There’s one problem left I know I haven’t solved correctly. Help me
figure out a decent strategy for this last piece and we can deploy
the new code.
The outstanding issue is how to handle character sets for the
constructed message. You’ll see in the code below that I just pull
the charset param from the original message, but after looking at a
few messages, I realize that this doesn’t make sense. For example,
here are the relevant portions of a recent post that wasn’t gated
correctly:
Content-Type: multipart/alternative; boundary=Apple-Mail-18-445454026
–Apple-Mail-18-445454026
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
charset=US-ASCII;
delsp=yes;
format=flowed
As you can see, the overall email doesn’t have a charset but each
text portion can. If we are going to merge these parts, what’s the
best strategy for handling the charset?
I thought of trying to convert them all to UTF-8 with Iconv, but I’m
not sure what to do if a type doesn’t declare a charset or when Iconv
chokes on what is declared? Please share your opinions.
If you are feeling really adventurous, rewrite the relevant portion
of the code below which I will bracket with a FIX ME comments.
Here’s the script:
#!/usr/bin/env ruby
written by James Edward G. II [email protected]
$KCODE = “u”
GATEWAY_DIR = File.join(File.dirname(FILE), “…”).freeze
$LOAD_PATH << File.join(GATEWAY_DIR, “config”) << File.join
(GATEWAY_DIR, “lib”)
require “tmail”
require “servers_config”
require “nntp”
require “logger”
require “timeout”
prepare log
log = Logger.new(ARGV.shift || $stdout)
log.datetime_format = "%Y-%m-%d %H:%M "
build incoming and outgoing message object
incoming = TMail::Mail.parse($stdin.read)
outgoing = TMail::Mail.new
skip any flagged messages
if incoming[“X-Rubymirror”].to_s == “yes”
log.info “Skipping message ##{incoming.message_id}, sent by
news_to_mail”
exit
elsif incoming[“X-Spam-Status”].to_s =~ /\AYes/
log.info "Ignoring Spam ##{incoming.message_id}: " +
“#{incoming.subject}–#{incoming.from}”
exit
end
only allow certain headers through
%w[from subject in_reply_to transfer_encoding date].each do |header|
outgoing.send(“#{header}=”, incoming.send(header))
end
outgoing.message_id = incoming.message_id.sub(/.+>$/, “>”)
%w[X-ML-Name X-Mail-Count X-X-Sender].each do |header|
outgoing[header] = incoming[header].to_s if incoming.key?header
end
doctor headers for Ruby T.
outgoing.references = if incoming.key? “References”
incoming.references
else
if incoming.key? “In-Reply-To”
incoming.reply_to
else
if incoming.subject =~ /^Re:/
outgoing.reply_to = “this_is_a_dummy_message-id@rubygateway”
end
end
end
outgoing[“X-Ruby-Talk”] = incoming.message_id
outgoing[“X-Received-From”] = <<END_GATEWAY_DETAILS.gsub(/\s+/, " ")
This message has been automatically forwarded from the ruby-talk
mailing list by
a gateway at #{ServersConfig::NEWSGROUP}. If it is SPAM, it did not
originate at
#{ServersConfig::NEWSGROUP}. Please report the original sender, and
not us.
Thanks! For more details about this gateway, please visit:
END_GATEWAY_DETAILS
outgoing[“X-Rubymirror”] = “Yes”
translate the body of the message, if needed
if incoming.multipart? and incoming.sub_type == “alternative”
FIX ME
handle multipart/alternative messages
extract body
body = “”
extract_text = lambda do |message_or_part|
if message_or_part.multipart?
message_or_part.each_part { |part| extract_text[part] }
elsif message_or_part.content_type == “text/plain”
body += message_or_part.body
end
end
extract_text[incoming]
if body.empty?
outgoing.body = "Note: the content-type of this message was
altered by " +
“the gateway.\n\n#{incoming.body}”
else
outgoing.body = "Note: non-text portions of this message were
stripped " +
“by the gateway.\n\n#{body}”
end
set the content type of the new message
outgoing.set_content_type( “text”, “plain”,
“charset” => incoming.type_param
(“charset”) )
END FIX ME
else
%w[content_type body].each do |header|
outgoing.send(“#{header}=”, incoming.send(header))
end
end
log.info "Sending message ##{incoming.message_id}: " +
“#{incoming.subject}–#{incoming.from}…”
log.info “Message looks like:\n#{outgoing.encoded}”
connect to NNTP host
begin
nntp = nil
Timeout.timeout(30) do
nntp = Net::NNTP.new( ServersConfig::NEWS_SERVER,
Net::NNTP::NNTP_PORT,
ServersConfig::NEWS_USER,
ServersConfig::NEWS_PASS )
end
rescue Timeout::Error
log.error “The NNTP connection timed out”
exit -1
rescue
log.fatal “Unable to establish connection to NNTP host:
#{$!.message}”
exit -1
end
attempt to send newsgroup post
unless $DEBUG
begin
result = nil
Timeout.timeout(30) { result = nntp.post(outgoing.encoded) }
rescue Timeout::Error
log.error “The NNTP post timed out”
exit -1
rescue
log.fatal “Unable to post to NNTP host: #{$!.message}”
exit -1
end
log.info “… Sent. nntp.post() result: #{result}”
end
END
Thanks for the help.
James Edward G. II
But if there are plans to have the list itself do
