Commit de3fecb4 authored by paz's avatar paz

Merge branch 'wip/fix-encoding-issues' into 'master'

ensure a proper encoding set - fix parsing of some UTF-8 mails

Closes #409

See merge request schleuder/schleuder!301
parents c726a35e 92c7ead4
Pipeline #32797 passed with stages
in 15 minutes and 30 seconds
......@@ -15,7 +15,7 @@ This project adheres to [Semantic Versioning](http://semver.org/).
* Allow Jenkins job notifications to reach lists. Before, such mails were rejected due to being "auto-submitted".
* Do not recognize sudo messages as automated message (#248)
* Fixed using x-attach-listkey with emails from Thunderbird that include protected headers.
* Ensure ASCII-8BIT as external encoding, this should ensure that plain text emails in different charsets can be parsed (#409)
## [3.4.1] / 2019-09-16
......
# default to ASCII-8BIT encoding as early as possible for external
# data.
#
# this should ensure we are able to parse most incoming
# plain text mails in different charsets.
Encoding.default_external = Encoding::ASCII_8BIT
# Stdlib
require 'fileutils'
require 'singleton'
......
MIME-Version: 1.0
Subject: =?UTF-8?B?44G+44G/44KA44KB44KC?=
From: sender@example.com
To: =?UTF-8?B?44G/44GR44KL?= <schleuder@example.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: base64
44GL44GN44GP44GI44GTCgotLSAKaHR0cDovL2xpbmRzYWFyLm5ldC8KUmFpbHMsIFJTcGVjIGFu
ZCBMaWZlIGJsb2cuLi4uCg==
\ No newline at end of file
MIME-Version: 1.0
Received: by 10.231.35.72 with HTTP; Fri, 16 Oct 2009 05:39:34 -0700 (PDT)
Date: Fri, 16 Oct 2009 23:39:34 +1100
Delivered-To: schleuder@example.org
Message-ID: <57a815bf0910160539m64240421gb35ea52e101aedbc@mail.gmail.com>
Subject: testing
From: sender@example.com
To: Schleuder <schleuder@example.org>
Content-Type: multipart/mixed; boundary=00032557395e3572cf04760cb060
--00032557395e3572cf04760cb060
Content-Type: text/plain; charset=UTF-8
testing
--
http://lindsaar.net/
Rails, RSpec and Life blog....
--00032557395e3572cf04760cb060
Content-Type: text/plain; charset=UTF-8; name="=?UTF-8?B?44Gm44GZ44GoLnR4dA==?="
Content-Disposition: attachment; filename="=?UTF-8?B?44Gm44GZ44GoLnR4dA==?="
Content-Transfer-Encoding: base64
X-Attachment-Id: f_g0uxfl510
dGhpcyBpcyBhIHRlc3QK44GT44KM44KP44Gm44GZ44Go
--00032557395e3572cf04760cb060--
\ No newline at end of file
Delivered-To: schleuder@example.org
Received: by 10.231.12.67 with SMTP id w3cs164325ibw;
Fri, 30 Oct 2009 01:11:12 -0700 (PDT)
Received: by 10.150.44.2 with SMTP id r2mr2367210ybr.77.1256890271939;
Fri, 30 Oct 2009 01:11:11 -0700 (PDT)
Return-Path: <mikel@test.lindsaar.net>
Received: from mx1.test.lindsaar.net.au (mx1.test.lindsaar.net.au [210.14.110.240])
by mx.google.com with ESMTP id 25si7923673gxk.34.2009.10.30.01.11.11;
Fri, 30 Oct 2009 01:11:11 -0700 (PDT)
Received-SPF: neutral (google.com: 210.14.110.240 is neither permitted nor denied by domain of mikel@test.lindsaar.net) client-ip=210.14.110.240;
Authentication-Results: mx.google.com; spf=neutral (google.com: 210.14.110.240 is neither permitted nor denied by domain of mikel@test.lindsaar.net) smtp.mail=mikel@test.lindsaar.net
Received: from [192.168.4.253] (60-241-138-146.static.tpgi.com.au [60.241.138.146])
(using TLSv1 with cipher AES128-SHA (128/128 bits))
(No client certificate requested)
(Authenticated sender: mikel)
by mx1.test.lindsaar.net.au (Postfix) with ESMTPSA id 5C0186DD4CD
for <schleuder@example.org>; Fri, 30 Oct 2009 19:11:08 +1100 (EST)
Subject: =?utf-8?B?44G+44G/44KA44KB44KC44G+44G/44KA44KB44KC44G+44G/44KA?=
=?utf-8?B?44KB44KC44G+44G/44KA44KB44KC44G+44G/44KA44KB44KC44G+?=
=?utf-8?B?44G/44KA44KB44KC44G+44G/44KA44KB44KC44G+44G/44KA44KB?=
=?utf-8?B?44KC44G+44G/44KA44KB44KC44G+44G/44KA44KB44KC?=
From: sender@example.com
Content-Type: multipart/mixed; boundary=Apple-Mail-6--589811753
Message-Id: <60A112A8-F26C-4E23-95B8-4EB9F139D6A0@test.lindsaar.net>
Date: Fri, 30 Oct 2009 19:11:02 +1100
To: Schleuder <schleuder@example.org>
Mime-Version: 1.0 (Apple Message framework v1076)
X-Mailer: Apple Mail (2.1076)
--Apple-Mail-6--589811753
Content-Disposition: attachment;
filename*0*=utf-8''%E3%81%8B%E3%81%8D%E3%81%8F%E3%81%91%E3%81%93%E3%81%8B%E3%81%8D%E3%81%8F%E3%81%91%E3%81%93%E3%81%8B%E3%81%8D%E3%81%8F%E3%81%91%E3%81%93%E3%81%8B%E3%81%8D%E3%81%8F%E3%81%91%E3%81%93%E3%81%8B;
filename*1*=%E3%81%8D%E3%81%8F%E3%81%91%E3%81%93.txt
Content-Type: text/plain;
x-unix-mode=0644;
name="=?utf-8?B?44GL44GN44GP44GR44GT44GL44GN44GP44GR44GT44GL44GN44GP?=
=?utf-8?B?44GR44GT44GL44GN44GP44GR44GT44GL44GN44GP44GR44GTLnR4?=
=?utf-8?B?dA==?="
Content-Transfer-Encoding: 7bit
this is the data
--Apple-Mail-6--589811753--
\ No newline at end of file
MIME-Version: 1.0
Subject: =?UTF-8?B?44G+44G/44KA44KB44KC?=
From: sender@example.com
To: =?UTF-8?B?44G/44GR44KL?= <schleuder@example.org>
Content-Type: text/plain;
charset=iso-2022-jp
Content-Transfer-Encoding: 7bit
$B$9$_$^$;$s!#(B
Delivered-To: schleuder@example.org
Date: Wed, 28 May 2014 17:18:19 +0900 (JST)
From: sender@example.com
To: schleuder@example.org
Subject: test
Message-ID: <xxxxx@docomo.ne.jp>
MIME-Version: 1.0
Content-Type: text/plain; charset="Shift_JIS"
Content-Transfer-Encoding: 8bit
あいうえお
このメールはテスト用のメールです。
今後ともよろしくお願い申し上げます!
Delivered-To: schleuder@example.org
Date: Wed, 28 May 2014 17:18:19 +0900 (JST)
From: sender@example.com
To: schleuder@example.org
Subject: test
Message-ID: <from@example.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="ks_c_5601-1987"
Content-Transfer-Encoding: 8bit
Ƽ
Date: Wed, 28 May 2014 17:18:19 +0900 (JST)
From: sender@example.com
To: schleuder@example.org
Subject: test
Message-ID: <xxxxx@docomo.ne.jp>
MIME-Version: 1.0
Content-Type: text/plain; charset="Shift_JIS"
Content-Transfer-Encoding: 8bit
あいうえお
このメールはテスト用のメールです。
今後ともよろしくお願い申し上げます!
MIME-Version: 1.0
Subject: =?UTF-8?B?44G+44G/44KA44KB44KC?=
From: Mikel Lindsaar <raasdnil@gmail.com>
To: =?UTF-8?B?44G/44GR44KL?= <raasdnil@gmail.com>
Content-Type: text/plain;
charset=iso-2022-jp
Content-Transfer-Encoding: 7bit
$B$9$_$^$;$s!#(B
Subject: test
To: schleuder@example.org
From: sender@example.com
Date: Wed, 6 Nov 2019 22:04:23 -0200
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859
Content-Language: en-US
Content-Transfer-Encoding: 8bit
é
Subject: test
To: schleuder@example.org
From: sender@example.com
Date: Wed, 6 Nov 2019 22:04:23 -0200
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
é
require "spec_helper"
describe "user sends emails with different charsets" do
Dir["spec/fixtures/mails/charset_mails/*.eml"].each do |f|
it "works with #{File.basename(f,'.eml')}" do
start_smtp_daemon
list = create(:list)
list.subscribe("admin@example.org", nil, true)
# Clean any LANG from env as this is usually the case for MUAs
# https://0xacab.org/schleuder/schleuder/issues/409
with_env(ENV.delete_if {|key, value| key =~ /LANG/ || key =~ /LC/ }) do
error = run_schleuder(:work, list.email, f)
mails = Dir.glob("#{smtp_daemon_outputdir}/mail-*")
expect(error).to be_empty
expect(mails.size).to eq 1
end
stop_smtp_daemon
end
end
end
......@@ -136,6 +136,14 @@ RSpec.configure do |config|
`SCHLEUDER_ENV=test SCHLEUDER_CONFIG=spec/schleuder.yml bin/schleuder #{command} 2>&1`
end
def with_env(env)
backup = ENV.to_hash
ENV.replace(env)
yield
ensure
ENV.replace(backup)
end
def process_mail(msg, recipient)
output = nil
begin
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment