Freeform and double-byte (Japanese) characters

Looks like I managed to fix the problem by playing with the core email file, core.email.php.

What worked for me: - Deleting, or commenting out, the $charset = “UTF-8”; line - Changing the $encoding variable to “7bit”

var $encoding = "7bit";     // Default bit depth (8bit = non-US char sets)

• At line 80, add “ISO-2022-JP”:

var $base_charsets = array('iso-8859-1', 'us-ascii', 'ISO-2022-JP');

• At line 212 (the reply_to function), replace

$name = '"'.$name.'"';

with

$name = mb_encode_mimeheader(mb_convert_encoding($name, $this->charset, "auto"));

• At line 1272 (the send_with_mail function), add

$this->finalbody = mb_convert_encoding($this->finalbody, $this->charset, "auto");

This is based on the suggestion made at the EE Japan site: http://expression-engine.jp/forums/viewthread/7/#8

Hi,

Will this hack work only for Japanese?

Im having a real problem with Norwegian - å æ ø, Å Æ Ø

Sent mail wont encode these letters.

regards, Kippi

I’m having trouble with ISO-2022-JP mail encoding as well. All of my apple machines are fine with UTF-8, but unfortunately my customers are generally running windows and they are having a rough time with garbled up text.

I’m running 1.6.8 here and I’ve tried numerous ways to get the above hacks to core.email.php to take, but it seems no matter what I do, the From (name)/Reply field as well as the subject field on every email comes up garbled. The message field itself is working as ISO-2022-JP, but I have no idea whats up with the name and subject ones.

Interestingly enough though, is that I notice that if I keep the number of japanese characters input into the name or title field below about 8 characters then they come out fine. Beyond that they show up as “=e3=82=b9=e3=82=ad=e3=83=ab=e3=8” etc. I also have noticed that whenever i input some standard roman characters (i.e., english) in front of the japanese it also seems to come out fine.

The last thing I can report is that when viewing the raw source headers on a garbled up email, the name and subject field both start with this: =?utf-8?Q?

but the header also correctly contains content-type: text/plain; charset=ISO-2022-JP, Content-Transfer-Encoding: 7bit, so I would guess that is why the actual message content is coming through OK.

Does this make any sense to anyone? It would be a huge help if someone could help me work out a solution to getting email properly send out in Japanese.

Cheers, Matthew.

Matthew,

How about the other email settings in the control panel? For one of my setups, in addition to the above hack, I did the following:

• In System preferences => Email configuration, I set Email configuration to “ISO-2022-JP”.
• I set the Email protocol to “PHP mail”.

Do you have similar settings?

I previously tried sending email using sendmail, with the Email protocol set to “Sendmail”, but I was also having problems with garbled text. After installing postfix on my server and setting the Email protocol to “PHP mail”, things started working better.

Although I don’t know if this is related, I would also try toggling “Automatically Convert High ASCII Text to Entities” on and off to see if anythings happens (Admin => Weblog Administration => Global Weblog preferences). However, changing this setting may have effects elsewhere in EE (I remember Japanese text in FF Matrix not working well a while ago when High ASCII was turned off). I wish Japan would just drop ISO-2022-JP support altogether and go with UTF-8… maybe in 10 years or so 😉

Nicolas,

Thank you very much for the quick reply.

I feel your pain in regards to the status-quo here of ISO-2022-JP. UTF-8 is so much easier to deal with.

I checked my setup in the CP and can confirm that the email config is setup as you have above: ISO-2022-JP and the email protocol is PHP mail.

One thing I mentioned earlier that I cant seem to wrap my head around is why the from and subject fields will only garble up the JP text after you input over about 8 characters or so. For most people, as long as the subject is fixed and the name field only requires the standard 4 or 5 kanji characters for a name, the problem goes unnoticed. However, as soon as you go over 8 characters or so, it comes out as one big long mess.

Im not sure about the high ASCII text options there. I’m not really doing much with FF Matrix, but I do use Playa and one of my clients requires WYGWAM for page creation and both of those run as Fieldframe types. If those two were to break on me, it would certainly cause trouble.

The server however is a DV server @ mediatemple, so I’m running qmail instead of postfix.
I’m starting to wonder if there is something on the server that I need to setup to get things moving forward. However, if possible, I would really like to take care of this in EE. As I mentioned in my previous post, I’m running 1.6.8 with MSM and all the sites under that installation are Japan based, so I dont mind so much if I have to hack away at the core files. Granted, it makes upgrading troublesome, but I dont imagine that the sites that I am running are going anywhere right away.

I dont know if the Japanese support is any better in EE2, but I have purchased two licenses to have a go with a couple of new projects. If EE2 turns out to be a lot better, then perhaps I’ll think about moving my 1.6.8 sites.

Again, thanks for your assistance. I’ll keep looking into this.

One thing I mentioned earlier that I cant seem to wrap my head around is why the from and subject fields will only garble up the JP text after you input over about 8 characters or so.

Maybe that’s a limitation of ISO-2022-JP? Although I’ve received a lot of junk magazines emails in Japanese with foot-long subject titles…

Have you checked that emails are sent in text-only (not HTML)? Looks like that’s what often used with Japanese emails (and unfortunately replaced with noisy ASCII art). Just from the top of my head, don’t know if that would solve anything.

Also, just to be sure, have you hacked the core files already?

Maybe there is something that can be done on the server side. If you could try out postfix or even sendmail, that might give you a hint of where the problem might be. Worst case, I would avoid Japanese text in the email subject completely, but that might not always be an option.

Good to know about EE2, thanks.