I am working on a project which needs to deal with email encoding/decoding in different charsets. A python code for this can be shown in the below:
from email.header import Header, decode_header, make_header
from charset import text_to_utf8
class ....
def decode_header(self, header):
decoded_header = decode_header(header)
if decoded_header[0][1] is None:
return text_to_utf8(decoded_header[0][0]).decode("utf-8", "replace")
else:
return decoded_header[0][0].decode(decoded_header[0][1].replace("windows-", "cp"), "replace")
Basically, for the text like "=?iso-2022-jp?b?GyRCRW1CQE86GyhCIDxtb21vQHRhcm8ubmUuanA=?="; the "decode_header" function just tries to find the encoding: 'iso-2022-jp'; then it will use the "decode" function to decode the charset to unicode.
Now, in go, i can do something similar to like:
import "mime"
dec := new(mime.WordDecoder)
text := "=?utf-8?q?=C3=89ric?= <eric@example.org>, =?utf-8?q?Ana=C3=AFs?= <anais@example.org>"
header, err := dec.DecodeHeader(text)
Seems that there mime.WordDecoder allow to put a charset decoder "hook":
type WordDecoder struct {
// CharsetReader, if non-nil, defines a function to generate
// charset-conversion readers, converting from the provided
// charset into UTF-8.
// Charsets are always lower-case. utf-8, iso-8859-1 and us-ascii charsets
// are handled by default.
// One of the the CharsetReader's result values must be non-nil.
CharsetReader func(charset string, input io.Reader) (io.Reader, error)
}
I am wondering is there any library which can allow me to convert arbitrary charset like the "decode" function in python as shown in the above example. I don't want to write a big "switch-case"like the one being used in mime/encodedword.go:
func (d *WordDecoder) convert(buf *bytes.Buffer, charset string, content []byte) error {
switch {
case strings.EqualFold("utf-8", charset):
buf.Write(content)
case strings.EqualFold("iso-8859-1", charset):
for _, c := range content {
buf.WriteRune(rune(c))
}
....
Any help would be very appreciated.
Thanks.
I'm not sure it is what you are looking for but there is golang.org/x/text
package which I'm using to convert Windows-1251 to UTF-8. Code looks like
import (
"golang.org/x/text/encoding/charmap"
"golang.org/x/text/transform"
"io/ioutil"
"strings"
)
func convert(s string) string {
sr := strings.NewReader(s)
tr := transform.NewReader(sr, charmap.Windows1251.NewDecoder())
buf, err := ioutil.ReadAll(tr)
if err != nil {
return ""
}
return string(buf)
}
I think in your case if you want to avoid "a big 'switch-case'" you can create kind of map with full list of available encodings and just make something like:
var encodings = map[string]transform.Transformer{
"win-1251": charmap.Windows1251.NewDecoder(),
}
func convert(s, charset string) string {
buf, err := ioutil.ReadAll(transform.NewReader(strings.NewReader(s), encodings[charset]))
if err != nil {
return ""
}
return string(buf)
}
Thanks. It seems that the package golang.org/x/net/html/charset already provided a map with available encoding. The following code works for me:
import "golang.org/x/net/html/charset"
CharsetReader := func (label string, input io.Reader) (io.Reader, error) {
label = strings.Replace(label, "windows-", "cp", -1)
encoding, _ := charset.Lookup(label)
return encoding.NewDecoder().Reader(input), nil
}
dec := mime.WordDecoder{CharsetReader: CharsetReader}
text := "=?iso-2022-jp?b?GyRCRW1CQE86GyhCIDxtb21vQHRhcm8ubmUuanA=?="
header, err := dec.DecodeHeader(text)
Thanks for your help!