Smart quotes, curly quotes, UTF8, and web posting errors

David Traver

Sometimes people like to write their postings in Word or some other word processor and then cut and paste it onto the web. This is usually done for speed, ease of use, and for spell checking. That happens here with at least two frequent posters. Because Word and other document generators use a text coding other than UTF8, when the cut and paste is performed, the curly quotes and "smart quotes, apostrophes, etc. turn into junk. This happens because the web is changing over to UTF8 character encoding, which has a transliteration mismatch with the word processors for those characters. That is, an "a" is an "a" in both systems, but a curly quote does not match up with the right character in UTF8. There are right and left quotes in UTF8 but they do not line up with the same codes used by Word and other processors. Why? Why should they?

Often you will see these untranslated characters show up as some sort of black diamond with a strange character inside, or a short string of incomprehensible characters where a quote or apostrophe should be, such as â€œ and â€?. This might not show up in the document when it is first posted, but users of RSS feeds and readers with one of the many popular browsers will see variations of your posting that make it somewhat goofy. This started happening on Connect when we upgraded our server last month. The new character set used on Connect is UTF8. I had to go though all of my blog items (Such as "experts" and "welcome" in the left menu and remove the non-UTF8 characters. This was a bummer, and I missed a few no doubt.) UTF8 is very compatible with non-traditional browsers such as cell phones and aids for the blind. So, by using UTF8, we are making Connect more accessible.

The solution for posters is to turn off the smart quotes and curly quotes in your word processors. Then, the problem goes away when you cut and paste because you are not moving the poorly translatable characters from one realm to another.

Another handy way to do this is to use a word processor that supports UTF8 character encoding from the getgo.

Also, you can save your document in html format selecting the UTF8 character set in the dropdown box at the bottom of the save screen (depending on what kind of word processor you use.)

In any event, to learn more, search Google for curly quotes, UTF8, word, etc. and you will likely find a solution that fits the bill.

See generally, http://std.dkuug.dk/JTC1/SC2/WG2/docs/n2043.pdf