SMS, Twilio, and special characters

So a client of mine uses twilio to send SMS messages to its users. The client would create a product and the user could get more information about it from a text message. This was working fine until the need came to selling some Mourvèdre wine. When we tested it out it came out like this:

MourvIdre

Now, the è exists in the UTF8 character set but when I tried to send it as a UTF8 character, the è would become a question mark. Attempting to encode via iconv, etc all either created a garbage message, a question mark, or an “I”. I would then try to manually insert UTF8 characters, emoji, Japanese characters into the message, and those would come out fine, but trying to send Mourvèdre would still not work. Then I tried sending this as a body:

“Mourvèdre 🐶”

And what came out on my phone looked like this:

Mourvèdre 🐶 

It worked!, I tried simply removing the dog from the same text string, and it comes out as:

MourvIdre

So it would appear that sending a string with special characters via Twilio would require it to contain a UTF8 character above the normal ranges. This essentially puts the message in a “UTF8-mode”. But also UTF8 characters below a certain range, such as è would not trigger this mode. Our solution was to send the hair space character (\x200A) with the messages, to ensure it gets sent as UTF-8. However, adding this created another problem.

For phones, the \x200A is not treated as a space, and so if the message contained a link at the end, the \x200A character would be part of the link, and when clicked would likely result in a 404. Even better. After much hair-pulling as to why our SMS links stopped working, we ended up putting a space between the original message and the hair space character. It’s good to know about this workaround just in case you need to send SMS mesages with a variety of characters.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.