Streaming UTF-8 (with node.js)

http://debuggable.com/posts/streaming-utf-8-with-node-js:4bf28e8b-a290-432f-a222-11c1cbdd56cb

by felixge • 16 years ago

22 13 16 years ago

debuggable.com

axod3• 16 years ago

slightly related - quick way to get the number of bytes in a string in javascript when the string is encoded in utf8:

var utf8_length = encodeURIComponent(data).replace(/%../g, 'x').length;

felixge3• 16 years ago

Don't use this, it's broken. Any % sign in your text will screw things. 3-4 byte characters are not handled by it at all.

axod3• 16 years ago

Care to elaborate?

% signs are handled fine - they get encoded into %25 which then gets shortened to a single 'x' character - thus counted as a single byte.

Similarly, for example € gets encoded to "%E2%82%AC" which then gets correctly counted as 3 bytes.

I'm not quite sure what value of 'broken' you're thinking of? Did you try this and find it didn't work in some particular circumstance?

BrianHammond• 16 years ago

For those using Node.js, the same is

    var len = Buffer.byteLength(a_string, 'utf8')

Sephr• 16 years ago

This script is unnecessary, as JavaScript already has functions which can facillitate encoding and decoding of UTF-8. Encoding UTF-8 to a bytestring is unescape(encodeURIComponent(string)) and decoding UTF-8 is decodeURIComponent(escape(string)).

div• 16 years ago

But this script handles streaming utf-8 encoded strings correctly.

codehero• 16 years ago

code does not work, does not check for invalid UTF-8 characters

felixge3• 16 years ago

That is done in Buffer.toString(), this stream filter just makes sure that no string conversion is attempted on a multibyte character that is still incomplete (valid or not).

oscarduignan2• 16 years ago

What an awesome RSS feed icon they have, if you haven't already, try hovering over it a few times (and then a few more.)

riobard• 16 years ago

What's the YouTube video about? It says the video is not available.

oscarduignan2• 16 years ago

I didn't stick around after reading the title when the youtube page loaded, however I'm pretty sure it was Rick Astley singing Never Gonna Give You Up.

Streaming UTF-8 (with node.js)

Related Stories