Text Read from Any Encoding

Text rarely reaches a program as letters. It arrives as bytes, from a file, a database or another program, and the same sentence gives different bytes depending on the encoding it was saved in. Here one line holding a euro sign, a dash and a check mark is written as bytes seven ways, in UTF-8 and in the several forms of UTF-16 and UTF-32, with and without the mark that announces them. Each is read back and set on the page, and all seven come out the same. A character cut in half is refused rather than set as a wrong letter.

Some text is written in plain letters only, with a short code standing in for every other sign: one for the euro sign, one for a dash, one for a check mark, one for a Greek letter. One such line is read with each kind of code on its own, with none and with all four, and a code that stands for nothing is left as it was written. Last, the word café is written twice, once with its é as a single character and once as an e followed by an accent, and the page says which form each one is in.

What This Example Shows

  • One line received in seven encodings, set the same each time.
  • The codes of a plain-letter line read, or left as they are.
  • An accented letter as one character or as two.

The Source Code, Complete

Nothing is left out and nothing is simplified: this is the program exactly as it runs, for the developer who uses this library.

Read the Rust source Read the Python source

The Document Itself

Open the document in English Open the document in French Open the document in German Open the document in Spanish Open the document in Italian Open the document in Portuguese

Text Read from Any Encoding, first page
Open the PDF Download PDF

The Same Document in Other Languages

This document weighs 16.8 kB

See the size of every example document.

Examples Near This One

Breaking an Overlong Word

A word longer than the column that holds it: the library leaves it whole, cuts it at the column's edge, or cuts it and closes the line on a hyphen.

Break Points After a Slash or a Hyphen

A justified column set four times: file paths and compounds running past the edge with no break points, wrapping where a break after a slash or a hyphen is allowed, then wherever their punctuation allows one, and last with no-break hyphens that keep each compound whole.

A Stack Laid out by Hand, One Box at a Time

A delivery slip poured into two boxes, the second taking up where the first stopped, and a poster whose picture no box can hold, each box saying what became of the stack in it.

Every example, by subject