The Text Encoding Submenu Is Gone

For context, please see Character Encoding Menu in 2014, Text Encoding Menu in 2021, and A Look at Encoding Detection and Encoding Menu Telemetry from Firefox 86.

Firefox 91 was released two weeks ago. This is the first release that does not have a Text Encoding submenu. Instead, the submenu has been replaced with a single menu item called to Repair Text Encoding. It performs the action that was previously performed by the item Automatic in the Text Encoding submenu: It runs chardetng with UTF-8 as a permitted outcome and ignoring the top-level domain.

The Repair Text Encoding menu item is in the View menu, which is hidden by default on Windows and Linux. The action is also available as an optional toolbar button (invoke the context menu on empty space in the toolbar and choose Customize Toolbar…). On Windows and Linux, you can invoke the menu item from the keyboard by pressing the v key while holding the alt key and then pressing the c key. (The keys may vary with the localization.)

What Problem Does “Repair Text Encoding” Solve?

  1. Sometimes the declared encoding is wrong, and the Web Platform would become more brittle if we started second-guessing the declared encoding automatically without user action.

    The typical case is that university faculty have created content over the years that is still worthwhile to read, and the old content is in a legacy encoding. However, independently of the faculty, the administrator has either explicitly or as a side effect of server software update caused the server configuration to claim UTF-8 server-wide even though this is wrong for old content. When the context is in the Latin script, the result is still readable. When the content is in a non-Latin script, the result is completely unreadable (without this feature).

  2. For non-Latin scripts, unlabeled UTF-8 is completely unreadable. Fixing this problem without requiring user action and also without making the Web Platform more brittle is a hard problem. There is a separate write-up on that topic alone. This problem might get solved one day in a way that does not involve user action but not today.

Why Remove the Other Submenu Items?

Why Not Remove the Whole Thing?

Chrome removed their menu altogether as part of what they called Project Eraser. (Amusingly, this lead to a different department of Google publishing a support article about using other browsers to access this functionality.) Mobile versions of Chrome, Safari, and Firefox don’t have the menu, either. So why not just follow Chrome?

Every time something in this area breaks intentionally or accidentally, feedback from Japan shows up relatively quickly. That’s the main reason why I believe users in Japan still care about having the ability to override the encoding of misconfigured pages. (That’s without articulating any particular numeric telemetry threshold for keeping the feature. However, telemetry confirms that the feature is relevant to the largest number of distinct telemetry submitters, both in absolute numbers and in region-total-relative numbers, in Japan.)

If we removed the feature, we’d remove a reason for these users to stay with Firefox. Safari and Gnome Web still have more elaborate encoding override UI built in (the list of encodings in both is questionably curated but the lists satisfy the Japanese use cases), and there are extensions for Chrome.

Shouldn’t This Be an Extension?

The built-in UI action in Firefox is more discoverable, more usable, and safer against the user getting baited into self-XSS than the Chrome extensions. Retaining the safety properties but moving the UI to an extension would increase implementation complexity while reducing discoverability—i.e. would help fewer users at a higher cost.

Removing the engine feature and leaving to an extension to rewrite headers of the HTTP responses (as in Chrome) would: