Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:113651
To: internals@lists.php.net
References: <f313c9c4-f8a2-0b39-7499-30620d80cecd@gmail.com>
 <3a4d89fc-c5f8-4720-b2e0-f6f3c28684f9@www.fastmail.com>
Message-ID: <5f5fd136-e181-d5d3-fe40-1a4cc5c668f2@gmail.com>
Date: Sun, 21 Mar 2021 18:13:11 +0000
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.8.1
MIME-Version: 1.0
In-Reply-To: <3a4d89fc-c5f8-4720-b2e0-f6f3c28684f9@www.fastmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-GB
Subject: Re: [PHP-DEV] What should we do with utf8_encode and utf8_decode?
From: rowan.collins@gmail.com (Rowan Tommins)

On 21/03/2021 16:51, Larry Garfield wrote:
> As Rowan notes, what people actually*want*  most of the time is "I got this string from a user and have NFI what it's encoding is, but my system needs UTF-8, so gimmie this string in UTF-8."  And they use utf8_encode(), which then fails*sometimes*  in exciting and mysterious ways, because that's not what it is.
>
> [...]
>
> If we're removing a bad answer to the problem, we should also replace it with a good answer.


This is indeed my main concern with complete deprecation. The problem is 
that detecting string encoding is a Really Hard Problem™

The fundamental problem is that any sequence of bytes is valid in any 
single-byte encoding. If you're expecting printable characters only, you 
can rule out some candidates if you're lucky - e.g. if your string 
contains a byte in the range 0x80 to 0x9F, it's not any part of ISO 8859 
- but the string "\xB0\xC0\xD0" is both valid and printable in any of 
dozens of 8-bit encodings.

I recently came across a Python library implementing a clever approach 
to the problem, which originated at Mozilla. Its concise FAQ is worth 
reading: https://chardet.readthedocs.io/en/latest/faq.html The approach 
Mozilla came up with is to decide which encoding leads to something most 
likely to be natural human text - e.g. don't suggest an encoding common 
for Cyrillic if the result would be completely unpronounceable in Russian.


The only function I know of which even attempts encoding detection in 
PHP is mb_detect_encoding, and it does a pretty bad job. For instance:

echo mb_detect_encoding("\x80500", ['Windows-1252', 'ISO-8859-15', 
'ISO-8859-1']);

...picks ISO-8859-15, where 0x80 is a rarely-used control character, 
rather than Windows-1252, where it's the Euro symbol.


On the other hand, if you know what encoding you do have, either of the 
following will work fine:

echo mb_convert_encoding("\x80500", 'UTF-8', 'Windows-1252');
echo iconv('Windows-1252', 'UTF-8', "\x80500");

Either of these functions (passed ISO-8859-1) can be used as a polyfill 
for correct uses of utf8_encode/utf8_decode, but neither is going to do 
the magic trick which people always *hope* those functions will.


Regards,

-- 
Rowan Tommins
[IMSoP]