# What should we do with utf8_encode and utf8_decode?

- Thread: https://externals.io/message/113645
- Original archive: https://news-web.php.net/php.internals/113645
- Started by Rowan Tommins on 2021-03-21T14:18:20+00:00
- Messages: 40
- RFC: Deprecate and remove utf8_decode() and utf8_encode() (Implemented (PHP 8.2)) — https://externals.io/rfc/remove_utf8_decode_and_utf8_encode.md
- Page 1 of 2: messages 1 to 25 (25 per page, add `?page=N` to the URL)
- Next page: https://externals.io/message/113645.md?page=2
- Reply by email to internals@lists.php.net
- More for agents: search with https://externals.io/search.md?q={query} · RFCs at https://externals.io/rfc.md · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt

---

## #113645 — Rowan Tommins — 2021-03-21T14:18:20+00:00

URL: https://externals.io/message/113645#113645

Hi all,

The functions utf8_encode and utf8_decode are historical oddities, which 
almost certainly would not be accepted if proposed today:

* Their names do not describe their functionality, which is to convert 
to/from one specific single-byte encoding. This leads to a common 
confusion that they can be used to "fix" UTF-8 encoding problems, which 
they generally make worse.
* That single-byte encoding is ISO 8859-1, not its common cousins 
Windows-1252 or ISO 88159-15. This means, for instance, that they do not 
handle the Euro sign: utf8_decode('€') returns '?' (i.e. unmappable)  
not "\x80" (Windows-1252) or "\xA4" (8859-15)

On the other hand, they are commonly used, both correctly and 
incorrectly, so removing them is not easy.

A previous proposal to remove them [1] resulted in Andrea making two 
significant improvements: moving them from ext/xml to ext/standard [2] 
and rewriting the documentation to explain them properly [3]. My genuine 
thanks for that.

However, it hasn't stopped people misunderstanding them, and quite 
reasonably: you shouldn't need to look up every function you use in the 
manual, to make sure it actually does what its name suggests.


I can see three ways forward:

A) Raise a deprecation notice in 8.1, and remove in 9.0. Do not provide 
a specific replacement, but recommend people look at `iconv()` or 
 `mb_convert_encoding()`. There is precedent for this, such as 
convert_cyr_string(), but it may frustrate those who are using the 
functions correctly.

B) Introduce new names, such as utf8_to_iso_8859_1 and 
iso_8859_1_to_utf8; immediately make those the primary names in the 
manual, with utf8_encode / utf8_decode as aliases. Raise deprecation 
notices for the old names, either immediately or in some future release. 
This gives a smoother upgrade path, but commits us to having these 
functions as outliers in our standard library.

C) Leave them alone forever. Treat it as the user's fault if they mess 
things up by misunderstanding them.


I am happy to put together an RFC for either A or B, if it has a chance 
of reaching consensus. I would really like to avoid option C.


[1] https://externals.io/message/95166
[2] https://github.com/php/php-src/pull/2160
[3] 
https://github.com/php/doc-en/commit/838941f6cce51f3beda16012eb497b26295a8238

Regards,

-- 
Rowan Tommins
[IMSoP]

---

## #113646 — Benjamin Morel — 2021-03-21T14:32:36+00:00

URL: https://externals.io/message/113645#113646 · In reply to: #113645

> I can see three ways forward:
>
> A) Raise a deprecation notice in 8.1, and remove in 9.0. Do not provide
> a specific replacement, but recommend people look at `iconv()` or
> `mb_convert_encoding()`. There is precedent for this, such as
> convert_cyr_string(), but it may frustrate those who are using the
> functions correctly.
>
> B) Introduce new names, such as utf8_to_iso_8859_1 and
> iso_8859_1_to_utf8; immediately make those the primary names in the
> manual, with utf8_encode / utf8_decode as aliases. Raise deprecation
> notices for the old names, either immediately or in some future release.
> This gives a smoother upgrade path, but commits us to having these
> functions as outliers in our standard library.
>

Hi, I'm personally fine with A or B, both of which have pros & cons:

- A is probably the cleanest way as, as you said, these functions should
never have existed (locked to a single encoding that will only benefit a
portion of users), but that's quite a BC break
- B has is less of a BC break as it gives users a chance to rename their
function calls, but leaves an oddity in the standard library

I'm a bit worried that either way, we'll start seeing some "polyfills"
appear on Packagist to re-introduce the old functions, but at least they
will be gone from the core.

— Benjamin

---

## #113647 — Ayesh Karunaratne — 2021-03-21T15:01:06+00:00

URL: https://externals.io/message/113645#113647 · In reply to: #113645

Thank you for opening this conversation, these functions have stung me
in the past, and I would be so happy to see them gone :)

Personally, I would very much like to go with Plan A.

- XML parsers that often deal with non-UTF-8 character encodings
frequently use these functions. However, any parser worth their salt
is better off using mbstring or iconv because of the lack of
Windows-1252 support that is assumed elsewhere for ISO-8859. If we
have a `utf8_encode` that supports Windows-1252 as often expected, I
think plan B would be the more smoother upgrade.

 - On Packagist top 1000 downloads, stripe-php, phpcpd, pdepend,
carbon, monolog, php-cs-fixer, htmlpurifier, and aws-php-sdk use
`utf8_encode`. Some of these libraries depend on `ext-mbstring` or
Symfony mbstring polyfill, so we are left with even fewer libraries
that cannot assume `iconv()` or `mb_convert_encoding` availability.


>
> Hi all,
>
> The functions utf8_encode and utf8_decode are historical oddities, which
> almost certainly would not be accepted if proposed today:
>
> * Their names do not describe their functionality, which is to convert
> to/from one specific single-byte encoding. This leads to a common
> confusion that they can be used to "fix" UTF-8 encoding problems, which
> they generally make worse.
> * That single-byte encoding is ISO 8859-1, not its common cousins
> Windows-1252 or ISO 88159-15. This means, for instance, that they do not
> handle the Euro sign: utf8_decode('€') returns '?' (i.e. unmappable)
> not "\x80" (Windows-1252) or "\xA4" (8859-15)
>
> On the other hand, they are commonly used, both correctly and
> incorrectly, so removing them is not easy.
>
> A previous proposal to remove them [1] resulted in Andrea making two
> significant improvements: moving them from ext/xml to ext/standard [2]
> and rewriting the documentation to explain them properly [3]. My genuine
> thanks for that.
>
> However, it hasn't stopped people misunderstanding them, and quite
> reasonably: you shouldn't need to look up every function you use in the
> manual, to make sure it actually does what its name suggests.
>
>
> I can see three ways forward:
>
> A) Raise a deprecation notice in 8.1, and remove in 9.0. Do not provide
> a specific replacement, but recommend people look at `iconv()` or
> `mb_convert_encoding()`. There is precedent for this, such as
> convert_cyr_string(), but it may frustrate those who are using the
> functions correctly.
>
> B) Introduce new names, such as utf8_to_iso_8859_1 and
> iso_8859_1_to_utf8; immediately make those the primary names in the
> manual, with utf8_encode / utf8_decode as aliases. Raise deprecation
> notices for the old names, either immediately or in some future release.
> This gives a smoother upgrade path, but commits us to having these
> functions as outliers in our standard library.
>
> C) Leave them alone forever. Treat it as the user's fault if they mess
> things up by misunderstanding them.
>
>
> I am happy to put together an RFC for either A or B, if it has a chance
> of reaching consensus. I would really like to avoid option C.
>
>
> [1] https://externals.io/message/95166
> [2] https://github.com/php/php-src/pull/2160
> [3]
> https://github.com/php/doc-en/commit/838941f6cce51f3beda16012eb497b26295a8238
>
> Regards,
>
> --
> Rowan Tommins
> [IMSoP]
>
> --
> 
> To unsubscribe, visit: https://www.php.net/unsub.php

---

## #113648 — Ben Ramsey — 2021-03-21T15:55:04+00:00

URL: https://externals.io/message/113645#113648 · In reply to: #113646

> 
> 
> 
> 
> > […]
> 
> Hi, I'm personally fine with A or B, both of which have pros & cons:
> 
> - A is probably the cleanest way as, as you said, these functions should
> never have existed (locked to a single encoding that will only benefit a
> portion of users), but that's quite a BC break
> - B has is less of a BC break as it gives users a chance to rename their
> function calls, but leaves an oddity in the standard library
> 
> I'm a bit worried that either way, we'll start seeing some "polyfills"
> appear on Packagist to re-introduce the old functions, but at least they
> will be gone from the core.


I prefer option A, and the emergence of userland polyfills doesn’t worry
me. IMO, that’s the right way for the community to handle the BC break.

Cheers,
Ben

---

## #113650 — Larry Garfield — 2021-03-21T16:51:25+00:00

URL: https://externals.io/message/113645#113650 · In reply to: #113645

> Hi all,
> 
> The functions utf8_encode and utf8_decode are historical oddities, which 
> almost certainly would not be accepted if proposed today:
> 
> * Their names do not describe their functionality, which is to convert 
> to/from one specific single-byte encoding. This leads to a common 
> confusion that they can be used to "fix" UTF-8 encoding problems, which 
> they generally make worse.
> * That single-byte encoding is ISO 8859-1, not its common cousins 
> Windows-1252 or ISO 88159-15. This means, for instance, that they do not 
> handle the Euro sign: utf8_decode('€') returns '?' (i.e. unmappable)  
> not "\x80" (Windows-1252) or "\xA4" (8859-15)
> 
> On the other hand, they are commonly used, both correctly and 
> incorrectly, so removing them is not easy.
> 
> A previous proposal to remove them [1] resulted in Andrea making two 
> significant improvements: moving them from ext/xml to ext/standard [2] 
> and rewriting the documentation to explain them properly [3]. My genuine 
> thanks for that.
> 
> However, it hasn't stopped people misunderstanding them, and quite 
> reasonably: you shouldn't need to look up every function you use in the 
> manual, to make sure it actually does what its name suggests.
> 
> 
> I can see three ways forward:
> 
> A) Raise a deprecation notice in 8.1, and remove in 9.0. Do not provide 
> a specific replacement, but recommend people look at `iconv()` or 
> `mb_convert_encoding()`. There is precedent for this, such as 
> convert_cyr_string(), but it may frustrate those who are using the 
> functions correctly.
> 
> B) Introduce new names, such as utf8_to_iso_8859_1 and 
> iso_8859_1_to_utf8; immediately make those the primary names in the 
> manual, with utf8_encode / utf8_decode as aliases. Raise deprecation 
> notices for the old names, either immediately or in some future release. 
> This gives a smoother upgrade path, but commits us to having these 
> functions as outliers in our standard library.
> 
> C) Leave them alone forever. Treat it as the user's fault if they mess 
> things up by misunderstanding them.
> 
> 
> I am happy to put together an RFC for either A or B, if it has a chance 
> of reaching consensus. I would really like to avoid option C.
> 
> 
> [1] https://externals.io/message/95166
> [2] https://github.com/php/php-src/pull/2160
> [3] 
> https://github.com/php/doc-en/commit/838941f6cce51f3beda16012eb497b26295a8238
> 
> Regards,

I lost several days of my life to exactly this problem, many years ago.  I am still triggered by it.

I am mostly OK with option A, but with a big caveat:

The root problem here is "You keep using that function.  I do not think it means what you think it means."

As Rowan notes, what people actually *want* most of the time is "I got this string from a user and have NFI what it's encoding is, but my system needs UTF-8, so gimmie this string in UTF-8."  And they use `utf8_encode()`, which then fails *sometimes* in exciting and mysterious ways, because that's not what it is.

Removing `utf8_encode()` may keep people from misusing it, but that doesn't mean the problem space they were trying to solve goes away.  If anything, people who still don't realize that it's the wrong solution will get angry that we're taking away a "useful" tool and replacing it with "meh, go look at library X," which is admittedly a pretty rude answer.

If we're removing a bad answer to the problem, we should also replace it with a good answer.

Someone will, I'm sure, pop in at this point and declare "if you don't know the character encoding you're receiving, then you're doing it wrong and are already lost and we can't help you."  While that may be technically correct, it's also an entirely useless answer because strings received over HTTP very frequently do not tell you what their encoding is, or they lie about what their encoding is.  (The header may say it's ISO8859, or UTF8, or whatever, but someone copy-pasted from MS Word into a text box and now it's Windows-1252 within a wrapper that says ISO8859 but is mostly UTF8 except for the Windows-1252 part.  Like, that's literally the problem I lost several days to.)  "Your own fault" is not even an accurate answer at that point.

So if we're going to take away people's broken hammer, we need to be very clear about what hammer to use instead.

The initial answer is probably "here's how to use a series of mb_string functions together to produce a reasonably good guess-my-encoding-and-convert-to-utf8 routine" documentation.  Which... may exist, but if it does I've never found it.  So at bare minimum the encode_utf8() documentation needs to include a "use this code snippet instead" description, and not just link to the mbstring extension.  Glancing through the mbstring docs right now, it looks like it's not already a single function call, but some combination of several, and has some global flags that get set (via `mb_detect_order()`), I think.  It's not as easy to use as `utf8_encode()`, even if `utf8_encode()` is wrong.  That suggests we may want to try and simplify the mbstring API, or internalize some function that handles the most common case in a way that doesn't rely on global flags.

So, let's make that easier to use, so that we can change "this function is wrong, we're taking it away from you" to "this function is wrong, here's a way better alternative that you can use instead (while we quietly take the wrong one away from you while you're distracted by the new shiny)."

I don't know the mbstring API well enough to say what that alternative ideally looks like, but if we can answer that it would make killing off the old functions much more palatable.

--Larry Garfield

---

## #113651 — Rowan Tommins — 2021-03-21T18:13:11+00:00

URL: https://externals.io/message/113645#113651 · In reply to: #113650

> As Rowan notes, what people actually*want*  most of the time is "I got this string from a user and have NFI what it's encoding is, but my system needs UTF-8, so gimmie this string in UTF-8."  And they use `utf8_encode()`, which then fails*sometimes*  in exciting and mysterious ways, because that's not what it is.
>
> [...]
>
> If we're removing a bad answer to the problem, we should also replace it with a good answer.


This is indeed my main concern with complete deprecation. The problem is 
that detecting string encoding is a Really Hard Problem™

The fundamental problem is that any sequence of bytes is valid in any 
single-byte encoding. If you're expecting printable characters only, you 
can rule out some candidates if you're lucky - e.g. if your string 
contains a byte in the range 0x80 to 0x9F, it's not any part of ISO 8859 
- but the string "\xB0\xC0\xD0" is both valid and printable in any of 
dozens of 8-bit encodings.

I recently came across a Python library implementing a clever approach 
to the problem, which originated at Mozilla. Its concise FAQ is worth 
reading: https://chardet.readthedocs.io/en/latest/faq.html The approach 
Mozilla came up with is to decide which encoding leads to something most 
likely to be natural human text - e.g. don't suggest an encoding common 
for Cyrillic if the result would be completely unpronounceable in Russian.


The only function I know of which even attempts encoding detection in 
PHP is mb_detect_encoding, and it does a pretty bad job. For instance:

echo mb_detect_encoding("\x80500", ['Windows-1252', 'ISO-8859-15', 
'ISO-8859-1']);

...picks ISO-8859-15, where 0x80 is a rarely-used control character, 
rather than Windows-1252, where it's the Euro symbol.


On the other hand, if you know what encoding you do have, either of the 
following will work fine:

echo mb_convert_encoding("\x80500", 'UTF-8', 'Windows-1252');
echo iconv('Windows-1252', 'UTF-8', "\x80500");

Either of these functions (passed ISO-8859-1) can be used as a polyfill 
for correct uses of utf8_encode/utf8_decode, but neither is going to do 
the magic trick which people always *hope* those functions will.


Regards,

-- 
Rowan Tommins
[IMSoP]

---

## #113652 — Kamil Tekiela — 2021-03-21T19:08:02+00:00

URL: https://externals.io/message/113645#113652 · In reply to: #113651

Option A, please.

I have never had a reason to use either of these two functions. I assume
there's plenty of valid applications for converting between ISO-8859-1 and
UTF-8, but that function causes more harm than good.
I have seen plenty of people use it, but I have never seen anyone use it
properly. Most of the time people use it to fix their mojibake text when
they forget to set the connection charset in PDO or mysqli. I was a little
surprised to learn that these functions had something to do with XML.

The reason why I consider them dangerous is that people using them are most
likely solving the wrong problem. The problem isn't the conversion from ISO
to UTF but having the text in the wrong format in the first place. They are
used as some kind of magical solution that fixes an annoying problem. I
would have no quarrel with them if they were named correctly though.
Another reason why I do not like these functions is that they let you shoot
yourself in the foot very easily. They don't warn about invalid or missing
code points, which often leads to more data corruption. When doing the same
with ICONV you at least get a notice.

I think we really do not need to keep these functions. As for the
alternative that we can offer, iconv seems to be doing exactly the same
thing and even better. mb_convert_encoding does the same but also silently
ignores invalid characters. So we already offer plenty of alternatives. We
don't need to add anything new.

-- Kamil

---

## #113653 — Max Semenik — 2021-03-21T21:00:37+00:00

URL: https://externals.io/message/113645#113653 · In reply to: #113652

> I think we really do not need to keep these functions. As for the
> alternative that we can offer, iconv seems to be doing exactly the same
> thing and even better. mb_convert_encoding does the same but also silently
> ignores invalid characters. So we already offer plenty of alternatives. We
> don't need to add anything new.
>

Just a quick reminder that it's possible to compile PHP without mbstring
and intl, which means that some hosts will provide PHP without these
extensions, and some packagers make them available as separate packages
that users can't or don't know how to install. Maybe we've got an
opportunity to think about making these extensions mandatory?

-- 
Best regards,
Max Semenik

---

## #113654 — Rowan Tommins — 2021-03-21T21:39:14+00:00

URL: https://externals.io/message/113645#113654 · In reply to: #113653

> Just a quick reminder that it's possible to compile PHP without 
> mbstring and intl, which means that some hosts will provide PHP 
> without these extensions, and some packagers make them available as 
> separate packages that users can't or don't know how to install. Maybe 
> we've got an opportunity to think about making these extensions mandatory?


It's somewhat relevant that until PHP 7.2, it was also possible for 
utf8_encode and utf8_decode to be missing, because they were in ext/xml, 
which is also optional.

Bundling mbstring sounds great, until you look into the details of 
what's in there and how it works. Its origin as a PHP 4 extension for 
handling Japanese-specific character encodings is visible in parts of 
its design - there's a lot of global state, and very little support for 
the nuances of Unicode.

Bundling intl would be great, but it's a wrapper around ICU, which is 
huge (because Unicode is complicated). I have read that incorporating 
that into core was one of the icebergs that sunk PHP 6. It's also 
extremely sparsely documented (if someone's looking for a project, it 
would be great to fill in all the manual stubs with a few details from 
the corresponding ICU documentation).

For what its worth, it seems these would be the relevant polyfills:

function utf8_encode(string $string) { return 
UConverter::transcode($string, 'UTF8', 'ISO-8859-1'); }
function utf8_decode(string $string) { return 
UConverter::transcode($string, 'ISO-8859-1', 'UTF8'); }


Regards,

-- 
Rowan Tommins
[IMSoP]

---

## #113655 — Sara Golemon — 2021-03-22T01:15:18+00:00

URL: https://externals.io/message/113645#113655 · In reply to: #113645

On Sun, Mar 21, 2021 at 9:18 AM Rowan Tommins <rowan.collins@gmail.com>
wrote:

> A) Raise a deprecation notice in 8.1, and remove in 9.0. Do not provide
> a specific replacement, but recommend people look at `iconv()` or
> `mb_convert_encoding()`. There is precedent for this, such as
> convert_cyr_string(), but it may frustrate those who are using the
> functions correctly.
>
> B) Introduce new names, such as utf8_to_iso_8859_1 and
> iso_8859_1_to_utf8; immediately make those the primary names in the
> manual, with utf8_encode / utf8_decode as aliases. Raise deprecation
> notices for the old names, either immediately or in some future release.
> This gives a smoother upgrade path, but commits us to having these
> functions as outliers in our standard library.
>
> C) Leave them alone forever. Treat it as the user's fault if they mess
> things up by misunderstanding them.
>
>
My preference is for a deprecation notice (but not necessarily removal ever
-- We can argue that part a little).

As for what users should use instead, obviously there are multiple options
already in core (which you referenced), but those all have third party deps
and can't be guaranteed the way utf8_en/decode() can (this was the point of
moving them from xml).

While I'm normally in favor of userspace things belonging in userspace
(this particular conversion is trivial since it's a 1:1 mapping), I'm
actually willing to see this added under a new, clearer name in
ext/standard since this is something that's in long use, but used
incorrectly.

As for details, I don't love iso_8859_1_to_utf8(), but we can use the
common alias for iso-8859-1 known as latin1 and call the new functions:
utf8_from_latin1() and utf8_to_latin1() with the caveat that the later will
throw a ValueError for codepoints which are out of range (one of the more
problematic issues with `utf8_decode()`).  That makes this not just a simple
rename for clarity, but what I'd consider a bug-fix for an unfortunately
unfixable function.

-Sara

---

## #113656 — Hans Henrik Bergan — 2021-03-22T08:28:51+00:00

URL: https://externals.io/message/113645#113656 · In reply to: #113655

i would prefer to soft-deprecate them like we did with the mysql_ api,
where they do not generate `E_DEPRECATED` for quite some time, but the
documentation say
"this function is deprecated, instead use mb_convert_encoding ( $str ,
"UTF-8", "ISO-8859-1" );  or iconv("ISO-8859-1","UTF-8", $str)"
and.. make it go `E_DEPRECATED` in the distant future..


Rowan said "they are commonly used, both correctly and
incorrectly", in my experience, no it's not used correctly, people who are
using it, are using it incorrectly to convert Windows-1252 to utf-8, not
ISO-8859-1...





> On Sun, Mar 21, 2021 at 9:18 AM Rowan Tommins <rowan.collins@gmail.com>
> wrote:
>
> > […]
> My preference is for a deprecation notice (but not necessarily removal ever
> -- We can argue that part a little).
>
> As for what users should use instead, obviously there are multiple options
> already in core (which you referenced), but those all have third party deps
> and can't be guaranteed the way utf8_en/decode() can (this was the point of
> moving them from xml).
>
> While I'm normally in favor of userspace things belonging in userspace
> (this particular conversion is trivial since it's a 1:1 mapping), I'm
> actually willing to see this added under a new, clearer name in
> ext/standard since this is something that's in long use, but used
> incorrectly.
>
> As for details, I don't love iso_8859_1_to_utf8(), but we can use the
> common alias for iso-8859-1 known as latin1 and call the new functions:
> utf8_from_latin1() and utf8_to_latin1() with the caveat that the later will
> throw a ValueError for codepoints which are out of range (one of the more
> problematic issues with `utf8_decode()`).  That makes this not just a simple
> rename for clarity, but what I'd consider a bug-fix for an unfortunately
> unfixable function.
>
> -Sara

---

## #113660 — Rowan Tommins — 2021-03-22T10:24:00+00:00

URL: https://externals.io/message/113645#113660 · In reply to: #113655

> My preference is for a deprecation notice (but not necessarily removal 
> ever -- We can argue that part a little).


I'm strongly against any concept of "indefinite deprecation". I consider 
any deprecation notice a commitment to remove the feature in the future, 
even if a specific timeline for that removal is not given.

If we want to have a separate status of "will be kept indefinitely, but 
you shouldn't use it", then we need a separate E_DISCOURAGED, or some 
boilerplate in the manual which doesn't use the word "deprecated".


> As for details, I don't love iso_8859_1_to_utf8(), but we can use the 
> common alias for iso-8859-1 known as latin1 and call the new 
> functions: utf8_from_latin1() and utf8_to_latin1() with the caveat 
> that the later will throw a ValueError for codepoints which are out of 
> range (one of the more problematic issues with `utf8_decode()`). That 
> makes this not just a simple rename for clarity, but what I'd consider 
> a bug-fix for an unfortunately unfixable function.


While I can see the temptation here, I'm not sure who the target 
audience for the new function would be:

* People who just want to replace calls to utf8_decode won't want to go 
through every call and make it exception safe.
* People who want to write a polyfill couldn't use it, because they 
wouldn't be able to recover the remainder of the string after an error 
is thrown.
* People who want transcoding without any optional extensions will be 
disappointed to find only this one encoding supported.

You'd effectively be adding a completely new core function just for 
those people who work with Latin1 text, and are confident that it's not 
Windows-1252 in disguise.

It's tempting to make any C1 control characters an error as well - 
although technically valid in Latin1, these are very rarely used, and 
it's much more likely that any bytes in that range are intended as 
characters in Windows-1252. But that would feel very odd without having 
a corresponding utf8_from_windows1252 function to use instead, at which 
point we're into designing a whole new conversion library. And of 
course, once you've got that UTF-8 string, you can't do much with it, 
because PHP's native string functions are all byte-based, so you've 
basically got to re-invent large chunks of ext/mbstring...


Regards,

-- 
Rowan Tommins
[IMSoP]

---

## #113661 — bjorn.x.larsson@telia.com — 2021-03-22T10:28:40+00:00

URL: https://externals.io/message/113645#113661 · In reply to: #113654

Den 2021-03-21 kl. 22:39, skrev Rowan Tommins:
> 
> > […]
> 
> 
> It's somewhat relevant that until PHP 7.2, it was also possible for 
> utf8_encode and utf8_decode to be missing, because they were in ext/xml, 
> which is also optional.
> 
> Bundling mbstring sounds great, until you look into the details of 
> what's in there and how it works. Its origin as a PHP 4 extension for 
> handling Japanese-specific character encodings is visible in parts of 
> its design - there's a lot of global state, and very little support for 
> the nuances of Unicode.
> 
> Bundling intl would be great, but it's a wrapper around ICU, which is 
> huge (because Unicode is complicated). I have read that incorporating 
> that into core was one of the icebergs that sunk PHP 6. It's also 
> extremely sparsely documented (if someone's looking for a project, it 
> would be great to fill in all the manual stubs with a few details from 
> the corresponding ICU documentation).
> 
> For what its worth, it seems these would be the relevant polyfills:
> 
> function utf8_encode(string $string) { return 
> UConverter::transcode($string, 'UTF8', 'ISO-8859-1'); }
> function utf8_decode(string $string) { return 
> UConverter::transcode($string, 'ISO-8859-1', 'UTF8'); }
> 
> 
> Regards,
> 

In our case we use the utf8_decode functions to convert from UTF8 in
the client to ISO-8859-1 on the server, since the site is encoded in
latin1.

Our usage of that function is working flawlessly, so for us it's super
important to have a clear migration path with a good polyfill!

r//Björn L

---

## #113662 — Rowan Tommins — 2021-03-22T11:12:46+00:00

URL: https://externals.io/message/113645#113662 · In reply to: #113661

Hi Björn,


> In our case we use the utf8_decode functions to convert from UTF8 in
> the client to ISO-8859-1 on the server, since the site is encoded in
> latin1.
>
> Our usage of that function is working flawlessly, so for us it's super
> important to have a clear migration path with a good polyfill! 


I realise you can't speak for anyone else, but as a point of interest, 
would you be OK with a polyfill having a requirement on ext/mbstring or 
ext/iconv, or would you have a strong preference for a replacement built 
into the core (i.e. guaranteed available without any optional packages)?

Regards,

-- 
Rowan Tommins
[IMSoP]

---

## #113665 — bjorn.x.larsson@telia.com — 2021-03-22T12:58:06+00:00

URL: https://externals.io/message/113645#113665 · In reply to: #113662

Den 2021-03-22 kl. 12:12, skrev Rowan Tommins:
> Hi Björn,
> 
> 
> > […]
> 
> 
> I realise you can't speak for anyone else, but as a point of interest, 
> would you be OK with a polyfill having a requirement on ext/mbstring or 
> ext/iconv, or would you have a strong preference for a replacement built 
> into the core (i.e. guaranteed available without any optional packages)?
> 
> Regards,
> 

Well, both these extensions are part of our environment so I presume it
will also be so in the future.

Now if we have a polyfill dependent on these libraries it's a question
on how these libraries are maintained and that they are not EOL. Just 
speaking from a general point here. I'm in slight favour of mbstring,
since I have a small experience of it.

What's important for us is that the polyfill has a simple API and 
doesn't have any surprises / side effects. I think though there is
a case for improving these functions and keep them in the core.

We wrap these functions in one place so it's relatively easy to change 
the wrapper to accomodate new functionality in the utf8_* functions as
long as we get the same end result.

I also think one should consider which opensource libraries that are
using these functions. E.g. the Revive ad server v5.2 are using both.

r//Björn L

---

## #113666 — Sara Golemon — 2021-03-22T13:10:23+00:00

URL: https://externals.io/message/113645#113666 · In reply to: #113660

On Mon, Mar 22, 2021 at 5:24 AM Rowan Tommins <rowan.collins@gmail.com>
wrote:
> I'm strongly against any concept of "indefinite deprecation". I consider
> any deprecation notice a commitment to remove the feature in the future,
> even if a specific timeline for that removal is not given.
>

I don't feel strongly about indefinite deprecation.  If you wanna nuke it
in 9.0, have fun.  I'm just saying I don't necessarily see the need to do
so.  The problem being addressed here is that *some* users of this function
are probably misusing it, so it's worth putting guiderails on.  I'm
hesitant to punish the ones who know exactly what they're doing as a result
of that well-meaning intention.

> * People who just want to replace calls to utf8_decode won't want to go
> through every call and make it exception safe.
>

Then they shouldn't use these replacements, it's not for them. It's for
people using iso-8859-1.

> * People who want to write a polyfill couldn't use it, because they
> wouldn't be able to recover the remainder of the string after an error
> is thrown.
>

If you're writing a polyfill, then write a polyfill.   The polyfill for the
old functions is trivial, I could have written it a dozen times in the
course of writing this email reply.
So this replacement is also not for them.

> * People who want transcoding without any optional extensions will be
> disappointed to find only this one encoding supported.
>
This function isn't for them.It's for people using iso-8859-1.

There's a theme in here. :)

> You'd effectively be adding a completely new core function just for
> those people who work with Latin1 text, and are confident that it's not
> Windows-1252 in disguise.
>

Yes.  I'm specifically addressing the people who have been using
utf8_en/decode() correctly all this time.  They shouldn't be punished for
the stupidity of others.

> It's tempting to make any C1 control characters an error as well -
> although technically valid in Latin1, these are very rarely used, and
> it's much more likely that any bytes in that range are intended as
> characters in Windows-1252. But that would feel very odd without having
> a corresponding utf8_from_windows1252 function to use instead, at which
> point we're into designing a whole new conversion library. And of
> course, once you've got that UTF-8 string, you can't do much with it,
> because PHP's native string functions are all byte-based, so you've
> basically got to re-invent large chunks of ext/mbstring...
>

I disagree that you'd need to add utf8_from/to_windows1252 "for
completeness".  The goal isn't to provide all possible conversion
utilities.  The goal is only to not punish users by taking away a valid API
that they were using correctly (for those users who were using it
correctly).

> -Sara

---

## #113667 — Sara Golemon — 2021-03-22T13:13:50+00:00

URL: https://externals.io/message/113645#113667 · In reply to: #113662

On Mon, Mar 22, 2021 at 6:12 AM Rowan Tommins <rowan.collins@gmail.com>
wrote:
> I realise you can't speak for anyone else, but as a point of interest,
> would you be OK with a polyfill having a requirement on ext/mbstring or
> ext/iconv, or would you have a strong preference for a replacement built
> into the core (i.e. guaranteed available without any optional packages)?
>

Can you clarify what *YOU* mean by a polyfill?  Because you're talking
about dependence on iconv/mbstring/icu which implies you want a polyfill
that does something other than what the original utf8_en/decode() functions
do, and those functions certainly do not need external dependencies.
They're really just not that complex.

-Sara

---

## #113668 — Nicolas Grekas — 2021-03-22T13:18:42+00:00

URL: https://externals.io/message/113645#113668 · In reply to: #113667

Le lun. 22 mars 2021 à 14:14, Sara Golemon <pollita@php.net> a écrit :

> On Mon, Mar 22, 2021 at 6:12 AM Rowan Tommins <rowan.collins@gmail.com>
> wrote:
> > […]
>
> Can you clarify what *YOU* mean by a polyfill?  Because you're talking
> about dependence on iconv/mbstring/icu which implies you want a polyfill
> that does something other than what the original utf8_en/decode() functions
> do, and those functions certainly do not need external dependencies.
> They're really just not that complex.
>

Shameless plug: the polyfill exists, without any dependency, see
https://github.com/symfony/polyfill-php72/blob/main/Php72.php

;)
Nicolas

---

## #113669 — bjorn.x.larsson@telia.com — 2021-03-22T13:52:07+00:00

URL: https://externals.io/message/113645#113669 · In reply to: #113666

Den 2021-03-22 kl. 14:10, skrev Sara Golemon:
> On Mon, Mar 22, 2021 at 5:24 AM Rowan Tommins <rowan.collins@gmail.com>
> wrote:
> > […]
> 
> I don't feel strongly about indefinite deprecation.  If you wanna nuke it
> in 9.0, have fun.  I'm just saying I don't necessarily see the need to do
> so.  The problem being addressed here is that *some* users of this function
> are probably misusing it, so it's worth putting guiderails on.  I'm
> hesitant to punish the ones who know exactly what they're doing as a result
> of that well-meaning intention.
> 
> > […]
> 
> Then they shouldn't use these replacements, it's not for them. It's for
> people using iso-8859-1.
> 
> > […]
> 
> If you're writing a polyfill, then write a polyfill.   The polyfill for the
> old functions is trivial, I could have written it a dozen times in the
> course of writing this email reply.
> So this replacement is also not for them.
> 
> > […]
> This function isn't for them.It's for people using iso-8859-1.
> 
> There's a theme in here. :)
> 
> > […]
> 
> Yes.  I'm specifically addressing the people who have been using
> utf8_en/decode() correctly all this time.  They shouldn't be punished for
> the stupidity of others.
> 
> > […]
> 
> I disagree that you'd need to add utf8_from/to_windows1252 "for
> completeness".  The goal isn't to provide all possible conversion
> utilities.  The goal is only to not punish users by taking away a valid API
> that they were using correctly (for those users who were using it
> correctly).
> 
> > […]
> 

Think I'm one such user :-) So keeping them and improving a little would
be fine with me!

r//Björn L

---

## #113670 — Rowan Tommins — 2021-03-22T14:30:33+00:00

URL: https://externals.io/message/113645#113670 · In reply to: #113666

>
> > […]
>
> Then they shouldn't use these replacements, it's not for them. It's 
> for people using iso-8859-1.


This is a non-sequitur. Someone using the function correctly to convert 
to ISO 8859-1 may also be relying on the documented and consistent 
error-handling behaviour. Substituting the character may not always be 
the best approach, but in some cases it's more useful than discarding 
the entire string, let alone aborting the entire process with an 
unhandled Throwable.


> The goal is only to not punish users by taking away a valid API that 
> they were using correctly (for those users who were using it correctly).


I'm sympathetic to that aim, but if the new function is not the same, 
you *are* taking away the existing API, and introducing a new one. 
Neither of the following seems like it would be accepted:

- Make `utf8_decode()` throw errors for unrepresentable characters.
- Introduce a function specifically for converting from UTF-8 to 
Latin-1, if we didn't already have one.

So it feels questionable to me to design a new function, which is 
neither compatible with what we have, nor a reasonable addition on its 
own merits.


Regards,

-- 
Rowan Tommins
[IMSoP]

---

## #113671 — Rowan Tommins — 2021-03-22T14:32:20+00:00

URL: https://externals.io/message/113645#113671 · In reply to: #113668

>
> Shameless plug: the polyfill exists, without any dependency, see
> https://github.com/symfony/polyfill-php72/blob/main/Php72.php 
> <https://github.com/symfony/polyfill-php72/blob/main/Php72.php>


Ah, thanks for sharing that. I realised while trying to get to sleep 
that a pure-PHP implementation would be fairly straight-forward because 
of the relationship between Latin1 and Unicode.

Regards,

-- 
Rowan Tommins
[IMSoP]

---

## #113673 — Aleksander Machniak — 2021-03-22T15:04:35+00:00

URL: https://externals.io/message/113645#113673 · In reply to: #113670

> - Make `utf8_decode()` throw errors for unrepresentable characters.

I'm not sure I understand this, but it sounds like it would be a BC
break for my case.

I'm using `utf8_encode()`/utf8_decode() to make input string safe to be
stored in DB, and back. In most cases the input is utf-8, but it
occasionally may contain "broken characters".

    $str = '';
    for ($x=0; $x<256; $x++) {
        $str .= chr($x);
    }

    $this->assertSame($str, utf8_decode(utf8_encode($str)));

    $str = "グーグル谷歌中信фδοκιμήóźdźрöß😁😃";

    $this->assertSame($str, utf8_decode(utf8_encode($str)));

Could anyone point to a sample input that will not work with my use-case?

-- 
Aleksander Machniak
Kolab Groupware Developer        [https://kolab.org]
Roundcube Webmail Developer  [https://roundcube.net]
----------------------------------------------------
PGP: 19359DC1 # Blog: https://kolabian.wordpress.com

---

## #113675 — Kamil Tekiela — 2021-03-22T15:34:49+00:00

URL: https://externals.io/message/113645#113675 · In reply to: #113673

>
> I'm using `utf8_encode()`/utf8_decode() to make input string safe to be
> stored in DB, and back. In most cases the input is utf-8, but it
> occasionally may contain "broken characters".
>

What exactly do you mean by making the input string safe? If I understand
correctly utf8_decode(utf8_encode($str)) should just be an identity
function. Could you please explain what is the purpose of using these
functions in such a way?

---

## #113677 — Rowan Tommins — 2021-03-22T15:41:27+00:00

URL: https://externals.io/message/113645#113677 · In reply to: #113673

> I'm using `utf8_encode()`/utf8_decode() to make input string safe to be
> stored in DB, and back. In most cases the input is utf-8, but it
> occasionally may contain "broken characters".


That is not what this function does, at all. The fact that its name 
makes you think that is exactly why I want to get rid of that name.


>      $str = "グーグル谷歌中信фδοκιμήóźdźрöß😁😃";
>
>      $this->assertSame($str, utf8_decode(utf8_encode($str)));


Let's write that out with a more descriptive function name:

$str = "グーグル谷歌中信фδοκιμήóźdźрöß😁😃";

$this->assertSame($str, utf8_to_latin1(latin1_to_utf8($str)));


Since Latin-1 does not contain any Chinese, Japanese, or Emoji 
characters, running latin1_to_uft8 on that string is clearly nonsensical.

The only reason it doesn't give you any errors is that every possible 
byte is a valid character in Latin1, and every Latin1 character has a 
Unicode code point. So the "グ" is interpreted as three Latin-1 
characters: E3, 82, and B0; those then become the corresponding Unicode 
code points U+00E3, U+00821, and U+00B0, represented in UTF-8. You then 
run utf8_to_latin1, and they get converted back.

That code will never do anything useful.

Regards,

-- 
Rowan Tommins
[IMSoP]

---

## #113685 — Aleksander Machniak — 2021-03-22T16:52:00+00:00

URL: https://externals.io/message/113645#113685 · In reply to: #113677

> That code will never do anything useful.

I already proved it is useful, regardless of it's name/intention.

This is old code, not even mine, so maybe when it's been written the PHP
documentation wasn't that clear about the function(s) intention. Or the
intention was different.

ps. to Kamil,

We use `utf8_encode()` to make the string safe to be put in utf-8 database
column/table. We use `utf8_decode()` to convert that back to what it was
before.

The tests prove that the conversion is lossless.

-- 
Aleksander Machniak
Kolab Groupware Developer        [https://kolab.org]
Roundcube Webmail Developer  [https://roundcube.net]
----------------------------------------------------
PGP: 19359DC1 # Blog: https://kolabian.wordpress.com

---

Next page: https://externals.io/message/113645.md?page=2
