Deprecate declare(encoding='...') + zend.multibyte + zend.script_encoding + zend.detect_unicode ?

btw if we come to some consensus to my php2utf8.php script is actually
worthwhile to expand on, i can volunteer to add more encodings (SJIS,
BIG5, anything supported by mbstring),
but it wouldn't surprise me if a better approach exist and the script
should be rewritten entirely~

add that what's special about UTF-8 isn't that it's "fixed-endian".

should've added this to the last post, but the "zend.detect_unicode"
ini-option is specifically to scan for BOMs, and BOMs are
significantly less useful in fixed-endian encodings (like UTF8) than
bi-endian encodings (like UTF16/UTF32) ^^

What is the migration path for legacy code that use those directives?

The migration path is to convert the legacy-encoding PHP files to UTF-8.
Luckily this can be largely automated, here is my attempt:
https://github.com/divinity76/php2utf8/blob/main/src/php2utf8.php
but that code definitely needs some proof-reading and additions - idk
if the approach used is even a good approach, it was just the first i
could think of, feel free to write one from scratch

Can you share a little more details about how this works?

I hope someone else can do that, but it allows PHP to parse and
execute scripts not written in UTF-8 and scripts utilizing
BOM/byte-order-masks.

add that what's special about UTF-8 isn't that it's "fixed-endian".

one of multiple good things about UTF-8 is that it's fixed-endian, and
UTF8 don't need a BOM to specify endianess (unlike UTF16 and UTF32
which are bi-endian, and a BOM helps identify endianess used~)

If the solution is as easy as just converting the encoding of the
source file, then why did we even need to have this setting at all?
Why did PHP parser support encodings that demanded the introduction of

I've read your question but don't have an answer to it, hopefully
someone else knows.

Le 28 nov. 2023 à 20:56, Kamil Tekiela tekiela246@gmail.com a écrit :

Convert your PHP source files to UTF-8.

If the solution is as easy as just converting the encoding of the
source file, then why did we even need to have this setting at all?
Why did PHP parser support encodings that demanded the introduction of
this declare?

It is not necessary as simple: because your code base may contain literal strings, and changing the encoding of the source file will effectively change the contents of the strings.

—Claude

On Tue, Nov 28, 2023 at 12:48 PM Hans Henrik Bergan
divinity76@gmail.com wrote:

If the solution is as easy as just converting the encoding of the
source file, then why did we even need to have this setting at all?
Why did PHP parser support encodings that demanded the introduction of

I've read your question but don't have an answer to it, hopefully
someone else knows.

These settings predate the ubiquity of UTF-8, which did not begin to
see widespread adoption until the mid-to-late 2000s, and did not reach
ubiquity until the mid-2010s:
https://en.wikipedia.org/wiki/Popularity_of_text_encodings

mbstring.script_encoding was introduced with this commit and released
in PHP 4.3 (renamed to zend.script_encoding in PHP 5.4):
https://github.com/php/php-src/commit/f30b722f14521fbad2fabe5fdcaa2b60fe97eebb

zend.detect_unicode introduced in this commit, released with PHP 5.1:
https://github.com/php/php-src/commit/a8c6b992b8894763c59276c1142971aa9a314500

zend.multibyte introduced with this commit, released with PHP 5.4:
https://github.com/php/php-src/commit/ab93d8c621645e05d6a6a431d52ac64eda956673

declare(encoding) appears to predate all of the PHP 4.0 tagged
releases, including the pre-release ones.

• Mark Trapp

Le 28 nov. 2023 à 21:47, Hans Henrik Bergan divinity76@gmail.com a écrit :

What is the migration path for legacy code that use those directives?

The migration path is to convert the legacy-encoding PHP files to UTF-8.
Luckily this can be largely automated, here is my attempt:
https://github.com/divinity76/php2utf8/blob/main/src/php2utf8.php
but that code definitely needs some proof-reading and additions - idk
if the approach used is even a good approach, it was just the first i
could think of, feel free to write one from scratch

Hi,

Converting the character encoding of php files is by no means sufficient, except in the simplest cases.

Strings of text are to be found in various places, such as:

1. in the php files, as literals;
2. inside memory, at runtime;
3. in non-php data files stored on the server;
4. in the database;
5. as presented to the user (e.g. html document) and as received from them (e.g. form submission);
6. etc.

If you change the character encoding in (1), you necessarily change the encoding in (2), unless you wrap your literals with some function that performs the conversion in the other direction at runtime. And if you change the encoding in (2), you should be very careful when your text flows from and to (3), (4), (5) and (6): you should either change the encoding at those places, or make sure that proper conversion is done at the boundaries of those domains.

Also, mechanical conversion is not the whole story. For example, if you change the encoding in (5), you should not forget to adapt the <meta charset> tag and/or the content-type http header.

Also, all strings are not text, and only a human can decide whether the literal “\xe9” in a random location is meant to encode the raw byte 0xE9 or the character “é” in latin-1.

Of course, because we live in an interesting world, there will be situations where the encoding is unknown or ambiguous. Yuya mentioned the case of Shift-JIS which has various incompatible variants, and I am happy not to have encountered such ambiguities (only unknownnesses) when I converted our code base from windows-1252 (aka latin-1) to utf-8 a few years ago.

—Claude

The migration path is to convert the legacy-encoding PHP files to UTF-8.

Please take a look at the following code.
This is a part of the code that I am actually maintaining in the
latest version of php.

<?php
pg_connect(/* omission */);

// The database server expects clients to perform queries in SJIS.
// Depending on the settings, it may not be necessary to specify it explicitly.
pg_set_client_encoding('SJIS');

$res = pg_query('select * from 表');

Unfortunately, this code breaks if I simply convert it to UTF-8.

In the "Usage statistics of character encodings for websites"
published by W3Techs, it is true that encodings other than UTF-8 are
rarely used. However, this is only within the range that can be
observed from the outside as a website .

As the code above shows, PHP covers a much wider area. In addition to
external connections, for example, SimpleXML and DOMDocument also
handle character codes internally, so they can break down using the
same logic as in the example above.

As Yuya says, the conversion itself is difficult, and even if you can
convert it, it may not be enough, so as a php user from a culture that
uses multi-byte characters, please be aware of this.

Kentaro Takeda