Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:121862
Message-ID: <2FF6C60C-300D-4900-84B8-0DF4311143A7@gmail.com>
Content-Type: multipart/alternative;
	boundary="Apple-Mail=_14D56F02-1DBB-48CD-87A5-DDE57C4CD5C1"
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\))
Date: Wed, 29 Nov 2023 21:14:13 +0100
In-Reply-To: <CAJmy8YG91ki8M-n7id=DUDUsG+1XBvtpSNdf6sJx3Y1B1Rytsg@mail.gmail.com>
Cc: Kamil Tekiela <tekiela246@gmail.com>,
 Dusk <dusk@woofle.net>,
 PHP internals <internals@lists.php.net>
To: Hans Henrik Bergan <divinity76@gmail.com>
References: <CAJmy8YFy5RmW1yWuA6akPU31xRz_i1io5uWgt1GWo0n5rzY9EQ@mail.gmail.com>
 <1BA05C1A-AFAE-4E86-BAA2-420B22549519@gmail.com>
 <0D8856BC-DDEE-47F8-8C59-7F4DC7A64237@woofle.net>
 <CAGBsUrd=5WhKERySavP=9yQM-RSkVCwhTJ2oQ4tAQOKTNFoCew@mail.gmail.com>
 <A00495F3-EEB5-4D49-9433-84E94C6920D0@gmail.com>
 <CAJmy8YG91ki8M-n7id=DUDUsG+1XBvtpSNdf6sJx3Y1B1Rytsg@mail.gmail.com>
Subject: Re: [PHP-DEV] Deprecate declare(encoding='...') + zend.multibyte +
 zend.script_encoding + zend.detect_unicode ?
From: claude.pache@gmail.com (Claude Pache)

--Apple-Mail=_14D56F02-1DBB-48CD-87A5-DDE57C4CD5C1
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8



> Le 28 nov. 2023 =C3=A0 21:47, Hans Henrik Bergan =
<divinity76@gmail.com> a =C3=A9crit :
>=20
>> What is the migration path for legacy code that use those directives?
>=20
> The migration path is to convert the legacy-encoding PHP files to =
UTF-8.
> Luckily this can be largely automated, here is my attempt:
> https://github.com/divinity76/php2utf8/blob/main/src/php2utf8.php
> but that code definitely needs some proof-reading and additions - idk
> if the approach used is even a good approach, it was just the first i
> could think of, feel free to write one from scratch

Hi,

Converting the character encoding of php files is by no means =
sufficient, except in the simplest cases.

Strings of text are to be found in various places, such as:

1. in the php files, as literals;
2. inside memory, at runtime;
3. in non-php data files stored on the server;
4. in the database;
5. as presented to the user (e.g. html document) and as received from =
them (e.g. form submission);
6. etc.

If you change the character encoding in (1), you necessarily change the =
encoding in (2), unless you wrap your literals with some function that =
performs the conversion in the other direction at runtime. And if you =
change the encoding in (2), you should be very careful when your text =
flows from and to (3), (4), (5) and (6): you should either change the =
encoding at those places, or make sure that proper conversion is done at =
the boundaries of those domains.

Also, mechanical conversion is not the whole story. For example, if you =
change the encoding in (5), you should not forget to adapt the <meta =
charset> tag and/or the content-type http header.

Also, all strings are not text, and only a human can decide whether the =
literal =E2=80=9C\xe9=E2=80=9D in a random location is meant to encode =
the raw byte 0xE9 or the character =E2=80=9C=C3=A9=E2=80=9D in latin-1.

Of course, because we live in an interesting world, there will be =
situations where the encoding is unknown or ambiguous. Yuya mentioned =
the case of Shift-JIS which has various incompatible variants, and I am =
happy not to have encountered such ambiguities (only unknownnesses) when =
I converted our code base from windows-1252 (aka latin-1) to utf-8 a few =
years ago.

=E2=80=94Claude


--Apple-Mail=_14D56F02-1DBB-48CD-87A5-DDE57C4CD5C1--