Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:121860
MIME-Version: 1.0
References: <CAJmy8YFy5RmW1yWuA6akPU31xRz_i1io5uWgt1GWo0n5rzY9EQ@mail.gmail.com>
 <1BA05C1A-AFAE-4E86-BAA2-420B22549519@gmail.com> <0D8856BC-DDEE-47F8-8C59-7F4DC7A64237@woofle.net>
 <CAGBsUrd=5WhKERySavP=9yQM-RSkVCwhTJ2oQ4tAQOKTNFoCew@mail.gmail.com>
 <A00495F3-EEB5-4D49-9433-84E94C6920D0@gmail.com> <CAJmy8YG91ki8M-n7id=DUDUsG+1XBvtpSNdf6sJx3Y1B1Rytsg@mail.gmail.com>
 <CAJmy8YF=cpXJAWsUKh-eYBembDAWu+X4_q24qV_xfTJLzAztiw@mail.gmail.com>
 <CAEPPVa2xgCh-Y2ovTL3=gX5vtTyiSAiNDnKLWPpohYCJ5Tr-cQ@mail.gmail.com>
 <CAJmy8YFsqvbBGTQJGUiBfKH2CoJ1sHdJYfindhtQKiUwubJwpw@mail.gmail.com>
 <CAEPPVa3ZNeA4dy4uCmh1ppxaiZ_wbRNMdd9wnQX5UQTZmkcB=w@mail.gmail.com>
 <CAJmy8YG4oBn1za1JaMpANXm+uugrwdMLCM5xO12eTjytkQK4Pw@mail.gmail.com>
 <CAEPPVa0SKpp9fX2QY5dxy0gxNg-Frk2pvDRzHt4=+YP_dEgFSA@mail.gmail.com>
 <CAEPPVa1WjRpwhqVEa9_3=OOL+b2DiG2Obb-=CHqMKnsVRwp+nQ@mail.gmail.com>
 <CAJmy8YEzyQzO1mPgpsTrErLTEL5Tos886-doMqZAYxA-UwMf0w@mail.gmail.com> <CAEPPVa0UTH_n=Rjo51-5c8N0OkpG6tQfQ38TyBw7oMao8yXMYQ@mail.gmail.com>
In-Reply-To: <CAEPPVa0UTH_n=Rjo51-5c8N0OkpG6tQfQ38TyBw7oMao8yXMYQ@mail.gmail.com>
Date: Wed, 29 Nov 2023 20:07:06 +0700
Message-ID: <CANLbj-pxX0fRHj=E2ZKUAeRw-oX5WzL-vteurS989zzMriTL8A@mail.gmail.com>
To: youkidearitai <youkidearitai@gmail.com>
Cc: internals@lists.php.net
Content-Type: text/plain; charset="UTF-8"
Subject: Re: [PHP-DEV] Deprecate declare(encoding='...') + zend.multibyte +
 zend.script_encoding + zend.detect_unicode ?
From: ayesh@php.watch (Ayesh Karunaratne)

> Shift_JIS is very ambiguous, What will we do if SJIS-2004 or SJIS-win comes?
> How do we guess(detect) SJIS-2004, SJIS-win and SJIS-mac?

I'm not the person you replied to in your previous email, but I
thought to weigh in with what I can. My native language also uses
multiple bytes, and have done a fair bit of character encoding
conversions from one to another.

The very reason why we have character encoding sets is to be able to
reassign the same byte values to multiple real-life characters, so
changing the character encodings from a non-UTF charset always carries
some sort of "risk" of detecting the wrong source text encoding. Like
Yuya Hamada mentioned in the rest of the previous email, 0xFC40 for
example can map to two different characters. These are quite common
occurrences, and there is even a word (Mojibake) for it!

The most robust projects in this space are probably `enca` and
`Chardet` (Python). However, theoretically, all tools can only guess
the text encoding by inspecting common patterns and by checking if all
bytes map to a meaningful glyph. When there is not a lot of text to
inspect, these tools are very prone to make wrong results.

When the source encoding is correctly detected or known, it's easy to
re-encode files using `iconv`, followed by a quick `sed` to remove the
`declare()` calls.

---

That said, I'm hugely in favor of dropping support for non-UTF8
encodings. Because the source encoding is present in the INI settings
or the declare statement, the site owners should be able to
mass-encode text to UTF-8. Many languages like Rust only support UTF-8
(https://doc.rust-lang.org/reference/input-format.html), and I don't
think any new PHP developers will expect PHP to work with non-UTF8
encodings in the first place.