Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:125009
Feedback-ID: id5114917:Fastmail
Date: Sat, 17 Aug 2024 00:37:43 +0100
To: internals@lists.php.net
Subject: =?US-ASCII?Q?Re=3A_=5BPHP-DEV=5D=5BDiscussion=5D_Should_All_S?=
 =?US-ASCII?Q?tring_Functions_Become_Multi-Byte_Safe=3F?=
User-Agent: K-9 Mail for Android
In-Reply-To: <8360937cc7ca31bf3bd0f8e3050c53cb32663428.camel@ageofdream.com>
References: <CAPn3h8LunRRAEfQXbfuDzMsxwGeFZfsHXs7oB=DTMhN2fgphUg@mail.gmail.com> <1AFE8300-D363-43D8-A989-15D001B9879C@newclarity.net> <dc2d9656-557a-4bf2-9971-245b30b1aa30@rwec.co.uk> <270D6057-626D-4720-B44A-3CB7A7B9320B@newclarity.net> <BC11D97A-F56E-4C22-ABCE-C4A44680BCA7@rwec.co.uk> <8360937cc7ca31bf3bd0f8e3050c53cb32663428.camel@ageofdream.com>
Message-ID: <36BC79B6-718D-4A01-B23C-8F0C652ED1C2@rwec.co.uk>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain;
 charset=utf-8
Content-Transfer-Encoding: quoted-printable
From: imsop.php@rwec.co.uk ("Rowan Tommins [IMSoP]")



On 16 August 2024 23:11:18 BST, Nick Lockheart <lists@ageofdream=2Ecom> wr=
ote:
>I used the rather broad title "Should All String Functions Become
>Multi-Byte Safe" because there are many smaller related topics, but my
>intention was to discuss multi-byte in general

I think it was probably not the best choice, because it seems like what yo=
u're specifically interested in is mostly not about existing functions, and=
 not particularly about encodings being more than one byte wide=2E

For instance, even good old 7-bit-per-character ASCII contains control cha=
racters you might want help sanitising out; and plenty of 8-bit-per-charact=
er encodings include more than one script, even more than one writing direc=
tion (e=2Eg=2E ISO 8859-8 Latin/Hebrew)=2E

But, the specific topic of safe input handling is definitely an interestin=
g one=2E And focussing on Unicode, rather than every possible encoding (mul=
tibyte or not) makes sense in modern usage=2E


>There's a lot of potential pitfalls for dealing with Unicode input, and
>there are some best practices per the Unicode Consortium

It's worth looking into whether the ICU library has explicit functions to =
help with those recommendations (if you can navigate its slightly patchy do=
cumentation)=2E Since most of ext/intl is just a thin wrapper on that libra=
ry, that could make our lives a lot easier=2E


>For example, there should be a function that removes unassigned code
>points=2E
>
>There should also be a function that removes "scripts" (as defined by
>Unicode)=2E
>
>We should have an easy way to remove private use code points (unless
>you're running a Star Trek fan site and really do need Klingon)=2E

These all seem like good ideas=2E I think you can do at least some of it w=
ith regular expressions, but dedicated functions have potential to be both =
easier to use and more efficient=2E


>And the default replacement character for `mb_scrub` shouldn't be `?`=2E

This is trickier, and where mixing the terms "multibyte" and "Unicode" act=
ually matters=2E The mbstring extension supports a number of different text=
 encodings, most of which don't have a dedicated replacement character to u=
se=2E It also has the ability to set the default in global state with mb_su=
bstitute_character() so it's not immediately obvious how a different defaul=
t could be applied based on the specified encoding=2E (I'm not a fan of tha=
t API design, but it's what we've got!)


>Each of these and other ideas could be part of an RFC, or we could
>brainstorm a Unicode built-in class that handles lots of the common use
>cases=2E

I don't think a single class that tries to "do Unicode" makes sense; it wo=
uld be like having a "maths class" that contains methods for anything deali=
ng with numbers=2E

In fact, I think the group of functions you're suggesting are a great illu=
stration of what I was saying in my last message to Rob: they make perfect =
sense as standalone features, and don't need any grand plan to "have Unicod=
e in core" before we proceed with them=2E

Regards,
Rowan Tommins
[IMSoP]