Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:121922
MIME-Version: 1.0
References: <CAMZNyFfiP5y8QAcNBCo235xG30Ra-PgCKcYfsZVuvgwy9Re-8w@mail.gmail.com>
 <CAFPFaMJeLD1WVzczjfXa4iKg0QtgKxhH0e3mzOMjsa9POf=C9w@mail.gmail.com>
 <CACsECNcbe+D=8WMa=LG-ts6_Rq1W91hneb43i07L58xC5WUt3w@mail.gmail.com>
 <CAMZNyFe0YywB5WGt3ga6s1SFOytNLFfF3=eEfMhnpWTDh24C9w@mail.gmail.com> <CAPzBOBNDhMkF1_qMHpSeZF=1+Gb8ZoobKnU7rLRuHN-9t3BGkQ@mail.gmail.com>
In-Reply-To: <CAPzBOBNDhMkF1_qMHpSeZF=1+Gb8ZoobKnU7rLRuHN-9t3BGkQ@mail.gmail.com>
Date: Mon, 4 Dec 2023 23:05:32 +0900
Message-ID: <CAEPPVa3rYBWokrFr--YAZFPfG5awNGetJDBUq3sW59JLy+7iwg@mail.gmail.com>
To: internals@lists.php.net
Cc: Alex <alexinbeijing@gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Subject: Re: [PHP-DEV] Inconsistency mbstring functions
From: youkidearitai@gmail.com (youkidearitai)

2023=E5=B9=B412=E6=9C=884=E6=97=A5(=E6=9C=88) 22:25 Robert Landers <landers=
.robert@gmail.com>:
>
> On Mon, Dec 4, 2023 at 1:51=E2=80=AFPM Stefan Schiller via internals
> <internals@lists.php.net> wrote:
> >
> > On Sat, Dec 2, 2023 at 6:13=E2=80=AFAM Alex <alexinbeijing@gmail.com> w=
rote:
> > >
> > > Dear Stefan, and Dear Gina,
> > >
> > > Thanks for the message. Yes, Stefan has rediscovered an interesting q=
uirk of the mbstring library. I have been aware of this for a long time, an=
d other mbstring developers have too. It dates back to the origin of the li=
brary; actually, even before the origin of mbstring, since mbstring was bas=
ed on another library called libmbfl, and this behavior originates from lib=
mbfl.
> > >
> > > Pull your chair up around the fire and let me tell you the tale of li=
bmbfl. Once upon a time, there was a text-processing library called libmbfl=
. libmbfl was based on a collection of text-decoding routines (which conver=
ted bytes to codepoints) and text-encoding routines (which converted codepo=
ints to bytes). Each such routine was structured as a stateful "filter". Th=
ese filters could be assembled into "chains", whereby the output values gen=
erated by one routine would automatically be passed to the next. libmbfl co=
uld perform many wonderful text-processing tasks by substituting a differen=
t final filter at the end of the chain.
> > >
> > > But all was not well. Since libmbfl's filters processed text only one=
 byte or codepoint at a time, and each routine had to save its state before=
 returning, and restore its state upon entry, libmbfl was slow. Slow as a t=
urtle, slow as a snail, slow as whatever-slowly-moving-thing-you-can-think-=
of. Oh, what was libmbfl to do? A clever plan was hatched: give libmbfl a 2=
56-entry table called a "mblen_table" for each supported text encoding with=
 the property that the byte length of a character can be determined from it=
s first byte. Then, text-processing tasks which were not dependent on the a=
ctual content of a string, but only on the number of codepoints, could be p=
erformed without ever invoking those wonderful, but painfully slow filters!=
 libmbfl could skip through a string while just examining the first byte of=
 each character. (Of course, this only worked for text encodings with an mb=
len_table.) For valid strings, the new method worked identically to the pre=
vious one. For invalid strings, there were significant differences in behav=
ior, but libmbfl tried to ignore these and bravely pressed on.
> > >
> > > The story ends with an ironic twist. Many years later, I became inter=
ested in mbstring and reimplemented its internals, replacing the libmbfl co=
de with fresh new code which ran many times faster. The new code was so muc=
h faster that in some cases, the mblen_table optimization actually became a=
 pessimization! In other cases, the mblen_table-based code is still faster,=
 but not by a large amount. But now mbstring was haunted by the spectre of =
Hyrum's Law (https://www.hyrumslaw.com/). With a huge body of legacy code r=
elying on mbstring, almost any observable behavior change runs a significan=
t risk of breaking someone's code. And when this happens, they will not hes=
itate to vent their rage on the hapless maintainers.
> > >
> > > Notwithstanding the rage of the users, about a year ago, I did remove=
 the mblen_table-based code in one place where benchmarks clearly showed it=
 was acting as a pessimization. I don't remember which mbstring function wa=
s affected and would need to check the commit log to confirm.
> >
> > Hi Alex,
> >
> > Thank you very much for sharing this background context.
> >
> > >
> > > Personally, I think the real issue here is not the inconsistency betw=
een mbstring functions which are based on the mblen_tables and those which =
are not. I think a lot of mbstring operations should not be used on invalid=
 strings at all, and that for such operations, mbstring would do well to th=
row an exception if it receives invalid input. (Like mb_strpos; how do you =
define the "position of a UTF-8 substring" when the parent string is not UT=
F-8 at all?) But that would be a huge BC break.
> > >
> >
> > My biggest concern is that this quirk can cause security issues in
> > user code. I came across this in the first place when discovering an
> > exploitable security vulnerability in an application. From my point of
> > view, this is not only about inconsistent behavior but also violates
> > the documentation for specific functions like mb_strstr. I agree that
> > a lot of mbstring operations should not be used on invalid strings,
> > and an exception seems to be an appropriate answer despite the huge BC
> > impact.
>
> I think it is only a security issue when people accidentally think
> mb_* functions should be used if it is available. I've seen people do
> mb_strlen() on binary data, for example, not realizing the differences
> between mb_strlen and strlen. Or using mb_* functions and then passing
> them off to cryptographic functions.
>
> Robert Landers
> Software Engineer
> Utrecht NL
>
> --
> PHP Internals - PHP Runtime Development Mailing List
> To unsubscribe, visit: https://www.php.net/unsub.php
>

Hi, Internals.

Sorry if I'm off topic.
I don't know if it will be helpful, Japanese mbstring user if use
these mb_* functions, we use mb_check_encoding.
If character encoding is invalid, then occur error.

```
<?php

if (mb_check_encoding($string, $encoding) !=3D=3D true)) {
    // throw Exception or exit(not zero)
}

mb_strlen($string);
// do something
```

Anyway, I agree if invalid encoding then mbstring is raise exception.
Because not only people who are familiar with character codes.
However, this is BC break, I think need more discussion.

Regards
Yuya

--=20
---------------------------
Yuya Hamada (tekimen)
- https://tekitoh-memdhoi.info
- https://github.com/youkidearitai
-----------------------------