[RFC] Locale-independent case conversion

I wonder if either JIT could be used for the intrinsics support,
adding neon, sse[2-4.2] or avc256/512 (the latter would basically
allow most common strings to be converted in one go.

If not, maybe split implementation however runtime cpu support would
be better. Many distributions built with SSE2 flags but are actually
ran on much recent CPUs, same for ARM (ie. graviton/neon f.e.).
Thoughts?

SSE2 is already scary-fast. I benchmarked the SSE2 tolower code on my
laptop: it was chewing through strings at 18 GiB/s, i.e. 50ps per
byte. This was a benchmark written in C -- you would have a lot of
trouble making a PHP tight loop in which SSE2 case conversion is the
slow part.

-- Tim Starling

Am 04.10.2021 um 12:08 schrieb Nikita Popov nikita.ppv@gmail.com:

On Thu, Sep 23, 2021 at 8:32 AM Tim Starling <tstarling@wikimedia.org mailto:tstarling@wikimedia.org>
wrote:

Please consider my RFC for locale-independent case conversion.

https://wiki.php.net/rfc/strtolower-ascii
https://github.com/php/php-src/pull/7506

The RFC and associated PR ended up going some way beyond the original
scope, because for consistency, it's best if everything has the same
concept of case folding. I saw this as an opportunity to clean up a
common kind of locale-dependence in PHP which was previously inconsistent.

So not only will strtolower() and strtoupper() become
locale-independent, converting only ASCII, but also stristr, stripos,
strripos, lcfirst, ucfirst, ucwords, str_ireplace, the array sorting
functions with SORT_FLAG_CASE, and array_change_key_case.

Also, I changed a number of internal functions to use ASCII case
folding, giving rise to a range of effects in callers throughout the
core tree. The effects are all documented in the RFC.

I am proposing that locale-sensitive case conversion be provided with
the new names ctype_tolower() and ctype_toupper(). Those names might
seem odd at first glance, but they are wrappers for functions in
ctype.h and work in a very similar way to the rest of the ctype extension.

Hi Tim,

Thanks for creating this proposal, it looks great!

I think this is a very beneficial change, and the amount of incorrect
locale-dependent calls we had just in php-src further convinced me of this:
We're generally aware of the problem, and we still made this mistake. Many
times.

I definitely agree that it's good to make these functions locale-insentive.

The only open question I have is regarding the ctype_* functions. One might
argue that these functions should be locale-independent as well. Certainly,
whenever I have used ctype_digit() I only intended it to match [0-9]. It
seems like some people try to use ctype_alpha() in a locale-sensitive way (
https://stackoverflow.com/questions/19929965/php-setlocale-not-working-for-ctype-alpha-check https://stackoverflow.com/questions/19929965/php-setlocale-not-working-for-ctype-alpha-check)
and then fail because it doesn't support UTF-8.

On that topic, do we also want to add mb_ucfrist, mb_lcfirst and mb_ucwords?
Then you have also proper local independent functions handling these use cases.

This is only tangentially related to the current RFC, so feel free to ignore this.

Bob

Hi Tim,

Thanks for creating this proposal, it looks great!

I think this is a very beneficial change, and the amount of
incorrect locale-dependent calls we had just in php-src further
convinced me of this: We're generally aware of the problem, and we
still made this mistake. Many times.

The only open question I have is regarding the ctype_* functions.
One might argue that these functions should be locale-independent as
well. Certainly, whenever I have used ctype_digit() I only intended
it to match [0-9]. It seems like some people try to use
ctype_alpha() in a locale-sensitive way
(https://stackoverflow.com/questions/19929965/php-setlocale-not-working-for-ctype-alpha-check
https://stackoverflow.com/questions/19929965/php-setlocale-not-working-for-ctype-alpha-check)
and then fail because it doesn't support UTF-8.

OK, I removed ctype_tolower() and ctype_toupper() from the RFC and the
PR since they would be incompatible with a move towards a
locale-independent ctype extension.

The non-controversial parts of the PR were split and merged, so I
rebased the PR and updated the RFC accordingly.

Do you think the RFC is ready for voting now?

PS: Regarding escapeshellarg(), are you aware of the array command
support for proc_open() that was added in PHP 7.4? That does away
the need to escape arguments.

It doesn't really help us. I recently wrote a new shell command
execution system for MediaWiki called Shellbox. As part of that
project, I reviewed how shell execution is used in the MediaWiki
ecosystem. There are a lot of callers which are using shell features,
for example redirecting inputs or outputs, or constructing pipelines.
I didn't really want to break them all or reimplement those features
without the shell. And we have security and containerization wrappers
which depend on construction of a shell command string. So we need to
be able to construct shell command strings safely.

After studying locale sensitivity for this RFC, I decided to get rid
of escapeshellarg() from MediaWiki. Instead we are doing our own shell
escaping:

https://gerrit.wikimedia.org/r/c/mediawiki/libs/Shellbox/+/722548

I also made MediaWiki use a fixed locale, instead of being configurable.

-- Tim Starling