# [VOTE] Locale-independent case conversion

- Thread: https://externals.io/message/116497
- Original archive: https://news-web.php.net/php.internals/116497
- Started by Tim Starling on 2021-11-25T05:05:37+00:00
- Messages: 15
- RFC: Locale-independent case conversion (Implemented (PHP 8.2)) — https://externals.io/rfc/strtolower-ascii.md
- Reply by email to internals@lists.php.net
- More for agents: search with https://externals.io/search.md?q={query} · RFCs at https://externals.io/rfc.md · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt

---

## #116497 — Tim Starling — 2021-11-25T05:05:37+00:00

URL: https://externals.io/message/116497#116497

Voting is now open for my RFC on locale-independent case conversion.

https://wiki.php.net/rfc/strtolower-ascii

Voting will close in two weeks, on 2021-12-09.

-- Tim Starling

---

## #116499 — Nicolas Grekas — 2021-11-25T08:55:36+00:00

URL: https://externals.io/message/116497#116499 · In reply to: #116497

Le jeu. 25 nov. 2021 à 06:05, Tim Starling <tstarling@wikimedia.org> a
écrit :

> Voting is now open for my RFC on locale-independent case conversion.
>
> https://wiki.php.net/rfc/strtolower-ascii
>
> Voting will close in two weeks, on 2021-12-09.
>

Hi Tim,

I voted yes because I want to see this happen but I raised a point in
https://externals.io/message/116141#116259 and didn't get an answer:

Despite their name, I never used natcase functions for natural language
> processing. I use them eg to sort lists of files in a directory, to account
> for numbers mainly. But that's not what I would call natural language
> processing. I'm not aware of anyone using them for that actually. I'm
> wondering if it's a good idea to postpone migrating them to an hypothetical
> future as to me, the whole reasoning of the RFC applies to them.
>

I wish the `strnatcasecmp()` and `natcasesort()` function, but also the
 `SORT_NATURAL` flag were also covered by this RFC.
Is that possible? Would it make sense?

Nicolas

---

## #116500 — come@chilliet.eu — 2021-11-25T08:57:33+00:00

URL: https://externals.io/message/116497#116500 · In reply to: #116497

Le jeudi 25 novembre 2021, 06:05:37 CET Tim Starling a écrit :
> Voting is now open for my RFC on locale-independent case conversion.
> 
> https://wiki.php.net/rfc/strtolower-ascii

Hello,

The RFC is missing information about alternatives:
Do all of these function have an mbstring version?
Are those locale dependant or have an option for it?

To reuse the example from the RFC, if I want to convert a UTF string to uppercase using Turkish rules and get dotted capital I, what should I use?

Côme

---

## #116501 — Hans Henrik Bergan — 2021-11-25T09:08:32+00:00

URL: https://externals.io/message/116497#116501 · In reply to: #116500

btw why is this code *not* getting dotted capital i on 3v4l?
https://3v4l.org/D1WG1#v7.4.26
it gets ["res_hex"]=> string(2) "49"

<?php
setlocale(LC_ALL, "Turkish");
$str="i";
$res=strtoupper($str);
var_dump([
    "str"=>$str,
    "str_hex"=>bin2hex($str),
    "res"=>$res,
    "res_hex"=>bin2hex($res),
]);
?>



> Le jeudi 25 novembre 2021, 06:05:37 CET Tim Starling a écrit :
> > […]
>
> Hello,
>
> The RFC is missing information about alternatives:
> Do all of these function have an mbstring version?
> Are those locale dependant or have an option for it?
>
> To reuse the example from the RFC, if I want to convert a UTF string to
> uppercase using Turkish rules and get dotted capital I, what should I use?
>
> Côme
>
> --
> 
> To unsubscribe, visit: https://www.php.net/unsub.php

---

## #116502 — Dusk — 2021-11-25T09:17:59+00:00

URL: https://externals.io/message/116497#116502 · In reply to: #116501

> btw why is this code *not* getting dotted capital i on 3v4l?
> https://3v4l.org/D1WG1#v7.4.26
> it gets ["res_hex"]=> string(2) "49"
> 
> <?php
> setlocale(LC_ALL, "Turkish");

Because "Turkish" isn't a locale. "tr_TR" is.

https://3v4l.org/GD91W#v7.4.26

Notice that the output doesn't show up correctly, as it is not UTF-8. (Which is part of the problem addressed by this RFC.)

---

## #116503 — Tim Starling — 2021-11-25T09:47:21+00:00

URL: https://externals.io/message/116497#116503 · In reply to: #116499

>
> I voted yes because I want to see this happen but I raised a point
> in https://externals.io/message/116141#116259
> <https://externals.io/message/116141#116259> and didn't get an answer:
>
>     Despite their name, I never used natcase functions for natural
>     language
>     processing. I use them eg to sort lists of files in a directory,
>     to account
>     for numbers mainly. But that's not what I would call natural
>     language
>     processing. I'm not aware of anyone using them for that
>     actually. I'm
>     wondering if it's a good idea to postpone migrating them to an
>     hypothetical
>     future as to me, the whole reasoning of the RFC applies to them.
>
>
> I wish the `strnatcasecmp()` and `natcasesort()` function, but also the
> `SORT_NATURAL` flag were also covered by this RFC.
> Is that possible? Would it make sense?


I'm not going to migrate those functions at this time. It's just a
project scope decision.

-- Tim Starling

---

## #116504 — Nicolas Grekas — 2021-11-25T09:58:41+00:00

URL: https://externals.io/message/116497#116504 · In reply to: #116503

Le jeu. 25 nov. 2021 à 10:47, Tim Starling <tstarling@wikimedia.org> a
écrit :

> 
>
>
> I voted yes because I want to see this happen but I raised a point in
> https://externals.io/message/116141#116259 and didn't get an answer:
>
> Despite their name, I never used natcase functions for natural language
> > […]
>
> I wish the `strnatcasecmp()` and `natcasesort()` function, but also the
> `SORT_NATURAL` flag were also covered by this RFC.
> Is that possible? Would it make sense?
>
>
> I'm not going to migrate those functions at this time. It's just a project
> scope decision.
>

Why?

The RFC says:
> because they also use isdigit() and isspace(),

Does that mean "too much work needed"? I would totally understand that of
course but I hope someone could do these last miles.

> and because they are intended for natural language processing

I definitely do not agree with this argument and it should be removed from
the RFC to me as it might add confusion in the future.

Nicolas

---

## #116505 — Christoph M. Becker — 2021-11-25T10:34:04+00:00

URL: https://externals.io/message/116497#116505 · In reply to: #116504

> Le jeu. 25 nov. 2021 à 10:47, Tim Starling <tstarling@wikimedia.org> a
> écrit :
>
> > […]
>
> I definitely do not agree with this argument and it should be removed from
> the RFC to me as it might add confusion in the future.

Yeah, the PHP manual says[1]:

| This function implements a comparison algorithm that orders
| alphanumeric strings in the way a human being would, this is described
| as a "natural ordering".

[1] <https://www.php.net/manual/en/function.strnatcmp.php>

--
Christoph M. Becker

---

## #116506 — Nicolas Grekas — 2021-11-25T10:39:49+00:00

URL: https://externals.io/message/116497#116506 · In reply to: #116505

Le jeu. 25 nov. 2021 à 11:34, Christoph M. Becker <cmbecker69@gmx.de> a
écrit :

> 
>
> > […]
> from
> > […]
>
> Yeah, the PHP manual says[1]:
>
> | This function implements a comparison algorithm that orders
> | alphanumeric strings in the way a human being would, this is described
> | as a "natural ordering".
>
> [1] <https://www.php.net/manual/en/function.strnatcmp.php>
>

Yep, yet "natural language processing" means processing sentences we write
as humans, e.g. processing this very message. natcase sorting functions are
not done for that. They're useful to sort items in a list - typically file
names - in a way that makes sense to humans. This is very different from
"natural language processing". Having "natsort" vary by locale doesn't make
more sense than having "sort()" vary by locale. That's my point. The
argument doesn't stand against implementing locale-insensitivity for these
functions and I think the RFC shouldn't make it (the argument.)

Nicolas

---

## #116507 — Tim Starling — 2021-11-25T11:14:01+00:00

URL: https://externals.io/message/116497#116507 · In reply to: #116500

> Hello,
>
> The RFC is missing information about alternatives:
> Do all of these function have an mbstring version?

The following functions have an mbstring version: strtolower,
strtoupper, stristr, stripos, strripos.

 `mb_convert_case()` provides functionality equivalent to lcfirst,
ucfirst and ucwords.

There is no mbstring version of str_ireplace, that is
https://bugs.php.net/bug.php?id=75225

There is no mbstring equivalent for the array sorting functions with
SORT_FLAG_CASE, but there is Collator::asort() in intl.

> Are those locale dependant or have an option for it?

The mbstring functions are locale-independent.

Unfortunately there do not seem to be PHP wrappers for the family of
case conversion functions in ICU's ustring.h. There is
IntlChar::tolower() and IntlChar::toupper(), but they provide
locale-independent case conversion, equvialent to mbstring. It's not
ideal to change the case of a string character by character, since
some languages have multi-character mappings. ICU calls this
context-sensitive case conversion.

Considering the lack of wide character support or context-sensitive
case conversion in the existing strtoupper/strtolower, I would
consider this missing functionality rather than functionality which I
am removing.

> To reuse the example from the RFC, if I want to convert a UTF string to uppercase using Turkish rules and get dotted capital I, what should I use?

For case-insensitive comparison you can use Collator. But for display
you just have to do it yourself. For the Turkish Wikipedia and other
Turkic language websites we are currently using `str_replace()`.

-- Tim Starling

---

## #116508 — Tim Starling — 2021-11-25T11:23:43+00:00

URL: https://externals.io/message/116497#116508 · In reply to: #116504

>
> The RFC says:
> > […]
>
> Does that mean "too much work needed"? I would totally understand
> that of course but I hope someone could do these last miles.
>
Yes.

> > […]
>
> I definitely do not agree with this argument and it should be
> removed from the RFC to me as it might add confusion in the future.

Done.

-- Tim Starling

---

## #116509 — Nicolas Grekas — 2021-11-25T11:31:59+00:00

URL: https://externals.io/message/116497#116509 · In reply to: #116508

Le jeu. 25 nov. 2021 à 12:23, Tim Starling <tstarling@wikimedia.org> a
écrit :

> 
>
>
> The RFC says:
> > […]
>
> Does that mean "too much work needed"? I would totally understand that of
> course but I hope someone could do these last miles.
>
> Yes.
>
> > […]
>
> I definitely do not agree with this argument and it should be removed from
> the RFC to me as it might add confusion in the future.
>
> Done.
>

Great, thanks!

---

## #116510 — Paul Crovella — 2021-11-25T12:34:27+00:00

URL: https://externals.io/message/116497#116510 · In reply to: #116507

>
> 
>
> > […]
>
> For case-insensitive comparison you can use Collator. But for display
> you just have to do it yourself. For the Turkish Wikipedia and other
> Turkic language websites we are currently using `str_replace()`.

Any particular reason not to use transliterators? https://3v4l.org/I038T

---

## #116511 — Dan Ackroyd — 2021-11-25T14:22:38+00:00

URL: https://externals.io/message/116497#116511 · In reply to: #116497

>
> Voting is now open for my RFC on locale-independent case conversion.
>

It seems popular, and likely to pass, but I voted no as the "Backward
Incompatible Changes" section is missing which makes it hard to
evaluate the impact.

cheers
Dan
Ack

---

## #116518 — Tim Starling — 2021-11-25T22:26:07+00:00

URL: https://externals.io/message/116497#116518 · In reply to: #116510

> 
> > […]
> Any particular reason not to use transliterators? https://3v4l.org/I038T

Thanks, I missed that.

You would need to do your own mapping from language code to
transliterator name, since it only has converters for az/tr, el, lt
and "Any", with no fallbacks. For example if you did
Transliterator::create("en-Upper")->transliterate('a') you would get a
fatal error.

Presumably if I submitted a PR adding wrappers for u_strToUpper()
etc., it would not be rejected on the basis that we already have
transliterators.

-- Tim Starling
