[RFC] grapheme cluster for str_split, grapheme_str_split function

2024年3月27日(水) 6:18 Casper Langemeijer langemeijer@php.net:

Many of these already exist, such as grapheme_substr. We can't simply change the behaviour of the already existing functions due to BC reasons.

Wow. I feel very stupid. I feel I should have known about grapheme_*, but I didn't. Oh my, the manual says since PHP 5.3 no less. From what I've seen around being used, I'm far from the only one though. In an attempt to justify my own stupidity I searched its use and it's bad.

Searching on github with language:PHP:
mb_strlen 84k files, grapheme_strlen 680

Then a big number of first 100 of these files are stubs/polyfills/phpstan metadata. I've seen no framework except Symphony (but they might be further in the searchresults)

The grapheme_str_split function, as well as other intl extension functions is what should replace mbstring really.

YES!

I'm sorry to have wasted your time. If you need someone to help for the grapheme_ marketing team, let me know.

Hi, Casper

I think still useful mbstring functions. Because mbstring functions is
still valid as a bridge to non-Unicode character codes.
We think it makes sense for mbstring to calculate in Unicode code point units.

Therefore, I think make sense that separate mbstring functions and
grapheme functions.

Regards
Yuya

--

Yuya Hamada (tekimen)

• https://tekitoh-memdhoi.info
• https://github.com/youkidearitai

If you need someone to help for the grapheme_ marketing team, let me know.

I think a big part of the problem is that very few people dig into the
complexities of text encoding, and so don't know that a "grapheme" is
what they're looking for.

Unicode documentation is, generally, very careful with its terminology -
distinguishing between "code points", "code units" "graphemes" ,
"grapheme clusters", "glyphs", etc. Pretty much everyone else just says
"character", and assumes that everyone knows what they mean.

As a case in point, looking at the PHP manual pages for strlen,
mb_strlen, and grapheme_strlen:

Short summary:

• strlen — Get string length
• mb_strlen — Get string length
• grapheme_strlen — Get string length in grapheme units

Description:

• Returns the length of the given string.
• Gets the length of a string.
• Get string length in grapheme units (not bytes or characters)

The first two don't actually say what units they're measuring in. Maybe
it's millimetres? ;)

The last one uses the term "grapheme" without explaining what it means,
and makes a contrast with "characters", which is confusing, as one of
the definitions in the Unicode glossary
[https://unicode.org/glossary/#grapheme] is:

What a user thinks of as a character.

The mb_strlen documentation has a bit more explanation in its Return
Values section:

Returns the number of characters in string string having character
encoding encoding. A multi-byte character is counted as 1.

For Unicode in particular, this is a poor description; it is completely
missing the term "code point", which is what it actually counts.

That's probably because ext/mbstring wasn't written with Unicode in
mind, it was "developed to handle Japanese characters", back in 2001;
and it still does support several pre-Unicode "multi-byte encodings".
For a bit of nostalgia:
http://web.archive.org/web/20010605075550/http://www.php.net/manual/en/ref.mbstring.php

So... if you want to help make people more aware of the grapheme_*
functions, one place to start would be editing the documentation for the
various string, mbstring, and grapheme functions to use consistent
terminology, and sign-post each other more clearly.
http://doc.php.net/tutorial/

Regards,

--
Rowan Tommins
[IMSoP]