Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:78120
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain gmail.com designates 209.85.220.171 as permitted sender)
MIME-Version: 1.0
Sender: nicolas.grekas@gmail.com
In-Reply-To: <543E8E91.7070805@gmail.com>
References: <543CE705.7030203@gmail.com> <4575A816-43F4-462D-8150-A2D35516D914@ajf.me>
 <543D64E5.8000706@gmail.com> <543D8528.1060605@gmail.com> <543D8FFA.8080408@gmail.com>
 <543DAA29.8040701@gmail.com> <68E97150-8840-4C31-B271-3E8C8BE933DB@gmail.com>
 <543E4626.3000102@gmail.com> <543E6F6E.8080006@gmail.com> <543E8E91.7070805@gmail.com>
Date: Thu, 16 Oct 2014 19:03:04 +0200
Message-ID: <CAOWwgpk=iXrEcN+XqbBUm2KFa+r1hWfHTNaj1rHjpeCFkQ-oFw@mail.gmail.com>
To: Aleksey Tulinov <aleksey.tulinov@gmail.com>, Rowan Collins <rowan.collins@gmail.com>
Cc: "internals@lists.php.net" <internals@lists.php.net>
Content-Type: multipart/alternative; boundary=bcaec5016663007a4205058d3ab0
Subject: Re: [PHP-DEV] Unicode support
From: nicolas.grekas+php@gmail.com (Nicolas Grekas)

--bcaec5016663007a4205058d3ab0
Content-Type: text/plain; charset=UTF-8

Hello,

I think that Rowan is right: PHP users need to manipulate grapheme clusters
first (and code points in some rare situations). The fact that most of us
live in a world were NFC composes all our characters only hides this
reality.

A typical use case is a template engine: nearly all string manipulations
there need grapheme awareness: cutting strings for getting excerpt,
inserting a space between every "character", changing the case, etc. A
typical use case for a PHP app.
An other use case is if you want to implement text indexing in PHP: you
need to normalize before indexing, handle case folding, and thus think in
terms of graphemes. I'm not sure this is frequent in PHP though.

Like already said, alongside with grapheme clusters, we should also deals
with string matching: collations are out of scope, but normalization and
case folding is in. Please do not forget the turkish alphabet
<https://github.com/nicolas-grekas/Patchwork-UTF8/blob/master/class/Patchwork/TurkishUtf8.php>
also...
This is required IMHO to have what user expects for str_replace, strpos,
strcmp, etc.

I wrote a quite successful PHP lib to deal with this in PHP:
https://github.com/nicolas-grekas/Patchwork-UTF8

My experience from this is the following:
- dealing with grapheme clusters in current PHP is ok with grapheme_*()
functions, but these require intl. It would be great to have them (or an
equivalent) in core,
- NFC normalization of all input is required to deal with string
comparisons, so having Normalizer in core looks required also,
- almost everybody uses mbstring when dealing with utf8 strings, but almost
all cases should use a grapheme_*() instead.


 To be clear, I am suggesting that we aim to be the language which gets
>> this right, where other languages get it wrong.
>>
>
> Thank you for explaining this. I also think it could do better. I think
> Unicode-aware strrev() shouldn't be too complicated to do.



Perl 6 identified the subject very well and invented what they call "NFG",
which is NFC + dynamic internal code points for non-composable grapheme
clusters:
http://docs.parrot.org/parrot/latest/html/docs/pdds/pdd28_strings.pod.html

Maybe worth looking at?

Cheers,
Nicolas

--bcaec5016663007a4205058d3ab0--