Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:72613
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: error (pb1.pair.com: domain lerdorf.com from 209.85.216.52 cause and error)
Message-ID: <52FE6FEA.5050204@lerdorf.com>
Date: Fri, 14 Feb 2014 14:35:06 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0
MIME-Version: 1.0
To: Rowan Collins <rowan.collins@gmail.com>, internals@lists.php.net
References: <50100EC8.3040102@ajf.me> <CAH-PCH5BNg6i2TcZbiWbi0bYKkSd9i8q5e3nfniS=aFL9V+fXA@mail.gmail.com> <52FDF7BC.8050408@lsces.co.uk> <52FE46D2.4060903@gmail.com>
In-Reply-To: <52FE46D2.4060903@gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Subject: Re: [PHP-DEV] PHP6 wiki page
From: rasmus@lerdorf.com (Rasmus Lerdorf)

On 02/14/2014 11:39 AM, Rowan Collins wrote:
> Lester Caine wrote (on 14/02/2014):
>> But more fundamentally I don't think there was agreement on whether we
>> simply standardise on unicode in the core, or allow a single byte
>> mode? 8 years on, I feel that the amount of utf8 material that is
>> floating around, the easiest route IS unicode only?
> 
> The question is not whether to be "Unicode only", it's *how* to
> implement Unicode. It's not just a case of making all your strings
> wider, every function that manipulates a string in any way has to be
> thought through, and every input and output has to be converted to/from
> whatever encoding is chosen as the internal implementation.
> 
> While updating the Wikipedia article [1] I came across this slide set
> [2], which has a fairly decent explanation of the issues and why the
> previous implementation was abandoned.
> 
> If somebody comes up with an implementation proposal of Unicode strings,
> whether to have a mode that doesn't use it can be discussed, but right
> now there doesn't seem to be such a live proposal.

What we really need is an awesome small and fast Unicode library that
does everything ICU does but faster and in less code while using UTF-8
as its internal storage so we don't have to convert on each and every
operation. There are a ton of non-obvious things beyond simple string
manipulation. String collation alone is massively complicated, for example.

-Rasmus