Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:47267
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain gmail.com designates 74.125.82.170 as permitted sender)
DomainKey-Status: bad
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=WMqhaENFZvfNFYjyMDguQGl7IPiZjUFYdJsNJpCFUKm2YI5Xt4+8rX0OgkxYzq5cTZ
         pT0M/76tVfRRspoLl9LKe4GO8hIfUEVXSfap1puz5DAnXxnn6P63oOhJozHjzJbvwTIZ
         d92n3ZaGJGnTCFYEskbiWhAs/Vg/NcqmoWQJI=
MIME-Version: 1.0
In-Reply-To: <4bcbf4711003140723s712c2653xa61e8f6053983553@mail.gmail.com>
References: <4B9C9007.1080802@lsces.co.uk> <4B9C91D7.2050402@rowe-clan.net>
	 <13008E62F851429F84B9FE2F3F230286@pc>
	 <4bcbf4711003140723s712c2653xa61e8f6053983553@mail.gmail.com>
Date: Sun, 14 Mar 2010 15:33:19 +0100
Message-ID: <fe05d1541003140733s3ea7a45ds4c12648d7eb1d9a5@mail.gmail.com>
To: Jordi Boggiano <j.boggiano@seld.be>
Cc: Stan Vassilev <sv_forums@fmethod.com>, "William A. Rowe Jr." <wrowe@rowe-clan.net>, internals@lists.php.net
Content-Type: text/plain; charset=ISO-8859-1
Subject: Re: [PHP-DEV] Where are we ACTUALLY on Unicode?
From: pierre.php@gmail.com (Pierre Joye)

On Sun, Mar 14, 2010 at 3:23 PM, Jordi Boggiano <j.boggiano@seld.be> wrote:
> On Sun, Mar 14, 2010 at 12:03 PM, Stan Vassilev <sv_forums@fmethod.com> wrote:
>> UTF8 also takes 4 bytes for representing characters in the higher bit
>> planes, as quite a lot of bits are lost for every char in order to describe
>> how long the code point is, and when it ends and so on. This means
>> memory-wise it may not be of big benefit to asian countries.
>
> I remember Brian Aker saying that they chose to work internally with
> UTF-8 for Drizzle. His explanation of it was that asian countries have
> so much english content mixed in that on average even for them UTF-8
> still had a lower footprint than UTF-16/32. I do not know where the
> stats came from, but if it holds any truth it is worth considering.

The idea behind his reasonning was to about optimizing the 90% of the
cases while being "fast enough" for the last 10% (could have been
other numbers, but that's the idea). For what I remember about our
discussions, he also mentioned fast UTF-8 capable string processing
implementation (as fast as what UTF-16 could be). I like this the
90/10 approach especially as it actually matches what we have in PHP.

Cheers,
-- 
Pierre

@pierrejoye | http://blog.thepimp.net | http://www.libgd.org