Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:47340
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain gmail.com designates 209.85.220.215 as permitted sender)
DomainKey-Status: bad
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type:content-transfer-encoding;
        b=lzK220ukwCebX8G8UUidbjT7+w+IlfD7+K6YTRgUXkvCbU0J1eoQHjZ+0rJFwoQo1l
         4KGEvjpU2GVnqFAOO3Crz2SH4p5D85PwgTl3KrY7LhjQ7S7D6U9cgwf4Bd+5dzMYZkMc
         JiM++29BEkVTn8qcVb9jKvZp5XqAdsopyjjKY=
MIME-Version: 1.0
In-Reply-To: <b54d0abe1003161304u35958c0scefb87a5607853b5@mail.gmail.com>
References: <4B9C9007.1080802@lsces.co.uk> <4B9EC3B2.7070901@zend.com> 
	<4B9F4196.9030404@lsces.co.uk> <4B9FD68B.5020900@zend.com> 
	<b54d0abe1003161304u35958c0scefb87a5607853b5@mail.gmail.com>
Date: Tue, 16 Mar 2010 20:43:29 +0000
Message-ID: <99cf22521003161343o21262736s801bd2e99ac2b6a8@mail.gmail.com>
To: Ferenc Kovacs <tyra3l@gmail.com>
Cc: Stanislav Malyshev <stas@zend.com>, Lester Caine <lester@lsces.co.uk>, 
	PHP internals <internals@lists.php.net>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Subject: Re: [PHP-DEV] Where are we ACTUALLY on Unicode?
From: dreamcat4@gmail.com (dreamcat four)

And remember,

Its not just the number of times its send to ICU for conversion. Its
also the number of times your UTF-16 string has to be converted back
into utf-8 afterwards. This is why Apple makes its utf-16 strings
immutable. So they are read-only, and the utf-8 representation can be
cached afterward.

Think of it this way:

1. Load a utf-8 string from DB or file
2. Convert it to utf-16
3. Perform ICU conv 3-5 times
4. Page gets hit by memcache
5. utf-16 is converted back to utf-8
6. Something changes
 ? String was cached ?
7. need to spit out another utf-8 version of the string again

And a persistent web application can be held for many hours in memory.
Are we converting back to utf-8 every time? Then it might be better to
wrap the string conversions just around ICU.

I'd suggest selecting a real (but still as easy-to-work with as can be
found) unicode php app. One that has been written to use a unicode php
module. Then getting a single, representative page from it. By that I
mean the kind of page that gets accessed the most. So for imdb that
would be a movie's page, etc. The smalled 'slice' of the app, not the
whole thing. Dummy-out the other stuff.

Then convert that part (for rendering one page) into the current php6
unicode scheme. And can see what's what.



On Tue, Mar 16, 2010 at 8:04 PM, Ferenc Kovacs <tyra3l@gmail.com> wrote:
> On Tue, Mar 16, 2010 at 8:05 PM, Stanislav Malyshev <stas@zend.com> wrote=
:
>> Hi!
>>
>>> On disk storage should probably be UTF-8 without any question? Windows
>>> use of widestrings for some files simple doubles up the on disk storage
>>
>> As file content, it's OK (an it'd be easy to add option to specify conte=
nt
>> transformation if we wanted), but prescribing filenames as UTF-8 would
>> probably be not workable, since different systems (and maybe even differ=
ent
>> filesystems inside same OS?) can have different opinions on that.
>>
>>> '3' is not a very processor friendly number, so working with 4 even
>>> though wasteful on memory, does make perfect sense. How long is it sinc=
e
>>
>> I'm not sure it does. Most of PHP strings are short, so memory loss woul=
d be
>> very significant. Also, take into account that CPU caches aren't as big =
as
>> the main memory, and not fitting your data into the cache is expensive.
>>
>>> we had a 640k limit on working memory? SERVERS should have a good amoun=
t
>>
>> It doesn't matter how much memory you have, in numbers. Until we find an
>> unlimited source of computer memory left by the aliens in Himalayas, mem=
ory
>> costs money. It doesn't matter how much memory do you have - however man=
y
>> gigs you have, you'll be able to run 3 times less PHP processes in new
>> version on the same hardware than in old version, which means new PHP wo=
uld
>> cost you more to run. "Memory is cheap" is a very misunderstood expressi=
on -
>> it's only cheap if you always have much more than you need.
>>
>>> Probably 90% of the time a string will come in and go out without
>>> requiring any processing at all, so leave it as UTF-8 ? The only time w=
e
>>
>> It might be great if we could do that. The problem might be that right n=
ow
>> AFAIK we don't have a good library to work with utf-8 strings (please
>> correct me if I'm wrong here).
> http://source.icu-project.org/repos/icu/icuhtml/trunk/design/strings/icu_=
utf8.html
> from ICU 3.6 changelog =3D> The UTF-8 transformation functions and
> macros are faster.
> from 4.2 =3D> UTF-8 friendly internal data structure for Unicode data loo=
kup
> so it's seems that guys at ICU tries to close the gap between the
> UTF-16 and UTF-8 performance, so maybe it would be a good idea, to
> check out the current situation.
>
> Tyrael
>> --
>> Stanislav Malyshev, Zend Software Architect
>> stas@zend.com =C2=A0 http://www.zend.com/
>> (408)253-8829 =C2=A0 MSN: stas@zend.com
>>
>> --
>> PHP Internals - PHP Runtime Development Mailing List
>> To unsubscribe, visit: http://www.php.net/unsub.php
>>
>>
>
> --
> PHP Internals - PHP Runtime Development Mailing List
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>