Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:37518
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: error (pb1.pair.com: domain gravitonic.com from 74.125.46.30 cause and error)
Message-ID: <4821DD47.9030900@gravitonic.com>
Date: Wed, 07 May 2008 09:48:07 -0700
User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070807)
MIME-Version: 1.0
To: Tomas Kuliavas <tokul@users.sourceforge.net>
CC:  internals@lists.php.net
References: <4BD5A050-02F2-46BD-B867-FA8CA12FF1BD@macvicar.net>    <48988.78.61.224.253.1209918881.nsm@avilys.eik.lt>    <alpine.DEB.0.98.0805041848040.5353@kossu.ez.no> <60526.78.61.224.253.1209928511.nsm@avilys.eik.lt>
In-Reply-To: <60526.78.61.224.253.1209928511.nsm@avilys.eik.lt>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Subject: Re: [PHP-DEV] Removal of unicode_semantics
From: andrei@gravitonic.com (Andrei Zmievski)

Tomas Kuliavas wrote:
> If I remain silent, others will have arguments that "everybody agrees on
> removal of unicode_semantics".
> 
> I write and maintain charset decoding and encoding functions.
> unicode_semantics breaks every mapping table and other functions that
> operate with binary 8bit strings.

Just curious, do these decoding/encoding functions do something that 
Unicode support won't do?

> In slides by Andrei Zmievski Unicode symbols are written with \u. Why are
> they written with \x(hex) and \(octal) in current PHP6?

\x and \(octal) inside Unicode strings are assumed to specify Unicode 
characters. This is one of the contention points, since a few people 
have said that they should specify individual bytes rather than 
characters, but in my opinion it's kind of dangerous since it may lead 
to broken/invalid Unicode strings.

> ---
> <?php
> echo "\xC3\200";
> ---
> I am not writing U+00C3 and U+0080, I am writing U+00C0 in UTF-8.

This should work fine inside binary strings..

> I can bypass it by adding one line to every script that operates with
> binary strings, but where are warranties that you won't dump declare()
> support just like you dump unicode_semantics.

It won't get dumped. Unicode_semantics is a BC/transition switch. 
declare() is crucial to proper script parsing.

> What happens to your new
> Unicode aware string functions, if I lie about strings' charset to PHP
> interpreter?

You will get in trouble.

> mb_strlen can't calculate correct $string length even when I
> set correct charset in mb_strlen() arguments. If above code works as I
> want in PHP6 unicode_semantics=on, mb_strlen($string,'utf-8') returns 2
> and not 1.

I don't know what mbstring does or does not with unicode_semantics 
switch, since it's meant to be deprecated.

-Andrei