Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:37869
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: softfail (pb1.pair.com: domain zend.com does not designate 64.97.136.137 as permitted sender)
Message-ID: <00da01c8be9b$4f4ddb90$4401a8c0@foxbox>
Reply-To: "Steph Fox" <steph@zend.com>
To: =?utf-8?Q?Johannes_Schl=C3=BCter?= <johannes@php.net>
Cc: "Andrei Zmievski" <andrei@gravitonic.com>,
	"Antony Dovgal" <tony@daylessday.org>,
	<internals@lists.php.net>
Date: Sun, 25 May 2008 20:12:41 +0100
Organization: Zend Technologies
MIME-Version: 1.0
Content-Type: text/plain;
	format=flowed;
	charset="utf-8";
	reply-type=response
Content-Transfer-Encoding: 7bit
Subject: Re: [PHP-DEV] Unicode progress [Was: unicode.semantics adinfinitum]
From: steph@zend.com ("Steph Fox")

Hi again Johannes,

Last one on this subject I promise...

> If the script encoding doesn't match up with whatever's set in INI, I 
> don't get as far as that stuff anyway:
>
> Warning: Illegal or truncated character in input: offset 0, state=0 in 
> C:\sandbox\php-src\Debug_TS\help.php on line 5

You were right to point out that runtime encoding isn't reliable. (Now I 
finally got the thing working in scripts here - there's something invisible 
going on with UTF-8 somewhere down the line that had me confused for a 
while.)

A call to zend_unicode_to_string_ex() that explicitly passes the fallback 
converter should be good enough:

      } else if (c == 'S' && Z_TYPE_PP(arg) != IS_NULL /* NULL is ok */) {
       UErrorCode status = U_ZERO_ERROR;

       zend_unicode_to_string_ex(UG(fallback_encoding_conv), p, pl, 
Z_USTRVAL_PP(arg), Z_USTRLEN_PP(arg), &status);

       if (U_FAILURE(status)) {
        efree(p);
        return "strictly a binary string";
       }
       break;
      }

It doesn't make sense to set unicode.fallback_encoding at script level 
because - as I wrote earlier - the parser may respond differently depending 
on its value. If your INI fallback setting clashes with your script encoding 
you *can't* override it locally. That being the case, I don't see how a 
conversion that uses the fallback encoding can be wrong here.

- Steph