Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:58863
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: error (pb1.pair.com: domain lerdorf.com from 209.85.213.42 cause and error)
Message-ID: <4F5DAFCE.8020600@lerdorf.com>
Date: Mon, 12 Mar 2012 01:11:58 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2
MIME-Version: 1.0
To: Stas Malyshev <smalyshev@sugarcrm.com>
CC: PHP internals <internals@lists.php.net>
References: <4F5D9C77.3030000@lerdorf.com> <4F5DA152.10109@sugarcrm.com> <4F5DA894.8060606@lerdorf.com> <4F5DAB49.3030808@sugarcrm.com>
In-Reply-To: <4F5DAB49.3030808@sugarcrm.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Subject: Re: [PHP-DEV] default charset confusion
From: rasmus@lerdorf.com (Rasmus Lerdorf)

On 03/12/2012 12:52 AM, Stas Malyshev wrote:
> Hi!
> 
>> Ignoring 5.4 for a second, if you in 5.3 do this:
>>
>> echo htmlspecialchars($string);
>> echo htmlspecialchars($string, NULL, "ISO-8859-1");
>> echo htmlspecialchars($string, NULL, "UTF-8");
>>
>> You will see that the first two output the escaped string with the
>> GB2312 bytes intact within it and the UTF-8 calls returns false because
>> it correctly recognizes that GB2312 is not UTF-8. We don't have any such
>> check for 8859-1, so yes, saying UTF-8 and 8859-1 are the same for
>> htmlspecialchars() is wrong for PHP 5.3 as well as for 5.4.
> 
> So the difference is that ISO8859-1 does not validate but UTF-8 validates?
> I'm not sure what GB2312 encoding does but isn't it dangerous to do
> htmlspecialchars() with wrong encoding? Wouldn't htmlentities() also
> produce wrong result when used with wrong encoding?

Not sure you can validate 8859-1 since it isn't multibyte, can you? Is
there any byte that is explicitly forbidden in 8859-1?

And yes, it may very well be dangerous to use the wrong charset and now
that we have better support for GB2312 and other asian charsets in the
entities functions in 5.4 it is even more prudent to choose the right
one so we should provide some way to help people get it right short of
changing every call.

Gustavo suggested we could use the multibyte encoding setting.
Unfortunately only zend.script_encoding is available and I think
internal_encoding is closer to what we need here, but that is only
available as mbstring.internal_encoding.

-Rasmus