Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:58890
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: error (pb1.pair.com: domain lerdorf.com from 209.85.213.42 cause and error)
Message-ID: <4F5E5893.9030903@lerdorf.com>
Date: Mon, 12 Mar 2012 13:12:03 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2
MIME-Version: 1.0
To: Stas Malyshev <smalyshev@sugarcrm.com>
CC: PHP internals <internals@lists.php.net>
References: <4F5D9C77.3030000@lerdorf.com> <4F5DA152.10109@sugarcrm.com> <4F5DA894.8060606@lerdorf.com> <4F5DAB49.3030808@sugarcrm.com> <4F5DAFCE.8020600@lerdorf.com> <4F5E5148.4030106@sugarcrm.com> <4F5E5219.7080501@lerdorf.com> <4F5E53C3.8060502@sugarcrm.com>
In-Reply-To: <4F5E53C3.8060502@sugarcrm.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Subject: Re: [PHP-DEV] default charset confusion
From: rasmus@lerdorf.com (Rasmus Lerdorf)

On 03/12/2012 12:51 PM, Stas Malyshev wrote:
> Hi!
> 
>> But you can't necessarily hardcode the encoding if you are writing
>> portable code. That's a bit like hardcoding a timezone. In order to
>> write portable code you need to give people the ability to localize it.
> 
> No, it's not like timezone at all. I have to support all timezones in a
> global app, but I don't have to internally support every encoding on
> Earth - having everything internally in UTF-8 works quite well, and a
> lot of applications do exactly that - they have everything internally in
> UTF-8 and only may convert when importing or exporting the data. I don't
> see anything in using UTF-8 throughout the app/library that makes it
> non-portable. However, if we allow to change defaults in
> htmlspecialchars() etc. that essentially makes having defaults useless
> as I'd have so explicitly specify UTF-8 each time - otherwise it's a
> gamble what encoding I'd actually get.

If everything was UTF-8 we wouldn't have any of these issues.
Unfortunately that isn't the case. The question is what to do with apps
that need to deal with non UTF-8 data. Are we going to provide any help
to them beyond just telling them to convert everything to UTF-8?

We took steps in 5.4 to improve htmlspecialchars to understand more
encodings and we have the concept of script_encoding and
internal_encoding that is used both in the engine and in mbstring.
Currently internal_encoding isn't checked by htmlspecialchars. If you
pass it '' it checks script_encoding and default_charset which is a bit
odd since neither directly relate to the encoding of the internal data
you are feeding to it. So maybe a way to tackle this is to use the
mbstring internal encoding when it is set as the htmlspecialchars
default when it is called without an encoding arg.

-Rasmus