Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:58908
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain gmail.com designates 209.85.161.170 as permitted sender)
MIME-Version: 1.0
Sender: yohgaki@gmail.com
In-Reply-To: <4F5E15D6.6080302@lerdorf.com>
References: <4F5D9C77.3030000@lerdorf.com> <4F5DA152.10109@sugarcrm.com>
 <4F5DA894.8060606@lerdorf.com> <4F5DAB49.3030808@sugarcrm.com>
 <4F5DAFCE.8020600@lerdorf.com> <CAGa2bXbNOqoDdQsmyWrXR+Qw8E6o2uxWbrnKXBwJewwjtpMJ_g@mail.gmail.com>
 <4F5E15D6.6080302@lerdorf.com>
Date: Tue, 13 Mar 2012 09:52:00 +0900
Message-ID: <CAGa2bXYSEk2UTKMnMGvA4R_mJrk1kO6BigSoC5U+aByW9eapEw@mail.gmail.com>
To: Rasmus Lerdorf <rasmus@lerdorf.com>
Cc: Stas Malyshev <smalyshev@sugarcrm.com>, PHP internals <internals@lists.php.net>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Subject: Re: [PHP-DEV] default charset confusion
From: yohgaki@ohgaki.net (Yasuo Ohgaki)

2012/3/13 Rasmus Lerdorf <rasmus@lerdorf.com>:
> On 03/12/2012 03:05 AM, Yasuo Ohgaki wrote:
>> I thought default_charset became UTF-8, so I was expecting
>> following HTTP header.
>>
>> content-type =C2=A0text/html; charset=3DUTF-8
>>
>> However, I got empty charset (missing 'charset=3DUTF-8').
>> So I looked up to source and found the line in SAPI.h
>>
>> 293 =C2=A0 #define SAPI_DEFAULT_CHARSET =C2=A0 =C2=A0 =C2=A0 =C2=A0""
>>
>> Empty string should be "UTF-8", isn't it?
>
> No, we can't force an output charset on people since it would end up
> breaking a lot of sites.

Right, so may be for the next major release? 5.5.0?

As the first XSS advisory in 2000 states, explicitly setting char coding wi=
ll
prevent certain XSS. Recent browsers have much better encoding handing,
but setting encoding explicitly is better for security still.

> PHP 5.3's determine_charset behaves exactly like 5.4's. In 5.3 we have:
>
> =C2=A0 =C2=A0if (charset_hint =3D=3D NULL)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return cs_8859_1;
>
> and in 5.4 we have:
>
> =C2=A0 =C2=A0if (charset_hint =3D=3D NULL)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return cs_utf_8;
>
> So there is no difference in their guessing when there is no hint, the
> only difference is that in 5.4 we choose utf8 and in 5.3 we choose
> 8859-1 in that case.

I got this with 5.3
<?php
echo htmlentities('<=E6=97=A5=E6=9C=AC=E8=AA=9EUTF-8>',ENT_QUOTES);
echo htmlentities('<=E6=97=A5=E6=9C=AC=E8=AA=9EUTF-8>',ENT_QUOTES, 'UTF-8')=
;

&lt;&aelig;=EF=BF=BD&yen;&aelig;=EF=BF=BD&not;&egrave;&ordf;=EF=BF=BDUTF8
&gt;&lt;=E6=97=A5=E6=9C=AC=E8=AA=9EUTF-8&gt;

So people migrating from 5.3 to 5.4 should not have problems.
Migration older than 5.3 to 5.4 will be problematic.

I always set all parameters for htmlentities/htmlspecialchars, therefore
I haven't noticed this was changed from 5.3. They may be migrating from
5.2 or older. (RHEL5 uses 5.1)

Since PHP does not have default multibyte module, it may be good for having

input_encoding
internal_encoding
output_encoding

php.ini settings and make multibyte modules use them when they are set.
Or just make mbstring default, alternatively.

Rather big change for released version, but this is simple easy change.

Regards,

--
Yasuo Ohgaki