Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:62428
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: error (pb1.pair.com: domain lerdorf.com from 209.85.212.42 cause and error)
Message-ID: <5036551E.1030804@lerdorf.com>
Date: Thu, 23 Aug 2012 09:06:54 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20120714 Thunderbird/14.0
MIME-Version: 1.0
To: PHP internals <internals@lists.php.net>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Subject: Default input encoding for htmlspecialchars/htmlentities
From: rasmus@lerdorf.com (Rasmus Lerdorf)

htmlspecialchars(), htmlentities(), html_entity_decode() and
get_html_translation_table() all take an encoding parameter that used to
default to iso-8859-1. We changed the default in PHP 5.4 to UTF-8. This
is a much more sensible default and in the case of the encoding
functions more secure as it prevents invalid UTF-8 from getting through.
If you use 8859-1 as the default but your app is actually in UTF-8 or
worse, some encoding that isn't low-ascii compatible then
htmlspecialchars()/htmlentities() aren't doing what you think they are
and you have a glaring security hole in your app.

However, people are understandably lazy and don't want to think about
this stuff. They don't want to explicitly provide their input encoding
to these calls. We provided a solution to this and a way to write
portable apps and that was to pass in an empty string "" as the
encoding. If we saw this we would set the input encoding to match the
output encoding specified by the "default_charset" ini setting. We
couldn't just default to this default_charset because input and output
encodings may very well be different and we would risk making existing
apps insecure. For example an app using BIG5/CJK for its output encoding
might very well be pulling data from 8859/UTF-8 data sources and if we
invisibly switched htmlspecialchars/entities to match their output
encoding we would have problems. Invisibly switching them from 8859-1 to
UTF-8 could still be problematic, but it at least it fails safe in that
it doesn't let invalid UTF-8 through and encodes low-ascii the same way
it did before.

The problem is that there is a lot of legacy code out there that doesn't
explicitly set the encoding on those calls and it is a lot of work to go
through and specify it on each call. I still personally prefer to have
people be explicit here, but I think it is slowing 5.4 adoption (see bug
61354).

In PHP 6 we tried to introduce separate input, script and output
encoding settings. Currently in 5.4 we don't have that, but we have
those 3 separately for mbstring and for iconv:

iconv.input_encoding
iconv.internal_encoding
iconv.output_encoding
mbstring.http_input
mbstring.internal_encoding
mbstring.http_output

Ideally we should be getting rid of the per-feature encoding settings
and have a single set of them that we refer to when we need them. This
is one of these places where we really need a default input encoding
setting. We could have it check mbstring.http_input, but there is a
wrinkle here that it has a fancy "auto" setting which we don't really
want in this case. So we could set it to iconv.input_encoding, but that
seems rather random and unintuitive.

So do we create a new default_input_encoding ini directive mid-stream in
5.4 for this? Of course with the longer-term in mind that this will be
part of a unified set of encoding settings in 5.5 and beyond.

-Rasmus