Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:112796
Content-Type: text/plain;
	charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.40.0.2.32\))
In-Reply-To: <d975512f-0d57-20d9-bc0d-58183b947992@users.sourceforge.net>
Date: Thu, 7 Jan 2021 14:24:06 +0000
Cc: Claude Pache <claude.pache@gmail.com>,
 Nikita Popov <nikita.ppv@gmail.com>,
 Tomas Kuliavas <tokul@users.sourceforge.net>
Content-Transfer-Encoding: quoted-printable
Message-ID: <982075D8-2B6B-4157-86FD-502D65144830@craigfrancis.co.uk>
References: <CAFv4g+GezouDRdyeyOV2eTvNi0K-jko+jf2QBc+MhjbwNiqR1Q@mail.gmail.com>
 <CAF+90c-Zhsw-s1VjN1NXHEkuYO+wOTVwbkMbNOJ=EA3mdnQGwQ@mail.gmail.com>
 <99C71641-5A5B-49C8-8D96-F0C080352B91@gmail.com>
 <d975512f-0d57-20d9-bc0d-58183b947992@users.sourceforge.net>
To: PHP internals <internals@lists.php.net>
Subject: Re: [PHP-DEV] ENT_COMPAT for htmlentities and htmlspecialchars
From: craig@craigfrancis.co.uk (Craig Francis)

On Sat, Dec 26, 2020 at 12:03 PM Craig Francis =
<craig@craigfrancis.co.uk> wrote:
> Could htmlspecialchars() use ENT_QUOTES by default?
> [...]
> I'd also be tempted to suggest ENT_SUBSTITUTE should be included, as I =
prefer to keep as much of the valid data (rather than losing =
everything), but that's not as important as escaping the apostrophe by =
default.


On Thu, 7 Jan 2021 at 09:00, Claude Pache <claude.pache@gmail.com> =
wrote:
> For ENT_SUBSTITUTE, there has been =
https://bugs.php.net/bug.php?id=3D69450, but I don=E2=80=99t understand =
the objection in that bug report. Maybe there is some issue related to =
non-Unicode multibyte encodings?



On Thu, 7 Jan 2021 at 09:29, Tomas Kuliavas =
<tokul@users.sourceforge.net> wrote:
> Only ISO-2022 encodings got bytes that can match symbols sanitized by =
htmlspecialchars.
>=20
> Bug objection insist that utf-8 parsing rules should be enacted by =
sanitizing function and not by application which displays text. And PHP =
code is enacting those rules in most unfriendly API way.



Does anyone have an example where ENT_SUBSTITUTE could be used to create =
an issue? ideally a security issue, but anything will do.




With `htmlspecialchars($user_value)`, I don't think it would matter if =
it ended with <multibyte start byte>, like the example from Rasmus =
(0xE0), because that end byte would be replaced by U+FFFD.

With `htmlspecialchars($user_value . $system_value)`, if $user_value =
ends with <multibyte start byte>, it's possible some characters at the =
beginning of $system_value could be replaced. But I can't find a way to =
do that with UTF-8; and even if it was possible, I would have thought =
some characters being replaced by U+FFFD, would be a much better =
solution than everything being lost (noting that $system_value will not =
contain any HTML characters, because they are escaped as well).

    echo '<p>' . htmlspecialchars($user . ' is lying to you') . '</p>';
   =20
    With: '<p>ABC=EF=BF=BDs lying to you</p>'
    Without: '<p></p>'

And, in both cases, the output is valid UTF-8, and shouldn't affect =
anything it's concatenated with (i.e. the HTML context).

Personally, I think every part of our processes (input, processing, and =
output) should do its best to handle encoding issues (incase something =
is missed). I believe ENT_SUBSTITUTE is the best way to deal with it =
during output. I don't think it's realistic to expect every single PHP =
developer to check for invalid characters in every single bit of input.

That said (and just to make things even more complicated), considering =
this is HTML encoding, we could go even further and add ENT_DISALLOWED. =
As Hans noted, some characters, such as 0x01, can be seen as valid in =
general, but not valid for HTML (where Text Nodes "must not contain =
control characters other than space characters"). All browsers seem to =
handle these control characters (by ignoring them), so I'm not too =
worried, but if it makes things safer, why not?

Craig