Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:125055
Content-Type: multipart/alternative;
	boundary="Apple-Mail=_FF52ADD5-65BA-4B75-9DFC-087DFE5A2036"
Precedence: bulk
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.600.62\))
Subject: [PHP-DEV] Re: [RFC] Decoding HTML and the Ambiguous Ampersand
Date: Mon, 19 Aug 2024 15:45:53 -0700
References: <CAA8B983-FD39-4AB1-B314-62608B0C0EF4@a8c.com>
To: Internals <internals@lists.php.net>
In-Reply-To: <CAA8B983-FD39-4AB1-B314-62608B0C0EF4@a8c.com>
Message-ID: <76D9E1DA-57CE-45C3-8E3E-B08A0B70FB60@a8c.com>
From: dennis.snell@automattic.com (Dennis Snell)


--Apple-Mail=_FF52ADD5-65BA-4B75-9DFC-087DFE5A2036
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8


> On Jul 9, 2024, at 4:55=E2=80=AFPM, Dennis Snell =
<dennis.snell@a8c.com> wrote:
>=20
> Greetings all,
>=20
> The `html_entity_decode( =E2=80=A6 ENT_HTML5 =E2=80=A6 )` function has =
a number of issues that I=E2=80=99d like to correct.
>=20
>  - It=E2=80=99s missing 720 of HTML5=E2=80=99s specified named =
character references.
>  - 106 of these are named character references which do not require a =
trailing semicolon, such as `&acute`
>  - It=E2=80=99s unaware of the ambiguous ampersand rule, which allows =
these 106 in special circumstances.
>=20
> HTML5 asserts that the list of named character references will not =
expand in the future. It can be found authoritatively at the following =
URL:
>=20
> https://html.spec.whatwg.org/entities.json
>=20
> The ambiguous ampersand rule smoothes over legacy behavior from before =
HTML5 where ampersands were not properly encoded in attribute values, =
specifically in URL values. For example, in a query string for a search, =
one might find `?q=3Ddog&not=3Dcat`. The `&not` in that value would =
decode to U+AC (=C2=AC), but since it=E2=80=99s in an attribute value it =
will be left as plaintext. Inside normal HTML markup it would transform =
into `?q=3Ddog=C2=AC=3Dcat`. There are related nuances when numeric =
character references are found at the end of a string or boundary =
without the semicolon.
>=20
> The function signature of `html_entity_decode()` does not currently =
allow for correcting this behavior. I=E2=80=99d like to propose an RFC =
or a bug fix which either extends the function (perhaps by adding a new =
flag like `ENT_AMBIGUOUS_AMPERSAND`) or preferably creates a new =
function. For the missing character references I wonder if it would be =
enough to add them to the list of default translatable references.
>=20
> One challenge with the existing function is that the concept of the =
translation table stands in contrast with the fixed and static nature of =
HTML5=E2=80=99s replacement tables. A new function or set of functions =
could open up spec-compliant decoding while providing helpful methods =
that are necessary in many common server-side operations:
>=20
>   - `html_decode( =E2=80=98attribute=E2=80=99 | =E2=80=98data=E2=80=99, =
$raw_text, $input_encoding =3D =E2=80=98utf-8' )`
>   - `html_text_contains( =E2=80=98attribute=E2=80=99 | =E2=80=98data=E2=80=
=99, $raw_haystack, $needle, $input_encoding =3D =E2=80=98utf-8=E2=80=99 =
)`
>   - `html_text_starts_with( =E2=80=98attribute=E2=80=99 | =E2=80=98data=E2=
=80=99, $raw_haystack, $needle, $input_encoding =3D =E2=80=98utf-8=E2=80=99=
 )`
>=20
> These methods are handy for inspecting things like encoded attribute =
values in a memory-efficient and processing-efficient way, when it=E2=80=99=
s not necessary to decode the entire value. In common situations, one =
encounters data-URIs with potentially megabytes of image data and =
processing only the first few or tens of bytes can save a lot of =
overhead.
>=20
> We=E2=80=99re exploring pure-PHP solutions to these problems in =
WordPress in attempts to improve the reliability and safety of handling =
HTML. I=E2=80=99d love to hear your thoughts and know if anyone is =
willing to work with me to create an RFC or directly propose patches. =
We=E2=80=99ve created a step function which allows finding the next =
character reference and decoding it separately, enabling some novel =
features like highlighting the character references in source text.
>=20
> Should I propose an RFC for this?
>=20
> Warmly,
> Dennis Snell
> Automattic Inc.

Thanks everyone for your feedback so far on the `decode_html()` RFC =
[https://wiki.php.net/rfc/decode_html]

I=E2=80=99ve updated it replacing the new constants with a new =
`HtmlContext` enum, and the interface seems much nicer this way. I =
particularly like how PHP enforces passing a valid value, vs. hoping =
that the right flag is used.

Additionally I added a section that I previously forgot, which =
highlights the source of the infamous mojibake/gremlins. HTML has =
special rules for remapping the C1 control characters, as if they had =
been stored or recorded for Windows-1251.

Warmly,
Dennis Snell


--Apple-Mail=_FF52ADD5-65BA-4B75-9DFC-087DFE5A2036
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8

<html><head><meta http-equiv=3D"content-type" content=3D"text/html; =
charset=3Dutf-8"></head><body style=3D"overflow-wrap: break-word; =
-webkit-nbsp-mode: space; line-break: =
after-white-space;"><br><div><blockquote type=3D"cite"><div>On Jul 9, =
2024, at 4:55=E2=80=AFPM, Dennis Snell &lt;dennis.snell@a8c.com&gt; =
wrote:</div><br class=3D"Apple-interchange-newline"><div><meta =
http-equiv=3D"content-type" content=3D"text/html; charset=3Dutf-8"><div =
style=3D"overflow-wrap: break-word; -webkit-nbsp-mode: space; =
line-break: after-white-space;"><div style=3D"caret-color: rgb(54, 61, =
76); color: rgb(54, 61, 76); font-size: 15px;">Greetings all,</div><div =
style=3D"caret-color: rgb(54, 61, 76); color: rgb(54, 61, 76); =
font-size: 15px;"><br></div><div style=3D"caret-color: rgb(54, 61, 76); =
color: rgb(54, 61, 76); font-size: 15px;">The `html_entity_decode( =E2=80=A6=
 ENT_HTML5 =E2=80=A6 )` function has a number of issues that I=E2=80=99d =
like to correct.</div><div style=3D"caret-color: rgb(54, 61, 76); color: =
rgb(54, 61, 76); font-size: 15px;"><br></div><div style=3D"caret-color: =
rgb(54, 61, 76); color: rgb(54, 61, 76); font-size: 15px;">&nbsp;- =
It=E2=80=99s missing 720 of HTML5=E2=80=99s specified named character =
references.</div><div style=3D"caret-color: rgb(54, 61, 76); color: =
rgb(54, 61, 76); font-size: 15px;">&nbsp;- 106 of these are named =
character references which do not require a trailing semicolon, such as =
`&amp;acute`</div><div style=3D"caret-color: rgb(54, 61, 76); color: =
rgb(54, 61, 76); font-size: 15px;">&nbsp;- It=E2=80=99s unaware of the =
ambiguous ampersand rule, which allows these 106 in special =
circumstances.</div><div style=3D"caret-color: rgb(54, 61, 76); color: =
rgb(54, 61, 76); font-size: 15px;"><br></div><div style=3D"caret-color: =
rgb(54, 61, 76); color: rgb(54, 61, 76); font-size: 15px;">HTML5 asserts =
that the list of named character references will not expand in the =
future. It can be found authoritatively at the following URL:</div><div =
style=3D"caret-color: rgb(54, 61, 76); color: rgb(54, 61, 76); =
font-size: 15px;"><br></div><div style=3D"caret-color: rgb(54, 61, 76); =
color: rgb(54, 61, 76); font-size: 15px;"><a =
href=3D"https://html.spec.whatwg.org/entities.json" style=3D"color: =
rgb(0, 128, =
255);">https://html.spec.whatwg.org/entities.json</a></div><div =
style=3D"caret-color: rgb(54, 61, 76); color: rgb(54, 61, 76); =
font-size: 15px;"><br></div><div style=3D"caret-color: rgb(54, 61, 76); =
color: rgb(54, 61, 76); font-size: 15px;">The ambiguous ampersand rule =
smoothes over legacy behavior from before HTML5 where ampersands were =
not properly encoded in attribute values, specifically in URL values. =
For example, in a query string for a search, one might find =
`?q=3Ddog&amp;not=3Dcat`. The `&amp;not` in that value would decode to =
U+AC (=C2=AC), but since it=E2=80=99s in an attribute value it will be =
left as plaintext. Inside normal HTML markup it would transform into =
`?q=3Ddog=C2=AC=3Dcat`. There are related nuances when numeric character =
references are found at the end of a string or boundary without the =
semicolon.</div><div style=3D"caret-color: rgb(54, 61, 76); color: =
rgb(54, 61, 76); font-size: 15px;"><br></div><div style=3D"caret-color: =
rgb(54, 61, 76); color: rgb(54, 61, 76); font-size: 15px;">The function =
signature of `html_entity_decode()` does not currently allow for =
correcting this behavior. I=E2=80=99d like to propose an RFC or a bug =
fix which either extends the function (perhaps by adding a new flag like =
`ENT_AMBIGUOUS_AMPERSAND`) or preferably creates a new function. For the =
missing character references I wonder if it would be enough to add them =
to the list of default translatable references.</div><div =
style=3D"caret-color: rgb(54, 61, 76); color: rgb(54, 61, 76); =
font-size: 15px;"><br></div><div style=3D"caret-color: rgb(54, 61, 76); =
color: rgb(54, 61, 76); font-size: 15px;">One challenge with the =
existing function is that the concept of the translation table stands in =
contrast with the fixed and static nature of HTML5=E2=80=99s replacement =
tables. A new function or set of functions could open up spec-compliant =
decoding while providing helpful methods that are necessary in many =
common server-side operations:</div><div style=3D"caret-color: rgb(54, =
61, 76); color: rgb(54, 61, 76); font-size: 15px;"><br></div><div =
style=3D"caret-color: rgb(54, 61, 76); color: rgb(54, 61, 76); =
font-size: 15px;">&nbsp; - `html_decode( =E2=80=98attribute=E2=80=99 | =
=E2=80=98data=E2=80=99, $raw_text, $input_encoding =3D =E2=80=98utf-8' =
)`</div><div style=3D"caret-color: rgb(54, 61, 76); color: rgb(54, 61, =
76); font-size: 15px;">&nbsp; - `html_text_contains( =E2=80=98attribute=E2=
=80=99 | =E2=80=98data=E2=80=99, $raw_haystack, $needle, $input_encoding =
=3D =E2=80=98utf-8=E2=80=99 )`</div><div style=3D"caret-color: rgb(54, =
61, 76); color: rgb(54, 61, 76); font-size: 15px;">&nbsp; - =
`html_text_starts_with( =E2=80=98attribute=E2=80=99 | =E2=80=98data=E2=80=99=
, $raw_haystack, $needle, $input_encoding =3D =E2=80=98utf-8=E2=80=99 =
)`</div><div style=3D"caret-color: rgb(54, 61, 76); color: rgb(54, 61, =
76); font-size: 15px;"><br></div><div style=3D"caret-color: rgb(54, 61, =
76); color: rgb(54, 61, 76); font-size: 15px;">These methods are handy =
for inspecting things like encoded attribute values in a =
memory-efficient and processing-efficient way, when it=E2=80=99s not =
necessary to decode the entire value. In common situations, one =
encounters data-URIs with potentially megabytes of image data and =
processing only the first few or tens of bytes can save a lot of =
overhead.</div><div style=3D"caret-color: rgb(54, 61, 76); color: =
rgb(54, 61, 76); font-size: 15px;"><br></div><div style=3D"caret-color: =
rgb(54, 61, 76); color: rgb(54, 61, 76); font-size: 15px;">We=E2=80=99re =
exploring pure-PHP solutions to these problems in WordPress in attempts =
to improve the reliability and safety of handling HTML. I=E2=80=99d love =
to hear your thoughts and know if anyone is willing to work with me to =
create an RFC or directly propose patches. We=E2=80=99ve created a step =
function which allows finding the next character reference and decoding =
it separately, enabling some novel features like highlighting the =
character references in source text.</div><div style=3D"caret-color: =
rgb(54, 61, 76); color: rgb(54, 61, 76); font-size: =
15px;"><br></div><div style=3D"caret-color: rgb(54, 61, 76); color: =
rgb(54, 61, 76); font-size: 15px;">Should I propose an RFC for =
this?</div><div style=3D"caret-color: rgb(54, 61, 76); color: rgb(54, =
61, 76); font-size: 15px;"><br></div><div style=3D"caret-color: rgb(54, =
61, 76); color: rgb(54, 61, 76); font-size: 15px;">Warmly,</div><div =
style=3D"caret-color: rgb(54, 61, 76); color: rgb(54, 61, 76); =
font-size: 15px;">Dennis Snell</div><div style=3D"caret-color: rgb(54, =
61, 76); color: rgb(54, 61, 76); font-size: 15px;">Automattic =
Inc.</div></div></div></blockquote><br></div><div>Thanks everyone for =
your feedback so far on the `decode_html()` RFC [<a =
href=3D"https://wiki.php.net/rfc/decode_html">https://wiki.php.net/rfc/dec=
ode_html</a>]</div><div><br></div><div>I=E2=80=99ve updated it replacing =
the new constants with a new `HtmlContext` enum, and the interface seems =
much nicer this way. I particularly like how PHP enforces passing a =
valid value, vs. hoping that the right flag is =
used.</div><div><br></div><div>Additionally I added a section that I =
previously forgot, which highlights the source of the infamous =
mojibake/gremlins. HTML has special rules for remapping the C1 control =
characters, as if they had been stored or recorded for =
Windows-1251.</div><div><br></div><div>Warmly,</div><div>Dennis =
Snell</div><br></body></html>=

--Apple-Mail=_FF52ADD5-65BA-4B75-9DFC-087DFE5A2036--