Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:124326
Date: Tue, 09 Jul 2024 17:00:24 -0700 (PDT)
To: Internals <internals@lists.php.net>
Subject: [PHP-DEV] Decoding HTML and the Ambiguous Ampersand
Message-ID: <80EA6CA9-E14E-4672-A88A-46EFE9E2F3F0@automattic.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="=_F7889AA9-F1EB-48AE-9C71-74EF34A4EBBB"
From: dennis.snell@automattic.com (Dennis Snell)


--=_F7889AA9-F1EB-48AE-9C71-74EF34A4EBBB
Content-Type: text/plain;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable

Greetings all,


The `html_entity_decode( =E2=80=A6 ENT_HTML5 =E2=80=A6 )` function has a nu=
mber of issues that I=E2=80=99d like to correct.


=C2=A0- It=E2=80=99s missing 720 of HTML5=E2=80=99s specified named charact=
er references.
=C2=A0- 106 of these are named character references which do not require a =
trailing semicolon, such as `&acute`
=C2=A0- It=E2=80=99s unaware of the ambiguous ampersand rule, which allows =
these 106 in special circumstances.


HTML5 asserts that the list of named character references will not expand i=
n the future. It can be found authoritatively at the following URL:


https://html.spec.whatwg.org/entities.json


The ambiguous ampersand rule smoothes over legacy behavior from before HTML=
5 where ampersands were not properly encoded in attribute values, specifica=
lly in URL values. For example, in a query string for a search, one might f=
ind `?q=3Ddog&not=3Dcat`. The `&not` in that value would decode to U+AC (=
=C2=AC), but since it=E2=80=99s in an attribute value it will be left as pl=
aintext. Inside normal HTML markup it would transform into `?q=3Ddog=C2=AC=
=3Dcat`. There are related nuances when numeric character references are fo=
und at the end of a string or boundary without the semicolon.


The function signature of `html_entity_decode()` does not currently allow f=
or correcting this behavior. I=E2=80=99d like to propose an RFC or a bug fi=
x which either extends the function (perhaps by adding a new flag like `ENT=
_AMBIGUOUS_AMPERSAND`) or preferably creates a new function. For the missin=
g character references I wonder if it would be enough to add them to the li=
st of default translatable references.


One challenge with the existing function is that the concept of the transla=
tion table stands in contrast with the fixed and static nature of HTML5=
=E2=80=99s replacement tables. A new function or set of functions could ope=
n up spec-compliant decoding while providing helpful methods that are neces=
sary in many common server-side operations:


=C2=A0 - `html_decode( =E2=80=98attribute=E2=80=99 | =E2=80=98data=E2=80=99=
, $raw_text, $input_encoding =3D =E2=80=98utf-8' )`
=C2=A0 - `html_text_contains( =E2=80=98attribute=E2=80=99 | =E2=80=98data=
=E2=80=99, $raw_haystack, $needle, $input_encoding =3D =E2=80=98utf-8=
=E2=80=99 )`
=C2=A0 - `html_text_starts_with( =E2=80=98attribute=E2=80=99 | =E2=80=98dat=
a=E2=80=99, $raw_haystack, $needle, $input_encoding =3D =E2=80=98utf-8=
=E2=80=99 )`


These methods are handy for inspecting things like encoded attribute values=
 in a memory-efficient and processing-efficient way, when it=E2=80=99s not =
necessary to decode the entire value. In common situations, one encounters =
data-URIs with potentially megabytes of image data and processing only the =
first few or tens of bytes can save a lot of overhead.


We=E2=80=99re exploring pure-PHP solutions to these problems in WordPress i=
n attempts to improve the reliability and safety of handling HTML. I=
=E2=80=99d love to hear your thoughts and know if anyone is willing to work=
 with me to create an RFC or directly propose patches. We=E2=80=99ve create=
d a step function which allows finding the next character reference and dec=
oding it separately, enabling some novel features like highlighting the cha=
racter references in source text.


Should I propose an RFC for this?


Warmly,
Dennis Snell
Automattic Inc.






--=_F7889AA9-F1EB-48AE-9C71-74EF34A4EBBB
Content-Type: text/html;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w=
3.org/TR/REC-html40/loose.dtd">
<html>
<head>
<meta charset=3D"utf-8">
<style>body{font-family:-apple-system,Helvetica,Arial,sans-serif;}blockquot=
e{margin:0;padding:0 0 0 12px;border-left:1px solid #aaa;color:#aaa;}hr.uni=
box-forward{border:0;color:#888;background-color:#888;height:1px;}</style>
</head>
<body>
<div>
<div>Greetings all,</div>
<div><br></div>
<div>The `html_entity_decode( =E2=80=A6 ENT_HTML5 =E2=80=A6 )` function has=
 a number of issues that I=E2=80=99d like to correct.</div>
<div><br></div>
<div>=C2=A0- It=E2=80=99s missing 720 of HTML5=E2=80=99s specified named ch=
aracter references.</div>
<div>=C2=A0- 106 of these are named character references which do not requi=
re a trailing semicolon, such as `&amp;acute`</div>
<div>=C2=A0- It=E2=80=99s unaware of the ambiguous ampersand rule, which al=
lows these 106 in special circumstances.</div>
<div><br></div>
<div>HTML5 asserts that the list of named character references will not exp=
and in the future. It can be found authoritatively at the following URL:</d=
iv>
<div><br></div>
<div>https://html.spec.whatwg.org/entities.json</div>
<div><br></div>
<div>The ambiguous ampersand rule smoothes over legacy behavior from before=
 HTML5 where ampersands were not properly encoded in attribute values, spec=
ifically in URL values. For example, in a query string for a search, one mi=
ght find `?q=3Ddog&amp;not=3Dcat`. The `&amp;not` in that value would decod=
e to U+AC (=C2=AC), but since it=E2=80=99s in an attribute value it will be=
 left as plaintext. Inside normal HTML markup it would transform into `?q=
=3Ddog=C2=AC=3Dcat`. There are related nuances when numeric character refer=
ences are found at the end of a string or boundary without the semicolon.</=
div>
<div><br></div>
<div>The function signature of `html_entity_decode()` does not currently al=
low for correcting this behavior. I=E2=80=99d like to propose an RFC or a b=
ug fix which either extends the function (perhaps by adding a new flag like=
 `ENT_AMBIGUOUS_AMPERSAND`) or preferably creates a new function. For the m=
issing character references I wonder if it would be enough to add them to t=
he list of default translatable references.</div>
<div><br></div>
<div>One challenge with the existing function is that the concept of the tr=
anslation table stands in contrast with the fixed and static nature of HTML=
5=E2=80=99s replacement tables. A new function or set of functions could op=
en up spec-compliant decoding while providing helpful methods that are nece=
ssary in many common server-side operations:</div>
<div><br></div>
<div>=C2=A0 - `html_decode( =E2=80=98attribute=E2=80=99 | =E2=80=98data=
=E2=80=99, $raw_text, $input_encoding =3D =E2=80=98utf-8' )`</div>
<div>=C2=A0 - `html_text_contains( =E2=80=98attribute=E2=80=99 | =E2=80=98d=
ata=E2=80=99, $raw_haystack, $needle, $input_encoding =3D =E2=80=98utf-8=
=E2=80=99 )`</div>
<div>=C2=A0 - `html_text_starts_with( =E2=80=98attribute=E2=80=99 | =
=E2=80=98data=E2=80=99, $raw_haystack, $needle, $input_encoding =3D =
=E2=80=98utf-8=E2=80=99 )`</div>
<div><br></div>
<div>These methods are handy for inspecting things like encoded attribute v=
alues in a memory-efficient and processing-efficient way, when it=E2=80=99s=
 not necessary to decode the entire value. In common situations, one encoun=
ters data-URIs with potentially megabytes of image data and processing only=
 the first few or tens of bytes can save a lot of overhead.</div>
<div><br></div>
<div>We=E2=80=99re exploring pure-PHP solutions to these problems in WordPr=
ess in attempts to improve the reliability and safety of handling HTML. I=
=E2=80=99d love to hear your thoughts and know if anyone is willing to work=
 with me to create an RFC or directly propose patches. We=E2=80=99ve create=
d a step function which allows finding the next character reference and dec=
oding it separately, enabling some novel features like highlighting the cha=
racter references in source text.</div>
<div><br></div>
<div>Should I propose an RFC for this?</div>
<div><br></div>
<div>Warmly,</div>
<div>Dennis Snell</div>
<div>Automattic Inc.</div>
</div>
<div><br></div>
<div class=3D"unibox-signature"></div>
</body>
</html>
--=_F7889AA9-F1EB-48AE-9C71-74EF34A4EBBB--