Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:125191
Message-ID: <90B08F35-06D5-4D5C-BA7B-B7116EE18769@automattic.com>
Content-Type: multipart/alternative;
	boundary="Apple-Mail=_784EE0E2-94F8-47D6-9D26-DDED17B097D5"
Precedence: bulk
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51\))
Subject: Re: [PHP-DEV] [RFC] Decoding HTML and the Ambiguous Ampersand
Date: Sat, 24 Aug 2024 15:31:17 -0500
In-Reply-To: <CAJmjfvXvBpxc-t2SkrK587Cm+=iOY9tNmpt-SC-pgg7pALJ9XA@mail.gmail.com>
Cc: Niels Dossche <dossche.niels@gmail.com>,
 Internals <internals@lists.php.net>
To: Jakob Givoni <jakob@givoni.dk>
References: <CAA8B983-FD39-4AB1-B314-62608B0C0EF4@a8c.com>
 <76D9E1DA-57CE-45C3-8E3E-B08A0B70FB60@a8c.com>
 <efaf4c62-a552-4232-8a22-410578c13b8d@gmail.com>
 <7ED2EE07-D7C6-43A4-A4E1-E9928E8B8D31@automattic.com>
 <CAJmjfvXvBpxc-t2SkrK587Cm+=iOY9tNmpt-SC-pgg7pALJ9XA@mail.gmail.com>
From: dennis.snell@automattic.com (Dennis Snell)


--Apple-Mail=_784EE0E2-94F8-47D6-9D26-DDED17B097D5
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

On Aug 24, 2024, at 2:56=E2=80=AFPM, Jakob Givoni <jakob@givoni.dk> =
wrote:
>=20
> Hi Dennis,
>=20
> Overall it sounds like a reasonable RFC.
>  =20
> > Dennis:
> >
> > > Niels:
> > >
> > > I'm not so sure that the name "decode_html" is self-descriptive =
enough, it sounds very generic.
> >
> > The name is not very important to me. For the sake of history, the =
reason I have chosen =E2=80=9Cdecode HTML=E2=80=9D is because, unlike an =
HTML parser, this is focused on taking a snippet of HTML =E2=80=9Ctext=E2=80=
=9D content and decoding it into a =E2=80=9Cplain PHP string.=E2=80=9D
>=20
> Why not make it two methods called "decode_html_text" and =
"decode_html_attribute"?
> Consider the following reasons:
> 1. The function doesn't actually decode html as such, it decodes =
either an html text node string or an html attribute string.

Thanks Jakob. In WordPress I did just this.
https://developer.wordpress.org/reference/classes/wp_html_decoder/

Part of the reason for that was the inability to require something like =
an enum (due to PHP version support requirements). The Enum solution =
feels very nice too.

> 2. Saves the $context parameter and the constants/enums, making the =
call significantly shorter.=20

In my PR I=E2=80=99ve actually expanded the Enum to include a few other =
contexts. I feel like there=E2=80=99s a balance we have to do if we want =
to ride the line between fully reliable and fully convenient. On one =
hand, we could say =E2=80=9Cdon=E2=80=99t send the text content of a =
SCRIPT element to this function!=E2=80=9D But on the other hand, that =
kind of forces people to expect that SCRIPT content is different.

With the Enum there is that in-built training material when someone =
looks and finds `Attribute | BodyText | ForeignText | Script | Style` =
(the contexts I=E2=80=99ve explored in my PR).=20

We could make the same argument for `decode_html_script()` and =
`decode_foreign_text_node()` and `decode_html_style()`. Somehow the =
context feels cleaner to me, and like a single entry point for learning =
instead of five.

> 3. It feels like decoding either text or attribute are two =
significantly different things. I admit I could be wrong, if code like =
decode_html($e->isAttritbute() ? HtmlContext::Attribute : =
HtmlContext::Text, $e->getContent()) is likely to be seen.

None of these contexts are significantly different, which is one of the =
major dangers of using `html_entity_decode()`. The results will look =
just about right most of the time. It=E2=80=99s the subtle differences =
that matter most, I suppose. Thankfully, in most places I=E2=80=99ve =
seen them blurred together, the intent of the code someone is writing =
understands which is which.

    preg_replace_callback(
        =E2=80=98~<a[^>]+href=3D=E2=80=9C([^=E2=80=9D]+)=E2=80=9D[^>]*>([^=
<]+)</a>~=E2=80=99,
        function ( $m ) {
            $title =3D str_replace( =E2=80=98]=E2=80=99, =E2=80=98\]=E2=80=
=99, html_entity_decode( $m[2] ) );
            $url =3D str_replace( =E2=80=98)=E2=80=99, =E2=80=98\)=E2=80=99=
, html_entity_decode( $m[1] ) );
            return =E2=80=9C[{$title}]({$url})=E2=80=9D;
        }
        $post_content
    );

The lesson I have drawn is that people frequently have what they =
understand to be a text node or an attribute value, but they aren=E2=80=99=
t aware that they are supposed to decode differently, and they also =
aren=E2=80=99t reaching to interact with a full parser to get these =
values. If PHP could train people as they use these functions, purely =
through their interfaces, I think that could help elevate the level of =
reliability out there in the wild, as long as they aren=E2=80=99t too =
cumbersome (hence explicitly no default context argument _or_ using =
separately-named functions).

Having the Enum I think enhances the ease with which people can reliably =
also decode things like SCRIPT and STYLE nodes. =E2=80=9CI know =
`html_decode_text()` but I don=E2=80=99t know what the rules for SCRIPT =
are or if they=E2=80=99re different so I=E2=80=99ll just stick with =
that.=E2=80=9D vs =E2=80=9CMy IDE suggests that `Script` is a different =
context, that=E2=80=99s interesting, I=E2=80=99ll try that and see how =
it=E2=80=99s different."

>  But I somehow don't foresee a lot of situations where text and =
attribute strings end up in the same code path?

The underlying reason I started this work was in support of building an =
HTML parser. We have a streaming parser which relies on a different =
parsing model than those built purely on the state machine in the =
specification, taking advantage of what we can to eek out performance in =
PHP code. For this, the strings are in the same path, and in this work =
I=E2=80=99ve come across a number of other common use-cases where the =
flow is the same but the decoder needs to know the context.

 - Normalizing HTML from =E2=80=9Ctag soup=E2=80=9D to standard =
serialized form.
 - Sanitizing code wanting to inspect values from different parts of the =
markup.
 - Sanitizing rules engines providing configurations or DSLs for =
sanitization.
 - Live optimizers or analyzers to improve the output HTML leaving a =
server.

It=E2=80=99s one of those things that when it becomes trivial to start =
getting reliable transforms from the HTML syntax to the decoded text, =
more opportunities appear that never seemed practical before.

>=20
> A couple of other options that would silence anyone opposed to =
implicitly favouring utf-8:
> html_text_to_utf8 and html_attribute_to_utf8

The names started with these =F0=9F=98=80. I do agree that it gets a bit =
excessive though to the point where it risks people not adopting them =
purely because they don=E2=80=99t want to type that long of a name every =
time they use it. Perhaps some of these =F0=9F=99=83

    str_from_html( HtmlContext $context, string $html ): string {}

    utf8_from_html( HtmlContext $context, string $html ): string {}

    html_to_utf8( HtmlContext $context, string $html ): string {}

>=20
> Best,
> Jakob
> =20

Thanks for your input. I=E2=80=99m grateful for the discussions and that =
people are sharing.

Dennis Snell


--Apple-Mail=_784EE0E2-94F8-47D6-9D26-DDED17B097D5
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8

<html><head><meta http-equiv=3D"content-type" content=3D"text/html; =
charset=3Dutf-8"></head><body style=3D"overflow-wrap: break-word; =
-webkit-nbsp-mode: space; line-break: after-white-space;">On Aug 24, =
2024, at 2:56=E2=80=AFPM, Jakob Givoni &lt;jakob@givoni.dk&gt; =
wrote:<br><div><blockquote type=3D"cite"><br =
class=3D"Apple-interchange-newline"><div><div dir=3D"ltr">Hi =
Dennis,<br><br>Overall it sounds like a reasonable RFC.<div>&nbsp; =
<br>&gt; Dennis:</div><div>&gt;<br>&gt; &gt; Niels:</div><div>&gt; =
&gt;<br>&gt; &gt; I'm not so sure that the name "decode_html" is =
self-descriptive enough, it sounds very generic.<br>&gt;<br>&gt; The =
name is not very important to me. For the sake of history, the reason I =
have chosen =E2=80=9Cdecode HTML=E2=80=9D is because, unlike an HTML =
parser, this is focused on taking a snippet of HTML =E2=80=9Ctext=E2=80=9D=
 content and decoding it into a =E2=80=9Cplain PHP string.=E2=80=9D<br><br=
>Why not make it two methods called "decode_html_text" and =
"decode_html_attribute"?</div><div>Consider the following =
reasons:</div><div>1. The function doesn't actually decode html as such, =
it decodes either an html text node string or an html attribute =
string.</div></div></div></blockquote><div><br></div><div>Thanks Jakob. =
In WordPress I did just this.</div><div><a =
href=3D"https://developer.wordpress.org/reference/classes/wp_html_decoder/=
">https://developer.wordpress.org/reference/classes/wp_html_decoder/</a></=
div><div><br></div><div>Part of the reason for that was the inability to =
require something like an enum (due to PHP version support =
requirements). The Enum solution feels very nice =
too.</div><br><blockquote type=3D"cite"><div><div dir=3D"ltr"><div>2. =
Saves the $context parameter and the constants/enums, making the call =
significantly =
shorter.&nbsp;</div></div></div></blockquote><div><br></div><div>In my =
PR I=E2=80=99ve actually expanded the Enum to include a few other =
contexts. I feel like there=E2=80=99s a balance we have to do if we want =
to ride the line between <i>fully reliable</i>&nbsp;and <i>fully =
convenient</i>. On one hand, we could say =E2=80=9Cdon=E2=80=99t send =
the text content of a SCRIPT element to this function!=E2=80=9D But on =
the other hand, that kind of forces people to expect that SCRIPT content =
is different.</div><div><br></div><div>With the Enum there is that =
in-built training material when someone looks and finds `Attribute | =
BodyText | ForeignText | Script | Style` (the contexts I=E2=80=99ve =
explored in my PR).&nbsp;</div><div><br></div><div>We could make the =
same argument for `decode_html_script()` and =
`decode_foreign_text_node()` and `decode_html_style()`. Somehow the =
context feels cleaner to me, and like a single entry point for learning =
instead of five.</div><br><blockquote type=3D"cite"><div><div =
dir=3D"ltr"><div>3. It feels like decoding either text or attribute are =
two significantly different things. I admit I could be wrong, if code =
like decode_html($e-&gt;isAttritbute() ? HtmlContext::Attribute : =
HtmlContext::Text, $e-&gt;getContent()) is likely to be =
seen.</div></div></div></blockquote><div><br></div><div>None of these =
contexts are <i>significantly</i>&nbsp;different, which is one of the =
major dangers of using `html_entity_decode()`. The results will look =
just about right most of the time. It=E2=80=99s the subtle differences =
that matter most, I suppose. Thankfully, in most places I=E2=80=99ve =
seen them blurred together, the intent of the code someone is writing =
understands which is which.</div><div><br></div><div>&nbsp; &nbsp; =
preg_replace_callback(</div><div>&nbsp; &nbsp; &nbsp; &nbsp; =
=E2=80=98~&lt;a[^&gt;]+href=3D=E2=80=9C([^=E2=80=9D]+)=E2=80=9D[^&gt;]*&gt=
;([^&lt;]+)&lt;/a&gt;~=E2=80=99,</div><div>&nbsp; &nbsp; &nbsp; &nbsp; =
function ( $m ) {</div><div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; =
$title =3D str_replace( =E2=80=98]=E2=80=99, =E2=80=98\]=E2=80=99, =
html_entity_decode( $m[2] ) );</div><div>&nbsp; &nbsp; &nbsp; &nbsp; =
&nbsp; &nbsp; $url =3D str_replace( =E2=80=98)=E2=80=99, =E2=80=98\)=E2=80=
=99, html_entity_decode( $m[1] ) );</div><div>&nbsp; &nbsp; &nbsp; =
&nbsp; &nbsp; &nbsp; return =E2=80=9C[{$title}]({$url})=E2=80=9D;</div><di=
v>&nbsp; &nbsp; &nbsp; &nbsp; }</div><div>&nbsp; &nbsp; &nbsp; &nbsp; =
$post_content</div><div>&nbsp; &nbsp; );</div><div><br></div><div>The =
lesson I have drawn is that people frequently have what they understand =
to be a text node or an attribute value, but they aren=E2=80=99t aware =
that they are supposed to decode differently, and they also aren=E2=80=99t=
 reaching to interact with a full parser to get these values. If PHP =
could train people as they use these functions, purely through their =
interfaces, I think that could help elevate the level of reliability out =
there in the wild, as long as they aren=E2=80=99t =
<i>too</i>&nbsp;cumbersome (hence explicitly no default context argument =
_or_ using separately-named functions).</div><div><br></div><div>Having =
the Enum I think enhances the ease with which people can reliably also =
decode things like SCRIPT and STYLE nodes. =E2=80=9CI know =
`html_decode_text()` but I don=E2=80=99t know what the rules for SCRIPT =
are or if they=E2=80=99re different so I=E2=80=99ll just stick with =
that.=E2=80=9D vs =E2=80=9CMy IDE suggests that `Script` is a different =
context, that=E2=80=99s interesting, I=E2=80=99ll try that and see how =
it=E2=80=99s different."</div><div><br></div><blockquote =
type=3D"cite"><div dir=3D"ltr">&nbsp;But I somehow don't foresee a lot =
of situations where text and attribute strings end up in the same code =
path?</div></blockquote><div><br></div><div>The underlying reason I =
started this work was in support of building an HTML parser. We have a =
streaming parser which relies on a different parsing model than those =
built purely on the state machine in the specification, taking advantage =
of what we can to eek out performance in PHP code. For this, the strings =
are in the same path, and in this work I=E2=80=99ve come across a number =
of other common use-cases where the flow is the same but the decoder =
needs to know the context.</div><div><br></div><div>&nbsp;- Normalizing =
HTML from =E2=80=9Ctag soup=E2=80=9D to standard serialized =
form.</div><div>&nbsp;- Sanitizing code wanting to inspect values from =
different parts of the markup.</div><div>&nbsp;- Sanitizing rules =
engines providing configurations or DSLs for =
sanitization.</div><div>&nbsp;- Live optimizers or analyzers to improve =
the output HTML leaving a server.</div><div><br></div><div>It=E2=80=99s =
one of those things that when it becomes trivial to start getting =
reliable transforms from the HTML syntax to the decoded text, more =
opportunities appear that never seemed practical =
before.</div><br><blockquote type=3D"cite"><div><div =
dir=3D"ltr"><div><br></div><div>A couple of other options that would =
silence anyone opposed to implicitly favouring =
utf-8:</div><div>html_text_to_utf8 and =
html_attribute_to_utf8</div></div></div></blockquote><div><br></div><div>T=
he names started with these =F0=9F=98=80. I do agree that it gets a bit =
excessive though to the point where it risks people =
<i>not</i>&nbsp;adopting them purely because they don=E2=80=99t want to =
type that long of a name every time they use it. Perhaps some of these =
=F0=9F=99=83</div><div><br></div><div>&nbsp; &nbsp; str_from_html( =
HtmlContext $context, string $html ): string =
{}</div><div><br></div><div>&nbsp; &nbsp; utf8_from_html( HtmlContext =
$context, string $html ): string {}</div><div><br></div><div>&nbsp; =
&nbsp; html_to_utf8( HtmlContext $context, string $html ): string =
{}</div><br><blockquote type=3D"cite"><div><div =
dir=3D"ltr"><div><br></div><div>Best,</div><div>Jakob</div><div>&nbsp;</di=
v></div>
</div></blockquote><br></div><div>Thanks for your input. I=E2=80=99m =
grateful for the discussions and that people are =
sharing.</div><div><br></div><div>Dennis Snell</div><br></body></html>=

--Apple-Mail=_784EE0E2-94F8-47D6-9D26-DDED17B097D5--