# [RFC] Improve HTML escape
- Thread: https://externals.io/message/71968
- Original archive: https://news-web.php.net/php.internals/71968
- Started by Yasuo Ohgaki on 2014-02-02T03:09:43+00:00
- Messages: 37
- Page 1 of 2: messages 1 to 25 (25 per page, add `?page=N` to the URL)
- Next page: https://externals.io/message/71968.md?page=2
- Reply by email to internals@lists.php.net
- More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt
---
## #71968 — Yasuo Ohgaki — 2014-02-02T03:09:43+00:00
URL: https://externals.io/message/71968#71968
Hi all,
This is a little improvement for HTML escape.
https://wiki.php.net/rfc/secure-html-escape
"/" escape is recommended by OWASP and we may follow them.
Any comments?
Regards,
--
Yasuo Ohgaki
yohgaki@ohgaki.net
---
## #71969 — Sara Golemon — 2014-02-02T03:15:00+00:00
URL: https://externals.io/message/71968#71969 · In reply to: #71968
> This is a little improvement for HTML escape.
> https://wiki.php.net/rfc/secure-html-escape
>
> "/" escape is recommended by OWASP and we may follow them.
>
Could you include some samples of malicious input and what the output
would actually look like? It's not obvious from the RFC or the link
referenced.
-Sara
---
## #71970 — Andrea Faulds — 2014-02-02T03:27:44+00:00
URL: https://externals.io/message/71968#71970 · In reply to: #71968
> "/" escape is recommended by OWASP and we may follow them.
Surely if this is to stop >,
then we'd have to escape ' ' too?
--
Andrea Faulds
http://ajf.me/
---
## #71971 — Yasuo Ohgaki — 2014-02-02T03:31:19+00:00
URL: https://externals.io/message/71968#71971 · In reply to: #71969
Hi Sara,
>
> > […]
> Could you include some samples of malicious input and what the output
> would actually look like? It's not obvious from the RFC or the link
> referenced.
They don't explain as code. AFAIK This is the case for generating invalid
HTML that destroys HTML tag structure.
>
When $str is
sometext /
Produced HTML would be
and tag is closed.
The code is broken in first place since attribute must be enclosed by
"(HTML5/XHTML) or '(HTML4), but many (if not most) browsers just allows
attributes without qoutes.
As long as user don't have other mistakes, it's not a security issue. It's
not vulnerable by itself, but it may be possible do some bad thing on some
implementations. It's just a precaution. It's good precaution as it does
not break any existing browsers. IMHO.
Regards,
--
Yasuo Ohgaki
yohgaki@ohgaki.net
---
## #71972 — Yasuo Ohgaki — 2014-02-02T03:33:37+00:00
URL: https://externals.io/message/71968#71972 · In reply to: #71970
Hi Andrea,
>
>
> > […]
>
> Surely if this is to stop >,
> then we'd have to escape ' ' too?
Making `ENT_QUOTES` as a default is good idea also.
I should have add this to the RFC.
Regards,
--
Yasuo Ohgaki
yohgaki@ohgaki.net
---
## #71973 — Sara Golemon — 2014-02-02T03:35:46+00:00
URL: https://externals.io/message/71968#71973 · In reply to: #71971
> They don't explain as code. AFAIK This is the case for generating invalid
> HTML that destroys HTML tag structure.
>
> >
>
> When $str is
>
> sometext /
>
> Produced HTML would be
>
>
>
> and tag is closed.
>
Oh, I see what the RFC is suggesting now: Include encoding '/' by
default. Yeah, I got no problem with that. Seems completely
reasonable. ((Though I twitch every time I see unquoted attributes))
-Sara
---
## #71981 — pajousek@gmail.com — 2014-02-02T09:38:22+00:00
URL: https://externals.io/message/71968#71981 · In reply to: #71971
> The code is broken in first place since attribute must be enclosed by
> "(HTML5/XHTML) or '(HTML4), but many (if not most) browsers just allows
> attributes without qoutes.
Well, the HTML5 specification says attribute values can be left
unquoted, so I'd say that the "code is broken" statement is invalid.
http://www.w3.org/TR/html-markup/syntax.html#syntax-attr-unquoted
---
## #71984 — Yasuo Ohgaki — 2014-02-02T09:54:38+00:00
URL: https://externals.io/message/71968#71984 · In reply to: #71981
Hi Pavel,
>
> > […]
>
> Well, the HTML5 specification says attribute values can be left
> unquoted, so I'd say that the "code is broken" statement is invalid.
> http://www.w3.org/TR/html-markup/syntax.html#syntax-attr-unquoted
Thank you for heads up!
We must have this change as a security fix, then.
Regards,
--
Yasuo Ohgaki
yohgaki@ohgaki.net
---
## #71986 — Stas Malyshev — 2014-02-02T10:19:03+00:00
URL: https://externals.io/message/71968#71986 · In reply to: #71971
Hi!
> They don't explain as code. AFAIK This is the case for generating invalid
> HTML that destroys HTML tag structure.
>
> >
`htmlentities()` is no good for encoding unquoted tags. This is obvious
since it does not encode space and space is a significant character with
unquoted tags. So if you have code like the above, it's game over for
you, no need to do anything further.
> As long as user don't have other mistakes, it's not a security issue. It's
> not vulnerable by itself, but it may be possible do some bad thing on some
> implementations. It's just a precaution. It's good precaution as it does
> not break any existing browsers. IMHO.
I don't see any reason so far to do this. The code above is broken with
and without quoting /, and quoting / adds nothing to its security as far
as I can see.
--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
---
## #71987 — Stas Malyshev — 2014-02-02T10:21:31+00:00
URL: https://externals.io/message/71968#71987 · In reply to: #71972
Hi!
> Making `ENT_QUOTES` as a default is good idea also.
> I should have add this to the RFC.
Why is it a good idea? Could you explain what it adds to the security of
this function?
--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
---
## #71991 — me@rouvenwessling.de — 2014-02-02T11:08:22+00:00
URL: https://externals.io/message/71968#71991 · In reply to: #71987
> > […]
>
> Why is it a good idea? Could you explain what it adds to the security of
> this function?
I suppose the argument could be made for "safe by default", since single quotes are now valid for HTML attributes as well. (I miss XHTML...)
More interesting to me, what's the use case for ENT_NOQUOTES? This one causes issues whatever attribute syntax one chooses.
Best regards
Rouven
---
## #71997 — padraic.brady@gmail.com — 2014-02-02T13:55:32+00:00
URL: https://externals.io/message/71968#71997 · In reply to: #71991
Hi all,
It's dated but: https://wiki.php.net/rfc/escaper I see Yasuo edited it a
wee bit in September on its 1 year anniversary to add ext/filter as an
option. I had hoped Anthony would get around to it but c'est la vie.
Without quotes you need to escape almost ALL non alphanumeric characters in
an attribute value just to make sure you cover every known and unknown
browser parsing oddity. It's just a bad practice full stop despite HTML5
allowing it.
`ENT_QUOTES` should be the default for obvious reasons. It escapes quotes.
`htmlentities()` doesn't anything more than `htmlspecialchars()` unless you
count turning "Pádraic Ó'Brádaigh" into "Pádraic
Ó'Brádaigh" as a positive benefit to the Irish language and
the size of its webpages :P.
Paddy
--
Pádraic Brady
http://blog.astrumfutura.com
http://www.survivethedeepend.com
Zend Framework Community Review Team
Zend Framework PHP-FIG Representative
---
## #72020 — me@rouvenwessling.de — 2014-02-02T22:55:21+00:00
URL: https://externals.io/message/71968#72020 · In reply to: #71997
> Without quotes you need to escape almost ALL non alphanumeric characters in an attribute value just to make sure you cover every known and unknown browser parsing oddity. It's just a bad practice full stop despite HTML5 allowing it.
>
> `ENT_QUOTES` should be the default for obvious reasons. It escapes quotes.
Just to be clear, the current default (ENT_COMPAT) does escape double quotes. The change to `ENT_QUOTES` would escape single quotes as well.
Best regards
Rouven
---
## #72029 — Yasuo Ohgaki — 2014-02-03T01:43:48+00:00
URL: https://externals.io/message/71968#72029 · In reply to: #71987
Hi Stas,
On Sun, Feb 2, 2014 at 7:21 PM, Stas Malyshev wrote:
> > […]
>
> Why is it a good idea? Could you explain what it adds to the security of
> this function?
Users can do
and this is valid. I think there is no reason not to escape ' by default.
I agree that user should not use unquoted attributes in general.
'/' escape could be still useful. For example, user may have validation
code that allows printable ASCII chars w/o spaces. '/' escape may protect
apps from generating invalid tag in this case.
We could say "your application is broken in first place".
However, both "'" and '/" escapes do not break apps at all, yet it
covers some issues.
There is no reason not to escape these chars by default. IMHO.
Regards,
--
Yasuo Ohgaki
yohgaki@ohgaki.net
---
## #72038 — Yasuo Ohgaki — 2014-02-03T05:33:55+00:00
URL: https://externals.io/message/71968#72038 · In reply to: #72029
Hi all,
> On Sun, Feb 2, 2014 at 7:21 PM, Stas Malyshev wrote:
>
> > […]
>
>
> Users can do
>
>
>
> and this is valid. I think there is no reason not to escape ' by default.
>
> I agree that user should not use unquoted attributes in general.
>
> '/' escape could be still useful. For example, user may have validation
> code that allows printable ASCII chars w/o spaces. '/' escape may protect
> apps from generating invalid tag in this case.
>
> We could say "your application is broken in first place".
> However, both "'" and '/" escapes do not break apps at all, yet it
> covers some issues.
>
> There is no reason not to escape these chars by default. IMHO.
>
Even we may deprecate `ENT_COMPAT` and ENT_QUOTES. We may ignore
them and escape all chars recommended by OWASP always. (Except ENT_NOQUOTES)
I think ENT_COMPAT/ENT_QUOTES does bad things rather than good.
Regards,
--
Yasuo Ohgaki
yohgaki@ohgaki.net
---
## #72055 — Stas Malyshev — 2014-02-03T08:17:58+00:00
URL: https://externals.io/message/71968#72055 · In reply to: #72029
Hi!
> Users can do
>
>
They also can do and . That's
not what they _should_ be doing, but they _can_ do it. That doesn't mean
there's something wrong with echo or PHP compiler.
> and this is valid. I think there is no reason not to escape ' by default.
>
> I agree that user should not use unquoted attributes in general.
>
> '/' escape could be still useful. For example, user may have validation
I don't see how it would be useful.
> code that allows printable ASCII chars w/o spaces. '/' escape may protect
> apps from generating invalid tag in this case.
This seems to be a very contrives scenario invented to fit your point.
If they already pre-filter input, they could also remove / or other
special characters. The fact is that htmlentities is useless as security
feature in this context, and removing / does not make it useful. Saying
"we'll add escape so that it would be safe" is magic-quotes kind of
mistake - it gives the users wrong impression that it's OK to do things
that they should not be doing.
> There is no reason not to escape these chars by default. IMHO.
There is a reason - there's no reason to escape them. In every scenario
that htmlentites should be used, escaping them is useless. In every
scenario where espacing / is useful, htmlentities should not be used. By
promoting usage of htmlentities in scenarios where it should absolutely
not be used, we are only doing the users a disservice.
--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
---
## #72079 — padraic.brady@gmail.com — 2014-02-03T11:06:22+00:00
URL: https://externals.io/message/71968#72079 · In reply to: #72055
Hi Stas,
>
> Hi!
>
> > […]
>
> They also can do and . That's
> not what they _should_ be doing, but they _can_ do it. That doesn't mean
> there's something wrong with echo or PHP compiler.
I don't believe this has anything to do with the question at hand.
> > […]
>
> I don't see how it would be useful.
I'm not sure it is either. OWASP definitely notes it, but it's not an attribute
termination character inside quotes.
> > […]
>
> There is a reason - there's no reason to escape them. In every scenario
> that htmlentites should be used, escaping them is useless. In every
> scenario where espacing / is useful, htmlentities should not be used. By
> promoting usage of htmlentities in scenarios where it should absolutely
> not be used, we are only doing the users a disservice.
There are three ways to present an attribute value validly in HTML5:
1. Double quoted
2. Single quoted
3. Unquoted.
Bearing in mind that people who use `htmlentities()` make a mockery of UTF-8 by
overescaping and increasing output page size for no good reason whatsoever, both
`htmlspecialchars()` and `htmlentities()` only work by default for the first option.
They do not work by default for the last two options.
In userland, virtually all security-concious libraries and frameworks cover TWO
options: 1 and 2 by setting ENT_QUOTES. It seems reasonable for PHP to make the
change also unless it has some hitherto unmentioned downside.
Also, for reference, here is the actual paragraph from the OWASP XSS cheatsheet:
"Escape the following characters with HTML entity encoding to prevent switching
into any execution context, such as script, style, or event handlers. Using hex
entities is recommended in the spec. In addition to the 5 characters significant
in XML (&, <, >, ", '), the forward slash is included as it helps to end an HTML
entity."
I read "entity" as "tag".
--
Pádraic Brady
http://blog.astrumfutura.com
http://www.survivethedeepend.com
Zend Framework Community Review Team
Zend Framework PHP-FIG Representative
---
## #72137 — Yasuo Ohgaki — 2014-02-03T22:04:07+00:00
URL: https://externals.io/message/71968#72137 · In reply to: #72055
Hi Stas,
On Mon, Feb 3, 2014 at 5:17 PM, Stas Malyshev wrote:
> > […]
>
> They also can do and . That's
> not what they _should_ be doing, but they _can_ do it. That doesn't mean
> there's something wrong with echo or PHP compiler.
>
> > […]
>
> I don't see how it would be useful.
>
> > […]
>
> This seems to be a very contrives scenario invented to fit your point.
> If they already pre-filter input, they could also remove / or other
> special characters. The fact is that htmlentities is useless as security
> feature in this context, and removing / does not make it useful. Saying
> "we'll add escape so that it would be safe" is magic-quotes kind of
> mistake - it gives the users wrong impression that it's OK to do things
> that they should not be doing.
>
> > […]
>
> There is a reason - there's no reason to escape them. In every scenario
> that htmlentites should be used, escaping them is useless. In every
> scenario where espacing / is useful, htmlentities should not be used. By
> promoting usage of htmlentities in scenarios where it should absolutely
> not be used, we are only doing the users a disservice.
I think we have different perspectives.
Some users has to confirm standard like PCI DSS.
PCI DSS requires to follow security standards and guidelines from OWASP,
SANS, etc.
Why not make PHP standard compliant?
It does not hart existing applications at all and this is simple enough
change.
Regards,
--
Yasuo Ohgaki
yohgaki@ohgaki.net
---
## #72141 — Stas Malyshev — 2014-02-03T22:14:27+00:00
URL: https://externals.io/message/71968#72141 · In reply to: #72137
Hi!
> Some users has to confirm standard like PCI DSS.
> PCI DSS requires to follow security standards and guidelines from OWASP,
> SANS, etc.
>
> Why not make PHP standard compliant?
> It does not hart existing applications at all and this is simple enough
> change.
I'm sorry, could you please quote me a standard that requires PHP to
escape / in function called htmlentites? If there's no such standard,
the argument of "but the standard requires it" is void. No standard can
require you to use htmlentites where it should not be used. Putting
stuff into language just because somebody in the internet mentioned in
different context that it might be a good idea - is not. We should
understand _why_ it is done and _why_ it is a good idea, especially when
we're talking about security. In this case, the proposed use case should
_never_ be used with htmlentities, due to obvious gaping security hole.
Adding code to enable such scenario is just not right. Instead, we
should tell people "Never ever do it. Ever.".
--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
---
## #72143 — Yasuo Ohgaki — 2014-02-03T22:15:31+00:00
URL: https://externals.io/message/71968#72143 · In reply to: #72079
Hi Padraic,
On Mon, Feb 3, 2014 at 8:06 PM, Pádraic Brady wrote:
> There are three ways to present an attribute value validly in HTML5:
>
> 1. Double quoted
> 2. Single quoted
> 3. Unquoted.
>
Unquoted is really bad standard with respect to security. I don't
understand why
they allow unquoted attributes, but I think we need to address this some
how.
htmlentities/htmlspecialchars may have ENT_NO_SPACE as an option. If
there is space char, null string is returned. Standard allows space before
attributes. User may write
>
Use of this option is not recommended, but there is the standard. We may
support it even if we don't recommend it.
Regards,
--
Yasuo Ohgaki
yohgaki@ohgaki.net
---
## #72145 — Stas Malyshev — 2014-02-03T22:21:08+00:00
URL: https://externals.io/message/71968#72145 · In reply to: #72143
Hi!
> Use of this option is not recommended, but there is the standard. We may
> support it even if we don't recommend it.
Nowhere in any standard it says we must use htmlentities to support
every possible context. There are contexts where htmlentities is
completely unsuitable - such as unquoted attributes, Javascript, CSS,
etc. In these contexts, other ways of escaping output should be used.
I get an impression you're trying to fit a square peg into a round hole
here. There are other ways to escape things and they should match the
context the output is used in. Trying to serve every scenario with one
function would not work.
--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
---
## #72147 — Yasuo Ohgaki — 2014-02-03T22:21:45+00:00
URL: https://externals.io/message/71968#72147 · In reply to: #72141
Hi Stas,
On Tue, Feb 4, 2014 at 7:14 AM, Stas Malyshev wrote:
> > […]
>
> I'm sorry, could you please quote me a standard that requires PHP to
> escape / in function called htmlentites?
I've already written the URL to OWASP.
PCI DSS v3 states in section 6.5
Develop applications based on secure coding guidelines.
Note: The vulnerabilities listed at 6.5.1 through 6.5.10 were current with
industry best
practices when this version of PCI DSS was published. However, as industry
best
practices for vulnerability management are updated (for example, the OWASP
Guide,
SANS CWE Top 25, CERT Secure Coding, etc.), the current best practices must
be used
for these requirements.
Regards,
--
Yasuo Ohgaki
yohgaki@ohgaki.net
---
## #72148 — Stas Malyshev — 2014-02-03T22:24:22+00:00
URL: https://externals.io/message/71968#72148 · In reply to: #72147
Hi!
> I've already written the URL to OWASP.
>
> PCI DSS v3 states in section 6.5
>
> Develop applications based on secure coding guidelines.
Secure coding guidelines in this case is to not use htmlentities in this
context. If you already violate this requirement, why would you expect
PHP to un-violate it for you?
--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
---
## #72149 — Yasuo Ohgaki — 2014-02-03T22:26:47+00:00
URL: https://externals.io/message/71968#72149 · In reply to: #72145
Hi Stas,
On Tue, Feb 4, 2014 at 7:21 AM, Stas Malyshev wrote:
> > […]
>
> Nowhere in any standard it says we must use htmlentities to support
> every possible context. There are contexts where htmlentities is
> completely unsuitable - such as unquoted attributes, Javascript, CSS,
> etc. In these contexts, other ways of escaping output should be used.
>
> I get an impression you're trying to fit a square peg into a round hole
> here. There are other ways to escape things and they should match the
> context the output is used in. Trying to serve every scenario with one
> function would not work.
We may or may not support unquoted attributes.
I think it's really dangerous, therefore we my not support it ;)
It may be good for PHP to declare "We support HTML5!", though.
Regards,
--
Yasuo Ohgaki
yohgaki@ohgaki.net
---
## #72150 — padraic.brady@gmail.com — 2014-02-03T22:31:14+00:00
URL: https://externals.io/message/71968#72150 · In reply to: #72141
Stas,
> Hi!
>
> > […]
>
> I'm sorry, could you please quote me a standard that requires PHP to
> escape / in function called htmlentites? If there's no such standard,
> the argument of "but the standard requires it" is void. No standard can
> require you to use htmlentites where it should not be used. Putting
> stuff into language just because somebody in the internet mentioned in
> different context that it might be a good idea - is not. We should
> understand _why_ it is done and _why_ it is a good idea, especially when
> we're talking about security. In this case, the proposed use case should
> _never_ be used with htmlentities, due to obvious gaping security hole.
> Adding code to enable such scenario is just not right. Instead, we
> should tell people "Never ever do it. Ever.".
There is far too much going on here...
1. Bear in mind that `htmlentities()` and `htmlspecialchars()` are
equivalent for HTML special characters.
2. PCI DSS is a real standard that real people apply in real
applications. You can google it if you don't believe me. It's at
version 3.0.
3. PCI DSS specifically notes OWASP guides as a source of best
practice as part of Requirement 6 (which covers XSS among other
things).
4. The OWASP guide for XSS mentions escaping the forward slash.
5. We do not currently escape the forward slash.
While I'm dubious about forward slash escaping myself and think it
might have been OWASP veering into overkill, it doesn't change the
fact that Yasuo's argument is perfectly sound. Nor does it change the
state of single quote escaping which is very obvious out of sync with
best practice.
Ignoring all this is tantamount to telling people not to use
`htmlspecialchars()` or `htmlentities()` at all. As in, never! Which,
coincidentally, is almost the current best practice in PHP where we
wrap those functions for the sake of insulating ourselves from its
more colourful behaviours. Yep, the standard is not to use these
functions as-is. It would be nice to see them fixed or replaced, but
I'm not holding my breath.
Paddy
--
Pádraic Brady
http://blog.astrumfutura.com
http://www.survivethedeepend.com
Zend Framework Community Review Team
Zend Framework PHP-FIG Representative
---
Next page: https://externals.io/message/71968.md?page=2