# bug #43941
- Thread: https://externals.io/message/40032
- Original archive: https://news-web.php.net/php.internals/40032
- Started by Stanislav Malyshev on 2008-08-21T01:23:21+00:00
- Messages: 10
- Reply by email to internals@lists.php.net
- More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt
---
## #40032 — Stanislav Malyshev — 2008-08-21T01:23:21+00:00
URL: https://externals.io/message/40032#40032
Hi!
Are there any objections to incorporating bugfix for #43941 (fix for how
json handles invalid UTF-8 sequences) into 5.2? I had some requests
about it, right now it's only in 5.3+.
--
Stanislav Malyshev, Zend Software Architect
stas@zend.com http://www.zend.com/
(408)253-8829 MSN: stas@zend.com
---
## #40033 — William A. Rowe — 2008-08-21T01:34:48+00:00
URL: https://externals.io/message/40032#40033 · In reply to: #40032
Stanislav Malyshev wrote:
> Hi!
>
> Are there any objections to incorporating bugfix for #43941 (fix for how
> json handles invalid UTF-8 sequences) into 5.2? I had some requests
> about it, right now it's only in 5.3+.
Is there the alternative of substituting an unmappable character FFFD in
place of the invalid sequence? This a a reasonable alternative behavior
for some less stringent cases.
(Yes, the fix is better than the status quo, but just taking this a step
further).
---
## #40034 — Rasmus Lerdorf — 2008-08-21T01:41:54+00:00
URL: https://externals.io/message/40032#40034 · In reply to: #40032
Stanislav Malyshev wrote:
> Hi!
>
> Are there any objections to incorporating bugfix for #43941 (fix for how
> json handles invalid UTF-8 sequences) into 5.2? I had some requests
> about it, right now it's only in 5.3+.
Go ahead. It is an obvious bug fix.
-Rasmus
---
## #40043 — david.zuelke@bitextender.com — 2008-08-21T10:50:02+00:00
URL: https://externals.io/message/40032#40043 · In reply to: #40033
Am 21.08.2008 um 03:34 schrieb William A. Rowe, Jr.:
> Stanislav Malyshev wrote:
> > […]
>
> Is there the alternative of substituting an unmappable character
> FFFD in
> place of the invalid sequence? This a a reasonable alternative
> behavior
> for some less stringent cases.
>
> (Yes, the fix is better than the status quo, but just taking this a
> step
> further).
I agree, that would be quite reasonable and also more consistent with
how UTF-8 works in other apps (browsers etc).
David
---
## #40055 — david.zuelke@bitextender.com — 2008-08-21T16:16:44+00:00
URL: https://externals.io/message/40032#40055
Am 21.08.2008 um 18:08 schrieb Rasmus Lerdorf:
> David Zülke wrote:
> > […]
>
> Well, using browsers as the benchmark here is a bad idea. IE is
> absolutely braindead about dealing with illegal UTF-8 chars. It
> will accept just about any sequence of bytes as a valid UTF-8 char
> which causes all sorts of problems.
I was talking about the common representation of an invalid sequence.
That's the question mark sign you usually see in a browser when the
encoding is incorrect.
According to the Unicode standard, U+FFFD is supposed to be used as
the replacement character instead of simply stripping invalid ones:
> Replacement Character. A character used as a substitute for an
> uninterpretable character from another encoding. The Unicode
> Standard uses U+FFFD replacement character for this function.
says http://unicode.org/glossary/#replacement_character
> Rendering software which cannot process a Unicode character
> appropriately most often display it as only an open rectangle, or
> the Unicode “replacement character” (U+FFFD, �), to indicate
> the position of the unrecognized character.
says http://en.wikipedia.org/wiki/Unicode#Standardized_subsets
Also see http://www.fileformat.info/info/unicode/char/fffd/index.htm
As always, I consider sticking to specs good practice, so doing it in
the above case would be wise :)
Hope that helps,
David
---
## #40057 — Rasmus Lerdorf — 2008-08-21T16:41:10+00:00
URL: https://externals.io/message/40032#40057 · In reply to: #40055
David Zülke wrote:
> Am 21.08.2008 um 18:08 schrieb Rasmus Lerdorf:
>
> > […]
>
> I was talking about the common representation of an invalid sequence.
> That's the question mark sign you usually see in a browser when the
> encoding is incorrect.
Yes, but it all comes down to how you do it. Say you have a 3 byte
sequence that starts with 0xE0 (E0 indicates the start of a 3-byte utf-8
char) but the 3 bytes together don't actually make up a valid utf-8
char. Id you substitute those 3 bytes with a ? or some other character
you have just created a nasty XSS vector for web apps.
And yes, that is exactly what IE does and it has caused us no end of
headaches over the years.
-Rasmus
---
## #40058 — david.zuelke@bitextender.com — 2008-08-21T16:44:33+00:00
URL: https://externals.io/message/40032#40058 · In reply to: #40057
Am 21.08.2008 um 18:41 schrieb Rasmus Lerdorf:
> David Zülke wrote:
> > […]
>
> Yes, but it all comes down to how you do it. Say you have a 3 byte
> sequence that starts with 0xE0 (E0 indicates the start of a 3-byte
> utf-8 char) but the 3 bytes together don't actually make up a valid
> utf-8 char. Id you substitute those 3 bytes with a ? or some other
> character you have just created a nasty XSS vector for web apps.
You don't substitute it with "a ? or some other character", you
replace it with U+FFFD (0xEF 0xBF 0xBD in UTF-8). I'd love to hear how
that causes an attack vector.
David
---
## #40060 — david.zuelke@bitextender.com — 2008-08-21T16:56:10+00:00
URL: https://externals.io/message/40032#40060
Am 21.08.2008 um 18:50 schrieb Rasmus Lerdorf:
> David Zülke wrote:
> > […]
>
> It doesn't matter what you replace it with. If the byte sequence is:
>
> 0xE0 " >
>
> And you replace those bytes with some other byte in this sort of
> context:
>
>
>
>
> Now do your silly replacement:
>
>
>
> That now means that IE interprets the value attribute of the foo
> element as: value="0xEF 0xBF 0xBD
should never be regarded a valid sequence since neither " nor > are in
the range above 0x7F...
David
---
## #40062 — david.zuelke@bitextender.com — 2008-08-21T17:08:40+00:00
URL: https://externals.io/message/40032#40062
Am 21.08.2008 um 19:03 schrieb Rasmus Lerdorf:
> David Zülke wrote:
> > […]
>
> But that's what we are talking about. What to do with invalid
> sequences. The E0 says that the following 2 bytes are part of the
> UTF-8 character. So this is a 3-byte sequence. Together these 3
> bytes are not valid, so Microsoft chose to replace those 3 with some
> other character. And yes, Microsoft is notoriously bad at reading
> specs, but I don't think it is completely clear what to do here, but
> we do know that we shouldn't do that.
Well to me, the invalid part would be "0xE0" since it is incomplete
(0x7F and below are never part of multi-byte sequences, so they don't
count into the sequence here), so 0xE0 would be replaced by 0xEF 0xBF
0xBD, and then you don't have an XSS unless I'm mistaken :)
If Microsoft regards 0x7F and below as valid sequence members, then
that is unfortunate, but that shouldn't stop PHP from doing it
properly, as we all know better now, don't we :)
I mean what does the patch do in that case? Strip it all? Then it's
the same problem. Strip 0xE0 only? Then we could just as well insert U
+FFFD instead. No difference.
David
---
## #40064 — William A. Rowe — 2008-08-21T23:21:19+00:00
URL: https://externals.io/message/40032#40064 · In reply to: #40060
David Zülke wrote:
> Am 21.08.2008 um 18:50 schrieb Rasmus Lerdorf:
>
> > […]
>
> Interesting. I assume that was a weakness in the respective
> implementation, right? Since
>
> 0xE0 " >
>
> should never be regarded a valid sequence since neither " nor > are in
> the range above 0x7F...
This is (obviously) given to multiple intepretations.
But when I suggested the feature, I mentioned for "less stringent apps".
Rasmus' case, the URL, should be more stringent and reject those which
contained wholly invalid utf-8 sequences, for short sequences, overlong
sequences and outright unmappable bytes.