bug #43941

Hi!

Are there any objections to incorporating bugfix for #43941 (fix for how
json handles invalid UTF-8 sequences) into 5.2? I had some requests
about it, right now it's only in 5.3+.

Stanislav Malyshev, Zend Software Architect
stas@zend.com http://www.zend.com/
(408)253-8829 MSN: stas@zend.com

Am 21.08.2008 um 18:08 schrieb Rasmus Lerdorf:

David Zülke wrote:

Am 21.08.2008 um 03:34 schrieb William A. Rowe, Jr.:

Stanislav Malyshev wrote:

Hi!
Are there any objections to incorporating bugfix for #43941 (fix
for
how json handles invalid UTF-8 sequences) into 5.2? I had some
requests about it, right now it's only in 5.3+.

Is there the alternative of substituting an unmappable character
FFFD in
place of the invalid sequence? This a a reasonable alternative
behavior
for some less stringent cases.

(Yes, the fix is better than the status quo, but just taking this
a step
further).

I agree, that would be quite reasonable and also more consistent with
how UTF-8 works in other apps (browsers etc).

Well, using browsers as the benchmark here is a bad idea. IE is
absolutely braindead about dealing with illegal UTF-8 chars. It
will accept just about any sequence of bytes as a valid UTF-8 char
which causes all sorts of problems.

I was talking about the common representation of an invalid sequence.
That's the question mark sign you usually see in a browser when the
encoding is incorrect.

According to the Unicode standard, U+FFFD is supposed to be used as
the replacement character instead of simply stripping invalid ones:

Replacement Character. A character used as a substitute for an
uninterpretable character from another encoding. The Unicode
Standard uses U+FFFD replacement character for this function.
says http://unicode.org/glossary/#replacement_character

Rendering software which cannot process a Unicode character
appropriately most often display it as only an open rectangle, or
the Unicode “replacement character” (U+FFFD, �), to indicate
the position of the unrecognized character.

says http://en.wikipedia.org/wiki/Unicode#Standardized_subsets

Also see http://www.fileformat.info/info/unicode/char/fffd/index.htm

As always, I consider sticking to specs good practice, so doing it in
the above case would be wise :)

Hope that helps,

David

Am 21.08.2008 um 18:50 schrieb Rasmus Lerdorf:

David Zülke wrote:

Am 21.08.2008 um 18:41 schrieb Rasmus Lerdorf:

David Zülke wrote:

Am 21.08.2008 um 18:08 schrieb Rasmus Lerdorf:

David Zülke wrote:

Am 21.08.2008 um 03:34 schrieb William A. Rowe, Jr.:

Stanislav Malyshev wrote:

Hi!
Are there any objections to incorporating bugfix for #43941
(fix for
how json handles invalid UTF-8 sequences) into 5.2? I had some
requests about it, right now it's only in 5.3+.

Is there the alternative of substituting an unmappable character
FFFD in
place of the invalid sequence? This a a reasonable alternative
behavior
for some less stringent cases.

(Yes, the fix is better than the status quo, but just taking
this a
step
further).

I agree, that would be quite reasonable and also more
consistent with
how UTF-8 works in other apps (browsers etc).

Well, using browsers as the benchmark here is a bad idea. IE is
absolutely braindead about dealing with illegal UTF-8 chars. It
will
accept just about any sequence of bytes as a valid UTF-8 char
which
causes all sorts of problems.

I was talking about the common representation of an invalid
sequence.
That's the question mark sign you usually see in a browser when the
encoding is incorrect.

Yes, but it all comes down to how you do it. Say you have a 3 byte
sequence that starts with 0xE0 (E0 indicates the start of a 3-byte
utf-8 char) but the 3 bytes together don't actually make up a valid
utf-8 char. Id you substitute those 3 bytes with a ? or some other
character you have just created a nasty XSS vector for web apps.

You don't substitute it with "a ? or some other character", you
replace
it with U+FFFD (0xEF 0xBF 0xBD in UTF-8). I'd love to hear how that
causes an attack vector.

It doesn't matter what you replace it with. If the byte sequence is:

0xE0 " >

And you replace those bytes with some other byte in this sort of
context:

<input type=text name=foo value="0xE0"> <input type=text name=bar value="$data">

Now do your silly replacement:

<input type=text name=foo value="0xEF 0xBF 0xBD
<input type=text name=bar value="$data">

That now means that IE interprets the value attribute of the foo
element as: value="0xEF 0xBF 0xBD <input type=text name=bar value="
And now $data is suddenly outside the quoted value attribute! Oops!
Major XSS. Google Groups and Yahoo were both hit by this last year.

Interesting. I assume that was a weakness in the respective
implementation, right? Since

0xE0 " >

should never be regarded a valid sequence since neither " nor > are in
the range above 0x7F...

David

Am 21.08.2008 um 19:03 schrieb Rasmus Lerdorf:

David Zülke wrote:

Interesting. I assume that was a weakness in the respective
implementation, right? Since

0xE0 " >

should never be regarded a valid sequence since neither " nor > are
in
the range above 0x7F...

But that's what we are talking about. What to do with invalid
sequences. The E0 says that the following 2 bytes are part of the
UTF-8 character. So this is a 3-byte sequence. Together these 3
bytes are not valid, so Microsoft chose to replace those 3 with some
other character. And yes, Microsoft is notoriously bad at reading
specs, but I don't think it is completely clear what to do here, but
we do know that we shouldn't do that.

Well to me, the invalid part would be "0xE0" since it is incomplete
(0x7F and below are never part of multi-byte sequences, so they don't
count into the sequence here), so 0xE0 would be replaced by 0xEF 0xBF
0xBD, and then you don't have an XSS unless I'm mistaken :)

If Microsoft regards 0x7F and below as valid sequence members, then
that is unfortunate, but that shouldn't stop PHP from doing it
properly, as we all know better now, don't we :)

I mean what does the patch do in that case? Strip it all? Then it's
the same problem. Strip 0xE0 only? Then we could just as well insert U
+FFFD instead. No difference.

David