Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:59336
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain adamharvey.name designates 209.85.210.170 as permitted sender)
MIME-Version: 1.0
Sender: adam@adamharvey.name
In-Reply-To: <4F7A5684.4050102@lerdorf.com>
References: <7EBB3287C8464A93B2B264FDAD2B6756@charliesomerville.com> <4F7A5684.4050102@lerdorf.com>
Date: Tue, 3 Apr 2012 09:56:18 +0800
Message-ID: <CADajX4AFEveDroZV66WDuVSNmtYQqVfiAbvBoeXVp2QygOe1MA@mail.gmail.com>
To: Rasmus Lerdorf <rasmus@lerdorf.com>
Cc: Charlie Somerville <charlie@charliesomerville.com>, internals@lists.php.net
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Subject: Re: [PHP-DEV] json_encode() and non-UTF8 strings
From: aharvey@php.net (Adam Harvey)

On 3 April 2012 09:46, Rasmus Lerdorf <rasmus@lerdorf.com> wrote:
> On 04/02/2012 06:35 PM, Charlie Somerville wrote:
>> I've created a pull request (https://github.com/php/php-src/pull/33) tha=
t changes json_encode to fall back to ASCII for strings that are not valid =
UTF-8.
>>
>> I ran into an issue in a production application involving PayPal IPN cal=
lbacks (which are sent encoded as windows-1252) and json_encode(). If there=
 was an accented character present in the data, json_encode() would fail to=
 encode the string and serialize it as 'null'.
>>
>> I've modified the behaviour of the underlying json_escape_string() imple=
mentation to attempt to encode strings anyway while still producing a warni=
ng.
>
> JSON with non-Unicode strings is no longer JSON. The spec is explicit
> that all strings must be Unicode. The default encoding is UTF-8, but it
> could be UTF-16/32 as well.
>
> See http://www.ietf.org/rfc/rfc4627.txt

Agreed. I have a patch lying around for bug #61537 that I need to
finish up that actually goes the other way and changes the default
behaviour of json_encode() to match the documentation and return false
if a string is invalid UTF-8, rather than just nulling that string.

-1 from me on this. It's a regression from the current behaviour, IMO.

Adam