json_decode/encode should return full precision values by default

9 years ago by Yasuo Ohgaki — view source

unread

Hi all,

I had to work with Google Map API and need to handle values precisely.
Fortunately, it seems values are IEEE double, but I get rounded float
values by default.

For example,
<?php
$json = '
{
"results" : [
{
"elevation" : 1608.637939453125,
"location" : {
"lat" : 39.73915360,
"lng" : -104.98470340
},
"resolution" : 4.771975994110107
},
{
"elevation" : -50.78903579711914,
"location" : {
"lat" : 36.4555560,
"lng" : -116.8666670
},
"resolution" : 19.08790397644043
}
],
"status" : "OK"
}
';

var_dump(json_decode($json));
?>

object(stdClass)#5 (2) {
["results"]=>
array(2) {
[0]=>
object(stdClass)#1 (3) {
["elevation"]=>
float(1608.6379394531)
["location"]=>
object(stdClass)#2 (2) {
["lat"]=>
float(39.7391536)
["lng"]=>
float(-104.9847034)
}
["resolution"]=>
float(4.7719759941101)
}
[1]=>
object(stdClass)#3 (3) {
["elevation"]=>
float(-50.789035797119)
["location"]=>
object(stdClass)#4 (2) {
["lat"]=>
float(36.455556)
["lng"]=>
float(-116.866667)
}
["resolution"]=>
float(19.08790397644)
}
}
["status"]=>
string(2) "OK"
}

json_decode()/json_encode() must be able to handle precise IEEE double
value by default.

serialize() is changed to use max precision. json_decode/encode should do
the same at least.

I think this change should be provided as bug fix.
Any comments?

--
Yasuo Ohgaki
yohgaki@ohgaki.net

9 years ago by Yasuo Ohgaki — view source

unread

Hi all,

Created bug report for this issue.
https://bugs.php.net/bug.php?id=70137

--
Yasuo Ohgaki
yohgaki@ohgaki.net

Hi all,

I had to work with Google Map API and need to handle values precisely.
Fortunately, it seems values are IEEE double, but I get rounded float
values by default.

For example,
<?php
$json = '
{
"results" : [
{
"elevation" : 1608.637939453125,
"location" : {
"lat" : 39.73915360,
"lng" : -104.98470340
},
"resolution" : 4.771975994110107
},
{
"elevation" : -50.78903579711914,
"location" : {
"lat" : 36.4555560,
"lng" : -116.8666670
},
"resolution" : 19.08790397644043
}
],
"status" : "OK"
}
';

var_dump(json_decode($json));
?>

object(stdClass)#5 (2) {
["results"]=>
array(2) {
[0]=>
object(stdClass)#1 (3) {
["elevation"]=>
float(1608.6379394531)
["location"]=>
object(stdClass)#2 (2) {
["lat"]=>
float(39.7391536)
["lng"]=>
float(-104.9847034)
}
["resolution"]=>
float(4.7719759941101)
}
[1]=>
object(stdClass)#3 (3) {
["elevation"]=>
float(-50.789035797119)
["location"]=>
object(stdClass)#4 (2) {
["lat"]=>
float(36.455556)
["lng"]=>
float(-116.866667)
}
["resolution"]=>
float(19.08790397644)
}
}
["status"]=>
string(2) "OK"
}

json_decode()/json_encode() must be able to handle precise IEEE double
value by default.

serialize() is changed to use max precision. json_decode/encode should do
the same at least.

I think this change should be provided as bug fix.
Any comments?

--
Yasuo Ohgaki
yohgaki@ohgaki.net

9 years ago by Jakub Zelenka — view source

unread

Hi,

Hi all,

I had to work with Google Map API and need to handle values precisely.
Fortunately, it seems values are IEEE double, but I get rounded float
values by default.

For example,
<?php
$json = '
{
"results" : [
{
"elevation" : 1608.637939453125,
"location" : {
"lat" : 39.73915360,
"lng" : -104.98470340
},
"resolution" : 4.771975994110107
},
{
"elevation" : -50.78903579711914,
"location" : {
"lat" : 36.4555560,
"lng" : -116.8666670
},
"resolution" : 19.08790397644043
}
],
"status" : "OK"
}
';

var_dump(json_decode($json));
?>

object(stdClass)#5 (2) {
["results"]=>
array(2) {
[0]=>
object(stdClass)#1 (3) {
["elevation"]=>
float(1608.6379394531)
["location"]=>
object(stdClass)#2 (2) {
["lat"]=>
float(39.7391536)
["lng"]=>
float(-104.9847034)
}
["resolution"]=>
float(4.7719759941101)
}
[1]=>
object(stdClass)#3 (3) {
["elevation"]=>
float(-50.789035797119)
["location"]=>
object(stdClass)#4 (2) {
["lat"]=>
float(36.455556)
["lng"]=>
float(-116.866667)
}
["resolution"]=>
float(19.08790397644)
}
}
["status"]=>
string(2) "OK"
}

json_decode()/json_encode() must be able to handle precise IEEE double
value by default.

serialize() is changed to use max precision. json_decode/encode should do
the same at least.

I think this change should be provided as bug fix.
Any comments?

I don't think that this is a bug. Your example is also completely unrelated
to json because the place when the value is rounded is var_dump where it's
based on ini precision. You would get the same values with json_encode but
it's only because it uses the same ini ( precision ). Basically it has
nothing to do with json_decode.

I see that that the bug report you created (
https://bugs.php.net/bug.php?id=70137 ) is actually about using
serialize.precision instead of precision in json_encode . I agree that
using 'precision' ini is not the best solution. However I don't think that
start suddenly using serialize.precision is a good idea. First of all it's
a big BC break that will be quite difficult to find (There is no way this
should go to the point release because this is changing the generated
json). Also the json encode usage is much wider than serialization. There
might be use cases when you don't care about maximal precision and you
prefer to save some space. Imagine that you are transferring lots of float
numbers. Then it will result in increasing of the transfer size. It would
be great if the new ini was used when it was introduced but I'm afraid that
changing that now is not a good idea

I think that due to BC break, this shouldn't be changed. If you really want
a bigger precision, you can do ini_set('precision', 30) before json_encode
and then set it back. It means that this not a bug because you can still
change it if you want. That means that we can't change before PHP 8 or
whatever comes after 7... :)

Cheers

Jakub

9 years ago by Yasuo Ohgaki — view source

unread

Hi Jakub,

I don't think that this is a bug. Your example is also completely
unrelated to json because the place when the value is rounded is var_dump
where it's based on ini precision. You would get the same values with
json_encode but it's only because it uses the same ini ( precision ).
Basically it has nothing to do with json_decode.

OK. Better example.

[yohgaki@dev PHP-master]$ ./php-bin
<?php
$j = '{ "v": 0.1234567890123456789 }';
var_dump(json_encode(json_decode($j)));
ini_set('precision', 20);
var_dump(json_encode(json_decode($j)));
?>

string(22) "{"v":0.12345678901235}"
string(28) "{"v":0.12345678901234567737}"

I see that that the bug report you created (

https://bugs.php.net/bug.php?id=70137 ) is actually about using
serialize.precision instead of precision in json_encode . I agree that
using 'precision' ini is not the best solution. However I don't think that
start suddenly using serialize.precision is a good idea. First of all it's
a big BC break that will be quite difficult to find (There is no way this
should go to the point release because this is changing the generated
json). Also the json encode usage is much wider than serialization. There
might be use cases when you don't care about maximal precision and you
prefer to save some space. Imagine that you are transferring lots of float
numbers. Then it will result in increasing of the transfer size. It would
be great if the new ini was used when it was introduced but I'm afraid that
changing that now is not a good idea

I think that due to BC break, this shouldn't be changed. If you really
want a bigger precision, you can do ini_set('precision', 30) before
json_encode and then set it back. It means that this not a bug because you
can still change it if you want. That means that we can't change before PHP
8 or whatever comes after 7... :)

Same argument could be done for serialize, but it's changed to keep
more precise value. Manual recommend json_decode/encode for data
serialization also.

Warning
Do not pass untrusted user input to unserialize(). Unserialization can
result in code being loaded and executed due to object instantiation and
autoloading, and a malicious user may be able to exploit this. Use a safe,
standard data interchange format such as JSON (via json_decode() and
json_encode()) if you need to pass serialized data to the user.
http://jp2.php.net/manual/en/function.unserialize.php

It does not make sense having different serialization for float.
Losing effective precision is design bug. IMHO.

Currently users cannot handle even Google Map JSON data correctly by
default.
Fixing this is just a matter of using serialize.precision. If users need
larger setting
than 17 for whatever reason, they may do ini_set('serialize.precision', 30)
and the
rest of codes are unaffected.

BTW, larger precision setting does not mean precise, but current behavior
definitely lose effective values. Those who don't care for precise data
handling
wouldn't care for more precise data handling, do they?

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

9 years ago by Jakub Zelenka — view source

unread

Hi Yasuo,

Hi Jakub,

I don't think that this is a bug. Your example is also completely
unrelated to json because the place when the value is rounded is var_dump
where it's based on ini precision. You would get the same values with
json_encode but it's only because it uses the same ini ( precision ).
Basically it has nothing to do with json_decode.

OK. Better example.

[yohgaki@dev PHP-master]$ ./php-bin
<?php
$j = '{ "v": 0.1234567890123456789 }';
var_dump(json_encode(json_decode($j)));
ini_set('precision', 20);
var_dump(json_encode(json_decode($j)));
?>

string(22) "{"v":0.12345678901235}"
string(28) "{"v":0.12345678901234567737}"

Yeah you will loose precision. That's just the fact that double can't
contain such a big precision. This is of course the same for serialize and
unserialize. See this example:

http://3v4l.org/nb3mE

The only difference is that that it is using different ini but you can
still change it if you want. The precision lost is the same if the values
are the same though.

I see that that the bug report you created (

https://bugs.php.net/bug.php?id=70137 ) is actually about using
serialize.precision instead of precision in json_encode . I agree that
using 'precision' ini is not the best solution. However I don't think that
start suddenly using serialize.precision is a good idea. First of all it's
a big BC break that will be quite difficult to find (There is no way this
should go to the point release because this is changing the generated
json). Also the json encode usage is much wider than serialization. There
might be use cases when you don't care about maximal precision and you
prefer to save some space. Imagine that you are transferring lots of float
numbers. Then it will result in increasing of the transfer size. It would
be great if the new ini was used when it was introduced but I'm afraid that
changing that now is not a good idea

I think that due to BC break, this shouldn't be changed. If you really
want a bigger precision, you can do ini_set('precision', 30) before
json_encode and then set it back. It means that this not a bug because you
can still change it if you want. That means that we can't change before PHP
8 or whatever comes after 7... :)

Same argument could be done for serialize, but it's changed to keep
more precise value. Manual recommend json_decode/encode for data
serialization also.

Warning
Do not pass untrusted user input to unserialize(). Unserialization can
result in code being loaded and executed due to object instantiation and
autoloading, and a malicious user may be able to exploit this. Use a safe,
standard data interchange format such as JSON (via json_decode() and
json_encode()) if you need to pass serialized data to the user.
http://jp2.php.net/manual/en/function.unserialize.php

It does not make sense having different serialization for float.
Losing effective precision is design bug. IMHO.

As I said I think that using preicision constant was unfortunate but start
using serialize_precision is not a win IMHO. Using serialize and
json_encoding are not the same thing. In any case this is not a bug, it's
just an unfortunate decision and we should not definitely change it in a
bug fixing release. It's a BC break. Maybe if we do an RFC, it could be
considered for 7.1 but voters will need to agree that. I'd imagine 3 voting
choices:

keep it as it is (use precision)
use serialize_precision
introduce new json_precision

Currently users cannot handle even Google Map JSON data correctly by
default.
Fixing this is just a matter of using serialize.precision. If users need
larger setting
than 17 for whatever reason, they may do ini_set('serialize.precision',

and the
rest of codes are unaffected.

you can still do ini_set('pricision', 17) resp. ini_set('pricision', 30).

BTW, larger precision setting does not mean precise, but current behavior
definitely lose effective values. Those who don't care for precise data
handling
wouldn't care for more precise data handling, do they?

yeah but mind that the side effect of rising precision is an increasing
size of the json_encoded data which is not what everyone is happy with...

Cheers

Jakub

9 years ago by Anthony Ferrara — view source

unread

Yasuo,

Hi Jakub,

I don't think that this is a bug. Your example is also completely
unrelated to json because the place when the value is rounded is var_dump
where it's based on ini precision. You would get the same values with
json_encode but it's only because it uses the same ini ( precision ).
Basically it has nothing to do with json_decode.

OK. Better example.

[yohgaki@dev PHP-master]$ ./php-bin
<?php
$j = '{ "v": 0.1234567890123456789 }';
var_dump(json_encode(json_decode($j)));
ini_set('precision', 20);
var_dump(json_encode(json_decode($j)));
?>

string(22) "{"v":0.12345678901235}"
string(28) "{"v":0.12345678901234567737}"

I think you missed the point. Parsing isn't dependent upon precision
setting. Only dumping: http://3v4l.org/48VSt

$j = '{ "v": 0.1234567890123456789 }';
$d1 = json_decode($j);

ini_set('precision', 20);

$d2 = json_decode($j);

var_dump($d1, $d2);

//object(stdClass)#1 (1) { ["v"]=> float(0.12345678901234567737) }
//object(stdClass)#2 (1) { ["v"]=> float(0.12345678901234567737) }

Meaning that it's parsed correctly.

There is no bug here.

Anthony

9 years ago by Yasuo Ohgaki — view source

unread

Hi Anthony,

On Tue, Jul 28, 2015 at 6:54 AM, Anthony Ferrara ircmaxell@gmail.com
wrote:

Hi Jakub,

I don't think that this is a bug. Your example is also completely
unrelated to json because the place when the value is rounded is
var_dump
where it's based on ini precision. You would get the same values with
json_encode but it's only because it uses the same ini ( precision ).
Basically it has nothing to do with json_decode.

OK. Better example.

[yohgaki@dev PHP-master]$ ./php-bin
<?php
$j = '{ "v": 0.1234567890123456789 }';
var_dump(json_encode(json_decode($j)));
ini_set('precision', 20);
var_dump(json_encode(json_decode($j)));
?>

string(22) "{"v":0.12345678901235}"
string(28) "{"v":0.12345678901234567737}"

I think you missed the point. Parsing isn't dependent upon precision
setting. Only dumping: http://3v4l.org/48VSt

$j = '{ "v": 0.1234567890123456789 }';
$d1 = json_decode($j);

ini_set('precision', 20);

$d2 = json_decode($j);

var_dump($d1, $d2);

//object(stdClass)#1 (1) { ["v"]=> float(0.12345678901234567737) }
//object(stdClass)#2 (1) { ["v"]=> float(0.12345678901234567737) }

Meaning that it's parsed correctly.

There is no bug here.

I understands PHP uses pure IEEE data and arithmetic and
I disagree that this is not a bug.

What I'm pointing it out is "simple JSON operation truncates
valid/precise value".
IEEE double can store 17 digit fraction part precisely, but PHP truncates
it.

<?php
$j = '{ "v": 0.1234567890123456789 }';
var_dump(json_encode(json_decode($j)));
ini_set('precision', 20);
var_dump(json_encode(json_decode($j)));
?>

string(22) "{"v":0.12345678901235}"
string(28) "{"v":0.12345678901234567737}"

The same problem in serialize/unserialize was fixed as a bug.
Encode/decode should be as precise as possible by default. IMO.

What's the point of truncation and having broken value?

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

9 years ago by Yasuo Ohgaki — view source

unread

On Tue, Jul 28, 2015 at 6:54 AM, Anthony Ferrara ircmaxell@gmail.com
wrote:

Hi Jakub,

I don't think that this is a bug. Your example is also completely
unrelated to json because the place when the value is rounded is
var_dump
where it's based on ini precision. You would get the same values with
json_encode but it's only because it uses the same ini ( precision ).
Basically it has nothing to do with json_decode.

OK. Better example.

[yohgaki@dev PHP-master]$ ./php-bin
<?php
$j = '{ "v": 0.1234567890123456789 }';
var_dump(json_encode(json_decode($j)));
ini_set('precision', 20);
var_dump(json_encode(json_decode($j)));
?>

string(22) "{"v":0.12345678901235}"
string(28) "{"v":0.12345678901234567737}"

I think you missed the point. Parsing isn't dependent upon precision
setting. Only dumping: http://3v4l.org/48VSt

$j = '{ "v": 0.1234567890123456789 }';
$d1 = json_decode($j);

ini_set('precision', 20);

$d2 = json_decode($j);

var_dump($d1, $d2);

//object(stdClass)#1 (1) { ["v"]=> float(0.12345678901234567737) }
//object(stdClass)#2 (1) { ["v"]=> float(0.12345678901234567737) }

Meaning that it's parsed correctly.

There is no bug here.

I understands PHP uses pure IEEE data and arithmetic and
I disagree that this is not a bug.

What I'm pointing it out is "simple JSON operation truncates
valid/precise value".
IEEE double can store 17 digit fraction part precisely, but PHP truncates
it.

<?php
$j = '{ "v": 0.1234567890123456789 }';
var_dump(json_encode(json_decode($j)));
ini_set('precision', 20);
var_dump(json_encode(json_decode($j)));
?>

string(22) "{"v":0.12345678901235}"
string(28) "{"v":0.12345678901234567737}"

The same problem in serialize/unserialize was fixed as a bug.
Encode/decode should be as precise as possible by default. IMO.

What's the point of truncation and having broken value?

Get JSON data from Google maps and store the data using PHP, then
users lose last 2 digits of fraction part by default. The value is changed
and wrong. This is definitely a bug.

We can write

$old = ini_set('precision', 17);
json_encode($var);
ini_set('precision', $old);

everywhere to workaround this problem.

Question is "Is this the way it should be?".

--
Yasuo Ohgaki
yohgaki@ohgaki.net

9 years ago by Stanislav Malyshev — view source

unread

Hi!

We can write

$old = ini_set('precision', 17);
json_encode($var);
ini_set('precision', $old);

everywhere to workaround this problem.

If we're talking about encoding, that's what precision setting is - it
says how much we want to preserve in decimal encoding of float numbers.
Decoding, of course, should use whatever data available and not
truncate, but I understand that is already happening?

Question is "Is this the way it should be?".

I don't see why not. That's exactly what precision setting is for, as
far as I can see.

--
Stas Malyshev
smalyshev@gmail.com

9 years ago by Yasuo Ohgaki — view source

unread

Hi all,

On Tue, Jul 28, 2015 at 9:34 AM, Stanislav Malyshev smalyshev@gmail.com
wrote:

We can write

$old = ini_set('precision', 17);
json_encode($var);
ini_set('precision', $old);

everywhere to workaround this problem.

If we're talking about encoding, that's what precision setting is - it
says how much we want to preserve in decimal encoding of float numbers.
Decoding, of course, should use whatever data available and not
truncate, but I understand that is already happening?

Question is "Is this the way it should be?".

I don't see why not. That's exactly what precision setting is for, as
far as I can see.

JSON is var exporter/importer like serialize/var_export.

Both of serialize/var_export uses PG(serialize_precision), what's the point
of
lose / destroy original values while PHP could keep more precise values
than now?

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

9 years ago by Stanislav Malyshev — view source

unread

Hi!

JSON is var exporter/importer like serialize/var_export.

Both of serialize/var_export uses PG(serialize_precision), what's the
point of
lose / destroy original values while PHP could keep more precise values
than now?

So you're saying JSON should use serialize_precision setting instead of
precision setting? That makes sense, I think it would be a reasonable
change.

--
Stas Malyshev
smalyshev@gmail.com

9 years ago by Jakub Zelenka — view source

unread

Hi Yasuo

Get JSON data from Google maps and store the data using PHP, then
users lose last 2 digits of fraction part by default. The value is changed
and wrong. This is definitely a bug.

I don't really get why you use Google maps as an example. Do you actually
know what's the difference between two distances that differs after
rounding with precision 14?

I just tried it on
http://www.ig.utexas.edu/outreach/googleearth/latlong.html and set

Lat 1: 51.602097123457 ; Long 1: -0.120667
Lat 2: 51.602097123458 ; Long 2: -0.120667

The difference is 1.114e-10 km which is 0.0001114 millimetres. Are you
actually serious about that? :)

We can write

$old = ini_set('precision', 17);
json_encode($var);
ini_set('precision', $old);

You can set it once in your ini file if you need such precision.

everywhere to workaround this problem.

Question is "Is this the way it should be?".

I have already said that using precision ini wasn't the best idea. However
json_encode is not the same as serialize and we should not ever change its
output in a bug fixing release. Doing that could cause also other issues as
this is a BC break. Lets imagine that someone set low precision on purpose
just to limit precision and save some space when transferring data . If you
change it, then it's screwed up because it will use different ini. We don't
know what people do in their code and we should not break it. As I said
this is not a bug but we could consider changing that if the RFC proposing
such change passes.

Cheers

Jakub

9 years ago by Yasuo Ohgaki — view source

unread

Hi Jakub,

Get JSON data from Google maps and store the data using PHP, then
users lose last 2 digits of fraction part by default. The value is
changed
and wrong. This is definitely a bug.

I don't really get why you use Google maps as an example. Do you actually
know what's the difference between two distances that differs after
rounding with precision 14?

I just tried it on
http://www.ig.utexas.edu/outreach/googleearth/latlong.html and set

Lat 1: 51.602097123457 ; Long 1: -0.120667
Lat 2: 51.602097123458 ; Long 2: -0.120667

The difference is 1.114e-10 km which is 0.0001114 millimetres. Are you
actually serious about that? :)

It's error margin for GPS. How about scientific values?

I'm not comfortable with that values aren't processed/stored correctly (as
much as possible.
You know float can't be exactly precise) Even if GPS values may ignore some
fractions,
I'm very uncomfortable with ignoring/destroying original values while it
could be stored
exactly as it is.

In addition, it's inconsistent with other data exchange functions. i.e.
serialize/var_export.

We can write

$old = ini_set('precision', 17);
json_encode($var);
ini_set('precision', $old);

You can set it once in your ini file if you need such precision.

It's not a good idea. Most PHP codes depend on 14 precision and programmer
cannot assure other code works as expected.

everywhere to workaround this problem.

Question is "Is this the way it should be?".

I have already said that using precision ini wasn't the best idea. However
json_encode is not the same as serialize and we should not ever change its
output in a bug fixing release. Doing that could cause also other issues as
this is a BC break. Lets imagine that someone set low precision on purpose
just to limit precision and save some space

I've been fixed var_export precision issue as a bug.

[yohgaki@dev php-src]$ git show 3cf2682083fc1c8635b02c4c
commit 3cf2682083fc1c8635b02c4cf77bdf12c5e5da35
Merge: 5c89d5a 4c45e95
Author: Yasuo Ohgaki yohgaki@php.net
Date: Tue Oct 29 17:30:58 2013 +0900

Merge branch 'PHP-5.5'

* PHP-5.5:
  Fixed Bug 64760 `var_export()` does not use full precision for

floating-point numbers

Although I don't strongly insist merging fix to 5.6 branch, I think data
exchange function
that is not trying to be precise is bad thing.

when transferring data . If you change it, then it's screwed up because it
will use different ini. We don't know what people do in their code and we
should not break it. As I said this is not a bug but we could consider
changing that if the RFC proposing such change passes.

Strong json numeric validator may have problem with the change. However,
JSON RFC states

Numbers
This specification allows implementations to set limits on the range
and precision of numbers accepted. Since software that implements
IEEE 754-2008 binary64 (double precision) numbers [IEEE754] is
generally available and widely used, good interoperability can be
achieved by implementations that expect no more precision or range
than these provide, in the sense that implementations will
approximate JSON numbers within the expected precision. A JSON
number such as 1E400 or 3.141592653589793238462643383279 may indicate
potential interoperability problems, since it suggests that the
software that created it expects receiving software to have greater
capabilities for numeric magnitude and precision than is widely
available.

Note that when such software is used, numbers that are integers and
are in the range [-(253)+1, (253)-1] are interoperable in the
sense that implementations will agree exactly on their numeric
values.
https://tools.ietf.org/html/rfc7159

In order PHP to be more compatible, we should use IEEE 754 range or
our other data exchange standard, which is serialize_precision=17.

My position is

PHP7.0: We must use serialize_precision for JSON
PHP5.6: I strongly suggest to use serialize_precision for JSON

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

9 years ago by Jakub Zelenka — view source

unread

Hi Yasuo,

Hi Jakub,

On Mon, Jul 27, 2015 at 11:17 PM, Yasuo Ohgaki yohgaki@ohgaki.net
wrote:

Get JSON data from Google maps and store the data using PHP, then
users lose last 2 digits of fraction part by default. The value is
changed
and wrong. This is definitely a bug.

I don't really get why you use Google maps as an example. Do you actually
know what's the difference between two distances that differs after
rounding with precision 14?

I just tried it on
http://www.ig.utexas.edu/outreach/googleearth/latlong.html and set

Lat 1: 51.602097123457 ; Long 1: -0.120667
Lat 2: 51.602097123458 ; Long 2: -0.120667

The difference is 1.114e-10 km which is 0.0001114 millimetres. Are you
actually serious about that? :)

It's error margin for GPS. How about scientific values?

This is certainly not a margin error for GPS and definitely has no
influence on Google maps... :) Of course there are cases when you can need
slightly bigger precision. What I was trying to say is that you exaggerate
the problem.

I'm not comfortable with that values aren't processed/stored correctly (as
much as possible.
You know float can't be exactly precise) Even if GPS values may ignore
some fractions,
I'm very uncomfortable with ignoring/destroying original values while it
could be stored
exactly as it is.

Once again, you have got the option to change it if you don't like the
current default.

In addition, it's inconsistent with other data exchange functions. i.e.
serialize/var_export.

json_encode is not serialize nor var_export. It's for different things... I
have already said that you might care about output size and if you do, you
certainly don't want to be changed in the bug fixing release.

Question is "Is this the way it should be?".

I have already said that using precision ini wasn't the best idea.
However json_encode is not the same as serialize and we should not ever
change its output in a bug fixing release. Doing that could cause also
other issues as this is a BC break. Lets imagine that someone set low
precision on purpose just to limit precision and save some space

I've been fixed var_export precision issue as a bug.

[yohgaki@dev php-src]$ git show 3cf2682083fc1c8635b02c4c
commit 3cf2682083fc1c8635b02c4cf77bdf12c5e5da35
Merge: 5c89d5a 4c45e95
Author: Yasuo Ohgaki yohgaki@php.net
Date: Tue Oct 29 17:30:58 2013 +0900
Merge branch 'PHP-5.5'

* PHP-5.5:
  Fixed Bug 64760 `var_export()` does not use full precision for
floating-point numbers

Again it's not the same. json_encode is much more used the var_export IMHO
so the chance that you break someone's code is much bigger. Also it doesn't
mean that you did the right thing and it's not a precedence for such change
in json. I guess that there wasn't anyone who would disagree with you at
that time.

Although I don't strongly insist merging fix to 5.6 branch, I think data
exchange function
that is not trying to be precise is bad thing.

when transferring data . If you change it, then it's screwed up because
it will use different ini. We don't know what people do in their code and
we should not break it. As I said this is not a bug but we could consider
changing that if the RFC proposing such change passes.

Strong json numeric validator may have problem with the change. However,
JSON RFC states

Numbers
This specification allows implementations to set limits on the range
and precision of numbers accepted. Since software that implements
IEEE 754-2008 binary64 (double precision) numbers [IEEE754] is
generally available and widely used, good interoperability can be
achieved by implementations that expect no more precision or range
than these provide, in the sense that implementations will
approximate JSON numbers within the expected precision. A JSON
number such as 1E400 or 3.141592653589793238462643383279 may indicate
potential interoperability problems, since it suggests that the
software that created it expects receiving software to have greater
capabilities for numeric magnitude and precision than is widely
available.

Note that when such software is used, numbers that are integers and
are in the range [-(253)+1, (253)-1] are interoperable in the
sense that implementations will agree exactly on their numeric
values.
https://tools.ietf.org/html/rfc7159

In order PHP to be more compatible, we should use IEEE 754 range or
our other data exchange standard, which is serialize_precision=17.

We are still compatible. This is recommendation, not requirement. However I
have to agree that it will be probably good idea to change default. But
just only if agreed in RFC as this is not a bug! Personally I am against
changing that to serialize_precision. I'd much prefer introducing new ini
(e.g. json_precision) with the same default as serialize_precision. It
would give more flexibility and allow changing value just for json.

My position is

PHP7.0: We must use serialize_precision for JSON

PHP5.6: I strongly suggest to use serialize_precision for JSON

I'd prefer 7.1 as this is not something that would be so urgent.

As you see we don't agree here. It means that if you want to get it in,
please write RFC. You can add the version option as I did it for json
numeric to string RFC and you will see if others agree with you that this
is a bug.

Cheers

Jakub

9 years ago by Yasuo Ohgaki — view source

unread

Hi Jakub,

For me, JSON is one of a data exchange format just like
serialize/var_export.
Anyway, we are about to reach an agreement.

Question is "Is this the way it should be?".

I have already said that using precision ini wasn't the best idea.
However json_encode is not the same as serialize and we should not ever
change its output in a bug fixing release. Doing that could cause also
other issues as this is a BC break. Lets imagine that someone set low
precision on purpose just to limit precision and save some space

I've been fixed var_export precision issue as a bug.

[yohgaki@dev php-src]$ git show 3cf2682083fc1c8635b02c4c
commit 3cf2682083fc1c8635b02c4cf77bdf12c5e5da35
Merge: 5c89d5a 4c45e95
Author: Yasuo Ohgaki yohgaki@php.net
Date: Tue Oct 29 17:30:58 2013 +0900
Merge branch 'PHP-5.5'

* PHP-5.5:
  Fixed Bug 64760 `var_export()` does not use full precision for
floating-point numbers
Again it's not the same. json_encode is much more used the var_export IMHO
so the chance that you break someone's code is much bigger. Also it doesn't
mean that you did the right thing and it's not a precedence for such change
in json. I guess that there wasn't anyone who would disagree with you at
that time.

I agree that JSON is used more than var_export or even serialize.

I guess almost all users do not care much about preciseness of JSON numeric.
I was the one also until I had to deal with float values. Since almost
nobody cares
about preciseness, why not make it more precise?

I may ask general list see if there are users who mind this change.

Although I don't strongly insist merging fix to 5.6 branch, I think data
exchange function
that is not trying to be precise is bad thing.

when transferring data . If you change it, then it's screwed up because
it will use different ini. We don't know what people do in their code and
we should not break it. As I said this is not a bug but we could consider
changing that if the RFC proposing such change passes.

Strong json numeric validator may have problem with the change. However,
JSON RFC states

Numbers
This specification allows implementations to set limits on the range
and precision of numbers accepted. Since software that implements
IEEE 754-2008 binary64 (double precision) numbers [IEEE754] is
generally available and widely used, good interoperability can be
achieved by implementations that expect no more precision or range
than these provide, in the sense that implementations will
approximate JSON numbers within the expected precision. A JSON
number such as 1E400 or 3.141592653589793238462643383279 may indicate
potential interoperability problems, since it suggests that the
software that created it expects receiving software to have greater
capabilities for numeric magnitude and precision than is widely
available.

Note that when such software is used, numbers that are integers and
are in the range [-(253)+1, (253)-1] are interoperable in the
sense that implementations will agree exactly on their numeric
values.
https://tools.ietf.org/html/rfc7159

In order PHP to be more compatible, we should use IEEE 754 range or
our other data exchange standard, which is serialize_precision=17.

We are still compatible. This is recommendation, not requirement. However
I have to agree that it will be probably good idea to change default. But
just only if agreed in RFC as this is not a bug! Personally I am against
changing that to serialize_precision. I'd much prefer introducing new ini
(e.g. json_precision) with the same default as serialize_precision. It
would give more flexibility and allow changing value just for json.

I agree that it is implementation matter how "JSON number" is treated.
Since PHP is
general programming language, I think it's better to make it as generic as
possible
by default.

It's perfectly OK for me to have json_precision. I prefer to have it indeed!

My position is

PHP7.0: We must use serialize_precision for JSON

PHP5.6: I strongly suggest to use serialize_precision for JSON

I'd prefer 7.1 as this is not something that would be so urgent.

I guess your reason to postpone the change is "json_precision".
This could be done by 7.0 as the resolution requires a trivial change to
code.

I don't see reasons not to fix this issue for 5.6 if "serialize_precision"
is used.

As you see we don't agree here. It means that if you want to get it in,

please write RFC. You can add the version option as I did it for json
numeric to string RFC and you will see if others agree with you that this
is a bug.

No problem. I'll start RFC discussion shortly.

BTW, please don't care if this is a bug or not. We have lots of issues to be
resolved. Let's concentrate resolving issues.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

9 years ago by Nikita Popov — view source

unread

Hi Jakub,

For me, JSON is one of a data exchange format just like
serialize/var_export.
Anyway, we are about to reach an agreement.
Question is "Is this the way it should be?".

I have already said that using precision ini wasn't the best idea.
However json_encode is not the same as serialize and we should not ever
change its output in a bug fixing release. Doing that could cause also
other issues as this is a BC break. Lets imagine that someone set low
precision on purpose just to limit precision and save some space

I've been fixed var_export precision issue as a bug.

[yohgaki@dev php-src]$ git show 3cf2682083fc1c8635b02c4c
commit 3cf2682083fc1c8635b02c4cf77bdf12c5e5da35
Merge: 5c89d5a 4c45e95
Author: Yasuo Ohgaki yohgaki@php.net
Date: Tue Oct 29 17:30:58 2013 +0900
Merge branch 'PHP-5.5'

* PHP-5.5:
  Fixed Bug 64760 `var_export()` does not use full precision for
floating-point numbers
Again it's not the same. json_encode is much more used the var_export
IMHO
so the chance that you break someone's code is much bigger. Also it
doesn't
mean that you did the right thing and it's not a precedence for such
change
in json. I guess that there wasn't anyone who would disagree with you at
that time.
I agree that JSON is used more than var_export or even serialize.

I guess almost all users do not care much about preciseness of JSON
numeric.
I was the one also until I had to deal with float values. Since almost
nobody cares
about preciseness, why not make it more precise?

I may ask general list see if there are users who mind this change.

Although I don't strongly insist merging fix to 5.6 branch, I think data
exchange function
that is not trying to be precise is bad thing.

when transferring data . If you change it, then it's screwed up because
it will use different ini. We don't know what people do in their code
and
we should not break it. As I said this is not a bug but we could
consider
changing that if the RFC proposing such change passes.

Strong json numeric validator may have problem with the change. However,
JSON RFC states

Numbers
This specification allows implementations to set limits on the range
and precision of numbers accepted. Since software that implements
IEEE 754-2008 binary64 (double precision) numbers [IEEE754] is
generally available and widely used, good interoperability can be
achieved by implementations that expect no more precision or range
than these provide, in the sense that implementations will
approximate JSON numbers within the expected precision. A JSON
number such as 1E400 or 3.141592653589793238462643383279 may indicate
potential interoperability problems, since it suggests that the
software that created it expects receiving software to have greater
capabilities for numeric magnitude and precision than is widely
available.

Note that when such software is used, numbers that are integers and
are in the range [-(253)+1, (253)-1] are interoperable in the
sense that implementations will agree exactly on their numeric
values.
https://tools.ietf.org/html/rfc7159

In order PHP to be more compatible, we should use IEEE 754 range or
our other data exchange standard, which is serialize_precision=17.

We are still compatible. This is recommendation, not requirement. However
I have to agree that it will be probably good idea to change default. But
just only if agreed in RFC as this is not a bug! Personally I am against
changing that to serialize_precision. I'd much prefer introducing new ini
(e.g. json_precision) with the same default as serialize_precision. It
would give more flexibility and allow changing value just for json.

I agree that it is implementation matter how "JSON number" is treated.
Since PHP is
general programming language, I think it's better to make it as generic as
possible
by default.

It's perfectly OK for me to have json_precision. I prefer to have it
indeed!

My position is

PHP7.0: We must use serialize_precision for JSON

PHP5.6: I strongly suggest to use serialize_precision for JSON

I'd prefer 7.1 as this is not something that would be so urgent.

I guess your reason to postpone the change is "json_precision".
This could be done by 7.0 as the resolution requires a trivial change to
code.

I don't see reasons not to fix this issue for 5.6 if "serialize_precision"
is used.

As you see we don't agree here. It means that if you want to get it in,

please write RFC. You can add the version option as I did it for json
numeric to string RFC and you will see if others agree with you that this
is a bug.

No problem. I'll start RFC discussion shortly.

BTW, please don't care if this is a bug or not. We have lots of issues to
be
resolved. Let's concentrate resolving issues.

Instead of continuing to use serialize_precision, which will produce
unnecessarily long outputs for many values, why don't we just switch to
using the 0 mode of zend_dtoa, i.e. to return the shortest output that is
still accurate if interpreted in round-to-nearest. I think this is what
everybody else is using when they convert floating point numbers to
strings. I guess we may not be able to change normal floating point
printing to use this, but this seems like the best mode for anything using
serialize_precision now and everything that should be using it (like JSON,
and queries, etc).

Nikita

9 years ago by Yasuo Ohgaki — view source

unread

Hi Nikita,

Instead of continuing to use serialize_precision, which will produce
unnecessarily long outputs for many values, why don't we just switch to
using the 0 mode of zend_dtoa, i.e. to return the shortest output that is
still accurate if interpreted in round-to-nearest. I think this is what
everybody else is using when they convert floating point numbers to
strings. I guess we may not be able to change normal floating point
printing to use this, but this seems like the best mode for anything using
serialize_precision now and everything that should be using it (like JSON,
and queries, etc).

I prefer your proposal!
Your proposal is a lot better than now.
Anyone has opinion for this?

I'm writing the RFC and I would like to make this the first option. i.e.
serialize_precision=0 uses "zend_dtoa 0 mode" for all data exchange
functions (json/serialize/var_exrport. Anyone care about WDDX/XML_RPC?)

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

9 years ago by Yasuo Ohgaki — view source

unread

Hi all,

On Thu, Jul 30, 2015 at 1:13 AM, Nikita Popov nikita.ppv@gmail.com
wrote:

Instead of continuing to use serialize_precision, which will produce
unnecessarily long outputs for many values, why don't we just switch to
using the 0 mode of zend_dtoa, i.e. to return the shortest output that is
still accurate if interpreted in round-to-nearest. I think this is what
everybody else is using when they convert floating point numbers to
strings. I guess we may not be able to change normal floating point
printing to use this, but this seems like the best mode for anything using
serialize_precision now and everything that should be using it (like JSON,
and queries, etc).

I prefer your proposal!
Your proposal is a lot better than now.
Anyone has opinion for this?

I'm writing the RFC and I would like to make this the first option. i.e.
serialize_precision=0 uses "zend_dtoa 0 mode" for all data exchange
functions (json/serialize/var_exrport. Anyone care about WDDX/XML_RPC?)

I wrote draft RFC.

https://wiki.php.net/rfc/precise_float_value

Please comment. I would like to start RFC discussion shortly.
Thank you.

--
Yasuo Ohgaki
yohgaki@ohgaki.net

9 years ago by Ryan Pallas — view source

unread

Hi all,

On Thu, Jul 30, 2015 at 1:13 AM, Nikita Popov nikita.ppv@gmail.com
wrote:

Instead of continuing to use serialize_precision, which will produce
unnecessarily long outputs for many values, why don't we just switch to
using the 0 mode of zend_dtoa, i.e. to return the shortest output that
is
still accurate if interpreted in round-to-nearest. I think this is what
everybody else is using when they convert floating point numbers to
strings. I guess we may not be able to change normal floating point
printing to use this, but this seems like the best mode for anything
using
serialize_precision now and everything that should be using it (like
JSON,
and queries, etc).

I prefer your proposal!
Your proposal is a lot better than now.
Anyone has opinion for this?

I'm writing the RFC and I would like to make this the first option. i.e.
serialize_precision=0 uses "zend_dtoa 0 mode" for all data exchange
functions (json/serialize/var_exrport. Anyone care about WDDX/XML_RPC?)

I wrote draft RFC.

https://wiki.php.net/rfc/precise_float_value

Please comment. I would like to start RFC discussion shortly.
Thank you.

My thought would be why not -1 to make it obvious it has special meaning?
When I see precision=0 I tend to think that means my numbers will be
returned as ints no matter what (there is zero fractional precision)
whereas this takes a valid value for precision and gives it special
meaning. I fully support your end game, but giving a valid value a special
meaning I do not like.

--
Yasuo Ohgaki
yohgaki@ohgaki.net

9 years ago by Yasuo Ohgaki — view source

unread

Hi Ryan,

I wrote draft RFC.

https://wiki.php.net/rfc/precise_float_value

Please comment. I would like to start RFC discussion shortly.
Thank you.

My thought would be why not -1 to make it obvious it has special meaning?
When I see precision=0 I tend to think that means my numbers will be
returned as ints no matter what (there is zero fractional precision)
whereas this takes a valid value for precision and gives it special
meaning. I fully support your end game, but giving a valid value a special
meaning I do not like.

Thank you for your feedback. I'll update the RFC and use "-1" for it.
When I make the patch for this, I'll post new RFC discussion mail.
It will take a while. Comments are appreciated.

Regards.

--
Yasuo Ohgaki
yohgaki@ohgaki.net

9 years ago by Nikita Popov — view source

unread

Hi all,

On Thu, Jul 30, 2015 at 1:13 AM, Nikita Popov nikita.ppv@gmail.com
wrote:

Instead of continuing to use serialize_precision, which will produce
unnecessarily long outputs for many values, why don't we just switch to
using the 0 mode of zend_dtoa, i.e. to return the shortest output that is
still accurate if interpreted in round-to-nearest. I think this is what
everybody else is using when they convert floating point numbers to
strings. I guess we may not be able to change normal floating point
printing to use this, but this seems like the best mode for anything using
serialize_precision now and everything that should be using it (like JSON,
and queries, etc).

I prefer your proposal!
Your proposal is a lot better than now.
Anyone has opinion for this?

I'm writing the RFC and I would like to make this the first option. i.e.
serialize_precision=0 uses "zend_dtoa 0 mode" for all data exchange
functions (json/serialize/var_exrport. Anyone care about WDDX/XML_RPC?)

I wrote draft RFC.

https://wiki.php.net/rfc/precise_float_value

Please comment. I would like to start RFC discussion shortly.
Thank you.

Nice idea about using a special serialize_precision value for this. This
allows to keep BC for those that have tests for particular serialize output
or similar things.

I would suggest to default serialize_precision to -1 in PHP 7 -- if people
want the previous behavior they can still have it, but I think -1 is the
more reasonable default as it matches what one would naturally expect.

I don't see the need for having a separate setting for JSON. Having a dozen
different float precision settings will not help anyone.

Nikita

9 years ago by Yasuo Ohgaki — view source

unread

Hi Nikita,

Hi all,

On Thu, Jul 30, 2015 at 1:13 AM, Nikita Popov nikita.ppv@gmail.com
wrote:

Instead of continuing to use serialize_precision, which will produce
unnecessarily long outputs for many values, why don't we just switch to
using the 0 mode of zend_dtoa, i.e. to return the shortest output that is
still accurate if interpreted in round-to-nearest. I think this is what
everybody else is using when they convert floating point numbers to
strings. I guess we may not be able to change normal floating point
printing to use this, but this seems like the best mode for anything using
serialize_precision now and everything that should be using it (like JSON,
and queries, etc).

I prefer your proposal!
Your proposal is a lot better than now.
Anyone has opinion for this?

I'm writing the RFC and I would like to make this the first option. i.e.
serialize_precision=0 uses "zend_dtoa 0 mode" for all data exchange
functions (json/serialize/var_exrport. Anyone care about WDDX/XML_RPC?)

I wrote draft RFC.

https://wiki.php.net/rfc/precise_float_value

Please comment. I would like to start RFC discussion shortly.
Thank you.

Nice idea about using a special serialize_precision value for this. This
allows to keep BC for those that have tests for particular serialize output
or similar things.

I would suggest to default serialize_precision to -1 in PHP 7 -- if people
want the previous behavior they can still have it, but I think -1 is the
more reasonable default as it matches what one would naturally expect.

I don't see the need for having a separate setting for JSON. Having a
dozen different float precision settings will not help anyone.

Thank you for your feedback.
I agree it's better if PHP7 uses 0 mode by default. I'll update the RFC.
I have things to do on this weekend. I'll try to write patch this weekend
hopefully.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

9 years ago by Yasuo Ohgaki — view source

unread

Hi all,

On Thu, Jul 30, 2015 at 6:06 PM, Nikita Popov nikita.ppv@gmail.com
wrote:

Hi all,

On Thu, Jul 30, 2015 at 7:44 AM, Yasuo Ohgaki yohgaki@ohgaki.net
wrote:

On Thu, Jul 30, 2015 at 1:13 AM, Nikita Popov nikita.ppv@gmail.com
wrote:

Instead of continuing to use serialize_precision, which will produce
unnecessarily long outputs for many values, why don't we just switch to
using the 0 mode of zend_dtoa, i.e. to return the shortest output that is
still accurate if interpreted in round-to-nearest. I think this is what
everybody else is using when they convert floating point numbers to
strings. I guess we may not be able to change normal floating point
printing to use this, but this seems like the best mode for anything using
serialize_precision now and everything that should be using it (like JSON,
and queries, etc).

I prefer your proposal!
Your proposal is a lot better than now.
Anyone has opinion for this?

I'm writing the RFC and I would like to make this the first option. i.e.
serialize_precision=0 uses "zend_dtoa 0 mode" for all data exchange
functions (json/serialize/var_exrport. Anyone care about WDDX/XML_RPC?)

I wrote draft RFC.

https://wiki.php.net/rfc/precise_float_value

Please comment. I would like to start RFC discussion shortly.
Thank you.

Nice idea about using a special serialize_precision value for this. This
allows to keep BC for those that have tests for particular serialize output
or similar things.

I would suggest to default serialize_precision to -1 in PHP 7 -- if
people want the previous behavior they can still have it, but I think -1 is
the more reasonable default as it matches what one would naturally expect.

I don't see the need for having a separate setting for JSON. Having a
dozen different float precision settings will not help anyone.

Thank you for your feedback.
I agree it's better if PHP7 uses 0 mode by default. I'll update the RFC.
I have things to do on this weekend. I'll try to write patch this weekend
hopefully.

I sent work in progress PR for this and updated the RFC.

https://github.com/php/php-src/pull/1455
TODO: Add/modify tests. Add 0 mode support for PG(precision). Add
WDDX/XMLRPC PG(serialize_precision) support.

https://wiki.php.net/rfc/precise_float_value

Old behavior
MacBook-Pro:github-php-src yohgaki$ php -r
'$a=["v"=>123450000000000000000000000000.1234567890123];
var_dump(json_encode($a), serialize($a), var_export($a, true));';
string(16) "{"v":1.2345e+29}"
string(39) "a:1:{s:1:"v";d:1.2344999999999999E+29;}"
string(42) "array (
'v' => 1.2344999999999999E+29,
)"

New behavior
MacBook-Pro:github-php-src yohgaki$ ./php-bin -r
'$a=["v"=>123450000000000000000000000000.1234567890123];
var_dump(json_encode($a), serialize($a), var_export($a, true));';
string(16) "{"v":1.2345e+29}"
string(27) "a:1:{s:1:"v";d:1.2345E+29;}"
string(30) "array (
'v' => 1.2345E+29,
)"

Before I start working on PG(precision) 0 mode support, I would like to hear
comments for this change if any. The default (PG(precision)=14) will not be
changed
and there will be no BC. I think it's more consistent if both PG(precision)
and
PG(serialize_precision) support 0 mode.

Any comments for adding PG(precision) 0 mode support?

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

9 years ago by Yasuo Ohgaki — view source

unread

Hi all,

Sorry, I was a bit busy during August.

I sent work in progress PR for this and updated the RFC.

https://github.com/php/php-src/pull/1455
TODO: Add/modify tests. Add 0 mode support for PG(precision). Add
WDDX/XMLRPC PG(serialize_precision) support.

https://wiki.php.net/rfc/precise_float_value

The patch and wiki is updated.

It's better to minimize the change at this time.
I didn't look into WDDX/XMLRPC which uses EG(precision), if anyone
insist, I'll look into it.
I don't mind much if anyone suggests to drop PHP 5.6 part. I'll drop the
part before RFC discussion to focus PHP7.

It should only affects php_gcvt() and php.ini's precision setting.
i.e. var_dump/var_export/
json_encode/echo/print. Tests passes, but if anyone notice
misbehavior, please let me
know.

I would like to start RFC discussion in a few days.
Thank you.

--
Yasuo Ohgaki
yohgaki@ohgaki.net