# [RFC] [DISCUSSION] Locale-independent float to string cast - Thread: https://externals.io/message/109261 - Original archive: https://news-web.php.net/php.internals/109261 - Started by kocsismate90@gmail.com on 2020-03-24T09:40:18+00:00 - Messages: 15 - RFC: Locale-independent float to string cast (Implemented (PHP 8.0)) — https://externals.io/rfc/locale_independent_float_to_string.md - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · RFCs at https://externals.io/rfc.md · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #109261 — kocsismate90@gmail.com — 2020-03-24T09:40:18+00:00 URL: https://externals.io/message/109261#109261 Hi Internals, Together with George, I'd like to propose an RFC for a long-standing problem PHP has: casting floats to string depends on the locale settings. As this behaviour is nonsense, and because it can cause quite serious problems, we would like to get rid of locale-dependence in PHP 8. Please find our RFC at https://wiki.php.net/rfc/locale_independent_float_to_string. Regards, George and Máté --- ## #109262 — Lynn — 2020-03-24T10:03:17+00:00 URL: https://externals.io/message/109261#109262 · In reply to: #109261 Hi, This is a great RFC! Just one minor thing. > Outputting floats as strings in locales which change the decimal separator will have a slightly different output. In our opinion, the backward compatibility break won't be serious in practice In my opinion, this will be huge. I can't trace back where the thousands of possible conversions are used in legacy software. There's a variety of custom format functions, sprintf implementations, string casts, and number_format usages. Would it be possible to trigger a warning or deprecation when using a locale that would have a different result from what the new result will be? If I'd use a locale that results in `3.5`, I don't need a warning. If I use a locale that results in `3,5`, I would like to see a warning of sorts so I can fix this before untested legacy code will seriously break data exports. Regards, Lynn > Hi Internals, > > Together with George, I'd like to propose an RFC for a long-standing > problem PHP has: > casting floats to string depends on the locale settings. As this behaviour > is nonsense, and > because it can cause quite serious problems, we would like to get rid of > locale-dependence > in PHP 8. > > Please find our RFC at > https://wiki.php.net/rfc/locale_independent_float_to_string. > > Regards, > George and Máté --- ## #109265 — Nikita Popov — 2020-03-24T10:14:52+00:00 URL: https://externals.io/message/109261#109265 · In reply to: #109261 > Hi Internals, > > Together with George, I'd like to propose an RFC for a long-standing > problem PHP has: > casting floats to string depends on the locale settings. As this behaviour > is nonsense, and > because it can cause quite serious problems, we would like to get rid of > locale-dependence > in PHP 8. > > Please find our RFC at > https://wiki.php.net/rfc/locale_independent_float_to_string. > I'm obviously in favor of this proposal. Only really comment I have is on `printf()`: You're right that we have %f and %F to toggle locale-sensitivity, but the %e, %E, %g, %G formats are always locale-sensitive. It might make sense to introduce locale-insensitive variants of those, especially considering that %G is considered the "standard" floating point format. Internally we support %H for that, so we could expose that... (Alternatively, locale-sensitivity might be removed for e/E/g/G.) Regards, Nikita --- ## #109267 — Christoph M. Becker — 2020-03-24T10:53:11+00:00 URL: https://externals.io/message/109261#109267 · In reply to: #109262 > > […] > > In my opinion, this will be huge. I can't trace back where the thousands of > possible conversions are used in legacy software. There's a variety of > custom format functions, sprintf implementations, string casts, and > number_format usages. Would it be possible to trigger a warning or > deprecation when using a locale that would have a different result from > what the new result will be? If I'd use a locale that results in `3.5`, I > don't need a warning. If I use a locale that results in `3,5`, I would like > to see a warning of sorts so I can fix this before untested legacy code > will seriously break data exports. It would certainly be *possible* to raise a warning/notice if a (non SCCP optimized) float to string conversion would have different results. This would have a (small) performance impact though, so I'd would rather avoid enforcing it (i.e. maybe temporarily introduce an INI directive for this purpose). -- Christoph M. Becker --- ## #109278 — G. P. B. — 2020-03-24T18:41:28+00:00 URL: https://externals.io/message/109261#109278 · In reply to: #109262 > Hi, > > This is a great RFC! Just one minor thing. > > > […] > separator will have a slightly different output. In our opinion, the > backward compatibility break won't be serious in practice > > In my opinion, this will be huge. I can't trace back where the thousands of > possible conversions are used in legacy software. There's a variety of > custom format functions, sprintf implementations, string casts, and > number_format usages. Would it be possible to trigger a warning or > deprecation when using a locale that would have a different result from > what the new result will be? If I'd use a locale that results in `3.5`, I > don't need a warning. If I use a locale that results in `3,5`, I would like > to see a warning of sorts so I can fix this before untested legacy code > will seriously break data exports. > > Regards, > Lynn > We are not saying that this won't be prevalent,especially in legacy code, however, IMHO, this is not a serious issue as it is only a matter how one character is displayed to end users. As said by Christoph we did not consider a deprecation warning due to the performance impact this would lead to as float to string conversions are a common operation. The idea of a temporary INI setting which warn about these conversions is an interesting idea, but I personally would rather not introduce one. Moreover, with a temporary INI setting how long would it last, one minor version, one major version? Something in between the two? > I'm obviously in favor of this proposal. > > Only really comment I have is on `printf()`: You're right that we have %f and > %F to toggle locale-sensitivity, but the %e, %E, %g, %G formats are always > locale-sensitive. It might make sense to introduce locale-insensitive > variants of those, especially considering that %G is considered the > "standard" floating point format. Internally we support %H for that, so we > could expose that... (Alternatively, locale-sensitivity might be removed > for e/E/g/G.) > > Regards, > Nikita *sight* I was not aware of this issue ... Personally, I would prefer that %e and %E are not locale aware as they are meant to represent a standard notation. This would then mean that %g is locale aware as %f is and %G is not because %E and %F are not locale aware. But I'm open to suggestion considering this edge case. Best regards George P. Banyard --- ## #109279 — Lynn — 2020-03-24T18:59:45+00:00 URL: https://externals.io/message/109261#109279 · In reply to: #109278 > > > > […] > > We are not saying that this won't be prevalent,especially in legacy code, > however, IMHO, this is not a serious issue as it is only a matter how one > character is displayed to end users. > It's so much more than a character that's being displayed. There are automated data exports with prices, that will be automatically consumed on the other side. These kind of changes can seriously break if they were expected to be localized currency formats. We're talking massive amounts of data here. I've seen price formats go wrong often and seen them be fixed even up to a week ago. It's hard to track all usages in a 20 year old code base. As much as I love that this can be fixed, I rather not see it fixed at all if there's no upgrade path available. Regards, Lynn --- ## #109281 — kocsismate90@gmail.com — 2020-03-24T23:14:21+00:00 URL: https://externals.io/message/109261#109281 · In reply to: #109279 Hi Lynn, > There are automated data exports with prices, that will be automatically consumed on the other side. I agree that these cases can go horribly wrong. However, my reasoning is the following: - if a piece of code currently relies on locale-independence (e.g. automated data exports) then this change wouldn't cause any breakage since a workaround has already been in place there (e.g. the programmers use `var_export()` instead of casting) - if a piece of code relies on the locale-dependent string representation of floats then there will be a BC break, sure, however I believe that code isn't very sensitive to the change in the vast majority of the cases since that data is for presentation purposes only. Or do you have other locale-dependent use-cases in mind? I am sure there are some but I think the number of the situations where the change is problematic is less than what it first seems. Regards, Máté --- ## #109282 — Lynn — 2020-03-25T08:25:49+00:00 URL: https://externals.io/message/109261#109282 · In reply to: #109281 I'm very sorry, I pressed the reply instead of reply all button, I hope this fixes it! I agree that these cases can go horribly wrong. However, my reasoning is > the following: > - if a piece of code currently relies on locale-independence (e.g. > automated data exports) then this > change wouldn't cause any breakage since a workaround has already been in > place there (e.g. the > programmers use `var_export()` instead of casting) > - if a piece of code relies on the locale-dependent string representation > of floats then there will be > a BC break, sure, however I believe that code isn't very sensitive to the > change in the vast majority of the > cases since that data is for presentation purposes only. > I know it's for presentation purpose only, sadly it's not used for just presentation code. It's being consumed and parsed by automated imports that expect a format different than `3.5`, because that's how it organically evolved. Even when it was meant only for presentation, the consumer expected the `3,5` format and will keep expecting this until we notify them of a change (and then it will probably still take weeks before this is fixed on their side). A lot of code that I've encountered is stuff like: ``` $csv .= $product->getPrice() . ';'; // older code I've seen come by $product = get_product($$var5); $csv .= $product['price'] . ';'; ``` As much as I'd like to see this fixed and always give a `3.5`, sadly that's not the case if you use a different locale. Going through thousands of files to fix this, is not going to be an easy task, especially not as this is often old enough to not be usable by static code analyzers. > Or do you have other locale-dependent use-cases in mind? I am sure there > are some but I think the number of the situations where the change is > problematic is less than what it first seems. > No, this is the only issue I personally see with it. However, the impact could be severe enough to not be able to upgrade to a new PHP version in the foreseeable future, thus I would like to see an upgrade path, one that tells you where it would've gone wrong, instead of turning the behavior on or off. If I can gather the logs and thus find usages, I can pro-actively start fixing where this would go wrong. I'll take a performance hit over a breakage. Performance penalty would only have to apply if the locale is set to something we know will change the outcome. Maybe it turns out this is a small issue, maybe it's a big issue... I can't tell because it's either upgrade & break, or upgrade & work in the proposed scenario. Some breakage may take weeks to find out because scripts run periodically. I fully understand that you don't see this as a big issue, and I really wish I could say the same. I can't vote, so I can't change the outcome, please consider my use-case when moving forward, thanks! Regards, Lynn --- ## #109328 — kocsismate90@gmail.com — 2020-03-26T09:54:21+00:00 URL: https://externals.io/message/109261#109328 · In reply to: #109282 Hi Lynn, Thank you very much for the input! Then we could add a secondary vote to the RFC where an upgrade path was decided upon. The first such path is the one proposed by Christoph: Introduce a temporary ini setting with which a "debug" mode of float to string casting could be enabled. That is, when it's turned on and a casting is affected by locale settings then a warning is emitted. This setting would be removed in a (near) future version of PHP. Although PHP 8.1 would be an unusual candidate to do so, but it *might* be possible if we found a good way to warn people from day 0 that it's just a temporary flag? Maybe by an immediate deprecation? Is there any other idea for a sensible upgrade path? Cheers: Máté --- ## #109668 — G. P. B. — 2020-04-15T20:19:32+00:00 URL: https://externals.io/message/109261#109668 · In reply to: #109328 Hello all, We've updated the implementation [1] and RFC [2] to provide a temporary INI setting which will emit a warning every time a locale-aware float to string conversation would have been done in PHP 7. It's current name debug_local_sensitive_float_casts is also up to debate and will be voted on a secondary vote. How long this INI setting should remain in PHP is also up to discussion. One remaining question is if we should change the %e and %E formats for the printf family of functions to be locale independent or not and/or if we should expose the %H format which is similar to %G but with %E being locale independent. Moreover, we would like to open voting next week, so your input on this is very much appreciated so that we can amend this before going to voting. Best regards George P. Banyard [1] https://github.com/php/php-src/pull/5224 [2] https://wiki.php.net/rfc/locale_independent_float_to_string --- ## #109671 — Andrea Faulds — 2020-04-15T20:54:21+00:00 URL: https://externals.io/message/109261#109671 · In reply to: #109261 Hi, Máté Kocsis wrote: > Hi Internals, > > Together with George, I'd like to propose an RFC for a long-standing > problem PHP has: > casting floats to string depends on the locale settings. As this behaviour > is nonsense, and > because it can cause quite serious problems, we would like to get rid of > locale-dependence > in PHP 8. > > Please find our RFC at > https://wiki.php.net/rfc/locale_independent_float_to_string. > > Regards, > George and Máté > Just a little thing, saying “certain locales (notably the German, French, and Spanish ones)” perhaps understates the issue. Most non-English-speaking countries do not use . as the decimal separator! https://en.wikipedia.org/wiki/Decimal_separator#Arabic_numerals Anyway, thank you for this RFC, I think fixing this is long, long overdue. I expect there could be a serious BC impact for certain PHP users, but I don't think there is a good long-term solution for this short of forcing such code to be updated. As a potential improvement, I would suggest to try to be closer to comprehensive in documenting which functions' behaviour changes or does not. For example, this RFC doesn't currently say what happens to `settype()` and `print_r()`. Thanks, Andrea --- ## #109762 — kocsismate90@gmail.com — 2020-04-21T20:00:00+00:00 URL: https://externals.io/message/109261#109762 Hi Andrea, Thank you for the suggestions! The wording of the RFC has been changed, and now it includes a link to the list of countries which use . as a decimal separator. Also, I updated the list of functions that change behaviour, even though there are only two newly added items: `implode()` and an XML-RPC related function. We tried to come up with an exhaustive list of functions which are going to be affected, however due to the weak typing semantics it is nearly impossible to make an accurate exhaustive list, but we hope the list provided covers the most common cases. That said, we'd like to open the vote on Thursday, unless serious concerns arise in the meanwhile. Cheers, Máté --- ## #109769 — Guilliam Xavier — 2020-04-22T08:53:09+00:00 URL: https://externals.io/message/109261#109769 · In reply to: #109762 > > That said, we'd like to open the vote on Thursday, unless serious concerns > arise > in the meanwhile. > > Cheers, > Máté Hi, thanks for the work, Shouldn't the RFC mention the open question of printf formats %e/%E and %g/%G (and possible %H)? -- Guilliam Xavier --- ## #109772 — Nikita Popov — 2020-04-22T13:34:41+00:00 URL: https://externals.io/message/109261#109772 · In reply to: #109769 On Wed, Apr 22, 2020 at 10:53 AM Guilliam Xavier wrote: > On Tue, Apr 21, 2020 at 10:01 PM Máté Kocsis > wrote: > > […] > concerns > > […] > > Hi, thanks for the work, > > Shouldn't the RFC mention the open question of printf formats %e/%E > and %g/%G (and possible %H)? > I've started a separate thread to discuss related improvements to `printf()`. I believe the `printf()` improvements are useful independent of the outcome of this RFC, and don't need to block this RFC from going forward. Regards, Nikita --- ## #109773 — kocsismate90@gmail.com — 2020-04-22T13:38:34+00:00 URL: https://externals.io/message/109261#109773 · In reply to: #109769 Hi Guilliam, Thank you for the reminder, we certainly forgot to resolve that issue. After a quick investigation, it turned out that the %e and %E specifiers are already locale-insensitive, so we can leave them as-is. Furthermore, the definition of the %G specifier according to the documentation is "Like the g specifier but uses E and F", which is not true, since %G is always a locale-dependent format, while %F is not. That's why we think the way forward is to: - fix the documentation independently of this RFC - expose the %h and %H format specifiers which are "like the %g and %G specifiers, but really use %e/%E and %f/%F respectively". Nikita is currently working on adding support for these, so it's also not a dependency of the RFC. That's why the vote is still scheduled for tomorrow. Thanks, Máté (P.S: I've just seen that Nikita already wrote a similar answer)