# NULL Coercion Consistency - Thread: https://externals.io/message/117501 - Original archive: https://news-web.php.net/php.internals/117501 - Started by Craig Francis on 2022-04-08T17:34:52+00:00 - Messages: 90 - RFC: NULL Coercion Consistency (Withdrawn) — https://externals.io/rfc/null_coercion_consistency.md - Page 1 of 4: messages 1 to 25 (25 per page, add `?page=N` to the URL) - Next page: https://externals.io/message/117501.md?page=2 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · RFCs at https://externals.io/rfc.md · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #117501 — Craig Francis — 2022-04-08T17:34:52+00:00 URL: https://externals.io/message/117501#117501 Hi, I've written a new draft RFC to address the `NULL` coercion problems: https://wiki.php.net/rfc/null_coercion_consistency This is due to the result of the Allow `NULL` quiz: https://quiz.craigfrancis.co.uk/ 14 votes for Fatal Type Errors irrespective of `strict_types=1`; 13 votes for `NULL` coercion when not using `strict_types=1`; 8 votes to update some parameters to allow NULL; I appreciate some want to force strict type checking on everyone, but I want to make sure we have this properly documented, with names, and explanations. Breaking changes should be justified - if they aren't, they only make upgrading difficult and frustrating (bad for security). Craig --- ## #117502 — Mel Dafert — 2022-04-08T17:54:59+00:00 URL: https://externals.io/message/117501#117502 · In reply to: #117501 >Hi, > >I've written a new draft RFC to address the `NULL` coercion problems: > >https://wiki.php.net/rfc/null_coercion_consistency > >This is due to the result of the Allow `NULL` quiz: > >https://quiz.craigfrancis.co.uk/ > >14 votes for Fatal Type Errors irrespective of `strict_types=1`; >13 votes for `NULL` coercion when not using `strict_types=1`; >8 votes to update some parameters to allow NULL; > >I appreciate some want to force strict type checking on everyone, but I >want to make sure we have this properly documented, with names, and >explanations. > >Breaking changes should be justified - if they aren't, they only >make upgrading difficult and frustrating (bad for security). > >Craig Hello, While the RFC extensively documents the problem space, it's rather short on what exactly it proposes. In particular, does this propose changing user-defined functions under strict_types=0 to accept null for scalar types? Eg., this will be allowed (under strict_types=0): ``` function x(string $y, int $z) { ... } x(null, null); //no error, no warning ``` Regards, Mel --- ## #117503 — Craig Francis — 2022-04-08T18:07:37+00:00 URL: https://externals.io/message/117501#117503 · In reply to: #117502 > In particular, does this propose changing user-defined functions under > strict_types=0 to accept null for scalar types? > > Eg., this will be allowed (under strict_types=0): > ``` > function x(string $y, int $z) { > ... > } > x(null, null); //no error, no warning > ``` > tbh my focus has been on the problems that have come up with internal functions. With user defined functions, I think it's up for debate (still a draft), but I think those NULL's should be coerced to the specified type (as documented), where I don't think PHP should be doing type checking under strict_types=0. Craig --- ## #117505 — Craig Francis — 2022-04-09T09:29:42+00:00 URL: https://externals.io/message/117501#117505 · In reply to: #117503 > > > > […] > > With user defined functions, I think it's up for debate (still a draft), > but I think those NULL's should be coerced to the specified type (as > documented), where I don't think PHP should be doing type checking under > strict_types=0. > I've updated the RFC to note that user-defined functions don't cause a backwards compatibility issue; but I have added an example to highlight the coercion inconstancy: ```php function user_function(string $s, int $i, float $f, bool $b) { var_dump($s, $i, $f, $b); } user_function('1', '1', '1', '1'); // string(1) "1" / int(1) / float(1) / bool(true) user_function(2, 2, 2, 2); // string(1) "2" / int(2) / float(2) / bool(true) user_function(3.3, 3.3, 3.3, 3.3); // string(3) "3.3" / int(3) / float(3.3) / bool(true) user_function(false, false, false, false); // string(0) "" / int(0) / float(0) / bool(false) user_function(NULL, NULL, NULL, NULL); // Uncaught TypeError x4? ``` --- ## #117510 — Guilliam Xavier — 2022-04-11T12:05:41+00:00 URL: https://externals.io/message/117501#117510 · In reply to: #117505 Hi, > https://wiki.php.net/rfc/null_coercion_consistency Thanks for the draft. On Sat, Apr 9, 2022 at 11:30 AM Craig Francis wrote: > On Fri, 8 Apr 2022 at 19:07, Craig Francis > wrote: > > > […] > > > I've updated the RFC to note that user-defined functions don't cause a > backwards compatibility issue; but I have added an example to highlight > the coercion inconstancy: > You've updated the **Documentation** section (also: did you mean "inconsistency" rather than "inconstancy"?) but still not the **Proposal** (BTW all those sections between "Introduction" and "Proposal" would probably better be *sub*-sections of a section named "Problem" or "Current State" or something). And for the question: (currently) the RFC is named "NULL Coercion *Consistency*" and the Proposal says "Must keep the spirit of the original RFC, and *keep user-defined and internal functions consistent*.", which (to me) implies *not only* reverting the 8.1 deprecation on internal functions *but also* "changing user-defined functions under strict_types=0 to [coerce] null for scalar type[ declaration]s" indeed. In any case, that should be written clear in the RFC (either in "Proposal" or "Open Issues"). Regards, -- Guilliam Xavier --- ## #117514 — Craig Francis — 2022-04-11T15:11:38+00:00 URL: https://externals.io/message/117501#117514 · In reply to: #117510 On Mon, 11 Apr 2022 at 13:05, Guilliam Xavier wrote: > > […] > > You've updated the **Documentation** section (also: did you mean > "inconsistency" rather than "inconstancy"?) but still not the **Proposal** > (BTW all those sections between "Introduction" and "Proposal" would > probably better be *sub*-sections of a section named "Problem" or "Current > State" or something). > > And for the question: (currently) the RFC is named "NULL Coercion > *Consistency*" and the Proposal says "Must keep the spirit of the original > RFC, and *keep user-defined and internal functions consistent*.", which (to > me) implies *not only* reverting the 8.1 deprecation on internal functions > *but also* "changing user-defined functions under strict_types=0 to > [coerce] null for scalar type[ declaration]s" indeed. > In any case, that should be written clear in the RFC (either in "Proposal" > or "Open Issues"). > Thank you Guilliam, I've applied all of your suggestions (with a slight tweak to use `strict_types=1`, just because I find it easier to read). While I am open to for user-defined functions to keep the type check for `NULL` when not in `strict_types=1`, it does feel like a bug, and I think it would be better to be consistent (happy to discuss on or off list). Craig --- ## #117515 — G. P. B. — 2022-04-11T15:14:32+00:00 URL: https://externals.io/message/117501#117515 · In reply to: #117501 > Hi, > > I've written a new draft RFC to address the `NULL` coercion problems: > > https://wiki.php.net/rfc/null_coercion_consistency The statement that: > But the implementation caused a Type Error when coercing `NULL` for everyone > (even when not using *strict_types=1*), this seems more of an over-sight > is utterly wrong and was a conscious design choice based on the widely accepted view that nullable by default like Java is a mistake and defeats the purpose. Even the competing RFC from Zend et al [1] proposed this behaviour > A new set of coercion rules will apply to both user-land type hints and > internal type hints. The guiding principals behind these new rules are: > > 1. If the type of the value is an exact match to the type requested by > the hint - allow. > 2. If the value can be coerced to the type requested by the hint > without data loss and without creation of likely unintended data - allow. > 3. In all other cases - reject. > > And later to describe the internal function changes: > This RFC proposes to bring the rule-set described in the last section to > internal functions as well, through updates to the zend_parse_parameters() > function. > There was obviously the caveat about `NULL` being coerced for internals functions, but other than usage, another reasons stated: > However, since this requires substantial auditing of internal functions - > especially ones that have default values but don't explicitly declare > themselves as accepting NULLs - it's outside the scope of this RFC and > will be revisited for 7.1. Note that if the > https://wiki.php.net/rfc/nullable_types > is accepted it will further > reduce this discrepancy, by allowing user-land functions and internal > functions the same level of granularity in terms of accepting or rejecting > `NULL` values for function arguments. > This auditing was to a large extent performed in PHP 8.0 with the introduction of the stubs. You can disagree with some of it, but this again brings us back to amending functions to have nullable arguments, not a blanket change. I appreciate some want to force strict type checking on everyone, but I > want to make sure we have this properly documented, with names, and > explanations. > > Breaking changes should be justified - if they aren't, they only > make upgrading difficult and frustrating (bad for security). > The breaking change was justified and explained, and voted unanimously in favour with 46 votes. The burden of justification is now on your end. Moreover, the breaking change has also been announced with ample time to fix the issues until the next major version. As with the previous one, this RFC is getting a strong no from my end, changing the type system into a broken and inconsistent way, and reversing a unanimously voted RFC needs strong justification. Best, George Peter Banyard [1] https://wiki.php.net/rfc/coercive_sth --- ## #117517 — Craig Francis — 2022-04-11T18:22:16+00:00 URL: https://externals.io/message/117501#117517 · In reply to: #117515 > But the implementation caused a Type Error when coercing `NULL` for everyone > > […] > is utterly wrong and was a conscious design choice based on the widely > accepted view that nullable by default like Java is a mistake and defeats > the purpose. > Bit hash... anyway, I've replaced that paragraph (it wasn't clear enough), I just wanted to note how developers not using `strict_types=1` would see it as an over-sight that coercion works for string/int/float/bool but not null (despite what the documentation says), and how it's introduced a type check (that they didn't ask for). I assume you're using `strict_types=1`, and will be unaffected by this, so with all due respect, I'm not considering how you view or use NULL, nor do I care what Java does, I'm simply focused on how most developers use `NULL` in PHP. "2. If the value can be coerced to the type requested by the hint without > data loss and without creation of likely unintended data - allow" Yep, as noted under the Documentation heading, and copying from the manual, “PHP will coerce values of the wrong type into the expected scalar type declaration if possible”... which it can, and does; for example, coercion to a string "null is always converted to an empty string". https://www.php.net/manual/en/language.types.declarations.php#language.types.declarations.strict https://www.php.net/manual/en/language.types.string.php [...] but this again brings us back to amending functions to have nullable > arguments, not a blanket change. > As noted in the Introduction, second paragraph, unfortunately this won't work (especially considering the ~335 parameters affected by this)... just taking `htmlspecialchars()` as the most simple example, I asked a few developers who use `strict_types=1`, and they did not want `$string` to be a nullable argument, because `NULL` being passed to this argument suggests there's a bug elsewhere in their code. This view was also reflected in the quiz (the names are links to view their answers): https://quiz.craigfrancis.co.uk/ The breaking change was justified and explained, and voted unanimously in > favour with 46 votes. > The burden of justification is now on your end. > My whole RFC is the justification, from the lack of discussion of the impact (where I will note that I am in favour of its intention of consistency), the roughly 85% of scripts that do not use `strict_types=1`, the almost certainly less than 33% developers who use static analysis, how tools do not and will not fix this problem, how this goes against the documentation, and all of the examples. Keep in mind, if you're using `strict_types=1`, great, you're not affected (I'm not either)... however, I work with quite a few developers, and it's causing them a lot of problems, and their current approach is to either disable the deprecation warnings, or to stay with 8.0, simply because they do not have the time to make the changes... I will note that one team has been trying to update their code base, it's about 6 months later, and they still keep adding `strval()` to silence these deprecation warnings (yeah, I know, not good, but it's the easy solution). changing the type system into a broken and inconsistent way, and reversing > a unanimously voted RFC needs strong justification. > I really appreciate your time to read my draft RFC, and I know there is no way I can change your mind, but can you explain how would this be "broken and inconsistent"? Just take `NULL` to String coercion as an example, the documentation clearly and simply says “null is always converted to an empty string”, this needs to be updated to end with "... except when being passed to a function, in which case `NULL` will result in a Type Error" (I think that's "broken and inconsistent"). Craig --- ## #117518 — Mark Baker — 2022-04-11T18:56:56+00:00 URL: https://externals.io/message/117501#117518 · In reply to: #117517 > Keep in mind, if you're using `strict_types=1`, great, you're not affected > (I'm not either)... however, I work with quite a few developers, and it's > causing them a lot of problems, and their current approach is to either > disable the deprecation warnings, or to stay with 8.0, simply because they > do not have the time to make the changes... I will note that one team has > been trying to update their code base, it's about 6 months later, and they > still keep adding `strval()` to silence these deprecation warnings (yeah, I > know, not good, but it's the easy solution). This doesn't only apply to end user developers, but also to library and toolchain developers who need to maintain code covering a range of PHP versions, and who also need to make these trade-offs. It's those maintainers that have to deal with the impact of this type of change, whether needing to apply changes to their libraries to "suppress" the deprecation notice; or dealing with end-user developers raising the same issues time and again about it because they don't want to see deprecation notices appearing in their logs. Even though this was only a deprecation, most end-user developers don't recognise that difference from logged errors when using libraries, so the impact is felt most heavily by library and toolchain maintainers. This 8.1 deprecation was the one that created the most work, and the most stress for me; and even with an average of 85% code coverage with tests, I'm still not certain that I've resolved every instance using the dirty `$x ?? ''` approach when calling PHP core functions. In many ways, readying code for the 8.1 release because of this deprecation was more painful and time consuming than for the 8.0 release. With all the new deprecations being added for 8.2, I'm looking forward to it even less. That's also why I wish there was a better risk assessment made for every deprecation RFC; becase all too often there's little or none; and why I get frustrated whenever a question is raised about an RFC, and the person raising these concerns is too often shouted down for raising "a storm in a teacup". -- Mark Baker _________ |. \ \-3 |_J_/ PHP | || | __ | || |m| |m| I LOVE PHP --- ## #117519 — Andreas Leathley — 2022-04-11T19:08:42+00:00 URL: https://externals.io/message/117501#117519 · In reply to: #117517 > Just take `NULL` to String coercion as an example, the documentation > clearly > and simply says “null is always converted to an empty string”, this needs > to be updated to end with "... except when being passed to a function, in > which case `NULL` will result in a Type Error" (I think that's "broken and > inconsistent"). You are taking parts of the documentation out of context, and omitting the start of the whole "Converting to string" section: "A value can be converted to a string using the (string) cast or the `strval()` function. String conversion is automatically done in the scope of an expression where a string is needed. This happens when using the echo or print functions, or when a variable is compared to a string. The sections on Types and Type Juggling will make the following clearer." In the same section it is also defined how resources and arrays get converted to strings. Following your argument, you should be able to pass arrays and resources to a string parameter because there is a clear definition in the documentation of what then happens. I unfortunately really don't like the direction of your RFC, because I see null as a real type and because it increases the divide between strict_types and not using strict_types. I have never used strict_types - in modern PHP code I find it unnecessary, if you use parameter, property and return types. I am glad PHP has decreased the difference between using strict_types or not, so your direction of trying to increase the difference seems like the wrong way to go. Ideally I would rather want to see a future where strict_types can be removed/reconciled. --- ## #117521 — Craig Francis — 2022-04-13T13:36:42+00:00 URL: https://externals.io/message/117501#117521 · In reply to: #117519 > You are taking parts of the documentation out of context, and omitting > the start of the whole "Converting to string" section: > > "A value can be converted to a string using the (string) cast or the > `strval()` function. String conversion is automatically done in the scope > of an expression where a string is needed. This happens when using the > echo or print functions, or when a variable is compared to a string. The > sections on Types and Type Juggling will make the following clearer." > I'm sorry, I've read this several times now, and I'm not sure what this adds. My RFC simply quotes the paragraphs that explain how null is coerced (other than the small abbreviation for booleans, those paragraphs are included in their entirety), I don't think it needs any more context than that, it's not like I'm mis-representing how coercion works (and is used) in PHP. I could expand my first quote to include "... For example, a function that is given an int for a parameter that expects a string will get a variable of type string", but I don't think it's that useful. > In the same section it is also defined how resources and arrays get > converted to strings. Following your argument, you should be able to > pass arrays and resources to a string parameter because there is a clear > definition in the documentation of what then happens. > No, my RFC is about the problems this change has caused, how it is making it difficult to upgrade. I'm only referencing the documentation to confirm how `NULL` has always been coerced. I unfortunately really don't like the direction of your RFC, because I > see null as a real type and because it increases the divide between > strict_types and not using strict_types. I have never used strict_types > - in modern PHP code I find it unnecessary, if you use parameter, > property and return types. I am glad PHP has decreased the difference > between using strict_types or not, so your direction of trying to > increase the difference seems like the wrong way to go. Ideally I would > rather want to see a future where strict_types can be removed/reconciled. You can see `NULL` however you like, but most developers do not share that view. `NULL` has been passed to these functions, since, well, forever; and changing code to manually convert `NULL` to the relevant type is incredibly time consuming, and of questionable value (e.g. when developers simply add `strval()` everywhere). Either way, I'm not trying to "increase the difference" (this is how it has always worked). If you want type checking, good, it can be really useful, and I recommend it with static analysis... but it's enabled with `strict_types=1`. Craig --- ## #117522 — Craig Francis — 2022-04-13T14:00:51+00:00 URL: https://externals.io/message/117501#117522 · In reply to: #117518 > This doesn't only apply to end user developers, but also to library and > toolchain developers who need to maintain code covering a range of PHP > versions > Yep, and thank you for commenting. Library authors are the ones receiving pressure at the moment, and while users of the library sometimes help (e.g. pull requests), it's a fairly thankless and time consuming task, with little/no value for scripts not using `strict_types=1` (you're not the only one simply adding `?? ''` everywhere). Just one thing to add, this is from people who are complaining about a deprecation notice (which can and is being ignored), now think of the typical non-library code when it becomes a Fatal Error. Craig --- ## #117523 — G. P. B. — 2022-04-13T14:15:14+00:00 URL: https://externals.io/message/117501#117523 · In reply to: #117517 On Mon, 11 Apr 2022 at 19:22, Craig Francis wrote: > > > > […] > > > Bit hash... anyway, I've replaced that paragraph (it wasn't clear enough), > I just wanted to note how developers not using `strict_types=1` would see > it as an over-sight that coercion works for string/int/float/bool but not > null (despite what the documentation says), and how it's introduced a type > check (that they didn't ask for). > > I assume you're using `strict_types=1`, and will be unaffected by this, so > with all due respect, I'm not considering how you view or use NULL, nor do > I care what Java does, I'm simply focused on how most developers use `NULL` > in PHP. > If I indeed exclusively used strict_types I wouldn't care, but I don't because IMHO strict_types were a mistake and are harmful and induce some self inflicted problems. I've spent a large amount of time making coercive typing mode more sensible and aligning the behaviour as close to reasonably possible with strict_types so that the possibility of dropping strict_types all together wouldn't be outrageous. I've written about this elsewhere and on this list already but main points are located here: https://github.com/Girgias/unify-typing-modes-rfc Userland scalar types, however they would have been introduced, did not include coercion from `NULL` for *very* good reasons. Wanting to change this behaviour needs more justification than a paragraph in docs, as the docs should reflect the reality of the implementation. The second aspect is the general consensus that internals and userland should behave identically. Which is why the deprecation is here in the first place, to align internal behaviour with the accepted *better* design choice of userland. Your RFC is going against these two pillars. Moreover, the fact that we are deprecating this and not just changing the behaviour from one version to another is that we do recognize PHP developers use `NULL` in this way, and we are giving them advance notice of the change. Also a deprecation notice should, IMHO, never be promoted to an exception. This is maybe something we need to get better at teaching end users so that they don't pressure library maintainers (or themselves) to fix all of them when they still have a couple of years. Be that either on the php.net docs or elsewhere, to teach end users to either not do this, or at least exclude the vendor directory from their error handler. However, and I cannot stress this enough, we cannot just stop using this tool available to us to steer the language into being better. Because then the other two choices are doing hard breaks with no notice in a major version, or not breaking anything. Both things which are unreasonable to expect. There are behaviours of PHP that will probably never be "fixable", but this should not prevent us from improving the aspects we can. And if we only focused on how developers use PHP we would still have register globals, magic quotes, etc. Clearly it seems you won't listen to why these changes were made and want to reverse course, therefore good luck convincing enough people to accept such an RFC. I think I've said all there is to say and won't interact further with any such proposals. Regards, George P. Banyard --- ## #117524 — Andreas Leathley — 2022-04-13T19:07:59+00:00 URL: https://externals.io/message/117501#117524 · In reply to: #117521 > > > You are taking parts of the documentation out of context, and omitting > the start of the whole "Converting to string" section: > > "A value can be converted to a string using the (string) cast or the > `strval()` function. String conversion is automatically done in the > scope > of an expression where a string is needed. This happens when using the > echo or print functions, or when a variable is compared to a > string. The > sections on Types and Type Juggling will make the following clearer." > > > > I'm sorry, I've read this several times now, and I'm not sure what > this adds. My RFC simply quotes the paragraphs that explain how null > is coerced (other than the small abbreviation for booleans, those > paragraphs are included in their entirety), I don't think it needs any > more context than that, it's not like I'm mis-representing how > coercion works (and is used) in PHP. > Mentioning the documentation as a reason to be "consistent" (which comes up again and again in your arguments with this RFC) just seems like a bogus reason to me. It is nitpicking about specific sentences in the documentation without refering to the surrounding context. It would be nicer to argue according to real technical arguments instead of arguments about sentences taken out of context in the documentation. > You can see `NULL` however you like, but most developers do not share > that view. `NULL` has been passed to these functions, since, well, > forever; and changing code to manually convert `NULL` to the relevant > type is incredibly time consuming, and of questionable value (e.g. > when developers simply add `strval()` everywhere). "Most developers do not share that view". I find statements such as these as a tall order - did you interview the majority of developers around the world? Are you suggesting you are talking as a representative of a large population of PHP developers? How do you think you got such a role? I would prefer some humility and not assume you are an elected leader of an underrepresented group of developers and even knowing the intention behind their code. "NULL has been passed to these functions, since, well, forever" - this also goes for wrong values being passed to functions since forever leading to security issues and bugs. That is the whole point of developing a language: Reducing the surface for bugs, improving parts of it where a lot of bugs have happened historically and not making the same mistakes as other languages (and learning from the good parts of other languages). More people writing code in a certain way does not make that code better or safer. --- ## #117525 — Craig Francis — 2022-04-13T19:43:00+00:00 URL: https://externals.io/message/117501#117525 · In reply to: #117523 https://wiki.php.net/rfc/null_coercion_consistency > I've spent a large amount of time making coercive typing mode more > sensible and aligning the behaviour as close to reasonably possible with > strict_types so that the possibility of dropping strict_types all together > wouldn't be outrageous. > I've written about this elsewhere and on this list already but main points > are located here: https://github.com/Girgias/unify-typing-modes-rfc > Thank you George, I do appreciate your work on this, I agree with pretty much everything you have written, and I really do want PHP to move in that direction. The *only* thing I disagree with is `NULL` coercion, because it has worked perfectly well for years, and is used so often. Userland scalar types, however they would have been introduced, did not > include coercion from `NULL` for *very* good reasons. > Wanting to change this behaviour needs more justification than a paragraph > in docs, as the docs should reflect the reality of the implementation. > Some people see `NULL` as an invalid value, which can sometimes be useful in a `strict_types=1` environment, but that's the only "good reason" I can find... it's why I changed my mind about making some parameters nullable, and took a different approach with this RFC. But most developers do not use `NULL` like that, and that's why it's passed so often to the ~335 function parameters I've noted (most of them are on the list because they realistically receive user values, which can be `NULL` in most frameworks). The second aspect is the general consensus that internals and userland > should behave identically. > Yep, they really should... it's why I said we "Must keep the spirit of the original RFC, and keep user-defined and internal functions consistent". Moreover, the fact that we are deprecating this and not just changing the > behaviour from one version to another is that we do recognize PHP > developers use `NULL` in this way, and we are giving them advance notice of > the change. > Also a deprecation notice should, IMHO, never be promoted to an exception. > Unfortunately deprecation notices are being ignored (as noted in my RFC, with examples on how); and there is the intention to use Type Error Exceptions in 9.0 (the thing I'm trying to avoid). I'm approaching this from a security point of view - the last thing I want to do in ~5 years is auditing code/systems where they have stuck with 8.x because it's too hard to upgrade. There are behaviours of PHP that will probably never be "fixable", but this > should not prevent us from improving the aspects we can. > And if we only focused on how developers use PHP we would still have > register globals, magic quotes, etc. > I want the language to be better as well, and I was *very* glad to see the back of Register Globals and Magic Quotes (I know they were a pain to remove, but it was well justified, as they caused so many problems). Clearly it seems you won't listen to why these changes were made and want > to reverse course, therefore good luck convincing enough people to accept > such an RFC. > I have been listening/reading (it's why I've taken about 2 hours to read what you have written, and reply). I have also been discussing this with different people (on and off list)... unlike the original RFC discussion, where it was only Craig Duncan that mentioned this issue: https://externals.io/message/112327#112531 And it's because I've been listening to people that I changed my mind, and created this new RFC (as noted above). I think I've said all there is to say and won't interact further with any > such proposals. > That's a shame, as I'd like to discuss solutions to this problem you would be happy with. Craig --- ## #117526 — Robert Landers — 2022-04-14T07:31:34+00:00 URL: https://externals.io/message/117501#117526 · In reply to: #117525 > > https://wiki.php.net/rfc/null_coercion_consistency > > > > > […] > > > Thank you George, I do appreciate your work on this, I agree with pretty > much everything you have written, and I really do want PHP to move in that > direction. > > The *only* thing I disagree with is `NULL` coercion, because it has worked > perfectly well for years, and is used so often. > > > > Userland scalar types, however they would have been introduced, did not > > […] > > > Some people see `NULL` as an invalid value, which can sometimes be useful in > a `strict_types=1` environment, but that's the only "good reason" I can > find... it's why I changed my mind about making some parameters nullable, > and took a different approach with this RFC. > > But most developers do not use `NULL` like that, and that's why it's passed > so often to the ~335 function parameters I've noted (most of them are on > the list because they realistically receive user values, which can be `NULL` > in most frameworks). > > > > The second aspect is the general consensus that internals and userland > > […] > > > Yep, they really should... it's why I said we "Must keep the spirit of the > original RFC, and keep user-defined and internal functions consistent". > > > > Moreover, the fact that we are deprecating this and not just changing the > > […] > > > Unfortunately deprecation notices are being ignored (as noted in my RFC, > with examples on how); and there is the intention to use Type Error > Exceptions in 9.0 (the thing I'm trying to avoid). > > I'm approaching this from a security point of view - the last thing I want > to do in ~5 years is auditing code/systems where they have stuck with 8.x > because it's too hard to upgrade. > > > > There are behaviours of PHP that will probably never be "fixable", but this > > […] > > > I want the language to be better as well, and I was *very* glad to see the > back of Register Globals and Magic Quotes (I know they were a pain to > remove, but it was well justified, as they caused so many problems). > > > > Clearly it seems you won't listen to why these changes were made and want > > […] > > > I have been listening/reading (it's why I've taken about 2 hours to read > what you have written, and reply). I have also been discussing this with > different people (on and off list)... unlike the original RFC discussion, > where it was only Craig Duncan that mentioned this issue: > > https://externals.io/message/112327#112531 > > And it's because I've been listening to people that I changed my mind, and > created this new RFC (as noted above). > > > > I think I've said all there is to say and won't interact further with any > > […] > > > That's a shame, as I'd like to discuss solutions to this problem you would > be happy with. > > Craig I don't have a vote, so my opinion only goes so far. But here's my 2¢. > I see null as a real type This confuses me. I see "false" and "true" becoming their own types, so what is a boolean? The language currently has no way to express aliases of types, otherwise I'd say a boolean is an alias "true|false" but it doesn't. What happens if a function returns the "false" type and I pass it to a function that only allows "bool"? I expect it to work. So naturally, there is some coercion in there as well, and it would be great if null got the same treatment where old codebases benefit without taking away the advantage for newer codebases. I see this RFC as a balance in that regard, because real life is messy. I really hope it passes and good luck! Robert Landers Software Engineer Utrecht NL --- ## #117527 — Craig Francis — 2022-04-14T08:10:50+00:00 URL: https://externals.io/message/117501#117527 · In reply to: #117524 > Mentioning the documentation as a reason to be "consistent" (which comes > up again and again in your arguments with this RFC) just seems like a > bogus reason to me. It is nitpicking about specific sentences in the > documentation without refering to the surrounding context. It would be > nicer to argue according to real technical arguments instead of > arguments about sentences taken out of context in the documentation. > I don't, do I? I've checked multiple times, and I only use the documentation to note how `NULL` coercion works. I'm using the word "consistent" to be "consistent with scalar types"... as in, a string/int/float/bool can be coerced, but `NULL` coercion has now been deprecated in some cases, as shown with the examples. > "Most developers do not share that view". I find statements such as > these as a tall order - did you interview the majority of developers > around the world? Are you suggesting you are talking as a representative > of a large population of PHP developers? How do you think you got such a > role? My intro says "Roughly 85% scripts do not use strict_types=1", and "Roughly 33% of developers use static analysis" (source of numbers noted in my RFC)... while that does not guarantee anything, it's a fairly good indication that most developers do not care about types (in regards to coercion), or specifically, how you "see null as a real type". As an aside, I would like to include more stats (maybe WordPress install numbers?), but I couldn't think of any which were easy to source (ref reliability), or be that useful to the discussion. I would prefer some humility and not assume you are an elected > leader of an underrepresented group of developers and even knowing the > intention behind their code. > I'm simply describing the situation, and I have suggested two ways to avoid the upgrade problems (the suggestion to update some arguments to be nullable has been rejected by written feedback, and via my questionnaire). "NULL has been passed to these functions, since, well, forever" - this > also goes for wrong values being passed to functions since forever > leading to security issues and bugs. I say in the RFC that "Arrays, Resources, and Objects (without __toString()) cannot be coerced (for fairly obvious reasons)". For example `htmlspecialchars(array())` will, as of 8.0, throw a Type Error (and a warning before), that does represent a bug (it should not happen), and I'm very happy with that situation. But `NULL` coercion is different, it has worked well for years, and it is used a lot (see the examples in my RFC). Craig --- ## #117528 — Craig Francis — 2022-04-14T08:27:35+00:00 URL: https://externals.io/message/117501#117528 · In reply to: #117526 On Thu, 14 Apr 2022 at 08:31, Robert Landers wrote: > > […] > > This confuses me... Andreas is probably the best person to explain their view; oddly I see `NULL` as it's own type as well, because are there are times where it's useful to determine the difference between `NULL` and an Empty String; but when looking at examples like user values (e.g. GET/POST/COOKIE) being checked (e.g. `strlen()` or `preg_match()`), encoded (e.g. shown on a web page with `htmlspecialchars()`), or simply modified (e.g. `trim()`), I don't see why those possible `NULL` values should lead to a Fatal Type Error (unless you are talking about an environment that uses `NULL` as an invalid value, but I suspect that's quite rare). I see this RFC as a balance in that regard, because real life is > messy. I really hope it passes and good luck! > Thank you Robert, it's right the balance I'm trying to find. Craig --- ## #117529 — Andreas Leathley — 2022-04-14T09:01:06+00:00 URL: https://externals.io/message/117501#117529 · In reply to: #117527 > My intro says "Roughly 85% scripts do not use strict_types=1", and "Roughly > 33% of developers use static analysis" (source of numbers noted in my > RFC)... while that does not guarantee anything, it's a fairly good > indication that most developers do not care about types (in regards to > coercion), or specifically, how you "see null as a real type". > > As an aside, I would like to include more stats (maybe WordPress install > numbers?), but I couldn't think of any which were easy to source (ref > reliability), or be that useful to the discussion. > I have never used strict_types in any code I have ever written, and I care about types and type coercions. Yet I do not like the strict_types distinction and I am glad that I do not need to use it, and I think we are not that far away from even reconciling these two systems. I do not mind that the string "5" can be passed to an int parameter, while passing the string "hello" will throw a type exception no matter what mode you use. With some adjustments to certain coercions the advantage of strict_types would be even more neglible - as you can pass "hello" to a boolean parameter without problems currently, and I don't like that. Looking at usage numbers and assuming that supports your point seems like a big leap to me. You are assuming these developers are making a choice while it seems quite possible that a lot of code has been written without thinking about all the implications, which is where so many bugs are coming from. In the last months reviewing code I noticed quite a few developers are copy-pasting code like crazy, or using generated code from other tools - both of which are often not well reasoned about. So this has little to do with developers not caring about methods of type coercions and more about the language permitting such code, leading to many follow-up problems. Also, this "problem" only affects the internal functions, yet you want to change it for all functions. If you use a framework and pass a value to a method of that framework, it would already have created a type error with null and a non-nullable argument for quite some time. I think you are making this out to be a bigger problem than it is. Sure, in a codebase with lots of custom code where no incoming variable is checked if it even exists there will be problems, but even then it is easily fixable. In a codebase based on a common framework it is much less of a problem, as you will probably be using lots of framework components and not mainly internal functions, and there you already have the "limitation" of not being able to pass null to a non-nullable method. --- ## #117531 — Craig Francis — 2022-04-14T12:14:26+00:00 URL: https://externals.io/message/117501#117531 · In reply to: #117529 > I have never used strict_types in any code I have ever written, and I care > about types and type coercions. Yet I do not like the strict_types > distinction and I am glad that I do not need to use it, and I think we are > not that far away from even reconciling these two systems. I do not mind > that the string "5" can be passed to an int parameter, while passing the > string "hello" will throw a type exception no matter what mode you use. > With some adjustments to certain coercions the advantage of strict_types > would be even more neglible - as you can pass "hello" to a boolean > parameter without problems currently, and I don't like that. > Yep, I agree with everything you have said there, and it's what George seems to be working towards (which I also agree with): https://github.com/Girgias/unify-typing-modes-rfc > Looking at usage numbers and assuming that supports your point seems like > a big leap to me. You are assuming these developers are making a choice > while it seems quite possible that a lot of code has been written without > thinking about all the implications, Actually, that's exactly what I see happening - `NULL` is passed to these internal functions without any thought, and unlike arrays/resources/etc which cannot be sensibly coerced, `NULL` coercion has been fine, and has worked for (as long as I can tell) forever. Back to the most simple example I can think of: $search = filter_input(INPUT_GET, 'q'); // Or any of the framework examples. echo 'Results for ' . htmlspecialchars($search); It does not matter if the request explicitly set "/?q=" in the URL or not; `filter_input()` and frameworks show there is a difference by returning `NULL` or an Empty String... but most of the time, no thought is given to that difference, developers just expect it to work. which is where so many bugs are coming from. In the last months reviewing > code I noticed quite a few developers are copy-pasting code like crazy, or > using generated code from other tools - both of which are often not well > reasoned about. So this has little to do with developers not caring about > methods of type coercions and more about the language permitting such code, > leading to many follow-up problems. > Yep, and that's why I'm in favour of stopping the broken/weird coercions... it's just that `NULL` is different, and it's fine (as noted, there are some developers who use it part of their strict type checking, but that seems to be fairly unusual). > Also, this "problem" only affects the internal functions, yet you want to > change it for all functions. > My RFC is focused on the backwards compatibility problems for internal functions... user defined functions are only included for consistency reasons (keeping the spirit of the original RFC), because they probably should coerce NULL, like how they coerce integers to strings, etc. If you use a framework and pass a value to a method of that framework, it > would already have created a type > error with null and a non-nullable argument for quite some time. > Yep, and that has caused... I wouldn't say problems, but minor confusion? Craig --- ## #117537 — Rowan Tommins — 2022-04-16T11:17:17+00:00 URL: https://externals.io/message/117501#117537 · In reply to: #117501 >I've written a new draft RFC to address the `NULL` coercion problems: > >https://wiki.php.net/rfc/null_coercion_consistency I'm sympathetic to the general problem you're trying to solve, but I'm not convinced by the argument that this is about consistency, because user defined functions have been consistently rejecting null for non-nullable parameters since PHP 7.0, and even before that for arrays and objects. Consistency is a good argument for small changes that eliminate unusual edge cases, but I think far-reaching changes should be able to stand on their own merits - regardless of whether it's consistent, do we want the language to work this way? To me, the main defence of type coercion is that PHP operates in a world full of strings - URLs are strings, HTTP requests are strings, and a lot of API responses are strings - so making it easy to work with strings like '42' as numbers makes a lot of sense. It's not clear to me that this extends to null. I think a large part of the problem here is that null can mean many different things in different contexts - "unknown", "not provided", "invalid input", "default", "not applicable", etc. These differences are subtle, but lead to different expectations of behaviour: - Treat null as a specific case with its own meaning, distinct from any other valid value. This is what explicitly nullable parameters and union types allow. - Treat null the same as any other out of range value, and raise an error. This is what happens in user-defined functions in PHP, and in built-in functions expecting non-scalar arguments. Compare also out of range actions like division by zero. - Treat null as a special state that propagates through expressions, because any operation with an unknown input has an unknown output. This is the approach to null taken by SQL, and by IEEE floating point with NaN. It is also the basis of the ?-> operator, and of things like Optional.map in Swift. - Treat null as a generic default value which can be filled in implicitly based on requirements. This is the interpretation currently taken by internal functions for scalar arguments, and what you are proposing to make standard. It is also, as you point out, the way PHP treats null in some other contexts, such as many operators. Interestingly, one of your examples mentions filter_input, which takes the "propagate" approach, and htmlspecialchars, which doesn't. It would often be more useful to retain the information that a value is null than to have it silently converted to an empty string as a side-effect of some other operation. Perhaps it would be useful to have some function-call equivalent of the ?-> operator. I'm not sure what this would look like for normal function calls, but it would be easy to add if we had a pipe operator, e.g.: If this was equivalent to htmlspecialchars($foo) $foo |> htmlspecialchars(...) Then this could be equivalent to ($foo === null ? null : htmlspecialchars($foo)) $foo ?|> htmlspecialchars(...) I'm not set against this RFC, but I'm not quite convinced by the case it makes, and think there may still be other options to explore. Regards, -- Rowan Tommins [IMSoP] --- ## #117542 — Craig Francis — 2022-04-19T11:34:58+00:00 URL: https://externals.io/message/117501#117542 · In reply to: #117537 > On 8 April 2022 18:34:52 BST, Craig Francis > wrote: > > […] > > > I'm sympathetic to the general problem you're trying to solve, but I'm not > convinced by the argument that this is about consistency, because user > defined functions have been consistently rejecting null for non-nullable > parameters since PHP 7.0, and even before that for arrays and objects. Thanks for looking at this Rowan. I am open to discussing different solutions (this is my second attempt), and any suggestions are welcome. In regards to consistency, while I think my RFC does that, it's not my main motivation for creating this RFC - which is to address the backwards compatibility issues. But to expand on consistency, I appreciate user defined functions (since 7.0) didn't coerce NULL, and I do want internal/user defined functions to be consistent with each other (why I agree with the spirit of the original RFC), it's just odd that string/int/float/bool can be coerced, but `NULL` cannot be coerced in this context and can be in other contexts, e.g. ``` $i = 0; $n = NULL; if ($i == $n) { // Fine } print($i); // Fine print($n); // Fine printf('%s', $i); // Fine printf('%s', $n); // Fine echo 'ConCat ' . $i . $n; // Fine echo htmlspecialchars($i); // Fine echo htmlspecialchars($n . ''); // Fine echo htmlspecialchars($n); // Bad??? ``` Consistency is a good argument for small changes that eliminate unusual > edge cases, but I think far-reaching changes should be able to stand on > their own merits - regardless of whether it's consistent, do we want the > language to work this way? > Good question, while I mention consistency, my focus is on backwards compatibility, and how PHP has accepted `NULL` to these ~335 parameters since, well, forever... my fear is that the upgrade to PHP 9 will be so difficult (as noted with the lack of tooling to solve this), I'll be auditing PHP code that's stuck on 8.x for many years to come. https://twitter.com/mwop/status/1441044164880355342 To me, the main defence of type coercion is that PHP operates in a world > full of strings - URLs are strings, HTTP requests are strings, and a lot of > API responses are strings - so making it easy to work with strings like > '42' as numbers makes a lot of sense. It's not clear to me that this > extends to null. > Yep, PHP does work in a world of strings, and for simplicity string coercion works fairly well... but I think I can include `NULL` in that definition, because HTTP requests do not guarantee values have been provided, where PHP (via filter_input) and frameworks (Laravel, Symfony, CakePHP, CodeIgniter, etc), have used `NULL` to note when a value has not been provided... and many developers have simply not cared about that distinction (e.g. when passing to htmlspecialchars, trim, strlen, preg_match, etc). I think a large part of the problem here is that null can mean many > different things in different contexts - "unknown", "not provided", > "invalid input", "default", "not applicable", etc. > > These differences are subtle, but lead to different expectations of > behaviour: > > - Treat null as a specific case with its own meaning, distinct from any > other valid value. This is what explicitly nullable parameters and union > types allow. > - Treat null the same as any other out of range value, and raise an error. > This is what happens in user-defined functions in PHP, and in built-in > functions expecting non-scalar arguments. Compare also out of range actions > like division by zero. > - Treat null as a special state that propagates through expressions, > because any operation with an unknown input has an unknown output. This is > the approach to null taken by SQL, and by IEEE floating point with NaN. It > is also the basis of the ?-> operator, and of things like Optional.map in > Swift. > - Treat null as a generic default value which can be filled in implicitly > based on requirements. This is the interpretation currently taken by > internal functions for scalar arguments, and what you are proposing to make > standard. It is also, as you point out, the way PHP treats null in some > other contexts, such as many operators. > Thank you, that's a really good explanation. The developers I work with would assume the last definition, in the same way they can pass in an integer to `urlencode()`, where they just don't think about it being implicitly coerced into a string, it just works (TM)... that's not to say the `strict_types=1` style of checking, and `NULL` being treated as an invalid value, doesn't provide value for some developers (but as shown in the stats, they are in the minority). Interestingly, one of your examples mentions filter_input, which takes the > "propagate" approach, and htmlspecialchars, which doesn't. It would often > be more useful to retain the information that a value is null than to have > it silently converted to an empty string as a side-effect of some other > operation. Those are interesting points. I think `filter_input()` is a bit different, as it's a source of data (not exactly propagating); whereas all of the other functions, they either process that data (and return a sting, and do not propagate) or work with it (e.g. preg_match and strcmp returning a bool, strlen returning an integer, etc). Retaining `NULL` is interesting, but it would be a new thing... Perhaps it would be useful to have some function-call equivalent of the ?-> > operator. I'm not sure what this would look like for normal function calls, > but it would be easy to add if we had a pipe operator, e.g.: > > If this was equivalent to htmlspecialchars($foo) > $foo |> htmlspecialchars(...) > > Then this could be equivalent to ($foo === null ? null : > htmlspecialchars($foo)) > $foo ?|> htmlspecialchars(...) > ... adding that syntax might be useful in some contexts (when the developer wants to make `NULL` easier to propagate); but that would be a new feature, and I don't think it solves the backwards compatibility issue we currently face, and what I'm trying to address in this RFC. I'm not set against this RFC, but I'm not quite convinced by the case it > makes, and think there may still be other options to explore. > I'm open to any suggestions, I just want to avoid the situation where Fatal Type Errors happen seemingly randomly (as in, someone who has never used type checks before, and they use `NULL` without any thought about it). I'm even open to the idea of someone coming up with a tool to auto-update existing code, but so far I've only heard people say that is the way it should be done (with absolutely no detail on how it would even begin to work). I also cannot explain why `NULL` should be rejected, other than for those developers who see `NULL` as an invalid value and also use `strict_types=1`... as in, when a team of developers spends a few hundred hours adding `strval()` everywhere, what does that actually achieve? what do they tell the client? does it make the code easier to read? does it avoid any bugs? or is it only for 8.1 compatibility? Anyway, thanks again Rowan, I really appreciate your thoughts on this. Craig --- ## #117543 — Rowan Tommins — 2022-04-19T13:17:38+00:00 URL: https://externals.io/message/117501#117543 · In reply to: #117542 > The developers I work with would assume the last definition I think you've somewhat missed my point. I wasn't talking about people's habits or preferences, I was talking about different *scenarios* where null is used to mean different things. Yes, some people prefer languages that "fails early" and some are more interested in "do what I mean", but not everything is about that. > I also cannot explain why `NULL` should be rejected ... does it avoid > any bugs? Yes, sometimes. Imagine an array of values provided by the user; during validation, those which were optional and not provided get set to null. You then loop through and display those which were provided: foreach ( $fields as $name => $value ) {     if ( $value !== null ) {          echo "$name: $value
\n";     } } Then, you realise you forgot about escaping, and decide to run everything through `htmlspecialchars()`: $htmlFields = array_map('htmlspecialchars', $fields); foreach ( $htmlFields as $name => $value ) {     if ( $value !== null ) {          echo "$name: $value
\n";     } } Spot the bug? $value will now never be null, because `htmlspecialchars()` will silently turn the nulls into empty strings. I've also seen the opposite problem: a string function was removed because it was no longer needed, and the code broke because values, including nulls, were no longer being cast to string. Is protecting against this worth the backwards compatibility cost of changing the behaviour, and requiring extra code in other scenarios? Possibly not. But that's different from not having any benefit. Regards, -- Rowan Tommins [IMSoP] --- ## #117551 — Craig Francis — 2022-04-20T17:02:09+00:00 URL: https://externals.io/message/117501#117551 · In reply to: #117543 > > > […] > > > I think you've somewhat missed my point. I wasn't talking about people's > habits or preferences, I was talking about different *scenarios* where > null is used to mean different things. > Yep, and I'm thankful you listed them... I'm just trying to focus on how PHP has worked (implicitly based on requirements), and how changing that causes problems (and will be an even bigger problem when it becomes a Fatal Type Error). Yes, some people prefer languages that "fails early" and some are more > interested in "do what I mean", but not everything is about that. > Agreed, the only thing I'd add... failing early with `NULL` might help debug some problems (assuming there is a problem), but I believe static analysis and unit tests are much better at doing this (e.g. static analysis is able to see if a variable can be NULL, and apply those stricter checks, as noted by George with Girgias RFC). In contrast, failing early at runtime, on something that is not actually a problem, like the examples in my RFC, creates 2 problems (primarily upgrading; but also adding unnecessary code to explicitly cast those possible `NULL` values to the correct type, which isn't needed for the other scalar types). > > […] > > > Yes, sometimes. Imagine an array of values provided by the user; during > validation, those which were optional and not provided get set to null. > You then loop through and display those which were provided: > > foreach ( $fields as $name => $value ) { > if ( $value !== null ) { > echo "$name: $value
\n"; > } > } > > Then, you realise you forgot about escaping, and decide to run > everything through `htmlspecialchars()`: > > $htmlFields = array_map('htmlspecialchars', $fields); > foreach ( $htmlFields as $name => $value ) { > if ( $value !== null ) { > echo "$name: $value
\n"; > } > } > > Spot the bug? $value will now never be null, because `htmlspecialchars()` > will silently turn the nulls into empty strings. > Thank you... but I will add, while `htmlspecialchars()` rejecting `NULL` would get you to look at the code again, I wouldn't say it's directly picking up the problem, and it relies on there being a `NULL` value in $fields for this to be noticed (if that doesen't happen, you're now in a position where random errors will start happening in production). This is where I'd note the value of unit tests, as they are much better placed to check this feature. Bit of a tangent, I'm uncomfortable that `$name` is not being HTML encoded, which takes us to context aware templating engines, and how you can identify these mistakes via the `is_literal` RFC or the `literal-string` type. I've also seen the opposite problem: a string function was removed > because it was no longer needed, and the code broke because values, > including nulls, were no longer being cast to string. > > > Is protecting against this worth the backwards compatibility cost of > changing the behaviour, and requiring extra code in other scenarios? > Possibly not. But that's different from not having any benefit. That's fair, adding in an extra check might give some benefit in some cases, but I don't think it's reliable, or worth the cost in updating existing code (with no tooling to help) and the additional code that will be needed to cast these possible `NULL` values to the relevant type (which isn't needed for the other scalar types). That said, if you (or anyone) have any better ideas on how to address this problem, please let me know. Craig --- ## #117552 — Christoph M. Becker — 2022-04-20T17:26:42+00:00 URL: https://externals.io/message/117501#117552 · In reply to: #117551 > In contrast, failing early at runtime, on something that is not actually a > problem, like the examples in my RFC, creates 2 problems (primarily > upgrading; but also adding unnecessary code to explicitly cast those > possible `NULL` values to the correct type, which isn't needed for the other > scalar types). Null is *not* a scalar type[1], though. This is the reason why it is not coerced for userland functions using *scalar* type hints with coercive typing. [1] -- Christoph M. Becker --- Next page: https://externals.io/message/117501.md?page=2