# The case for transpiled generics - Thread: https://externals.io/message/111875 - Original archive: https://news-web.php.net/php.internals/111875 - Started by Brent Roose on 2020-09-17T11:33:23+00:00 - Messages: 10 - RFC: Constructor Property Promotion (Implemented (PHP 8.0)) — https://externals.io/rfc/constructor_promotion.md - RFC: Add generic class and method support (Inactive) — https://externals.io/rfc/generics.md - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · RFCs at https://externals.io/rfc.md · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #111875 — Brent Roose — 2020-09-17T11:33:23+00:00 URL: https://externals.io/message/111875#111875 Hello internals Today I'd like to hear your thoughts on what might be a controversial topic, though I think it's worth having this discussion. I want to make the case for adding generic syntax, without actually enforing any additional type checks at runtime. Please hear me out. We've been discussing generics for years now [1][2], all without any result. Nikita's latest attempt [3] stalled because, from what I gathered and amongst other things, doing generic type checks at runtime has a significant impact on performance. On the other hand, static analysers have been making their rise for a few years now. Granted: not the whole community might like this kind of type strictness, and PHP doesn't force them to; but still projects like PhpStorm acknowledge their significance — they will add built-in support for both psalm and PHPStan later this year [4]. Rasmus Lerdorf also showed interest in the idea of improving PHP's static analysis capabilities two years ago [5]. That all to say that there's a significant part of the PHP community who's interested in embracing the benefits of static analysis. If we look outside of our PHP bubble, we can see the same thing happening in JavaScript: the core benefit that TypeScript adds is its robust static analysis. Sure those developers need an extra compilation step to transpile their code to plain old JavaScript, but it seems that they are… fine with that? I'd like to discuss a similar idea for PHP. If runtime generics aren't possible because of performance issues, why not explore the other option: adding generic syntax that is ignored by the interpreter, but can be used by static analysis tools — third party of built-into PHP, that's another discussion. I realise this thought goes against the "PHP mindset" we've been programming with for more than 20 years, but we I think we shouldn't ignore what's happening in the PHP- and wider progamming community: static analysis is relevant, whether you want to use it or not, and a stricter type system is prefered by many. Now I know there are alternatives we can use today. Static analysers already support generics, using doc blocks. I'm not trying to argue that it's impossible to achieve the same results with the toolset we have, but rather that there's room for improvement from the developer's point of view. History has shown that such convenience additions to PHP have been a difficult pill to swallow for some, but on the other hand those kind of changes _have_ been happening more and more often anyway: property promotion, short closures, named arguments, attributes, yes even types themselves: you can write the same working PHP program without any of those features, and yet they have been proven so useful and wanted over the last years. As a sidenote: the idea of transpiling is already present in PHP. Looking at constructor property promotion: a purely syntactical feature, which is transformed to simpler PHP code at runtime. Nikita called this principle "desugaring" in the constructor property promotion RFC [6]. So here's my case for transpiled generics summarized: - There's no significant runtime performance impact - The PHP community is already embracing static analysis - Transpiling has been proved to be a viable workflow, thanks to TypeScript - As with all things-PHP: it's opt-in. You don't have to use the syntax if you don't want to and you won't experience any downsides So with all that being said, I'm looking forward to hearing your thoughts. Kind regards Brent [1] https://wiki.php.net/rfc/generics [2] https://wiki.php.net/rfc/generic-arrays [3] https://github.com/PHPGenerics/php-generics-rfc/issues/45 [4] https://blog.jetbrains.com/phpstorm/2020/07/phpstan-and-psalm-support-coming-to-phpstorm/ [5] https://externals.io/message/101477#101592 [6] https://wiki.php.net/rfc/constructor_promotion#desugaring Hello internals > > Today I'd like to hear your thoughts on what might be a controversial > topic, though I think it's worth having this discussion. I want to make the > case for adding generic syntax, without actually enforing any additional > type checks at runtime. Please hear me out. > > We've been discussing generics for years now [1][2], all without any > result. Nikita's latest attempt [3] stalled because, from what I gathered > and amongst other things, doing generic type checks at runtime has a > significant impact on performance. > > On the other hand, static analysers have been making their rise for a few > years now. Granted: not the whole community might like this kind of type > strictness, and PHP doesn't force them to; but still projects like PhpStorm > acknowledge their significance — they will add built-in support for both > psalm and PHPStan later this year [4]. Rasmus Lerdorf also showed interest > in the idea of improving PHP's static analysis capabilities two years ago > [5]. > > That all to say that there's a significant part of the PHP community who's > interested in embracing the benefits of static analysis. > > If we look outside of our PHP bubble, we can see the same thing happening > in JavaScript: the core benefit that TypeScript adds is its robust static > analysis. Sure those developers need an extra compilation step to transpile > their code to plain old JavaScript, but it seems that they are… fine with > that? > > I'd like to discuss a similar idea for PHP. If runtime generics aren't > possible because of performance issues, why not explore the other option: > adding generic syntax that is ignored by the interpreter, but can be used > by static analysis tools — third party of built-into PHP, that's another > discussion. I realise this thought goes against the "PHP mindset" we've > been programming with for more than 20 years, but we I think we shouldn't > ignore what's happening in the PHP- and wider progamming community: static > analysis is relevant, whether you want to use it or not, and a stricter > type system is prefered by many. > > Now I know there are alternatives we can use today. Static analysers > already support generics, using doc blocks. I'm not trying to argue that > it's impossible to achieve the same results with the toolset we have, but > rather that there's room for improvement from the developer's point of > view. History has shown that such convenience additions to PHP have been a > difficult pill to swallow for some, but on the other hand those kind of > changes _have_ been happening more and more often anyway: property > promotion, short closures, named arguments, attributes, yes even types > themselves: you can write the same working PHP program without any of those > features, and yet they have been proven so useful and wanted over the last > years. > > As a sidenote: the idea of transpiling is already present in PHP. Looking > at constructor property promotion: a purely syntactical feature, which is > transformed to simpler PHP code at runtime. Nikita called this principle > "desugaring" in the constructor property promotion RFC [6]. > > So here's my case for transpiled generics summarized: > > - There's no significant runtime performance impact > - The PHP community is already embracing static analysis > - Transpiling has been proved to be a viable workflow, thanks to TypeScript > - As with all things-PHP: it's opt-in. You don't have to use the syntax if > you don't want to and you won't experience any downsides > > So with all that being said, I'm looking forward to hearing your thoughts. > > Kind regards > Brent > > [1] https://wiki.php.net/rfc/generics > [2] https://wiki.php.net/rfc/generic-arrays < > https://wiki.php.net/rfc/generic-arrays> > [3] https://github.com/PHPGenerics/php-generics-rfc/issues/45 < > https://github.com/PHPGenerics/php-generics-rfc/issues/45> > [4] > https://blog.jetbrains.com/phpstorm/2020/07/phpstan-and-psalm-support-coming-to-phpstorm/ > < > https://blog.jetbrains.com/phpstorm/2020/07/phpstan-and-psalm-support-coming-to-phpstorm/> > > [5] https://externals.io/message/101477#101592 < > https://externals.io/message/101477#101592> > [6] https://wiki.php.net/rfc/constructor_promotion#desugaring < > https://wiki.php.net/rfc/constructor_promotion#desugaring> Hello Brent, I'm going to make the argument I've already done on Reddit once [1], IMHO TypeScript is just a nicer pipeline for a preprocessor and a static analyser and not a language per say. Let me explain with a simpler example than generics, adding immutable objects to PHP without native language support using a preprocessor and Psalm as the back-end for static-analysis. It would be rather trivial to do a search-and-replace of the word immutable and replace it with /** @psalm-immutable */ at the (currently non existent) preprocessor level then run Psalm and have it shout at you if "you are doing it wrong"TM. Now bundle the two tools into a nice CLI command run TypePHP root_of_project and voilà support for immutable by just writing immutable class Foo {} And I'm fairly certain that one can achieve support for generic templates in the same way, probably harder than adding support for immutable objects. It is also possible to have various back-ends for the static analysers which would change the doc-annotation to the one understood by the analyser asked. Most importantly this can be done purely in PHP and distributed as a composer package. Therefore needing no support from the language itself, thus this can even work in PHP 7. As such I see no benefit in supporting generics at the language level if they don't have any runtime checks. Best regards George P. Banyard [1] https://www.reddit.com/r/PHP/comments/i9h1v8/considering_php/g1h9umy?context=3 --- ## #111885 — Matthew Brown — 2020-09-17T15:21:12+00:00 URL: https://externals.io/message/111875#111885 · In reply to: #111875 Quick thing before I get into my own reaction: Transpiling is normally thought of as the process of converting one language into another. Tools like Babel transpile TypeScript to JavaScript. What's being proposed here (AFAICT) is type erasure – the generic type information would be erased during the conversion to opcodes. Python (the language) has type erasure for all its types (including its generic types), so no popular Python interpreters check that a function call's argument types match up with the function signature. PHP currently does not erase any types so the opcodes generated by PHP's interpreter include type checks for all the arguments passed to a typed function signature. Hack follows PHP's model, but erases generic types by default (though has more recently introduced the concept of reified generics) in much the same way you're proposing. --- ## #111886 — Larry Garfield — 2020-09-17T16:21:48+00:00 URL: https://externals.io/message/111875#111886 · In reply to: #111885 > Quick thing before I get into my own reaction: > > Transpiling is normally thought of as the process of converting one > language into another. Tools like Babel transpile TypeScript to JavaScript. > > What's being proposed here (AFAICT) is type erasure – the generic type > information would be erased during the conversion to opcodes. > > Python (the language) has type erasure for all its types (including its > generic types), so no popular Python interpreters check that a function > call's argument types match up with the function signature. > > PHP currently does not erase any types so the opcodes generated by PHP's > interpreter include type checks for all the arguments passed to a typed > function signature. > > Hack follows PHP's model, but erases generic types by default (though has > more recently introduced the concept of reified generics) in much the same > way you're proposing. I concur; this is type elidiing, not transpiling. Transpiling implies a user-triggered pre-run step. I know Sara has mused about it before but it's not happened yet. Basically what is being proposed (for those who need a concrete example) is you'd write this: class Collection { public function add(T $item) { ... } } $c = new Collection(). And linters would recognize that, and it would parse, but at runtime it would compile to: class Collection { public function add(mixed $item) { ... } } $c = new Collection(). And that's the opcodes that would be saved. (I'm hand-waving a lot here.) I would be on board with this, although I would ask if it's possible to provide some compile time checks; I'm thinking LSP validation, which is still relevant for inheritance with generics. Beyond that, leave the runtime to not enforce it. (I have no idea how viable that is, but Sara seemed to think it would work so it probably would.) Another possibility could be to desugar that original to: class Collection { public $__T; public function add(mixed $item) { if (!$item instanceof $this->__T) { throw new TypeError(); } } } $c = new Collection(). $c->__T == 'Product'; Which is essentially what you can do in userspace today, but automated. I don't know how viable that is, but it's another thing to consider. --Larry Garfield --- ## #111887 — Ilija Tovilo — 2020-09-17T17:01:19+00:00 URL: https://externals.io/message/111875#111887 · In reply to: #111875 Hi Brent > Today I'd like to hear your thoughts on what might be a controversial topic, though I think it's worth having this discussion. I want to make the case for adding generic syntax, without actually enforing any additional type checks at runtime. Please hear me out. I think having the option to omit runtime type checks could be valuable even for non-generic code. If your own internal classes all use static analysis the chance of passing a wrong type is pretty low. This would probably have to be on a per class or even per function basis as you can't reason about other people's code. I do think we need some form of generic runtime check. If we don't, how generics work is completely open to interpretation. Static analysis tools copy the behavior of PHP. If PHP has no behavior in this regard tools will deviate and generics won't have any real meaning at all. There are many nuances to generics (ask Matthew who wrote Psalm), we shouldn't underestimate that. With a hybrid approach we could also do full type checking in testing environments and disable it in more performance critical situations with more confidence. Ilija --- ## #111888 — Brent Roose — 2020-09-17T17:22:09+00:00 URL: https://externals.io/message/111875#111888 · In reply to: #111877 Hi George Thanks for the reply! > > > Hello Brent, > > I'm going to make the argument I've already done on Reddit once [1], IMHO > TypeScript is just a nicer pipeline for a preprocessor and a static > analyser and not a language per say. > > Let me explain with a simpler example than generics, adding immutable > objects to PHP without native language support using a preprocessor and > Psalm as the back-end for static-analysis. It would be rather trivial to do > a search-and-replace of the word immutable and replace it with /** > @psalm-immutable */ at the (currently non existent) preprocessor level then > run Psalm and have it shout at you if "you are doing it wrong"TM. > Now bundle the two tools into a nice CLI command run TypePHP root_of_project > and voilà support for immutable by just writing > > immutable class Foo {} > > And I'm fairly certain that one can achieve support for generic templates > in the same way, probably harder than adding support for immutable objects. > It is also possible to have various back-ends for the static analysers > which would change the doc-annotation to the one understood by the analyser > asked. > > Most importantly this can be done purely in PHP and distributed as a > composer package. > Therefore needing no support from the language itself, thus this can even > work in PHP 7. Preprocessing PHP has been done [1], and the reality is that it didn't work. Not because it's technically impossible, but because there's no tooling around it: no IDE integration and no static analysis. The pre project actually tried developing a plugin for PhpStorm, which failed due to plugin API limitations. Tooling and IDEs tend to follow the language spec, because there's no way around it. That power lies within the PHP project, and no one else in the community has the same amount of influence. > > As such I see no benefit in supporting generics at the language level if > they don't have any runtime checks. Like I said in my mail: it's not about being able to achieve a specific kind of functionality — static analysers already support generics — it's about giving developers an easy, integrated way of writing them, with as less overhead as possible. I think most new features added to PHP these days are a testament to that mindset — I've already mentioned them in my previous post. > > Best regards > > George P. Banyard > > [1] > https://www.reddit.com/r/PHP/comments/i9h1v8/considering_php/g1h9umy?context=3 King regards Brent [1] https://preprocess.io/#/ --- ## #111889 — Brent Roose — 2020-09-17T17:28:27+00:00 URL: https://externals.io/message/111875#111889 · In reply to: #111886 Hi Matthew and Larry > > > > > […] I looked up the definition [1] before writing my mail and it said > A […] transpiler is a type of translator that takes the source code of a program written in a programming language as its input and produces an equivalent source code in the same or a different programming language But I don't want to get stuck on phrasing, if elidiing is the right term as Larry suggests, let's go with it! > > […] I didn't know about Python's approach before, and I think it's validating the idea, it's at least something worth looking into. > > […] > > I concur; this is type elidiing, not transpiling. Transpiling implies a user-triggered pre-run step. I know Sara has mused about it before but it's not happened yet. > > Basically what is being proposed (for those who need a concrete example) is you'd write this: > > class Collection { > > public function add(T $item) { ... } > > } > $c = new Collection(). > > And linters would recognize that, and it would parse, but at runtime it would compile to: > > class Collection { > > public function add(mixed $item) { ... } > > } > $c = new Collection(). > > > And that's the opcodes that would be saved. (I'm hand-waving a lot here.) > > I would be on board with this, although I would ask if it's possible to provide some compile time checks; I'm thinking LSP validation, which is still relevant for inheritance with generics. Beyond that, leave the runtime to not enforce it. (I have no idea how viable that is, but Sara seemed to think it would work so it probably I agree that's an approach worth looking into. I didn't want to discuss specifics yet, and first get a feel of what people think of the general idea, but it's good that you're mentioning this, and I'm glad others are open to the idea. > > Another possibility could be to desugar that original to: > > class Collection { > public $__T; > > public function add(mixed $item) { > if (!$item instanceof $this->__T) { > throw new TypeError(); > } > } > } > $c = new Collection(). > $c->__T == 'Product'; > > Which is essentially what you can do in userspace today, but automated. I don't know how viable that is, but it's another thing to consider. > > --Larry Garfield > > -- > > To unsubscribe, visit: https://www.php.net/unsub.php > Kind regards Brent [1] https://en.wikipedia.org/wiki/Source-to-source_compiler --- ## #111890 — Brent Roose — 2020-09-17T17:30:45+00:00 URL: https://externals.io/message/111875#111890 · In reply to: #111887 Hi Ilija > > > Hi Brent > > > […] > > I think having the option to omit runtime type checks could be > valuable even for non-generic code. If your own internal classes all > use static analysis the chance of passing a wrong type is pretty low. > This would probably have to be on a per class or even per function > basis as you can't reason about other people's code. I agree, if there's enough interest in this, we might need to consider the scope of one or several RFCs. > > I do think we need some form of generic runtime check. If we don't, > how generics work is completely open to interpretation. Static > analysis tools copy the behavior of PHP. If PHP has no behavior in > this regard tools will deviate and generics won't have any real > meaning at all. There are many nuances to generics (ask Matthew who > wrote Psalm), we shouldn't underestimate that. With a hybrid approach > we could also do full type checking in testing environments and > disable it in more performance critical situations with more > confidence. Larry also shared this concern, and I agree that it's important to at least have an official spec from the PHP project. Whether part of it should be enforced at runtime is up for debate, but it's definitely important. > > Ilija > > -- > > To unsubscribe, visit: https://www.php.net/unsub.php > Kind regards Brent --- ## #111895 — Matthew Brown — 2020-09-17T21:00:26+00:00 URL: https://externals.io/message/111875#111895 · In reply to: #111875 I think the addition of runtime-erased generics would be a good thing for the language. I don’t think PHP should perform any sort of checking of generic parameters at compile time. No other equivalent interpreter performs those compile-time checks, and it’s also *very* complex — since adding support for generics in Psalm I’ve spent the ensuing two years fixing many bugs and filling various holes that have cropped up. Hack, introduced in 2014 and written by bunch of people far smarter than me, had at least one hole in its type-checker’s generics handling that was only patched last year. PHP could develop a separate official type-checker to be used alongside its interpreter, similar to Hack’s (hh_check) or TypeScript’s (typescript-checker), but that’s an entirely separate conversation. ——— If this proposal happens, the documentation should make it *super* clear that all generic types are erased at runtime. Erased generics would not support code like this: ``` function filter(array $arr) : array { $new = []; foreach ($arr as $a) { if ($a instanceof T) { $new[] = $a; } } } ``` Instead PHP would need to support a type classname (or similar) to allow passing generic params explicitly. ``` function filter(array $arr, classname $t_class) : array { $new = []; foreach ($arr as $a) { if ($a instanceof $t_class) { $new[] = $a; } } } ``` The implementation would also need a way of denoting covariant (pretty common) and contravariant (very rare) generic parameters, like Hack does. > > > Hello internals > > Today I'd like to hear your thoughts on what might be a controversial topic, though I think it's worth having this discussion. I want to make the case for adding generic syntax, without actually enforing any additional type checks at runtime. Please hear me out. > > We've been discussing generics for years now [1][2], all without any result. Nikita's latest attempt [3] stalled because, from what I gathered and amongst other things, doing generic type checks at runtime has a significant impact on performance. > > On the other hand, static analysers have been making their rise for a few years now. Granted: not the whole community might like this kind of type strictness, and PHP doesn't force them to; but still projects like PhpStorm acknowledge their significance — they will add built-in support for both psalm and PHPStan later this year [4]. Rasmus Lerdorf also showed interest in the idea of improving PHP's static analysis capabilities two years ago [5]. > > That all to say that there's a significant part of the PHP community who's interested in embracing the benefits of static analysis. > > If we look outside of our PHP bubble, we can see the same thing happening in JavaScript: the core benefit that TypeScript adds is its robust static analysis. Sure those developers need an extra compilation step to transpile their code to plain old JavaScript, but it seems that they are… fine with that? > > I'd like to discuss a similar idea for PHP. If runtime generics aren't possible because of performance issues, why not explore the other option: adding generic syntax that is ignored by the interpreter, but can be used by static analysis tools — third party of built-into PHP, that's another discussion. I realise this thought goes against the "PHP mindset" we've been programming with for more than 20 years, but we I think we shouldn't ignore what's happening in the PHP- and wider progamming community: static analysis is relevant, whether you want to use it or not, and a stricter type system is prefered by many. > > Now I know there are alternatives we can use today. Static analysers already support generics, using doc blocks. I'm not trying to argue that it's impossible to achieve the same results with the toolset we have, but rather that there's room for improvement from the developer's point of view. History has shown that such convenience additions to PHP have been a difficult pill to swallow for some, but on the other hand those kind of changes _have_ been happening more and more often anyway: property promotion, short closures, named arguments, attributes, yes even types themselves: you can write the same working PHP program without any of those features, and yet they have been proven so useful and wanted over the last years. > > As a sidenote: the idea of transpiling is already present in PHP. Looking at constructor property promotion: a purely syntactical feature, which is transformed to simpler PHP code at runtime. Nikita called this principle "desugaring" in the constructor property promotion RFC [6]. > > So here's my case for transpiled generics summarized: > > - There's no significant runtime performance impact > - The PHP community is already embracing static analysis > - Transpiling has been proved to be a viable workflow, thanks to TypeScript > - As with all things-PHP: it's opt-in. You don't have to use the syntax if you don't want to and you won't experience any downsides > > So with all that being said, I'm looking forward to hearing your thoughts. > > Kind regards > Brent > > [1] https://wiki.php.net/rfc/generics > [2] https://wiki.php.net/rfc/generic-arrays > [3] https://github.com/PHPGenerics/php-generics-rfc/issues/45 > [4] https://blog.jetbrains.com/phpstorm/2020/07/phpstan-and-psalm-support-coming-to-phpstorm/ > [5] https://externals.io/message/101477#101592 > [6] https://wiki.php.net/rfc/constructor_promotion#desugaring > > I think the addition of runtime-erased generics would be a good thing for the language. > > I don’t think PHP should perform any sort of checking of generic parameters at compile time. No other equivalent interpreter performs those compile-time checks, and it’s also *very* complex — since adding support for generics in Psalm I’ve spent the ensuing two years fixing many bugs and filling various holes that have cropped up. Hack, introduced in 2014 and written by bunch of people far smarter than me, had at least one hole in its type-checker’s generics handling that was only patched last year. > > PHP could develop a separate official type-checker to be used alongside its interpreter, similar to Hack’s (hh_check) or TypeScript’s (typescript-checker), but that’s an entirely separate conversation. > > ——— > > If this proposal happens, the documentation should make it *super* clear that all generic types are erased at runtime. > > Erased generics would not support code like this: > > ``` > function filter(array $arr) : array { > $new = []; > foreach ($arr as $a) { > if ($a instanceof T) { > $new[] = $a; > } > } > } > ``` > > Instead PHP would need to support a type classname (or similar) to allow passing generic params explicitly. > > ``` > function filter(array $arr, classname $t_class) : array { > $new = []; > foreach ($arr as $a) { > if ($a instanceof $t_class) { > $new[] = $a; > } > } > } > ``` > > The implementation would also need a way of denoting covariant (pretty common) and contravariant (very rare) generic parameters, like Hack does. > > > > […]