Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:123979
Content-Type: text/plain;
	charset=utf-8
Precedence: bulk
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.8\))
Subject: Re: [PHP-DEV] [Initial Feedback] PHP User Modules - An Adaptation of
 ES6 from JavaScript
In-Reply-To: <CADK1yX+E9mvaM_CHJ8tmEx=O70hf8xasbvngh4wvNM8ce8KKGQ@mail.gmail.com>
Date: Thu, 27 Jun 2024 20:16:24 -0400
Cc: Michael Morris <tendoaki@gmail.com>,
 PHP internals <internals@lists.php.net>
Content-Transfer-Encoding: quoted-printable
Message-ID: <B5DC69B8-E044-47FB-B81C-307E2C19CB66@newclarity.net>
References: <CAEUnE0fGZ3jFiBG=GyvnhBzH5a0iO4=n75PM=3+6zLAMrumEvA@mail.gmail.com>
 <CADK1yX+E9mvaM_CHJ8tmEx=O70hf8xasbvngh4wvNM8ce8KKGQ@mail.gmail.com>
To: Deleu <deleugyn@gmail.com>,
 Rob Landers <rob@bottled.codes>,
 "Rowan Tommins [IMSoP]" <imsop.php@rwec.co.uk>,
 Larry Garfield <larry@garfieldtech.com>
From: mike@newclarity.net (Mike Schinkel)

This is a long reply rather than send a bunch of shorter emails.

> On Jun 27, 2024, at 2:10 PM, Deleu <deleugyn@gmail.com> wrote:
>=20
> Overall, I think PHP has already reached the limit of surviving with =
only PSR-4 and Composer. Single class files were a great solution to get =
us out of the nightmare of `require` and `import` on top of PHP files. =
But more than once I have had the desire to declare a couple of =
interfaces in a single file, or a handful of Enums, etc.=20

This.=20

I cannot overemphasize how nice it is to work in Go where I can put =
almost any code I want in any file I want without having to think about =
autoloading.=20

It is great when writing proofs of concept and having all the code in =
one place makes it easier to reason about. Once fleshed out you can then =
organize into multiple files, but still get to keep highly related code =
in the same files.=20

> On Jun 27, 2024, at 3:02 PM, Michael Morris <tendoaki@gmail.com> =
wrote:
> Thanks. The sticking point is what degree of change should be =
occurring. PHP isn't as behind an 8-ball as JavaScript is since the dev =
can choose their PHP version and hence deprecation works most of the =
time for getting rid of old stuff. But not always. Changes that are =
incompatible with what came before need a way to do things the old way =
during transition.=20

As I understand the proposal, this would have no BC issues for code not =
in modules. PHP could then set rules for code in modules that would not =
to be directly compatible with code outside modules.=20

That's how it works in JavaScript, at least as I have experienced, and =
I'd say it works pretty well.

> Again, see PHP 6 and unicode, which snowballed until it was clear that =
even if PHP 6 had been completed it wouldn't be able to run most PHP 5 =
code.

At least to me this does not feel as big as trying to implement unicode.

>  2. No need for autoloaders with modules; I assume this would be =
obvious, right?
>=20
> Depends largely on whether modules can include and require to get =
access to old code. I also didn't discuss how they behave - do they =
share their variables with includes and requires?

I was presuming that all old code would use autoloaders but modules =
would be free to do it a better way.

If you need to call code from a namespace from inside a module, sure, =
the autoloader would be needed.

> 6. Modules should be directories, not .php files. Having each file be =
a module makes code org really hard.
>=20
> Yes, but that is how JavaScript currently handles things. It is =
currently necessary when making large packages to have an index.js that =
exports out the public members of the module. This entry point is =
configurable through the package.json of the module.

I am envisioning that there could be a module metadata file that would =
have everything that PHP needs to handle the module.  It could even be =
binary, using protobufs:

https://github.com/protobuf-c/protobuf-c

The php CLI could have an option to generate this file making it easy =
for IDEs to generate the file, or generic file watchers to generate.  =
This would mean that within a module there would be no need for an =
autoloader.=20

If the module metadata file does n0t exist, PHP could generate it on the =
fly. If the file is obviously out-of-date given a new file, PHP could =
re-generate.  If PHP can't write the file, such as on a production =
server, it throws a warning and regenerates for in-memory use each page =
load.

It iss also possible that instead of protobuf the module file could =
actually be a phar file, or the equivalent of a phar file optimized to =
allow PHP to load, access and execute that code as fast as possible.

> =20
> 7. Modules would have a symbol table metadata file generated by IDEs =
and during deployment.
>=20
> Node.js uses package.json and the attendant npm to do this sort of =
prep work.  And it's a critical part of this since modules can be =
versioned, and different modules may need to run different specific =
versions of other modules.

node_modules IMO is one of the worse things about the JavaScript =
ecosystem. Who has not seen the meme about node_modules being worse than =
a black hole?

I would argue that PHP itself not be involved in trying to manage =
versions. Let Composer do that, or whatever other tool developers =
currently use to manage versions, or new tools developed later.

> 9. .php files in modules as identified by metadata file should not be =
loadable via HTTP(S).=20
>=20
> Those are implementation details a little further down the road than =
we're ready for, I think.=20

But ensuring that it is possible to disallow loading needs to be =
contemplated in the design. PHP has to be able to know what is a module =
and what isn't without expensive processes.

> 10. Having exports separate from functions and classes seems like it =
would be problematic.
>=20
> Again, this is how they work in JavaScript. Not saying that's the best =
approach, but even if problematic it's a solved problem.

I have evidently not written enough JavaScript to realize that.

> I'm also interested in learning on how other module systems out there =
do work.

I am very familiar with modules (packages) in GoLang and think PHP could =
benefit from considering how they work, too.

> On Jun 27, 2024, at 3:22 PM, Michael Morris <tendoaki@gmail.com> =
wrote:
> Composer would need a massive rewrite to be a part of this since it =
currently requires the file once it determines it should do so. If we do =
a system where import causes the parser to act differently then that =
alone means imports can't be dealt with in the same manner as other =
autoloads.

That is why I am strongly recommending a modern symbol resolution system =
within modules vs. autoloading.

>> I'm not fond of this either.
>=20
> There will need to be a way to define the entrypoint php.  I think =
index.php is reasonable, and if another entry point is desired it can be =
called out -> "mypackage/myentry.php"

Why is an entry point needed?  If there is a module metadata file as I =
am proposing PHP can get all the information it needs from that file. =
Maybe that is the `.phm` file?

> On Jun 27, 2024, at 4:54 PM, Rob Landers <rob@bottled.codes> wrote:
>> Thanks. The sticking point is what degree of change should be =
occurring. PHP isn't as behind an 8-ball as JavaScript is since the dev =
can choose their PHP version and hence deprecation works most of the =
time for getting rid of old stuff. But not always. Changes that are =
incompatible with what came before need a way to do things the old way =
during transition. Again, see PHP 6 and unicode, which snowballed until =
it was clear that even if PHP 6 had been completed it wouldn't be able =
to run most PHP 5 code.
>=20
> It=E2=80=99s not just up to the dev, but the libraries we use and =
whether or not we can easily upgrade (or remove) them to upgrade the php =
version.

By "upgrade" then, do you mean convert them into modules, or just be =
able to use them as-is. =20

As I read it and am envisioning it, there would be no changes needed to =
be able to use them as-is.

> I think it would be a mistake to exclude old code and/or prevent =
templating. Not only are there now decades old code in some orgs, but =
how would you write an email sender that sent templated emails, provide =
html, generate code, etc? There has to be an output from the code to be =
useful.=20

Excluding old code or templates from modules would not exclude them from =
working as they currently do outside modules.  As I see it, modules =
would be more about exporting classes and functions, not generating =
output per se.=20

So all that decades of old code could continue to exist outside modules, =
as it currently does today.

> I think it=E2=80=99s fine to use js as an inspiration, but it isn=E2=80=99=
t the only one out there. There is some precedent to consider =
directories as modules (go calls them =E2=80=9Cpackages=E2=80=9D) and =
especially in PHP where namespaces (due to PSR-4 autoloading) typically =
match directory structures.

Totally agree about inspiration for modules outside JS, but not sure =
that PHP namespaces are the best place to look for inspiration. =20

Namespaces by their very nature were designed to enable autoloading with =
a one-to-one file to class or interface, and by nature add conceptual =
scope and complexity to a project that would not be required if a modern =
module/package system were added to PHP.=20

Modules could and IMO should be a rethink that learns the lessons other =
languages have learned over the past decade+.=20

>> Node.js uses package.json and the attendant npm to do this sort of =
prep work.  And it's a critical part of this since modules can be =
versioned, and different modules may need to run different specific =
versions of other modules.
>=20
> Please, please, please do not make a json file a configuration =
language. You can=E2=80=99t comment in them, you can=E2=80=99t handle =
=E2=80=9Cif php version <9, load this, or if this extension is =
installed, use this.=E2=80=9D
>=20
> Maybe that is desirable, but doing things slightly different based on =
extensions loaded is def a thing.=20

I don't think commenting is important in this file, or even desired.=20

As I proposed above, these could be protobuf or phar.  These should be =
build artifacts that can be generated on the fly during development or =
for newbies even during deployment, not hand-managed.

I could see the generation of two files; one in binary form and one that =
is readonly so a developer can double-check what is in the current =
protobuf or phar file.

>> Those are implementation details a little further down the road than =
we're ready for, I think.=20
>=20
> Personally, if these are going to have any special syntax, we probably =
shouldn=E2=80=99t call them .php files. Maybe .phm?

I was going to suggest that, and then remembered earlier PHP when there =
were multiple file extensions and that was a nightmare. =20

This does remind me to mention that I think there should be a required =
"module" declaration at the top of each file just like Go requires a =
"package" declaration at the top of each file. That would make it =
trivial for tooling to differentiate, even with grep.

> the only thing I don=E2=80=99t like about this import/export thing is =
that it reminds me of the days when we had to carefully order our =
require_once directives to make sure files were loaded before they were =
used. So, I think it is worth thinking about how loading will work and =
whether loading can be dynamic, hoisted out of function calls (like js), =
how order matters, whether packages can enrich other packages (like =
doctrine packages) and if so, how much they can gain access to internal =
state, etc. This is very much not =E2=80=9Ca solved problem.=E2=80=9D

That is why I proposed having a "compiled" module symbol table to =
eliminate most (all?) of those issues.

> On Jun 27, 2024, at 6:00 PM, Rowan Tommins [IMSoP] =
<imsop.php@rwec.co.uk> wrote:
> I do think PHP badly needs a native concept of "module" or "package" - =
in fact, I'm increasingly convinced it's the inevitable path we'll end =
up on at some point. BUT I think any such concept needs to be built on =
top of what we have right now. That means:
>=20
> - It should build on or work in harmony with namespaces, not ignore or =
replace them

It may be an unpopular opinion, but I would argue that namespaces were =
optimized for autoloading and the one class/interface per file paradigm, =
not to mention to regrettable choice of using the escape operator to =
seperate namespaces and that fact that PHP throws away a lot of =
information about namespaces at runtime.

IMO allowing modules to eventually deprecate namespaces =E2=80=94 at =
least in a defacto form of deprecation =E2=80=94 would allow modules to =
be much better than if the try to cling to a less desirable past.

> - It should be easy to take existing code, and convert it to a =
module/package

Maybe, but not if that means modules retain baggage that should really =
be jettisoned.

> and namespaces have proved an extremely successful way of sharing code =
without those names colliding.

At the expense of a lot more complexity than necessary, yes.=20

Managing symbols in a module need not be a hard problem if PHP =
recognizes modules internally rather than trying to munge everything =
into a global namespace like with namespaces.

> Other parts of your e-mail are essentially an unrelated idea, to have =
some new "PHP++" dialect, where a bunch of "bad" things are removed. =
You're not the first person to be tempted by this, but I think the =
history HHVM and Hack is educational here: initially, PHP and Hack were =
designed to interoperate on one run-time, but the more they tried to =
optimise for Hack, the harder it became to support PHP, and now Hack is =
a completely independent language.

While I agree that some things are unnecessary =E2=80=94 such as =
unifying scope resolution operators for existing concepts =E2=80=94 past =
failure does not guarantee future failure.

Hack tried to create an entirely new language yet still be PHP =
compatible. Learn from the Hack experience and rather than create an =
entirely new language, PHP modules could simply add constraints for code =
in modules, and then any "new" language features that are not =
module-specific by-nature should be considered to work everywhere in =
PHP, or not at all.

> On Jun 27, 2024, at 6:41 PM, Larry Garfield <larry@garfieldtech.com> =
wrote:
> What problem would packages/modules/whatever be solving that isn't =
already adequately solved?

Not speaking for Michael, obviously, but speaking for what I envision:

1. Adding a module/package system to PHP with modern module features
	- including module private, module function, and module =
properties
2. Providing an alternative to auto-loader-optimized namespaces.
	- better code management and better page load performance=20

> Do we want:
>=20
> 1. Packages and namespaces are synonymous?  (This is roughly how JVM =
languages work, I believe.)
> 2. Packages and files are synonymous?  (This is how Python and =
Javascript work.)
> 3. All packages correspond to a namespace, but not all namespaces are =
a package?

I would argue packages (modules) should be orthogonal to namespaces to =
allow modules to be optimized for what other languages have learned =
about packages/modules over the past decade+.

The fact that namespaces use the escape character as a separator, that =
PHP does not keep track of namespace after parsing is enough reason to =
move on from them, and that they were optimize for one-to-one symbol to =
file autoload are enough reasons IMO to envision a way to move on from =
them.

> And given the near-universality of PSR-4 file structure, what impact =
would each of those have in practice? =20

Orthogonal.  Old way vs new way.  But still completely usable, just not =
as modules without conversion.

The fact PSR-4 exists is an artifact of autoloading single-symbol files =
and thus a sunken cost does not mean that PHP should not cling to for =
modules just because they currently exist.

-Mike