Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:124118
Precedence: bulk
MIME-Version: 1.0
References: <CAEUnE0fGZ3jFiBG=GyvnhBzH5a0iO4=n75PM=3+6zLAMrumEvA@mail.gmail.com>
 <CADK1yX+E9mvaM_CHJ8tmEx=O70hf8xasbvngh4wvNM8ce8KKGQ@mail.gmail.com>
 <B5DC69B8-E044-47FB-B81C-307E2C19CB66@newclarity.net> <1917CF7C-26D8-4DBE-B05C-5AA650AC6C9F@rwec.co.uk>
 <551cd5b0-1c00-4818-a9ca-97f6b7e8c3dc@app.fastmail.com> <CAEUnE0fNxSHN4QGfm6ETMVdE5Mcm1AEebmWs7i9xDNgcWnvQvw@mail.gmail.com>
 <39B496F8-062E-4848-9B3B-529BE8D3415A@newclarity.net> <CAEUnE0fOfU1tJNKXuBqbSm8jsp7gqe_JY-SiipdUKNztcx0Rog@mail.gmail.com>
 <856F4F70-DC81-4098-82DD-5F6D47CDF3F0@newclarity.net> <CAEUnE0c0fW9xnQOtLgj1zTpvdfixSHuLeLA7HOqDTkpr4hJ84g@mail.gmail.com>
In-Reply-To: <CAEUnE0c0fW9xnQOtLgj1zTpvdfixSHuLeLA7HOqDTkpr4hJ84g@mail.gmail.com>
Date: Sun, 30 Jun 2024 19:12:50 -0400
Message-ID: <CAEUnE0fyOjQ4X_-X9nGqKnscUBPuJAV2PtPxdWJsRWJYLw+toA@mail.gmail.com>
Subject: [PHP-DEV] Packages Iteration 3: was Re: [PHP-DEV] [Initial Feedback]
 PHP User Modules - An Adaptation of ES6 from JavaScript
To: PHP internals <internals@lists.php.net>
Content-Type: multipart/alternative; boundary="00000000000055b45f061c239fcd"
From: tendoaki@gmail.com (Michael Morris)

--00000000000055b45f061c239fcd
Content-Type: text/plain; charset="UTF-8"

First off, in 10 years of using gmail I've never had it lose an email.
Well, it happened after I spent 4 hours on this.  So, this is sorta
iteration 4.  I'll type this up in Visual Studio code and then paste to
gmail.

The Wordpress discussion about composer and the decision not to use it keys
in the features a package management system must meet. These are

* Command Line should not be required
* Plugins using version X of a package should not be affected by plugins
using version Y
* Install and maintenance of the site should remain possible from the
browser alone.
* Backwards compatibility must be maintained.

A core PHP package management system should be a revision of what's come
before that doesn't disturb what has come before. PHP packages are as
follows:

* Extensions, written in C++
* PECL Libraries, which I believe can be in PHP or C, but are server-wide
* Phar archives
* Composer libraries, which are always in PHP and leverage namespaces,
PSR-4 convention and the autoload system.

Some of my prior discussions have been about what I'd like to see in a new
module system, but for scope, clarity, and sanity reasons I'm going to set
that aside. (Not to mention some of the more controversial pieces have been
there)

Within the package management system there is a single new keyword -
import. I'll go into details on it in a bit but suffice to say its behavior
is different from include/require.

In this schema a PHP application is a collection of packages.  Existing PHP
code that does not use the import statement or the special directories
mentioned below will not invoke anything discussed on this page and will
not need to change in any way.


Applications

The application is the root package. It is the package that imports to the
root namespace. When PHP is asked to parse a file it will look for a
`.php-packages` folder, first in the current working directory then in
parent directories.  If it doesn't find one, business as usual.  If we do
find one we follow its directives about setting up an application
environment.

The `.php-packages` folder is where PHP will put package related code for
the application at hand.  Code written explicitly for these changes will
also put their package related files there - composer's vendor directory,
composer.json, composer.lock, and so on - rather than putting those files
in the site root. The folder is hidden to prevent web servers like nginx or
apache from serving the files directly in any way.

The .php-packages directory will have a configuration file called
`php.mod`.  This tells the parser:

* How to initialize the application
* What autoloaders are to be used
* How to resolve import

Let's look at what such a file might look like for Drupal. For the moment
I'm going to use go.mod's syntax. The final syntax to be used, be it ini,
yaml, toml, json, is a discussion for another time. The part to focus in on
here is what type of information do we need.

  package Drupal

  php 10

  registry //packagist.org/packages composer

  init (
    composer install
  )

  require (
    ./vendor/autoload.php
  )

  imports (
    //getcomposer.org/composer.phar
  )

The directives do the following:
* package sets the name of the package if it is imported. Sort of
irrelevant here, but present for consistency
* registry sets up the default registry for the import statement and the
loader used to install packages
* init is the command(s) to run before the application is started for the
first time.
* require is the file(s) to require before running any file in the
application
* imports are the imports the package needs, in this case the composer.phar
to load composer in locally.

Now, given the popularity of composer some of these directives likely could
be put in as logical defines, particularly the init, require and imports
directives just being put in place if composer is selected as a loader.  I
don't rule out the possibility of a competitor to composer showing up one
day though, as yarn was introduced as an alternative to npm.

If this theoretical version of Drupal moves its composer.json and
composer.lock files into `.php-packages` then the autoloader doesn't have
to be required in the index.php file.  Also, the application can be started
without running `composer install`


Import statement
Given the Drupal application package in the previous section, we can have
an extension file call out its dependencies in code rather than in config.

  import "twig/twig"

But granted, there's not much point to this since Drupal already uses
Twig.  But suppose for the sake of argument that Symfony releases a Twig 4
with some major BC breaks, and we're working on a Drupal extension that
wants to use the new features of that Twig before Drupal itself upgrades
Twig in core. Well, now we can do this

  import "twig/twig ^4.0" as NewTwig

Note that we have to mount this package off the root using the alias syntax
above.  If the new package isn't written with this system in mind then
we'll end up having NewTwig prefixed on the normal namespace path which
would end up looking like this

  use NewTwig\Twig\Environment

However, if this new twig has a `php-mod` file then the name of the top
level namespace must be put there, and all the other files do not have to
call out that namespace at their top. This allows the alias to plug in more
seamlessly allowing for this use statement

  use NewTwig\Environment

Again, running two versions of the same package is not ideal, but sometimes
it's unavoidable. Especially in large ecosystems like WordPress.

Imported packages run on their own request thread (I think that's the best
way - I'm sure the guys working on the egine know best). They don't see or
affect anything outside themselves. They don't see global variables or even
the superglobals. Namespace resolution for them is - touchy.

We could do it like this: A symbol starting without a \ resolves locally,
then goes up the chain till it hits the application root. This is most
similar to the current namespace system, but it allows for hidden
dependencies and frankly, that worries me.

I'm more inclined to isolate package so that if they want to use something
they need to import it for themselves even if the hosting application
already has the lib. This means namespace resolution in a package stops at
the root of the package. We end up with something like this

  namespace MyExtension
  import "twig/twig ^3.0"
  use \Twig\Environment

The package manager should be able to provide this without needing to
download the package twice. Note also in this import that we specify we
want version 3 which we've tested our Extension with.  If that matches the
App, great!  Only one copy of the code needed. If it doesn't, it's
suboptimal but it will run.

As mentioned above, import also needs to be able to load extensions.
Currently extensions are locked into PHP - they can't advance at different
rates. This has led to several extension improvements getting abandoned
over BC concerns. However, this would be nice

  import "ext://mysql ^2.0"

Architecturally this would allow the PHP suite to move to a more mono-repo
style with the various extensions having defaults for the current PHP
distribution and newer versions available. It would also stop a repeat of
having to have both mysql and mysqli functions loaded at the same time.

This is enough to chew on for now.  I'll take notes from the conversation
that follows and iterate again.

--00000000000055b45f061c239fcd
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">First off, in 10 years of using gmail I&#39;ve never had i=
t lose an email. Well, it happened after I spent 4 hours on this.=C2=A0 So,=
 this is sorta iteration 4.=C2=A0 I&#39;ll type this up in Visual Studio co=
de and then paste to gmail.<br><br>The Wordpress discussion about composer =
and the decision not to use it keys in the features a package management sy=
stem must meet. These are<br><br>* Command Line should not be required<br>*=
 Plugins using version X of a package should not be affected by plugins usi=
ng version Y<br>* Install and maintenance of the site should remain possibl=
e from the browser alone.<br>* Backwards compatibility must be maintained.<=
br><br>A core PHP package management system should be a revision of what=
9;s come before that doesn&#39;t disturb what has come before. PHP packages=
 are as follows:<br><br>* Extensions, written in C++<br>* PECL Libraries, w=
hich I believe can be in PHP or C, but are server-wide<br>* Phar archives<b=
r>* Composer libraries, which are always in PHP and leverage namespaces, PS=
R-4 convention and the autoload system.<br><br>Some of my prior discussions=
 have been about what I&#39;d like to see in a new module system, but for s=
cope, clarity, and sanity reasons I&#39;m going to set that aside. (Not to =
mention some of the more controversial pieces have been there)<br><br>Withi=
n the package management system there is a single new keyword - import. I&#=
39;ll go into details on it in a bit but suffice to say its behavior is dif=
ferent from include/require.<br><br>In this schema a PHP application is a c=
ollection=C2=A0of packages.=C2=A0 Existing PHP code that does not use the i=
mport statement or the special directories mentioned below will not invoke =
anything discussed on this page and will not need to change in any way.<br>=
<br><br>Applications<br><br>The application is the root package. It is the =
package that imports to the root namespace. When PHP is asked to parse a fi=
le it will look for a `.php-packages` folder, first in the current working =
directory then in parent directories.=C2=A0 If it doesn&#39;t find one, bus=
iness as usual.=C2=A0 If we do find one we follow its directives about sett=
ing up an application environment.<br><br>The `.php-packages` folder is whe=
re PHP will put package related code for the application at hand.=C2=A0 Cod=
e written explicitly for these changes will also put their package related =
files there - composer&#39;s vendor directory, composer.json, composer.lock=
, and so on - rather than putting those files in the site root. The folder =
is hidden to prevent web servers like nginx or apache from serving the file=
s directly in any way.<br><br>The .php-packages directory will have a confi=
guration file called `php.mod`.=C2=A0 This tells the parser:<br><br>* How t=
o initialize the application<br>* What autoloaders are to be used<br>* How =
to resolve import<br><br>Let&#39;s look at what such a file might look like=
 for Drupal. For the moment I&#39;m going to use go.mod&#39;s syntax. The f=
inal syntax to be used, be it ini, yaml, toml, json, is a discussion for an=
other time. The part to focus in on here is what type of information do we =
need.<br><br><font face=3D"monospace">=C2=A0 package Drupal<br><br>=C2=A0 p=
hp 10<br><br>=C2=A0 registry //<a href=3D"http://packagist.org/packages">pa=
ckagist.org/packages</a> composer<br><br>=C2=A0 init (<br>=C2=A0 =C2=A0 com=
poser install<br>=C2=A0 )<br><br>=C2=A0 require (<br>=C2=A0 =C2=A0 ./vendor=
/autoload.php<br>=C2=A0 )<br><br>=C2=A0 imports (<br>=C2=A0 =C2=A0 //<a hre=
f=3D"http://getcomposer.org/composer.phar">getcomposer.org/composer.phar</a=
><br>=C2=A0 )<br></font><br>The directives do the following:<br>* package s=
ets the name of the package if it is imported. Sort of irrelevant here, but=
 present for consistency<br>* registry sets up the default registry for the=
 import statement and the loader used to install packages<br>* init is the =
command(s) to run before the application is started for the first time.<br>=
* require is the file(s) to require before running any file in the applicat=
ion<br>* imports are the imports the package needs, in this case the compos=
er.phar to load composer in locally.<br><br>Now, given the popularity of co=
mposer some of these directives likely could be put in as logical defines, =
particularly the init, require and imports directives just being put in pla=
ce if composer is selected as a loader.=C2=A0 I don&#39;t rule out the poss=
ibility of a competitor to composer showing up one day though, as yarn was =
introduced as an alternative to npm.<br><br>If this theoretical version of =
Drupal moves its composer.json and composer.lock files into `.php-packages`=
 then the autoloader doesn&#39;t have to be required in the index.php file.=
=C2=A0 Also, the application can be started without running `composer insta=
ll`<br><br><br>Import statement<br>Given the Drupal application package in =
the previous section, we can have an extension file call out its dependenci=
es in code rather than in config.<br><br><font face=3D"monospace">=C2=A0 im=
port &quot;twig/twig&quot;<br></font><br>But granted, there&#39;s not much =
point to this since Drupal already uses Twig.=C2=A0 But suppose for the sak=
e of argument that Symfony releases a Twig 4 with some major BC breaks, and=
 we&#39;re working on a Drupal extension that wants to use the new features=
 of that Twig before Drupal itself upgrades Twig in core. Well, now we can =
do this<br><br><font face=3D"monospace">=C2=A0 import &quot;twig/twig ^4.0&=
quot; as NewTwig</font><br><br>Note that we have to mount this package off =
the root using the alias syntax above.=C2=A0 If the new package isn&#39;t w=
ritten with this system in mind then we&#39;ll end up having NewTwig prefix=
ed on the normal namespace path which would end up looking like this<br><br=
><font face=3D"monospace">=C2=A0 use NewTwig\Twig\Environment</font><br><br=
>However, if this new twig has a `php-mod` file then the name of the top le=
vel namespace must be put there, and all the other files do not have to cal=
l out that namespace at their top. This allows the alias to plug in more se=
amlessly allowing for this use statement<br><br><font face=3D"monospace">=
=C2=A0 use NewTwig\Environment</font><br><br>Again, running two versions of=
 the same package is not ideal, but sometimes it&#39;s unavoidable. Especia=
lly in large ecosystems like WordPress.<br><br>Imported packages run on the=
ir own request thread (I think that&#39;s the best way - I&#39;m sure the g=
uys working on the=C2=A0egine=C2=A0know best). They don&#39;t see or affect=
 anything outside themselves. They don&#39;t see global variables or even t=
he superglobals. Namespace resolution for them is - touchy.<br><br>We could=
 do it like this: A symbol starting without a \ resolves locally, then goes=
 up the chain till it hits the application root. This is most similar to th=
e current namespace system, but it allows for hidden dependencies and frank=
ly, that worries me.<br><br>I&#39;m more inclined to isolate package so tha=
t if they want to use something they need to import it for themselves even =
if the hosting application already has the lib. This means namespace resolu=
tion in a package stops at the root of the package. We end up with somethin=
g like this<br><br><font face=3D"monospace">=C2=A0 namespace MyExtension<br=
>=C2=A0 import &quot;twig/twig ^3.0&quot;<br>=C2=A0 use \Twig\Environment</=
font><br><br>The package manager should be able to provide this without nee=
ding to download the package twice. Note also in this import that we specif=
y we want version 3 which we&#39;ve tested our Extension with.=C2=A0 If tha=
t matches the App, great!=C2=A0 Only one copy of the code needed. If it doe=
sn&#39;t, it&#39;s suboptimal but it will run.<br><br>As mentioned above, i=
mport also needs to be able to load extensions. Currently extensions are lo=
cked into PHP - they can&#39;t advance at different rates. This has led to =
several extension improvements getting abandoned over BC concerns. However,=
 this would be nice<br><br><font face=3D"monospace">=C2=A0 import &quot;ext=
://mysql ^2.0&quot;</font><br><br>Architecturally this would allow the PHP =
suite to move to a more mono-repo style with the various extensions having =
defaults for the current PHP distribution and newer versions available. It =
would also stop a repeat of having to have both mysql and mysqli functions =
loaded at the same time.=C2=A0=C2=A0<br><div><br></div><div>This is enough =
to chew on for now.=C2=A0 I&#39;ll take notes from the conversation that fo=
llows and iterate again.</div></div>

--00000000000055b45f061c239fcd--