Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:124207
Precedence: bulk
MIME-Version: 1.0
References: <09559430-4477-4516-8D78-6F4071E1AA6C@newclarity.net>
 <0182F3D6-F464-477F-9029-A2D0A8B50C71@koalephant.com> <GV1PR01MB10526D0B220ED0AC9D1C17473C2DD2@GV1PR01MB10526.eurprd01.prod.exchangelabs.com>
 <1AFD7AAE-8BEA-460D-88A8-15BB3D30A775@koalephant.com> <CAJp_myXHSHD6+76gyOs=d+cfVgwCmpuhKMtYcVBM6b-i3Bf0WQ@mail.gmail.com>
 <CAEUnE0dObGkb+jUZrVhNRg0JHaM5Kt9d93k3p_0D=jX7kxmYAg@mail.gmail.com>
 <e42cd46b-d98b-4567-a854-03ca54d4dbcf@app.fastmail.com> <CAEUnE0cQA-QrmDmAjpBm=sXkE58Y4ODbNwPYY=QBJV46Q8nR9g@mail.gmail.com>
In-Reply-To: <CAEUnE0cQA-QrmDmAjpBm=sXkE58Y4ODbNwPYY=QBJV46Q8nR9g@mail.gmail.com>
Date: Wed, 3 Jul 2024 20:16:15 -0400
Message-ID: <CAEUnE0eEjsBXmz8AEVG1SKE=i08-9dit1ze1K0mWrdyQt4okqQ@mail.gmail.com>
Subject: [PHP-DEV] [PHP-Dev] Versioned Packagers (Iteration IV)
To: PHP internals <internals@lists.php.net>
Content-Type: multipart/alternative; boundary="0000000000009ed84e061c60db52"
From: tendoaki@gmail.com (Michael Morris)

--0000000000009ed84e061c60db52
Content-Type: text/plain; charset="UTF-8"

Hello all. Hitting reset again as the primary problem at hand has become
clear.  Let's recap it.

Autoloading is great for loading packages, but it can't load different
versions of the same package at the same time.  Why would you want to do
that?

When you don't have full control of the code.

For example, consider Drupal.  It is running Twig at some version of 3 at
the moment. Suppose Twig 4 is introduced with significant backward
compatibility breaks (Not saying the authors would do such a thing) but
also wonderful features.

If you're writing a Drupal extension you might want to use this new Twig.
This is possible if you are willing to monkey-type the package - that is,
have a code package traverse over the entire package and change all
instances of `namespace Twig` in the files to `namespace NewTwig`. You can
then use the package at the namespace of \NewTwig.

This is painful, but the pain factor increases if multiple extension
developers choose to do the same thing.  Each extension using its own Twig
library is going to incur a performance hit.

One upshot of this is I've noted that major package distributors, like
Symfony, take BC into account with major releases - and may not develop new
features or change things in those releases out of fear of people not
wanting to upgrade.

Now don't get me wrong, changing things just because is a bad thing. If a
BC can be avoided it should be. But having a mechanism to move forward is
important.

In some ways versioning packages is like static typing variables. It
doesn't seem important at all until you are faced with a problem only it
can solve, or faced with a problem created by dynamic typing of variables.

What can be done in the engine?

Well first off, recognize that autoloading isn't going to work with a
versioned package scheme. Autoloaders, regardless of their resolution
schema be it PSR-0, PSR-4, or BobbysFirstAutoloader-Scheme can only have
one symbol per package, set by the namespace.

Can PHP support multiple packages without rewriting the whole engine?  I
think so, but it isn't trivial, and the side effects need to be cordoned
off so that those who need this complexity can have it while the beginning
and intermediate coders can ignore it just like they ignore strict
comparison operators and strict typing unless a library they are trying to
use foists it on them.

This is why I advocate a new keyword for this - import.  Import's behavior
is most similar to require_once, but it doesn't have to be the same.  Since
it is a new entrypoint into the engine the way the engine considers the
code can be different - whether slightly different or radically different
is a debate for another time. I'm going to stick with only those changes
that make sense in the context of package links.

Let's start with the simplest problem, importing this file.

  namespace A;
  function foo() { echo 'Hi'; }

To review, if we require_once this file we'll find the function at
\A\foo().  If our current file uses the same namespace we can just use foo()

At its root import would do the same. `import "file.php"` would do the same
as a require_once assuming there's no difference between the file structure
rules for import - again there is opportunity here, but it's not a
requirement.

If that's all it does, it's pointless.  However, import can alias.

  import 'file.php' as B;

Now we have \B\foo();  This makes it relatively easy to have two different
versions of the package running since in our own code we can always
reference the foo in the B namespace. But while that allows limited package
versioning, it doesn't solve the multiple extensions wanting to use the new
stuff problem outlined above.

So we have to call out the version in code, like so.

  import 'file.php v1.0.0';

A simple space separates the version from the file.  If the filename has a
space, well \ characters aren't just for namespaces.

Now for the first real behavior difference between import and require_once,
even if we aren't doing anything fancy.  Import cares about the namespace
it's invoked from.  Require_once does not.  To illustrate this behavior
he's some pseudocode - we are including the file.php given earlier

  namespace D;
  require_once 'file.php';

  \A\foo(); // Hi.

  import 'file.php';

  \D\A\foo(); // Hi.

See that? The namespace of the calling file is prepended to the namespace
contained in the import.

Why?  What's the value here?  I'll explain.

Now, let's suppose we do have two versions of file.php. So in addition to
the above, elsewhere in the code this happens

  namespace C;
  import 'file.php v2.0.0'

  A\foo(); // Welcome, since version 2 echoes welcome. Remember your
namespace resolution rules - this import is actually at:
  \C\A\foo(); Welcome, as this is the absolute path to the code we just
imported.
  \A\foo(); // Hi, as the package at root was brought in by require_once()
  \D\A\foo(); Hi, as that's what was imported into the D namespace.

Now for the kicker

  namespace E;
  import 'file.php';

  A\foo(); // Hi.

The engine can be left as is and this would work, but if the engine is
altered to support symbolic links on the symbol table then the performance
hit might be avoided.  That is, when a redundant import occurs that would
pull the same package the engine just quickly links up the new namespace.
Hence \E\A\foo() quietly points to \D\A\foo() as it was declared first.


What hasn't been discussed in this iteration are the following critical
points:
1) How the package path gets resolved in the first place. Does it work like
require and check locally then check the PHP include paths?
2) When does the code get downloaded from where it is downloaded?
3) Is a registry used like composer and npm, or are repos directly invoked
as in go (I don't remember how Python does it, but someone providing that
example might be useful)
4) The huge ball of wax that is the package definition file. Just look at
the properties of composer.json and package.json to get an idea of that
scope. How much of if any of this should PHP deal with.
5) Is import to be locked into loading other PHP files, or could it deal
with .so (Unix) or .dll (Windows) files? Phar files?

It's not like I'm not interested in any of these questions, but too many
questions at once is too much so I'd like to leave them aside for now.

And there are yet more questions as well raised in previous iterations, but
I've again left those out because they touched off controversy. While I'm
not afraid of such, I'm inclined to avoid it if possible.

A quick thank you to everyone who has participated in the thread, even the
torpedo tossers because it's forcing me to think this through entirely. And
I'm trying to take as much into consideration as possible.  And yes, this
remains a brainstorm for now, but each successive brainstorm is more tight
than the one before it.

--0000000000009ed84e061c60db52
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hello all. Hitting reset again as the primary problem at h=
and has become clear.=C2=A0 Let&#39;s recap it.<div><br></div><div>Autoload=
ing is great for loading packages, but it can&#39;t load different versions=
 of the same package at the same time.=C2=A0 Why would you want to do that?=
</div><div><br></div><div>When you don&#39;t have full control of the code.=
</div><div><br></div><div>For example, consider Drupal.=C2=A0 It is running=
 Twig at some version of 3 at the moment. Suppose Twig 4 is introduced with=
 significant backward compatibility breaks (Not saying the authors would do=
 such a thing) but also wonderful features.</div><div><br></div><div>If you=
&#39;re writing a Drupal extension you might want to use this new Twig.=C2=
=A0 This is possible if you are willing to monkey-type the package - that i=
s, have a code package traverse over the entire package and change all inst=
ances of `namespace Twig` in the files to `namespace NewTwig`. You can then=
 use the package at the namespace of \NewTwig.</div><div><br></div><div>Thi=
s is painful, but the pain factor increases if multiple extension developer=
s choose to do the same thing.=C2=A0 Each extension using its own Twig libr=
ary is going to incur a performance hit.</div><div><br></div><div>One upsho=
t of this is I&#39;ve noted that major package distributors, like Symfony, =
take BC into account with major releases - and may not develop new features=
 or change things in those releases out of fear of people not wanting to up=
grade.</div><div><br></div><div>Now don&#39;t get me wrong, changing things=
 just because is a bad thing. If a BC can be avoided it should be. But havi=
ng a mechanism to move forward is important.</div><div><br></div><div>In so=
me ways versioning packages is like static typing variables. It doesn&#39;t=
 seem important at all until you are faced with a problem only it can solve=
, or faced with a problem created by dynamic typing of variables.</div><div=
><br></div><div>What can be done in the engine?</div><div><br></div><div>We=
ll first off, recognize that autoloading isn&#39;t going to work with a ver=
sioned package scheme. Autoloaders, regardless of their resolution schema b=
e it PSR-0, PSR-4, or BobbysFirstAutoloader-Scheme=C2=A0can only have one s=
ymbol per package, set by the namespace.=C2=A0</div><div><br></div><div>Can=
 PHP support multiple packages without rewriting the whole engine?=C2=A0 I =
think so, but it isn&#39;t trivial, and the side effects need to be cordone=
d off so that those who need this complexity can have it while the beginnin=
g and intermediate coders can ignore it just like they ignore strict compar=
ison operators and strict typing unless a library they are trying to use fo=
ists it on them.</div><div><br></div><div>This is why I advocate a new keyw=
ord for this - import.=C2=A0 Import&#39;s behavior is most similar to requi=
re_once, but it doesn&#39;t have to be the same.=C2=A0 Since it is a new en=
trypoint into the engine the way the engine considers the code can be diffe=
rent - whether slightly different or radically different is a debate for an=
other time. I&#39;m going to stick with only those changes that make sense =
in the context of package links.</div><div><br></div><div>Let&#39;s start w=
ith the simplest problem, importing this file.</div><div><br></div><div>=C2=
=A0 namespace A;</div><div>=C2=A0 function foo() { echo &#39;Hi&#39;; }</di=
v><div><br></div><div>To review, if we require_once this file we&#39;ll fin=
d the function at \A\foo().=C2=A0 If our current file uses the same namespa=
ce we can just use foo()</div><div><br></div><div>At its root import would =
do the same. `import &quot;file.php&quot;` would do the same as a require_o=
nce assuming there&#39;s no difference between the file structure rules for=
 import - again there is opportunity here, but it&#39;s not a requirement.<=
/div><div><br></div><div>If that&#39;s all it does, it&#39;s pointless.=C2=
=A0 However, import can alias.</div><div><br></div><div>=C2=A0 import &#39;=
file.php&#39; as B;</div><div><br></div><div>Now we have \B\foo();=C2=A0 Th=
is makes it relatively easy to have two different versions of the package r=
unning since in our own code we can always reference the foo in the B names=
pace. But while that allows limited package versioning, it doesn&#39;t solv=
e the multiple extensions wanting to use the new stuff problem outlined abo=
ve.</div><div><br></div><div>So we have to call out the version in code, li=
ke so.</div><div><br></div><div>=C2=A0 import &#39;file.php v1.0.0&#39;;</d=
iv><div><br></div><div>A simple space separates the version from the file.=
=C2=A0 If the filename has a space, well \ characters aren&#39;t just for n=
amespaces.</div><div><br></div><div>Now for the first real behavior differe=
nce between import and require_once, even if we aren&#39;t doing anything f=
ancy.=C2=A0 Import cares about the namespace it&#39;s invoked from.=C2=A0 R=
equire_once does not.=C2=A0 To illustrate this behavior he&#39;s some pseud=
ocode=C2=A0- we are including the file.php given earlier</div><div><br></di=
v><div>=C2=A0 namespace D;</div><div>=C2=A0 require_once &#39;file.php&#39;=
;</div><div><br></div><div>=C2=A0 \A\foo(); // Hi.</div><div><br></div><div=
>=C2=A0 import &#39;file.php&#39;;</div><div><br></div><div>=C2=A0 \D\A\foo=
(); // Hi.</div><div><br></div><div>See that? The namespace of the calling =
file is prepended to the namespace contained in the import.</div><div><br><=
/div><div>Why?=C2=A0 What&#39;s the value here?=C2=A0 I&#39;ll explain.</di=
v><div><br></div><div>Now, let&#39;s suppose we do have two versions of fil=
e.php. So in addition to the above, elsewhere in the code this happens</div=
><div><br></div><div>=C2=A0 namespace C;</div><div>=C2=A0 import &#39;file.=
php v2.0.0&#39;</div><div><br></div><div>=C2=A0 A\foo(); // Welcome, since =
version 2 echoes welcome. Remember your namespace resolution rules - this i=
mport is actually at:<br>=C2=A0 \C\A\foo(); Welcome, as this is the absolut=
e path to the code we just imported.<br>=C2=A0 \A\foo(); // Hi, as the pack=
age at root was brought in by require_once()</div><div>=C2=A0 \D\A\foo(); H=
i, as that&#39;s what was imported into the D namespace.</div><div><br></di=
v><div>Now for the kicker</div><div><br></div><div>=C2=A0 namespace E;</div=
><div>=C2=A0 import &#39;file.php&#39;;</div><div><br></div><div>=C2=A0 A\f=
oo(); // Hi.</div><div><br></div><div>The engine can be left as is and this=
 would work, but if the engine is altered to support symbolic links on the =
symbol table then the performance hit might be avoided.=C2=A0 That is, when=
 a redundant import occurs that would pull the same package the engine just=
 quickly links up the new namespace.=C2=A0 Hence \E\A\foo() quietly points =
to \D\A\foo() as it was declared first.</div><div><br></div><div><br></div>=
<div>What hasn&#39;t been discussed in this iteration are the following cri=
tical points:<br>1) How the package path gets resolved in the first place. =
Does it work like require and check locally then check the PHP include path=
s?<br>2) When does the code get downloaded from where it is downloaded? <br=
>3) Is a registry used like composer and npm, or are repos directly invoked=
 as in go (I don&#39;t remember how Python does it, but someone providing t=
hat example might be useful)</div><div>4) The huge ball of wax that is the =
package definition file. Just look at the properties of composer.json and p=
ackage.json to get an idea of that scope. How much of if any of this should=
 PHP deal with.<br>5) Is import to be locked into loading other PHP files, =
or could it deal with .so (Unix) or .dll (Windows) files? Phar files?<br><b=
r></div><div>It&#39;s not like I&#39;m not interested in any of these quest=
ions, but too many questions at once is too much so I&#39;d like to leave t=
hem aside for now.</div><div><br></div><div>And there are yet more question=
s as well raised in previous iterations, but I&#39;ve again left those out =
because they touched off controversy. While I&#39;m not afraid of such, I&#=
39;m inclined to avoid it if possible.<br><br>A quick thank you to everyone=
 who has participated in the thread, even the torpedo tossers because it=
9;s forcing me to think this through entirely. And I&#39;m trying to take a=
s much into consideration as possible.=C2=A0 And yes, this remains a brains=
torm for now, but each successive brainstorm is more tight than the one bef=
ore it.<br><br>=C2=A0</div></div>

--0000000000009ed84e061c60db52--