Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:124271
Precedence: bulk
MIME-Version: 1.0
References: <09559430-4477-4516-8D78-6F4071E1AA6C@newclarity.net>
 <0182F3D6-F464-477F-9029-A2D0A8B50C71@koalephant.com> <GV1PR01MB10526D0B220ED0AC9D1C17473C2DD2@GV1PR01MB10526.eurprd01.prod.exchangelabs.com>
 <1AFD7AAE-8BEA-460D-88A8-15BB3D30A775@koalephant.com> <CAJp_myXHSHD6+76gyOs=d+cfVgwCmpuhKMtYcVBM6b-i3Bf0WQ@mail.gmail.com>
 <CAEUnE0dObGkb+jUZrVhNRg0JHaM5Kt9d93k3p_0D=jX7kxmYAg@mail.gmail.com>
 <e42cd46b-d98b-4567-a854-03ca54d4dbcf@app.fastmail.com> <CAEUnE0cQA-QrmDmAjpBm=sXkE58Y4ODbNwPYY=QBJV46Q8nR9g@mail.gmail.com>
 <CAEUnE0eEjsBXmz8AEVG1SKE=i08-9dit1ze1K0mWrdyQt4okqQ@mail.gmail.com>
In-Reply-To: <CAEUnE0eEjsBXmz8AEVG1SKE=i08-9dit1ze1K0mWrdyQt4okqQ@mail.gmail.com>
Date: Sun, 7 Jul 2024 19:25:45 -0700
Message-ID: <CAMrTa2EvD3qcTL5SoXwAJ6kT5NTu0QLSQjjJRa0z-esUMVxj6w@mail.gmail.com>
Subject: Re: [PHP-DEV] [PHP-Dev] Versioned Packagers (Iteration IV)
To: Michael Morris <tendoaki@gmail.com>
Cc: PHP internals <internals@lists.php.net>
Content-Type: multipart/alternative; boundary="00000000000027f783061cb322b7"
From: jordan.ledoux@gmail.com (Jordan LeDoux)

--00000000000027f783061cb322b7
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Wed, Jul 3, 2024 at 5:18=E2=80=AFPM Michael Morris <tendoaki@gmail.com> =
wrote:

> Hello all. Hitting reset again as the primary problem at hand has become
> clear.  Let's recap it.
>
> Autoloading is great for loading packages, but it can't load different
> versions of the same package at the same time.  Why would you want to do
> that?
>
> When you don't have full control of the code.
>
> For example, consider Drupal.  It is running Twig at some version of 3 at
> the moment. Suppose Twig 4 is introduced with significant backward
> compatibility breaks (Not saying the authors would do such a thing) but
> also wonderful features.
>
> If you're writing a Drupal extension you might want to use this new Twig.
> This is possible if you are willing to monkey-type the package - that is,
> have a code package traverse over the entire package and change all
> instances of `namespace Twig` in the files to `namespace NewTwig`. You ca=
n
> then use the package at the namespace of \NewTwig.
>
> This is painful, but the pain factor increases if multiple extension
> developers choose to do the same thing.  Each extension using its own Twi=
g
> library is going to incur a performance hit.
>
> One upshot of this is I've noted that major package distributors, like
> Symfony, take BC into account with major releases - and may not develop n=
ew
> features or change things in those releases out of fear of people not
> wanting to upgrade.
>
> Now don't get me wrong, changing things just because is a bad thing. If a
> BC can be avoided it should be. But having a mechanism to move forward is
> important.
>
> In some ways versioning packages is like static typing variables. It
> doesn't seem important at all until you are faced with a problem only it
> can solve, or faced with a problem created by dynamic typing of variables=
.
>
> What can be done in the engine?
>
> Well first off, recognize that autoloading isn't going to work with a
> versioned package scheme. Autoloaders, regardless of their resolution
> schema be it PSR-0, PSR-4, or BobbysFirstAutoloader-Scheme can only have
> one symbol per package, set by the namespace.
>
> Can PHP support multiple packages without rewriting the whole engine?  I
> think so, but it isn't trivial, and the side effects need to be cordoned
> off so that those who need this complexity can have it while the beginnin=
g
> and intermediate coders can ignore it just like they ignore strict
> comparison operators and strict typing unless a library they are trying t=
o
> use foists it on them.
>
> This is why I advocate a new keyword for this - import.  Import's behavio=
r
> is most similar to require_once, but it doesn't have to be the same.  Sin=
ce
> it is a new entrypoint into the engine the way the engine considers the
> code can be different - whether slightly different or radically different
> is a debate for another time. I'm going to stick with only those changes
> that make sense in the context of package links.
>
> Let's start with the simplest problem, importing this file.
>
>   namespace A;
>   function foo() { echo 'Hi'; }
>
> To review, if we require_once this file we'll find the function at
> \A\foo().  If our current file uses the same namespace we can just use fo=
o()
>
> At its root import would do the same. `import "file.php"` would do the
> same as a require_once assuming there's no difference between the file
> structure rules for import - again there is opportunity here, but it's no=
t
> a requirement.
>
> If that's all it does, it's pointless.  However, import can alias.
>
>   import 'file.php' as B;
>
> Now we have \B\foo();  This makes it relatively easy to have two differen=
t
> versions of the package running since in our own code we can always
> reference the foo in the B namespace. But while that allows limited packa=
ge
> versioning, it doesn't solve the multiple extensions wanting to use the n=
ew
> stuff problem outlined above.
>
> So we have to call out the version in code, like so.
>
>   import 'file.php v1.0.0';
>
> A simple space separates the version from the file.  If the filename has =
a
> space, well \ characters aren't just for namespaces.
>
> Now for the first real behavior difference between import and
> require_once, even if we aren't doing anything fancy.  Import cares about
> the namespace it's invoked from.  Require_once does not.  To illustrate
> this behavior he's some pseudocode - we are including the file.php given
> earlier
>
>   namespace D;
>   require_once 'file.php';
>
>   \A\foo(); // Hi.
>
>   import 'file.php';
>
>   \D\A\foo(); // Hi.
>
> See that? The namespace of the calling file is prepended to the namespace
> contained in the import.
>
> Why?  What's the value here?  I'll explain.
>
> Now, let's suppose we do have two versions of file.php. So in addition to
> the above, elsewhere in the code this happens
>
>   namespace C;
>   import 'file.php v2.0.0'
>
>   A\foo(); // Welcome, since version 2 echoes welcome. Remember your
> namespace resolution rules - this import is actually at:
>   \C\A\foo(); Welcome, as this is the absolute path to the code we just
> imported.
>   \A\foo(); // Hi, as the package at root was brought in by require_once(=
)
>   \D\A\foo(); Hi, as that's what was imported into the D namespace.
>
> Now for the kicker
>
>   namespace E;
>   import 'file.php';
>
>   A\foo(); // Hi.
>
> The engine can be left as is and this would work, but if the engine is
> altered to support symbolic links on the symbol table then the performanc=
e
> hit might be avoided.  That is, when a redundant import occurs that would
> pull the same package the engine just quickly links up the new namespace.
> Hence \E\A\foo() quietly points to \D\A\foo() as it was declared first.
>
>
> What hasn't been discussed in this iteration are the following critical
> points:
> 1) How the package path gets resolved in the first place. Does it work
> like require and check locally then check the PHP include paths?
> 2) When does the code get downloaded from where it is downloaded?
> 3) Is a registry used like composer and npm, or are repos directly invoke=
d
> as in go (I don't remember how Python does it, but someone providing that
> example might be useful)
> 4) The huge ball of wax that is the package definition file. Just look at
> the properties of composer.json and package.json to get an idea of that
> scope. How much of if any of this should PHP deal with.
> 5) Is import to be locked into loading other PHP files, or could it deal
> with .so (Unix) or .dll (Windows) files? Phar files?
>
> It's not like I'm not interested in any of these questions, but too many
> questions at once is too much so I'd like to leave them aside for now.
>
> And there are yet more questions as well raised in previous iterations,
> but I've again left those out because they touched off controversy. While
> I'm not afraid of such, I'm inclined to avoid it if possible.
>
> A quick thank you to everyone who has participated in the thread, even th=
e
> torpedo tossers because it's forcing me to think this through entirely. A=
nd
> I'm trying to take as much into consideration as possible.  And yes, this
> remains a brainstorm for now, but each successive brainstorm is more tigh=
t
> than the one before it.
>
>
>

I think it's strange that this discussion has driven deep down the tangent
of versioning, as if the selling point of any kind of module/package system
in PHP core would be to do what composer does. Let compose do what composer
does, it does it well.

Instead of building out the specific features like this that honestly
shouldn't be built into the language directly, I would think it makes more
sense for the discussion to be centered around the compiler and engine
features that would LET or ENABLE software like composer to easily meet
these requirements.

Things like separating global scope between importer and importee, managed
visibility of symbols and exports from modules/packages, allowing for
separate autoloaders for things which are called or included via an import,
etc. Those are the things that the language itself can do.

All this other stuff feels like a distraction.

Jordan

--00000000000027f783061cb322b7
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">=
<div dir=3D"ltr" class=3D"gmail_attr">On Wed, Jul 3, 2024 at 5:18=E2=80=AFP=
M Michael Morris &lt;<a href=3D"mailto:tendoaki@gmail.com">tendoaki@gmail.c=
om</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margi=
n:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex=
"><div dir=3D"ltr">Hello all. Hitting reset again as the primary problem at=
 hand has become clear.=C2=A0 Let&#39;s recap it.<div><br></div><div>Autolo=
ading is great for loading packages, but it can&#39;t load different versio=
ns of the same package at the same time.=C2=A0 Why would you want to do tha=
t?</div><div><br></div><div>When you don&#39;t have full control of the cod=
e.</div><div><br></div><div>For example, consider Drupal.=C2=A0 It is runni=
ng Twig at some version of 3 at the moment. Suppose Twig 4 is introduced wi=
th significant backward compatibility breaks (Not saying the authors would =
do such a thing) but also wonderful features.</div><div><br></div><div>If y=
ou&#39;re writing a Drupal extension you might want to use this new Twig.=
=C2=A0 This is possible if you are willing to monkey-type the package - tha=
t is, have a code package traverse over the entire package and change all i=
nstances of `namespace Twig` in the files to `namespace NewTwig`. You can t=
hen use the package at the namespace of \NewTwig.</div><div><br></div><div>=
This is painful, but the pain factor increases if multiple extension develo=
pers choose to do the same thing.=C2=A0 Each extension using its own Twig l=
ibrary is going to incur a performance hit.</div><div><br></div><div>One up=
shot of this is I&#39;ve noted that major package distributors, like Symfon=
y, take BC into account with major releases - and may not develop new featu=
res or change things in those releases out of fear of people not wanting to=
 upgrade.</div><div><br></div><div>Now don&#39;t get me wrong, changing thi=
ngs just because is a bad thing. If a BC can be avoided it should be. But h=
aving a mechanism to move forward is important.</div><div><br></div><div>In=
 some ways versioning packages is like static typing variables. It doesn=
9;t seem important at all until you are faced with a problem only it can so=
lve, or faced with a problem created by dynamic typing of variables.</div><=
div><br></div><div>What can be done in the engine?</div><div><br></div><div=
>Well first off, recognize that autoloading isn&#39;t going to work with a =
versioned package scheme. Autoloaders, regardless of their resolution schem=
a be it PSR-0, PSR-4, or BobbysFirstAutoloader-Scheme=C2=A0can only have on=
e symbol per package, set by the namespace.=C2=A0</div><div><br></div><div>=
Can PHP support multiple packages without rewriting the whole engine?=C2=A0=
 I think so, but it isn&#39;t trivial, and the side effects need to be cord=
oned off so that those who need this complexity can have it while the begin=
ning and intermediate coders can ignore it just like they ignore strict com=
parison operators and strict typing unless a library they are trying to use=
 foists it on them.</div><div><br></div><div>This is why I advocate a new k=
eyword for this - import.=C2=A0 Import&#39;s behavior is most similar to re=
quire_once, but it doesn&#39;t have to be the same.=C2=A0 Since it is a new=
 entrypoint into the engine the way the engine considers the code can be di=
fferent - whether slightly different or radically different is a debate for=
 another time. I&#39;m going to stick with only those changes that make sen=
se in the context of package links.</div><div><br></div><div>Let&#39;s star=
t with the simplest problem, importing this file.</div><div><br></div><div>=
=C2=A0 namespace A;</div><div>=C2=A0 function foo() { echo &#39;Hi&#39;; }<=
/div><div><br></div><div>To review, if we require_once this file we&#39;ll =
find the function at \A\foo().=C2=A0 If our current file uses the same name=
space we can just use foo()</div><div><br></div><div>At its root import wou=
ld do the same. `import &quot;file.php&quot;` would do the same as a requir=
e_once assuming there&#39;s no difference between the file structure rules =
for import - again there is opportunity here, but it&#39;s not a requiremen=
t.</div><div><br></div><div>If that&#39;s all it does, it&#39;s pointless.=
=C2=A0 However, import can alias.</div><div><br></div><div>=C2=A0 import &#=
39;file.php&#39; as B;</div><div><br></div><div>Now we have \B\foo();=C2=A0=
 This makes it relatively easy to have two different versions of the packag=
e running since in our own code we can always reference the foo in the B na=
mespace. But while that allows limited package versioning, it doesn&#39;t s=
olve the multiple extensions wanting to use the new stuff problem outlined =
above.</div><div><br></div><div>So we have to call out the version in code,=
 like so.</div><div><br></div><div>=C2=A0 import &#39;file.php v1.0.0&#39;;=
</div><div><br></div><div>A simple space separates the version from the fil=
e.=C2=A0 If the filename has a space, well \ characters aren&#39;t just for=
 namespaces.</div><div><br></div><div>Now for the first real behavior diffe=
rence between import and require_once, even if we aren&#39;t doing anything=
 fancy.=C2=A0 Import cares about the namespace it&#39;s invoked from.=C2=A0=
 Require_once does not.=C2=A0 To illustrate this behavior he&#39;s some pse=
udocode=C2=A0- we are including the file.php given earlier</div><div><br></=
div><div>=C2=A0 namespace D;</div><div>=C2=A0 require_once &#39;file.php=
9;;</div><div><br></div><div>=C2=A0 \A\foo(); // Hi.</div><div><br></div><d=
iv>=C2=A0 import &#39;file.php&#39;;</div><div><br></div><div>=C2=A0 \D\A\f=
oo(); // Hi.</div><div><br></div><div>See that? The namespace of the callin=
g file is prepended to the namespace contained in the import.</div><div><br=
></div><div>Why?=C2=A0 What&#39;s the value here?=C2=A0 I&#39;ll explain.</=
div><div><br></div><div>Now, let&#39;s suppose we do have two versions of f=
ile.php. So in addition to the above, elsewhere in the code this happens</d=
iv><div><br></div><div>=C2=A0 namespace C;</div><div>=C2=A0 import &#39;fil=
e.php v2.0.0&#39;</div><div><br></div><div>=C2=A0 A\foo(); // Welcome, sinc=
e version 2 echoes welcome. Remember your namespace resolution rules - this=
 import is actually at:<br>=C2=A0 \C\A\foo(); Welcome, as this is the absol=
ute path to the code we just imported.<br>=C2=A0 \A\foo(); // Hi, as the pa=
ckage at root was brought in by require_once()</div><div>=C2=A0 \D\A\foo();=
 Hi, as that&#39;s what was imported into the D namespace.</div><div><br></=
div><div>Now for the kicker</div><div><br></div><div>=C2=A0 namespace E;</d=
iv><div>=C2=A0 import &#39;file.php&#39;;</div><div><br></div><div>=C2=A0 A=
\foo(); // Hi.</div><div><br></div><div>The engine can be left as is and th=
is would work, but if the engine is altered to support symbolic links on th=
e symbol table then the performance hit might be avoided.=C2=A0 That is, wh=
en a redundant import occurs that would pull the same package the engine ju=
st quickly links up the new namespace.=C2=A0 Hence \E\A\foo() quietly point=
s to \D\A\foo() as it was declared first.</div><div><br></div><div><br></di=
v><div>What hasn&#39;t been discussed in this iteration are the following c=
ritical points:<br>1) How the package path gets resolved in the first place=
. Does it work like require and check locally then check the PHP include pa=
ths?<br>2) When does the code get downloaded from where it is downloaded? <=
br>3) Is a registry used like composer and npm, or are repos directly invok=
ed as in go (I don&#39;t remember how Python does it, but someone providing=
 that example might be useful)</div><div>4) The huge ball of wax that is th=
e package definition file. Just look at the properties of composer.json and=
 package.json to get an idea of that scope. How much of if any of this shou=
ld PHP deal with.<br>5) Is import to be locked into loading other PHP files=
, or could it deal with .so (Unix) or .dll (Windows) files? Phar files?<br>=
<br></div><div>It&#39;s not like I&#39;m not interested in any of these que=
stions, but too many questions at once is too much so I&#39;d like to leave=
 them aside for now.</div><div><br></div><div>And there are yet more questi=
ons as well raised in previous iterations, but I&#39;ve again left those ou=
t because they touched off controversy. While I&#39;m not afraid of such, I=
&#39;m inclined to avoid it if possible.<br><br>A quick thank you to everyo=
ne who has participated in the thread, even the torpedo tossers because it&=
#39;s forcing me to think this through entirely. And I&#39;m trying to take=
 as much into consideration as possible.=C2=A0 And yes, this remains a brai=
nstorm for now, but each successive brainstorm is more tight than the one b=
efore it.<br><br>=C2=A0</div></div></blockquote><div><br></div><div>I think=
 it&#39;s strange that this discussion has driven deep down the tangent of =
versioning, as if the selling point of any kind of module/package system in=
 PHP core would be to do what composer does. Let compose do what composer d=
oes, it does it well.</div><div><br></div><div>Instead of building out the =
specific features like this that honestly shouldn&#39;t be built into the l=
anguage directly, I would think it makes more sense for the discussion to b=
e centered around the compiler and engine features that would LET or ENABLE=
 software like composer to easily meet these requirements.</div><div><br></=
div><div>Things like separating global scope between importer and importee,=
 managed visibility of symbols and exports from modules/packages, allowing =
for separate autoloaders for things which are called or included via an imp=
ort, etc. Those are the things that the language itself can do.=C2=A0</div>=
<div><br></div><div>All this other stuff feels like a distraction.</div><di=
v><br></div><div>Jordan <br></div></div></div>

--00000000000027f783061cb322b7--