Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:124331
Content-Type: text/plain;
	charset=utf-8
Precedence: bulk
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.8\))
Subject: Re: [PHP-DEV] [PHP-Dev] Versioned Packagers (Iteration IV)
In-Reply-To: <7b40e925-d642-4cf4-83f8-f903a9964362@app.fastmail.com>
Date: Wed, 10 Jul 2024 02:38:13 -0400
Cc: php internals <internals@lists.php.net>
Content-Transfer-Encoding: quoted-printable
Message-ID: <EA1B29C9-2EDB-4449-8BC1-5ED09D29A336@newclarity.net>
References: <09559430-4477-4516-8D78-6F4071E1AA6C@newclarity.net>
 <0182F3D6-F464-477F-9029-A2D0A8B50C71@koalephant.com>
 <GV1PR01MB10526D0B220ED0AC9D1C17473C2DD2@GV1PR01MB10526.eurprd01.prod.exchangelabs.com>
 <1AFD7AAE-8BEA-460D-88A8-15BB3D30A775@koalephant.com>
 <CAJp_myXHSHD6+76gyOs=d+cfVgwCmpuhKMtYcVBM6b-i3Bf0WQ@mail.gmail.com>
 <CAEUnE0dObGkb+jUZrVhNRg0JHaM5Kt9d93k3p_0D=jX7kxmYAg@mail.gmail.com>
 <e42cd46b-d98b-4567-a854-03ca54d4dbcf@app.fastmail.com>
 <CAEUnE0cQA-QrmDmAjpBm=sXkE58Y4ODbNwPYY=QBJV46Q8nR9g@mail.gmail.com>
 <CAEUnE0eEjsBXmz8AEVG1SKE=i08-9dit1ze1K0mWrdyQt4okqQ@mail.gmail.com>
 <7B633CC7-C768-4852-A4D0-B252A04F7DE1@newclarity.net>
 <CAEUnE0fFfggyvtNGMytWATG7xUrwfjtXp5F0Sk9PvXo-1Rf+_g@mail.gmail.com>
 <0E11F373-99DC-496E-9BBC-2F8688B9F66A@newclarity.net>
 <CAEUnE0fMGzYimJr71VSFr44FQPBzszfd7FbScfwYx794GZMsrQ@mail.gmail.com>
 <4F720A7A-B7DD-4B31-B0C9-6907419B53A5@newclarity.net>
 <CAEUnE0cf2ekFTsQphH8GjzZ4Yds2DXwxS_DT37UZ4reMHg=D+Q@mail.gmail.com>
 <D412D0D3-AD94-4B06-A302-AC7215271367@newclarity.net>
 <7b40e925-d642-4cf4-83f8-f903a9964362@app.fastmail.com>
To: Larry Garfield <larry@garfieldtech.com>
From: mike@newclarity.net (Mike Schinkel)

> On Jul 9, 2024, at 1:15 PM, Larry Garfield <larry@garfieldtech.com> =
wrote:
> On Sat, Jul 6, 2024, at 1:12 AM, Mike Schinkel wrote:
>> WordPress differs from the apps many *(most?)* developers on PHP=20
>> Internals work with in the following way:
>>=20
>> WordPress =3D *User-managed app*
>> Most =3D *Developer-managed apps*
>>=20
>> In a* Developer-Managed app* developers choose which 3rd party=20
>> functionality will be incorporated into their sites whereas with a=20
>> *User-managed app* users choose which 3rd party functionality will be=20=

>> incorporated into their site. And that is the KEY difference.=20
>>=20
>> So I am wondering if we can get people on this PHP Internals list who=20=

>> dismiss the needs of WordPress developer BECAUSE it is WordPress to=20=

>> recognize that User-Managed apps ARE a class of PHP applications have=20=

>> needs that **deserve** to be addressed? =20
>> *
>> *
>> Two (2)* unmet needs of User-Managed apps *that *"standard" *PHP=20
>> currently does not address come to mind:
>>=20
>> User-managed apps needs to be able to handle both:
>> *
>> *
>> 1. *User-added add-ons* *("plugins" in WordPress, "modules" in =
Drupal)=20
>> *that have conflicting dependencies, and
>>=20
>> 2. *Add-on directory structures *that do not follow a PSR-4 directory =
hierarchy.
>>=20
>> As for #2, even if those apps could rearchitect their existing=20
>> directory structure they cannot realistically be expected to do with=20=

>> because of the huge BC issues their users would experience.=20
>>=20
>> And newly created User-managed apps may still find that a PSR-4=20
>> directory structure is not in the best interest of their project or=20=

>> their users. To elaborate, PSR-4 generally assumes that ALL code goes=20=

>> into ONE hierarchy and that any and all code that will be autoload =
gets=20
>> placed in that hierarchy.
>>=20
>> But with add-ons it makes a lot more sense to have the entire add-on=20=

>> contained in its own add-on directory. This is exactly where PSR-4=20
>> breaks down with respect to User-managed apps.
>>=20
>> Sure, you can have multiple PSR-4 autoloader root directories, but =
that=20
>> does not scale well to websites with a large number of add-ons as =
many=20
>> WordPress sites I worked on used. Some had over 100 plugins.  With a=20=

>> hierarchy of autoloader maps that Michael Morris is proposing =
WordPress=20
>> could collect up all the maps and create one map every time a plugin =
is=20
>> added, updated or deleted.
>>=20
>> </epiphany>
>=20
> I am going to jump in here on this point specifically, because it =
seems to be a mix of genuinely insightful observation (though not =
unique) and uninformed FUD.
>=20
> Some context: I haven't seriously used Wordpress in, ever.  However, I =
was a Drupal lead developer for many years, and wrote, among other =
things, Drupal's DBAL, Drupal's first autoloader, Drupal's PSR-3 =
implementation, was involved in Drupal's file organization guidelines =
for Drupal 8+ (when Drupal adopted a PSR-0/4 autoloader), and led the =
Drupal 8 "Modernize all the things" effort.  So I do have some =
non-trivial experience in this area.

Yes. I remember that about you as well, Crell.

> First, you're correct that there is an architectural difference =
between "projects that assume the owner has CLI access" and those that =
do not.  You are also correct that most of the Internals crowd comes =
from the former. =20
>=20
> However, I don't think it's fair to say that's why Internals folks =
"dismiss" Wordpress generally. =20

No, I said you dismiss the concerns of user-managed apps *because* you =
dismiss WordPress, which we know because you just gave a list of why you =
dismiss WordPress.

What I was hoping was to make the distinction that user-managed apps =E2=80=
=94 regardless of if they are WordPress, are some other user-managed app =
=E2=80=94 have needs that are unfair to demonize simply because you have =
contempt for the poster-child of user-managed apps.

> And yes, I fully agree that any module/package/thing needs to take =
into account the needs of both types of projects.  As Rowan has =
repeated, that means keeping the impact of any changes minimal, so that =
the Composer ecosystem and WP ecosystem can TYPO3 ecosystem can build =
their own tooling on top of it.
>=20
> You'll note I did not list Drupal there.  That's because modern Drupal =
is composer-based, and has been for many years.  I was the one that =
pushed hard for adopting Composer, its autoloader, and PSR-4 for Drupal =
8 in the first place.  While much of the transition happened after I =
left the project, the groundwork is over a decade old.  Composer is the =
preferred way to use Drupal, and to install Drupal modules.

Interesting regarding Drupal 8 and beyond. =20

The ONE place where I will accept your claims that my comments have been =
"uninformed" and/or "ignorant" has been WRT the details of using Drupal =
8+ and beyond. BTW, based on your comments about Drupal and my =
subsequent analysis it turns out that Drupal is now much more of a =
developer-managed app than a user-managed app.

I started my PHP career on Drupal in 2008 and worked with it for two =
years until the work dried up and I had to switch to WordPress. I later =
found a paying Drupal project but after two weeks I fired myself because =
working with Drupal was such a pain compared to working with WordPress =
(long story, but basically it came down to theming architectures =E2=80=94=
 Drupal's was coupled vs. WordPress' being decoupled =E2=80=94 making =
Drupal much harder to theme when compared to WordPress.)

Interesting that Drupal 8+ fully embraced PSR-4. I was aware of their =
embrace of Symfony in Drupal 8 and even wrote a blog post[1] right at 10 =
years ago on how that decision would result in a sizable decline in =
Drupal's user base over time. I drew on my experience watching the =
precipitous decline of Visual Basic when Microsoft transitioned from =
Visual Basic 6 to VB.NET =E2=80=94 and history shows I was right about =
Drupal, at least in predicting the sizable market decline. Visual Basic =
was once the most widely-used programming language in the world. But =
after moving to VB.NET it fell into almost complete obscurity.

In the past decade Drupal's marketshare of CMS has fallen significantly, =
from around 7.2% of CMS in Jan 2013 to 1.4% in July 2024. Of course =
there are far too many factors to say Drupal's decline was in-fact =
because of their embrace of complexity but it certainly correlates.  A =
more charitable analysis is that Drupal chose to go upmarket to the =
enterprise which needed that complexity, but that would also imply that =
the rest of the market moved on from Drupal likely because the =
complexity no longer met their needs.  #fwiw

> So the line is not as hard between those two models as you might =
think.
>=20
>> PSR-4 generally assumes that ALL code goes=20
>> into ONE hierarchy and that any and all code that will be autoload =
gets=20
>> placed in that hierarchy.
>=20
> This is flatly untrue, and belies a considerable ignorance about how =
PSR-4 and Composer work.

No, It shows that you ignored my use of the word "generally."

And the only thing it belies is that you chose the *least* charitable =
characterization of my knowledge of how PSR-4 and Composer work that you =
could have, instead of charitably presuming that maybe I actually do =
have valuable insight you do not have. =20

I do not presume the same of you =E2=80=94 knowing you have knowledge I =
do no have in various areas. I do however make exception for those areas =
where you both claim a lack of experience and then demonstrate a lack of =
knowledge, as I reveal in sections below.

Btw, if I revised my words to avoid being imprecise with "generally" I =
would write that PHP encourages **too many** autoloader callbacks, and =
the way most of the PHP community tries to manage that is to condense =
all those independent groups of code each with their own autoloader =
callback, down into as few PSR-4 hierarchies that with as few autoloader =
callbacks as possible.

Just like procedural __get and __set magic methods seemed like a good =
idea at the time but now declarative property hooks will supplant them =
so to I argue that procedural autoloaders =E2=80=94 which seemed like a =
good idea at the time =E2=80=94 have the same types of issues as =
procedural magic methods and that moving to a declarative form for =
autoloading would improve PHP significantly.


> PHP supports multiple autoloader callbacks, and has for over 15 years. =
 You absolutely can register multiple if you'd like, using whatever =
logic you like. PHP will call each one in turn until the class is =
loaded.

Of *course* PHP supports multiple autoloader callbacks.  I know, I had =
to work with many autoloaders over the years of developing WordPress =
sites and plugins.

But PHP supporting multiple autoloader callbacks is far from the benefit =
you are insinuating when they proliferate.

Consider a user-managed WordPress site =E2=80=94 and I worked on many =
like this after the site had gotten so slow the user had to hire a =
developer =E2=80=94 where the user has 50+ active plugins on the site. =
And then consider that up to 51 autoloaders had been registered, or on =
average 25 autoloaders run EVERY new symbol load attempt which results =
in a performance and complexity nightmare.

So if there is ignorance in this thread, it is the blissful ignorance of =
the issues that too many autoloader callbacks can cause. And that =
ignorance from someone who directly claims "I haven't seriously used =
Wordpress in, ever." =20

So please let us both acknowledge what each of us *do* know vs. what we =
do not, and then accept that the *other* likely has insight into the =
areas the opposite person does not know.

> All PSR-4 does is specify a directory structure that makes a common =
autoloader stupidly simple to write.  It's just a few lines long.  But =
you can already do any logic you like for an autoloader.  PHP doesn't =
care.
>=20
> However, nothing precludes you from registering multiple autoloaders, =
all using PSR-4, all using a different path root.  That has been =
trivially simple to do since 2009.  (OK, it was PSR-0 at the time, but =
the implications here are the same.)  So your statement above about "all =
code goes into one hierarchy" is simply flat out false.
>=20
> Of course, as you note, registering lots of separate autoloaders has a =
performance impact.  That is true.  Which is why I don't think anyone =
actually does that.

No one, well, except everyone who writes a plugin for WordPress. =20

And in production, except everyone who runs multiple plugins, which is =
practically everyone who runs a WordPress site.

Again, your admitted lack of ever using WordPress is showing your lack =
of experience on this topic.

> Composer, for instance, registers a single autoloader only.  That =
autoloader internally tracks many dozens of PSR-4 roots (one for each =
package, sometimes two per package), as well as files that will get =
force-loaded when the autoloader is registered, plus generated =
classmaps.
>=20
> Using class maps, you can put a hundred classes in one file and =
composer can handle that *today*.  That has always been possible.  That =
no one does so is a sign that there's little reason to do so in most =
cases.

What is your evidence of your claim that "no one does so?"

I just pulled up the source for one my most recently WordPress clients =
=E2=80=94 the first one I looked at since your email =E2=80=94 and =
searched for `spl_autoload_register(" in the `/wp-content/plugins/` =
directory. I found 43 different autoloaders being registered. Here is a =
screenshot of that:=20

https://postimg.cc/TyWjW6Px

So clearly yes, "people do so."  And many of them do.  True, they are =
not inside the bubble where you reside, a bubble where no other =
residents do that either.  But outside that bubble the practice is =
widespread as there is not a better way that is also realistic for the =
individual site builder to choose.

> In fact, if you use an optimized/dumped autoloader, then Composer =
simply builds an internal giant lookup table of what class maps to what =
file.  PSR-4 is then *completely irrelevant* at runtime.  It's already =
one giant O(1) lookup map.  That can be done *today*.  That *is* done =
today.

Yes, it can be done today. It *is* done today.  By. Developer. Managed. =
Apps.

> But what about systems like Drupal, that don't put code in `/vendor/`? =
 Drupal ties directly into Composer via its API,and has done so for a =
decade.  Drupal, a "user-managed application" as you describe it, has =
Composer baked in at a core level.
>=20
> It looks like the integration has evolved considerably since I was =
last involved, but have a look at:
>=20
> https://git.drupalcode.org/project/drupal/-/tree/11.x/composer

Like I previously said, if the team managing a project does not care =
about BC =E2=80=94 as Drupal apparently did not in its transition to =
Composer =E2=80=94 then tying Composer into the core is not a problem.=20=


But if they DO care about BC =E2=80=94 as WordPress very much cares =
about =E2=80=94 then fully integrating Composer into an existing =
user-managed app is a non-starter.  And as Drupal's decline in user base =
over the past decade shows, doing so also which carries a high level of =
risk of shedding user base, too.

> As of when I last looked at it (around 2016 or so), Drupal registers =
its module code roots with Composer directly, and then Composer takes =
over from there and integrates Drupal's code into its own indexes.  =
There is still only one single autoloader registered with PHP.  This is =
entirely fine.

Yes, as the Drupal module Ludwig[3] explains. It can be managed by a =
Drupal module, but even its own authors do not recommend doing so, =
stating (parenthetical mine): "However, please note that Composer (on =
the CLI) is highly recommended whenever possible!"

Which again, means current Drupal is now effectively a developer-managed =
app.

> I would encourage you to do your research before speaking =
pseudo-authoritatively on this topic, as you clearly are mis-stating =
both the problem and the tools involved today.

Back at you; see the above.

> What Drupal does not do is address the "different dependency version" =
question.  And neither does Wordpress.  Or TYPO3.  Or any other project. =
 Because that's a core PHP limitation. =20

That too.

> PHP works very very differently.  PHP has a single global list of =
symbols.  (Well, two, for classes and functions.)  Namespaces are just =
syntax sugar over very-long-names, nothing more.  There is no "local =
symbol table," so having different local symbol tables point to =
different code blocks using the same name is not even conceivable.
>=20
> If you want to change that, and give PHP multiple local symbol tables, =
then autoloading... is utterly irrelevant.  The question there is "how =
can we introduce local symbol tables in the engine without requiring 10 =
million developers to rewrite the file header of 1 billion PHP files =
across the world?" =20

You have not shown how having a local symbol table "requires 10 million =
developer to rewrite header files..." =20
As I envision it, no such thing would be required.

However, maybe I just do not understand enough about how PHP internals =
currently works? How about explaining your assertion that a local symbol =
table requires a rewrite of all PHP file's header?

> Honestly, I'm not convinced its even possible. Someone with more =
engine knowledge than I could be able to find away, maybe, but I am =
skeptical.  If it's even possible, I suspect it would be an absurdly =
large amount of work and necessarily include many hard BC breaks.
>=20
> If you'd like to prove me wrong, go for it.  But that's the problem to =
address.  Debating file paths is about four steps down the line before =
it's even relevant.  And even then... if you can't make a =
PSR-4-organized package (of which there are several hundred thousand) =
slot into that new model comfortably with zero effort on the part of the =
package author, it's doomed.

Now is a good time to summarize the request that you claim is "4 steps =
down the line" that I (and maybe Michael Morris?) wants to get PHP to =
incorporate, at least as I see it. =20

The request is to add class maps with a PHP-standardized format into PHP =
core so that when a library of code needs to register classes to be =
autoloaded they can contribute to a cascading of class maps where ONE =
internal function checks the single union of all class maps for a mapped =
symbol before turning over control if no such symbol is found in the map =
to the procedural autoloaders that are currently available. That way =
*any* PHP code could register its own class map without having to get =
the core app to rearchitect itself to allow that to happen.

It is really as simple as that, and the duplicate symbol issue is =
orthogonal, but relating them can see synergy in implementation.

> So please, spare us the ill-informed descriptions of how you think =
autoloaders work, when you have demonstrated you do not know how they =
work. =20
>=20
> Spare us the litany of complaints about PSR-4 when you have =
demonstrated you don't know what PSR-4 says.

No, spare me your bad-faith accusations of ignorance =E2=80=94  I =
demonstrated my knowledge above =E2=80=94 and your claims that "nobody =
is doing" exactly what I illustrated above that they are doing.

> Spare us the gnashing of teeth about how hard it is to use a Wordpress =
plugin that hasn't been updated in 10 years with a modern plugin because =
the former is still using a 10 year old abandoned version of some =
library, when that's not PHP's problem, that's a Wordpress maintenance =
problem.

Again, showing that your contempt of WordPress is allowing you to =
dismiss the needs of all user-manage apps as being "a maintenance =
problem."

> If you want to move this effort forward, here's your todo list:
>=20
> 1. Do some research in the engine to determine if local symbol tables =
are even possible without rewriting the engine.

Will do. =20

> 2. Work through the highly complex logic of handling three layer =
overlapping transitive dependencies in a diamond pattern with =
conflicting version requirements.

Why?  I do not see that as a problem that needs to be addressed here for =
PHP core to solve as it is too time-consuming to be resolved on load. It =
either needs to be handled by:

1.) A tool like Composer, which already throws an error when such =
conflicting requirements occur,

2.) A tool like WordPress' plugin manager, which will just throw an =
error in a sandbox and let the user or site administrator resolve the =
issue, or

3.) PHP itself which would just throw an error when such a conflict =
emerges and lets the developer figure it out.

Feels like you are just nerd-sniping here.

> 3. Investigate the performance impact of maintaining multiple versions =
of the same code in memory at once, when the order they get loaded will =
vary by request.

More nerd-sniping. =20

By the same token I could say to you "Investigate the performance impact =
of maintaining a large number of autoloaders in the same codebase" =E2=80=94=
 which I illustrated is happening =E2=80=94 except you have the =
status-quo on your side.

Performance impact due to "maintaining multiple versions of the same =
code in memory at once" is no different than the performance impact of =
maintaining multiple versions of equivalent code in memory at once that =
have had their namespaces rewritten. That is what those who have been =
forced to solve the same-symbol problem have been doing in userland. =
Most others have just given up on running two plugins when their =
dependencies conflict.

It is not up to PHP to address when people load too much code into =
memory other than for PHP to fail to continue running.  Yes, if people =
duplicate *exact* same code that should be addressed =E2=80=94 ideally =
by Composer or the app itself =E2=80=94 but if different versions cause =
performance problems its the site builder's responsibility to address, =
not PHP.=20

> 4. Think through how you'd support *both* composer-based and "user =
managed" applications with such a model, especially projects that are =
already architecturally a decade out of date (like Wordpress).

Already done.  I explained how an `spl_autoload_map()` would do exactly =
that above.

> When you have a proven that it's even possible to have multiple local =
symbol tables, we can talk.  Until then, please spare us.

My one useful takeaway from your email =E2=80=94 except that I already =
knew that =E2=80=94 was the need to figure out how PHP can handle =
multiple symbol tables.  Beyond that, your take your own advice and =
spare us (me) from your contempt and condescension as they are not good =
looks on anyone.

-Mike

[1] =
https://mikeschinkel.com/2014/the-decline-of-drupal-or-how-to-fix-drupal-8=
/
[2] =
https://w3techs.com/technologies/history_overview/content_management/ms/y
[3] https://www.drupal.org/project/ludwig=