Linking and autoloading of dependent extensions

11 years ago by Michael Wallner — view source

unread

Hi,

I've prepared a tiny POC which enables linking and autoloading of
dependent extensions:
https://github.com/m6w6/php-src/compare/module_deps

Anybody else other than me interested in this feature?

--
Regards,
Mike

11 years ago by johannes@schlueters.de — view source

unread

Hi,

I've prepared a tiny POC which enables linking and autoloading of
dependent extensions:
https://github.com/m6w6/php-src/compare/module_deps

Anybody else other than me interested in this feature?

This is a problem hurting me for a while and I'm happy you're looking
into it! I'm not sure about the solution here, though. This solution
assumes

  * the extension is inside extension_dir and
  * a specific naming scheme on the extension's shared library file.

Both assumptions might be wrong and might lead to loading the wrong file
when different ones are around. This will also lead to a misleading
error message ("PHP Warning: Module 'foo' already loaded in Unknown on
line 0") where the users just tries to load it once in wrong order.

I'm not sure this is better than completely offloading it to the user,
as we do now.

johannes

11 years ago by Julien Pauli — view source

unread

On Tue, May 27, 2014 at 2:32 PM, Johannes Schlüter johannes@schlueters.de
wrote:

Hi,

I've prepared a tiny POC which enables linking and autoloading of
dependent extensions:
https://github.com/m6w6/php-src/compare/module_deps

Anybody else other than me interested in this feature?

Hey

I once worked on extension improvements with Joe.
We stopped at extension dependency management, as this needs deep rethink.
I plan to write an RFC and work on the subject for PHP-Next, perhaps we
could merge our ideas ?
Code is here :
https://github.com/jpauli/php-src/compare/extensions_improvements

Cheers,

Julien

11 years ago by Michael Wallner — view source

unread

Hi,

I've prepared a tiny POC which enables linking and autoloading of
dependent extensions:
https://github.com/m6w6/php-src/compare/module_deps

Anybody else other than me interested in this feature?

Hey

I once worked on extension improvements with Joe.
We stopped at extension dependency management, as this needs deep rethink.
I plan to write an RFC and work on the subject for PHP-Next, perhaps we could merge our ideas ?
Code is here : https://github.com/jpauli/php-src/compare/extensions_improvements

Sure, I’d be glad to combine efforts. Yet, I don’t see you linking dependent extensions, so I’m not sure what your approach is to even load the top extension of the dependency chain?

Cheers,
Mike

11 years ago by Julien Pauli — view source

unread

On Tue, May 27, 2014 at 2:32 PM, Johannes Schlüter johannes@schlueters.de
wrote:

Hi,

I've prepared a tiny POC which enables linking and autoloading of
dependent extensions:
https://github.com/m6w6/php-src/compare/module_deps

Anybody else other than me interested in this feature?

Hey

I once worked on extension improvements with Joe.
We stopped at extension dependency management, as this needs deep rethink.
I plan to write an RFC and work on the subject for PHP-Next, perhaps we
could merge our ideas ?
Code is here :
https://github.com/jpauli/php-src/compare/extensions_improvements

Sure, I’d be glad to combine efforts. Yet, I don’t see you linking dependent
extensions, so I’m not sure what your approach is to even load the top
extension of the dependency chain?

Yeah, that was the idea we discussed but never implemented.
Beeing able to declare a weigthed heap of extensions as dependencies,
and having the system try to load them using the weigth as load
order, and recover cleanly in case on error. All should have taken
care of recursion.

Julien

11 years ago by Michael Wallner — view source

unread

I've prepared a tiny POC which enables linking and autoloading of
dependent extensions:
https://github.com/m6w6/php-src/compare/module_deps

Anybody else other than me interested in this feature?

This is a problem hurting me for a while and I'm happy you're looking
into it! I'm not sure about the solution here, though. This solution
assumes
 * the extension is inside extension_dir and
 * a specific naming scheme on the extension's shared library file.
Both assumptions might be wrong and might lead to loading the wrong file
when different ones are around. This will also lead to a misleading
error message ("PHP Warning: Module 'foo' already loaded in Unknown on
line 0") where the users just tries to load it once in wrong order.

Thanks Johannes, as always you’ve got a bigger picture than I have!
The warning should be easy to mitigate except for multiple extension INI entries; not sure if this is that important, though.

The second problem is a bit harder, except for source distribution installations where EXTENSION_DIR should be fine and PECL installs where I could use INI’s extension_dir.
For relocated extension_dirs like Linux dist packages, this might indeed be a problem.

Thank you,
Mike

11 years ago by Michael Wallner — view source

unread

I've prepared a tiny POC which enables linking and autoloading of
dependent extensions:
https://github.com/m6w6/php-src/compare/module_deps

Anybody else other than me interested in this feature?

This is a problem hurting me for a while and I'm happy you're looking
into it! I'm not sure about the solution here, though. This solution
assumes
 * the extension is inside extension_dir and
 * a specific naming scheme on the extension's shared library file.

Wait, what? I guess I read too fast. Did you see extensions not named that way?

--
Regards,
Mike

11 years ago by johannes@schlueters.de — view source

unread

 * the extension is inside extension_dir and
 * a specific naming scheme on the extension's shared library file.
Wait, what? I guess I read too fast. Did you see extensions not named that way?

I myself do this when keeping different versions around for testing etc.
and I've even different (commercial) extensions doing that for
distributing differently named files for different PHP versions.This
also happens when handling mixed cased names vs. lowercase (i.e. if you
build phar shared you get phar.so but it is called Phar, I assume there
are others)

johannes

11 years ago by Michael Wallner — view source

unread

 * the extension is inside extension_dir and
 * a specific naming scheme on the extension's shared library

file.

Wait, what? I guess I read too fast. Did you see extensions not named
that way?

I myself do this when keeping different versions around for testing etc.
and I've even different (commercial) extensions doing that for
distributing differently named files for different PHP versions.This
also happens when handling mixed cased names vs. lowercase (i.e. if you
build phar shared you get phar.so but it is called Phar, I assume there
are others)

Well, ok... so that's absolutely not the problem I'm trying to solve. You
can still load that stubbornly named extensions manually and nothing will
complain, even if something has a dependency on it.

I'm about e.g. pecl install & going ahead without reordering extension
entries...

Cheers,
Mike

11 years ago by johannes@schlueters.de — view source

unread

Well, ok... so that's absolutely not the problem I'm trying to solve.
You can still load that stubbornly named extensions manually and
nothing will complain, even if something has a dependency on it.

I'm about e.g. pecl install & going ahead without reordering extension
entries...

The issue is this can pull in the wrong extension giving the user a hard
time to understand how and why that happens.

And I don't think this will solve the "pecl install & going ahead" wish

what should "pecl install" do? - Add new extensions to the end of the
file? So it will be loaded two times?

I think the solution is more in the loading and startup logic. I think
we should only load explicitly mentioned extensions (or alternatively
all extension in extension dir, while that has issues, too, think i.e.
Windows where we distribute a bunch of dlls which require some 3rd party
libs in proper places), then analyze the dependencies, order the startup
call accordingly and start them up.

(this all won't solve issues where lazy linking doesn't work and we need
system level symbols from other extensions, though ... but I think
systems without working RTLD_LAZY are rare these days)

johannes

11 years ago by Michael Wallner — view source

unread

Well, ok... so that's absolutely not the problem I'm trying to solve.
You can still load that stubbornly named extensions manually and
nothing will complain, even if something has a dependency on it.

I'm about e.g. pecl install & going ahead without reordering extension
entries...

The issue is this can pull in the wrong extension giving the user a hard
time to understand how and why that happens.

Could you give an example? I can hardly imagine a likely scenario...

And I don't think this will solve the "pecl install & going ahead" wish

what should "pecl install" do? - Add new extensions to the end of the
file? So it will be loaded two times?

Wa? pecl will download, build and install the dependencies listed in
package.xml.
It is smart enough to not add the same extension line twice. Frankly, I'm
not sure what you are talking about.

I think the solution is more in the loading and startup logic. I think
we should only load explicitly mentioned extensions (or alternatively

Which is the way it currently is, isn't it?

all extension in extension dir, while that has issues, too, think i.e.

Sounds crazy. Distributions also ship lots of not enabled extensions.

Windows where we distribute a bunch of dlls which require some 3rd party
libs in proper places), then analyze the dependencies, order the startup
call accordingly and start them up.

(this all won't solve issues where lazy linking doesn't work and we need
system level symbols from other extensions, though ... but I think
systems without working RTLD_LAZY are rare these days)

My approach is quite unintrusive, should work in 90% of cases as before,
and in 10% improved as intended. (Given I solve the install_root dilemma)

Cheers,
Mike

11 years ago by johannes@schlueters.de — view source

unread

Well, ok... so that's absolutely not the problem I'm trying to solve.
You can still load that stubbornly named extensions manually and
nothing will complain, even if something has a dependency on it.

I'm about e.g. pecl install & going ahead without reordering extension
entries...

The issue is this can pull in the wrong extension giving the user a hard
time to understand how and why that happens.

Could you give an example? I can hardly imagine a likely scenario...

Whenever there are multiple versions of an extension around. This
happens for different (bad) reasons.

And I don't think this will solve the "pecl install & going ahead" wish

what should "pecl install" do? - Add new extensions to the end of the
file? So it will be loaded two times?

Wa? pecl will download, build and install the dependencies listed in package.xml.
It is smart enough to not add the same extension line twice. Frankly,
I'm not sure what you are talking about.

It is enough to add it once in the wrong place (too late) to produce a
"already loaded warning" - first the extension will be pulled in by the
new magic, then by the line from php.ini.

pecl install also won't know which php.ini file(s) to edit - some setups
use different files for cli and web server etc.

I think the solution is more in the loading and startup logic. I think
we should only load explicitly mentioned extensions (or alternatively

Which is the way it currently is, isn't it?

nope. php_load_extension() opensthe file, fetches the module structure
and starts it up in one go.

all extension in extension dir, while that has issues, too, think i.e.

Sounds crazy. Distributions also ship lots of not enabled extensions.

The could solve it by symlinking "activated" ones in a directory which
is configured as extension dir. Mentioned it mostly for brainstorming
purpose.

Windows where we distribute a bunch of dlls which require some 3rd party
libs in proper places), then analyze the dependencies, order the startup
call accordingly and start them up.

(this all won't solve issues where lazy linking doesn't work and we need
system level symbols from other extensions, though ... but I think
systems without working RTLD_LAZY are rare these days)

My approach is quite unintrusive, should work in 90% of cases as
before, and in 10% improved as intended. (Given I solve the
install_root dilemma)

... and gives harder to debug behavior in case a system isn't properly
setup, which likely means there is a less experienced user, which will
have an even harder time.

johannes

11 years ago by Michael Wallner — view source

unread

On 27 May 2014 17:47, "Johannes Schlüter" johannes@schlueters.de
wrote:

Well, ok... so that's absolutely not the problem I'm trying to
solve.
You can still load that stubbornly named extensions manually and
nothing will complain, even if something has a dependency on it.

I'm about e.g. pecl install & going ahead without reordering
extension
entries...

The issue is this can pull in the wrong extension giving the user a
hard
time to understand how and why that happens.

Could you give an example? I can hardly imagine a likely scenario...

Whenever there are multiple versions of an extension around. This
happens for different (bad) reasons.

So you are not able to provide an example either?
If you manage multiple versions of extensions, you're probably also
capable of managing your extension section to be loaded in proper manner.

And I don't think this will solve the "pecl install & going ahead"
wish

what should "pecl install" do? - Add new extensions to the end of
the
file? So it will be loaded two times?

Wa? pecl will download, build and install the dependencies listed in
package.xml.
It is smart enough to not add the same extension line twice. Frankly,
I'm not sure what you are talking about.

It is enough to add it once in the wrong place (too late) to produce a
"already loaded warning" - first the extension will be pulled in by the
new magic, then by the line from php.ini.

Which I already mentioned is probably possible to mitigate.

pecl install also won't know which php.ini file(s) to edit - some setups
use different files for cli and web server etc.

You obviously do not use the pecl command. It edits whichever file you tell
it to maintain.

I think the solution is more in the loading and startup logic. I think
we should only load explicitly mentioned extensions (or alternatively

Which is the way it currently is, isn't it?

nope. php_load_extension() opensthe file, fetches the module structure
and starts it up in one go.

Which is not possible for extensions that use symbols exportet from
dependent extensions, it won't dlopen.

all extension in extension dir, while that has issues, too, think
i.e.

Sounds crazy. Distributions also ship lots of not enabled extensions.

The could solve it by symlinking "activated" ones in a directory which
is configured as extension dir. Mentioned it mostly for brainstorming
purpose.

Windows where we distribute a bunch of dlls which require some 3rd
party
libs in proper places), then analyze the dependencies, order the
startup
call accordingly and start them up.

(this all won't solve issues where lazy linking doesn't work and we
need
system level symbols from other extensions, though ... but I think
systems without working RTLD_LAZY are rare these days)

My approach is quite unintrusive, should work in 90% of cases as
before, and in 10% improved as intended. (Given I solve the
install_root dilemma)

... and gives harder to debug behavior in case a system isn't properly
setup, which likely means there is a less experienced user, which will
have an even harder time.

I wouldn't call such a system not properly set up, but rather severely
fucked up :)

Re-reading the manpage it's now obvious why my first tests failed, which
caused me to abandon RTLD_LAZY. So back to the start.

Anyway, whatever I propose, I cannot fix your extension naming quirks.

Cheers, Mike

11 years ago by Hannes Magnusson — view source

unread

Hi,

I've prepared a tiny POC which enables linking and autoloading of
dependent extensions:
https://github.com/m6w6/php-src/compare/module_deps

Anybody else other than me interested in this feature?

I'm assuming the idea here is to fix the chicken and egg problem of
loading pdo_mysql before pdo? or mysqlnd before mysqli? and whatever
extensions that depend on ext/json...

Big +1 from me.

But I think the extension system needs more overhaul then just this.
We need to think about the next 10 years. pecl is clearly not a tool
that can tackle the next decade of PHP.

And why oh why do we have almost 80 extensions bundled with php?
There are plenty of pecl extensions that are used by gazillion more
people then half of them.

http://bjori.blogspot.com/2014/05/i-have-dream.html

Should we try to work on new system for PHPNG?
Something that could also be more friendly to things like composer? :]

-Hannes