Two-step creation for PHP modules View as Markdown

Again, another "modules" proposal, but in more steps.

(sorry, this is a very long post).

===

TL;DR: the final goal (in a long time, possibly) is to be able to do
something like this:

<?php declare(module=1);
// some_file.php

export class SomeClass {
public function someMessage() {
myInternalFunction();
}
}

function myInternalFunction() {
echo "Hello world!";
}

<?php
// index.php
import SomeClass from 'some_file.php';

$o = new SomeClass();
$o->someMessage(); // Works.
myInternalFunction(); // Fatal error: undefined function "myInternalFunction".

===

Trivia:

Recently, we had proposals for "Friend classes" and "Pure PHP files".
These suggestions aren't new at all, but they demonstrate some wish in
the PHP ecosystem to make certain changes for PHP to be closer to other
programming languages.

On my side, I've reviewed some of the discussions regarding "modules",
and though it's quite messy because there are lots of different views
and opinions, I think, maybe too optimistically, that there might be a
way to pave the way to "PHP modules" in another way than a huge change
in the entire engine.

I think we can implement actual "modules" in two steps:

• Implement definition files first, so they can be handled in the
  compilation step and have no runtime effect (see later).
• Implement "modules" with an "import" keyword, and make "import"-ed
  modules in a way that a "module" is only a definition file that can
  "export" some of its defined structures and/or "import" structures from
  other modules, with a completely enclosed and standalone scope/context.

===

First step: definition files.

One of the proposals for "modules" implied files with a different PHP
extension, to make them easily distinguishible from other files, and the
recent "pure files" suggestion follows the same idea: removing the
"<?php" tag so that PHP doesn't need it anymore.

However, these discussions had certain conclusions that I agree with:

• Making the extensions different will profoundly change how PHP
  includes files. Extensions like ".inc", ".php5" or alike were
  discouraged for specific reasons, and nowadays, most PHP handlers
  (apache, nginx, caddy, and possibly others) are defaulted to ".php"
  extension files, so adding a new extension means changing both the
  engine and the whole ecosystem. The conclusion is overall that changing
  the extension will not give any benefit, and only brings disadvantages.
• "pure" files, as in "no open/close tags" brings no real value, because
  having "<?php" is similar to having a shebang line in many other file
  types, and even PHP files themselves can contain a shebang line, so it's
  already a nice indicator, and ALL tooling around PHP code needs them to
  distinguish PHP code from "non-PHP" whatever-they-are characters.

On my side, when I first read the "pure" proposal, I was thinking mostly
about "pure" as in "has no side-effect".

Which brings another idea: what if include-ing a PHP file actually had
zero side-effect, apart its compilation process?

That's where I'm coming with this idea: definition files:

Notes:

• I will often refer to "built-in" in here, and "built-in" means
  "built in PHP or one of the enabled extensions", which implies
  "accessible at compile-time"
• When I say "global scope", I also imply "global namespace scope"
  for every namespace defined in the file.

• A PHP "definition" file is a file that has ZERO global state,
  calls, or mutable statements.
• It is declared with a declare(def=1); statement at the top of it.
• It can only contain declarations: (include|require)(_once)?
const, function, namespace, class, return, interface,
  trait, use, enum, etc.
• It can include/require a file, as long as this statement is a
  string literal (or a built-in constant)
• Since the file must not contain statements, the global scope of the
  file must not refer to any variable, and must not define variables
  either. Even superglobals.
• if/else/elseif, switch or match statements can also be
  allowed, only if they respect the previous points. This way, you can
  still define functions/constants/classes depending on PHP versions
  (see next points about constants). I'm not sure about iterators
  (for, while, do...while or foreach), because I see no proper
  use-case, but they can still be allowed if they imply no global call
  statements, though it seems very unlikely anyway.
• try/catch/finally are useless if global scope has no calls, so
  they can be safely forbidden.
• Statements like break, continue are not allowed either, because
  they implicitly expect a "parent context".
• new is allowed for built-in classes only, and can only receive
  literals (because it cannot refer to variables or user-based constants.)
• exit/die are also forbidden in the global scope, and should be
  replaced with exceptions.
• throw is allowed, but can only throw exceptions from built-in
  eceptions. User-created exceptions are not allowed.
• Global scope can never contain the closing tag ?>, as a safety
  against potential "echo" calls. It /might/ use it as only last
  characters of said file, but IMO it's much easier to handle the "no
  closing tag" case than "possible as last statement of the file". The
  problem is that having a closing tag at the end can mess with IDEs
  that ensure a line feed at the end of every file (file that will
  therefore have an echo "\n"; statement in it...)
• The file's global scope can refer to constants defined internally by
  PHP or its extensions. This means every constant that isn't in the
  user key when calling get_defined_constants(true) in PHP, as
  well as magic constants like __DIR__. Only constants that are
  always available at compile-time will be checked. This way, it can't
  accidentaly trigger an Undefined constant warning for userland,
  but it can trigger one if a native extension isn't enabled or
  doesn't have said constant, which can also be detected at compile-time.

This concept brings more advantages than previous proposals:

• Your file is still normal PHP, compatible as usual.
• Can still be interpreted by all IDEs that support PHP code
• Doesn't need a different file extension
• The fact that it's a definition file is explicitly visible at the
  beginning of the file when you open it
• It can still be included/required by any other file without the file
  itself knowing that it actually includes a "definition file", and is
  therefore fully eligible to be compatible with all current autoload
  setups, including Composer
• Still allows things that frameworks do for conditional
  function/class declaration (if it relies on the PHP version
  constant, for example). Will not be able to use version_compare()
  though, but there are workarounds.
• Potential compile-time built-in constant optimization (if not
  already done by the compiler, I didn't search for this yet)
• Everything that is not global/namespace-scope (functions, classes,
  etc.) can still contain whatever code they need, and theoretically
  it can even contain the PHP closing tag, since it's compiled as an
  "ECHO" statement.
• All potential errors when including/requiring such file will be
  compile-time errors, therefore if the file is "correct", compiling
  it definitely means that it has its place in the opcache for a very
  long time as no runtime can alter its global context.
• Having no actual call statements in the global/namespaced scope
  ensures no "echo", but overall has absolutely zero runtime impact
  other than compile-time errors, since there cannot be notice/warning
  errors that might also pollute the current buffer. (I might have
  forgotten what else can throw a notice/warning, but feel free to
  correct me if I do).

There are only a tiny amount of drawbacks to this (from what I've
thought about so far):

• All definition files will have to begin with <?php declare(def=1);
  (fair enough IMO, since some static analysers are already capable of
  adding "declare(strict_types=1)` automatically...)
• Potential tiny compile-time performance drop and/or memory
  consumption, because all global statements would have to be checked
  and analysed. And maybe a bit more if constants are also validated.

For end-users, a "definition" file has only one single advantage: it has
no runtime impact when being loaded, and only when its defined
structures are used. This is a guarantee of trust that can benefit all
frameworks and libraries.

But this advantage paves the way to "modules" in a very interesting
manner: all modules must be definition files in the first place.

===

Second step: PHP modules.

TL;DR: loading a "module" is similar to an "include/require"_once,
but at the compiler-level instead of the engine/runtime-level.

The concept of "module" in my mind in PHP is the following:

A PHP Module is a normal PHP file that is, at first, a definition
file, but instead starts with declare(module=1);. This declaration
automatically implies declare(def=1);, and the wrong combo
declare(module=1,def=0); can throw a compile-time error.

On the Module side:

• It has access to new keywords: import and export, as well as
  import ... as ... and export ... as ....
• Module names are useless, since the module is the file itself.
• A module can export whatever is declared in said file: constants,
  functions, classes, enums, interfaces..., as long as it's only a
  declaration and not an actual call.
• The export keyword must implicitly be in the global namespace,
  even if it is written inside another namespace.
  This implies that these variants have to be considered strictly
  equivalent:

  <?php declare(module=1);
  namespace My\Namespace;
  export class MyClass {};
  
  <?php declare(module=1);
  namespace My\Namespace {
       export class MyClass {};
  }
  
  <?php declare(module=1);
  namespace My\Namespace {
       class MyClass {};
  }
  export MyClass;
  

  is the exact same as the following:
• Composer can add a new package type named "php-module" that accepts
  only one single PHP file as input, that file must be a module.
• A module can import other modules.
• Conditional, or encapsulated import can also be resolved at
  compile-time, and since they can only contain compiler-accessible
  statements (string literals or built-in constants), the behavior
  will be similar to an "include" statement at runtime on a definition
  file anyway.
• Unused imports can be detected at compile-time and throw a notice
  message
• (my opinion, so definitely optional) Two exports must never have the
  same name. By this, I mean that you could use import someFunction, SomeClass, SOME_CONSTANT from 'file.php'; freely without having to
  specify what you are trying to import. This is of personal taste,
  to enforce users not to use the same names to avoid confusion in
  general. I see no proper use-case to allow constants and classes to
  have the same name, for example, but some people might dislike this.
  On the engine-side it will still properly define the expected
  structures from the module, and on the userland, errors will be
  thrown if said structure is used improperly anyway. To me,
  considering how big this feature is, this is just a way to
  "opinionate for better naming" :) (and it avoids ugly things like
  import const SOME_CONST as SOME_CONST_ALIAS, class SomeClass as SomeClassAlias, function someFunction as someFunctionAlias from 'file.php';, right?)

And on the userland-side:

• The import and import ... as ... keywords also becomes
  accessible to ANY other PHP file, whether it is a definition, a
  module, or a regular PHP file.
• Just like in definition files, the import keyword can only refer
  to string literals and/or built-in constants. This restriction is
  only applied for the new import keyword, and not to the rest of
  the file.
• The import keyword will explicitly make all imported structures
  accessible in the current file, just like if the module was loaded
  with include, but only export-ed structures are accessible.
• Imports can be placed anywhere in the file, and will be resolved at
  compile-time, since their only drawback is "adding more structures
  in memory".
• An import can define an alias that will only be accessible to the
  importer-file, like import SomeClass as MyAlias from 'file.php';.

Internally, there are other interesting things that happen:

• All definitions from inside a module will have be prefixed in the
  symbols table with a hash corresponding to the current module file's
  hash. It can be similar to how anonymous classes are registered
  internally. The goal is to make them inaccessible (as much as
  possible) from the global scope.
• All calls to module file's internal definitions will use this hash
  prefix to refer to said structures.
• When using "import", it will do 3 things:
  o Analyse import-ed statements, to retrieve only the structures
  that are asked by the end-user
  o Load the file (from file, or from opcache, if not already in memory)
  o Create the modules definitions (if not already in memory) with
  internal hashes as previously described, and tree-shake unused
  structures as of the list of import-ed ones. Can be done at
  importer-compile-time too.
  This way, it would behave similarly to (require|include)_once but
  at a more granular level: with only the structures that were
  imported by the current file. Any subsequent call to import for
  the same file will do the same thing, and since these files have no
  runtime impact and only contain definitions, it should have close to
  no loading impact. Subsequent imports with the same structures will
  load only the ones that are already in memory (because they can be
  referenced with the hash-prefix), and if a "new structure" is found
  that has not been loaded already in the global space, it will create
  it in the global scope at runtime. This makes sure that module files
  with 100 exports will not load /all/ structures in memory when the
  file is imported.
• Modules have no impact on autoload, since they don't function the
  same way.
• Since internal functions/constants/etc. are hash-prefixed, they will
  never conflict with other internal structures. This means that a
  module could define the str_contains function, if it wanted. And
  it could even reuse the native function by using use function str_contains as base_str_contains. Would also work for
  object-oriented structures (class, interface, etc.) as well as
  constants too.
• We can create a ReflectionModule class, which constructor
  accepts a file name, and throws an exception if the file is not a
  module. This class would expose the list of exported structures from
  said module. Maybe it can also contain the processed hash/prefix and
  the internal structures too, but having these available kinda
  defeats the purpose of having internal structures in the first
  place... But the exported structures would be ReflectionClass,
  ReflectionConstant, etc, with the "module" flag explained in next point:
• Other Reflection classes will contain an internal "defined in
  module" flag, as well as a nullable string corresponding to the path
  to the module file if the structure is effectively defined in a module.
• FQCNs will always resolve to the "global public name", and never to
  the internal hash-prefixed name.
• A new global function can be used: spl_register_module($prefix, $filePath);. This way, we could definitely imagine a flexible
  prefix to resolve to a module path for import statements. It
  allows Composer-package-compatible syntax like import Request from '@symfony/http-foundation/Request.php'; being registered with
  something like spl_register_module('@symfony/http-foundation', __DIR__.'/vendor/symfony/http-foundation/');
• The spl_register_module function can refer to structures directly,
  if needed: spl_register_module('@symfony/http-foundation/Request', __DIR__.'/vendor/symfony/http-foundation/Request.php');.
  This allows for no-extension imports like import Request from '@symfony/http-foundation/Request'; (but this is just for the fancy
  looks)
  The function itself will register the input as a prefix or as a
  module path based on whether the specified path is a directory or a
  file, checked at runtime.
• Multiple paths can be used for the same prefix, as long as they are
  all directories.
• If a module is registered as a file instead of a directory, there
  can be only one.
• (yeah, I know, this concept looks like a reinvention of
  include_paths, but hey, it's modules now!)
• Autoload-like features can be used for projects using Composer:
  o The composer.json file can contain new field: "modules" and
  "modules_dev".
  o This field would contain a key=>value list of prefix=>file_path
  items, that Composer will register through the aforementioned
  new spl_register_module() function.
  o Composer will (sorry folks) need a way to make sure two PHP
  packages don't contain the same module prefixes resolving to two
  paths, regardless of them being directories or files. Maybe
  Packagist (sorry again) will need this too. This is important to
  avoid vendor name squatting in modules.
• These module-autoload rules would not change anything at existing
  autoload, but they would mostly be here to map a PHP package with
  its exposed API.
• This also makes sure that any PHP package can say "All API exposed
  as a module is covered by BC policy, all the rest is not". Easier
  for maintainers to keep their internal stuff, and a bit easier to
  make the Open/Closed principle available at a package-level instead
  of a class-level.

===

I already worked on the first step to "PHP definition files" and made a
PR of it:
https://github.com/php/php-src/compare/master...Pierstoval:php-src:defs

With my tiny knowledge of PHP internals, I required the help of Cursor
for that, and I added a lot of .phpt test files to ensure the basics
are covered, built the project & ran all the tests on my LMDE 7 (Debian)
machine multiple times with different configs (embed, fpm, debug, etc.),
everything works so far.
Apart the tests, the PR seems quite light, but it obviously needs
thorough review (or rewrite...) before even being converted into an RFC.
It just had the advantage of being fully ready and thoroughly tested
(hopefully I didn't forget anything) in less than a day...

/> Note: I did NOT use any llm to write this message, it was only used
for some bits of code in the above PR, nothing more./

If you have read everything down to this line, thank you very much! It's
the fruit of quite some work!

Now to you folks, it's yours to take and talk :)

--
Alex "Pierstoval" Rock
Polydisciplinary professional web development and training