Generating arginfo from stub files

Hi internals,

In PHP 8 it will be possible to add reflectible argument and return type
information for internal functions (it was previously theoretically
possible as well, but forbidden by policy for php-src for multiple reasons,
which have now been resolved).

It will take quite a bit of effort to add this information for hundreds of
builtin functions. I would like to take this chance to improve the way in
which arginfo structures are specified, and make it more ergonomic and
future proof. Here is an example of a typed arginfo structure:

ZEND_BEGIN_ARG_WITH_RETURN_TYPE_INFO_EX(arginfo_class_alias, 0, 2,
_IS_BOOL, 0)
ZEND_ARG_TYPE_INFO(0, user_class_name, IS_STRING, 0)
ZEND_ARG_TYPE_INFO(0, alias_name, IS_STRING, 0)
ZEND_ARG_TYPE_INFO(0, autoload, _IS_BOOL, 0)
ZEND_END_ARG_INFO()

Rather than writing this out by hand, I would like arginfo structures to be
generated from PHP stub files that contain definitions like this:

function class_alias(string $user_class_name, string $alias_name, bool
$autoload = true): bool {}

I've created a proof of concept implementation for this at
https://github.com/php/php-src/pull/4284. Function signatures are specified
in a xyz.stub.php file from which xyz_arginfo.h is generated. This file can
then be included in the implementation. Nothing about the arginfo
implementation itself changes.

What do you think about providing this mechanism?

Nikita

Rather than writing this out by hand, I would like arginfo structures to be
generated from PHP stub files that contain definitions like this:

What do you think about providing this mechanism?

Sounds good.

But also, it's really a shame we don't have the union types RFC passed
yet, as it means that the return type for a significant number of
functions will be either incomplete or misleading.

strpos(string $haystack, $needle, int $offset = 0)
This is incomplete as the return isn't specified.

strpos(string $haystack, $needle, int $offset = 0) : int
This is misleading as it can return false;

strpos(string $haystack, $needle, int $offset = 0): int|false
Yay!, tools like PhpStan or Psalm can use this info, and don't need to
maintain their own info for internal functions.

Having the 'mixed' type would also be useful here, as there will
almost always be some things that can't be expressed in a type system,
and being able to use mixed shows that the type wasn't forgotten
about, it just can be expressed currently.

cheers
Dan
Ack

https://wiki.php.net/rfc/union_types
https://wiki.php.net/rfc/mixed-typehint

But also, it's really a shame we don't have the union types RFC passed
yet, as it means that the return type for a significant number of
functions will be either incomplete or misleading.

Agreed.

Hi Nikita,

I've created a proof of concept implementation for this at
https://github.com/php/php-src/pull/4284. Function signatures are
specified
in a xyz.stub.php file from which xyz_arginfo.h is generated. This file can
then be included in the implementation. Nothing about the arginfo
implementation itself changes.

I notice on the PR there was discussion of stubs elsewhere (e.g.
https://github.com/JetBrains/phpstorm-stubs) still being necessary because
they contain additional documentation. However, has the opposite been
considered: process an existing set of stubs to remove unsupported or
irrelevant annotations, and import into php-src with this tool? That way,
the manual effort for existing functions would be virtually zero.

Regards,

Rowan Collins
[IMSoP]

On Tue, Jun 18, 2019 at 7:38 PM Rowan Collins rowan.collins@gmail.com
wrote:

Hi Nikita,

I've created a proof of concept implementation for this at
https://github.com/php/php-src/pull/4284. Function signatures are
specified
in a xyz.stub.php file from which xyz_arginfo.h is generated. This file
can
then be included in the implementation. Nothing about the arginfo
implementation itself changes.

I notice on the PR there was discussion of stubs elsewhere (e.g.
https://github.com/JetBrains/phpstorm-stubs) still being necessary
because they contain additional documentation. However, has the opposite
been considered: process an existing set of stubs to remove unsupported or
irrelevant annotations, and import into php-src with this tool? That way,
the manual effort for existing functions would be virtually zero.

This is not really possible, because we need completely accurate type
information. phpstorm-stubs is a good approximation especially when it
comes to "useful" return types, but it doesn't model things like possible
"false" return values with sufficient accuracy. Any type information we add
in php-src needs a careful review against the implementation.

Nikita

This is not really possible, because we need completely accurate type
information. phpstorm-stubs is a good approximation especially when it
comes to "useful" return types, but it doesn't model things like possible
"false" return values with sufficient accuracy. Any type information we add
in php-src needs a careful review against the implementation.

I say this with a certain amount of ignorance, but what about the
existing manual?

As the authoritative source of user-facing documentation, does it not
contain an XML parseable signature for every method, in addition to a
return value section detailing types and a standard boilerplate for
things with falseable return values?

The main issue I would see is "mixed" argument types in the signature
which would require manual intervention.

I'd certainly like to see a more comprehensive format. Stubs are great
but stubs seem more like the end result rather than the original metadata.

An XML or JSON file capable of expressing the names, types, defaults etc
would have the added benefit of being easily extensible to cover things
like descriptions or unions with different requirements for each type.

Well-docblock'd stubs, C definitions, documentation pages etc would then
be the drawn from the JSON rather than the other way.

--
Mark Randall

This is not really possible, because we need completely accurate type
information. phpstorm-stubs is a good approximation especially when it
comes to "useful" return types, but it doesn't model things like possible
"false" return values with sufficient accuracy. Any type information we
add
in php-src needs a careful review against the implementation.

I say this with a certain amount of ignorance, but what about the
existing manual?

As the authoritative source of user-facing documentation, does it not
contain an XML parseable signature for every method, in addition to a
return value section detailing types and a standard boilerplate for
things with falseable return values?

The main issue I would see is "mixed" argument types in the signature
which would require manual intervention.

I'd certainly like to see a more comprehensive format. Stubs are great
but stubs seem more like the end result rather than the original metadata.

An XML or JSON file capable of expressing the names, types, defaults etc
would have the added benefit of being easily extensible to cover things
like descriptions or unions with different requirements for each type.

Well-docblock'd stubs, C definitions, documentation pages etc would then
be the drawn from the JSON rather than the other way.

--
Mark Randall

--

Sadly this wouldn't be easy to pull off as the documented return values
don't include any error return value such as false or null;
and even some times the parameter values aren't in accordance as seen with
the null documented default values which can at times provoke an invalid
param error.

A good idea which can't work :(

Best regards

George P. Banyard

Hi internals,

In PHP 8 it will be possible to add reflectible argument and return type
information for internal functions (it was previously theoretically
possible as well, but forbidden by policy for php-src for multiple reasons,
which have now been resolved).

It will take quite a bit of effort to add this information for hundreds of
builtin functions. I would like to take this chance to improve the way in
which arginfo structures are specified, and make it more ergonomic and
future proof. Here is an example of a typed arginfo structure:

ZEND_BEGIN_ARG_WITH_RETURN_TYPE_INFO_EX(arginfo_class_alias, 0, 2,
_IS_BOOL, 0)
ZEND_ARG_TYPE_INFO(0, user_class_name, IS_STRING, 0)
ZEND_ARG_TYPE_INFO(0, alias_name, IS_STRING, 0)
ZEND_ARG_TYPE_INFO(0, autoload, _IS_BOOL, 0)
ZEND_END_ARG_INFO()

Rather than writing this out by hand, I would like arginfo structures to
be generated from PHP stub files that contain definitions like this:

function class_alias(string $user_class_name, string $alias_name, bool
$autoload = true): bool {}

I've created a proof of concept implementation for this at
https://github.com/php/php-src/pull/4284. Function signatures are
specified in a xyz.stub.php file from which xyz_arginfo.h is generated.
This file can then be included in the implementation. Nothing about the
arginfo implementation itself changes.

What do you think about providing this mechanism?

Nikita

Any more feedback on this?

The existing discussion has been mostly around "can we get the type
information from some existing source?" to which the answer is "nope":
Nothing that already exists is accurate enough for php-src.

Nikita

Hi internals,

In PHP 8 it will be possible to add reflectible argument and return type
information for internal functions (it was previously theoretically
possible as well, but forbidden by policy for php-src for multiple
reasons,
which have now been resolved).

It will take quite a bit of effort to add this information for hundreds
of
builtin functions. I would like to take this chance to improve the way in
which arginfo structures are specified, and make it more ergonomic and
future proof. Here is an example of a typed arginfo structure:

ZEND_BEGIN_ARG_WITH_RETURN_TYPE_INFO_EX(arginfo_class_alias, 0, 2,
_IS_BOOL, 0)
ZEND_ARG_TYPE_INFO(0, user_class_name, IS_STRING, 0)
ZEND_ARG_TYPE_INFO(0, alias_name, IS_STRING, 0)
ZEND_ARG_TYPE_INFO(0, autoload, _IS_BOOL, 0)
ZEND_END_ARG_INFO()

Rather than writing this out by hand, I would like arginfo structures to
be generated from PHP stub files that contain definitions like this:

function class_alias(string $user_class_name, string $alias_name, bool
$autoload = true): bool {}

I've created a proof of concept implementation for this at
https://github.com/php/php-src/pull/4284. Function signatures are
specified in a xyz.stub.php file from which xyz_arginfo.h is generated.
This file can then be included in the implementation. Nothing about the
arginfo implementation itself changes.

What do you think about providing this mechanism?

Nikita

Any more feedback on this?

The existing discussion has been mostly around "can we get the type
information from some existing source?" to which the answer is "nope":
Nothing that already exists is accurate enough for php-src.

Nikita

Would it be too aggressive to have (in the php-src test suite, as a first
step) something that ensures that a stub exists for each exposed userland
symbol?

That would ensure that any newly added symbol or core extension has the
matching userland reflectable-ish structure, which later (hopefully, in the
years) could lead to having proper reflection on internal symbols...

Marco Pivetta

http://twitter.com/Ocramius

http://ocramius.github.com/

Would it be too aggressive to have (in the php-src test suite, as a first
step) something that ensures that a stub exists for each exposed userland
symbol?

That's an interesting point, at the risk of flogging a dead horse with
my ignorance of internals, what about using the unit tests as a means of
enumerating the arguments, their names and their return types?

If there's nothing that meets the requirements of php-src, what about
php-src itself? As almost all arg are handled via zpp, and almost all
the return values are handled by RETURN_XXX etc, what about a one-off
build that modifies those macros to report their usage to a global
state, which can be read back after and used to build the docs?

Something like a dozen years ago, I wrote a scripting language in C++
for my undergraduate thesis, and hit across a similar problem. I too
used macros to define functions arguments, what their types were, copied
them into native variables and so forth. When it came to documenting all
the functions I was quickly losing my sanity, and so modified my macros
so that if ran with a certain define active, they would dump all of
their information into a global store that I could dump after.

Obviously PHP's use of quasi-overloading doesn't help its situation
here, and Z_PARAM_ZVAL and friends could cause quite a headache, as it
would need to track what types were extracted through such macros as
Z_LVAL_XX.

Certainly not a catch-all situation, but might present an opportunity
for doing some of the leg work automatically.

Mark Randall

Hi internals,

In PHP 8 it will be possible to add reflectible argument and return type
information for internal functions (it was previously theoretically
possible as well, but forbidden by policy for php-src for multiple reasons,
which have now been resolved).

It will take quite a bit of effort to add this information for hundreds of
builtin functions. I would like to take this chance to improve the way in
which arginfo structures are specified, and make it more ergonomic and
future proof. Here is an example of a typed arginfo structure:

ZEND_BEGIN_ARG_WITH_RETURN_TYPE_INFO_EX(arginfo_class_alias, 0, 2,
_IS_BOOL, 0)
ZEND_ARG_TYPE_INFO(0, user_class_name, IS_STRING, 0)
ZEND_ARG_TYPE_INFO(0, alias_name, IS_STRING, 0)
ZEND_ARG_TYPE_INFO(0, autoload, _IS_BOOL, 0)
ZEND_END_ARG_INFO()

Rather than writing this out by hand, I would like arginfo structures to be
generated from PHP stub files that contain definitions like this:

function class_alias(string $user_class_name, string $alias_name, bool
$autoload = true): bool {}

I've created a proof of concept implementation for this at
https://github.com/php/php-src/pull/4284. Function signatures are specified
in a xyz.stub.php file from which xyz_arginfo.h is generated. This file can
then be included in the implementation. Nothing about the arginfo
implementation itself changes.

What do you think about providing this mechanism?

I like it. Thanks!

Christoph