# Generating arginfo from stub files

- Thread: https://externals.io/message/105970
- Original archive: https://news-web.php.net/php.internals/105970
- Started by Nikita Popov on 2019-06-18T15:10:01+00:00
- Messages: 11
- RFC: Remove functions with overloaded signatures (Accepted) — https://externals.io/rfc/deprecate_functions_with_overloaded_signatures.md
- Reply by email to internals@lists.php.net
- More for agents: search with https://externals.io/search.md?q={query} · RFCs at https://externals.io/rfc.md · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt

---

## #105970 — Nikita Popov — 2019-06-18T15:10:01+00:00

URL: https://externals.io/message/105970#105970

Hi internals,

In PHP 8 it will be possible to add reflectible argument and return type
information for internal functions (it was previously theoretically
possible as well, but forbidden by policy for php-src for multiple reasons,
which have now been resolved).

It will take quite a bit of effort to add this information for hundreds of
builtin functions. I would like to take this chance to improve the way in
which arginfo structures are specified, and make it more ergonomic and
future proof. Here is an example of a typed arginfo structure:

ZEND_BEGIN_ARG_WITH_RETURN_TYPE_INFO_EX(arginfo_class_alias, 0, 2,
_IS_BOOL, 0)
    ZEND_ARG_TYPE_INFO(0, user_class_name, IS_STRING, 0)
    ZEND_ARG_TYPE_INFO(0, alias_name, IS_STRING, 0)
    ZEND_ARG_TYPE_INFO(0, autoload, _IS_BOOL, 0)
ZEND_END_ARG_INFO()

Rather than writing this out by hand, I would like arginfo structures to be
generated from PHP stub files that contain definitions like this:

function class_alias(string $user_class_name, string $alias_name, bool
$autoload = true): bool {}

I've created a proof of concept implementation for this at
https://github.com/php/php-src/pull/4284. Function signatures are specified
in a xyz.stub.php file from which xyz_arginfo.h is generated. This file can
then be included in the implementation. Nothing about the arginfo
implementation itself changes.

What do you think about providing this mechanism?

Nikita

---

## #105971 — Dan Ackroyd — 2019-06-18T17:09:59+00:00

URL: https://externals.io/message/105970#105971

> Rather than writing this out by hand, I would like arginfo structures to be
> generated from PHP stub files that contain definitions like this:
>
> What do you think about providing this mechanism?

Sounds good.

But also, it's really a shame we don't have the union types RFC passed
yet, as it means that the return type for a significant number of
functions will be either incomplete or misleading.

strpos(string $haystack, $needle, int $offset = 0)
This is incomplete as the return isn't specified.

strpos(string $haystack, $needle, int $offset = 0) : int
This is misleading as it can return false;

strpos(string $haystack, $needle, int $offset = 0): int|false
Yay!, tools like PhpStan or Psalm can use this info, and don't need to
maintain their own info for internal functions.

Having the 'mixed' type would also be useful here, as there will
almost always be some things that can't be expressed in a type system,
and being able to use mixed shows that the type wasn't forgotten
about, it just can be expressed currently.

cheers
Dan
Ack

https://wiki.php.net/rfc/union_types
https://wiki.php.net/rfc/mixed-typehint

---

## #105972 — Benjamin Morel — 2019-06-18T17:13:36+00:00

URL: https://externals.io/message/105970#105972

>
> But also, it's really a shame we don't have the union types RFC passed
> yet, as it means that the return type for a significant number of
> functions will be either incomplete or misleading.
>
>
Agreed.

---

## #105973 — Rowan Collins — 2019-06-18T17:38:24+00:00

URL: https://externals.io/message/105970#105973

Hi Nikita,



> I've created a proof of concept implementation for this at
> https://github.com/php/php-src/pull/4284. Function signatures are
> specified
> in a xyz.stub.php file from which xyz_arginfo.h is generated. This file can
> then be included in the implementation. Nothing about the arginfo
> implementation itself changes.
>


I notice on the PR there was discussion of stubs elsewhere (e.g.
https://github.com/JetBrains/phpstorm-stubs) still being necessary because
they contain additional documentation. However, has the opposite been
considered: process an existing set of stubs to remove unsupported or
irrelevant annotations, and import into php-src with this tool? That way,
the manual effort for existing functions would be virtually zero.

Regards,
-- 
Rowan Collins
[IMSoP]

---

## #105974 — Nikita Popov — 2019-06-18T17:53:48+00:00

URL: https://externals.io/message/105970#105974

On Tue, Jun 18, 2019 at 7:38 PM Rowan Collins <rowan.collins@gmail.com>
wrote:

> Hi Nikita,
>
> 
>
> > […]
>
>
> I notice on the PR there was discussion of stubs elsewhere (e.g.
> https://github.com/JetBrains/phpstorm-stubs) still being necessary
> because they contain additional documentation. However, has the opposite
> been considered: process an existing set of stubs to remove unsupported or
> irrelevant annotations, and import into php-src with this tool? That way,
> the manual effort for existing functions would be virtually zero.
>

This is not really possible, because we need completely accurate type
information. phpstorm-stubs is a good approximation especially when it
comes to "useful" return types, but it doesn't model things like possible
"false" return values with sufficient accuracy. Any type information we add
in php-src needs a careful review against the implementation.

Nikita

---

## #105975 — Mark Randall — 2019-06-18T18:37:48+00:00

URL: https://externals.io/message/105970#105975

> This is not really possible, because we need completely accurate type
> information. phpstorm-stubs is a good approximation especially when it
> comes to "useful" return types, but it doesn't model things like possible
> "false" return values with sufficient accuracy. Any type information we add
> in php-src needs a careful review against the implementation.

I say this with a certain amount of ignorance, but what about the 
existing manual?

As the authoritative source of user-facing documentation, does it not 
contain an XML parseable signature for every method, in addition to a 
return value section detailing types and a standard boilerplate for 
things with falseable return values?

The main issue I would see is "mixed" argument types in the signature 
which would require manual intervention.

I'd certainly like to see a more comprehensive format. Stubs are great 
but stubs seem more like the end result rather than the original metadata.

An XML or JSON file capable of expressing the names, types, defaults etc 
would have the added benefit of being easily extensible to cover things 
like descriptions or unions with different requirements for each type.

Well-docblock'd stubs, C definitions, documentation pages etc would then 
be the drawn from the JSON rather than the other way.

--
Mark Randall

---

## #105978 — G. P. B. — 2019-06-18T20:08:41+00:00

URL: https://externals.io/message/105970#105978

> 
> > […]
> add
> > […]
>
> I say this with a certain amount of ignorance, but what about the
> existing manual?
>
> As the authoritative source of user-facing documentation, does it not
> contain an XML parseable signature for every method, in addition to a
> return value section detailing types and a standard boilerplate for
> things with falseable return values?
>
> The main issue I would see is "mixed" argument types in the signature
> which would require manual intervention.
>
> I'd certainly like to see a more comprehensive format. Stubs are great
> but stubs seem more like the end result rather than the original metadata.
>
> An XML or JSON file capable of expressing the names, types, defaults etc
> would have the added benefit of being easily extensible to cover things
> like descriptions or unions with different requirements for each type.
>
> Well-docblock'd stubs, C definitions, documentation pages etc would then
> be the drawn from the JSON rather than the other way.
>
> --
> Mark Randall
>
> --
> 
> 


Sadly this wouldn't be easy to pull off as the documented return values
don't include any error return value such as false or null;
and even some times the parameter values aren't in accordance as seen with
the null documented default values which can at times provoke an invalid
param error.

A good idea which can't work :(

Best regards

George P. Banyard

---

## #106252 — Nikita Popov — 2019-07-23T15:00:00+00:00

URL: https://externals.io/message/105970#106252

> Hi internals,
>
> In PHP 8 it will be possible to add reflectible argument and return type
> information for internal functions (it was previously theoretically
> possible as well, but forbidden by policy for php-src for multiple reasons,
> which have now been resolved).
>
> It will take quite a bit of effort to add this information for hundreds of
> builtin functions. I would like to take this chance to improve the way in
> which arginfo structures are specified, and make it more ergonomic and
> future proof. Here is an example of a typed arginfo structure:
>
> ZEND_BEGIN_ARG_WITH_RETURN_TYPE_INFO_EX(arginfo_class_alias, 0, 2,
> _IS_BOOL, 0)
>     ZEND_ARG_TYPE_INFO(0, user_class_name, IS_STRING, 0)
>     ZEND_ARG_TYPE_INFO(0, alias_name, IS_STRING, 0)
>     ZEND_ARG_TYPE_INFO(0, autoload, _IS_BOOL, 0)
> ZEND_END_ARG_INFO()
>
> Rather than writing this out by hand, I would like arginfo structures to
> be generated from PHP stub files that contain definitions like this:
>
> function class_alias(string $user_class_name, string $alias_name, bool
> $autoload = true): bool {}
>
> I've created a proof of concept implementation for this at
> https://github.com/php/php-src/pull/4284. Function signatures are
> specified in a xyz.stub.php file from which xyz_arginfo.h is generated.
> This file can then be included in the implementation. Nothing about the
> arginfo implementation itself changes.
>
> What do you think about providing this mechanism?
>
> Nikita
>

Any more feedback on this?

The existing discussion has been mostly around "can we get the type
information from some existing source?" to which the answer is "nope":
Nothing that already exists is accurate enough for php-src.

Nikita

---

## #106253 — Marco Pivetta — 2019-07-23T15:05:05+00:00

URL: https://externals.io/message/105970#106253

> 
>
> > […]
> reasons,
> > […]
> of
> > […]
>
> Any more feedback on this?
>
> The existing discussion has been mostly around "can we get the type
> information from some existing source?" to which the answer is "nope":
> Nothing that already exists is accurate enough for php-src.
>
> Nikita
>

Would it be too aggressive to have (in the php-src test suite, as a first
step) something that ensures that a stub exists for each exposed userland
symbol?

That would ensure that any newly added symbol or core extension has the
matching userland reflectable-ish structure, which later (hopefully, in the
years) could lead to having proper reflection on internal symbols...

Marco Pivetta

http://twitter.com/Ocramius

http://ocramius.github.com/

---

## #106285 — Mark Randall — 2019-07-24T15:46:24+00:00

URL: https://externals.io/message/105970#106285

> Would it be too aggressive to have (in the php-src test suite, as a first
> step) something that ensures that a stub exists for each exposed userland
> symbol?

That's an interesting point, at the risk of flogging a dead horse with 
my ignorance of internals, what about using the unit tests as a means of 
enumerating the arguments, their names and their return types?

If there's nothing that meets the requirements of php-src, what about 
php-src itself? As almost all arg are handled via zpp, and almost all 
the return values are handled by RETURN_XXX etc, what about a one-off 
build that modifies those macros to report their usage to a global 
state, which can be read back after and used to build the docs?

Something like a dozen years ago, I wrote a scripting language in C++ 
for my undergraduate thesis, and hit across a similar problem. I too 
used macros to define functions arguments, what their types were, copied 
them into native variables and so forth. When it came to documenting all 
the functions I was quickly losing my sanity, and so modified my macros 
so that if ran with a certain define active, they would dump all of 
their information into a global store that I could dump after.

Obviously PHP's use of quasi-overloading doesn't help its situation 
here, and Z_PARAM_ZVAL and friends could cause quite a headache, as it 
would need to track what types were extracted through such macros as 
Z_LVAL_XX.

Certainly not a catch-all situation, but might present an opportunity 
for doing some of the leg work automatically.

Mark Randall

---

## #106287 — Christoph M. Becker — 2019-07-24T19:19:36+00:00

URL: https://externals.io/message/105970#106287

> Hi internals,
>
> In PHP 8 it will be possible to add reflectible argument and return type
> information for internal functions (it was previously theoretically
> possible as well, but forbidden by policy for php-src for multiple reasons,
> which have now been resolved).
>
> It will take quite a bit of effort to add this information for hundreds of
> builtin functions. I would like to take this chance to improve the way in
> which arginfo structures are specified, and make it more ergonomic and
> future proof. Here is an example of a typed arginfo structure:
>
> ZEND_BEGIN_ARG_WITH_RETURN_TYPE_INFO_EX(arginfo_class_alias, 0, 2,
> _IS_BOOL, 0)
>     ZEND_ARG_TYPE_INFO(0, user_class_name, IS_STRING, 0)
>     ZEND_ARG_TYPE_INFO(0, alias_name, IS_STRING, 0)
>     ZEND_ARG_TYPE_INFO(0, autoload, _IS_BOOL, 0)
> ZEND_END_ARG_INFO()
>
> Rather than writing this out by hand, I would like arginfo structures to be
> generated from PHP stub files that contain definitions like this:
>
> function class_alias(string $user_class_name, string $alias_name, bool
> $autoload = true): bool {}
>
> I've created a proof of concept implementation for this at
> https://github.com/php/php-src/pull/4284. Function signatures are specified
> in a xyz.stub.php file from which xyz_arginfo.h is generated. This file can
> then be included in the implementation. Nothing about the arginfo
> implementation itself changes.
>
> What do you think about providing this mechanism?

I like it.  Thanks!

Christoph