[RFC] Configurable callback to dump results of expressions in `php -a`

3 years ago by tyson andre — view source

unread

Hi internals,

I've created a new RFC https://wiki.php.net/rfc/readline_interactive_shell_result_function
adding a way to configure a callback to be called to dump the value of single-expression statements from php -a
(and an ini setting that can be disabled to prevent that callback from being called).

Many REPLs (Read-Eval-Print Loops) for other programming languages that I'm familiar with
print a (possibly truncated) representation of the result of expressions, but PHP doesn't.
It would be useful to allow users to extend the functionality of the default interactive php shell (php -a),
possibly with auto_prepend_file or through use of command wrappers/aliases, or after loading the shell.
Prior to this RFC, there was no way to extend the interactive php shell in this way.
(I've seen https://github.com/bobthecow/psysh mentioned as an alternative for php -a while investigating this,
but that's a shell written from scratch, and doesn't have some functionality from php -a such as tolerance of fatal errors)

Because PHP's interactive shell is written in C, adding new features or bug fixes is inconvenient - it would require a lot
of time getting familiar with C programming, PHP's internals and memory management,
and with PHP's internal C ast representation.
It would be easier and more accessible to extend PHP's interactive shell through code written in PHP rather than code written in C.

This was among the proposed enhancements I brought up in https://externals.io/message/111073 .
An earlier version of the implementation was also created months ago - https://github.com/php/php-src/pull/5962

Thanks,

Tyson

3 years ago by tyson andre — view source

unread

Hi internals,

I've created a new RFC https://wiki.php.net/rfc/readline_interactive_shell_result_function
adding a way to configure a callback to be called to dump the value of single-expression statements from php -a
(and an ini setting that can be disabled to prevent that callback from being called).

Many REPLs (Read-Eval-Print Loops) for other programming languages that I'm familiar with
print a (possibly truncated) representation of the result of expressions, but PHP doesn't.
It would be useful to allow users to extend the functionality of the default interactive php shell (php -a),
possibly with auto_prepend_file or through use of command wrappers/aliases, or after loading the shell.
Prior to this RFC, there was no way to extend the interactive php shell in this way.
(I've seen https://github.com/bobthecow/psysh mentioned as an alternative for php -a while investigating this,
but that's a shell written from scratch, and doesn't have some functionality from php -a such as tolerance of fatal errors)

A reminder that I plan to start the vote on https://wiki.php.net/rfc/readline_interactive_shell_result_function on 2021-01-03

Thanks,

Tyson

3 years ago by Nikita Popov — view source

unread

On Sun, Dec 20, 2020 at 11:36 PM tyson andre tysonandre775@hotmail.com
wrote:

Hi internals,

I've created a new RFC
https://wiki.php.net/rfc/readline_interactive_shell_result_function
adding a way to configure a callback to be called to dump the value of
single-expression statements from php -a
(and an ini setting that can be disabled to prevent that callback from
being called).

Many REPLs (Read-Eval-Print Loops) for other programming languages that
I'm familiar with
print a (possibly truncated) representation of the result of expressions,
but PHP doesn't.
It would be useful to allow users to extend the functionality of the
default interactive php shell (php -a),
possibly with auto_prepend_file or through use of command
wrappers/aliases, or after loading the shell.
Prior to this RFC, there was no way to extend the interactive php shell in
this way.
(I've seen https://github.com/bobthecow/psysh mentioned as an alternative
for php -a while investigating this,
but that's a shell written from scratch, and doesn't have some
functionality from php -a such as tolerance of fatal errors)

Because PHP's interactive shell is written in C, adding new features or
bug fixes is inconvenient - it would require a lot
of time getting familiar with C programming, PHP's internals and memory
management,
and with PHP's internal C ast representation.
It would be easier and more accessible to extend PHP's interactive shell
through code written in PHP rather than code written in C.

This was among the proposed enhancements I brought up in
https://externals.io/message/111073 .
An earlier version of the implementation was also created months ago -
https://github.com/php/php-src/pull/5962

Reading through the linked earlier mail, you have quite a few additional
hooks in mind, which might need a significant amount of additional userland
code (such as a parser implementation) to usefully implement. At that point
I'm wondering what the benefit of this hybrid approach is, relatively to a
userland implementation like psysh.

That is: a) Assuming all the hooks have been implemented, what additional
functionality is the interactive shell implementation itself actually
providing? b) Is it possible to go the other way around, and expose that
additional functionality to userland instead? You do mention fatal error
tolerance as one distinguishing feature -- is there anything beyond that?

Regards,
Nikita

3 years ago by tyson andre — view source

unread

Reading through the linked earlier mail, you have quite a few additional hooks in mind, which might need a significant amount of additional userland code (such as a parser implementation) to usefully implement. At that point I'm wondering what the benefit of this hybrid approach is, relatively to a userland implementation like psysh.

That is:
a) Assuming all the hooks have been implemented, what additional functionality is the interactive shell implementation itself actually providing?
b) Is it possible to go the other way around, and expose that additional functionality to userland instead? You do mention fatal error tolerance as one distinguishing feature -- is there anything beyond that?

For https://wiki.php.net/rfc/readline_interactive_shell_result_function in particular,
compared to php -a in php 8.0, a subsequent RFC would provide additional functionality using these hooks: it would add default hooks to print a short representation of results of expressions to the interactive shell implementation (that could be turned off or replaced to dump objects)

The benefit is that dumping the result of expressions improves the default experience.
psysh wouldn't be installed by default when a new developer is learning php through the php manual, or when sshed into a remote server.
The reason to allow hooking it is because some objects wouldn't have a user-friendly representation for var_dump/var_export (e.g. recursive data structures)
Even if the code size is large, it may be doable by making the userland implementation part of ext/phpi/ (--enable-phpi), which is disabled by default and can be installed in separate packages created by operating system package maintainers (and shrunk by using minifiers)

Compared to psysh, the main distinguishing feature is definitely the ability to detect/tolerate fatal errors when compiling snippets or inheriting classes, and fewer dependencies to include to integrate an interactive shell with utilities for a project.
I don't think it should be exposed to regular processes or web servers, though, due to possible memory corruption or leaks after zend_error_noreturn (e.g. class inheritance errors after autoloading), etc.).

It would possibly be an improvement to throw an error instead of causing a fatal error for common mistakes in interactive shell sessions such as duplicate functions/parameters but I'm not sure how likely that is, especially since classes and functions currently get added as the file is being compiled.

Integrating userland shells like psysh deeply into php -a may wish to avoid readline entirely and call a callback instead of printing php> and directly processing input like those projects already do.
Two hooks may help with enabling that approach, which can be added in auto_prepend_file

A hook to call a callback instead of printing "php >" and C readline reading stdin.
e.g. readline_replace_interactive_shell_initializer(function () { ... read and process stdin in a loop })
Adding a hook to call a function every time an uncatchable fatal error was encountered, e.g. to resume the userland shell.
e.g. readline_replace_interactive_fatal_error_handler(function ($errcode, $errmsg, $file, $line, $errcount): bool { /* process or exit */ })

Thanks,

Tyson

3 years ago by Rowan Tommins — view source

unread

The benefit is that dumping the result of expressions improves the default experience.
psysh wouldn't be installed by default when a new developer is learning php through the php manual, or when sshed into a remote server.

It doesn't feel to me that you've really answered Nikita's question: if
all the code using these hooks is going to be distributed as userland
code anyway, then they're not going to improve the default experience.

It feels like we need to go in one of two directions:

a) Build a full-featured official REPL with all of these improvements
completely implemented out of the box. Limited extension hooks might
still be desirable to build custom versions for frameworks etc, but they
could be more targeted - for custom input, it could be "register
meta-command"; for custom output, we already have __debugInfo() at the
class level.

b) Expose the magic behaviour needed for something like PsySh to do
everything php -a already can, and leave the rest to userland. So far,
the only mentioned requirement is a special form of eval() that swallows
fatal errors.

My feeling is that the current mood of the community favours (b) rather
than (a); the most obvious example is that PHP used to bundle a PEAR
executable, but Composer remains an entirely external project. Is there
a reason not to aim for the same "de facto standard" for a REPL?

Regards,

--
Rowan Tommins
[IMSoP]

3 years ago by tyson andre — view source

unread

Hi Rowan Tommins,

The benefit is that dumping the result of expressions improves the default experience.
psysh wouldn't be installed by default when a new developer is learning php through the php manual, or when sshed into a remote server.

It doesn't feel to me that you've really answered Nikita's question: if
all the code using these hooks is going to be distributed as userland
code anyway, then they're not going to improve the default experience.

I was saying that I'd planned to propose defaults if this passed.
__debugInfo() is a slight improvement, but the human-readable representation of an object isn't always the same thing as the debug representation of the object.
A human-readable representation might be Point(x: 1, y: 2), where var_dump or var_export(__set_state)
is much longer, and I don't believe var_dump is a one size fits all solution for both simple and recursive data structures,
especially since __debugInfo predates the repl.

php > class Point { public function __construct(public int $x, public int $y) {} }
php > var_export(new Point(1, 2));
Point::__set_state(array(
   'x' => 1,
   'y' => 2,
))

It feels like we need to go in one of two directions:
a) Build a full-featured official REPL with all of these improvements
completely implemented out of the box. Limited extension hooks might
still be desirable to build custom versions for frameworks etc, but they
could be more targeted - for custom input, it could be "register
meta-command"; for custom output, we already have __debugInfo() at the
class level.

I'd be happy as long as we made progress on improving the interactive shell.

Psysh is 2.4MB as a compiled phar release and larger if distributed with library/application releases (e.g. on remote servers).
default extension hooks would likely be much smaller.

b) Expose the magic behaviour needed for something like PsySh to do
everything php -a already can, and leave the rest to userland. So far,
the only mentioned requirement is a special form of eval() that swallows
fatal errors.

That may or may not be possible to do through creating a new unsafe_eval PECL (or only exposing it for interactive sessions) - I'd have to check.
Something like unsafe_eval(string $code, array &$variables): mixed result (or throw UnsafeFatalError)
The usual caveats about not using it in production would apply - the php compiler treats eval differently in that it has access to the caller's scope.

https://github.com/bobthecow/psysh/blob/master/src/ExecutionLoopClosure.php uses eval(),
but there's actually a lot of heuristics to avoid calling eval() on code with common known fatal errors.
(it uses the readline/libedit PHP module if it's available, but doesn't load readline_cli.c)

The ext/readline/readline_cli.c overrides EG(bailout) with zend_try macros to recover from fatal errors

				zend_try {
					zend_eval_stringl(code, prompt_end - prompt_spec - 1, NULL, "php prompt code");
				} zend_end_try();

My feeling is that the current mood of the community favours (b) rather
than (a); the most obvious example is that PHP used to bundle a PEAR
executable, but Composer remains an entirely external project. Is there
a reason not to aim for the same "de facto standard" for a REPL?

Even if we do start distributing a better alternative shell with php or endorsing an alternative in docs,
it's still possible to incrementally improve php -a

It seems like a significant missing feature to omit printing expression results from php -a;
people who do use php -a may wish to continue using
that feature set (e.g. #setting_name=value to set/dump ini variables) but still benefit from minor improvements such as this RFC.
https://github.com/bobthecow/psysh/issues/462 mentions psysh isn't a drop-in replacement.

E.g. on shared hosting or when debugging on a remote host, it may be inconvenient to download psysh and faster to use php -a

https://www.php.net/manual/en/features.commandline.interactive.php only mentions php -a.
Perhaps it should mention external userland shells with a note that they're developed independently
if the REPL rarely receives updates in favor of other functionality.

Someone learning from the php.net manual or a tutorial with minimal dependencies wouldn't install psysh right now.

Aside: the php manual doesn't mention composer except in a few PECL extension docs,
but people seem to figure out how to use composer out of necessity and library installation docs.
Interactive shells wouldn't be as advertised

Cheers,
-Tyson

3 years ago by Rowan Tommins — view source

unread

I was saying that I'd planned to propose defaults if this passed.

Then I guess my feedback can be summarised as: propose these default
implementations first, rather than last. Start by improving the
out-of-the-box experience, and then look at what hooks userland tools
might want to customise that (which might involve something "smarter"
than replacing the entire output routine).

Psysh is 2.4MB as a compiled phar release and larger if distributed with library/application releases (e.g. on remote servers).
default extension hooks would likely be much smaller.

That was a lot of data when I used to carry floppy disks around in my
school backpack. It sounds pretty reasonable for a feature-rich
command-line tool in 2021.

Someone learning from the php.net manual or a tutorial with minimal dependencies wouldn't install psysh right now.

To be honest, I doubt most users ever come across php -a either, or know
that they need to install ext/readline to unlock its features (I'm
pretty sure that's not installed by default on Ubuntu, for instance).

I'm supportive of the idea of changing that, but I don't think the RFC
as currently presented particularly helps that cause.

Regards,

--
Rowan Tommins
[IMSoP]

3 years ago by Rowan Tommins — view source

unread

__debugInfo() is a slight improvement, but the human-readable representation of an object isn't always the same thing as the debug representation of the object.
A human-readable representation might be Point(x: 1, y: 2), where var_dump or var_export(__set_state)
is much longer, and I don't believe var_dump is a one size fits all solution for both simple and recursive data structures,
especially since __debugInfo predates the repl.

A couple of notes on this:

The readline-based shell for "php -a" was added in PHP 5.1
[https://www.php.net/manual/en/features.commandline.interactive.php] and
__debugInfo not until 5.6 [https://wiki.php.net/rfc/debug-info]
I agree that the existing debug outputs are quite verbose, but I don't
think that's a problem unique to the REPL. var_export() is constrained
to render valid PHP code, but print_r() and var_dump() could and perhaps
should represent objects more compactly.
That compact representation of a Point would be useful pretty much
everywhere anyone wanted debug output. It would also be possible to
build it on top of the existing functionality (key-value pairs from
__debugInfo if defined, else all properties).
Allowing objects to overload the output seems much preferable to the
formatting function having to know all the special cases, so I'm not
convinced of the need to hook the entire output for the shell.
I can't find any references off-hand, but I'm pretty sure popular
REPLs in other languages take that approach: use existing
pretty-printing mechanisms from the language, which in turn can be
overloaded by individual types / classes.

Regards,

--
Rowan Tommins
[IMSoP]

3 years ago by tyson andre — view source

unread

Hi internals,

I've created a new RFC https://wiki.php.net/rfc/readline_interactive_shell_result_function
adding a way to configure a callback to be called to dump the value of single-expression statements from php -a
(and an ini setting that can be disabled to prevent that callback from being called).

Many REPLs (Read-Eval-Print Loops) for other programming languages that I'm familiar with
print a (possibly truncated) representation of the result of expressions, but PHP doesn't.
It would be useful to allow users to extend the functionality of the default interactive php shell (php -a),
possibly with auto_prepend_file or through use of command wrappers/aliases, or after loading the shell.
Prior to this RFC, there was no way to extend the interactive php shell in this way.
(I've seen https://github.com/bobthecow/psysh mentioned as an alternative for php -a while investigating this,
but that's a shell written from scratch, and doesn't have some functionality from php -a such as tolerance of fatal errors)

Because PHP's interactive shell is written in C, adding new features or bug fixes is inconvenient - it would require a lot
of time getting familiar with C programming, PHP's internals and memory management,
and with PHP's internal C ast representation.
It would be easier and more accessible to extend PHP's interactive shell through code written in PHP rather than code written in C.

This was among the proposed enhancements I brought up in https://externals.io/message/111073 .
An earlier version of the implementation was also created months ago - https://github.com/php/php-src/pull/5962

Some of the feedback had mentioned that adding the ability to dump the results of expressions in the default shell
only made sense if there was a default implementation included with this RFC.

I plan to start a straw poll shortly, to measure interest with/without a proposed result expression dumper loaded by default.
https://wiki.php.net/rfc/readline_interactive_shell_result_function_straw_poll

Thanks,

Tyson

3 years ago by tyson andre — view source

unread

Hi internals,

I've created a new RFC https://wiki.php.net/rfc/readline_interactive_shell_result_function
adding a way to configure a callback to be called to dump the value of single-expression statements from php -a
(and an ini setting that can be disabled to prevent that callback from being called).

Many REPLs (Read-Eval-Print Loops) for other programming languages that I'm familiar with
print a (possibly truncated) representation of the result of expressions, but PHP doesn't.
It would be useful to allow users to extend the functionality of the default interactive php shell (php -a),
possibly with auto_prepend_file or through use of command wrappers/aliases, or after loading the shell.
Prior to this RFC, there was no way to extend the interactive php shell in this way.
(I've seen https://github.com/bobthecow/psysh mentioned as an alternative for php -a while investigating this,
but that's a shell written from scratch, and doesn't have some functionality from php -a such as tolerance of fatal errors)

I've updated https://wiki.php.net/rfc/readline_interactive_shell_result_function after feedback
and plan to start voting in a few days.
This RFC will make the interactive shell dump non-null expression results with var_dump()/var_export() by default
(https://externals.io/message/112568#112719 , https://wiki.php.net/rfc/readline_interactive_shell_result_function_straw_poll)

The dumping of expression results can be disabled through a system ini setting or at runtime

$ php -a
Interactive shell

php > 0.5*2;
=> 1.0

Thanks,

Tyson