Improving the usability of PHP's interactive shell? (completions, displaying expression results, custom syntax)

Hi internals,

php -a currently provides an interactive shell with tab completion written in C, in readline/libedit.
This currently has a number of limitations:

• Adding new features or bug fixes would require a time getting familiar with
  C programming, PHP's internals and memory management, and with how readline works internally.

  Making it possible to write replace more of the REPL implementation in PHP instead of C
  may open this up to more contributors.
  (at the cost of a completion performance slowdown/memory increase/installation size,
  which should be unimportant if only some interactive shells are affected or if there are few candidates for completion)

• Completions for syntax such as constants, member variables/properties, etc. ($x->) have been a TODO for years.

// ext/readline/readline_cli.c
static char *cli_completion_generator(const char *text, int index) /* {{{ */
{
/*
TODO:
- constants
- maybe array keys
- language constructs and other things outside a hashtable (echo, try, function, class, ...)
- object/class members

- future: respect scope ("php > function foo() { $[tab]" should only expand to local variables...)
*/

So, my ideas for improving the situation in 8.1 or later

• Allow overriding readline_completion_function() in auto_prepend_file
  (I think that that not working is a bug, and hope it could get fixed in 8.0)
  https://github.com/php/php-src/pull/5872

  https://github.com/phan/phan/commit/228827516df606de23e9ac94873c7b953f4bf4c1 (tool/phan_repl_helpers.php) includes a prototype
  of replacing PHP's C readline completion with a readline callback written in pure PHP.
  (I started working on that file today and it's missing some completion functionality, but it may be a good starting point)

• Add a hook to readline such as readline_set_php_result_handler_callback(function(mixed $result, string $snippet){...})
  that can be used to create a user-defined function to print/process the result of expressions.
  (Create a clone of zend_eval_stringl to support that)
  (The result value would instead be freed after the callback was called)

  Many other REPLs (Read-Eval-Print Loops) that I'm familiar with print a representation of the result of expressions, but PHP doesn't.

  Some values such as $GLOBALS, extremely long binary strings, etc. may need to have the representation truncated in some way.
  A tiny example implementation of readline_set_php_result_handler_callback would be if ($result !== null) { var_dump($result); }

• Add a hook such as readline_before_evaluate_php_snippet(string $snippet) : bool {}
  that gets called before attempting to evaluate snippets (whether or not they have parse errors).

  Returning true would indicate that the snippet should not be evaluated.

  This could be useful for adding meta commands that don't use PHP syntax, such as help; help SomeClass::someClassElement;, cd /dirname, etc.

• Add some way to provide the contents of all previous lines during php -a to provide better completions for multi-line snippets
  (e.g. suggest local variables in multi-line function declarations instead of global variables)

Finally, it would be useful to have something to tie those together:

• Provide an iphp, phpi, php-interactive, or some other entry point to prepend a bundled phar script
  that uses those hooks to provide a better user experience.
  (Analogous to how ipython and irb start a different interactive shell than python or ruby)

Miscellaneous thoughts on implementation details:

• Bundling an actual parser (e.g. https://github.com/nikic/PHP-Parser) would help in properly analyzing Foo::$var-><TAB>
  by being less reliant on heuristics (e.g. checking if $var was a variable or a property, making it easier to collect local variables, etc).

Is packaging a parser practical for a phpi binary (e.g. for package managers, maintainers of php, other reasons)?

• A parser may fail for code using new token types until the parser gets updated to handle the new token types. This stops being a concern after feature freezes.
  Looping over @token_get_all() and bailing out on an unknown token type may help with that.

• How would crash/bug fixes of phpi or the parser be handled in patch releases of php if this was released with php?

• Automatically rewriting the code to namespace the parser and its dependencies with \PHP\Internal\BundledPhpParser
  would let phpi be used with projects that depend on a different php-parser version.

  (clarifications may be necessary to indicate to end users that the bundled parser copy won't get updates or support outside of php minor releases,
  should not be used by libraries/applications and that it won't support newer php syntax, and possibly other clarifications)

• It may be useful to have an ini setting to disable these new hooks,
  in case bugs/crashes in libraries using those hooks interfered with debugging.

P.S. What do developers here use for an interactive shell?
I've seen https://github.com/bobthecow/psysh mentioned as an alternative for php -a while investigating options but I haven't gotten around to using it.
It's useful that php -a doesn't die from otherwise unrecoverable errors.

Thanks,

• Tyson