(I already send the mesage after the “----”, but unfortunately it showed
up as part of the “Capturing reasons for votes for historical sake?”
thread, so I'm re-posting it as a “new” message in the hopes it'll be
its own thread. Therefore, please reply to this message, not the
previous copy of it.)

Hi everyone,

I want to break off this thing from the “Capturing reasons for votes for
historical sake?” thread.

Background and problem

Rowan Tommins wrote:

Hi Mike,

Currently it takes herculean effort to get almost anything approved,
but it takes effectively zero effort to stifle the hard work someone
invests in trying to improve PHP. Is it really just that all their work
can be nullified by a simple thumbs down like an emperor deciding the
death of a gladiator? (sorry, couldn't resist using that analogy.

BTW, I am thinking of the outrageous amount of work Paul M Jones is
putting into Server-Side Request and Response Objects (v2) and fear for
him that all his effort will be for naught, and he won't even have a
concise list of reasons why it was voted down. The best he will be able
to do is infer from the comments in thousands of the messages why people
voted down. But he still won't know.

I wanted to pull this point out from further up the thread, because I
think it's a very real concern, but I think it's about something more
fundamental than how people vote.

Large RFCs, where there's significant implementation work, are
essentially software projects. If we ran them that way, we'd all
collaborate on some sequence of steps (or several agile iterations, or
whatever), such as:

• Identifying the problem or aim
• Defining the scope
• Identifying risks
• Agreeing an approach
• Initial implementation or prototype
• Discovering problems based on the initial work
• Testing
• Final sign-off for release (this is what RFC votes should be)

The problem is that the RFC process only really covers a small
fraction of that process, and mostly the later parts. Most of the
effort, and crucially most of the decisions, are left to whoever is
drafting the RFC. So we end up with a process that feels more like a
sales pitch for a shrink-wrapped product:

• The RFC author identifies a problem, defines the scope, tries to
  identify risks, designs a solution, maybe builds an initial
  implementation or prototype, and starts refining and testing it
• This is pitched to the community
• There is some negotiation, usually over details rather than
  substantial rewrites, and often involving the author defending the
  decisions they've already made
• The community decides whether to "buy" the proposal (this is what
  RFC votes often turn into)

Some RFCs are rejected because the community doesn't actually agree
on the definition of the problem or scope; capturing that in a reason
for voting No might help someone later, but it's already far too late to
save the effort of the author. Other RFCs are rejected because the
community agrees on the problem, but not the details of the solution;
writing that down encourages someone to try again, but repeatedly trying
variations until one passes might not be the most efficient process.

Just to be clear, I am not blaming authors for bringing ready-made
pitches like this - it's what the current process tells them to do. Nor
do I have a brilliant proposal that would change it overnight. But I do
think we need to spend more time collaborating on ideas, and less on
saying "yes" or "no" to each other's ready-made solutions.

Regards,

I agree with those quoted on this, it matches my own experience from
the… I want to say around two dozen?… RFCs I have created or contributed to.

For most ideas, they get one real chance at life — someone writes a
draft RFC, attempts implementation, puts it out to discussion, perhaps
makes minor changes after feedback, and then puts it to one big vote. If
the vote passes, the feature is finalised then and there. If the vote
fails, it is dead. Of course, it does not have to be this way in every
case, but it usually is.

There are several angles you could take to criticise this process. Here
are just two on my mind:

• Ideas that are popular enough to get voted through are not necessarily
  fully refined. The typical life of an RFC — from draft to final merged
  patch — might be as little as a few weeks. That is not a long time, and
  it is probably not long enough to fully evaluate how well the idea has
  been executed. The first significant code using the feature is probably
  going to be written long after it's too late to make significant changes.
• Ideas that do not get voted through can die an untimely death. Perhaps
  with further refineements, they would have been acccepted — sometimes
  there is agreement with an idea in principle, but not in execution.

Possible solution

I have been thinking about these problems for a while now, and wondering
if there is a better way. I am aware that some projects, e.g. the
Chromium web browser and the Rust programming language, land things on
trunk before they are fully “stable” and keep them gated behind a flag
until they are done. I must admit I am not that familiar with it, but it
seems like it could be the basis of a better approach.

Essentially, what if we don't have one vote on an RFC, but two? The
first vote would be the conclusion of a similar RFC process to now, but
instead of voting on merging the feature into the next release, it would
be on merging it to an official development/unstable branch of PHP (it
could just be master but with some sort of flag). Then, over the next
few months, interested parties could play around with this bleeding-edge
version of PHP, and the feature could be gradually improved. At some
point, a second vote can then be held on whether we think the feature in
its current state is now polished enough to be finalised (frozen in
place with a promise of compatibility) and released in a regular PHP
version.

The hope would be that this longer period of refinement could:

• Give better engagement with the community — they could actually have
  not only the opportunity (publicly available binaries?), but time to try
  out and write perhaps real code with proposed features before they are
  finalised
• Reduce rough edges on features that currently successfully pass as RFCs
• Give features that are in principle popular but fail in the RFC
  process due to implementation detail disagreements a second chance —
  maybe people would be more willing to vote for things if they know it is
  not going to be released in its current form, and with more
  experimentation there is a chance of finding an acceptable final version

This is just a rough idea though. And sorry for the long email!

How does the list feel about this idea?

Regards,
Andrea

How does the list feel about this idea?

From what I have read, userland testing of even RC versions seems to be as infrequent as to practically be non-existent outside the major frameworks, having a usable number of people playing about with it when it's not even in for certain seems unrealistic?

Would it not also lead us to a situation where at any given time this incubation branch, or god forbid master, could be in any of 2^n different states depending on which engine features were available or not? That's a lot of extra uncertainty and possible unexpected interactions.

To encourage userland testing of RC versions, I think it would be helpful to maintain Docker images of RCs.

I’m not suggesting the internals group take this on themselves. IMO, this is something that the community can take on and offer, and it’s something that needs to be done before we can expect significant testing of RCs.

Cheers,
Ben

I’m not suggesting the internals group take this on themselves. IMO, this is something that the community can take on and offer, and it’s something that needs to be done before we can expect significant testing of RCs.

We could fully automate this fairly easy I'd think. We already have the APIs and it's not much different from how the testing pipeline is set up for github.

The docker-official PHP repos have some nice extra tools to go with them for installing extensions mind.

The primary difference is in pulling the daily source from the master branch and running ./buildconf on it. The rest could be pretty much identical to the official Docker repos.

Cheers,
Ben