Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:99155
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain sammykmedia.com designates 209.85.213.45 as permitted sender)
MIME-Version: 1.0
Sender: sammyk@sammykmedia.com
In-Reply-To: <HE1PR02MB105261E560488C1136C3B688BAE40@HE1PR02MB1052.eurprd02.prod.outlook.com>
References: <CALYT1Yrs-9Xr0tyGGWdVBsm0r6h+w+GZcLT0UbVCSMrBYZ01bQ@mail.gmail.com>
 <HE1PR02MB105261E560488C1136C3B688BAE40@HE1PR02MB1052.eurprd02.prod.outlook.com>
Date: Wed, 24 May 2017 13:06:41 -0500
Message-ID: <CALYT1YqXoadxQkaKtpXH4NVqt1Vk0sfFTO4TT=amMxgSM5067w@mail.gmail.com>
To: Anatol Belski <weltling@outlook.de>
Cc: PHP Internals <internals@lists.php.net>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Subject: Re: [PHP-DEV] Implement formal process for run-tests.php refactor
From: me@sammyk.me (Sammy Kaye Powers)

On Thu, May 18, 2017 at 5:59 AM, Anatol Belski <weltling@outlook.de> wrote:
>
> Hi Sammy,
>
> > -----Original Message-----
> > From: sammyk@sammykmedia.com [mailto:sammyk@sammykmedia.com] On
> > Behalf Of Sammy Kaye Powers
> > Sent: Wednesday, May 17, 2017 4:29 PM
> > To: PHP Internals <internals@lists.php.net>
> > Subject: [PHP-DEV] Implement formal process for run-tests.php refactor
> >
> > Hello internals folks!
> >
> > As you may already know, run-tests.php is an old legacy app that is in =
serious
> > need of a refactor. Some things that are badly needed amongst other thi=
ngs are:
> >
> > - Better maintainability
> > - Unit tests
> > - Concurrency
> > - Prettier output
> >
> > I have seen numerous attempts of rewriting run-tests from scratch after=
 some
> > rallying at a UG or PHP conference only to see the initial excitement d=
ie off. The
> > general advice for refactoring legacy apps is to not start with a clean=
 slate,
> > rather refactor a little bit at a time. We should treat run-tests as we=
 would any
> > other legacy app.
> >
> > We need a structured process to allow the run-tests app to get refactor=
ed a
> > little bit at a time. I propose the following (open to discussion of co=
urse).
> >
> > 1) Create a new run-tests karma with access to php-src.git/run-tests.ph=
p (and
> > hopefully the soon-to-be created php-src.git/run-tests folder).
> >
> > 2) Elect 2-3 people and grant the new run-tests karma to them. They wil=
l
> > champion the PR's that incrementally improve run-tests.
> >
> > 3) The new run-tests champions (we'll call them "champions" because, wh=
y
> > not?) will run the entire test suite ensuring that the refactor didn't =
inadvertently
> > break the functionality and if all good, merge. (We might be able to ge=
t a solid
> > Docker container or an automated CI process for an end-to-end test -
> > thoughts?)
> >
> > 4) The run-tests champions could be elected for a year-long term with t=
he
> > possibility of being reelected for another year commitment. This would =
help
> > prevent run-test champion drop-off (because sometimes, life happens).
> >
> > Once we get a process in place, I've drafted up a possible proposal of =
next steps
> > to get run-tests refactoring kicked off:
> >
> > https://github.com/SammyK/run-tests#proposed-refactoring-path
> >
> > It's time to circle the wagons on run-tests! :)
> >
> That's a fair amount on pre evaluation already =F0=9F=98=89.
>
> AFM, the parallelism is the most wanted feature that would improve QA a l=
ot, so that is what should stand in the foreground. Having a better structu=
red app might help on it, yes, but having the parallelism to be targeted "s=
ome when" is probably not a good sign. Otherwise, an nice app with colors a=
nd styles, etc. would be indeed nice but less helpful, IMHO, as the current=
 run-tests.php is great for simplicity and still maintainable well.
>
> I would suggest therefore, to put the harder goals into the foreground, a=
s that would require and warrant a certain rewrite. It's not for nothing, t=
hat previous rewrite attempts targeting similar things was abandoned previo=
usly, and it's probably not wise to allow for the same mistakes. I'd see as=
 first step an app that
>
> - is a very minimal rewrite of run-tests.php, made OO and whatsoever
> - mimics all the existing features and options, but only them
> - has as less dependencies as possible, be it PHP extensions or PHP code =
itself, best requires pure core only
> - is compatible with the current test format
> - is able to run tests in parallel
> - some simple tests are ported for parallel runs already
>
> I'd see such a bare app first developed as a pull request, agreed upon so=
 then run-tests.php is completely replaced by it. Then where the main work =
can begin - the most of it will be anyway not the app itself, but the porti=
ng of the existing tests. The required karma would include the app itself, =
ext/*/tests, Zend/tests, tests, and other related areas. Perhaps, the tests=
 able to run in parallel can be marked as such with a new section or alike.=
 There are probably thousands of tests that have to be touched to run in pa=
rallel otherwise, be it DB tests utilizing same table name or built-in serv=
er based tests that want to occupy same port. If there were such an app mer=
ged into master, that supports current test format and can run a mix of tes=
ts in old/new ways, it'd allow an organic integration and step-by-step impr=
ovement of the test suite. Then also, the other features from your evaluati=
on could be developed while the test porting work is being done.
>
> Thanks
>
> Anatol
>
>

Hey Anatol!

I agree that parallelism should be the first and primary goal of the
refactor. I just mentioned maintainability and unit tests first since
I foresee those things happing as we refactor to parallelism. :)

I'm OK doing a very minimal "proof-of-concept" project that would
showcase a possible concurrency model so that we could discuss the
tradeoffs of the implementation details in more depth. But after we
agree on an implementation, I don't think we need to merge it
directly, instead we should make small refactorings of the live
run-tests.php that would eventually guide it toward the agreed-upon
proof-of-concept implementation.

As far as tests that can't be run in parallel, we could look to HHVM
which checks for an empty file with the same name as the test file
with the ".serial" extension. If it finds that, the test is put into a
special "run in serial" bucket. I know make has a ".NOTPARALLEL"
target to specify that things should be run serially. We could also
have a new section in the .phpt file "--NOTPARALLEL--" or a tag
similar to =3D=3D=3DDONE=3D=3D=3D so it's not parsed as a section:
"=3D=3D=3DNOTPARALLEL=3D=3D=3D". At any rate, we'll need to decide on somet=
hing
but I think an empty ".serial" file makes a lot of sense. :)

What would you say would be good next steps? Should I draft up a more
formal process in the WIKI so we can discuss specifics further?
https://wiki.php.net/qa/runtests