Moving `php-src/docs` forward

1 month ago by Nick Sdot — view source — reply

unread

Hey everyone,

I started updating, restructuring, and modernising php-src/docs, which
also includes migrating the existing Markdown based php-src/docs-old
into the book. The first step in my prerequisite series of PRs (please
check the PR desc to see the full scope of all PRs) give the current
docs a Markdown facelift:

https://github.com/php/php-src/pull/23368

Why Markdown:

not yet another syntax to deal with; we all write Markdown
renderable by default everywhere; including GitHub and IDEs
less bloated for titles, inline code, prose and such
we can easily extend it later if we want to

That said, Ilija -- who initially did setup the book 2.5 years ago --
considers the change as "churn" and asked me to better bring it up on
list. I didn't expect this to be a topic fo the list, but here I am. :)**

My Argument:

It's a light ~1,100 line, mechanical diff (see first PR). Though, if we
would not do it in the first PR, we would do it vice versa in the later
PR that integrates php-src/docs-old into the book -- which has pretty
much the exact same amount of lines. So we will have "churn" in one or
the other direction. These docs are so aged, outdated, incomplete that
there will be much more changes over the next weeks and months (see
last PR). Given that, I don't see any reason for holding back because of
"churn". This switch to Markdown reduces lines already by ~100 lines,
just for 17 existing documents. Not to forget, CODING_STANDARDS.md and
CONTRIBUTING.md which have contents that eventually should be integrated
into the book, are also already Markdown.

Where I agree with Ilija is that the reST table syntax is better. The PR
is already updated to use reST for tables -- both can co-exist nicely.

My Opinion

Please allow me this... If we actually want to move things forward, it
would be nice if we would not over-complicate and over-think everything.
Otherwise, things just stagnate and (almost) nothing happens as we see
with this started but never finished initiative. Ilija did a great job,
but it was merged it with the words:

some chapters first just to give an idea of the depth and language
that is expected

So to boil it down, these are internal docs, almost in the same "demo"
state as when they were introduced, not important C code. They were
introduced 2.5 years ago, yet they still have a big "work in progress"
banner, a bunch of todo's inline, and links to 404's (also meant as
todo's) in it. Plus, the overall activity to improve was rather
moderate. The only actual activity was in release-process.md -- which
is already Markdown, and not touched by my PRs at all.

From my own onboarding experience I know how complicated and confusing
things are; I strongly believe this is not the state these docs should
be in.

My Ask

Devs are understandably busy with other things, and no one can or want
to get it done. So here I am, offering myself to help out. Those who saw
my recent contributions might already know that I am interested in
making myself helpful by improving the test suite, CI, and general QA
stuff. I see these docs as a part of it, because A) docs = QA, B) a lot
of test related stuff has to be documented, and C) while I monitor PRs
for test related stuff I often see things pointed out in reviews that
are worth to be documented.

A preview of the "final" state of the prerequisite PR series can be
found here:
https://nicksdot.github.io/docs-restructure-final/

After these prerequisites are merged I can start to cook.
I'd appreciate if I could get everyone's blessing to get this done. 🥷

Cheers
Nick

1 month ago by Ilija Tovilo — view source — reply

unread

Hi Nick

Thanks for bringing this up on the list.

Just to state my thoughts officially:

Why Markdown:

not yet another syntax to deal with; we all write Markdown
reST is pretty popular, so not exactly something esoteric. Sure, I would
also guess that Markdown is more known, but reST is very hard to "learn"
either.

renderable by default everywhere; including GitHub and IDEs

I don't think this holds.

Before:
https://github.com/php/php-src/blob/master/docs/source/core/data-structures/reference-counting.rst
After:
https://github.com/NickSdot/php__php-src/blob/6730bde4ee9ddfe618213441f83af70d49dc347f/docs/source/core/data-structures/reference-counting.rst

The problem here being that this isn't pure reST or Markdown, but MyST.
I'd guess the same goes for IDEs. Switching the names to .md might look
better for a good chunk of the file, but then the reST-like parts won't
render well anymore.

Please allow me this... If we actually want to move things forward, it
would be nice if we would not over-complicate and over-think
everything. Otherwise, things just stagnate and (almost) nothing
happens as we see with this started but never finished initiative.
I don't think "let's not overcomplicate" should be an argument for
"let's not question changes". Asking for good reasons for the switching
from one stack to another seems reasonable.
So to boil it down, these are internal docs, almost in the same "demo"
state as when they were introduced, not important C code. They were
introduced 2.5 years ago, yet they still have a big "work in progress"
banner, a bunch of todo's inline, and links to 404's (also meant as
todo's) in it. Plus, the overall activity to improve was rather
moderate. The only actual activity was in release-process.md --
which is already Markdown, and not touched by my PRs at all.

None of this is wrong, but I don't think the docs have stalled because
the syntax is too hard. Rather, more time should be designated to them.
It's just hard in today's climate, e.g. LLM security reports have taken
over a large chunk of our time, and even before that issues and user
docs seem more pressing.

The auto-formatter was also removed in your PR (there might not be one
for MyST?), I think that's worth mentioning.

Ilija

1 month ago by Jordi Kroon — view source — reply

unread

None of this is wrong, but I don't think the docs have stalled because
the syntax is too hard. Rather, more time should be designated to them.
It's just hard in today's climate, e.g. LLM security reports have taken
over a large chunk of our time, and even before that issues and user
docs seem more pressing.

The auto-formatter was also removed in your PR (there might not be one
for MyST?), I think that's worth mentioning.

Ilija

I agree with Ilija. While my personal preference would be Markdown, I
don't see much benefit of rewriting/converting the docs to another
markup language. It's not hard to understand RST, nor I have a feeling
it's overcomplicated.

Besides the docs folder, RST is also currently used for UPGRADING,
UPGRADING.INTERNALS and NEWS. Changing these files to another format
will break things as they depend on that specific format (e.g
https://github.com/php/web-php/blob/master/bin/news2html internally, but
external sites parse these files as well).

So given that it does make sense that the docs are also written in the
same format for consistency. Not ignoring the fact that there are MD
files as well.

--
Regards,

Jordi Kroon

1 month ago by Nick Sdot — view source — reply

unread

Hey Jordi,

thanks for the feedback!

Besides the docs folder, RST is also currently used for UPGRADING,
UPGRADING.INTERNALS and NEWS. Changing these files to another format
will break things as they depend on that specific format (e.g
https://github.com/php/web-php/blob/master/bin/news2html internally,
but external sites parse these files as well).

This is a non-argument because these documents are not and will not be
part of the book. Different context, different consumers. They will not
be touched, and are not relevant here. Bringing those up here is like
saying we cannot migrate the internals book to Markdown because doc-en
is XML.

So given that it does make sense that the docs are also written in the
same format for consistency. Not ignoring the fact that there are MD
files as well.

Also doesn't make sense because you ignore that we unavoidably will have
"churn" in one or the other direction to integrate docs-old that are
already Markdown. Although I'd really love to be one, I am unfortunately
not a magician -- hence, I cannot make the impossible 'smallest diff
with no "churn' at all' possible because we have source documents in
both formats.

Making it Markdown is more sensible; you even say you prefer it yourself.

Cheers
Nick

1 month ago by Nick Sdot — view source — reply

unread

Hey Ilija,

thank you too for the feedback!

Hi Nick

Thanks for bringing this up on the list.

Just to state my thoughts officially:

Why Markdown:

not yet another syntax to deal with; we all write Markdown
reST is pretty popular, so not exactly something esoteric. Sure, I
would also guess that Markdown is more known, but reST is very hard to
"learn" either.

renderable by default everywhere; including GitHub and IDEs

I don't think this holds.

Before:
https://github.com/php/php-src/blob/master/docs/source/core/data-structures/reference-counting.rst

After:
https://github.com/NickSdot/php__php-src/blob/6730bde4ee9ddfe618213441f83af70d49dc347f/docs/source/core/data-structures/reference-counting.rst

The problem here being that this isn't pure reST or Markdown, but
MyST. I'd guess the same goes for IDEs. Switching the names to .md
might look better for a good chunk of the file, but then the reST-like
parts won't render well anymore.

That's only because you picked a commit from a non-final state that
solely exists to keep diffs small; mind the still existing .rst
extension. I would really appreciate if you could please at least once
look at the final PR instead of bit by bit arguing about non-final state.

Here is the real picture...

Before:
https://github.com/php/php-src/blob/master/docs/source/index.rst
After:
https://github.com/NickSdot/php__php-src/blob/docs/restructure-08/docs/source/index.md

Before:
https://github.com/php/php-src/blob/master/docs/source/core/data-structures/reference-counting.rst
After:
https://github.com/NickSdot/php__php-src/blob/docs/restructure-08/docs/source/core/memory-management/reference-counting.md

The only thing that cannot render perfectly are the non-standard tables
you asked for. Yet their contents are still very readable (in fact the
raw format you prefer in code) and still stand out (same as TOC; same in
reST/Markdown), reST renders eg important warnings/notes as plain text,
which is stupid That the Markdown syntax is much less noisy is proven by
the diff. Also, let me quote yourself from an internals message:

That said, reStructuredText is far from perfect itself.

So if you now gonna argue that:

rendering tables is more important than rendering important (!)
warnings/notes that by definition should be highlighted
using the syntax all of use daily is not better in general
suddenly reST which you called "far from perfect" has to be defended
at all cost

I will gonna fall off my chair for real.

Could you please address:

why the demo state of the docs would justify being so picky
why this is a blocker at all if we will have unavoidable "churn" one
or the other way because of docs-old
my point that much of this anyway will be touched/replaced in the
process of getting things updated

You are literally making any progress impossible by requiring no "churn".

Please allow me this... If we actually want to move things forward,
it would be nice if we would not over-complicate and over-think
everything. Otherwise, things just stagnate and (almost) nothing
happens as we see with this started but never finished initiative.
I don't think "let's not overcomplicate" should be an argument for
"let's not question changes". Asking for good reasons for the
switching from one stack to another seems reasonable.

As mentioned above, I'd really appreciate if you would get the full
picture, as in look at least once at the final PR, before questioning
bit by bit based on non-final state picks. I'd also appreciate if you
would not leave the good reasons I added unaddressed/ignored.

So to boil it down, these are internal docs, almost in the same
"demo" state as when they were introduced, not important C code. They
were introduced 2.5 years ago, yet they still have a big "work in
progress" banner, a bunch of todo's inline, and links to 404's (also
meant as todo's) in it. Plus, the overall activity to improve was
rather moderate. The only actual activity was in release-process.md
-- which is already Markdown, and not touched by my PRs at all.

None of this is wrong, but I don't think the docs have stalled because
the syntax is too hard. Rather, more time should be designated to
them. It's just hard in today's climate, e.g. LLM security reports
have taken over a large chunk of our time, and even before that issues
and user docs seem more pressing.

As I mentioned, understandable that y'all are busy with other things.
That's why I am offering to help out. :)

The auto-formatter was also removed in your PR (there might not be one
for MyST?), I think that's worth mentioning.

Nah, that's not really dropped. It's re-activated in a later branch.
Reason for deactivating temporarily was your smaller diff requirement
for the first PR. I didn't open the last branch PR yet because it is WIP
to handle feedback
(https://github.com/php/php-src/commit/0d4035b98bfd5e5d97c5b515cf6630d685b34545)
without requiring me to rewrite all commits in all branches each time.
And, it is literally mentioned in the very diff you saw the removal in.

Ilija

Cheers
Nick

1 month ago by Ilija Tovilo — view source — reply

unread

Why Markdown:

not yet another syntax to deal with; we all write Markdown
reST is pretty popular, so not exactly something esoteric. Sure, I
would also guess that Markdown is more known, but reST is very hard
to "learn" either.

renderable by default everywhere; including GitHub and IDEs

I don't think this holds.

Before:
https://github.com/php/php-src/blob/master/docs/source/core/data-structures/reference-counting.rst

After:
https://github.com/NickSdot/php__php-src/blob/6730bde4ee9ddfe618213441f83af70d49dc347f/docs/source/core/data-structures/reference-counting.rst

The problem here being that this isn't pure reST or Markdown, but
MyST. I'd guess the same goes for IDEs. Switching the names to .md
might look better for a good chunk of the file, but then the
reST-like parts won't render well anymore.

That's only because you picked a commit from a non-final state that
solely exists to keep diffs small; mind the still existing .rst
extension. I would really appreciate if you could please at least once
look at the final PR instead of bit by bit arguing about non-final state.

To be fair, you created 8 PRs and pinged me only in the first one. So
naturally that's what I look at and judge. But as I already said in my
response, switching the file name to .md will fix some formatting issues
while creating others. You'll need a custom MyST plugin to fix it.
That's fine, but I'd like to remind you that your original argument was
that reST is somehow not well supported by GitHub or IDEs, which is just
incorrect.

I won't respond to the rest of your e-mail, as it reads like a rant and
personal attack.

Ilija

1 month ago by Nick Sdot — view source — reply

unread

Hey Ilija,

That's only because you picked a commit from a non-final state that
solely exists to keep diffs small; mind the still existing .rst
extension. I would really appreciate if you could please at least
once look at the final PR instead of bit by bit arguing about
non-final state.

To be fair, you created 8 PRs and pinged me only in the first one. So
naturally that's what I look at and judge. But as I already said in my
response, switching the file name to .md will fix some formatting
issues while creating others. You'll need a custom MyST plugin to fix
it. That's fine, but I'd like to remind you that your original
argument was that reST is somehow not well supported by GitHub or
IDEs, which is just incorrect.

The first PR explains everything for the full series in its description.
What for do we need a custom MyST plugin? What is to fix? And what
formatting issues? I cannot address if I don't know what this is about.
Could you please be more specific?

My original argument are the 5 points in the PR description of the first
PR, not the single one you are mentioning now. And they were all
correct; that now tables are not rendered is because you asked for the
non-Markdown ones. Wo that's the trade off.

I won't respond to the rest of your e-mail, as it reads like a rant
and personal attack.

Nothing of what I wrote was a rant, nor meant personal? I calmly and
rationally answered your email. I was simply pointing out that you
apparently not looked at the whole thing, which you now confirmed. My
point is that it is not productive to talk about in between steps
because it leads to wrong assumptions. For me as a non-native speaker my
mail reads polite. So not at all sure what you mean, but sorry anyway if
it came over different than I meant it.

I'd appreciate if you could re-consider to answer, because I think it
had important bits; and as clarified above nothing of this was meant as
a rant or personal.

Ilija

Cheers
Nick

1 month ago by weilin-du@qq.com — view source — reply

unread

Hi Nick,

I don't object to the change, nor in favor of it.

However, since you've posted under the topic "Moving php‑src/docs forward",
I want to send you this email. I want to quote something here:

“the single worst strategic mistake that any software company can make: They
decided to rewrite the code from scratch”

Now, if the goal is to move the document forward, I don't see why we are
rewriting it from rst to md. I don't want to debate about whether rst if better
or md is, I would say that, famous doc hosters like readthedocs are rst users.
If you want to convince other that md is better than rst, in my honest opinion
your proposal is impossible to pass. To some extend, the statement is similar
to those crazy rust folks.

The point isn't rst is better, the point is if there are indeed benefits, is the
benefits worth the effort and time to put in, and worth to risk the break that
potentially exists? Note that you are basically adding new requirements here
to make your md file works.

In other words, if you and reviewers put the same effort into the content itself,
would it be more effective to the topic "Moving php‑src/docs forward" than
rewriting it in different language?

See?

I say I don't object to the change because I don't think I have a say on internal
doc, but I hope you can focus on the content itself instead of this if you want
to make the internal doc better. That is, update missing APIs through
UPGRAING.INTERNALS, building new chapters, writing articles for newcomers,
etc, etc.

That sounds way more better to me, and I am glad to see these contributions
happening

Cheers,
Weilin D.

1 month ago by Calvin Buckley — view source — reply

unread

In other words, if you and reviewers put the same effort into the content itself,
would it be more effective to the topic "Moving php‑src/docs forward" than
rewriting it in different language?

See?

I say I don't object to the change because I don't think I have a say on internal
doc, but I hope you can focus on the content itself instead of this if you want
to make the internal doc better. That is, update missing APIs through
UPGRAING.INTERNALS, building new chapters, writing articles for newcomers,
etc, etc.

That sounds way more better to me, and I am glad to see these contributions
happening

Cheers,
Weilin D.

Very much this. We have a ton of stale docs on i.e. the wiki, internals
book, etc. There was the effort to write more in docs/ (plus archiving
the old stuff in docs-old/). Changing the format is minor, rST I don't
think will pose a difficulty to people writing it more than Markdown in
the same place.

1 month ago by Nick Sdot — view source — reply

unread

Hey Weilin and Calvin,

thanks for the feedback!

In other words, if you and reviewers put the same effort into the content itself,
would it be more effective to the topic "Moving php‑src/docs forward" than
rewriting it in different language?

See?

I say I don't object to the change because I don't think I have a say on internal
doc, but I hope you can focus on the content itself instead of this if you want
to make the internal doc better. That is, update missing APIs through
UPGRAING.INTERNALS, building new chapters, writing articles for newcomers,
etc, etc.

That sounds way more better to me, and I am glad to see these contributions
happening

Cheers,
Weilin D.
Very much this. We have a ton of stale docs on i.e. the wiki, internals
book, etc. There was the effort to write more in docs/ (plus archiving
the old stuff in docs-old/). Changing the format is minor, rST I don't
think will pose a difficulty to people writing it more than Markdown in
the same place.

I totally get where you two come from, and I do not even disagree.
Adding new contents is my goal. But this step is a different task, and
it seems that I could not make entirely clear what it is about. Just in
case that's correct I'll try to make it clear now.

This task/effort started with this comment:
https://github.com/php/php-src/pull/13338#issuecomment-5263413385

Based on Ilijas answer these old docs must be checked and integrated
into the book, and that's what the PRs 23373 and 23374 do. So this whole
thing here is not "switch to Markdown because I like it so much", but
"get the old docs integrated". Those, however, are Markdown as many
other documents that must be integrated into the book. So the current
status is that we have Markdown /and/ reST documents. This means one of
either format must be converted to the other format, because we
logically do not want some pages to be rendered from Markdown and others
from reST. In fact, we have more Markdown lines than reST lines! And my
sole point is: since we must do one switch anyway, we should switch the
very few existing "demo state" book entries to Markdown instead the
other way around. For the multiple reasons I mentioned in the earlier
mails, and because as mentioned in the PR: "keep diffs when
moving/merging docs-old clean" -- ironically to avoid "churn". Because
the docs-old contents have content changes due to updates and removals
while integrating them; if those would be changed to reST in the same
time an understandable diff would be impossible.

To me that's just logical. That does make sense, doesn't it?

While typing all this I realise it was a bit unfortunate that I made the
Markdown PR the first in the stack; perhaps things would have been more
clear if the docs-old moves would have been in the first PR. But yeah,
I didn't expect that this will result in discussions in the first place,
so what I did was ordering the PRs in a way so that everything from
first to last PR remained fully renderable at all times.

Cheers
Nick