# re2c version(s)

- Thread: https://externals.io/message/102796
- Original archive: https://news-web.php.net/php.internals/102796
- Started by Dmitry Stogov on 2018-07-13T09:13:42+00:00
- Messages: 31
- Page 1 of 2: messages 1 to 25 (25 per page, add `?page=N` to the URL)
- Next page: https://externals.io/message/102796.md?page=2
- Reply by email to internals@lists.php.net
- More for agents: search with https://externals.io/search.md?q={query} · RFCs at https://externals.io/rfc.md · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt

---

## #102796 — Dmitry Stogov — 2018-07-13T09:13:42+00:00

URL: https://externals.io/message/102796#102796

Hi,


I think, many core developers saw unexpected changes in "zend_labguages_scanner.c" or "var_unserializer.c" after rebuilds.

This occurs, because we use different versions of re2c, and some of them produce really different code.

They also embed version number into the generate source. Currently different files in PHP source tree generated by different re2c versions:


ext/json/json_scanner.c 0.16

ext/date/lib/parse_date.c 0.15.3

ext/date/lib/parse_iso_intervals.c 0.15.3

ext/pdo/pdo_sql_parser.c 0.16

ext/phar/phar_path_check.c 1.0.3

sapi/phpdbg/phpdbg_lexer.c 0.16

ext/standard/url_scanner_ex.c 0.16

ext/standard/var_unserializer.c 1.0.1

Zend/zend_ini_scanner.c 0.15

Zend/zend_language_scanner.c 1.0.1


I propose, to change build scripts (in master and PHP-7.3) to require at least re2c version 1.0.0 (it seems 1.0.0-1.0.3 produce the same result) and suppress version output into the generated files.


I'm not sure about timelib files.


Thanks. Dmitry.

---

## #102797 — Nikita Popov — 2018-07-13T09:26:22+00:00

URL: https://externals.io/message/102796#102797 · In reply to: #102796

> Hi,
>
>
> I think, many core developers saw unexpected changes in
> "zend_labguages_scanner.c" or "var_unserializer.c" after rebuilds.
>
> This occurs, because we use different versions of re2c, and some of them
> produce really different code.
>
> They also embed version number into the generate source. Currently
> different files in PHP source tree generated by different re2c versions:
>
>
> ext/json/json_scanner.c 0.16
>
> ext/date/lib/parse_date.c 0.15.3
>
> ext/date/lib/parse_iso_intervals.c 0.15.3
>
> ext/pdo/pdo_sql_parser.c 0.16
>
> ext/phar/phar_path_check.c 1.0.3
>
> sapi/phpdbg/phpdbg_lexer.c 0.16
>
> ext/standard/url_scanner_ex.c 0.16
>
> ext/standard/var_unserializer.c 1.0.1
>
> Zend/zend_ini_scanner.c 0.15
>
> Zend/zend_language_scanner.c 1.0.1
>
>
> I propose, to change build scripts (in master and PHP-7.3) to require at
> least re2c version 1.0.0 (it seems 1.0.0-1.0.3 produce the same result) and
> suppress version output into the generated files.
>
>
> I'm not sure about timelib files.
>
>
> Thanks. Dmitry.
>

I don't think normalizing the version really solves anything. These files
should be dropped from version control entirely instead. Generated files do
not belong in version control.

re2c is widely available on Linux distros nowadays (probably specifically
because PHP uses it) and while there might have been historical ground to
bundle these generated files, there no longer is one.

Nikita

---

## #102798 — Derick Rethans — 2018-07-13T09:55:36+00:00

URL: https://externals.io/message/102796#102798 · In reply to: #102797

> 
> 
> > […]
> 
> I don't think normalizing the version really solves anything. These 
> files should be dropped from version control entirely instead. 
> Generated files do not belong in version control.

the timelib files *must* be generated with 0.15.3: 
https://github.com/mongodb/mongo/blob/master/src/third_party/scripts/timelib_get_sources.sh#L10-L11

> re2c is widely available on Linux distros nowadays (probably 
> specifically because PHP uses it) and while there might have been 
> historical ground to bundle these generated files, there no longer is 
> one.

The timelib files will need to be continued to be bundled. There is no 
rule in the Makefile either (unlike zend_language parser(s)), so this 
should not cause issues with GIT.

cheers,
Derick

-- 
https://derickrethans.nl | https://xdebug.org | https://dram.io
Like Xdebug? Consider a donation: https://xdebug.org/donate.php,
or become my Patron: https://www.patreon.com/derickr
twitter: @derickr and @xdebug

---

## #102800 — Dmitry Stogov — 2018-07-13T11:00:27+00:00

URL: https://externals.io/message/102796#102800

Hi Nikita,


OK. Dropping files and requirement for re2c 1.0.0. Right?


Thanks. Dmitry.

---

## #102801 — Christoph M. Becker — 2018-07-13T18:06:11+00:00

URL: https://externals.io/message/102796#102801 · In reply to: #102797

> 
>
> > […]
>
> I don't think normalizing the version really solves anything. These files
> should be dropped from version control entirely instead. Generated files do
> not belong in version control.

+1

-- 
Christoph M. Becker

---

## #102802 — Sara Golemon — 2018-07-13T18:22:09+00:00

URL: https://externals.io/message/102796#102802 · In reply to: #102800

> OK. Dropping files and requirement for re2c 1.0.0. Right?
>
+1 to removing them from git (except timelib as Derick noted).

The minimum requirement is reasonable in terms of reproducibility, but
should that be specific to master/7.3?

I ask, because my release builder for 7.2 (and 7.1 if Joe is using
Davey's builder) is using re2c 0.13.5 .   I don't imagine the risk of
updating re2c mid-relase is particularly high, but it is non-zero.

-Sara

---

## #102803 — Stanislav Malyshev — 2018-07-13T18:40:12+00:00

URL: https://externals.io/message/102796#102803 · In reply to: #102797

Hi!

> re2c is widely available on Linux distros nowadays (probably

On Linux distros on common platforms (Intel/AMD) - sure. But what if you
need an uncommon platform, or one that does not run Linux? It's those
platforms where you'd have to build PHP from source (after all, PHP is
also widely available as a package on Linux distros anyway) and adding
another hassle of figuring out how to build a third-party tool - I don't
think it's a good service to the community.

-- 
Stas Malyshev
smalyshev@gmail.com

---

## #102804 — Stanislav Malyshev — 2018-07-13T18:42:53+00:00

URL: https://externals.io/message/102796#102804 · In reply to: #102798

Hi!

> the timelib files *must* be generated with 0.15.3: 
> https://github.com/mongodb/mongo/blob/master/src/third_party/scripts/timelib_get_sources.sh#L10-L11

That comment says 0.16 was problematic. Is it still true for 1.0.*? Was
it reported to re2c?

-- 
Stas Malyshev
smalyshev@gmail.com

---

## #102805 — Stanislav Malyshev — 2018-07-13T18:46:20+00:00

URL: https://externals.io/message/102796#102805 · In reply to: #102803

Hi!

> On Linux distros on common platforms (Intel/AMD) - sure. But what if you
> need an uncommon platform, or one that does not run Linux? It's those
> platforms where you'd have to build PHP from source (after all, PHP is
> also widely available as a package on Linux distros anyway) and adding
> another hassle of figuring out how to build a third-party tool - I don't
> think it's a good service to the community.

BTW, looking at re2c source, I don't see any mention of Windows. Does it
even build on Windows? What would be prerequisites for it? I thought
Windows is still a supported platform? It won't be nice if one couldn't
build PHP from source on Windows anymore...

-- 
Stas Malyshev
smalyshev@gmail.com

---

## #102806 — Sara Golemon — 2018-07-13T19:07:46+00:00

URL: https://externals.io/message/102796#102806 · In reply to: #102803

> > […]
>
> On Linux distros on common platforms (Intel/AMD) - sure. But what if you
> need an uncommon platform, or one that does not run Linux? It's those
> platforms where you'd have to build PHP from source (after all, PHP is
> also widely available as a package on Linux distros anyway) and adding
> another hassle of figuring out how to build a third-party tool - I don't
> think it's a good service to the community.
>
I would offer that this is what official releases and/or snaps.php.net is for.

**Checks snaps.php.net**
...wait, is this not a thing anymore?

-Sara

---

## #102807 — Anatol Belski — 2018-07-13T19:08:18+00:00

URL: https://externals.io/message/102796#102807 · In reply to: #102802

Hi,

> -----Original Message-----
> From: php@golemon.com <php@golemon.com> On Behalf Of Sara Golemon
> Sent: Friday, July 13, 2018 8:22 PM
> To: Dmitry Stogov <dmitry@zend.com>
> Cc: Nikita Popov <nikita.ppv@gmail.com>; PHP internals list
> <internals@lists.php.net>; Stanislav Malyshev <smalyshev@gmail.com>;
> derick@derickrethans.nl; Christoph M. Becker <cmbecker69@gmx.de>
> Subject: Re: [PHP-DEV] re2c version(s)
> 
> 
> > […]
> +1 to removing them from git (except timelib as Derick noted).
> 
> The minimum requirement is reasonable in terms of reproducibility, but should
> that be specific to master/7.3?
>
The tool version is in most cases irrelevant for the generated files. I recall there was some issues with re2c 0.16 and ext/date in 7.1, but that's the only case I could recall. But limiting the tool version seems hard for another reason - LTS distros won't upgrade them. Fe. Jessie still has re2c 0.13.5, bison 3.0.2 by default. We'd effectively force anyone to either upgrade their distros, or to use some workarounds like manually installing the required tools into a prefix and prepending it to the path.

> I ask, because my release builder for 7.2 (and 7.1 if Joe is using
> Davey's builder) is using re2c 0.13.5 .   I don't imagine the risk of
> updating re2c mid-relase is particularly high, but it is non-zero.
> 
Normally RMs would define the tool versions that are expected to be stable and that are always used for the official tarballs. In the case of 7.0, it'd be always re2c 0.13.5 and bison 3.0.2, where ever I'd produce the tarball (even I don't always use Jessie for packaging). We probably should take this into the release process docs, if it isn't there already.

Following up on Dmitry's idea - what if we'd not remove the generated files, but instead would define some minimal tool versions that have to be used by default for the core? That however also needs to stand in unison with how the official tarballs are produced. Also, it might be possible, that some particular code needs some particular tool version, which should be stated explicitly.

Regards

Anatol

---

## #102808 — Nikita Popov — 2018-07-13T19:11:14+00:00

URL: https://externals.io/message/102796#102808 · In reply to: #102803

On Fri, Jul 13, 2018 at 8:40 PM, Stanislav Malyshev <smalyshev@gmail.com>
wrote:

> Hi!
>
> > […]
>
> On Linux distros on common platforms (Intel/AMD) - sure. But what if you
> need an uncommon platform, or one that does not run Linux? It's those
> platforms where you'd have to build PHP from source (after all, PHP is
> also widely available as a package on Linux distros anyway) and adding
> another hassle of figuring out how to build a third-party tool - I don't
> think it's a good service to the community.
>

We always bundle generated files in distributed sources, so end-users need
neither re2c nor bison to build releases.

re2c is not a problem for Windows either, it is bundled as part of the PHP
SDK.

Nikita

---

## #102809 — Anatol Belski — 2018-07-13T19:16:34+00:00

URL: https://externals.io/message/102796#102809 · In reply to: #102805

> -----Original Message-----
> From: Stanislav Malyshev <smalyshev@gmail.com>
> Sent: Friday, July 13, 2018 8:46 PM
> To: Nikita Popov <nikita.ppv@gmail.com>; Dmitry Stogov <dmitry@zend.com>
> Cc: PHP internals list <internals@lists.php.net>; derick@derickrethans.nl;
> Christoph M. Becker <cmbecker69@gmx.de>
> Subject: Re: [PHP-DEV] re2c version(s)
> 
> Hi!
> 
> > […]
> 
> BTW, looking at re2c source, I don't see any mention of Windows. Does it even
> build on Windows? What would be prerequisites for it? I thought Windows is still
> a supported platform? It won't be nice if one couldn't build PHP from source on
> Windows anymore...
> 
The binary SDK has undergone a major revamp in the last couple of years. It's now using the MSYS2 port, which works almost fine and is an active project. Before that, some tools like bison 2.4.1 was used, because there was no way to upgrade it. See https://github.com/Microsoft/php-sdk-binary-tools/tree/master/msys2/usr . I was repeatedly asking systems@ to move this repo to git.php.net, but unfortunately that didn't happen. Anyway, the approach is platform specific and was inspired by how mozbuild does it. One always has the full control on which tool versions are used and bad versions can be omitted.

The binary SDK for Windows currently uses bison 3.0.4 (3.0.5 in staging) and re2c 1.0.3 on all the branches. The exact versions run on AppVeyor and are tested continuously. Also every release is tested individually and snapshots from windows.php.net are tested, too. The binary SDK has all the tools that are needed to build PHP. The sources for Windows zipballs are distributed separately, which has its obvious reasons.

Regards

Anatol

---

## #102810 — Sara Golemon — 2018-07-13T20:27:24+00:00

URL: https://externals.io/message/102796#102810 · In reply to: #102807

> > […]
> Normally RMs would define the tool versions that are expected
> to be stable and that are always used for the official tarballs.
> In the case of 7.0, it'd be always re2c 0.13.5 and bison 3.0.2,
> where ever I'd produce the tarball (even I don't always use
> Jessie for packaging). We probably should take this into the
> release process docs, if it isn't there already.
>
Well, whichever version we've settled on, I've updated
sgolemon/php-release to allow using arbitrary versions of re2c.
https://github.com/sgolemon/php-release/commit/9197257bc6f2f607476f950733a0c13ab5e3e867

If RE2C_VERSION is set to 1.0.3, it'll swap in the pre-built 1.0.3
binary I've included in the package.
Otherwise, it'll clone the source repo, checkout the requested tag (or
commit hash) and build that).

I'll plan on doing 7.2.9+ with 1.0.3 (unless we decide differently by then).
Remi should stick to 0.13.5 (default) for 7.2.8-final as previously stated.
Doing a dry-run now to see how it looks.

-Sara

---

## #102811 — Anatol Belski — 2018-07-13T21:15:37+00:00

URL: https://externals.io/message/102796#102811 · In reply to: #102810

> -----Original Message-----
> From: php@golemon.com <php@golemon.com> On Behalf Of Sara Golemon
> Sent: Friday, July 13, 2018 10:27 PM
> To: Anatol Belski <ab@php.net>
> Cc: Dmitry Stogov <dmitry@zend.com>; Nikita Popov <nikita.ppv@gmail.com>;
> PHP internals list <internals@lists.php.net>; Stanislav Malyshev
> <smalyshev@gmail.com>; derick@derickrethans.nl; Christoph M. Becker
> <cmbecker69@gmx.de>
> Subject: Re: [PHP-DEV] re2c version(s)
> 
> 
> > […]
> Well, whichever version we've settled on, I've updated sgolemon/php-release to
> allow using arbitrary versions of re2c.
> https://github.com/sgolemon/php-
> release/commit/9197257bc6f2f607476f950733a0c13ab5e3e867
> 
> If RE2C_VERSION is set to 1.0.3, it'll swap in the pre-built 1.0.3 binary I've
> included in the package.
> Otherwise, it'll clone the source repo, checkout the requested tag (or commit
> hash) and build that).
> 
As long as it suffices for the platform targeted for packaging, that'd work. I'd wonder, if we could use this approach to have the exact tools version on Travis. Then it'd be matching with what is tested and released. If that'd be ok, still the Stas concern about other platforms were to be respected. Perhaps we could integrate such a mechanism into the core, but it should fallback to available re2c, bison and whatever, if the given platform can't build the required tool version. As it would be for dev, perhaps it could be ignored, as a release would deliver the generated files. Or, we should still keep the generated files in the repo.

Regards

Anatol

---

## #102812 — Alice Wonder — 2018-07-13T21:27:17+00:00

URL: https://externals.io/message/102796#102812 · In reply to: #102810

> 

*snip*

> Well, whichever version we've settled on, I've updated
> sgolemon/php-release to allow using arbitrary versions of re2c.
> https://github.com/sgolemon/php-release/commit/9197257bc6f2f607476f950733a0c13ab5e3e867
>
> If RE2C_VERSION is set to 1.0.3, it'll swap in the pre-built 1.0.3
> binary I've included in the package.
> Otherwise, it'll clone the source repo, checkout the requested tag (or
> commit hash) and build that).
>
> I'll plan on doing 7.2.9+ with 1.0.3 (unless we decide differently by then).
> Remi should stick to 0.13.5 (default) for 7.2.8-final as previously stated.
> Doing a dry-run now to see how it looks.
>
> -Sara
>

I build 7.1.x packages for CentOS 7 linked against LibreSSL (I don't do 
7.2 because I do every other major release)

I use re2c 0.14.3 because, well, that's what EPEL for CentOS 7 ships.

Is there a real world benefit to upgrading that to the 1.0.3 version in 
my build system? (I suspect relatively easy to do)

Is it likely 7.3 will require newer re2c? My LibreSSL PHP packaging is 
likely switches to 7.3.x branch when 7.3.2 is released.

---

## #102813 — Christoph M. Becker — 2018-07-13T21:32:55+00:00

URL: https://externals.io/message/102796#102813 · In reply to: #102806

> **Checks snaps.php.net**
> ....wait, is this not a thing anymore?

IIRC, that was already gone before I got my php.net account.  Nowadays
users are supposed to checkout from Git.

-- 
Christoph M. Becker

---

## #102814 — Zeev Suraski — 2018-07-13T21:48:32+00:00

URL: https://externals.io/message/102796#102814 · In reply to: #102797

> -----Original Message-----
> From: Nikita Popov [mailto:nikita.ppv@gmail.com]
> Sent: Friday, July 13, 2018 12:26 PM
> To: Dmitry Stogov <dmitry@zend.com>
> Cc: PHP internals list <internals@lists.php.net>; Stanislav Malyshev
> <smalyshev@gmail.com>; derick@derickrethans.nl; Christoph M. Becker
> <cmbecker69@gmx.de>
> Subject: Re: [PHP-DEV] re2c version(s)
> 
> 
> 
> > […]
> 
> I don't think normalizing the version really solves anything. These files should be
> dropped from version control entirely instead. Generated files do not belong in
> version control.
>
> re2c is widely available on Linux distros nowadays (probably specifically because
> PHP uses it) and while there might have been historical ground to bundle these
> generated files, there no longer is one.

I agree that these files shouldn't be in version control, but I think normalizing the version does buy us something - it ensures (or at least tries to) that whomever does generate these files does so with a version that we believe is suitable for the job.  This is of course especially important for the RM's machine - but it's also important for whomever else might be building from a direct checkout and not a source package (I think the files we currently see in source control are indicative of that). 

So why not do both - remove these files from version control, but also update the re2c requirements in configure and makedist..?

And of course we still want to bundle these in our distros - just not track them in our source control.

Zeev

---

## #102819 — Jan Ehrhardt — 2018-07-14T04:02:07+00:00

URL: https://externals.io/message/102796#102819 · In reply to: #102803

Stanislav Malyshev in php.internals (Fri, 13 Jul 2018 11:40:12 -0700):
> > […]
>
>On Linux distros on common platforms (Intel/AMD) - sure. But what if you
>need an uncommon platform, or one that does not run Linux? It's those
>platforms where you'd have to build PHP from source (after all, PHP is
>also widely available as a package on Linux distros anyway) and adding
>another hassle of figuring out how to build a third-party tool - I don't
>think it's a good service to the community.

Directadmin just makes the tarballs from php.net available at every new
release. re2c is not even installed on my Directadmin powered CentOS 6
systems and bison is not used in the build process.

The Directadmin admins surely will not be happy if the generated files
are removed from the tarballs. And the Directadmin users will probably
not be happy either, because chances are high that new releases will
have a time delay. Currently a new PHP release is distributed through
Directadmin on the day of release.
-- 
Jan

---

## #102820 — Sara Golemon — 2018-07-14T04:17:00+00:00

URL: https://externals.io/message/102796#102820 · In reply to: #102819

> 
> 
> Stanislav Malyshev in php.internals (Fri, 13 Jul 2018 11:40:12 -0700):
> > […]
> 
> Directadmin just makes the tarballs from php.net available at every new
> release. re2c is not even installed on my Directadmin powered CentOS 6
> systems and bison is not used in the build process.
> 
> The Directadmin admins surely will not be happy if the generated files
> are removed from the tarballs. And the Directadmin users will probably
> not be happy either, because chances are high that new releases will
> have a time delay. Currently a new PHP release is distributed through
> Directadmin on the day of release.

We're not talking about removing from tarballs, only git.  RMs will continue generating bison and re2c targets so that these are not build requirements for normal users.

-Sara

---

## #102826 — Stanislav Malyshev — 2018-07-14T19:24:37+00:00

URL: https://externals.io/message/102796#102826 · In reply to: #102806

Hi!

> I would offer that this is what official releases and/or snaps.php.net is for.

Do you mean releases would contain the generated files but the regular
source won't? I am not sure then why - if we say our "official" source
has generated files, why keep VCS out of sync with the source? Also,
what if you want to build a version that has not been released yet? I
think keeping VCS in a state where it can not be built in a major system
is not a good thing.

-- 
Stas Malyshev
smalyshev@gmail.com

---

## #102827 — Stanislav Malyshev — 2018-07-14T19:26:22+00:00

URL: https://externals.io/message/102796#102827 · In reply to: #102808

Hi!

> re2c is not a problem for Windows either, it is bundled as part of the
> PHP SDK.

OK then, Windows is not a problem, that makes it better.

-- 
Stas Malyshev
smalyshev@gmail.com

---

## #102875 — Remi Collet — 2018-07-17T05:04:21+00:00

URL: https://externals.io/message/102796#102875 · In reply to: #102814

Le 13/07/2018 à 23:48, Zeev Suraski a écrit :

> So why not do both - remove these files from version control, but also update the re2c requirements in configure and makedist..?
> 
> And of course we still want to bundle these in our distros - just not track them in our source control.

I agree

Perhaps we can also add all the generated files (including configure) in
the tagged versions, so the tag will have same content than the official
archive.


Remi

---

## #102882 — Sara Golemon — 2018-07-17T11:01:11+00:00

URL: https://externals.io/message/102796#102882 · In reply to: #102875

> Le 13/07/2018 à 23:48, Zeev Suraski a écrit :
> Perhaps we can also add all the generated files (including configure) in
> the tagged versions, so the tag will have same content than the official
> archive.
>
Ick, no.  That's the worst outcome IMO.  I don't think they need to be
in git, but if we're going to have them there during tags, then they
should always be there.  One way or the other, not some
middle-of-the-road thing.

-Sara

---

## #102885 — Zeev Suraski — 2018-07-17T11:51:57+00:00

URL: https://externals.io/message/102796#102885 · In reply to: #102882

> On Tue, Jul 17, 2018 at 1:04 AM, Remi Collet <remi@fedoraproject.org>
> wrote:
> > […]
> Ick, no.  That's the worst outcome IMO.  I don't think they need to be
> in git, but if we're going to have them there during tags, then they
> should always be there.  One way or the other, not some
> middle-of-the-road thing.


I can explain why I think that what I propose is the best outcome:
- It ensures that the correct versions of re2c are always used.
- It doesn't track generated files in source control.
- It allows users to build PHP from source on platforms where re2c is
unavailable.

Correct me if I'm wrong, but isn't that exactly what we do with
zend_language_parser.c?  makedist is responsible for generating it so that
it's available in distributions, but it isn't tracked.  Why would
zend_language_scanner.c be any different?  My guess is that it's probably
because at the time we moved to re2c it wasn't nearly as ubiquitous as it
is today and even most developers didn't have access to it, but now that's
changed.

I'm also fine with what Remi proposed which is adding these files
specifically to the source control at the time of tagging, in the spirit of
tracking everything that we actually end up releasing in source control
(and perhaps do that for zend_language_parser.c if we decide that this is
the right thing to do).

Either way - the first step (normalizing re2c versions - updating our r2ec
requirements) seems to make sense.  We can decide about whether or not we
track the generated files in git as we do today, only during tagging or not
at all independently of that.

Zeev

---

Next page: https://externals.io/message/102796.md?page=2
