# Official Preprocessor

- Thread: https://externals.io/message/119451
- Original archive: https://news-web.php.net/php.internals/119451
- Started by someniatko on 2023-02-02T14:19:40+00:00
- Messages: 11
- Reply by email to internals@lists.php.net
- More for agents: search with https://externals.io/search.md?q={query} · RFCs at https://externals.io/rfc.md · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt

---

## #119451 — someniatko — 2023-02-02T14:19:40+00:00

URL: https://externals.io/message/119451#119451

Hi Internals

The main gist:
--------------

I suggest to introduce an official type-checker / static analyser /
preprocessor / transpiler for PHP, somewhat similar to TypeScript in
the JavaScript world, which will allow to introduce the features
otherwise impossible for the PHP world, at the cost of including an
additional transpilation step.

The reasoning:
--------------

You all know there are a lot of problems with the language, especially
large gaps with the current type system, which have large community
demand to overcome, but still cannot be solved due to the paradigm
chosen by PHP. And the paradigm is to be statically typed (to some
extent), but dynamically checked language, which limits the number of
available options due to performance reasons.

The introduction of static types have gradually eliminated the need of
using external means of communicating the types (phpdoc @param and
@return annotations), replacing them with native language constructs,
making the language more expressive and enterprise-ready. However,
there are a lot of things still missing, like types of arrays,
including lists, hash-maps and structure-like arrays, and of course
generic classes and functions. These are still covered nowadays with
external means like phpdoc annotations and static analysers / type
checkers to validate them. This leaves PHP in the odd mixture of
native types and externally validated annotations via comments.

Paradigm of a dynamically checked language also leaves out the problem
of type-checking before running the code. Even though the types are
defined statically, you still have to physically run the code, with
all possible execution paths, to catch a type error. To avoid that you
still would have to use an external typechecker / static analyser,
even if PHP could cover 100% of type demand natively.

Python solves this problem nicely I think. It is dynamically typed,
but it allows a special syntax for static types and also has a
separate static type checker. I think this is really a good approach
and it would be great if PHP followed this path for the beginning, but
we have what we have already.

There were attempts to create preprocessors for PHP (like PHP++ IIRC),
which allowed features like generics, but they did not gain enough
popularity, weren't baked by large enough companies, didn't make their
way to major IDEs support. I believe the only solution is to make an
official one.

Therefore, I suggest the following thing:
1. Introduce a new #declare directive which will mark the file as
requiring pre-processing.
2. Let PHP introduce special features that only work in such
"pre-processing required" files, such as typed lists and generics.
3. Implement a static type-checker in PHP that will verify those typed
lists and generics are indeed used in a valid way.
4. Make the PHP preprocessor output the "normal" PHP file somewhere,
which has list types and generics erased. I think something like that
was suggested earlier many times.

As an alternative to a new #declare feature + validation +
transpilation in "plain old PHP", the Python way of just ignoring
statically checked stuff could be applied. This approach though looks
odd to me in PHP world, because having some stuff checked in runtime,
and some only statically may confuse developers a lot.

Basically, this will split PHP in two languages, one being superset of
other, but checked statically. I realize this might be much of a
paradigm shift, it may require lots of work if accepted (I can only
dream of it though), but still I want to share my thoughts with you
and propose something for consideration.

Regards,
Illia / someniatko

---

## #119452 — someniatko — 2023-02-02T14:21:53+00:00

URL: https://externals.io/message/119451#119452 · In reply to: #119451

I'd like to also emphasize that the concept of compilation and static
type-checking / analysis is not foreign to the PHP community. Major
frameworks like Symfony already have a concept of compiling e.g. a DI
container, and static analyzers like Psalm, PHPStan and others are
actively used.

---

## #119454 — Deleu — 2023-02-02T20:59:55+00:00

URL: https://externals.io/message/119451#119454 · In reply to: #119452

> I'd like to also emphasize that the concept of compilation and static
> type-checking / analysis is not foreign to the PHP community. Major
> frameworks like Symfony already have a concept of compiling e.g. a DI
> container, and static analyzers like Psalm, PHPStan and others are
> actively used.
>
> --
> 
> To unsubscribe, visit: https://www.php.net/unsub.php


I think the idea overall is pretty solid and could get a few community
supporters.  But the devil is in the details.

1. Making it official is a trade-off. It's the highest chance of widespread
adoption and the heaviest burden on PHP Internals.

2. Who's going to fund it, develop it, maintain it? It's a significant
project and a heavy burden.

3. What's the MVP? How do we measure usefulness, adoption and maintenance
complexity?

4. Is it an aborttable idea? If things go south and can't be supported
anymore, how bad for PHP adopters would it be?

5. How would voting on the spec be like? Complete years-work-hours spec or
small incremental functionalities?

6. Is versioning tightly coupled with PHP version or does it have its own
version support system targeting multiple php versions at the same time?

Overall, I think PHP has been constantly losing popularity in favor of
Typescript and a transpiler could create an exciting solution for some of
PHP's problems, but it does not seem like a pet project.

---

## #119455 — Rowan Tommins — 2023-02-02T22:20:18+00:00

URL: https://externals.io/message/119451#119455 · In reply to: #119451

> I suggest to introduce an official type-checker / static analyser /
> preprocessor / transpiler for PHP, somewhat similar to TypeScript in
> the JavaScript world, which will allow to introduce the features
> otherwise impossible for the PHP world, at the cost of including an
> additional transpilation step.


This is something I have thought about a lot over the years, and in some 
ways, it is very appealing; but it does have some problems.

One of the biggest is the concept of "type erasure": in a pre-processed 
or transpiled language, features like Reflection are crippled by not 
knowing the full information present in the source code. More 
importantly, dynamic language features - of which PHP has many - also 
have no access to the type information. That means that a List<Dog> can 
sometimes end up with a Cat in it at run-time, because a dynamic call 
doesn't know that's a problem.

I think it's interesting to look around at different approaches to this 
elsewhere:

You mention Python, which I think has probably the *worst* approach: it 
has official syntax which appears to convey type information, but 
doesn't actually provide any guarantees if you forget to run an external 
tool. Outside of that tool, the type information is just another type of 
comment, just less clearly so than the annotations understood by tools 
like PhpStan and Psalm in today's PHP.

Another approach is TypeScript, which is essentially a complete language 
in its own right, with JavaScript as a compilation target. As I 
understand it, you can call JS code from inside a TS project (and vice 
versa, once the TS is transpiled), but that won't always be seamless, so 
the two languages will tend to end up with separate ecosystems of 
libraries and tooling. Implementing a new language also takes time, and 
ultimately money - in this case, Microsoft's money.

A similar project, closer to PHP's heart, is Hack; the main difference 
being that rather than transpiling to PHP, it initially aimed to compile 
both languages to a common bytecode, to be executed in a shared VM. 
However, it turns out that as well as *adding* features on top of PHP, 
it was necessary for Hack to *remove* features of PHP that weren't 
compatible with a static typing system, and eventually the 
interoperability was dropped altogether. It also has money behind it, in 
this case from Facebook.

Dart, which was written completely from scratch (the money this time is 
Google's), has a type system checked *mostly* at compile-time, but with 
certain run-time assertions inserted where the compiler can't prove 
soundness. The approach seems appealing, but PHP would probably end up 
with a lot of such cases - all the features that the Hack team decided 
to remove - leading to poor performance and confusing developer experience.

I don't have a conclusion to this e-mail, I just thought I'd throw out 
these thoughts.

Regards,

-- 
Rowan Tommins
[IMSoP]

---

## #119456 — olleharstedt@gmail.com — 2023-02-03T09:18:54+00:00

URL: https://externals.io/message/119451#119456 · In reply to: #119451

2023-02-02 15:19 GMT+01:00, someniatko <someniatko@gmail.com>:
> Hi Internals
>
> The main gist:
> --------------
>
> I suggest to introduce an official type-checker / static analyser /
> preprocessor / transpiler for PHP, somewhat similar to TypeScript in
> the JavaScript world, which will allow to introduce the features
> otherwise impossible for the PHP world, at the cost of including an
> additional transpilation step.
>
> The reasoning:
> --------------
>
> You all know there are a lot of problems with the language, especially
> large gaps with the current type system, which have large community
> demand to overcome, but still cannot be solved due to the paradigm
> chosen by PHP. And the paradigm is to be statically typed (to some
> extent), but dynamically checked language, which limits the number of
> available options due to performance reasons.
>
> The introduction of static types have gradually eliminated the need of
> using external means of communicating the types (phpdoc @param and
> @return annotations), replacing them with native language constructs,
> making the language more expressive and enterprise-ready. However,
> there are a lot of things still missing, like types of arrays,
> including lists, hash-maps and structure-like arrays, and of course
> generic classes and functions. These are still covered nowadays with
> external means like phpdoc annotations and static analysers / type
> checkers to validate them. This leaves PHP in the odd mixture of
> native types and externally validated annotations via comments.
>
> Paradigm of a dynamically checked language also leaves out the problem
> of type-checking before running the code. Even though the types are
> defined statically, you still have to physically run the code, with
> all possible execution paths, to catch a type error. To avoid that you
> still would have to use an external typechecker / static analyser,
> even if PHP could cover 100% of type demand natively.
>
> Python solves this problem nicely I think. It is dynamically typed,
> but it allows a special syntax for static types and also has a
> separate static type checker. I think this is really a good approach
> and it would be great if PHP followed this path for the beginning, but
> we have what we have already.
>
> There were attempts to create preprocessors for PHP (like PHP++ IIRC),
> which allowed features like generics, but they did not gain enough
> popularity, weren't baked by large enough companies, didn't make their
> way to major IDEs support. I believe the only solution is to make an
> official one.
>
> Therefore, I suggest the following thing:
> 1. Introduce a new #declare directive which will mark the file as
> requiring pre-processing.
> 2. Let PHP introduce special features that only work in such
> "pre-processing required" files, such as typed lists and generics.
> 3. Implement a static type-checker in PHP that will verify those typed
> lists and generics are indeed used in a valid way.

No need to waste resources implementing a new one when we already have
two very competent already: Psalm and Phpstan? Or why is those not
enough for your use-case?

"The greatest enemy to a perfect solution is a good enough solution
already in place." ;)

Olle

---

## #119460 — flaviohbatista@gmail.com — 2023-02-03T16:38:31+00:00

URL: https://externals.io/message/119451#119460 · In reply to: #119456

> 2023-02-02 15:19 GMT+01:00, someniatko <someniatko@gmail.com>:
> > […]
>
> No need to waste resources implementing a new one when we already have
> two very competent already: Psalm and Phpstan? Or why is those not
> enough for your use-case?
>
> "The greatest enemy to a perfect solution is a good enough solution
> already in place." ;)
>
> Olle
>
> --
> 
> To unsubscribe, visit: https://www.php.net/unsub.php
>
>
Why not follow the steps of other projects that have been created
independently from the core and,
at some point, gained enough support/traction that were incorporated into
it?

---

## #119461 — olleharstedt@gmail.com — 2023-02-03T18:30:00+00:00

URL: https://externals.io/message/119451#119461 · In reply to: #119460

2023-02-03 17:38 GMT+01:00, Flávio Heleno <flaviohbatista@gmail.com>:
> On Fri, Feb 3, 2023 at 6:19 AM Olle Härstedt <olleharstedt@gmail.com>
> wrote:
>
> > […]
> Why not follow the steps of other projects that have been created
> independently from the core and,
> at some point, gained enough support/traction that were incorporated into
> it?

You'd have to rewrite Psalm or Phpstan in C if you want it to be part
of php-src. Or write a completely new project.

Olle

---

## #119462 — Rowan Tommins — 2023-02-03T19:17:27+00:00

URL: https://externals.io/message/119451#119462 · In reply to: #119461

>You'd have to rewrite Psalm or Phpstan in C if you want it to be part
>of php-src. Or write a completely new project.

I don't think that's necessarily true. There are already tools like run-tests written in PHP and shipped with php-src, PECL is written in PHP, and there have even been suggestions that rewriting some included functions in PHP would be a good long-term goal.

The far larger concerns are governance, resources, and release cycles.

Regards,

-- 
Rowan Tommins
[IMSoP]

---

## #119464 — thomas.pakkanen@posteo.de — 2023-02-03T22:48:14+00:00

URL: https://externals.io/message/119451#119464 · In reply to: #119462

Hi,

why not using the existing Typescript programming language and transpile 
it to PHP? :)

There are community examples for doing that:
"ts2php"
https://www.npmjs.com/package/ts2php

Maybe creating an own PHP transpiler for Typescript written in pure PHP 
(community friendly) would be an alternative of creating a complete new 
high ievel language developed in C-programming langage.

Best regards,
Thomas

Am 03.02.2023 20:17 schrieb Rowan Tommins:
> On 3 February 2023 18:30:00 GMT, "Olle Härstedt" 
> <olleharstedt@gmail.com> wrote:
> > […]
> 
> I don't think that's necessarily true. There are already tools like
> run-tests written in PHP and shipped with php-src, PECL is written in
> PHP, and there have even been suggestions that rewriting some included
> functions in PHP would be a good long-term goal.
> 
> The far larger concerns are governance, resources, and release cycles.
> 
> Regards,
> 
> --
> Rowan Tommins
> [IMSoP]

---

## #119467 — someniatko — 2023-02-04T13:23:19+00:00

URL: https://externals.io/message/119451#119467 · In reply to: #119454

Hi,

These all are valid concerns! I cannot speak about management and cost
issues though, as I am not an active PHP dev team member.

> 3. What's the MVP? How do we measure usefulness, adoption and maintenance complexity?

As a shot in the dark, the first step could be implementation of typed
lists in my opinion. Do not implement full-blown static analyser nor
the type checker for each possible case, just only check the usage of
typed lists within the pre-processed files. Assume that whatever comes
from the non-preprocessed ones is correct. Make only simple naive
assumptions for all internal array_ functions as well as basic array
operations. If there is some runtime magic going on with constructing
the list, just fallback to infering the type as a simple array. For
such cases, allow a special attribute which will tell the type checker
this variable / property etc will indeed contain the typed list, like
a @var annotation for IDEs / static analysers nowadays.

The preprocessor / type checker **should not aim to 100% guarantee the
correct types will be in the runtime**: TypeScript does this neither.
Rowan has mentioned Dart language in a parallel thread, which makes
most of the checks statically, but for some cases the compiler still
adds runtime checks. Rowan warned that PHP might have a lot of such
cases possible though. But I still think this also might be a good
idea for MVP.


> 4. Is it an aborttable idea? If things go south and can't be supported anymore, how bad for PHP adopters would it be?

I don't think any of PHP new features that go live are abortable. This
affects not only my suggestion, but also everything what has been done
for PHP in past. Once a feature is in the language and released to the
public, you cannot revert it. There must be a long multi-year process
preceded with a deprecation at least. A concept of experimental
features was suggested earlier, but this is completely different
topic, so I believe we should not go into that rabbit hole here.
Overall, I consider this question orthogonal (or irrelevant if you
prefer) to the original topic, let's take as an axiom that there is no
way back.


> 5. How would voting on the spec be like? Complete years-work-hours spec or small incremental functionalities?

Small incremental functionalities. Gradually more stuff could be
proven statically, more complex cases covered, and while they are not,
an attribute / static assertion could be used. Maybe some of such
assertions could be allowed to be preserved in runtime, at the cost of
some performance penalty, IDK. But I believe we should leave such
choice to developers, whether they want the checks in runtime as well.


> 6. Is versioning tightly coupled with PHP version or does it have its own version support system targeting multiple php versions at the same time?

I strongly believe it must go hand in hand with the PHP version, and
not as a separate tool, because I see it as a feature of a PHP
language itself.


> Overall, I think PHP has been constantly losing popularity in favor of Typescript and a transpiler could create an exciting solution for some of PHP's problems, but it does not seem like a pet project.

Sure, it should be considered seriously and requires some upfront
planning. Though, well, PHP itself started as a pet project :D

---

## #119466 — Rowan Tommins — 2023-02-04T14:07:27+00:00

URL: https://externals.io/message/119451#119466 · In reply to: #119464

>Hi,
>
>why not using the existing Typescript programming language and transpile it to PHP? :)
>
>There are community examples for doing that:
>"ts2php"
>https://www.npmjs.com/package/ts2php


I can think of a few reasons.

Firstly, TypeScript is a language designed with a very specific purpose: to build a statically typed language on top of the existing functionality of JavaScript. Although some of the decisions that follow from that would also aid compatibility with PHP, some would lead to serious mismatches - for example, JavaScript has an entirely different object model, with prototypes rather than classes [1], and methods more closely resembling closures bound to the object.

Secondly, designing the language is only half the battle, probably less. It requires expertise to do well, but the bulk of the time is spent *implementing* that design. It can be deceptively easy to implement a proof of concept that can run some carefully chosen sample code, but actually making something production ready is a much bigger effort. It doesn't surprise me at all that the project you linked had some initial beta releases, and then went quiet for several years.

Thirdly, if this approach made sense, TypeScript wouldn't need to exist - Microsoft could have written tooling to translate C# to JavaScript instead. The reason languages like TypeScript are created is specifically to bridge existing code and existing knowledge from an existing language. That might mean running in the same run-time as the older language, or just being similar enough to allow a supervised one-off conversion. For PHP, that's the thinking behind Hack (from HHVM), Zephir (from the Phalcon framework), and more recently Ara (https://ara-lang.io/).

To succeed, a new language needs a rare combination of committed resources (which generally means commercial backing to pay multiple full time developers), plus a critical mass of users. Zephir lacked both; Hack had backing inside Facebook, but didn't gain traction outside it; I wish Saif luck with Ara, but I'm not holding my breath.


[1] Yes, there's now a "class" keyword, but it's just syntactic sugar; the fundamental inheritance model still consists of prototype chains.

-- 
Rowan Tommins
[IMSoP]
