Hi internals,
I would like to gauge reaction before writing an RFC.
PHP's scanner defines identifiers on bytes rather than code points:
LABEL [a-zA-Z_\x80-\xff][a-zA-Z0-9_\x80-\xff]*
Every byte >= 0x80 is accepted, so UTF-8 identifiers work by accident;
the manual says as much. There is no encoding validation, no
normalisation requirement and no UAX #31 conformance. As a result
${"\xFF\xFE"} is a valid variable name, and $x and $x<U+00A0> are two
distinct variables that render identically.
I am not proposing that PHP accept more characters -- it already accepts
everything. I am proposing a per-file declare under which the accepted
set is specified: well-formed UTF-8, UAX31-R1-2 with the standard
Default-Ignorable Exclusion Profile, and NFC required rather than
applied. Identifiers consisting only of bytes below 0x80 are never
examined, so existing code pays nothing.
To find out what this would break I surveyed the 250 most-downloaded
Packagist packages and 250 GitHub repositories -- 168,604 PHP files --
using ext/tokenizer. The Packagist corpus contains exactly one
non-ASCII identifier, and no identifier in either corpus is non-NFC.
Tooling, raw data and per-identifier CSVs are here:
https://github.com/Otzie2023/PHP
The patch would not touch the re2c scanner: the byte rule is already
maximally permissive and cannot split a UTF-8 sequence, so a strict
mode only ever rejects and the check can run after the token is formed.
About 13.7 KiB of generated tables, with no dependency on intl, ICU or
mbstring. I would write and maintain it.
Is this worth an RFC, or is there an objection I should know about
before I spend the time?
Regards,
Luca Rodenhaeuser
-----Original Message-----
From: otzelot2021@outlook.de otzelot2021@outlook.de
Sent: Wednesday, August 26, 2026 6:15 PM
To: internals@lists.php.net
Subject: [PHP-DEV] [Concept] declare(strict_identifiers=1)[..] I am proposing a per-file declare under which the accepted set
is specified: well-formed UTF-8, UAX31-R1-2 with the standard Default-
Ignorable Exclusion Profile, and NFC required rather than applied. Identifiers
consisting only of bytes below 0x80 are never examined, so existing code
pays nothing.
Hey Luca,
To prevent errors? I must admit I don't rly understand all the terms.
I assume it implies identifiers should be more visible/readable, right?
To find out what this would break I surveyed the 250 most-downloaded
Why would anything break if it's per-file?
Packagist packages and 250 GitHub repositories -- 168,604 PHP files -- using
ext/tokenizer. The Packagist corpus contains exactly one non-ASCII identifier,
and no identifier in either corpus is non-NFC.
Do I understand it correctly that by adding that declare to 168604 you would uncover
a single risky identifier? Not that convincing...
Would it be fair to say that the same constraints can be enforced by linters/cs tooling?
BR,
Juris
Hey Juris,
To prevent errors? I must admit I don't rly understand all the terms.
I assume it implies identifiers should be more visible/readable, right?
Not readability -- unambiguity. Three concrete things, no jargon:
-
Two identifiers that look identical on screen are currently two
different identifiers. $x and $x followed by U+00A0 NO-BREAK SPACE
are separate variables. So are "a" plus a combining diaeresis and the
single character U+00E4; both display as a-umlaut. -
Text that is not valid UTF-8 at all is currently a valid identifier.
${"\xFF\xFE"} = 1; compiles. -
Nobody can state what a PHP identifier is except by copying the byte
class out of the scanner. The manual does exactly that, and so does
PHP-Parser.
The declare says: in this file, an identifier is well-formed UTF-8, is a
Unicode identifier in the sense of UAX #31 (the Unicode annex that
defines this for programming languages), and is written in one canonical
spelling. Anything else is a compile error.
Why would anything break if it's per-file?
Nothing breaks. That was bad wording on my part.
The survey answers a different question: if a maintainer adds the declare
to a file they already have, does it still compile? That is adoption
friction, not breakage. It is also a calibration check -- a rule that
rejected a lot of legitimate existing code would be the wrong rule, and I
wanted to know that before proposing it rather than after.
Do I understand it correctly that by adding that declare to 168604 you
would uncover a single risky identifier? Not that convincing...
That number is the cost, not the benefit, and I should have separated the
two more clearly.
The single Packagist finding is what would stop compiling: symfony/cache
declares a class whose entire name is the single byte 0xA9. That is the
whole measured adoption cost across the 250 most-installed packages.
What the rule catches can only be measured where non-ASCII identifiers
actually occur, which is not in the top Packagist packages. In the GitHub
corpus, 33 of 136 non-ASCII identifiers fail the rule and 11 contain an
invisible character. One is live code: the Alipay OpenAPI SDK has
$chrtext<U+00A0> = null;
// ...
openssl_public_encrypt($block, $chrtext<U+00A0>, $res);
with a no-break space inside the name. It works only because the typo is
consistent throughout the function. Anyone who types $chrtext normally
gets a different variable, passed by reference, that stays null. Five
vendored copies across four unrelated projects in my sample.
But you are right that these are small numbers, and I would rather say so
than dress them up. The case does not rest on the bug count.
Would it be fair to say that the same constraints can be enforced by
linters/cs tooling?
Partly yes, and I will concede that plainly: UTF-8 validity, NFC and
UAX #31 conformance are all statically checkable. My survey tool is
exactly such a linter, written in PHP, and it is in the repository.
Three things it cannot do.
It cannot define the language. "What is a valid PHP identifier" currently
has no answer other than "whatever bytes the scanner happened to accept",
which is why the manual, PHP-Parser, every IDE and every static analyser
separately reverse-engineer the same byte class. A declare makes it a
versioned, testable statement.
It does not travel with the code. A declare is in the source file; a lint
configuration is in someone's toolchain. The file keeps its guarantee
after composer install, and the rule also covers generated and eval'd
code that never reaches a linter.
And it cannot touch semantics. Case-insensitive symbol lookup folds ASCII
only, so Stra<U+00DF>e and STRA<U+00DF>E are the same class while
Stra<U+1E9E>e is not, and Strasse and STRASSE are. No linter can fix
that, and I do not think it can sensibly be fixed before there is a
definition of what an identifier is. I deliberately kept case folding out
of this proposal, but that is the thing underneath it.
The same objection would apply to strict_types -- static analysers check
types, so why does the engine need a per-file declaration? I do not think
the answer there was "it doesn't", though I accept the parallel is not
exact, since strict_types changes runtime behaviour and this does not.
This is the weakest point of the proposal and you found it on the first
reading. If the list's view is that specifying the rule and leaving
enforcement to tooling is the right scope, that is a smaller and possibly
better RFC, and I would rather establish that now than after writing the
patch.
Thanks for the questions.
Regards,
Luca
The survey answers a different question: if a maintainer adds the declare
to a file they already have, does it still compile? That is adoption
friction, not breakage. It is also a calibration check -- a rule that
rejected a lot of legitimate existing code would be the wrong rule, and I
wanted to know that before proposing it rather than after.Do I understand it correctly that by adding that declare to 168604 you
would uncover a single risky identifier? Not that convincing...That number is the cost, not the benefit, and I should have separated the
two more clearly.The single Packagist finding is what would stop compiling: symfony/cache
declares a class whose entire name is the single byte 0xA9. That is the
whole measured adoption cost across the 250 most-installed packages.
If this is so rarely seen in the wild (something that should be verified with more than 250 packages), why make it an option? Just plan that PHP 9 will enforce UTF-8-or-GTFO rules on identifiers, Symfony updates one oddball class, and we move on with life. 99.99% of developers won't notice anything happened.
--Larry Garfield
The survey answers a different question: if a maintainer adds the declare
to a file they already have, does it still compile? That is adoption
friction, not breakage. It is also a calibration check -- a rule that
rejected a lot of legitimate existing code would be the wrong rule, and I
wanted to know that before proposing it rather than after.Do I understand it correctly that by adding that declare to 168604 you
would uncover a single risky identifier? Not that convincing...That number is the cost, not the benefit, and I should have separated the
two more clearly.The single Packagist finding is what would stop compiling: symfony/cache
declares a class whose entire name is the single byte 0xA9. That is the
whole measured adoption cost across the 250 most-installed packages.If this is so rarely seen in the wild (something that should be verified with more than 250 packages), why make it an option? Just plan that PHP 9 will enforce UTF-8-or-GTFO rules on identifiers, Symfony updates one oddball class, and we move on with life. 99.99% of developers won't notice anything happened.
--Larry Garfield
Is it important enough to have this memory footprint added to each PHP process though?
About 13.7 KiB of generated tables
cheers
Derick
در تاریخ پنجشنبه ۲۷ اوت ۲۰۲۶، ۰۱:۲۳ Derick Rethans derick@php.net نوشت:
On 26 August 2026 22:24:06 BST, Larry Garfield larry@garfieldtech.com
wrote:The survey answers a different question: if a maintainer adds the
declare
to a file they already have, does it still compile? That is adoption
friction, not breakage. It is also a calibration check -- a rule that
rejected a lot of legitimate existing code would be the wrong rule, and
I
wanted to know that before proposing it rather than after.Do I understand it correctly that by adding that declare to 168604
you
would uncover a single risky identifier? Not that convincing...That number is the cost, not the benefit, and I should have separated
the
two more clearly.The single Packagist finding is what would stop compiling: symfony/cache
declares a class whose entire name is the single byte 0xA9. That is the
whole measured adoption cost across the 250 most-installed packages.If this is so rarely seen in the wild (something that should be verified
with more than 250 packages), why make it an option? Just plan that PHP 9
will enforce UTF-8-or-GTFO rules on identifiers, Symfony updates one
oddball class, and we move on with life. 99.99% of developers won't notice
anything happened.--Larry Garfield
Is it important enough to have this memory footprint added to each PHP
process though?About 13.7 KiB of generated tables
cheers
Derick
Hi Derick,
Good point. The ~13.7 KiB footprint comes from the static lookup tables
generated for fast classification.
A couple of aspects regarding how this is handled / can be optimized:
- Shared Memory (
.rodata): Since these tables are declared asstatic const, in standard multi-process setups (e.g. PHP-FPM), they reside in
read-only memory pages shared across processes rather than allocating
per-process heap memory. - Compacting / Range Encoding: We can definitely look into compressing the
lookup tables (e.g., using run-length/interval encoding or two-stage lookup
tables) to bring the table size well under a few kilobytes if the raw table
footprint is a concern.
I’m happy to explore compressing the tables or benchmarking the memory
impact across different setups to ensure the footprint remains negligible.
Best regards,
Sepehr
Is it important enough to have this memory footprint added to each PHP
process though?About 13.7 KiB of generated tables
It is not added to each process, and I should have been clearer about what
the number is.
The tables are static const arrays, so they land in .rodata. That section
is mapped read-only, and every process started from the same binary maps
the same physical pages. The marginal cost of an additional FPM worker is
page table entries, not the data.
For scale, measured on Ubuntu 24.04 with PHP 8.3.6:
mbstring.so 1,209,216 bytes
of which .rodata 780,276 bytes
php binary 5,784,016 bytes
PHP therefore already carries roughly 760 KiB of read-only Unicode tables
in mbstring alone, for character set conversion. On the source side the
largest single files in php-src are unicode_table_uhc.h at 402 KB,
unicode_table_cns11643.h at 373 KB, unicode_table_jis.h at 308 KB and
unicode_data.h at 267 KB.
13.7 KiB is 1.16 % of mbstring.so and 0.24 % of the php binary.
Two caveats I would rather state than have found.
The 13.7 KiB is a naive figure: sorted uint32 range pairs, 693 ranges for
Start, 807 for Continue, 251 for the NFC quick check, with the profile
subtraction already applied. A two-stage table would be smaller. I have
not measured by how much and would rather not quote a number I have not
measured.
The tables are linked unconditionally, so the footprint does not depend on
whether any file actually uses the declare. For the same reason it does
not scale with the number of processes either.
Runtime cost is a separate question and is zero for ASCII. The first thing
the check does is scan for a byte >= 0x80; an identifier made only of
ASCII returns immediately, which is every identifier in essentially all
existing code.
One clarification for the list, since messages have appeared in this
thread that read as if speaking for the proposal: I am the only person
working on it, none of those replies were coordinated with me, and I would
ask that my own messages be taken as its position.
Regards,
Luca
در تاریخ پنجشنبه ۲۷ اوت ۲۰۲۶، ۰۰:۵۶ Larry Garfield larry@garfieldtech.com
نوشت:
The survey answers a different question: if a maintainer adds the declare
to a file they already have, does it still compile? That is adoption
friction, not breakage. It is also a calibration check -- a rule that
rejected a lot of legitimate existing code would be the wrong rule, and I
wanted to know that before proposing it rather than after.Do I understand it correctly that by adding that declare to 168604 you
would uncover a single risky identifier? Not that convincing...That number is the cost, not the benefit, and I should have separated the
two more clearly.The single Packagist finding is what would stop compiling: symfony/cache
declares a class whose entire name is the single byte 0xA9. That is the
whole measured adoption cost across the 250 most-installed packages.If this is so rarely seen in the wild (something that should be verified
with more than 250 packages), why make it an option? Just plan that PHP 9
will enforce UTF-8-or-GTFO rules on identifiers, Symfony updates one
oddball class, and we move on with life. 99.99% of developers won't notice
anything happened.--Larry Garfield
Fair point, Larry.
Moving to enforce strict UTF-8 identifier rules in PHP 9 would definitely
simplify things and eliminate edge cases cleanly. The main goal here was to
highlight the current ambiguity and explore whether a transitional path or
immediate strictness is preferred. Doing a broader ecosystem check before
finalizing the PHP 9 deprecation path makes total sense.
If this is so rarely seen in the wild (something that should be verified
with more than 250 packages), why make it an option? Just plan that
PHP 9
will enforce UTF-8-or-GTFO rules on identifiers, Symfony updates one
oddball class, and we move on with life. 99.99% of developers won't
notice anything happened.
I ran it: the top 5,000 Packagist packages by downloads, 4,863 of them
resolvable and non-empty, 520,802 PHP files, 2.4 GB. Raw scanner output,
per-identifier CSVs and the corpus manifest are in the repository.
It is not one oddball class.
1,447 non-ASCII identifiers across 25 packages. 1,312 of them -- 91 % --
are in markrogoyski/math-php, and they are neither fixtures nor accidents.
They are variable names in src/, and they spell the formula:
$n! = self::factorial($n);
$∑ = 0;
protected $d₁;
$π = \M_PI;
$│∑│ = $∑->det();
$√⟮2π⟯ᵏ│∑│ = \sqrt((2 * $π) ** $k * $│∑│);
1,247 of the 1,312 are T_VARIABLE. 888 of them would stop compiling: the
offenders are U+27EE/U+27EF mathematical flattened parentheses, subscript
and superscript digits, U+2211 summation, U+2212 minus. None of those is
in XID_Continue.
So the honest answer to "99.99 % won't notice" is that one library would
notice 888 times, and its author chose that style deliberately and has
shipped it for years.
The rest of the picture argues the other way, though, and I would rather
give you both halves.
Outside math-php there are 135 non-ASCII identifiers, and only 25 fail --
15 of which are one test file in hoa/console with arrow characters in
method names (case_move_↑, U+2191). Everything else already conforms:
mjaschen/phpgeo 22, all conforming: $φ, $λ, $sinλ, $cos2σM in
src/Bearing/BearingEllipsoidal.php -- Vincenty
geodesy, Greek letters only, no symbols
tracy/tracy 14, all conforming: $ʟ_tmp, $ʟ_tag in Latte
templates, U+029F as a namespace prefix so internal
variables cannot collide with user ones
wsdltophp/... 44, all conforming: setСубъектРФ, ApiАдресРФ --
generated accessors over a Russian WSDL schema,
where the names come from the schema
And across all 520,802 files, zero identifiers are not in NFC.
So the cost of making this mandatory is specific and nameable rather than a
long tail: it outlaws mathematical symbol notation in identifiers. Two
packages do that. Everything else in the top 5,000 that uses non-ASCII
identifiers already satisfies the rule and would not notice.
That may still be the right call -- "PHP identifiers are letters, not
notation" is a defensible position and I am not arguing against it. But it
is a decision to withdraw something that works today, not a cleanup of an
oddity, and I would rather the list took it with the number in front of it.
One correction to something I said in the other subthread: I told Chris
there was not one identifier mixing Latin with Cyrillic or Greek. At 250
packages that was true. At 5,000 there are 41 outside math-php, and they
are benign -- a code generator putting an ASCII "set" in front of a
Cyrillic schema element. My mixed-script metric flags the script pair, not
the risk, and I should have said so.
Regards,
Luca
Not readability -- unambiguity. Three concrete things, no jargon:
Two identifiers that look identical on screen are currently two
different identifiers. $x and $x followed by U+00A0 NO-BREAK SPACE
are separate variables. So are "a" plus a combining diaeresis and the
single character U+00E4; both display as a-umlaut.Text that is not valid UTF-8 at all is currently a valid identifier.
${"\xFF\xFE"} = 1; compiles.Nobody can state what a PHP identifier is except by copying the byte
class out of the scanner. The manual does exactly that, and so does
PHP-Parser.The declare says: in this file, an identifier is well-formed UTF-8, is a
Unicode identifier in the sense of UAX #31 (the Unicode annex that
defines this for programming languages), and is written in one canonical
spelling. Anything else is a compile error.
Hi Luca & list,
Reading this and seeing you talk about making it a compile time error,
raises the question for me of how this will interact with variable
variables which don't comply with the proposed rules - AFAICS those
wouldn't be able to be a compile time error and they also wouldn't have
been found in the scan of Packagist files.
I imagine "on the fly" class creation, like when mocking code may also
run into issues with this up to a point ?
Those are also the things which static analysis of code would not be
able to find or flag (if this were left to static analysis).
Curious to hear your thoughts on this.
Smile,
Juliette
raises the question for me of how this will interact with variable
variables which don't comply with the proposed rules - AFAICS those
wouldn't be able to be a compile time error and they also wouldn't have
been found in the scan of Packagist files.
Correct on both counts. Claude Pache gave me better vocabulary for the
first half than I had, and I went and measured the second half rather than
guessing at it.
PHP distinguishes names from identifiers. An identifier is a lexical token
the scanner produces from source text; a name is any string that reaches a
symbol table, and that set is far larger -- you can create a variable named
"+!" or a class alias named "" today. What I described governs identifiers
only. $$name, define(), class_alias() and property names materialised by
json_decode() are untouched. That is not a hole I carved out for
convenience; it is a line PHP already draws and already enforces
syntactically.
On your second point: you were right that my scan could not see these, so
I extended it. Names built from a run-time string remain unmeasurable by
anyone, but the statically visible subset is not -- literals in
name-creating and name-looking-up positions: ${'...'}, ->{'...'},
?->{'...'}, Foo::${'...'}, define(), constant(), class_alias(),
property_exists(), method_exists() and friends.
Across the same 4,863 packages and 520,802 files, that turns up five
non-ASCII names, in two packages. All five, in full:
halaxa/json-machine, src/TokensWithDebugging.php:39, under the author's own
comment "Treat UTF-8 BOM bytes as whitespace":
${"\xEF"} = ${"\xBB"} = ${"\xBF"} = 0;
Three variables named after the individual bytes of the UTF-8 BOM, used as
a lookup table alongside ${' '}, ${"\n"}, ${'{'} and so on. None of the
three is valid UTF-8 on its own, and none of them could be written as an
identifier at all.
rowbot/url (vendored into wp-php-toolkit/data-liberation),
tests/WhatWg/URLSearchParamsConstructorTest.php:209-210:
$obj3->{"c\u{D83D}"} = '23';
$obj3->{"d\u{1234}"} = 'foo';
WHATWG URL conformance test data. U+D83D is a lone high surrogate, which
PHP encodes as ED A0 BD and which is not well-formed UTF-8; U+1234 is
ordinary Ethiopic. Four of the five would fail the identifier rule, and
that is the point rather than a problem: every one of them is a
deliberate use of the name syntax precisely because it is not an
identifier.
So as far as I can measure it, the existing split is doing its job. People
who need non-identifier names reach for the name syntax, rarely and on
purpose. I would not have believed that number before running it, and I
would not have run it if you had not asked.
I imagine "on the fly" class creation, like when mocking code may also
run into issues with this up to a point ?
This one has a definite answer, which I checked rather than assumed: code
passed to eval() does not inherit strict_types from the calling file. It
is compiled as its own unit and may carry its own declare; include behaves
the same way. So under an opt-in model, PHPUnit, Mockery and Prophecy
generating classes at run time sit outside it entirely. Under a mandatory
model they would be checked, but a generated mock name derives from the
mocked class, so a non-conforming name only appears if the class being
mocked already had one.
Those are also the things which static analysis of code would not be able
to find or flag (if this were left to static analysis).
That is the strongest argument for putting any of this in the engine that
anyone has made in this thread, including me, and I do not want to claim
more from it than it gives.
A linter can never reach run-time names. But the engine as I described it
does not reach them either. The difference is that the engine could be
extended there and a linter could not -- and extending it means validating
every symbol table insertion, a cost paid on every dynamic property write,
and json_decode() throwing on untrusted input. I think that is a much
larger and probably worse proposal. It is, though, a real option rather
than an impossible one, which is more than the tooling route offers.
The scanner change and the raw output are in the repository.
Regards,
Luca
Am 26.08.2026 um 17:15 schrieb otzelot2021@outlook.de:
I am not proposing that PHP accept more characters -- it already accepts
everything. I am proposing a per-file declare under which the accepted
set is specified: well-formed UTF-8, UAX31-R1-2 with the standard
Default-Ignorable Exclusion Profile, and NFC required rather than
applied. Identifiers consisting only of bytes below 0x80 are never
examined, so existing code pays nothing.
What problem would this restriction solve?
Is it about an code smuggling attack vector using code obfuscation with indistinguishable Unicode sequences?
The patch would not touch the re2c scanner: the byte rule is already
maximally permissive and cannot split a UTF-8 sequence, so a strict
mode only ever rejects and the check can run after the token is formed.
About 13.7 KiB of generated tables, with no dependency on intl, ICU or
mbstring. I would write and maintain it.Is this worth an RFC, or is there an objection I should know about
before I spend the time?
Not sure if I think it is worth the effort but maybe you can shine some light on why we want this.
I'm currently leaning to -1 on this,
- Chris
What problem would this restriction solve?
Is it about an code smuggling attack vector using code obfuscation with
indistinguishable Unicode sequences?
No, and I would rather decline that framing than borrow it.
I did look for it. Across both corpora -- 168,604 files -- there is not
one identifier mixing Latin with Cyrillic or Greek, and not one that maps
to a plausible ASCII identifier under homoglyph substitution. The
proposal does not reject confusables and the draft says so explicitly.
UAX #31 takes the same position: for programming languages, spoofing is
better addressed by higher-level diagnostics than in the lexer.
What it does solve, ordered by how much I think each is actually worth:
-
PHP has no definition of an identifier. The only answer to "what is a
valid PHP identifier" is "whatever bytes the scanner accepted", which
is why the manual, PHP-Parser, every IDE and every static analyser
each copy out the same byte class. That is a language-definition gap,
not a bug report. -
Identifiers that render identically are different identifiers. A
no-break space or a decomposed umlaut inside a name is invisible in
every editor. Real, but rare: 11 instances in 168,604 files. -
Case-insensitive lookup folds ASCII only. Stra<U+00DF>e and
STRA<U+00DF>E are the same class; Stra<U+1E9E>e is not, and Strasse is
not. That rule is coherent only if identifiers are ASCII.
Taken alone, (2) is thin. Juris made the same point in the other
subthread and he is right; I am not going to inflate it.
Which leaves the question I think has to be answered before this is worth
either of our time:
PHP accepts non-ASCII identifiers today, and not by decision. The manual
says outright that PHP "doesn't support Unicode variable names" and that
they work because of how UTF-8 happens to encode. So either they are a
supported feature, in which case they need a definition and (3) is an
inconsistency in the engine -- or they are not, in which case that ought
to be a decision somebody made rather than a note explaining an accident,
and I will drop this.
I would rather have a clear answer than a favourable one, because
anything else I might propose in this area depends on it.
If your -1 rests on "not supported, and we are not going to build on it",
I would sooner have that now than after writing a patch. It is a
legitimate answer and it settles the question.
Regards,
Luca
Am 26.08.2026 um 21:57 schrieb Luca Rodenhäuser otzelot2021@outlook.de:
What it does solve, ordered by how much I think each is actually worth:
- PHP has no definition of an identifier. The only answer to "what is a
valid PHP identifier" is "whatever bytes the scanner accepted", which
is why the manual, PHP-Parser, every IDE and every static analyser
each copy out the same byte class. That is a language-definition gap,
not a bug report.
I'm not sure why you consider a formal definition like
LABEL [a-zA-Z_\x80-\xff][a-zA-Z0-9_\x80-\xff]*
not a definition. Personally I find this a simpler definition (and easier to implement in a parser) than something like
UTF-8, UAX31-R1-2 with the standard Default-Ignorable Exclusion Profile
But then again I'm not really using non-ASCII identifiers today.
- Identifiers that render identically are different identifiers. A
no-break space or a decomposed umlaut inside a name is invisible in
every editor. Real, but rare: 11 instances in 168,604 files.
I understand your point. But I'm not so worried about accidental mixups here. And this is also something an LSP or Linter can help you with if it a real concern for you.
- Case-insensitive lookup folds ASCII only. Stra<U+00DF>e and
STRA<U+00DF>E are the same class; Stra<U+1E9E>e is not, and Strasse is
not. That rule is coherent only if identifiers are ASCII.
Case-insensitive folding adds another problem: Would you be using IntlChar::FOLD_CASE_DEFAULT or IntlChar::FOLD_CASE_EXCLUDE_SPECIAL_I to fold "I"? Or would you base it on a language setting?
In general I think most people consider the case-folding for identifiers nowadays to be a bug, not a feature, so I would probably rather try to reduce than extend it.
Regards,
- Chris
در تاریخ پنجشنبه ۲۷ اوت ۲۰۲۶، ۰۲:۵۷ Christian Schneider <
cschneid@cschneid.com> نوشت:
Am 26.08.2026 um 21:57 schrieb Luca Rodenhäuser otzelot2021@outlook.de:
What it does solve, ordered by how much I think each is actually worth:
- PHP has no definition of an identifier. The only answer to "what is a
valid PHP identifier" is "whatever bytes the scanner accepted", which
is why the manual, PHP-Parser, every IDE and every static analyser
each copy out the same byte class. That is a language-definition gap,
not a bug report.I'm not sure why you consider a formal definition like
LABEL [a-zA-Z_\x80-\xff][a-zA-Z0-9_\x80-\xff]*
not a definition. Personally I find this a simpler definition (and easier
to implement in a parser) than something like
UTF-8, UAX31-R1-2 with the standard Default-Ignorable Exclusion
ProfileBut then again I'm not really using non-ASCII identifiers today.
- Identifiers that render identically are different identifiers. A
no-break space or a decomposed umlaut inside a name is invisible in
every editor. Real, but rare: 11 instances in 168,604 files.I understand your point. But I'm not so worried about accidental mixups
here. And this is also something an LSP or Linter can help you with if it a
real concern for you.
- Case-insensitive lookup folds ASCII only. Stra<U+00DF>e and
STRA<U+00DF>E are the same class; Stra<U+1E9E>e is not, and Strasse is
not. That rule is coherent only if identifiers are ASCII.Case-insensitive folding adds another problem: Would you be using
IntlChar::FOLD_CASE_DEFAULT or IntlChar::FOLD_CASE_EXCLUDE_SPECIAL_I to
fold "I"? Or would you base it on a language setting?In general I think most people consider the case-folding for identifiers
nowadays to be a bug, not a feature, so I would probably rather try to
reduce than extend it.Regards,
- Chris
Hi Chris,
Thanks for the solid points. Let me clarify the perspective behind these:
-
Lexer simplicity vs. Semantic definition:
[a-zA-Z_\x80-\xff] is indeed trivial for the lexer, but it isn't an
identifier specification in terms of character semantics—it's essentially
"ASCII identifiers plus any high byte". This was originally a pragmatic way
to allow Latin-1 / UTF-8 bytes to pass through unchanged. The problem
arises when we consider what an identifier semantically is across tooling,
ASTs, and static analyzers versus raw byte streaming. -
Invisible characters and Linters:
You're right that linters/LSPs can catch these, but language specifications
usually define identifier boundaries (such as TR31 / UAX #31) precisely so
that the baseline definition of a valid symbol doesn't require third-party
tooling to reject canonically confusing or invisible code points. -
Case Folding:
I completely agree with your assessment here. Extending ASCII case-folding
to full Unicode casing (with all the locale subtleties like the
dotted/dotless Turkish I) would be opening Pandora's box. The argument
wasn't necessarily to expand case-folding to Unicode, but rather to
highlight the existing inconsistency: PHP treats identifiers as
case-insensitive on the ASCII plane while allowing non-ASCII bytes that are
strictly case-sensitive.
If the consensus leans toward treating case-insensitivity as historical
baggage, clarifying the identifier grammar and transition paths (especially
looking ahead to PHP 9 / UTF-8 requirements) is exactly the right
discussion to have.
Best regards,
Sepehr
I'm not sure why you consider a formal definition like
LABEL [a-zA-Z_\x80-\xff][a-zA-Z0-9_\x80-\xff]*
not a definition. Personally I find this a simpler definition (and easier
to implement in a parser) than something like
UTF-8, UAX31-R1-2 with the standard Default-Ignorable Exclusion
Profile
You are right and I overreached. It is a definition, it is precise, and it
is far simpler to implement -- one character class against roughly 13.7 KiB
of generated tables. That is a real cost and I should not have written as
if PHP had nothing.
What I should have said is that it is a complete definition of bytes and
a silent one about characters. It fixes no encoding, so the same file is
two different programs depending on how you read it, and it cannot tell a
character from half of one.
The part that actually bothers me is that PHP does not stay at the byte
level. Case-insensitive symbol lookup is defined over characters -- ASCII
characters. So there is already one character-level rule bolted onto a
byte-level definition, and the seam between them is where
Stra<U+00DF>e/STRA<U+00DF>E/Stra<U+1E9E>e/Strasse comes from.
But then again I'm not really using non-ASCII identifiers today.
Then nothing here would ever fire for you. The check I described never
looks at an identifier whose bytes are all below 0x80, which is every
identifier in most codebases. That is not a footnote -- it is why the thing
can be considered at all.
And this is also something an LSP or Linter can help you with if it a
real
concern for you.
I answered that at length to Juris and to Rowan and will not repeat it.
Two things I did not know then and do now: the measurement is 68 identifiers
with an invisible character across 520,802 files, and not all of them are
accidents -- math-php puts U+00A0 inside variable names deliberately. That
makes it warning-shaped rather than error-shaped, which is closer to your
position than to my original one.
Case-insensitive folding adds another problem: Would you be using
IntlChar::FOLD_CASE_DEFAULT or IntlChar::FOLD_CASE_EXCLUDE_SPECIAL_I to
fold "I"? Or would you base it on a language setting?
PHP has already answered this, which I think settles it in your favour.
The 8.2 RFC "Locale-independent case conversion" used precisely the Turkish
dotted I as its motivating example: before PHP 8.0 the engine inherited the
system locale, so case folding varied by installation. The resolution was
to make folding ASCII-only everywhere, deliberately and by vote.
So the answer to "which folding" is that PHP tried locale-sensitive folding,
found it unworkable, and retreated. I would not want to walk back into it.
In general I think most people consider the case-folding for identifiers
nowadays to be a bug, not a feature, so I would probably rather try to
reduce than extend it.
Agreed, and I want to be clear I never proposed extending it. I listed it
as evidence that the current rule is incoherent, not as something to grow.
Incoherent is equally an argument for reducing.
Worth saying though that reducing is not cheap either: making class and
function names case-sensitive is a far larger break than anything discussed
in this thread. I have no proposal there, only the observation that the
rule as it stands makes sense only if identifiers are ASCII, and 1,447 of
them in the top 5,000 packages are not.
Regards,
Luca
Le 26 août 2026 à 17:15, otzelot2021@outlook.de a écrit :
Hi internals,
I would like to gauge reaction before writing an RFC.
PHP's scanner defines identifiers on bytes rather than code points:
LABEL [a-zA-Z_\x80-\xff][a-zA-Z0-9_\x80-\xff]*Every byte >= 0x80 is accepted, so UTF-8 identifiers work by accident;
the manual says as much. There is no encoding validation, no
normalisation requirement and no UAX #31 conformance. As a result
${"\xFF\xFE"} is a valid variable name, and $x and $x<U+00A0> are two
distinct variables that render identically.
Hi,
There is some confusion here: A valid name is made up of any string, not just strings that are identifiers.
Example of use of a variable named "+!" : https://3v4l.org/QSJmh
Example of use of a class (more precisely a class alias) named "" (yes, the empty string): https://3v4l.org/TQL18
Of course, names that are not identifiers cannot appear in constructs that accept an identifier only, such as an extends clause.
(...). I am proposing a per-file declare under which the accepted
set is specified: well-formed UTF-8, UAX31-R1-2 with the standard
Default-Ignorable Exclusion Profile, and NFC required rather than
applied. (...)
I don’t think that a per-file declare is a reasonable option. I would love if the PHP compiler could complain with a clearer error whenever I type accidentally a non-breaking space. But I wouldn’t add a declare clause at the top of each and every file just for that.
—Claude
There is some confusion here: A valid name is made up of any string, not
just strings that are identifiers.
Example of use of a variable named "+!" (...)
Of course, names that are not identifiers cannot appear in constructs
that
accept an identifier only, such as anextendsclause.
You are right, and I have been sloppy about it throughout this thread.
PHP does distinguish the two: an identifier is a lexical token the scanner
produces from source text, a name is whatever string ends up in a symbol
table, and the second set is much larger than the first. Your ${'+!'} and
the empty-string class alias make that concrete in a way my wording did
not.
That distinction helps the proposal rather than undermining it. What I am
describing governs identifiers only. Names created at run time through
$$name, define(), class_alias() or json_decode() are untouched -- and that
is not a gap I invented to make the scope convenient, it is a line PHP
already draws and enforces syntactically, exactly as you say with
extends. Juliette raised the same question from the other side in
another subthread; I will answer her with your terminology rather than
mine.
I will fix the wording in the draft.
I don't think that a per-file declare is a reasonable option. I would
love
if the PHP compiler could complain with a clearer error whenever I type
accidentally a non-breaking space. But I wouldn't add a declare clause at
the top of each and every file just for that.
You are the third person to push back on the declare. Larry asked why it
is optional at all, and Rowan is circling the same ground from the
direction of rejecting versus normalising. Three people arriving there
independently is a signal, and I am no longer confident the declare is the
right vehicle.
What you describe wanting is also narrower than what I proposed, and the
two come apart cleanly. A rule that rejects has to be opt-in: the
5,000-package survey turned up a live, maintained library whose variable
names are mathematical formulae, which a mandatory rule would break 888
times. A diagnostic that warns about an accidental invisible character
needs no opt-in at all, and would catch your no-break space without asking
anything of you.
I would rather not choose between those in a reply. Rowan has asked me to
state the problem before the solution, which is fair, and your message is
evidence for the same point -- I started from a mechanism and have been
arguing backwards from it ever since. Let me answer him properly first,
and then come back to whether a declare is what any of this actually
needs.
Regards,
Luca
Hi Luca,
... well-formed UTF-8, UAX31-R1-2 with the standard
Default-Ignorable Exclusion Profile, and NFC required rather than
applied
A couple of people have touched on this, but I don't think it's been directly addressed, so I'll ask it more explicitly: what do these terms mean?
- UAX31-R1-2
- the standard Default-Ignorable Exclusion Profile
- NFC
I think that's important context for this discussion, but it's also relevant to the eventual user experience: what is the summary that goes into the manual and error messages? "Class name doesn't meet UAX31-R1-2" would be about as meaningful to most people as the infamous "Unexpected T_PAAMAYIM_NEKUDOTAYIM".
The other thing that I'm not entirely clear on is how much of this is or should be about rejecting names, and how much about normalising them - bearing in mind we already perform some normalisation in the form of ASCII case folding.
Perhaps we need to step back and define the problem statement more clearly, rather than starting with a solution and trying to define its benefits?
Regards,
Rowan Tommins
[IMSoP]
A couple of people have touched on this, but I don't think it's been
directly addressed, so I'll ask it more explicitly: what do these terms
mean?
- UAX31-R1-2
- the standard Default-Ignorable Exclusion Profile
- NFC
In plain terms:
NFC. Unicode can write some characters more than one way. An o-umlaut
is either one character, U+00F6, or two: a plain o followed by a combining
diaeresis. Normalization Form C is the form that uses the single character
wherever one exists. Requiring NFC means one spelling per name.
UAX #31. The Unicode annex that says which characters a programming
language should allow in identifiers: letters, digits, marks and connecting
punctuation, but not general punctuation, symbols or formatting characters.
"R1-2" only means "we follow it with a stated list of changes" rather than
"we follow it exactly".
Default-Ignorable Exclusion Profile. One of those stated changes, and a
standard one that Unicode itself defines rather than something I made up:
drop the characters that are invisible by design -- zero-width joiners,
variation selectors and the like.
what is the summary that goes into the manual and error messages?
"Class name doesn't meet UAX31-R1-2" would be about as meaningful to most
people as the infamous "Unexpected T_PAAMAYIM_NEKUDOTAYIM".
You are right that this is the actual deliverable, and I had not written
it. Attempting it, with the standard's name appearing nowhere:
Identifier contains U+00A0 NO-BREAK SPACE, which is not allowed in a
name
Identifier is not valid UTF-8 (invalid byte 0xA9 at offset 0)
Identifier "gro<U+0308>sse" is not in Unicode normalization form C;
write it as "gr<U+00F6>sse"
The third one bears on your second question.
The other thing that I'm not entirely clear on is how much of this is or
should be about rejecting names, and how much about normalising them
- bearing in mind we already perform some normalisation in the form of
ASCII case folding.
That is the sharpest thing anyone has said in this thread, and I did not
have it clear in my own head. Sorting the four things I have been bundling
together along that axis:
not valid UTF-8 only rejectable; there is nothing to normalise to
invisible chars only rejectable; removing them would change meaning
not NFC either -- and because the engine knows the composed
spelling, a rejection can print it, which gets most
of the benefit of normalising without the engine
quietly editing your source
ASCII case folding already normalisation, and incomplete:
Stra<U+00DF>e and STRA<U+00DF>E are one class,
Stra<U+1E9E>e is another, Strasse is a third
They also have four different costs, which I can now put numbers to, from
4,863 packages and 520,802 files:
not valid UTF-8 2 identifiers
invisible chars 68, and not all accidental -- math-php spells
variables like
<U+27EE>1<U+00A0><U+2212><U+00A0>p<U+27EF><U+02E3>,
with U+00A0 inside the name, on purpose
not NFC 0, across 627,515 files in both corpora
case divergence 19, in case-insensitive positions
Bundling four rules with four cost profiles behind one mechanism was the
mistake. Larry, Claude Pache and you have each pushed on a different corner
of the same thing.
Perhaps we need to step back and define the problem statement more
clearly, rather than starting with a solution and trying to define its
benefits?
Yes. I started from a mechanism and have been arguing backwards from it all
week. Trying it the other way round:
PHP's identifier rule is expressed in bytes and says nothing about
characters. Three things follow. Two identifiers a reader cannot tell
apart may be distinct to the engine. An identifier may be text that is not
well-formed in any encoding. And the engine's own case-insensitive
matching, which is a normalisation, is defined over a 26-letter subset of
what an identifier may contain.
Underneath that sits a question nobody has answered: are non-ASCII
identifiers a supported feature of PHP? The manual says they are not, and
explains that they work because of how UTF-8 happens to encode. 1,447 of
them, in 25 of the 5,000 most-installed packages, say otherwise -- Vincenty
geodesy in mjaschen/phpgeo, Latte's U+029F prefix in tracy/tracy, Russian
schema accessors in wsdltophp/packagegenerator, and mathematical formulae
as variable names in markrogoyski/math-php.
I do not think any one of those four items justifies a language change on
its own, and I would rather say so than keep hunting for an argument that
makes it sound bigger. Together they say that PHP's identifier rule was
never designed, only inherited, and that the ecosystem has quietly built on
it anyway. Whether that is worth fixing, and in which direction, is a
question for the list rather than for me.
One observation, and then I will stop reaching for mechanisms. What falls
out of your reject/normalise split is not one feature but three, with three
different audiences: a diagnostic for invisible characters, which needs no
opt-in and is exactly what Claude Pache described wanting; a rule about
well-formedness, which has to reckon with symfony/cache; and a conformance
rule about which characters are permitted at all, which is the only part
that would break math-php 888 times and therefore the only part that
plausibly needs opting into. NFC costs nothing either way and can ride
along with whichever of those happens.
But take the problem statement first. I owe the thread that much before
proposing anything further.
Tooling, raw scanner output and per-identifier CSVs, if anyone wants to
check the numbers rather than take them: https://github.com/Otzie2023/PHP
Regards,
Luca