# [VOTE][RFC] Unicode Codepoint Escape Syntax - Thread: https://externals.io/message/79482 - Original archive: https://news-web.php.net/php.internals/79482 - Started by Andrea Faulds on 2014-12-08T23:51:37+00:00 - Messages: 28 - Page 1 of 2: messages 1 to 25 (25 per page, add `?page=N` to the URL) - Next page: https://externals.io/message/79482.md?page=2 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #79482 — Andrea Faulds — 2014-12-08T23:51:37+00:00 URL: https://externals.io/message/79482#79482 Good evening, I’m opening voting on the Unicode Codepoint Escape Syntax RFC. There’s been some discussion in the last two weeks since I introduced the RFC, but there’s nothing left which I feel needs changing. For the character name syntax suggestion (i.e. something like \N{arabic letter alef}), if that’s desired, it could be done later in a different RFC. Please read through the RFC and cast your vote if you wish to do so: https://wiki.php.net/rfc/unicode_escape Voting starts today (2014-12-08) and ends in 10 days’ time (2014-12-18). Thanks! -- Andrea Faulds http://ajf.me/ --- ## #79483 — Alain Williams — 2014-12-09T00:11:56+00:00 URL: https://externals.io/message/79482#79483 · In reply to: #79482 > Good evening, > > I’m opening voting on the Unicode Codepoint Escape Syntax RFC. There’s been some discussion in the last two weeks since I introduced the RFC, but there’s nothing left which I feel needs changing. For the character name syntax suggestion (i.e. something like \N{arabic letter alef}), if that’s desired, it could be done later in a different RFC. > > Please read through the RFC and cast your vote if you wish to do so: > > https://wiki.php.net/rfc/unicode_escape > > Voting starts today (2014-12-08) and ends in 10 days’ time (2014-12-18). I vote 'yes'. -- Alain Williams Linux/GNU Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer. +44 (0) 787 668 0256 http://www.phcomp.co.uk/ Parliament Hill Computers Ltd. Registration Information: http://www.phcomp.co.uk/contact.php #include > > I vote 'yes’. At the risk of stating the obvious: I don’t see your vote on the page’s voting widget. Please vote there. Thanks! -- Andrea Faulds http://ajf.me/ --- ## #79485 — Thomas Bley — 2014-12-09T00:44:06+00:00 URL: https://externals.io/message/79482#79485 · In reply to: #79482 thanks for the rfc! Maybe you can add to the documentation that older PHP versions can use json_decode with the limit to 4 hex digits: php -r "echo json_decode('\"man\u0303ana\"');" php -r "echo json_decode('\"ma\u00F1ana\"');" Regards Thomas Andrea Faulds wrote on 09.12.2014 00:51: > Good evening, > > I’m opening voting on the Unicode Codepoint Escape Syntax RFC. There’s been > some discussion in the last two weeks since I introduced the RFC, but there’s > nothing left which I feel needs changing. For the character name syntax > suggestion (i.e. something like \N{arabic letter alef}), if that’s desired, > it could be done later in a different RFC. > > Please read through the RFC and cast your vote if you wish to do so: > > https://wiki.php.net/rfc/unicode_escape > > Voting starts today (2014-12-08) and ends in 10 days’ time (2014-12-18). > > Thanks! > -- > Andrea Faulds > http://ajf.me/ --- ## #79486 — Alain Williams — 2014-12-09T00:50:53+00:00 URL: https://externals.io/message/79482#79486 · In reply to: #79484 > Hi Alain, > > > […] > > At the risk of stating the obvious: I don’t see your vote on the page’s voting widget. Please vote there. I looked ... I now see that I need voting Karma - which I don't have. I suppose that I might qualify under 'regular participant of internals discussions' - but that is not for me to decide. -- Alain Williams Linux/GNU Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer. +44 (0) 787 668 0256 http://www.phcomp.co.uk/ Parliament Hill Computers Ltd. Registration Information: http://www.phcomp.co.uk/contact.php #include : > > https://wiki.php.net/rfc/unicode_escape Still leaves unmentioned that there was already an established Unicode escape syntax. PCRE provides \x{1F520} for codepoints in conjunction to plain \xFF for byte escapes. Maybe there should be more elaboration on why PHP itself should go with the \u{xxxx} ECMAScript representaton, thus introducing a syntax disparity with our most major string handling extension. --- ## #79488 — Andrea Faulds — 2014-12-09T02:44:33+00:00 URL: https://externals.io/message/79482#79488 · In reply to: #79487 Hi! > > > 2014-12-09 0:51 GMT+01:00 Andrea Faulds : > > […] > > > Still leaves unmentioned that there was already an established Unicode > escape syntax. PCRE provides \x{1F520} for codepoints in conjunction to > plain \xFF for byte escapes. Interesting, I was unaware of that until now, thanks for pointing this out. > Maybe there should be more elaboration on why PHP itself should go with > the \u{xxxx} ECMAScript representaton, thus introducing a syntax disparity > with our most major string handling extension. Well, PCRE does what it does probably because of its name: *Perl-Compatible* Regular Expressions. Perl has the \x syntax. But PCRE’s syntax comes from what suits Perl, not PHP, so I don’t see why we should necessarily match its behaviour. If we add \x{xxxxx} syntax to PHP’s string literals, then we’ll break existing code which uses double quoted strings for regular expressions. I think \x{xxxx} is misleading anyway - \xXX is always single-byte/character, yet Unicode code points can’t be represented in PHP strings as single bytes when encoded in UTF-8 (unless they’re below U+0100, of course). If I saw "\x{abcd}” I'd expect it to be the same as "\xab\xbc”. Plus, while Perl has \x{xxxx} syntax, Ruby and ECMAScript 6 have the \u{xxxx} syntax, so \u{xxxx} is already more popular. The ‘u’ in \u{xxxx} also makes it more obviously “Unicode”. Thanks! -- Andrea Faulds http://ajf.me/ --- ## #79490 — mario@include-once.org — 2014-12-09T07:07:29+00:00 URL: https://externals.io/message/79482#79490 · In reply to: #79488 Tue, 9 Dec 2014 02:44:33 +0000 Andrea Faulds : > > Well, PCRE does what it does probably because of its name: > *Perl-Compatible* Regular Expressions. Perl has the \x syntax. But > PCRE’s syntax comes from what suits Perl, not PHP, so I don’t see why > we should necessarily match its behaviour. If we add \x{xxxxx} syntax > to PHP’s string literals, then we’ll break existing code which uses > double quoted strings for regular expressions. Actually the opposite seems alarming. For double quoted strings it'd be irrelevant if \u{} or \x{} was priorly handled by PHP or left to PCREs interpretation. (Having an alternative there is even beneficial.) However, single-quoted strings are more commonly and habitually used for regexps. And with \u{} going to be used regularily, then unknowingly or accidentially in regex context, is where it would trigger PCRE failures. preg_match('~\u{bad}~umixUs') Both \u{} and \x{} are used in fringe cases only of course. Consistently settling on one would still benefit forward compatibility here. > > I think \x{xxxx} is misleading anyway - \xXX is always > single-byte/character, yet Unicode code points can’t be represented > in PHP strings as single bytes when encoded in UTF-8 (unless they’re > below U+0100, of course). If I saw "\x{abcd}” I'd expect it to be the > same as "\xab\xbc”. Plus, while Perl has \x{xxxx} syntax, Ruby and > ECMAScript 6 have the \u{xxxx} syntax, so \u{xxxx} is already more > popular. The ‘u’ in \u{xxxx} also makes it more obviously “Unicode”. There's no question really about \u being more common and therefore recognize~ and preferrable. Taking the cue from Ruby is appreciated! Since the RFC rightly discounts the standard \uFFFF due to compatibility reasons, there's however little visual and semantic distinction between the {}-embellished variant \u{hhhhh} and a hypothetical \x{hhhhh}. Not sure why or who would misinterpret \x{abc} as multi-bytes, really. It's well understood and working for PCRE. The advantage of overloading \x is a much lessened likelihood to ever encounter a residual "\x{" in PHP strings. Whereas "\u" is new, and never had an implicit payload constraint, thus could run into a preexisting "\u{xxxx}" that was formerly targeted at a later/distinct context. Going with the Ruby theme; when piping a string there or receiving one it's irrelevant who uses which syntax to preinterpret it. It's only really interesting when exchanging string literals. But the RFC and the patch don't cover `stripcslashes()` or `addcslashes()` for instance. So there's no direct string syntax interoperability earmarked for. Which is why I brought forward \x{hhhhh} as alternative for within-PHP consistency at least. (Not bent on lobbying for x, as \u{…} is visually more pleasing; just unsure about its scope.) \u{1F44B} --- ## #79491 — Lester Caine — 2014-12-09T08:15:31+00:00 URL: https://externals.io/message/79482#79491 · In reply to: #79488 > > […] > Well, PCRE does what it does probably because of its name: *Perl-Compatible* Regular Expressions. Perl has the \x syntax. But PCRE’s syntax comes from what suits Perl, not PHP, so I don’t see why we should necessarily match its behaviour. If we add \x{xxxxx} syntax to PHP’s string literals, then we’ll break existing code which uses double quoted strings for regular expressions. > > I think \x{xxxx} is misleading anyway - \xXX is always single-byte/character, yet Unicode code points can’t be represented in PHP strings as single bytes when encoded in UTF-8 (unless they’re below U+0100, of course). If I saw "\x{abcd}” I'd expect it to be the same as "\xab\xbc”. Plus, while Perl has \x{xxxx} syntax, Ruby and ECMAScript 6 have the \u{xxxx} syntax, so \u{xxxx} is already more popular. The ‘u’ in \u{xxxx} also makes it more obviously “Unicode”. If ICU is to be adopted as the base for unicode support, then surely everything else should follow those rules? \uhhhh and \Uhhhhhhhh are defined along with \x{hhhhhh} so does it make sense to add something which is not part of ICU? -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk --- ## #79495 — Andrea Faulds — 2014-12-09T13:07:34+00:00 URL: https://externals.io/message/79482#79495 · In reply to: #79491 > > > If ICU is to be adopted as the base for unicode support, then surely > everything else should follow those rules? > \uhhhh and \Uhhhhhhhh are defined along with \x{hhhhhh} so does it make > sense to add something which is not part of ICU? Er, where does ICU define \uXXXX and \UXXXXXX? I don't unferstand. -- Andrea Faulds http://ajf.me/ --- ## #79497 — Lester Caine — 2014-12-09T13:35:07+00:00 URL: https://externals.io/message/79482#79497 · In reply to: #79495 > > > […] > > Er, where does ICU define \uXXXX and \UXXXXXX? I don't unferstand. http://userguide.icu-project.org/strings/regexp -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk --- ## #79498 — Andrea Faulds — 2014-12-09T14:07:22+00:00 URL: https://externals.io/message/79482#79498 · In reply to: #79497 > > > > […] > > http://userguide.icu-project.org/strings/regexp We aren't using ICU regular expressions, and ICU is merely an implementation detail anyway. -- Andrea Faulds http://ajf.me/ --- ## #79501 — Lester Caine — 2014-12-09T15:07:58+00:00 URL: https://externals.io/message/79482#79501 · In reply to: #79498 > > > […] > > We aren't using ICU regular expressions, and ICU is merely an implementation detail anyway. Has THAT been agreed on? Surely if using ICU fully in PHP7 in place of the patchwork of current fixes for unicode then we don't want to be breaking thing again by odd differences from the core code for unicode? I though the agreement was that there was no resource to create an alternative from scratch? -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk --- ## #79503 — Rowan Collins — 2014-12-09T15:30:00+00:00 URL: https://externals.io/message/79482#79503 · In reply to: #79501 Lester Caine wrote on 09/12/2014 15:07: > > > […] > Has THAT been agreed on? Surely if using ICU fully in PHP7 in place of > the patchwork of current fixes for unicode then we don't want to be > breaking thing again by odd differences from the core code for unicode? > I though the agreement was that there was no resource to create an > alternative from scratch? I think what Andrea's getting at is that the fact that ICU is in use under the hood shouldn't be particularly visible to users. If PHP gets "Unicode support" (whatever that turns out to mean), what the user should see is *PHP's Unicode facilities*; only core devs and package maintainers will need to know that those are implemented using ICU. As such, there's no automatic need for PHP to do everything the same way as ICU. -- Rowan Collins [IMSoP] --- ## #79506 — Lester Caine — 2014-12-09T16:00:42+00:00 URL: https://externals.io/message/79482#79506 · In reply to: #79503 > Lester Caine wrote on 09/12/2014 15:07: > > […] > > I think what Andrea's getting at is that the fact that ICU is in use > under the hood shouldn't be particularly visible to users. If PHP gets > "Unicode support" (whatever that turns out to mean), what the user > should see is *PHP's Unicode facilities*; only core devs and package > maintainers will need to know that those are implemented using ICU. As > such, there's no automatic need for PHP to do everything the same way as > ICU. That was the reason for asking ... What is the point of all these piecemeal patches when the underlying base has not yet been agreed on? That we are using ICU in things like the database interfaces for unicode support would point to it being somewhat useful if those processes produced the same code as the same actions in PHP. ICU is well established and it's API already in use in the same platform as PHP is running on ... so can we please treat all of these 'patches' in the light of a proper debate on the bigger picture. Forcing something like this through now simply does not make sense, and while there may be no 'automatic need' for the database interface to work the same as other parts, it would perhaps be worth a little consideration? -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk --- ## #79507 — Rowan Collins — 2014-12-09T16:24:33+00:00 URL: https://externals.io/message/79482#79507 · In reply to: #79506 Lester Caine wrote on 09/12/2014 16:00: > > > […] > That was the reason for asking ... > What is the point of all these piecemeal patches when the underlying > base has not yet been agreed on? That we are using ICU in things like > the database interfaces for unicode support would point to it being > somewhat useful if those processes produced the same code as the same > actions in PHP. ICU is well established and it's API already in use in > the same platform as PHP is running on ... so can we please treat all of > these 'patches' in the light of a proper debate on the bigger picture. > Forcing something like this through now simply does not make sense, and > while there may be no 'automatic need' for the database interface to > work the same as other parts, it would perhaps be worth a little > consideration? > I see what you mean, but I think in this case, it would make very little difference what other Unicode pieces are added, since the Unicode escape syntax will only ever be interpreted by the compiler, and no other functions will ever see what it looks like. The only exception would be things like PCRE (not ICU) regexes, where - in a single-quoted string - a visually similar syntax might exist, but there are already lots of differences between what backslash-something means in a regex and what it means in a double-quoted string literal. -- Rowan Collins [IMSoP] --- ## #79509 — Derick Rethans — 2014-12-09T16:54:29+00:00 URL: https://externals.io/message/79482#79509 · In reply to: #79488 > I think \x{xxxx} is misleading anyway - \xXX is always > single-byte/character, yet Unicode code points can’t be represented in > PHP strings as single bytes when encoded in UTF-8 (unless they’re > below U+0100, of course). You mean below U+0080 surely? Only the "first 7 bits" can be represented as a single byte with UTF-8. U+0080 is for example 0xC2 0x80 in UTF-8. cheers, Derick --- ## #79510 — Andrea Faulds — 2014-12-09T17:42:13+00:00 URL: https://externals.io/message/79482#79510 · In reply to: #79509 Hi Derick, > > > > […] > > You mean below U+0080 surely? Only the "first 7 bits" can be represented > as a single byte with UTF-8. U+0080 is for example 0xC2 0x80 in UTF-8. Ah, yes, my bad. I was probably getting confused with how all Unicode codepoints up to U+0100 match Latin-1. Thanks! -- Andrea Faulds http://ajf.me/ --- ## #79535 — Sara Golemon — 2014-12-10T18:55:45+00:00 URL: https://externals.io/message/79482#79535 · In reply to: #79482 > Please read through the RFC and cast your vote if you wish to do so: > > https://wiki.php.net/rfc/unicode_escape > > Voting starts today (2014-12-08) and ends in 10 days’ time (2014-12-18). > I was just updating my HHVM patch to match your PHP implementation and an issue came up. The following code, which is valid in PHP5: > > > > […] > I was just updating my HHVM patch to match your PHP implementation and > an issue came up. The following code, which is valid in PHP5: > > echo json_decode("\"ma\u00F1ana\""); > > Will throw a fatal compiler error as "\u00F1" is an invalid unicode > escape sequence. Since this represents an unnecessary BC break, I'd > like to propose the error handling be modified to match \x, which is > to say: Pass the value through unmodified. I was wondering about that case. Previously, the patch just raised a warning but let it through unmodified. But then old code would be littered with warnings, and I felt it was better just to throw an error. Part of the problem is that I’d rather mistakes in string literals be caught at compile time, to prevent someone accidentally echoing ‘\u00F1’ somewhere. A possible compromise might be to let ‘\u’ through but not ‘\u{‘. -- Andrea Faulds http://ajf.me/ --- ## #79537 — Alain Williams — 2014-12-10T19:03:47+00:00 URL: https://externals.io/message/79482#79537 · In reply to: #79536 > A possible compromise might be to let ‘\u’ through but not ‘\u{‘. +1 I can see that some people might have \u (for what reason I do not know), but it would be more unlikely for \u{ to be found in 'legacy' code. -- Alain Williams Linux/GNU Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer. +44 (0) 787 668 0256 http://www.phcomp.co.uk/ Parliament Hill Computers Ltd. Registration Information: http://www.phcomp.co.uk/contact.php #include > […] > > I was wondering about that case. Previously, the patch just raised a warning > but let it through unmodified. But then old code would be littered with warnings, > and I felt it was better just to throw an error. Part of the problem is that I’d rather > mistakes in string literals be caught at compile time, to prevent someone > accidentally echoing ‘\u00F1’ somewhere. > Well, consistency would be not raising a warning at all (like with \x or unknown \ sequences), but I get your concern with helping coders catch mistakes early. I don't have the stats on how many projects are using \uXXXX sequences (with the `json_decode()` or similar hack to parse them), but I know FB's codebase is absolutely covered in them. > A possible compromise might be to let ‘\u’ through but not ‘\u{‘. > Still don't like it from the inconsistency with existing escape sequence handlers pov, but it'd cover the biggest set of BC issues, so I'd be happy with it. -Sara --- ## #79544 — Andrea Faulds — 2014-12-10T19:56:29+00:00 URL: https://externals.io/message/79482#79544 · In reply to: #79540 > > > > […] > Still don't like it from the inconsistency with existing escape > sequence handlers pov, but it'd cover the biggest set of BC issues, so > I'd be happy with it. I’ve updated the patches for php-src and the specification to implement this, along with their tests, and I’ve also updated the RFC. Now this won’t error: "\"\u202e\"" But this still will: "\u{foobar" I think this is an acceptable compromise. -- Andrea Faulds http://ajf.me/ --- ## #79545 — Sara Golemon — 2014-12-10T20:09:57+00:00 URL: https://externals.io/message/79482#79545 · In reply to: #79544 > I’ve updated the patches for php-src and the specification to implement this, along with their tests, and I’ve also updated the RFC. > > Now this won’t error: > "\"\u202e\"" > > But this still will: > "\u{foobar" > > I think this is an acceptable compromise. > Groovy, thanks. -Sara --- ## #79621 — Markus Fischer — 2014-12-13T08:57:42+00:00 URL: https://externals.io/message/79482#79621 · In reply to: #79482 > Good evening, > > I’m opening voting on the Unicode Codepoint Escape Syntax RFC. There’s been some discussion in the last two weeks since I introduced the RFC, but there’s nothing left which I feel needs changing. For the character name syntax suggestion (i.e. something like \N{arabic letter alef}), if that’s desired, it could be done later in a different RFC. > > Please read through the RFC and cast your vote if you wish to do so: > > https://wiki.php.net/rfc/unicode_escape > > Voting starts today (2014-12-08) and ends in 10 days’ time (2014-12-18). The RFC is really a good writeup, very much appreciated. I've voted no because I'm not entirely convinced the current approach suits PHP long-term. I absolutely appreciate the hard work going into it, no doubt, and I rather have a solution yesterday than tomorrow. But I think a proper Unicode implementation in PHP must not rely on puzzle pieces put together without a clear goal where's we're overall heading. I know that the latter has been tried and failed (multiple times) but that's how I feel about the situation. - Markus --- Next page: https://externals.io/message/79482.md?page=2