# [RFC] Unicode Escape Syntax

- Thread: https://externals.io/message/79133
- Original archive: https://news-web.php.net/php.internals/79133
- Started by Andrea Faulds on 2014-11-24T22:09:47+00:00
- Messages: 32
- Page 1 of 2: messages 1 to 25 (25 per page, add `?page=N` to the URL)
- Next page: https://externals.io/message/79133.md?page=2
- Reply by email to internals@lists.php.net
- More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt

---

## #79133 — Andrea Faulds — 2014-11-24T22:09:47+00:00

URL: https://externals.io/message/79133#79133

Good evening,

Here’s a new RFC: https://wiki.php.net/rfc/unicode_escape

It has a rationale section explaining why certain decisions were made, that I’d recommend you read in full.

Thanks!
--
Andrea Faulds
http://ajf.me/

---

## #79134 — Andrea Faulds — 2014-11-24T22:18:50+00:00

URL: https://externals.io/message/79133#79134 · In reply to: #79133

> 
> 
> Here’s a new RFC: https://wiki.php.net/rfc/unicode_escape

My apologies to you all, a small correction: The title of that email should’ve been “[RFC] Unicode Codepoint Escape Syntax” to match the title of the RFC, I missed out the “Codepoint".
--
Andrea Faulds
http://ajf.me/

---

## #79135 — Sara Golemon — 2014-11-24T22:21:37+00:00

URL: https://externals.io/message/79133#79135 · In reply to: #79133

> Here’s a new RFC: https://wiki.php.net/rfc/unicode_escape
>
I'm okay with producing UTF-8 even though our strings are technically
binary.  As you state, UTF-8 is the de-facto encoding, and recognizing
this is pretty reasonable.

You may want to make it a requirement that strings containing \u
escapes are denoted as:   u"blah blah"    We set aside this format
back in the PHP6 days (note that b"blah" is equivalent to "blah" for
binary strings).

On the BMP versus SMP issue of \uXXXX styles, we addressed this in
PHP6 by making \u denote 4 hexit BMP codepoints, while \U denoted six
hexit codepoints.   e.g.    "\u1234" === "\U001234"   I'd rather
follow this style than making \u special and different from hex and
octal notations by using braces.

-Sara

---

## #79136 — Andrea Faulds — 2014-11-24T22:28:30+00:00

URL: https://externals.io/message/79133#79136 · In reply to: #79135

> 
> 
> 
> > […]
> I'm okay with producing UTF-8 even though our strings are technically
> binary.  As you state, UTF-8 is the de-facto encoding, and recognizing
> this is pretty reasonable.

On that note, it strikes me now that we assume an encoding anyway for all escape sequences. If I’m using EBCDIC or UTF-16, “\n” isn’t going to help me much!

> You may want to make it a requirement that strings containing \u
> escapes are denoted as:   u"blah blah"    We set aside this format
> back in the PHP6 days (note that b"blah" is equivalent to "blah" for
> binary strings).

I’d rather keep u"blah blah” for if/when we add actual Unicode strings. 

> On the BMP versus SMP issue of \uXXXX styles, we addressed this in
> PHP6 by making \u denote 4 hexit BMP codepoints, while \U denoted six
> hexit codepoints.   e.g.    "\u1234" === "\U001234"   I'd rather
> follow this style than making \u special and different from hex and
> octal notations by using braces.

That is something I’d thought about. \U takes 8 hex digits in every other language which has it, though.

I suppose we could do this, it resolves the BMP issue, certainly. Still, I think the brace syntax has its advantages because it’s completely unambiguous and it means we only have one syntax for this, not two different ones (less mental overhead). Plus, it’s worth noting that \u would still be different from \ooo and \xXX anyway, as it’d be fixed-length while octal and hex aren’t.

--
Andrea Faulds
http://ajf.me/

---

## #79137 — Adam Harvey — 2014-11-24T22:30:15+00:00

URL: https://externals.io/message/79133#79137 · In reply to: #79135

> 
> > […]
> I'm okay with producing UTF-8 even though our strings are technically
> binary.  As you state, UTF-8 is the de-facto encoding, and recognizing
> this is pretty reasonable.

I'm also OK with this, although I do wonder if we should be respecting
the user's default_charset setting instead. (Since default_charset
defaults to "UTF-8", in practice this isn't a significant difference
for the average user.)

> You may want to make it a requirement that strings containing \u
> escapes are denoted as:   u"blah blah"    We set aside this format
> back in the PHP6 days (note that b"blah" is equivalent to "blah" for
> binary strings).

It seems to me that the point of \u and \U escapes is to embed Unicode
in potentially non-Unicode strings, so using u"" doesn't feel right.

> On the BMP versus SMP issue of \uXXXX styles, we addressed this in
> PHP6 by making \u denote 4 hexit BMP codepoints, while \U denoted six
> hexit codepoints.   e.g.    "\u1234" === "\U001234"   I'd rather
> follow this style than making \u special and different from hex and
> octal notations by using braces.

I think I prefer the brace style, personally. Non-BMP codepoints have
become more important since PHP 6 (thanks, emoji), and having \u and
\U be case sensitive when \x isn't seems confusing.

Adam

---

## #79138 — Andrea Faulds — 2014-11-24T22:35:23+00:00

URL: https://externals.io/message/79133#79138 · In reply to: #79137

> 
> 
> 
> > […]
> 
> I'm also OK with this, although I do wonder if we should be respecting
> the user's default_charset setting instead. (Since default_charset
> defaults to "UTF-8", in practice this isn't a significant difference
> for the average user.)

Ooh, that would be a possibility. That or using whatever encoding the source file is specified to be with declare(), so it matches the encoding of other characters in the string.

This’d add significant complexity to it, though (would we have to require ICU or something? D:), plus the vast majority of Unicode characters will only be supported by Unicode encodings… and of those, only UTF-8 is really in much use here anyway.

> > […]
> 
> It seems to me that the point of \u and \U escapes is to embed Unicode
> in potentially non-Unicode strings, so using u"" doesn't feel right.

I don’t really see where you’re coming from, it also makes just as much sense within Unicode strings. There are plenty of cases (like the U+202E or mañana examples in the RFC) where you’d want a Unicode escape in a Unicode string.

--
Andrea Faulds
http://ajf.me/

---

## #79139 — Adam Harvey — 2014-11-24T22:41:19+00:00

URL: https://externals.io/message/79133#79139 · In reply to: #79138

>
> > […]
>
> Ooh, that would be a possibility. That or using whatever encoding the source file is specified to be with declare(), so it matches the encoding of other characters in the string.
>
> This’d add significant complexity to it, though (would we have to require ICU or something? D:), plus the vast majority of Unicode characters will only be supported by Unicode encodings… and of those, only UTF-8 is really in much use here anyway.

We would have to require ICU, but that might be worthwhile for PHP 7
anyway. Having at least one i18n API that's guaranteed to be available
would be nice.

> > […]
>
> I don’t really see where you’re coming from, it also makes just as much sense within Unicode strings. There are plenty of cases (like the U+202E or mañana examples in the RFC) where you’d want a Unicode escape in a Unicode string.

I probably worded that badly — I just mean that I don't think \u and
\U should be limited to only u"" strings, but should work in normal
strings as well. (In other words, I'm agreeing with what's in your
RFC, not with Sara.)

Adam

---

## #79140 — Sara Golemon — 2014-11-24T23:19:12+00:00

URL: https://externals.io/message/79133#79140 · In reply to: #79139

> We would have to require ICU, but that might be worthwhile for PHP 7
> anyway. Having at least one i18n API that's guaranteed to be available
> would be nice.
>
It's 2014.  I think requiring ICU is reasonable at this point.

Orthogonal to this RFC, but I'd be in favor of deprecating all the
non-ICU intl stuff sometime soon.

> I probably worded that badly — I just mean that I don't think \u and
> \U should be limited to only u"" strings, but should work in normal
> strings as well. (In other words, I'm agreeing with what's in your
> RFC, not with Sara.)
>
I don't feel strongly about the u"" requirement, it doesn't make the
world a darker place if we're more permissive.

> Plus, it’s worth noting that \u would still be different from \ooo and \xXX anyway,
> as it’d be fixed-length while octal and hex aren’t.
>
And I really wish that weren't true. :p

-Sara

---

## #79141 — Andrea Faulds — 2014-11-24T23:22:05+00:00

URL: https://externals.io/message/79133#79141 · In reply to: #79140

> 
> 
> > […]
> It's 2014.  I think requiring ICU is reasonable at this point.

I also think it would be reasonable to require ICU, especially as it means we could perhaps enable Joe Watkins’s UString by default, assuming it actually makes it into PHP 7.

That said, I don’t think we should go down the route of making \u convert to the current encoding. It doesn’t make much sense, if any, for non-Unicode encodings, and nobody is using UTF-16 or UTF-32. Plus, it’d be inconsistent, given we don’t convert any of the other escape sequences in strings anyway! It would be quite weird if \u{77} converted yet \x77 did not.

--
Andrea Faulds
http://ajf.me/

---

## #79142 — Alain Williams — 2014-11-24T23:29:11+00:00

URL: https://externals.io/message/79133#79142 · In reply to: #79135

> 
> > […]
> I'm okay with producing UTF-8 even though our strings are technically
> binary.  As you state, UTF-8 is the de-facto encoding, and recognizing
> this is pretty reasonable.
> 
> You may want to make it a requirement that strings containing \u
> escapes are denoted as:   u"blah blah"    We set aside this format
> back in the PHP6 days (note that b"blah" is equivalent to "blah" for
> binary strings).
> 
> On the BMP versus SMP issue of \uXXXX styles, we addressed this in
> PHP6 by making \u denote 4 hexit BMP codepoints, while \U denoted six
> hexit codepoints.   e.g.    "\u1234" === "\U001234"   I'd rather
> follow this style than making \u special and different from hex and
> octal notations by using braces.

There is a big difference with \u or \U and \x or \o and that is the number of
characters that follow the escape. \x has 2, \o has 3 - both are short and easy
to count with the eye. \U012345 is quite long and it is not so visually obvious
where it should end.

Ergo: I prefer Andrea's "\u{0123}" as it is going to be more robust against typos.


One other thing that we could do is to allow code points to be named, with \U
(capital 'U') eg:

echo "\U{arabic letter alef}\n";

If you think that it is a bad idea, please update the RFC to say why this is a
bad idea and so why it is not going to happen - for now.

It would be nice since a code point is just a big number without any really obvious
meaning, but a name makes for greater clarity.

However: I suspect that interpretting this might be considerably slower which
means slower compilation.

Regards

-- 
Alain Williams
Linux/GNU Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer.
+44 (0) 787 668 0256  http://www.phcomp.co.uk/
Parliament Hill Computers Ltd. Registration Information: http://www.phcomp.co.uk/contact.php
#include <std_disclaimer.h

---

## #79143 — Andrea Faulds — 2014-11-24T23:36:28+00:00

URL: https://externals.io/message/79133#79143 · In reply to: #79142

> 
> 
> There is a big difference with \u or \U and \x or \o and that is the number of
> characters that follow the escape. \x has 2, \o has 3 - both are short and easy
> to count with the eye. \U012345 is quite long and it is not so visually obvious
> where it should end.
> 
> Ergo: I prefer Andrea's "\u{0123}" as it is going to be more robust against typos.

Typos are an angle I hadn’t quite considered, but yes, this syntax is better against that. Importantly, it’s a compile error if you produce a broken literal, while if you screwed up the brace-free style you’d probably just get a mangled string.

> One other thing that we could do is to allow code points to be named, with \U
> (capital 'U') eg:
> 
> echo "\U{arabic letter alef}\n”;

Ooh, that’s an interesting idea. I believe Perl actually has this already, although it uses the \N syntax:

http://perldoc.perl.org/perlreref.html#ESCAPE-SEQUENCES

Is something like that what you have in mind?

> If you think that it is a bad idea, please update the RFC to say why this is a
> bad idea and so why it is not going to happen - for now.
> 
> It would be nice since a code point is just a big number without any really obvious
> meaning, but a name makes for greater clarity.
> 
> However: I suspect that interpretting this might be considerably slower which
> means slower compilation.

I’ll add it to the Future Scope part.

One issue with this, however, is that we’d have to include a Unicode info database from somewhere with the names of the characters. That’d probably mean requiring ICU or something like it, which the current patch doesn’t do.
--
Andrea Faulds
http://ajf.me/

---

## #79144 — Alain Williams — 2014-11-24T23:45:25+00:00

URL: https://externals.io/message/79133#79144 · In reply to: #79143

> 
> > […]

> > […]
> 
> Ooh, that’s an interesting idea. I believe Perl actually has this already, although it uses the \N syntax:
> 
> http://perldoc.perl.org/perlreref.html#ESCAPE-SEQUENCES
> 
> Is something like that what you have in mind?

Exactly.

Confession: it was looking at the perl documentation that led me to suggest it.

-- 
Alain Williams
Linux/GNU Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer.
+44 (0) 787 668 0256  http://www.phcomp.co.uk/
Parliament Hill Computers Ltd. Registration Information: http://www.phcomp.co.uk/contact.php
#include <std_disclaimer.h

---

## #79145 — Sara Golemon — 2014-11-25T02:37:10+00:00

URL: https://externals.io/message/79133#79145 · In reply to: #79133

> Here’s a new RFC: https://wiki.php.net/rfc/unicode_escape
>
I've linked a provisional HHVM implementation from that page.
Planning to match whatever PHP7 does, of course, but for the moment
I've added named entity support since it's being discussed.

https://github.com/sgolemon/hhvm/compare/unicode-escape

---

## #79147 — Ivan Enderlin @ Hoa — 2014-11-25T07:29:01+00:00

URL: https://externals.io/message/79133#79147 · In reply to: #79133

Le 24/11/2014 23:09, Andrea Faulds a écrit :
> Good evening,
>
> Here’s a new RFC: https://wiki.php.net/rfc/unicode_escape
>
> It has a rationale section explaining why certain decisions were made, that I’d recommend you read in full.
Excellent RFC, thank you for this proposal.
I would suggest this talk 
https://speakerdeck.com/mathiasbynens/hacking-with-unicode (you might 
already know) but interesting concepts and limitations of current 
Unicode implementations are mentioned.
The usage of `\u{…}` fixes most limitations and I could not be more 
agree with that notation!

Cheers.

-- 
Ivan Enderlin
Developer of Hoa
http://hoa-project.net/

PhD. at DISC/Femto-ST (Vesontio) and INRIA (Cassis)
http://disc.univ-fcomte.fr/ and http://www.inria.fr/

Member of HTML and WebApps Working Group of W3C
http://w3.org/

---

## #79148 — Dmitry Stogov — 2014-11-25T08:33:01+00:00

URL: https://externals.io/message/79133#79148 · In reply to: #79133

May be I misunderstood something, but why to introduce unicode escapes if
PHP engine doesn't support Unicode.

Always converting such escapes into UTF-8 encoding, doesn't make any sense
for people who use other encodings for output, databases, etc.

Thanks. Dmitry.






> Good evening,
>
> Here’s a new RFC: https://wiki.php.net/rfc/unicode_escape
>
> It has a rationale section explaining why certain decisions were made,
> that I’d recommend you read in full.
>
> Thanks!
> --
> Andrea Faulds
> http://ajf.me/

---

## #79149 — Markus Fischer — 2014-11-25T08:33:19+00:00

URL: https://externals.io/message/79133#79149 · In reply to: #79133

> Good evening,
> 
> Here’s a new RFC: https://wiki.php.net/rfc/unicode_escape

I think the choice of \u{xx} is interesting, i.e. using '{' and '}'.

Afaik, one of the current best practices is to use `json_decode()`, like so:

$ cat test.php
<?php var_dump( json_decode( '"\u00c4"' ) );

$ php test.php
string(2) "Ä"

So your choice definitely will not influence this practice which is
good, IMHO. OTOH I'm not sure if not using '{' and '}' would actually
cause BC problems, but it's of no concern with your RFC.

thanks,
- Markus

---

## #79150 — Andrea Faulds — 2014-11-25T10:00:29+00:00

URL: https://externals.io/message/79133#79150 · In reply to: #79148

> 
> 
> May be I misunderstood something, but why to introduce unicode escapes if PHP engine doesn't support Unicode.

We don't have Unicode strings which are made of codepoints rather than bytes, sure. But we do usually treat these strings as UTF-8. The idea of doing this in a language without Unicode strings isn't new, C/C++ have the u8"" syntax for making UTF-8 strings.

> Always converting such escapes into UTF-8 encoding, doesn't make any sense for people who use other encodings for output, databases, etc.

If you're using other encodings, why do you want to use a Unicode codepoints? Most Unicode codepoints will not supported by another character set.

--
Andrea Faulds
http://ajf.me/

---

## #79152 — Andrea Faulds — 2014-11-25T10:03:54+00:00

URL: https://externals.io/message/79133#79152 · In reply to: #79149

> 
> 
> > […]
> 
> I think the choice of \u{xx} is interesting, i.e. using '{' and '}'.
> 
> Afaik, one of the current best practices is to use `json_decode()`, like so:
> 
> $ cat test.php
> <?php var_dump( json_decode( '"\u00c4"' ) );
> 
> $ php test.php
> string(2) "Ä"

json_decode is an interesting workaround, but JSON suffers from JavaScript, where it got its syntax, having poor Unicode handling. JSON's Unicode escapes only allow BMP characters and, worse, non-BMP characters must be encoded using surrogate pairs.

--
Andrea Faulds
http://ajf.me/

---

## #79154 — Derick Rethans — 2014-11-25T10:32:39+00:00

URL: https://externals.io/message/79133#79154 · In reply to: #79135

> 
> > […]
> I'm okay with producing UTF-8 even though our strings are technically
> binary.  As you state, UTF-8 is the de-facto encoding, and recognizing
> this is pretty reasonable.
> 
> You may want to make it a requirement that strings containing \u
> escapes are denoted as:   u"blah blah"    We set aside this format
> back in the PHP6 days (note that b"blah" is equivalent to "blah" for
> binary strings).
> 
> On the BMP versus SMP issue of \uXXXX styles, we addressed this in
> PHP6 by making \u denote 4 hexit BMP codepoints, while \U denoted six
> hexit codepoints.   e.g.    "\u1234" === "\U001234"   I'd rather
> follow this style than making \u special and different from hex and
> octal notations by using braces.

I agree with this fully. No need to reinvent a wheel (that we left 
behind on the road)...

cheers,
Derick

---

## #79155 — Dmitry Stogov — 2014-11-25T10:41:48+00:00

URL: https://externals.io/message/79133#79155 · In reply to: #79150

>
> > […]
> if PHP engine doesn't support Unicode.
>
> We don't have Unicode strings which are made of codepoints rather than
> bytes, sure. But we do usually treat these strings as UTF-8. The idea of
> doing this in a language without Unicode strings isn't new, C/C++ have the
> u8"" syntax for making UTF-8 strings.
>

u8"string" tells that the whole string is UTF-8 encoded.
Your escape Unicode proposal  assumes just UTF-8 codepoint, but the whole
string encoding is still undefined.


>
> > […]
> sense for people who use other encodings for output, databases, etc.
>
> If you're using other encodings, why do you want to use a Unicode
> codepoints? Most Unicode codepoints will not supported by another character
> set.
>

Agree, this Unicode escapes are not going to be used for anything except
UTF-8 encoded strings.
I'm not completely against it. It's just an incomplete solution.

echo "\u{1F602}"; // won't output 😂 if the output encoding is not UTF-8

echo "Привет \u{1F602}"; // won't output anything useful if script
encoding is not UTF-8

The second problem present even for European counties that use Windows-1250
codepage.

echo "mañana \u{1F602}"; // won't output anything useful if script
encoding is not UTF-8

Thanks. Dmitry.


>
> --
> Andrea Faulds
> http://ajf.me/

---

## #79156 — Andrea Faulds — 2014-11-25T11:09:02+00:00

URL: https://externals.io/message/79133#79156 · In reply to: #79154

> 
> 
> 
> 
> > […]
> 
> I agree with this fully. No need to reinvent a wheel (that we left 
> behind on the road)…

The \u{} style has its advantages, like not being case-sensitive and being clearly delimited, though. Plus, PHP 6 is dead, and non-BMP code points are more important than ever.

--
Andrea Faulds
http://ajf.me/

---

## #79157 — Andrea Faulds — 2014-11-25T11:18:57+00:00

URL: https://externals.io/message/79133#79157 · In reply to: #79155

> 
> 
> u8"string" tells that the whole string is UTF-8 encoded.
> Your escape Unicode proposal  assumes just UTF-8 codepoint, but the whole string encoding is still undefined.

True. There’s an assumption there that you’re using a UTF-8-compatible source file. Actually, for other encodings, do we even guarantee that “\n” produces an ASCII LF just now? It certainly will on most Windows and Unix systems, but since we’re just using C’s ‘\n’ (http://lxr.php.net/xref/PHP_TRUNK/Zend/zend_language_scanner.l#885), it might produce the newline character of some other encoding like EBCDIC in the right environment.

> > […]
> 
> Agree, this Unicode escapes are not going to be used for anything except UTF-8 encoded strings.
> I'm not completely against it. It's just an incomplete solution.
> 
> echo "\u{1F602}"; // won't output 😂 if the output encoding is not UTF-8
> 
> echo "Привет \u{1F602}"; // won't output anything useful if script encoding is not UTF-8
> The second problem present even for European counties that use Windows-1250 codepage.
> echo "mañana \u{1F602}"; // won't output anything useful if script encoding is not UTF-8
> Thanks. Dmitry.

Yeah, that’s unfortunate. Although I don’t think there’s much we can do about it here. We can’t really convert, as if most Unicode characters won’t be available in the codepage you’re using.

Even if we did have Unicode strings like the fabled PHP6 would have had, you still have this problem when you’re outputting in non-Unicode encodings.

Although it’s worth noting that mbstring *should* handle this, since if you have an internal encoding of UTF-8 and an output encoding of, say, Windows-1250, you can use UTF-8 in your strings it should convert that for you on output. How well this works in practice, however, I have no idea.
--
Andrea Faulds
http://ajf.me/

---

## #79158 — Alain Williams — 2014-11-25T11:20:50+00:00

URL: https://externals.io/message/79133#79158 · In reply to: #79155

> I'm not completely against it. It's just an incomplete solution.
> 
> echo "\u{1F602}"; // won't output ? if the output encoding is not UTF-8
> 
> echo "Привет \u{1F602}"; // won't output anything useful if script
> encoding is not UTF-8
> 
> The second problem present even for European counties that use Windows-1250
> codepage.

I think that we need to clarify what we are talking about.

What Andrea has proposed is a way of writing string constants. These characters
in these strings will still be 8 bits big, this means that there needs to be
some way of encoding characters with code points that will not fit in 8 bits.
The only way of avoiding that would be to use, internally, 32 bit characters --
which would be a huge change.

So: we need to have some form of encoding.

As I started ''a way of writing string constants'' - ie a *compile* time action.

With the code below it is likely that at *run-time* `mb_internal_encoding()` has
been called before the echo is executed or the 'Content-Type:' header specifies
some encoding.

> echo "mañana \u{1F602}"; // won't output anything useful if script
> encoding is not UTF-8

This is not something that the compiler can guess.

It is even worse if my proposal of \U{arabic letter alef} types is added, how is
that encoded ? UTF-8 or iso-8859-6 or .... ?

So, how do we fix the problem ?

* mb_internal_encoding($new_encoding) finds every string (variable and constant)
  and converts from the previous encoding to the $new_encoding.

  Possible, but horribly slow and would prob break things (eg strings that
  contain binary data).

  Not a good idea.

* Decide that UTF-8 is king.
  That is what I have decided - but I do not have any legacy code to worry about
  -- being a Brit I don't have to worry much.

* Rely on the programmer to understand encoding and know what the eventual
  output encoding will be and if it is not UTF-8 write characters using \Xxx or
  use mb_convert_encoding($string, $output_encoding, 'utf-8').

If we decide to support non-utf-8 encoding at compile time then we could extend
the syntax a bit to allow the encoding to be specified, eg:

    \U{utf-8: arabic letter alef}

    \U{iso-8859-6: arabic letter alef}

Ie, allow this to be optionally specified and terminated by ':'. If not
specified then assume utf-8.


-- 
Alain Williams
Linux/GNU Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer.
+44 (0) 787 668 0256  http://www.phcomp.co.uk/
Parliament Hill Computers Ltd. Registration Information: http://www.phcomp.co.uk/contact.php
#include <std_disclaimer.h

---

## #79160 — Andrea Faulds — 2014-11-25T11:25:17+00:00

URL: https://externals.io/message/79133#79160 · In reply to: #79158

> 
> 
> I think that we need to clarify what we are talking about.
> 
> What Andrea has proposed is a way of writing string constants. These characters
> in these strings will still be 8 bits big, this means that there needs to be
> some way of encoding characters with code points that will not fit in 8 bits.
> The only way of avoiding that would be to use, internally, 32 bit characters --
> which would be a huge change.
> 
> So: we need to have some form of encoding.
> 
> As I started ''a way of writing string constants'' - ie a *compile* time action.
> 
> With the code below it is likely that at *run-time* `mb_internal_encoding()` has
> been called before the echo is executed or the 'Content-Type:' header specifies
> some encoding.
> 
> > […]
> 
> This is not something that the compiler can guess.

Well, we *do* already have a compile-time system for declaring encoding, the declare() construct.

> It is even worse if my proposal of \U{arabic letter alef} types is added, how is
> that encoded ? UTF-8 or iso-8859-6 or .... ?
> 
> So, how do we fix the problem ?
> 
> * mb_internal_encoding($new_encoding) finds every string (variable and constant)
>  and converts from the previous encoding to the $new_encoding.
> 
>  Possible, but horribly slow and would prob break things (eg strings that
>  contain binary data).
> 
>  Not a good idea.

I also agree this isn’t a good idea.

> * Decide that UTF-8 is king.
>  That is what I have decided - but I do not have any legacy code to worry about
>  -- being a Brit I don't have to worry much.
> 
> * Rely on the programmer to understand encoding and know what the eventual
>  output encoding will be and if it is not UTF-8 write characters using \Xxx or
>  use mb_convert_encoding($string, $output_encoding, 'utf-8').
> 
> If we decide to support non-utf-8 encoding at compile time then we could extend
> the syntax a bit to allow the encoding to be specified, eg:
> 
>    \U{utf-8: arabic letter alef}
> 
>    \U{iso-8859-6: arabic letter alef}
> 
> Ie, allow this to be optionally specified and terminated by ':'. If not
> specified then assume utf-8.

There are only two sane options:

  * Always UTF-8
  * Whatever source file encoding we’ve specified with declare()

Of those, I’d prefer UTF-8, as nobody’s using UTF-16 or UTF-32.

--
Andrea Faulds
http://ajf.me/

---

## #79161 — Christoph Becker — 2014-11-25T11:37:19+00:00

URL: https://externals.io/message/79133#79161 · In reply to: #79147

Ivan Enderlin @ Hoa wrote:

> Le 24/11/2014 23:09, Andrea Faulds a écrit :
> > […]
> Excellent RFC, thank you for this proposal.
> I would suggest this talk
> https://speakerdeck.com/mathiasbynens/hacking-with-unicode (you might
> already know) but interesting concepts and limitations of current
> Unicode implementations are mentioned.
> The usage of `\u{…}` fixes most limitations and I could not be more
> agree with that notation!

I don't see that the proposed \u{...} notation fixes any limitation.
Its only advantage would be slightly better readability opposed to
inserting the desired UTF-8 byte sequences as octal or hexadecimal
escapes.  For instance, all of the following would print Ä (in an UTF-8
context):

  echo "\u{00C4}";
  echo "\xC3\x84";
  echo "\303\204";

-- 
Christoph M. Becker

---

Next page: https://externals.io/message/79133.md?page=2
