# PHP6 wiki page - Thread: https://externals.io/message/61768 - Original archive: https://news-web.php.net/php.internals/61768 - Started by Andrew Faulds on 2012-07-25T15:20:40+00:00 - Messages: 17 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #61768 — Andrew Faulds — 2012-07-25T15:20:40+00:00 URL: https://externals.io/message/61768#61768 Is there a PHP 6 wiki page for co-ordinating development of and collecting ideas for PHP6 development? There are a lot of idea being thrown around, would be nice if we had a page (there might be one, but I can't find it). Thanks. -- Andrew Faulds http://ajf.me/ --- ## #61779 — Ferenc Kovacs — 2012-07-25T15:58:28+00:00 URL: https://externals.io/message/61768#61779 · In reply to: #61768 > Is there a PHP 6 wiki page for co-ordinating development of and collecting > ideas for PHP6 development? > > There are a lot of idea being thrown around, would be nice if we had a > page (there might be one, but I can't find it). > > Thanks. > We had one for the last php6 attempt: https://wiki.php.net/todo/php60 but we didn't started a new one about/after the recent '6.0 And Moving Forward' thread. -- Ferenc Kovács @Tyr43l - http://tyrael.hu --- ## #72590 — Lester Caine — 2014-02-14T11:02:20+00:00 URL: https://externals.io/message/61768#72590 · In reply to: #61779 Ferenc Kovacs wrote: > > […] > We had one for the last php6 attempt:https://wiki.php.net/todo/php60 but > we didn't started a new one about/after the recent '6.0 And Moving Forward' > thread. Yes I know this is an old thread, but it is probably indicative of the state of play? It was last modified in 2006! Is it worth going through the points there or would it be better simply starting with a clean piece of paper and leaving this as history? Yasuo has already brought up one of the unicode problems that will need to be addressed in the phar file name thread. But more fundamentally I don't think there was agreement on whether we simply standardise on unicode in the core, or allow a single byte mode? 8 years on, I feel that the amount of utf8 material that is floating around, the easiest route IS unicode only? Perhaps with a compiler switch to disable it if people want the option, rather than an ini setting? And I only use 127 character English :) but probably half the emails I'm processing have unicode of some sort these days? Certainly all of my websites require utf8 without any discussion. Even 64bit integers get a mention near the bottom! Although it is probably worth flagging that most higher end processors these days do have a 256bit integer capability! So while 128bit has been mentioned in recent posts, perhaps a little more in depth coverage is appropriate if PHP6 is going to be the base for the next 10 years? Many of the bullet points certainly need updating, but how many have actually already been addressed in PHP5.X? -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk --- ## #72606 — Rowan Collins — 2014-02-14T16:39:46+00:00 URL: https://externals.io/message/61768#72606 · In reply to: #72590 Lester Caine wrote (on 14/02/2014): > But more fundamentally I don't think there was agreement on whether we > simply standardise on unicode in the core, or allow a single byte > mode? 8 years on, I feel that the amount of utf8 material that is > floating around, the easiest route IS unicode only? The question is not whether to be "Unicode only", it's *how* to implement Unicode. It's not just a case of making all your strings wider, every function that manipulates a string in any way has to be thought through, and every input and output has to be converted to/from whatever encoding is chosen as the internal implementation. While updating the Wikipedia article [1] I came across this slide set [2], which has a fairly decent explanation of the issues and why the previous implementation was abandoned. If somebody comes up with an implementation proposal of Unicode strings, whether to have a mode that doesn't use it can be discussed, but right now there doesn't seem to be such a live proposal. [1] http://en.wikipedia.org/wiki/PHP#PHP_6_and_Unicode [2] http://www.slideshare.net/andreizm/the-good-the-bad-and-the-ugly-what-happened-to-unicode-and-php-6 Regards, -- Rowan Collins [IMSoP] --- ## #72607 — Julien Pauli — 2014-02-14T16:44:37+00:00 URL: https://externals.io/message/61768#72607 · In reply to: #61768 > Is there a PHP 6 wiki page for co-ordinating development of and collecting > ideas for PHP6 development? > > There are a lot of idea being thrown around, would be nice if we had a page > (there might be one, but I can't find it). Feel free to create a new php5++ one. We should keep the history of what's been done on old PHP60. Perhaps it is better to rename php60 to php60ld and get php60 back alive ? Julien --- ## #72608 — Rowan Collins — 2014-02-14T16:49:36+00:00 URL: https://externals.io/message/61768#72608 · In reply to: #72607 Julien Pauli wrote (on 14/02/2014): > > > […] > Feel free to create a new php5++ one. +1 for calling it "PHP 5++", with the first item on the todo list being "decide on version number". I have a book next to me titled "PHP 6 and MySQL 5"... > We should keep the history of what's been done on old PHP60. > Perhaps it is better to rename php60 to php60ld and get php60 back alive ? How about "php6_2010", which is when the Unicode implementation was officially abandoned? -- Rowan Collins [IMSoP] --- ## #72610 — Julien Pauli — 2014-02-14T16:57:00+00:00 URL: https://externals.io/message/61768#72610 · In reply to: #72608 > Julien Pauli wrote (on 14/02/2014): > > > […] > > > +1 for calling it "PHP 5++", with the first item on the todo list being > "decide on version number". > > I have a book next to me titled "PHP 6 and MySQL 5"... That's what happens when some author want to run faster than time. Silly people... > > > […] > > > How about "php6_2010", which is when the Unicode implementation was > officially abandoned? Why not, this is just a naming anyway, nothing really important. Julien --- ## #72612 — Lester Caine — 2014-02-14T17:52:58+00:00 URL: https://externals.io/message/61768#72612 · In reply to: #72606 Rowan Collins wrote: > If somebody comes up with an implementation proposal of Unicode strings, whether > to have a mode that doesn't use it can be discussed, but right now there doesn't > seem to be such a live proposal. I think it is now accepted that the mistake was UTF16? Personally I always thought it was the wrong choice as other projects had already shown and so was not likely to work. If we look at UTF8 as a starting point, then in the large majority of places all that results is longer strings? Modern tools will just display them and the bulk of PHP simply works without a problem already? I've mentioned one point already in other threads ... if you are simply looking to match a string, then equal/not-equal is all that is required. The current compare also looks to supply 'order' as well but in many cases this is simply not needed? Drop the 'order' and it does not mater if the string has strange characters in it. The slide show recognises that converting to UTF8 and then back again is something that simply slots in on the periphery, and so should just work. I throw out any of the date and currency styling ... that is a different problem and is already covered well and can return uncode strings if required! To be honest I can't see why these are bundled with the unicode problem at all? This just leaves 'sort' and more complex string handling? I accept that the major brick wall here is 'case-insensitivity' since unicode string length may well change when making a conversion. That is not exactly a 'PHP' problem, but a fact of life with the languages we are dealing with? strtolower already has problems with the more complex single byte character sets, but there is no reason that it could not follow the unicode defined rules as a starting point? Moving to 'character handling' specifically I have always viewed that as a place where 'under the hood' the string being handled becomes UTF32 so that we are back looking at individual characters rather than 'bytes'? But I am getting out of my own depth when the character is fabricated from more than one unicode character. The bit that Yasuo outlined earlier re NFC/NFD normalization? This is another variation on the 'case-insensitive' problem? If an accent is added additionally to a base character, then I would tend to defaulting to combine them when processing a string, but that may not be correct ... The bottom line is - I think - that we already know what works and what does not, and we can define a 'default' simple sort where case-sensitivity is restricted to fixed length string conversions to get a base system. Moving to sort routines that respect different locals is then a layer on top? But all of the groundwork for a default system does already exist? We have the filters to convert to and from UTF8, and we have all the basic processes for handling utf8 strings. It is just a matter of agreeing a method of pulling it together? -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk --- ## #72613 — Rasmus Lerdorf — 2014-02-14T19:35:06+00:00 URL: https://externals.io/message/61768#72613 · In reply to: #72606 > Lester Caine wrote (on 14/02/2014): > > […] > > The question is not whether to be "Unicode only", it's *how* to > implement Unicode. It's not just a case of making all your strings > wider, every function that manipulates a string in any way has to be > thought through, and every input and output has to be converted to/from > whatever encoding is chosen as the internal implementation. > > While updating the Wikipedia article [1] I came across this slide set > [2], which has a fairly decent explanation of the issues and why the > previous implementation was abandoned. > > If somebody comes up with an implementation proposal of Unicode strings, > whether to have a mode that doesn't use it can be discussed, but right > now there doesn't seem to be such a live proposal. What we really need is an awesome small and fast Unicode library that does everything ICU does but faster and in less code while using UTF-8 as its internal storage so we don't have to convert on each and every operation. There are a ton of non-obvious things beyond simple string manipulation. String collation alone is massively complicated, for example. -Rasmus --- ## #72614 — Lester Caine — 2014-02-14T21:06:50+00:00 URL: https://externals.io/message/61768#72614 · In reply to: #72613 Rasmus Lerdorf wrote: > What we really need is an awesome small and fast Unicode library that > does everything ICU does but faster and in less code while using UTF-8 > as its internal storage so we don't have to convert on each and every > operation. There are a ton of non-obvious things beyond simple string > manipulation. String collation alone is massively complicated, for example. Surely the bottom line is that to cover every fine detail, ICU has to be used as the smaller libraries tend to make few assumptions to make life easy? But my point was that most of the time you only need the simple stuff? Simply using UTF8 strings in place of the byte based ones in all of the relevant string? Remove the need to 'lowercase' by dropping case-insensitivity and things are simplified somewhat? I've found the comment I was looking for finally while searching around ... "UTF-8 is specially designed so that many byte-oriented string functions continue to work or only need minor modifications." This is why people can put unicode characters in many places in PHP now without it actually breaking? I've seen a few comments about switching to C++ and http://utfcpp.sourceforge.net/ caught my eye, but http://www.public-software-group.org/utf8proc-documentation came to light when I started looking at NDF/NDC but I've been looking for a suitable unicode string handler for doing substring clipping and all of that. I AM right in thinking that mbstring is basically overkill if everything being worked with has already been converted to UTF8? While I was aware of accent code points, I'd not quite appreciated how complicated they can get. Up until now I've just been looking at text cut and pasted from UTM8 messages. If one simply ignores the transcoding in and out, leaving the core only to handle clean UTF8 strings what non-trivial things are left? Could this be a candidate for a SOC project? -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk --- ## #72623 — Pierre Joye — 2014-02-15T05:43:16+00:00 URL: https://externals.io/message/61768#72623 · In reply to: #72613 > > What we really need is an awesome small and fast Unicode library that > does everything ICU does but faster and in less code while using UTF-8 > as its internal storage so we don't have to convert on each and every > operation. There are a ton of non-obvious things beyond simple string > manipulation. String collation alone is massively complicated, for example. http://www.public-software-group.org/utf8proc-documentation looks interesting. There are other but it has to be chosen very carefully :) Cheers, Pierre --- ## #72624 — Lester Caine — 2014-02-15T09:50:22+00:00 URL: https://externals.io/message/61768#72624 · In reply to: #72623 Pierre Joye wrote: > > […] > example. > > http://www.public-software-group.org/utf8proc-documentation looks > interesting. There are other but it has to be chosen very carefully :) I'd looked at that last night http://lsces.co.uk/wiki/PHP6+unicode+core is the start of a crib sheet but searching with google really is painful so any other links would be welcome! -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk --- ## #72627 — Pierre Joye — 2014-02-15T11:46:49+00:00 URL: https://externals.io/message/61768#72627 · In reply to: #72623 > > > > > […] > > > […] example. > > http://www.public-software-group.org/utf8proc-documentation looks interesting. There are other but it has to be chosen very carefully :) https://github.com/josephg/librope claims to be fast and compliant. Added to my list. > Cheers, > Pierre --- ## #72629 — Lester Caine — 2014-02-15T12:32:06+00:00 URL: https://externals.io/message/61768#72629 · In reply to: #72627 Pierre Joye wrote: > > > […] > example. > > […] > interesting. There are other but it has to be chosen very carefully :) > > https://github.com/josephg/librope claims to be fast and compliant. Added > to my list. If I'm reading that correctly it does the substring stuff on already converted UTF8 strings? It needs a UTF8 conversion of even a UTF16 string to work ... I think? -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk --- ## #72640 — Lester Caine — 2014-02-16T13:08:42+00:00 URL: https://externals.io/message/61768#72640 · In reply to: #72614 Lester Caine wrote: > If one simply ignores the transcoding in and out, leaving the core only to > handle clean UTF8 strings what non-trivial things are left? Could this be a > candidate for a SOC project? Has anybody looked at the U_CHARSET_IS_UTF8 flag in ICU? The only reference I've found to it is on the UTF-8 page of the ICU site http://userguide.icu-project.org/strings/utf-8 but it would seem to be their attempt to remove the overheads of UTF-16 conversions when the base character set is already UTF-8? The bit I'm having trouble with is it's link to UCONFIG_NO_CONVERSION which would seem to disable any conversion filter, but we still want to convert into and export from UTF-8 in the outside world, so I don't see why that is appropriate? U_WCHAR_IS_UTF32 would seem to simplify codepoint based activity by using UTF-32 'strings' when looking at character based processing. This is how I've been viewing handling 'character' based string handling anyway. Rather than introducing the problems UTF-16 seems to create here, but I'm not sure what happens on windows based platforms here. It seems UTF-16 is the default for windows API in ICU. My simplistic view of things seems to think there are basically three string lengths ... 1/ Number of bytes for buffer 2/ Number of code points ( characters + control and embellishment ? ) 3/ Number of glyphs ( option to display or hide control codes as in ASCII ) But this has now been confused by the introduction of NFD/NFC/NFKC/NFKD? Which will vary all of the above in some cases? Being somewhat linguistically challenged, while I understand the concepts such as accents, Would standardising on say NFD help with actions like lower/upper conversion, or does accenting a character sometimes change it's alphabetical order so collations need the 'NFC' form to sort by? I think that what is clear is that while there may be a single 'UTF-8' writing standard, sorting collations are even more diverse than the previous codesets? Firebird has always managed COLLATION as a separate filter to CHARACTER SET, and allows individual fields to have their own COLLATION so we can index on different languages within the one table. I'm thinking that this may be required when adding sorting in a UTF-8 based setup? Rather than specifying 'encoding' one simply specifies 'collation' where it varies from the basic rules? -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk --- ## #72641 — Stas Malyshev — 2014-02-17T03:13:54+00:00 URL: https://externals.io/message/61768#72641 · In reply to: #72613 Hi! > operation. There are a ton of non-obvious things beyond simple string > manipulation. String collation alone is massively complicated, for example. Oh yes, and if somebody thinks case sensitivity is weird now, wait until Unicode gets into play. There for some chars when you change the case string length changes, and for some conversion is not roundtrip-safe. And you have various long form/short form combining issues which means you need to normalize everything on every corner. So letting Unicode into things like identifiers opens a huge container of worms. Also, if one wants to appreciate what other cans of worms are hiding there, I recommend this oldie but goodie: http://stackoverflow.com/a/6163129/214196 It's about Perl, but we'd have many of the same issues. -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 --- ## #72644 — Yasuo Ohgaki — 2014-02-17T04:56:57+00:00 URL: https://externals.io/message/61768#72644 · In reply to: #72641 Hi Stas, On Mon, Feb 17, 2014 at 12:13 PM, Stas Malyshev wrote: > > […] > example. > > Oh yes, and if somebody thinks case sensitivity is weird now, wait until > Unicode gets into play. There for some chars when you change the case > string length changes, and for some conversion is not roundtrip-safe. > And you have various long form/short form combining issues which means > you need to normalize everything on every corner. So letting Unicode > into things like identifiers opens a huge container of worms. > Also, if one wants to appreciate what other cans of worms are hiding > there, I recommend this oldie but goodie: > http://stackoverflow.com/a/6163129/214196 > It's about Perl, but we'd have many of the same issues. Nice article. I mostly agree. "Code that converts unknown characters to ? is broken, stupid, braindead, and runs contrary to the standard recommendation, which says NOT TO DO THAT! RTFM for why not." While I agree this (It's BAD to accept broken text as valid input), there are situations that programmer has to handle broken text. Ruby finally admits scrab method is needed. It's available from Ruby 2.1.0. Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net