# [RFC] Unicode Text Processing - Thread: https://externals.io/message/119149 - Original archive: https://news-web.php.net/php.internals/119149 - Started by Derick Rethans on 2022-12-15T15:34:01+00:00 - Messages: 38 - RFC: Unicode Text Processing (Inactive) — https://externals.io/rfc/unicode_text_processing.md - Page 1 of 2: messages 1 to 25 (25 per page, add `?page=N` to the URL) - Next page: https://externals.io/message/119149.md?page=2 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · RFCs at https://externals.io/rfc.md · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #119149 — Derick Rethans — 2022-12-15T15:34:01+00:00 URL: https://externals.io/message/119149#119149 Hi, I have just published an initial draft of the "Unicode Text Processing" RFC, a proposal to have performant unicode text processing always available to PHP users, by introducing a new "Text" class. You can find it at: https://wiki.php.net/rfc/unicode_text_processing I'm looking forwards to hearing your opinions, additions, and suggestions — the RFC specifically asks for these in places. cheers, Derick -- https://derickrethans.nl | https://xdebug.org | https://dram.io Author of Xdebug. Like it? Consider supporting me: https://xdebug.org/support Host of PHP Internals News: https://phpinternals.news mastodon: @derickr@phpc.social @xdebug@phpc.social twitter: @derickr and @xdebug --- ## #119150 — Andreas Heigl — 2022-12-15T16:05:57+00:00 URL: https://externals.io/message/119149#119150 · In reply to: #119149 Hey Derick, Hey all. > Hi, > > I have just published an initial draft of the "Unicode Text Processing" > RFC, a proposal to have performant unicode text processing always > available to PHP users, by introducing a new "Text" class. > > You can find it at: > https://wiki.php.net/rfc/unicode_text_processing > > I'm looking forwards to hearing your opinions, additions, and > suggestions — the RFC specifically asks for these in places. Thanks for tackling this immense topic. I see a few challenges in the approach. My first question was: Why do we need a new implementation of the ICU library? Creating a userland implementation that wraps the currently existing mb-string and ICU functions into a class that allows better usability shouldn't add that much of a performance penalty. And including the mb-string and the intl extension by default wouldn't hurt. That way there would be no added maintenance burden on the core developers. In addition to that it looked to me that there are multiple things mixed up in this Text-class. If we want a Text-class to handle Unicode strings in a better way, why does the string itself need to be Locale-aware? The string itself is a collection of Unicode-Codepoints referencing Characters and Graphemes. Does the string itself need to be aware of a locale to aid in sorting? It needs to be aware of the internal normalization form for character-comparison for sure. But I would rather see a Normalizer handle normalization of the Text-content instead of the Text-class handling that itself. Similarily I'd see the Transliteration done by a separate class. Which then strongly looks similar to the Intl-extension. Which brings me back to the question: Do we really need a second Intl-extension in the core? I'm ambivalent about this. On the one hand it could make some things for sure easier. On the other hand it adds burden onto the core-developers that could be avoided by providing the intl (and mb-string) extension by default instead of having to add them separately. And then find a group if people willing to build a userland implementation. And yes, I know the intl-extension is everything but easy to use. Especially in the quirky edge-cases regarding Transliteration and Normalization. But the issue usually isn't using it but finding the appropriate documentation on the ICU page. Helping the ICU to improve on that documentation would also be a huge benefit. To all those trying to use the Intl-extension right now. But that's just my 0.02€ Cheers Andreas > > cheers, > Derick > > > > -- ,,, (o o) +---------------------------------------------------------ooO-(_)-Ooo-+ | Andreas Heigl | | mailto:andreas@heigl.org N 50°22'59.5" E 08°23'58" | | https://andreas.heigl.org | +---------------------------------------------------------------------+ | https://hei.gl/appointmentwithandreas | +---------------------------------------------------------------------+ | GPG-Key: https://hei.gl/keyandreasheiglorg | +---------------------------------------------------------------------+ --- ## #119151 — Deleu — 2022-12-15T16:26:06+00:00 URL: https://externals.io/message/119149#119151 · In reply to: #119149 > Hi, > > I have just published an initial draft of the "Unicode Text Processing" > RFC, a proposal to have performant unicode text processing always > available to PHP users, by introducing a new "Text" class. > > You can find it at: > https://wiki.php.net/rfc/unicode_text_processing > > I'm looking forwards to hearing your opinions, additions, and > suggestions — the RFC specifically asks for these in places. > > cheers, > Derick > I appreciate the work behind this RFC. Although I can't comment much on the utility itself as I never really used `ext-intl` or much of UTF-8/16 stuff, I think PHP lacks A LOT of built-in classes to make matters simpler and welcoming. Sure, anybody in userland could write a class, but that leads to lots of implementations and an overwhelming amount of choices to be made. If this class can cover 80% of the use-case, folks can extend it to build their remaining 20% advanced use-case and PHP becomes easier. I really look forward to more basic utility-classes built-in and ready to go. -- Marco Deleu --- ## #119152 — tim@bastelstu.be — 2022-12-15T16:40:09+00:00 URL: https://externals.io/message/119149#119152 · In reply to: #119149 Hi > You can find it at: > https://wiki.php.net/rfc/unicode_text_processing > > I'm looking forwards to hearing your opinions, additions, and > suggestions — the RFC specifically asks for these in places. Some first remarks: ---------- replaceText(): In the replaceText() section the description refers to a non-existent parameters '$maxReplacements' and '$collator'. Also the second paragraph in that section looks like a word is missing there. ---------- getPositionOfFirstOccurrence(): I agree this is too long. How about: - findOffset() - findOffsetLast() And for returnFromFirstOccurence(): - startingWith() - startingWithLast() ---------- firstToTitle(): I don't see how this differs from toTitle() and/or firstToUpper(). An example would likely be helpful. ---------- wordsToTitle(): Likewise. ---------- The return of many methods is not explicitly listed. I'd put it everywhere to make it 100% clear that all those functions return 'Text'. Best regards Tim Düsterhus --- ## #119153 — Christoph M. Becker — 2022-12-15T16:56:27+00:00 URL: https://externals.io/message/119149#119153 · In reply to: #119149 > I have just published an initial draft of the "Unicode Text Processing" > RFC, a proposal to have performant unicode text processing always > available to PHP users, by introducing a new "Text" class. > > You can find it at: > https://wiki.php.net/rfc/unicode_text_processing > > I'm looking forwards to hearing your opinions, additions, and > suggestions — the RFC specifically asks for these in places. | As the implementation requires ICU, this would also mean that PHP will | depend on the ICU library. Our current stance is that a minimal PHP should be buildable without requiring any "non-standard" libraries; this is the reason why we bundle PCRE. If we wanted to stick with that policy, we would need to bundle ICU, what might not be the best idea – it's generally not great to have bundled libraries which are still maintained outside of php-src, and especially for such huge libraries. -- Christoph M. Becker --- ## #119154 — Paul Crovella — 2022-12-15T17:15:19+00:00 URL: https://externals.io/message/119149#119154 · In reply to: #119149 > https://wiki.php.net/rfc/unicode_text_processing A few quick thoughts: > The constructor will also convert the given text to Unicode Canonical Form. By this do you mean Normalization Form C (NFC)? "Unicode Canonical Form" isn't a phrase I'm familiar with. Assuming so, are modified texts (e.g. via join, replaceText, reverse) re-normalized? --- > The constructor will also strip out a BOM (Byte-Order-Mark) character, if present. This is also known as ZWNBSP (Zero Width No-Break Space). Will only a leading instance be stripped? If so, how can someone search for it (or a substring beginning with it) given that: > If an argument to any of the methods is listed as string|Text, passing in a string value will have the same semantics as replacing the passed value with new Text($string). and all the search methods take `string|Text $search`. --- Why is this being introduced directly into PHP core rather than first an extension where it's easier to shake out the interface and behavior? --- ## #119155 — tim@bastelstu.be — 2022-12-15T18:16:36+00:00 URL: https://externals.io/message/119149#119155 · In reply to: #119150 Hi > I see a few challenges in the approach. My first question was: Why do we > need a new implementation of the ICU library? Creating a userland > > […] > > I'm ambivalent about this. On the one hand it could make some things for > sure easier. On the other hand it adds burden onto the core-developers > that could be avoided by providing the intl (and mb-string) extension by > default instead of having to add them separately. And then find a group > if people willing to build a userland implementation. Because a programming language needs a standard library, otherwise one could just use JavaScript and pull in a dependency for 'is-odd' or left-padding. The biggest advantage this proposal has compared to ext/intl is that it *adds a new data type*. If you receive a 'Text' object then you are guaranteed to have valid Unicode/UTF-8 inside of it. It also provides a OO API around text/string processing functionality, which is something users have desired for quite some time already ("scalar objects"). The addition of a new data type is also a reason why this cannot usefully be implemented in userland alone: It would require every developer to standardize on a single userland implementation, as otherwise you need bridges to convert between the different representations of various userland libraries (or need to round-trip through the standard 'string' type), which I consider to be a non-starter for something as fundamental as text processing. Both because it adds complexity and because it will kill performance. As the RFC notes, an explicit design goal is to keep the API simple and focused, so I don't expect much ongoing maintenance burden here. Especially if all the heavy lifting is off-loaded to ICU. Any convenience functionality can then be be provided in userland based on the building blocks provided by PHP itself, with the benefit that userland libraries are going to be fully interoperable because they all use the standard 'Text' type that is guaranteed to be available [1]. Best regards Tim Düsterhus [1] The 'Text' class should likely be made final, because folks might otherwise rely on a specific userland extension, preventing actual interoperability. --- ## #119156 — Sara Golemon — 2022-12-15T18:41:16+00:00 URL: https://externals.io/message/119149#119156 · In reply to: #119149 > I have just published an initial draft of the "Unicode Text Processing" > RFC, a proposal to have performant unicode text processing always > available to PHP users, by introducing a new "Text" class. > > You can find it at: > https://wiki.php.net/rfc/unicode_text_processing > > I'm looking forwards to hearing your opinions, additions, and > suggestions — the RFC specifically asks for these in places. > > Obviously, hurdle one is making the ICU library a requirement for building PHP. I'd almost make that it's own milestone in this project with the introduction of the Text class as a separate followon. A very casual (IANAL) read of the ICU license doesn't seem to make this a problem, so it may be more of a question of whether we put this on people wanting to build PHP. ICU is pretty widely available and used, so I also don't see this as a major stumbling block. Question 2 is that class. I know folks have been clammoring for a `String` class for some time and this actually fills that niche quite well. A part of me wonders if we can overload it a little to provide a psuedo locale of "binary" so that users can, optionally, treat it like a more generalized String class in specific cases, storing a normal `char*` zend_string under the hood in that case. Possibly as a specialzation tree. /* names as examples only */ interface Stringy { /* define all those APIs */ } class Text implements Stringy { /* ... */ } class BinaryString implements Stringy { /* ... */ } I think you'd get a lot more buy-in from the folks who worry that UTF16 is overhead they don't want, but who do like the idea of an OOPy string. It also provides a migration path to avoid having to rethink byte vs grapheme conversions up front, instead deferring that part of a migration till later. Overall, I'm more positive on this than negative, and I eagerly await the rest of this thread. -Sara --- ## #119158 — Deleu — 2022-12-15T18:51:49+00:00 URL: https://externals.io/message/119149#119158 · In reply to: #119155 > > [1] The 'Text' class should likely be made final, because folks might > otherwise rely on a specific userland extension, preventing actual > interoperability. > I'm fond of final classes but in here I think it *adds* burden to core developers. As you said it yourself having a Type within PHP will help interoperability. Having this type be final will hurt interoperability because everyone's wrapper will be different. This may lead to the community requesting more changes to core. --- ## #119159 — tim@bastelstu.be — 2022-12-15T19:53:25+00:00 URL: https://externals.io/message/119149#119159 · In reply to: #119158 Hi > > […] > > I'm fond of final classes but in here I think it *adds* burden to core > developers. As you said it yourself having a Type within PHP will help > interoperability. Having this type be final will hurt interoperability > because everyone's wrapper will be different. This may lead to the > community requesting more changes to core. The wrappers may be different, but they would all be expected to provide a method to retrieve the underlying 'Text' object that every wrapper would know how to use, without the need to actually *convert* between different representations. Furthermore if the class is not final, then every future (method) addition to the class API would be a possible breaking change, if the method is already in use in a userland subclass. For the kind of methods you would add to the 'Text' class there usually is only one reasonable name (there's only so many possible names for "replace", "pad", "split" or similar). Thus any future additions would result in headaches, because (1) either userland breaks, because (2) some less-than-great method name needs to be used or because (3) the addition is not going to happen at all, because (1) or (2) are both unacceptable. Then there's also the point about subclasses not necessarily be a drop-in replacement (either because methods are overriden if not final, or because the immutability is violated). We already had a similar discussion recently with regard to readonly classes, but I'm not sure a gentleman's agreement to not create "incompatible" subclasses is strong enough of a guarantee for something that is a similarly basic build block as the 'string' type itself [1]. Best regards Tim Düsterhus [1] Java's 'String' class is final, I would assume for the same reasons, see also: https://softwareengineering.stackexchange.com/q/97437. --- ## #119160 — Jakub Zelenka — 2022-12-15T20:19:32+00:00 URL: https://externals.io/message/119149#119160 · In reply to: #119153 On Thu, Dec 15, 2022 at 4:56 PM Christoph M. Becker wrote: > > > > […] > > | As the implementation requires ICU, this would also mean that PHP will > | depend on the ICU library. > > Our current stance is that a minimal PHP should be buildable without > requiring any "non-standard" libraries; this is the reason why we bundle > PCRE. If we wanted to stick with that policy, we would need to bundle > ICU, what might not be the best idea – it's generally not great to have > bundled libraries which are still maintained outside of php-src, and > especially for such huge libraries. > > I agree with this. Bundling ICU doesn't seem like a good idea. Wouldn't be better to base on something smaller that can be bundled and does the job? For example NJS and QuickJS use their own implementations which seem to be fine. Especially https://github.com/bellard/quickjs/blob/master/libunicode.c seems like something that we could fork and maintain potentially. Cheers Jakub --- ## #119161 — Rowan Tommins — 2022-12-15T22:20:15+00:00 URL: https://externals.io/message/119149#119161 · In reply to: #119149 > I have just published an initial draft of the "Unicode Text Processing" > RFC, a proposal to have performant unicode text processing always > available to PHP users, by introducing a new "Text" class. > > You can find it at: > https://wiki.php.net/rfc/unicode_text_processing > > I'm looking forwards to hearing your opinions, additions, and > suggestions — the RFC specifically asks for these in places. As others have said already, thank you for taking a stab at this important topic. I agree that it would be a really useful feature for the language, but it's also a really difficult one to get right. Here are my initial thoughts... # Design Process Rather than designing the whole class "on paper", I think this really needs to be built as a prototype, where we can build up documentation and tests, plug variations into some real life scenarios, and have separate discussions about different details. If we limit ourselves initially to features already exposed by ext/intl (I think everything proposed so far is?), a prototype doesn't even need to be an extension, it can be in pure PHP. Then once the design is finalised, you have a ready-made polyfill for older PHP versions, and a set of tests for the native version :) We might also want to do some general investigation of what other languages and frameworks provide, and which decisions have proven good or bad in practice. # Lossy Transforms Automatic normalisation and stripping of BOMs seems useful, but it immediately rules out use of this class for anything where you want to get back what you put in. For instance, if an ORM used Text instances for strings in data models, it would generate extra Update queries on the database even when the string wasn't otherwise changed. I think it would be better to make this easy but explicit. # UTF-8 on the outside, UTF-16 on the inside I know this will be a very common combination, but it feels odd that an application which actually wanted to work with UTF-16 would need to perform round-trips through UTF-8 just to use this class. It should at least be possible to specify the encoding on input and output. Ruby takes an interesting approach where strings are tagged with their current binary encoding, and only converted to another form if actually required. If your input layer says "$name = new Text($_GET['name'], 'Windows-1252');" and your output layer says "echo $name->asBytes('Windows-1252');" the overhead of converting to UTF-16 can be skipped entirely, unless something in between says "$name = $name->wordsToUpper()". This also removes another source of lossy transformation, since some encoding conversions aren't perfectly reversible (e.g. the source encoding has more than one byte sequence mapped to the same Unicode code point). # Internationalisation Having locale and collation as state on the object, rather than parameters on relevant methods, feels like muddling responsibilities. It makes it hard to reason about what exactly some of the methods will do: Can I trust that this object will give me a sensible result from compareWith, or has it been assigned a collation somewhere else? What exactly will be the definition of "replace" or "contains" for this pair of objects? How users will work with these also needs careful thought - your first listed design goal is "keep it simple", but under locales and Internationalisation is the worrying sentence "This will require extensive documentation". This is one of those places where "doing it right" is really hard to combine with "making it easy", because language is inherently complex, but users will expect a simple answer to "how do I make it case-insensitive?" # Allowing other abstractions I 100% approve of your use of grapheme clusters, rather than code points, as the primary unit; so many implementations get that wrong. However, when interacting with other systems, reasoning about bytes (or sometimes even codepoints) is essential. One function that I would really like to see, for instance, is a grapheme-aware version of mb_strcut, to solve tasks like: "encode this abstract Unicode string as UTF-16BE, truncated to at most 200 bytes, without breaking apart any grapheme clusters". Thanks again for getting the ball rolling, and I look forward to helping iterate the design. Regards, -- Rowan Tommins [IMSoP] --- ## #119162 — Tim Starling — 2022-12-16T03:21:03+00:00 URL: https://externals.io/message/119149#119162 · In reply to: #119149 > Hi, > > I have just published an initial draft of the "Unicode Text Processing" > RFC, a proposal to have performant unicode text processing always > available to PHP users, by introducing a new "Text" class. Using "collator" and "locale" interchangeably seems imprecise. If the input is an ICU locale string, then I think you should just call it locale. Then the user will be armed with the correct terminology when they go looking for more information in the ICU manual. In ICU, case conversion and BreakIterator need a locale, not a collator. I'm concerned about the time order of using grapheme offsets. For example, is subString() O(N) in $offset? If the idea is to be easy to use and performant, you don't want to have subtle algorithmic complexity traps. I'm probably not the target audience for this class, since I'm generally looking for maximum flexibility, not minimum complexity. As such, I'd like intl to have better documentation and more features. The RFC has a family of locale-aware case conversion functions which do not exist in intl. This was raised as an issue during the discussion on my ASCII case conversion RFC. It would be great if intl could get those functions too. I think you should consider making this Text class a part of the intl extension. You're adding a class which is similar to the classes in that extension. In terms of data, it's like IntlChar, except it's for strings not characters. Its constructor takes an ICU locale string, just like IntlBreakIterator or MessageFormatter. I can understand if you don't want to follow all the existing conventions of the intl extension. But if that is the rationale for the RFC, I'd like to see a discussion of the specific usability problems with the intl extension. -- Tim Starling --- ## #119164 — Daniel Wolfe — 2022-12-16T05:09:49+00:00 URL: https://externals.io/message/119149#119164 · In reply to: #119149 > Hi, > > I have just published an initial draft of the "Unicode Text Processing" > RFC, a proposal to have performant unicode text processing always > available to PHP users, by introducing a new "Text" class. > > You can find it at: > https://wiki.php.net/rfc/unicode_text_processing > > I'm looking forwards to hearing your opinions, additions, and > suggestions — the RFC specifically asks for these in places. > > cheers, > Derick > > I love the idea, but we may want to think about using the terms `trimStart` and `trimEnd` instead of `trimLeft` and `trimRight`. Obviously, when the text in the parameter is in a right-to-left language (e.g., Hebrew, Arabic), `trimLeft` and `trimRight` are going to do the opposite of what one may expect from the names of the functions. I know that identifiers like `ltrim` and `rtrim` are longstanding conventions which everyone understands that they **really** mean trimming from the start and end respectively, but this may be an opportunity to amend the inaccuracy. If this ever gets expanded to add padding functions, those should probably be named `padStart` and `padEnd` as well. --- ## #119165 — Rowan Tommins — 2022-12-16T11:21:46+00:00 URL: https://externals.io/message/119149#119165 · In reply to: #119162 > > I'm concerned about the time order of using grapheme offsets. For > example, is subString() O(N) in $offset? If the idea is to be easy to > use and performant, you don't want to have subtle algorithmic > complexity traps. > This is a good point; it's certainly true of existing functions, like `grapheme_strlen()`, and indeed `mb_strlen()`, which has to iterate variable width code points. Perhaps we could take advantage of having a stateful object and internally optimise this in some way, such as caching a partial lookup table of graphemes to byte offsets. For instance, the table might look like this: 10: 22 20: 50 30: 70 35: 82; LAST Then $string->subString(23, 20) would: * take a pointer to byte 50 * pass it to the ICU grapheme iterator to skip over 3 graphemes; let's say that takes us to byte 58 * since 23 + 20 > 35, the rest of the string is included * the new object could construct an offset table without examining the string: 7: 12 (grapheme 30 - 23; byte 70 - 58) 12: 24; LAST (grapheme 35 - 23; byte 82 - 58) Whether this complexity would pay off in real-world scenarios, I don't know, but if people started using this for all the text on an application, I can see longer strings becoming a more common use case. Regards, -- Rowan Tommins [IMSoP] --- ## #119167 — Derick Rethans — 2022-12-16T12:55:46+00:00 URL: https://externals.io/message/119149#119167 · In reply to: #119160 > On Thu, Dec 15, 2022 at 4:56 PM Christoph M. Becker > wrote: > > > […] > I agree with this. Bundling ICU doesn't seem like a good idea. > Wouldn't be better to base on something smaller that can be bundled > and does the job? For example NJS and QuickJS use their own > implementations which seem to be fine. Especially > https://github.com/bellard/quickjs/blob/master/libunicode.c seems like > something that we could fork and maintain potentially. I have no intentions of bundling ICU. That'd be a crazy thing to do. Instead, the current proposal is to make PHP depend on libicu. I realise that this is against our current stance, but considering that 1. most (if not all) Linux distributions ignore our bundled libraries any way as per their policies; 2. libicu is pretty much available everywhere; and 3. I am not proposing to require the latest and greatest, I believe we can safely rely on it being available. I'm not opposed to using something else than ICU Most of the other unicode related libraries that I had a quick look at, either provide a small subset — either just character properties, or graphemes, none of them also take care of collation/locales and transliteration. I am also weary about some of these library's development and future proofness. ICU won't have these problems. cheers, Derick -- https://derickrethans.nl | https://xdebug.org | https://dram.io Author of Xdebug. Like it? Consider supporting me: https://xdebug.org/support Host of PHP Internals News: https://phpinternals.news mastodon: @derickr@phpc.social @xdebug@phpc.social twitter: @derickr and @xdebug --- ## #119168 — Derick Rethans — 2022-12-16T13:25:39+00:00 URL: https://externals.io/message/119149#119168 > > > […] > > A few quick thoughts: > > > […] > > By this do you mean Normalization Form C (NFC)? "Unicode Canonical Form" isn't > a phrase I'm familiar with. Yes. I've seen both phrases used, so I'll add NFC in brackets. > Assuming so, are modified texts (e.g. via join, replaceText, reverse) > re-normalized? Yes — although I do not expect that to change anything, as normalisation usually happens *in* a grapheme, and not between them. I suspect there might be some indian languages where that is proven wrong though. > > […] > > This is also known as ZWNBSP (Zero Width No-Break Space). Will only a > leading instance be stripped? If so, how can someone search for it (or > a substring beginning with it) given that: > > > […] > > and all the search methods take `string|Text $search`. I hadn't realised this is now used for both use cases. I've just read[1] "If the BOM character appears in the middle of a data stream, Unicode says it should be interpreted as a "zero-width non-breaking space" (inhibits line-breaking between word-glyphs). In Unicode 3.2, this usage is deprecated in favor of the "Word Joiner" character, U+2060.[1] This allows U+FEFF to be used only as a BOM. " This indicates that this might not be a problem. Would you have a better suggestion? > --- > > Why is this being introduced directly into PHP core rather than first an > extension where it's easier to shake out the interface and behavior? It will be developed as an extension inside the ext/ branch, pretty much like ext/standard or ext/date; but if it is not in core, very few people will use it, defeauting the whole point of the effort. cheers, Derick [1] https://en.wikipedia.org/wiki/Byte_order_mark#Usage -- https://derickrethans.nl | https://xdebug.org | https://dram.io Author of Xdebug. Like it? Consider supporting me: https://xdebug.org/support Host of PHP Internals News: https://phpinternals.news mastodon: @derickr@phpc.social @xdebug@phpc.social twitter: @derickr and @xdebug --- ## #119169 — Derick Rethans — 2022-12-16T13:28:56+00:00 URL: https://externals.io/message/119149#119169 · In reply to: #119156 > > > > […] > Question 2 is that class. I know folks have been clammoring for a > `String` class for some time and this actually fills that niche quite > well. A part of me wonders if we can overload it a little to provide > a psuedo locale of "binary" so that users can, optionally, treat it > like a more generalized String class in specific cases, storing a > normal `char*` zend_string under the hood in that case. Possibly as a > specialzation tree. An alternative could be to just have this as an implementation detail, in case the associated locale/collation is C/root. Then nobody needs to worry about it, *but* it would mean implementing everything twice. Which I am not too keen on, especially because we have such a wide array of operations on strings already. cheers, Derick -- https://derickrethans.nl | https://xdebug.org | https://dram.io Author of Xdebug. Like it? Consider supporting me: https://xdebug.org/support Host of PHP Internals News: https://phpinternals.news mastodon: @derickr@phpc.social @xdebug@phpc.social --- ## #119170 — Derick Rethans — 2022-12-16T13:54:40+00:00 URL: https://externals.io/message/119149#119170 · In reply to: #119155 > [1] The 'Text' class should likely be made final, because folks might > otherwise rely on a specific userland extension, preventing actual > interoperability. Yes, I intended to do this, but forgot to include it. I've updated the RFC. cheers, Derick -- https://derickrethans.nl | https://xdebug.org | https://dram.io Author of Xdebug. Like it? Consider supporting me: https://xdebug.org/support Host of PHP Internals News: https://phpinternals.news mastodon: @derickr@phpc.social @xdebug@phpc.social twitter: @derickr and @xdebug --- ## #119171 — Derick Rethans — 2022-12-16T13:54:47+00:00 URL: https://externals.io/message/119149#119171 > > > […] > I love the idea, but we may want to think about using the terms > `trimStart` and `trimEnd` instead of `trimLeft` and `trimRight`. > Obviously, when the text in the parameter is in a right-to-left > language (e.g., Hebrew, Arabic), `trimLeft` and `trimRight` are going > to do the opposite of what one may expect from the names of the > functions. I know that identifiers like `ltrim` and `rtrim` are > longstanding conventions which everyone understands that they **really** > mean trimming from the start and end respectively, but this may be an > opportunity to amend the inaccuracy. That's an excellent point. And I've updated these names. cheers, Derick -- https://derickrethans.nl | https://xdebug.org | https://dram.io Author of Xdebug. Like it? Consider supporting me: https://xdebug.org/support Host of PHP Internals News: https://phpinternals.news mastodon: @derickr@phpc.social @xdebug@phpc.social twitter: @derickr and @xdebug --- ## #119172 — Derick Rethans — 2022-12-16T13:54:55+00:00 URL: https://externals.io/message/119149#119172 · In reply to: #119162 > > > […] > > Using "collator" and "locale" interchangeably seems imprecise. If the > input is an ICU locale string, then I think you should just call it > locale. Then the user will be armed with the correct terminology when > they go looking for more information in the ICU manual. In ICU, case > conversion and BreakIterator need a locale, not a collator. Yeah, the terms are currently used interchangably (sort of). I will update that. Although I really would not suggest that users look at the ICU manual, as it's really hard to find things in it :-) > I'm concerned about the time order of using grapheme offsets. For > example, is subString() O(N) in $offset? Yes. It would have to scan the Text. > I'm probably not the target audience for this class, since I'm > generally looking for maximum flexibility, not minimum complexity. As > such, I'd like intl to have better documentation and more features. > The RFC has a family of locale-aware case conversion functions which > do not exist in intl. This was raised as an issue during the > discussion on my ASCII case conversion RFC. It would be great if intl > could get those functions too. AFAIK Intl can do all of these things, but yes, its documentation is "sparse". However, that's not in scope of this RFC. > I think you should consider making this Text class a part of the intl > extension. You're adding a class which is similar to the classes in > that extension. In terms of data, it's like IntlChar, except it's for > strings not characters. Its constructor takes an ICU locale string, > just like IntlBreakIterator or MessageFormatter. I did consider that, and rejected that idea. Intl, although powerful, does not have an approcable API. It is also not installed or available by default, and I am not suggesting we do that. That than means that it doesn't fit the design goals here (having it always available). cheers, Derick -- https://derickrethans.nl | https://xdebug.org | https://dram.io Author of Xdebug. Like it? Consider supporting me: https://xdebug.org/support Host of PHP Internals News: https://phpinternals.news mastodon: @derickr@phpc.social @xdebug@phpc.social twitter: @derickr and @xdebug --- ## #119173 — Derick Rethans — 2022-12-16T13:55:02+00:00 URL: https://externals.io/message/119149#119173 · In reply to: #119161 > > > […] > > > As others have said already, thank you for taking a stab at this important > topic. I agree that it would be a really useful feature for the language, but > it's also a really difficult one to get right. Here are my initial thoughts... > > # Design Process > > Rather than designing the whole class "on paper", I think this really needs to > be built as a prototype, where we can build up documentation and tests, plug > variations into some real life scenarios, and have separate discussions about > different details. If we limit ourselves initially to features already exposed > by ext/intl (I think everything proposed so far is?), a prototype doesn't even > need to be an extension, it can be in pure PHP. Then once the design is > finalised, you have a ready-made polyfill for older PHP versions, and a set of > tests for the native version :) I do not want a polyfill. These already exist for intl and friends. I had no intention to design everything up front though, and it is likely that I missed useful methods. This is not going to be right in a single implementation. > # UTF-8 on the outside, UTF-16 on the inside > > I know this will be a very common combination, but it feels odd that an > application which actually wanted to work with UTF-16 would need to perform > round-trips through UTF-8 just to use this class. It should at least be > possible to specify the encoding on input and output. I disgree. Users should not care what is used in the implementation. It's only UTF-16 because that is what ICU's API use. I do not want the complexity of having different in/ex encodings. Perhaps 15 years ago that was useful to have, but right now, everything should be UTF-8 on the interface layer, that is, if you care about internationalisation. > # Internationalisation > > Having locale and collation as state on the object, rather than > parameters on relevant methods, feels like muddling responsibilities. > It makes it hard to reason about what exactly some of the methods will > do: Can I trust that this object will give me a sensible result from > compareWith, or has it been assigned a collation somewhere else? What > exactly will be the definition of "replace" or "contains" for this > pair of objects? A locale/collator is an inherent property of Text (we're dealing with Text here, not strings). I do need to tidy up the wording about what locales and collations are, as I've so far used them sparingly interchangably. > How users will work with these also needs careful thought - your first listed > design goal is "keep it simple", but under locales and Internationalisation is > the worrying sentence "This will require extensive documentation". This phrase is meant to mean that the *format of the locale/collator name* needs extensive documentation. > One function that I would really like to see, for instance, is a > grapheme-aware version of mb_strcut, to solve tasks like: "encode this > abstract Unicode string as UTF-16BE, truncated to at most 200 bytes, > without breaking apart any grapheme clusters". For that to work, you need a methods that instantly returns UTF-8 strings, and not UTF-16. In the RFC, the current subString() uses int $length to mean grapheme clusters. Adding another methods to do something else, is of course possible. I'll think about it (and noted in "Open Issues"). cheers, Derick -- https://derickrethans.nl | https://xdebug.org | https://dram.io Author of Xdebug. Like it? Consider supporting me: https://xdebug.org/support Host of PHP Internals News: https://phpinternals.news mastodon: @derickr@phpc.social @xdebug@phpc.social twitter: @derickr and @xdebug --- ## #119174 — Derick Rethans — 2022-12-16T13:55:11+00:00 URL: https://externals.io/message/119149#119174 · In reply to: #119152 > > > […] > > Some first remarks: > > ---------- > > replaceText(): > > In the replaceText() section the description refers to a non-existent > parameters '$maxReplacements' and '$collator'. Also the second paragraph in > that section looks like a word is missing there. I will update this. the ''$collator'' would be an argument to the constructor of the Text/$search argument, not the method itself. > ---------- > > getPositionOfFirstOccurrence(): > > I agree this is too long. How about: > > - findOffset() > - findOffsetLast() > > And for returnFromFirstOccurence(): > > - startingWith() > - startingWithLast() I have included these as suggested names. I suspect we'll get more :-) > ---------- > > firstToTitle(): > > I don't see how this differs from toTitle() and/or firstToUpper(). An example > would likely be helpful. It doesn't differ from firstToUpper, so I will remove it. > ---------- > > wordsToTitle(): > > Likewise. It doesn't differ either from toTitle, so I will remove it. I have also added examples for the others. > ---------- > > The return of many methods is not explicitly listed. I'd put it everywhere to > make it 100% clear that all those functions return 'Text'. I will update that. cheers, Derick -- https://derickrethans.nl | https://xdebug.org | https://dram.io Author of Xdebug. Like it? Consider supporting me: https://xdebug.org/support Host of PHP Internals News: https://phpinternals.news mastodon: @derickr@phpc.social @xdebug@phpc.social twitter: @derickr and @xdebug --- ## #119175 — tim@bastelstu.be — 2022-12-16T15:21:46+00:00 URL: https://externals.io/message/119149#119175 · In reply to: #119169 Hi > > […] > > An alternative could be to just have this as an implementation detail, > in case the associated locale/collation is C/root. Then nobody needs to > worry about it, *but* it would mean implementing everything twice. Which > I am not too keen on, especially because we have such a wide array of > operations on strings already. > I rather not see this either, because if a 'Text' object may contain binary data, the type safety is lost and users cannot rely on "'Text' implies valid UTF-8" (see sibling thread). Best regards Tim Düsterhus --- ## #119176 — Andreas Heigl — 2022-12-16T15:27:41+00:00 URL: https://externals.io/message/119149#119176 · In reply to: #119175 Hey > Hi > > > > […] > > I rather not see this either, because if a 'Text' object may contain > binary data, the type safety is lost and users cannot rely on "'Text' > implies valid UTF-8" (see sibling thread). Does Text contain valid UTF-8? Or valid Unicode? As IIRC the idea was to internally use UTF-16 as encoding. In the end the internal encoding should be irrelevant to the user as long as we can assert that __toString() returns a Unicode-String in a valid encoding. And I'm with you that UTF-8 might be the best choice for that. Cheers Andreas -- ,,, (o o) +---------------------------------------------------------ooO-(_)-Ooo-+ | Andreas Heigl | | mailto:andreas@heigl.org N 50°22'59.5" E 08°23'58" | | https://andreas.heigl.org | +---------------------------------------------------------------------+ | https://hei.gl/appointmentwithandreas | +---------------------------------------------------------------------+ | GPG-Key: https://hei.gl/keyandreasheiglorg | +---------------------------------------------------------------------+ --- Next page: https://externals.io/message/119149.md?page=2