Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:119178
Message-ID: <bda68471-3106-2c9b-9494-062cf74f6b9d@bastelstu.be>
Date: Fri, 16 Dec 2022 16:48:38 +0100
MIME-Version: 1.0
Content-Language: en-US
To: Andreas Heigl <andreas@heigl.org>, internals@lists.php.net
References: <alpine.DEB.2.23.453.2212151531360.462551@singlemalt.home.derickrethans.nl>
 <CAESVnVqtSMFCJ_Uxa5Ts7+O=WqGoF+sdrjtcVEUOK2+SHi0z6g@mail.gmail.com>
 <alpine.DEB.2.23.453.2212161327180.462551@singlemalt.home.derickrethans.nl>
 <24ba312e-5965-8b8c-2506-2db9767aa3d5@bastelstu.be>
 <68e8a528-2d7b-5d0b-43b9-bd36da7a46a5@heigl.org>
In-Reply-To: <68e8a528-2d7b-5d0b-43b9-bd36da7a46a5@heigl.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Subject: Re: [PHP-DEV] [RFC] Unicode Text Processing
From: tim@bastelstu.be (=?UTF-8?Q?Tim_D=c3=bcsterhus?=)

Hi

On 12/16/22 16:27, Andreas Heigl wrote:
>> I rather not see this either, because if a 'Text' object may contain
>> binary data, the type safety is lost and users cannot rely on "'Text'
>> implies valid UTF-8" (see sibling thread).
> 
> Does Text contain valid UTF-8? Or valid Unicode? As IIRC the idea was to
> internally use UTF-16 as encoding.
> 
> In the end the internal encoding should be irrelevant to the user as
> long as we can assert that __toString() returns a Unicode-String in a
> valid encoding. And I'm with you that UTF-8 might be the best choice for
> that.

The RFC already specifies that the inputs (__construct()) and outputs 
(__toString()) must/will be UTF-8 strings in 
https://wiki.php.net/rfc/unicode_text_processing#basics.

So for all intents and purposes "'Text' implies valid UTF-8" is what 
this guarantees, because the internal representation will not be visible 
to the user.

Best regards
Tim Düsterhus