Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:121176
Message-ID: <1f3948c1-7b4c-430c-b686-031274dbd328@gmail.com>
Date: Fri, 29 Sep 2023 18:38:27 +0200
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Content-Language: en-US
To: internals@lists.php.net
References: <c7a7f77b-8187-23f7-7c02-c92f1b538039@gmail.com>
 <48c7bb29-a52c-416e-b855-be2746dc7a84@gmail.com>
 <39900ce4-56b1-2397-ee9c-c9b7086b33cb@mabe.berlin>
 <0c2d7e79-2c74-4263-81ec-ac8832ca50bb@gmail.com>
 <eda0b582-e452-d72b-bf69-1c3a6f9a5ffd@bastelstu.be>
In-Reply-To: <eda0b582-e452-d72b-bf69-1c3a6f9a5ffd@bastelstu.be>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Subject: Re: [PHP-DEV] Re: [RFC] [Discussion] DOM HTML5 parsing and
 serialization support
From: dossche.niels@gmail.com (Niels Dossche)

Hi Tim

On 29/09/2023 18:06, Tim Düsterhus wrote:
> Hi
> 
> On 9/29/23 17:45, Niels Dossche wrote:
>> Right, we follow the HTML spec in this regard. Roughly speaking we determine the charset in the following order of priorities.
>> If one option fails, it will fall through to the next one.
>> 1. The Content-Type HTTP header from which you loaded the document.
> 
> How would the new document classes make use of that? The HTTP header is transmitted out-of-band with regard to the actual payload.
> 
> Is this referring to passing a `http://` path to HTMLDocument::createFromFile()? This would be unusable for everyone who manually downloads the document, e.g. using a PSR-18 HTTP Client.

When the stream wrapper contains header information that information is used indeed.
That would unfortunately indeed mean it's unusable when manually passed in.

> 
> It might actually be necessary to add an encoding parameter to these functions, but it would need to take priority over anything implicit. The current $encoding of the global \DOMDocument has the problem that it doesn't take priority/is ignored entirely. Manually converting the document to UTF-8 before passing it to \DOMDocument has the problem that the meta tag in the document takes priority.
> 
> In fact I've run into this issue before for the implementation of a rich embed feature. We're downloading the websites using Guzzle and attempt to make sense of them with \DOMDocument. However we can't reliably force the encoding given within the 'content-type' response header, so in some cases we obtain mojibake.
> 
> This encoding parameter would likely need to be `?string $encoding = null` with everything non-null overwriting implicit detection and null meaning implicit detection in the order of priorities you mentioned.

I agree.
I'll add the optional arguments `?string $override_encoding = null` to XML/HTMLDocument::createFromString and XML/HTMLDocument::createFromFile.
I'd call it override_encoding to emphasize it's about overriding the behaviour.

> 
>> 2. BOM sniffing in the content. I.e. UTF-8 with BOM and UTF-16 LE/BE prepend the content with byte markers. This is used to detect encoding.
>> 3. Meta tag in the content.
> 
> Best regards
> Tim Düsterhus

Kinds regards
Niels