Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:121175
Message-ID: <eda0b582-e452-d72b-bf69-1c3a6f9a5ffd@bastelstu.be>
Date: Fri, 29 Sep 2023 18:06:52 +0200
MIME-Version: 1.0
Content-Language: en-US
To: Niels Dossche <dossche.niels@gmail.com>, internals@lists.php.net
References: <c7a7f77b-8187-23f7-7c02-c92f1b538039@gmail.com>
 <48c7bb29-a52c-416e-b855-be2746dc7a84@gmail.com>
 <39900ce4-56b1-2397-ee9c-c9b7086b33cb@mabe.berlin>
 <0c2d7e79-2c74-4263-81ec-ac8832ca50bb@gmail.com>
In-Reply-To: <0c2d7e79-2c74-4263-81ec-ac8832ca50bb@gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Subject: Re: [PHP-DEV] Re: [RFC] [Discussion] DOM HTML5 parsing and
 serialization support
From: tim@bastelstu.be (=?UTF-8?Q?Tim_D=c3=bcsterhus?=)

Hi

On 9/29/23 17:45, Niels Dossche wrote:
> Right, we follow the HTML spec in this regard. Roughly speaking we determine the charset in the following order of priorities.
> If one option fails, it will fall through to the next one.
> 1. The Content-Type HTTP header from which you loaded the document.

How would the new document classes make use of that? The HTTP header is 
transmitted out-of-band with regard to the actual payload.

Is this referring to passing a `http://` path to 
HTMLDocument::createFromFile()? This would be unusable for everyone who 
manually downloads the document, e.g. using a PSR-18 HTTP Client.

It might actually be necessary to add an encoding parameter to these 
functions, but it would need to take priority over anything implicit. 
The current $encoding of the global \DOMDocument has the problem that it 
doesn't take priority/is ignored entirely. Manually converting the 
document to UTF-8 before passing it to \DOMDocument has the problem that 
the meta tag in the document takes priority.

In fact I've run into this issue before for the implementation of a rich 
embed feature. We're downloading the websites using Guzzle and attempt 
to make sense of them with \DOMDocument. However we can't reliably force 
the encoding given within the 'content-type' response header, so in some 
cases we obtain mojibake.

This encoding parameter would likely need to be `?string $encoding = 
null` with everything non-null overwriting implicit detection and null 
meaning implicit detection in the order of priorities you mentioned.

> 2. BOM sniffing in the content. I.e. UTF-8 with BOM and UTF-16 LE/BE prepend the content with byte markers. This is used to detect encoding.
> 3. Meta tag in the content.

Best regards
Tim Düsterhus