Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:92829
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain gmail.com designates 209.85.192.67 as permitted sender)
MIME-Version: 1.0
Sender: yohgaki@gmail.com
In-Reply-To: <CAESVnVouYE9o9d+39mF+RAKqioTJ=cTQNvn48ziciDYq_apLHQ@mail.gmail.com>
References: <CAESVnVo-BV7BOWV2H4ie1z48XofKnejArHm++zB9wGRPC4rB6w@mail.gmail.com>
 <CAGa2bXZSc=RkrBzwMg5K+MLfxc6XAL4ZSBRs7yjzkSLiPXeE2g@mail.gmail.com> <CAESVnVouYE9o9d+39mF+RAKqioTJ=cTQNvn48ziciDYq_apLHQ@mail.gmail.com>
Date: Wed, 27 Apr 2016 13:11:08 +0900
Message-ID: <CAGa2bXaD67Y-FixO6BUvVcXjBcKKbCGdaqoX16AUACwmArMRYQ@mail.gmail.com>
To: Sara Golemon <pollita@php.net>
Cc: PHP internals <internals@lists.php.net>
Content-Type: text/plain; charset=UTF-8
Subject: Re: [PHP-DEV] [RFC] IntlCharsetDetector
From: yohgaki@ohgaki.net (Yasuo Ohgaki)

Hi Sara,

On Wed, Apr 27, 2016 at 1:10 AM, Sara Golemon <pollita@php.net> wrote:
> On Tue, Apr 26, 2016 at 2:06 AM, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote:
>> Things might have been changed, but as you've mentioned encoding
>> detection is unstable and ICU is poor compared to mbstring's detection
>> at least for Japanese encodings.
>>
> For me, the difference is that I expect further work to be done on
> improving ICU, while I lack that confidence for mbstring.  If the API
> is in place early on, the library can improve underneath it to the
> point it becomes more trustworthy later, but still be usable on older
> versions of PHP (linked against newer libicu).
>
> Maybe, I dunno.  I lack the motivation to push this feature forward
> atm, merely because it's not trust-worthy now.

I can understand this.

>
>> Developers should not rely on encoding detector, but they should validate
>> encoding.
>>
> I think everyone agrees on that. :)
>
>> Problem is there are cases that developers cannot determine used encoding...
>> If we are going to have this API, it would be better to validate string with
>> detected encoding by default and disable encoding validation optionally.
>> There are cases that developers have to deal with broken string data
>> on occasion.
>>
> What do you have in mind?  Full-on pre-request input filtering?
> 'cause that's never worked right (we tried really hard to make PHP6 do
> that and it failed badly)

I'm not.

>
> Or do you mean something like wrapping the ucsdet API in a coercer
> function that only returned the original string if it detected at high
> confidence and then validated against that detection?  'cause
> honestly, that should also be left to the application IMO.

I don't have problem with this approach. Developers must be responsible
for this.

For normal web apps, developers must validate encoding if it is expected
one or not. Developers do not have guess encoding for most cases.

Developers may need to detect encoding for uploaded text files, for example.
If they use encoding detection, then they should validate text data by
detected encoding.

Experienced developers will detect encoding then validate text data with
detected encoding before saving uploaded text files. However, many developers
will detect encoding and assume text file char encoding is valid. This is the
reason why I suggest

 - detect encoding (This is done by using only the beginning of data usually)
 - then validate the text data by detected encoding
 - if validation is OK return encoding name, otherwise return error.

This would reduce chance of storing invalid text data in system.
It's not strictly required, but I think it is more developer friendly.
It's just a suggestion.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net