Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:92826
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain gmail.com designates 209.85.192.176 as permitted sender)
To: Sara Golemon <pollita@php.net>, Yasuo Ohgaki <yohgaki@ohgaki.net>
References: <CAESVnVo-BV7BOWV2H4ie1z48XofKnejArHm++zB9wGRPC4rB6w@mail.gmail.com>
 <CAGa2bXZSc=RkrBzwMg5K+MLfxc6XAL4ZSBRs7yjzkSLiPXeE2g@mail.gmail.com>
 <CAESVnVouYE9o9d+39mF+RAKqioTJ=cTQNvn48ziciDYq_apLHQ@mail.gmail.com>
Cc: PHP internals <internals@lists.php.net>
Message-ID: <571FF0DE.9060600@gmail.com>
Date: Tue, 26 Apr 2016 15:51:10 -0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:38.0)
 Gecko/20100101 Thunderbird/38.7.2
MIME-Version: 1.0
In-Reply-To: <CAESVnVouYE9o9d+39mF+RAKqioTJ=cTQNvn48ziciDYq_apLHQ@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Subject: Re: [PHP-DEV] [RFC] IntlCharsetDetector
From: smalyshev@gmail.com (Stanislav Malyshev)

Hi!

> For me, the difference is that I expect further work to be done on
> improving ICU, while I lack that confidence for mbstring.  If the API

My experience over the years has been that established supported
libraries like ICU usually have better track record in improving and
maintenance than more niche libraries, but it differs a lot from case to
case. I have no idea though how good/bad is ICU in detecting Asian
languages and encodings.

>> Developers should not rely on encoding detector, but they should validate
>> encoding.
>>
> I think everyone agrees on that. :)

True, but also incomplete. There's ideal case, and there's real world.
In ideal case, you know encodings of everything and everything is nicely
specified and shiny and rainbows and unicorns abound. Real data, though,
is messy and unpredictable and comes from places and practices that
makes one shudder. And when it comes to that we can either give the
developers at least something - an imperfect encoding detector, with all
caveats - or just ignore it and not give them anything, because it is
not matching our theories. and leave them to implement even worse hacks.
I think the former is much better approach.

And of course, detection and validation is a different thing. A text may
look like valid string in encoding A but actually be encoding B. "Tell
me if this data looks like Russian text in KOI-8 or Japanese text in
Shift-JIS" and "tell me if this is a valid or invalid UTF-8" are two
completely different tasks.
-- 
Stas Malyshev
smalyshev@gmail.com