Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:67543
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain gmail.com designates 209.85.217.170 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <61BC4F17-86D9-4CBD-B185-58A2D4AFAE5F@rouvenwessling.de>
References: <61BC4F17-86D9-4CBD-B185-58A2D4AFAE5F@rouvenwessling.de>
Date: Mon, 27 May 2013 08:40:17 +0200
Message-ID: <CAEZPtU6qB7HNvM9AnhwGVbFo3nTvfGM71B2_09aO714JwFtYcA@mail.gmail.com>
To: =?ISO-8859-1?Q?Rouven_We=DFling?= <me@rouvenwessling.de>
Cc: PHP internals <internals@lists.php.net>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Subject: Re: [PHP-DEV] Proposal for better UTF-8 handling
From: pierre.php@gmail.com (Pierre Joye)

hi!

On Fri, May 24, 2013 at 3:17 AM, Rouven We=DFling <me@rouvenwessling.de> wr=
ote:
> Hi Internals!
>
> First let me introduce myself, my name is Rouven We=DFling, I'm a student=
 at RWTH Aachen University and I'm one of the maintainers of the Joomla! Fr=
amework (n=E9e Platform). I've been following the internals list for a few =
months and started brushing of my C skills for the past couple of months so=
 I can start contributing.
>
> To me one of the most annoying things about working with PHP is the (lack=
 of) unicode support. In Joomla! we've been discussing switching from PHP U=
TF-8 to Patchwork UTF-8 for our needs of handling UTF-8. Both are libraries=
 abstracting the multibyte extension and supplementing it with a number of =
functions. They also provide userland replacements for when multibyte is no=
t available (Patchwork will also use iconv and intl if available). All of t=
his is a huge pain.
>
> To ease this situation I'd like to make a new start at better unicode sup=
port for PHP, this time focusing on UTF-8 as the dominant web encoding. As =
a first step I'd like to propose adding a set of functions for handling UTF=
-8 strings. This should keep applications from implementing these algorithm=
s in PHP (also many of these are quite a bit faster, see benchmark results =
below). Once the algorithms are in place I'd like to look into creating a c=
lass for unicode strings and eventually Python like unicode literals.
>
> Before I write an RFC I'd like to get some feedback what you think about =
adding the following functions to PHP 5.6 (possibly more to follow): utf8_i=
s_valid, utf8_strlen,  utf8_substr, utf8_strpos, utf8_strrpos, utf8_str_spl=
it, utf8_strrev, utf8_recover, utf8_chr, utf8_ord, string_is_ascii.
>
> Most of them (exceptions are utf8_chr, utf8_is_valid, utf8_recover and st=
ring_is_ascii) are currently written in a way that they emit a warning when=
 they encounter invalid UTF-8 and return with null. This should encourage a=
pplications to check their input with utf8_is_valid and either stop further=
 processing or to fall back to utf8_recover to get a valid string. This sho=
uld improve security since there are attack vectors when malformed sequence=
s get interpreted as another encoding.
>
> You can find the code I've written so far here: https://github.com/realit=
yking/pecl-utf8
> You can find benchmark results here: http://realityking.github.io/pecl-ut=
f8/results.html

Without judging your extension, I wonder if you have looked at the
intl extension, for the php core parts. There are also some exts to
deal with non ascii strings in pecl.

I always promoted intl usage as it handles UTF-8 or other very well
and for everything needed to fully support Unicode, their data is kept
updated and the APIs are very stable. It is also available since PHP
5.3 which makes it a very good choice to begin with.

Cheers,
--
Pierre

@pierrejoye | http://www.libgd.org