Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:67551
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain gmail.com designates 209.85.220.178 as permitted sender)
MIME-Version: 1.0
Sender: nicolas.grekas@gmail.com
In-Reply-To: <CAEZPtU6qB7HNvM9AnhwGVbFo3nTvfGM71B2_09aO714JwFtYcA@mail.gmail.com>
References: <61BC4F17-86D9-4CBD-B185-58A2D4AFAE5F@rouvenwessling.de> <CAEZPtU6qB7HNvM9AnhwGVbFo3nTvfGM71B2_09aO714JwFtYcA@mail.gmail.com>
Date: Mon, 27 May 2013 10:33:51 +0200
Message-ID: <CAOWwgp=cveQVNwpXtdiODBmnLHD1NTFr1BYsfE-g_ktSc356-w@mail.gmail.com>
To: Pierre Joye <pierre.php@gmail.com>
Cc: =?ISO-8859-1?Q?Rouven_We=DFling?= <me@rouvenwessling.de>, 
	PHP internals <internals@lists.php.net>
Content-Type: multipart/alternative; boundary=089e012948f8680d8504ddaf03a2
Subject: Re: [PHP-DEV] Proposal for better UTF-8 handling
From: nicolas.grekas+php@gmail.com (Nicolas Grekas)

--089e012948f8680d8504ddaf03a2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Btw, I hit a bug on grapheme_substr() that got no attention:
https://bugs.php.net/bug.php?id=3D62759

There is also https://bugs.php.net/bug.php?id=3D61860 that waits for a fix.

Nicolas



On Mon, May 27, 2013 at 8:40 AM, Pierre Joye <pierre.php@gmail.com> wrote:

> hi!
>
> On Fri, May 24, 2013 at 3:17 AM, Rouven We=DFling <me@rouvenwessling.de>
> wrote:
> > Hi Internals!
> >
> > First let me introduce myself, my name is Rouven We=DFling, I'm a stude=
nt
> at RWTH Aachen University and I'm one of the maintainers of the Joomla!
> Framework (n=E9e Platform). I've been following the internals list for a =
few
> months and started brushing of my C skills for the past couple of months =
so
> I can start contributing.
> >
> > To me one of the most annoying things about working with PHP is the
> (lack of) unicode support. In Joomla! we've been discussing switching fro=
m
> PHP UTF-8 to Patchwork UTF-8 for our needs of handling UTF-8. Both are
> libraries abstracting the multibyte extension and supplementing it with a
> number of functions. They also provide userland replacements for when
> multibyte is not available (Patchwork will also use iconv and intl if
> available). All of this is a huge pain.
> >
> > To ease this situation I'd like to make a new start at better unicode
> support for PHP, this time focusing on UTF-8 as the dominant web encoding=
.
> As a first step I'd like to propose adding a set of functions for handlin=
g
> UTF-8 strings. This should keep applications from implementing these
> algorithms in PHP (also many of these are quite a bit faster, see benchma=
rk
> results below). Once the algorithms are in place I'd like to look into
> creating a class for unicode strings and eventually Python like unicode
> literals.
> >
> > Before I write an RFC I'd like to get some feedback what you think abou=
t
> adding the following functions to PHP 5.6 (possibly more to follow):
> utf8_is_valid, utf8_strlen,  utf8_substr, utf8_strpos, utf8_strrpos,
> utf8_str_split, utf8_strrev, utf8_recover, utf8_chr, utf8_ord,
> string_is_ascii.
> >
> > Most of them (exceptions are utf8_chr, utf8_is_valid, utf8_recover and
> string_is_ascii) are currently written in a way that they emit a warning
> when they encounter invalid UTF-8 and return with null. This should
> encourage applications to check their input with utf8_is_valid and either
> stop further processing or to fall back to utf8_recover to get a valid
> string. This should improve security since there are attack vectors when
> malformed sequences get interpreted as another encoding.
> >
> > You can find the code I've written so far here:
> https://github.com/realityking/pecl-utf8
> > You can find benchmark results here:
> http://realityking.github.io/pecl-utf8/results.html
>
> Without judging your extension, I wonder if you have looked at the
> intl extension, for the php core parts. There are also some exts to
> deal with non ascii strings in pecl.
>
> I always promoted intl usage as it handles UTF-8 or other very well
> and for everything needed to fully support Unicode, their data is kept
> updated and the APIs are very stable. It is also available since PHP
> 5.3 which makes it a very good choice to begin with.
>
> Cheers,
> --
> Pierre
>
> @pierrejoye | http://www.libgd.org
>
> --
> PHP Internals - PHP Runtime Development Mailing List
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>

--089e012948f8680d8504ddaf03a2--