Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:71283
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain gmail.com designates 209.85.215.46 as permitted sender)
MIME-Version: 1.0
Sender: yohgaki@gmail.com
In-Reply-To: <CAEZPtU5roOCqdJPnDbPU4yU85HrnFGYXjpSBhPQ+BmYHQ7qh4Q@mail.gmail.com>
References: <CAGa2bXauxj6Sca8qYk2cxQ=TeG4bjWgcv-Aqy3cJe35ap72Yfw@mail.gmail.com>
 <CAF+90c-ZgsCczU71TJap1yeaNhKgQeoUR5oLKwwHHaoOD8z_ig@mail.gmail.com>
 <CAGa2bXb6jLX8HMN-rL7t5j5R_0SaeRkd5r4_MrmQFfsJxxaSxQ@mail.gmail.com>
 <CAF+90c-JBP5iY_DmkSq7Ao6w4rkWoFp9xdwJ9frtQGdxyLyztQ@mail.gmail.com>
 <CAGa2bXb56tka6Ac6TkwxXfjNLUAVbuj3fDQa1SZLE6j8VwOFLw@mail.gmail.com> <CAEZPtU5roOCqdJPnDbPU4yU85HrnFGYXjpSBhPQ+BmYHQ7qh4Q@mail.gmail.com>
Date: Mon, 20 Jan 2014 07:48:08 +0900
Message-ID: <CAGa2bXYvCkA2UxFjQUzxMgSNKgDvGoU_HSVR1HedJttgpoN0-g@mail.gmail.com>
To: Pierre Joye <pierre.php@gmail.com>
Cc: Nikita Popov <nikita.ppv@gmail.com>, 
	"internals@lists.php.net" <internals@lists.php.net>
Content-Type: multipart/alternative; boundary=001a11c233ec147ac804f05a94ce
Subject: Re: [PHP-DEV] [RFC] Multibyte char handling
From: yohgaki@ohgaki.net (Yasuo Ohgaki)

--001a11c233ec147ac804f05a94ce
Content-Type: text/plain; charset=UTF-8

Hi Pierre,

On Mon, Jan 20, 2014 at 1:09 AM, Pierre Joye <pierre.php@gmail.com> wrote:

> On Thu, Jan 16, 2014 at 11:47 PM, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote:
> > Hi Nikita,
> >
> > On Fri, Jan 17, 2014 at 7:38 AM, Nikita Popov <nikita.ppv@gmail.com>
> wrote:
> >
> >> No, I don't want a locale-based approach. I want the string functions to
> >> stay as is. Multibyte variants of the functions can be added to the
> >> multibyte extension.
> >
> >
> > Creating mb_*() function would not solve security issues of
> > multibyte char handling since multibyte aware functions are
> > optional feature.
>
> We never supported nor claimed that these functions are multi bytes
> safe. However I actually fully understand that we should solve this
> problem, in one way or another.
>
> > However, it may work if PHP compiles mbstring by default and
> > discourage use of addslashes()/var_export()/stripslashes()
> > in favor of mb_*() variants.
>
> I do not think we should discourage the use of these functions but
> clearly document to rely on mb_* APIs as long as multi bytes support
> is required.
>
> I join other about not making any optional arguments in the existing
> APIs, for a couple of reasons:
>
> 1. it does not solve anything as people still have to update their
> code, and they won't unless maybe if they read the doc/changelog
> 2. It is really not a clean solution
> 3. we already have many duplicate functions in mb, it has worked well
> so far and we can add the ones discussed here
>

I'll leave existing ext/standard functions alone.

The last question was about relying on locale. This is absolutely not
> a solution. Locale has been proven to be totally unreliable, buggy and
> unsafe. Let alone the total lack of real posix locale support on
> Windows.
>

mb_escape_shell_arg()/mb_escape_shell_cmd() need locale based
solution, since there aren't good way to detect terminal encoding. I'll
make mb version explicitly overrides this behavior by explicitly specifying
encoding.

On UNIXes, UTF-8 encoding is popular terminal encoding, but there
would be systems using other encoding such as EUC, or even SJIS, BIG5.
Windows uses different encoding for terminal encoding according to locale,
so it's much more complex.

This is the reason why I would use locale. However, this implementation
is debatable.

We could say "Users should explicitly specify terminal encoding
by themselves". In fact, I prefer this even if I am about to implement
mb_escape_shell_*() using locale for automatic encoding detection.

It may be better to raise E_NOTICE at least if encoding parameter is
omitted for mb_escape_shell_*().

For anything related to locale, formats or encoding, we should rely on
> intl (ICU) and not on systems's locale. This is the only way to be
> portable, safe and updated.


I agree.
I also would like to propose

https://wiki.php.net/rfc/altmbstring   - ICU version of mbstring

for future release. Most work has done by Moriyoshi. We may try to
switch to it now, but I suppose there is not enough time for 5.6.

It's supposed to work the same as current mbstring mostly. It may be
better mbstring compile as optional in favor of ICU implementation.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

--001a11c233ec147ac804f05a94ce--