Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:108898
To: internals@lists.php.net
References: <09dd1b84-ed33-a059-82f9-5efd179e69d6@gmx.de>
 <CA+kxMuSBsC1ZfPrOWdWhc92Q0Dx_1vF22f5Dv888jr0vo9usqg@mail.gmail.com>
Message-ID: <3952f4f7-a782-b392-50f7-27c2ef05fbb2@gmail.com>
Date: Sun, 8 Mar 2020 15:42:35 +0000
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101
 Thunderbird/68.5.0
MIME-Version: 1.0
In-Reply-To: <CA+kxMuSBsC1ZfPrOWdWhc92Q0Dx_1vF22f5Dv888jr0vo9usqg@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-GB
Subject: Re: [PHP-DEV] iconv vs. mbstring
From: rowan.collins@gmail.com (Rowan Tommins)

On 08/03/2020 14:08, Dan Ackroyd wrote:
> Related to this discussion, please could someone remind me why the
> mbstring extension is an extension and not part of core PHP?
>
> I realise at the time it was introduced, UTF-8 was far less widely
> used: https://en.wikipedia.org/wiki/UTF-8#/media/File:Utf8webgrowth.svg
>
> But now UTF-8 is pretty much the default for the vast majority of
> projects, so does that decision to keep it as an optional extension
> still hold up?


 From what I can make out, mbstring was not actually built for Unicode 
string-handling, but for what we would now consider "legacy encodings". 
Its original niche seems to have been support for various Japanese text 
encodings, and UTF-8 support was added relatively late.

That has some implications for its design:

- every function takes encoding as a parameter, and defaults to a 
run-time global setting
- on the other hand, there is no support for locales in functions which 
would benefit, e.g. mb_convert_case, mb_stripos
- Unicode is treated as just another character encoding, so there is no 
support for concepts like normalisation, graphemes, character 
properties, etc
- instead, there are lots of niche functions for CJK languages like 
mb_convert_kana and mb_strwidth

It also includes some things which probably wouldn't pass review if 
proposed today:

- a lot of global state, with combined get-or-set functions like 
mb_detect_order(), mb_substitute_character(), etc
- mb_send_mail seems oddly specific, and has its own concept of 
"language" not shared by anything else
- there's an entire regex implementation, with its own API and some 
compatibility with the removed ereg_* functions; I believe the preg_* 
functions included in core already support UTF-8


For handling of Unicode, ext/intl is generally superior, with a more 
structured API based on Unicode-specific concepts, rather than 
attempting to map them to concepts used in older character encodings. 
There may be a need for a more user-friendly subset of this (a "UString" 
class is a common suggestion), but it shouldn't look like ext/mbstring, 
IMHO.

I believe both extensions require fairly large external libraries, which 
probably justifies them being optional. From what I've read, ICU, which 
ext/intl is built on, would have been bundled with PHP 6, but its size 
and performance contributed to the failure of that project.


Regards,

-- 
Rowan Tommins (né Collins)
[IMSoP]