# iconv vs. mbstring - Thread: https://externals.io/message/108835 - Original archive: https://news-web.php.net/php.internals/108835 - Started by Christoph M. Becker on 2020-03-03T22:16:54+00:00 - Messages: 7 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #108835 — Christoph M. Becker — 2020-03-03T22:16:54+00:00 URL: https://externals.io/message/108835#108835 Hi all, we still have 2 bundled extensions for working with strings in different encodings: ext/mbstring and ext/iconv. While working on bug #79200[1], I've noticed that the implementation of many of the iconv_*() functions is rather suboptimal. This is mostly because `iconv()` is meant just for character encoding *conversion*, but ext/iconv puts several other useful string functions on top of that, but can't have these really optimized, because the extension doesn't really know anything about those character encodings. For instance, `iconv_strlen()` is basically implemented by converting the input string to UCS-4, and then simply counting the UCS-4 characters. On the other hand, `mb_strlen()` makes use of length tables (where appropriate), and as such does not even need to convert the string in many typical cases. Some quick benchmarks on getting the string length of UTF-8 strings show that `mb_strlen()` is roughly 10 times faster than `iconv_strlen()`. Now it would be trivially possible to improve the `iconv_strlen()` implementation by converting a larger number of characters in one go (instead of currently up to two only[2]), which would make the function much faster (roughly 3 to 4 times for a 1024 character buffer), but still `mb_strlen()` would obviously beat that. The situation for the other iconv_*() functions is similar, more or less. However, it seems that `iconv()` can be much faster than `mb_convert_encoding()`. Quick benchmarks show a factor of 2 to 3. So I wonder if we wouldn't be better off if we unbundle ext/iconv, but move the `iconv()` function (and possibly the convert.iconv.* stream filter) into ext/standard. It shouldn't be hard to update code which uses any of the iconv_*() functions to use respective mb_*() functions, and users who couldn't do this, or don't want to for whatever reason, could still use the iconv package available from PECL. However, users who would switch to mbstring would likely get better performance for their applications. For core developers that would obviously save time to maintain both extensions. For users learning PHP, and also for new code, it would be beneficial to not have to decide which of these extensions to use; if they need character encoding conversion, `iconv()` would be preferable; for more general string functionality, it would be ext/mbstring. Thoughts? [1] [2] -- Christoph M. Becker --- ## #108866 — Aleksander Machniak — 2020-03-05T11:59:45+00:00 URL: https://externals.io/message/108835#108866 · In reply to: #108835 > For users learning PHP, and also for new code, it would be beneficial to > not have to decide which of these extensions to use; if they need > character encoding conversion, `iconv()` would be preferable; for more > general string functionality, it would be ext/mbstring. From my experience iconv does not support all charsets e.g. UTF7-IMAP or ISO-2022-JP-MS, that mbstring does. Also, I have a case in which iconv_* functions were much much slower than mbstring. See wordwrap implementation in https://github.com/roundcube/roundcubemail/blob/master/program/lib/Roundcube/rcube_mime.php#L589 I did not do any performance comparison for `iconv()` function itself and I'm not sure it should be considered preferable. I saw a lot of performance improvements in mbstring in the last year or so. Do anyone have some perf. comparison for charset conversion cases? -- Aleksander Machniak Kolab Groupware Developer [https://kolab.org] Roundcube Webmail Developer [https://roundcube.net] ---------------------------------------------------- PGP: 19359DC1 # Blog: https://kolabian.wordpress.com --- ## #108870 — Rowan Tommins — 2020-03-05T14:33:28+00:00 URL: https://externals.io/message/108835#108870 · In reply to: #108866 > > Also, I have a case in which iconv_* functions were much much slower > than mbstring. See wordwrap implementation in > > https://github.com/roundcube/roundcubemail/blob/master/program/lib/Roundcube/rcube_mime.php#L589 > > Either I misread, or you did: I thought that's exactly what Christoph was saying, that iconv_* functions will be slower because they basically do convert-process-unconvert rather than having implementations for each encoding. So mb_strlen will always be faster than iconv_strlen. `iconv()` vs `mb_convert_encoding()` doesn't have the same penalty. It seems quite plausible that a library dedicated to converting charsets would be more optimised for that job than a single function in a larger library mainly focussed on working with one charset at a time. Regards, -- Rowan Tommins [IMSoP] --- ## #108872 — Christoph M. Becker — 2020-03-05T15:13:55+00:00 URL: https://externals.io/message/108835#108872 · In reply to: #108870 > > > > […] > > Either I misread, or you did: I thought that's exactly what Christoph was > saying, that iconv_* functions will be slower because they basically do > convert-process-unconvert rather than having implementations for each > encoding. So mb_strlen will always be faster than iconv_strlen. > > `iconv()` vs `mb_convert_encoding()` doesn't have the same penalty. It seems > quite plausible that a library dedicated to converting charsets would be > more optimised for that job than a single function in a larger library > mainly focussed on working with one charset at a time. I should add that all my testing was done solely with a current libiconv version (on Windows); I don't know how libc's ivonc() and other implementations might perform. -- Christoph M. Becker --- ## #108897 — Dan Ackroyd — 2020-03-08T14:08:04+00:00 URL: https://externals.io/message/108835#108897 · In reply to: #108835 > > for more > general string functionality, it would be ext/mbstring. > > Thoughts? > Related to this discussion, please could someone remind me why the mbstring extension is an extension and not part of core PHP? I realise at the time it was introduced, UTF-8 was far less widely used: https://en.wikipedia.org/wiki/UTF-8#/media/File:Utf8webgrowth.svg But now UTF-8 is pretty much the default for the vast majority of projects, so does that decision to keep it as an optional extension still hold up? cheers Dan Ack --- ## #108898 — Rowan Tommins — 2020-03-08T15:42:35+00:00 URL: https://externals.io/message/108835#108898 · In reply to: #108897 > Related to this discussion, please could someone remind me why the > mbstring extension is an extension and not part of core PHP? > > I realise at the time it was introduced, UTF-8 was far less widely > used: https://en.wikipedia.org/wiki/UTF-8#/media/File:Utf8webgrowth.svg > > But now UTF-8 is pretty much the default for the vast majority of > projects, so does that decision to keep it as an optional extension > still hold up? From what I can make out, mbstring was not actually built for Unicode string-handling, but for what we would now consider "legacy encodings". Its original niche seems to have been support for various Japanese text encodings, and UTF-8 support was added relatively late. That has some implications for its design: - every function takes encoding as a parameter, and defaults to a run-time global setting - on the other hand, there is no support for locales in functions which would benefit, e.g. mb_convert_case, mb_stripos - Unicode is treated as just another character encoding, so there is no support for concepts like normalisation, graphemes, character properties, etc - instead, there are lots of niche functions for CJK languages like mb_convert_kana and mb_strwidth It also includes some things which probably wouldn't pass review if proposed today: - a lot of global state, with combined get-or-set functions like `mb_detect_order()`, `mb_substitute_character()`, etc - mb_send_mail seems oddly specific, and has its own concept of "language" not shared by anything else - there's an entire regex implementation, with its own API and some compatibility with the removed ereg_* functions; I believe the preg_* functions included in core already support UTF-8 For handling of Unicode, ext/intl is generally superior, with a more structured API based on Unicode-specific concepts, rather than attempting to map them to concepts used in older character encodings. There may be a need for a more user-friendly subset of this (a "UString" class is a common suggestion), but it shouldn't look like ext/mbstring, IMHO. I believe both extensions require fairly large external libraries, which probably justifies them being optional. From what I've read, ICU, which ext/intl is built on, would have been bundled with PHP 6, but its size and performance contributed to the failure of that project. Regards, -- Rowan Tommins (né Collins) [IMSoP] --- ## #108900 — Tomas Kuliavas — 2020-03-08T17:40:14+00:00 URL: https://externals.io/message/108835#108900 · In reply to: #108897 2020.03.08 16:08 Dan Ackroyd rašė: > On Tue, 3 Mar 2020 at 22:17, Christoph M. Becker > wrote: > > […] > > Related to this discussion, please could someone remind me why the > mbstring extension is an extension and not part of core PHP? > > I realise at the time it was introduced, UTF-8 was far less widely > used: https://en.wikipedia.org/wiki/UTF-8#/media/File:Utf8webgrowth.svg > > But now UTF-8 is pretty much the default for the vast majority of > projects, so does that decision to keep it as an optional extension > still hold up? Majority is not 100%. Your German folders in webmails are not in UTF-8. Even if application operates in utf-8, it still must be able to work not with utf-8 and PHP functions that changed default from ascii/iso-8859-1 to utf-8 were not designed to operate in real world. Reason why mbstring are not base string functions is why PHP6 does not exist. -- Tomas