# adding GB18030 support for mbstring - Thread: https://externals.io/message/46896 - Original archive: https://news-web.php.net/php.internals/46896 - Started by KITAZAKI Shigeru on 2010-01-28T09:41:13+00:00 - Messages: 11 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #46896 — KITAZAKI Shigeru — 2010-01-28T09:41:13+00:00 URL: https://externals.io/message/46896#46896 We made a patch to add a mbfilter for GB18030 encoding for PHP-5.3.1. Please take a look at our blog: http://developer.cybozu.co.jp/oss/2010/01/php-mbstring-pa.html We would appreciate if you take this patch into the mainline. BTW, our blog has various other patches for PHP in addition to this one. Feel free to mail me if you are interested in some of them. Regards, KITAZAKI Shigeru : > We made a patch to add a mbfilter for GB18030 encoding for PHP-5.3.1. > Please take a look at our blog: >  http://developer.cybozu.co.jp/oss/2010/01/php-mbstring-pa.html > > We would appreciate if you take this patch into the mainline. > > BTW, our blog has various other patches for PHP in addition to this one. > Feel free to mail me if you are interested in some of them. > > Regards, > KITAZAKI Shigeru > > -- > PHP Unicode & I18N Mailing List (http://www.php.net/) --- ## #46899 — Christopher Jones — 2010-01-28T16:29:54+00:00 URL: https://externals.io/message/46896#46899 · In reply to: #46896 KITAZAKI Shigeru wrote: > BTW, our blog has various other patches for PHP in addition to this one. > Feel free to mail me if you are interested in some of them. > > Regards, > KITAZAKI Shigeru > To highlight and record those patches, can you create a bug report for each problem/patch and add the URL to your patch? Some readers of the internals list would also suggest emailing an short summary of the patch here. Chris -- Blog: http://blogs.oracle.com/opal Twitter: http://twitter.com/ghrd --- ## #46918 — Moriyoshi Koizumi — 2010-02-01T07:29:16+00:00 URL: https://externals.io/message/46896#46918 · In reply to: #46896 Kitazaki-san, First thank you for your effort. But, I am under the impression that the conversion table looks too huge to include in a distribution (>30MB). Is there any way to get this more compressed? BTW, I created an extension that is near-compatible with mbstring and based on ICU that of course supports GB18030. See http://github.com/moriyoshi/mbstring-ng for detail. Regards, Moriyoshi 2010/1/28 KITAZAKI Shigeru : > We made a patch to add a mbfilter for GB18030 encoding for PHP-5.3.1. > Please take a look at our blog: >  http://developer.cybozu.co.jp/oss/2010/01/php-mbstring-pa.html > > We would appreciate if you take this patch into the mainline. > > BTW, our blog has various other patches for PHP in addition to this one. > Feel free to mail me if you are interested in some of them. > > Regards, > KITAZAKI Shigeru > > -- > PHP Unicode & I18N Mailing List (http://www.php.net/) --- ## #46919 — Pierre Joye — 2010-02-01T08:52:03+00:00 URL: https://externals.io/message/46896#46919 hi, Thanks for the patches :) 2010/2/1 KITAZAKI Shigeru : > Thank you for your suggestion. > Let me introduce some patches which we published. They might be useful on > specific environments although they are not bugs. > If I had better to report below issues as bugs, please tell me. > > * php_syslog.patch >  syslog() function cannot properly send UTF-8 strings to event log on >  Windows. This patch changes the internal API. We, however, must set >  UTF-8 on 'mbstring.internal_incoding'. >  In addition, this changes the severity of 'LOG_ERR' from eventlog's >  warning to eventlog's error. It has to be optional. Maybe a mbstring.syslog* option to set a given encoding for the syslog functions. > * php_ldap.patch >  This changes the internal API to use Windows native LDAP API instead of >  OpenLDAP one. This is because OpenLDAP has dropped Windows support. That's something I have done too but I don't think we can do it in a minor version as it introduces some BC breaks. For example with SSL, as far as I know it is not possible to use custom cerficates in custom places (via openssl). Where did you read that openldap stops to support windows? Last time I checked I saw that they stoped (they never really did) to stupport VC++ but they did support windows. Some openldap contributors even provide windows builds as commercial packages. Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- ## #46923 — Moriyoshi Koizumi — 2010-02-01T10:32:53+00:00 URL: https://externals.io/message/46896#46923 2010/2/1 KITAZAKI Shigeru : > * php_syslog.patch >  syslog() function cannot properly send UTF-8 strings to event log on >  Windows. This patch changes the internal API. We, however, must set >  UTF-8 on 'mbstring.internal_incoding'. >  In addition, this changes the severity of 'LOG_ERR' from eventlog's >  warning to eventlog's error. It seems this doesn't relies on any mbstring settings, but just changed `syslog()` to take strings encoded in UTF-8 instead of the system's default encoding. It'd look good to me if it had a new flag causing `syslog()` to switch to the new behaviour. Regards, Moriyoshi --- ## #46925 — Tex Texin — 2010-02-01T19:59:01+00:00 URL: https://externals.io/message/46896#46925 · In reply to: #46918 Since ICU supports many conversions, including gb18030, and is regularly updated and is already a part of php, it makes no sense to include individually written conversions. ICU also gets considerable testing and review. Conversions should all be driven through icu. -----Original Message----- From: Moriyoshi Koizumi [mailto:mozo@mozo.jp] Sent: Sunday, January 31, 2010 11:29 PM To: KITAZAKI Shigeru Cc: php-i18n@lists.php.net; internals@lists.php.net Subject: Re: [PHP-I18N] adding GB18030 support for mbstring Kitazaki-san, First thank you for your effort. But, I am under the impression that the conversion table looks too huge to include in a distribution (>30MB). Is there any way to get this more compressed? BTW, I created an extension that is near-compatible with mbstring and based on ICU that of course supports GB18030. See http://github.com/moriyoshi/mbstring-ng for detail. Regards, Moriyoshi 2010/1/28 KITAZAKI Shigeru : > We made a patch to add a mbfilter for GB18030 encoding for PHP-5.3.1. > Please take a look at our blog: >  http://developer.cybozu.co.jp/oss/2010/01/php-mbstring-pa.html > > We would appreciate if you take this patch into the mainline. > > BTW, our blog has various other patches for PHP in addition to this one. > Feel free to mail me if you are interested in some of them. > > Regards, > KITAZAKI Shigeru > > -- > PHP Unicode & I18N Mailing List (http://www.php.net/) > > > -- PHP Unicode & I18N Mailing List (http://www.php.net/) --- ## #46926 — Pierre Joye — 2010-02-01T20:11:12+00:00 URL: https://externals.io/message/46896#46926 · In reply to: #46925 hi, > Since ICU supports many conversions, including gb18030, and is regularly > updated and is already a part of php, it makes no sense to include > individually written conversions. ICU also gets considerable testing and > review. > > Conversions should all be driven through icu. In an ideal world yes. But in our not so ideal world mbstring is still here, is still used in many places inside PHP and I think it makes sense too add more encoding if there is a need for them. However it is obvious that the mid/long term goal should be to replace it completely with ICU. Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- ## #46927 — Tex Texin — 2010-02-01T20:25:15+00:00 URL: https://externals.io/message/46896#46927 · In reply to: #46926 mbstring can call out to icu to do the work. -----Original Message----- From: Pierre Joye [mailto:pierre.php@gmail.com] Sent: Monday, February 01, 2010 12:11 PM To: Tex Texin Cc: Moriyoshi Koizumi; KITAZAKI Shigeru; php-i18n@lists.php.net; internals@lists.php.net Subject: Re: [PHP-DEV] RE: [PHP-I18N] adding GB18030 support for mbstring hi, > Since ICU supports many conversions, including gb18030, and is regularly > updated and is already a part of php, it makes no sense to include > individually written conversions. ICU also gets considerable testing and > review. > > Conversions should all be driven through icu. In an ideal world yes. But in our not so ideal world mbstring is still here, is still used in many places inside PHP and I think it makes sense too add more encoding if there is a need for them. However it is obvious that the mid/long term goal should be to replace it completely with ICU. Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- ## #46928 — Pierre Joye — 2010-02-01T20:32:48+00:00 URL: https://externals.io/message/46896#46928 · In reply to: #46927 hi, > mbstring can call out to icu to do the work. Right, pls read the thread, that's Moriyoshi is working on. But can we do it in a minor release? I don't think so. Cheers, > -----Original Message----- > From: Pierre Joye [mailto:pierre.php@gmail.com] > Sent: Monday, February 01, 2010 12:11 PM > To: Tex Texin > Cc: Moriyoshi Koizumi; KITAZAKI Shigeru; php-i18n@lists.php.net; > internals@lists.php.net > Subject: Re: [PHP-DEV] RE: [PHP-I18N] adding GB18030 support for mbstring > > hi, > > > > […] > > In an ideal world yes. But in our not so ideal world mbstring is still > here, is still used in many places inside PHP and I think it makes > sense too add more encoding if there is a need for them. > > However it is obvious that the mid/long term goal should be to replace > it completely with ICU. > > Cheers, > -- > Pierre > > @pierrejoye | http://blog.thepimp.net | http://www.libgd.org > > -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- ## #46930 — Pierre Joye — 2010-02-02T13:54:10+00:00 URL: https://externals.io/message/46896#46930 2010/2/2 KITAZAKI Shigeru : > hi, > > Pierre Joye wrote: > > […] > Thank you for your advise. It sounds good to be optional. > Let me consider about it again. > > > […] > I see. If you remember your work, please give me the points such as > summary of URL or patches and the test suites. > > > […] > Ah, sorry to misguide you. You are right. > My concern is why we cannot build it with VC++ with official documents. > I want to use the same compiler and to keep distributing PHP binary as > small as possible. And, OpenLDAP had supported VC++ in several years ago > as much as I know. > But, this is not a issue of PHP but OpenLDAP, I guess. By the way, I patched the openldap libraries to build them with VC6/9/10. You can find the dev files here: http://pecl2.php.net/downloads/php-windows-builds/php-libs/VC9/x86/ and the sources here: http://pecl2.php.net/downloads/php-windows-builds/source/ It could be nice to port the 2.4.x libs instead of 2.3. That should not be too hard :) Would you like to give me a hand here? Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org