adding GB18030 support for mbstring

15 years ago by KITAZAKI Shigeru — view source

unread

We made a patch to add a mbfilter for GB18030 encoding for PHP-5.3.1.
Please take a look at our blog:
http://developer.cybozu.co.jp/oss/2010/01/php-mbstring-pa.html

We would appreciate if you take this patch into the mainline.

BTW, our blog has various other patches for PHP in addition to this one.
Feel free to mail me if you are interested in some of them.

Regards,
KITAZAKI Shigeru <shigeru_kitazaki@cybozu.co.jp

15 years ago by Butrus Damaskus — view source

unread

Hm, I just wonder why yet another encoding. Shouldn't unicode be
enough for every one?

2010/1/28 KITAZAKI Shigeru shigeru_kitazaki@cybozu.co.jp:

We made a patch to add a mbfilter for GB18030 encoding for PHP-5.3.1.
Please take a look at our blog:
http://developer.cybozu.co.jp/oss/2010/01/php-mbstring-pa.html

We would appreciate if you take this patch into the mainline.

BTW, our blog has various other patches for PHP in addition to this one.
Feel free to mail me if you are interested in some of them.

Regards,
KITAZAKI Shigeru shigeru_kitazaki@cybozu.co.jp

--
PHP Unicode & I18N Mailing List (http://www.php.net/)

15 years ago by Christopher Jones — view source

unread

KITAZAKI Shigeru wrote:

BTW, our blog has various other patches for PHP in addition to this one.
Feel free to mail me if you are interested in some of them.

Regards,
KITAZAKI Shigeru shigeru_kitazaki@cybozu.co.jp

To highlight and record those patches, can you create a bug report for
each problem/patch and add the URL to your patch?

Some readers of the internals list would also suggest emailing an
short summary of the patch here.

Chris

--
Blog: http://blogs.oracle.com/opal
Twitter: http://twitter.com/ghrd

15 years ago by Moriyoshi Koizumi — view source

unread

Kitazaki-san,

First thank you for your effort. But, I am under the impression that
the conversion table looks too huge to include in a distribution
(>30MB). Is there any way to get this more compressed?

BTW, I created an extension that is near-compatible with mbstring and
based on ICU that of course supports GB18030. See
http://github.com/moriyoshi/mbstring-ng for detail.

Regards,
Moriyoshi

2010/1/28 KITAZAKI Shigeru shigeru_kitazaki@cybozu.co.jp:

We made a patch to add a mbfilter for GB18030 encoding for PHP-5.3.1.
Please take a look at our blog:
http://developer.cybozu.co.jp/oss/2010/01/php-mbstring-pa.html

We would appreciate if you take this patch into the mainline.

BTW, our blog has various other patches for PHP in addition to this one.
Feel free to mail me if you are interested in some of them.

Regards,
KITAZAKI Shigeru shigeru_kitazaki@cybozu.co.jp

--
PHP Unicode & I18N Mailing List (http://www.php.net/)

15 years ago by Tex Texin — view source

unread

Since ICU supports many conversions, including gb18030, and is regularly
updated and is already a part of php, it makes no sense to include
individually written conversions. ICU also gets considerable testing and
review.

Conversions should all be driven through icu.

-----Original Message-----
From: Moriyoshi Koizumi [mailto:mozo@mozo.jp]
Sent: Sunday, January 31, 2010 11:29 PM
To: KITAZAKI Shigeru
Cc: php-i18n@lists.php.net; internals@lists.php.net
Subject: Re: [PHP-I18N] adding GB18030 support for mbstring

Kitazaki-san,

First thank you for your effort. But, I am under the impression that
the conversion table looks too huge to include in a distribution
(>30MB). Is there any way to get this more compressed?

BTW, I created an extension that is near-compatible with mbstring and
based on ICU that of course supports GB18030. See
http://github.com/moriyoshi/mbstring-ng for detail.

Regards,
Moriyoshi

2010/1/28 KITAZAKI Shigeru shigeru_kitazaki@cybozu.co.jp:

We made a patch to add a mbfilter for GB18030 encoding for PHP-5.3.1.
Please take a look at our blog:
http://developer.cybozu.co.jp/oss/2010/01/php-mbstring-pa.html

We would appreciate if you take this patch into the mainline.

BTW, our blog has various other patches for PHP in addition to this one.
Feel free to mail me if you are interested in some of them.

Regards,
KITAZAKI Shigeru shigeru_kitazaki@cybozu.co.jp

--
PHP Unicode & I18N Mailing List (http://www.php.net/)

--
PHP Unicode & I18N Mailing List (http://www.php.net/)

15 years ago by Pierre Joye — view source

unread

hi,

Since ICU supports many conversions, including gb18030, and is regularly
updated and is already a part of php, it makes no sense to include
individually written conversions. ICU also gets considerable testing and
review.

Conversions should all be driven through icu.

In an ideal world yes. But in our not so ideal world mbstring is still
here, is still used in many places inside PHP and I think it makes
sense too add more encoding if there is a need for them.

However it is obvious that the mid/long term goal should be to replace
it completely with ICU.

Cheers,

Pierre

@pierrejoye | http://blog.thepimp.net | http://www.libgd.org

15 years ago by Tex Texin — view source

unread

mbstring can call out to icu to do the work.

-----Original Message-----
From: Pierre Joye [mailto:pierre.php@gmail.com]
Sent: Monday, February 01, 2010 12:11 PM
To: Tex Texin
Cc: Moriyoshi Koizumi; KITAZAKI Shigeru; php-i18n@lists.php.net;
internals@lists.php.net
Subject: Re: [PHP-DEV] RE: [PHP-I18N] adding GB18030 support for mbstring

hi,

Since ICU supports many conversions, including gb18030, and is regularly
updated and is already a part of php, it makes no sense to include
individually written conversions. ICU also gets considerable testing and
review.

Conversions should all be driven through icu.

In an ideal world yes. But in our not so ideal world mbstring is still
here, is still used in many places inside PHP and I think it makes
sense too add more encoding if there is a need for them.

However it is obvious that the mid/long term goal should be to replace
it completely with ICU.

Cheers,

Pierre

@pierrejoye | http://blog.thepimp.net | http://www.libgd.org

15 years ago by Pierre Joye — view source

unread

hi,

mbstring can call out to icu to do the work.

Right, pls read the thread, that's Moriyoshi is working on. But can we
do it in a minor release? I don't think so.

Cheers,

-----Original Message-----
From: Pierre Joye [mailto:pierre.php@gmail.com]
Sent: Monday, February 01, 2010 12:11 PM
To: Tex Texin
Cc: Moriyoshi Koizumi; KITAZAKI Shigeru; php-i18n@lists.php.net;
internals@lists.php.net
Subject: Re: [PHP-DEV] RE: [PHP-I18N] adding GB18030 support for mbstring

hi,

Since ICU supports many conversions, including gb18030, and is regularly
updated and is already a part of php, it makes no sense to include
individually written conversions. ICU also gets considerable testing and
review.

Conversions should all be driven through icu.

In an ideal world yes. But in our not so ideal world mbstring is still
here, is still used in many places inside PHP and I think it makes
sense too add more encoding if there is a need for them.

However it is obvious that the mid/long term goal should be to replace
it completely with ICU.

Cheers,

Pierre

@pierrejoye | http://blog.thepimp.net | http://www.libgd.org

--
Pierre

@pierrejoye | http://blog.thepimp.net | http://www.libgd.org

15 years ago by Pierre Joye — view source

unread

hi,

Thanks for the patches :)

2010/2/1 KITAZAKI Shigeru shigeru_kitazaki@cybozu.co.jp:

Thank you for your suggestion.
Let me introduce some patches which we published. They might be useful on
specific environments although they are not bugs.
If I had better to report below issues as bugs, please tell me.

php_syslog.patch
syslog() function cannot properly send UTF-8 strings to event log on
Windows. This patch changes the internal API. We, however, must set
UTF-8 on 'mbstring.internal_incoding'.
In addition, this changes the severity of 'LOG_ERR' from eventlog's
warning to eventlog's error.

It has to be optional. Maybe a mbstring.syslog* option to set a given
encoding for the syslog functions.

php_ldap.patch
This changes the internal API to use Windows native LDAP API instead of
OpenLDAP one. This is because OpenLDAP has dropped Windows support.

That's something I have done too but I don't think we can do it in a
minor version as it introduces some BC breaks. For example with SSL,
as far as I know it is not possible to use custom cerficates in custom
places (via openssl).

Where did you read that openldap stops to support windows? Last time I
checked I saw that they stoped (they never really did) to stupport
VC++ but they did support windows. Some openldap contributors even
provide windows builds as commercial packages.

Cheers,

Pierre

@pierrejoye | http://blog.thepimp.net | http://www.libgd.org

15 years ago by Moriyoshi Koizumi — view source

unread

2010/2/1 KITAZAKI Shigeru shigeru_kitazaki@cybozu.co.jp:

php_syslog.patch
syslog() function cannot properly send UTF-8 strings to event log on
Windows. This patch changes the internal API. We, however, must set
UTF-8 on 'mbstring.internal_incoding'.
In addition, this changes the severity of 'LOG_ERR' from eventlog's
warning to eventlog's error.

It seems this doesn't relies on any mbstring settings, but just
changed syslog() to take strings encoded in UTF-8 instead of the
system's default encoding. It'd look good to me if it had a new flag
causing syslog() to switch to the new behaviour.

Regards,
Moriyoshi

15 years ago by Pierre Joye — view source

unread

2010/2/2 KITAZAKI Shigeru shigeru_kitazaki@cybozu.co.jp:

hi,

Pierre Joye wrote:

php_syslog.patch
It has to be optional. Maybe a mbstring.syslog* option to set a given
encoding for the syslog functions.

Thank you for your advise. It sounds good to be optional.
Let me consider about it again.

php_ldap.patch
That's something I have done too but I don't think we can do it in a
minor version as it introduces some BC breaks. For example with SSL,
as far as I know it is not possible to use custom cerficates in custom
places (via openssl).

I see. If you remember your work, please give me the points such as
summary of URL or patches and the test suites.

Where did you read that openldap stops to support windows? Last time I
checked I saw that they stoped (they never really did) to stupport
VC++ but they did support windows. Some openldap contributors even
provide windows builds as commercial packages.

Ah, sorry to misguide you. You are right.
My concern is why we cannot build it with VC++ with official documents.
I want to use the same compiler and to keep distributing PHP binary as
small as possible. And, OpenLDAP had supported VC++ in several years ago
as much as I know.
But, this is not a issue of PHP but OpenLDAP, I guess.

By the way, I patched the openldap libraries to build them with
VC6/9/10. You can find the dev files here:

http://pecl2.php.net/downloads/php-windows-builds/php-libs/VC9/x86/

and the sources here:

http://pecl2.php.net/downloads/php-windows-builds/source/

It could be nice to port the 2.4.x libs instead of 2.3. That should
not be too hard :) Would you like to give me a hand here?

Cheers,

Pierre

@pierrejoye | http://blog.thepimp.net | http://www.libgd.org