13 years ago by Yasuo Ohgaki — view source

unread

Hi all,

I noticed that "Zend Multibyte Support" won't be on with

./sapi/cli/php -d zend.multibyte=1
nor
zend.multibyte=on (in php.ini)

This happens both php-src and php-src-5.4.

According to php.ini-production from php-src:
; If enabled, scripts may be written in encodings that are incompatible with
; the scanner. CP936, Big5, CP949 and Shift_JIS are the examples of such
; encodings. To use this feature, mbstring extension must be enabled.
; Default: Off
;zend.multibyte = Off

I thought it became runtime option.
Is this a bug or am I missing something?

--
Yasuo Ohgaki
yohgaki@ohgaki.net

13 years ago by Rui Hirokawa — view source

unread

Hi,

It is almost same for me.
What kind of configure option you are using ?

For me (Ubuntu 11.10),
if I made PHP-5.4 with 'configure --enable-mbstring', it works fine.

But, if I made it with 'configure --enable-mbstring
--with-apxs2=/usr/bin/apxs2', a Shift_JIS encoded PHP script causes the
parser error.

For both cases, zend.multibyte and zend.script_encoding can be specified.

I believe that it was OK in Ubuntu 11.04 with the same option
(--with-axps2).

Rui

Yasuo Ohgaki wrote:

Hi all,

I noticed that "Zend Multibyte Support" won't be on with

./sapi/cli/php -d zend.multibyte=1
nor
zend.multibyte=on (in php.ini)

This happens both php-src and php-src-5.4.

According to php.ini-production from php-src:
; If enabled, scripts may be written in encodings that are incompatible with
; the scanner. CP936, Big5, CP949 and Shift_JIS are the examples of such
; encodings. To use this feature, mbstring extension must be enabled.
; Default: Off
;zend.multibyte = Off

I thought it became runtime option.
Is this a bug or am I missing something?

--
Yasuo Ohgaki
yohgaki@ohgaki.net

13 years ago by Dmitry Stogov — view source

unread

Hi Yasuo,

how did you see that "Zend Multibyte Support" weren't enabled?

$ sapi/cli/php -d zend.multibyte=1 -i | grep -i multibyte

Zend Multibyte Support => provided by mbstring
zend.multibyte => On => On ******
Multibyte Support => enabled
Multibyte string engine => libmbfl
Multibyte (japanese) regex support => enabled
Multibyte regex (oniguruma) backtrack check => On
Multibyte regex (oniguruma) version => 4.7.1

Thanks. Dmitry.

Hi all,

I noticed that "Zend Multibyte Support" won't be on with

./sapi/cli/php -d zend.multibyte=1
nor
zend.multibyte=on (in php.ini)

This happens both php-src and php-src-5.4.

According to php.ini-production from php-src:
; If enabled, scripts may be written in encodings that are incompatible with
; the scanner. CP936, Big5, CP949 and Shift_JIS are the examples of such
; encodings. To use this feature, mbstring extension must be enabled.
; Default: Off
;zend.multibyte = Off

I thought it became runtime option.
Is this a bug or am I missing something?

--
Yasuo Ohgaki
yohgaki@ohgaki.net

13 years ago by Yasuo Ohgaki — view source

unread

Hi Dimity & Rui,

I was trying to see if PHP 5.4 and trunk was also affected by this bug report.

--with-zend-multibyte and --enable-debug reports LEAK with run-test.php
https://bugs.php.net/bug.php?id=60194

So I configured as "./configure --enable-debug"

Sorry for being a lazy reader. I could turn on zend.multibyte with
"--enable-mbstring".
Now I see PHP 5.3 doesn't have dependency to mbstring, but PHP 5.4+ has.
I haven't read the last sentence.Thanks.

It looks like php-src-5.4 doesn't has leaks as PHP 5.3 does.
(I haven't completed the test, since it seems zend.multibyte is NOT working)

$ TEST_PHP_EXECUTABLE=./sapi/cli/php ./run-tests.php -m -c ./php.ini

php-src-5.4
Extra dirs :
VALGRIND : valgrind-3.6.1

TIME START 2011-11-03 16:36:57

PASS EXPECT [tests/run-test/test001.phpt]
PASS EXPECTF [tests/run-test/test002.phpt]
PASS EXPECTREGEX [tests/run-test/test003.phpt]
.....

However, I got the same parse error, as Rui mentioned, with SJIS source script.

sjis.php

<?php
echo '表';

(You cannot copy&paste, since this would be UTF-8. If you need
file, let me know. I'll directly mail you.)

表 is valid SJIS char code and has \ as second byte.
Since PHP complains with parse error, it seems zend.multibyte is not working on
PHP 5.4.

$ ./sapi/cli/php -i | grep multibyte
zend.multibyte => On => On
$ ./sapi/cli/php /usr/local/apache2.0/htdocs/sjis.php
PHP Parse error: syntax error, unexpected ''�';'
(T_ENCAPSED_AND_WHITESPACE) in /usr/local/apache2.0/htdocs/sjis.php on
line 2

It seems like working with zend.multibyte=Off.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

Hi Yasuo,

how did you see that "Zend Multibyte Support" weren't enabled?

$ sapi/cli/php -d zend.multibyte=1 -i | grep -i multibyte

Zend Multibyte Support => provided by mbstring
zend.multibyte => On => On ******
Multibyte Support => enabled
Multibyte string engine => libmbfl
Multibyte (japanese) regex support => enabled
Multibyte regex (oniguruma) backtrack check => On
Multibyte regex (oniguruma) version => 4.7.1

Thanks. Dmitry.

Hi all,

I noticed that "Zend Multibyte Support" won't be on with

./sapi/cli/php -d zend.multibyte=1
nor
zend.multibyte=on (in php.ini)

This happens both php-src and php-src-5.4.

According to php.ini-production from php-src:
; If enabled, scripts may be written in encodings that are incompatible
with
; the scanner. CP936, Big5, CP949 and Shift_JIS are the examples of such
; encodings. To use this feature, mbstring extension must be enabled.
; Default: Off
;zend.multibyte = Off

I thought it became runtime option.
Is this a bug or am I missing something?

--
Yasuo Ohgaki
yohgaki@ohgaki.net

13 years ago by Dmitry Stogov — view source

unread

I suppose PHP can't autodetect SJIS encoding and needs a hint

for 5.4

$ php -d zend.multibyte=1 -d zend.script_encoding=SJIS sjis.php

5.3 must be compiled with --enable-zend-multibyte, then

$ php -d mbstring.script_encoding=SJIS for 5.3

(I've just tested 5.4 but not 5.3. Just don't have 5.3 compiled with
--enable-zend-multibyte).

Thanks. Dmitry.

Hi Dimity& Rui,

I was trying to see if PHP 5.4 and trunk was also affected by this bug report.

--with-zend-multibyte and --enable-debug reports LEAK with run-test.php
https://bugs.php.net/bug.php?id=60194

So I configured as "./configure --enable-debug"

Sorry for being a lazy reader. I could turn on zend.multibyte with
"--enable-mbstring".
Now I see PHP 5.3 doesn't have dependency to mbstring, but PHP 5.4+ has.
I haven't read the last sentence.Thanks.

It looks like php-src-5.4 doesn't has leaks as PHP 5.3 does.
(I haven't completed the test, since it seems zend.multibyte is NOT working)

$ TEST_PHP_EXECUTABLE=./sapi/cli/php ./run-tests.php -m -c ./php.ini

=====================================================================
PHP : ./sapi/cli/php
PHP_SAPI : cli
PHP_VERSION : 5.4.0RC1-dev
ZEND_VERSION: 2.4.0
PHP_OS : Linux - Linux dev.inter.es-i.jp 2.6.35.14-2m.mo7.x86_64
#1 SMP Mon Sep 12 11:09:50 JST 2011 x86_64
INI actual : /home/yohgaki/ext/svn/oss/php.net/php-src-5.4/php.ini
More .INIs :
CWD : /home/yohgaki/ext/svn/oss/php.net/php-src-5.4
Extra dirs :
VALGRIND : valgrind-3.6.1

TIME START 2011-11-03 16:36:57

PASS EXPECT [tests/run-test/test001.phpt]
PASS EXPECTF [tests/run-test/test002.phpt]
PASS EXPECTREGEX [tests/run-test/test003.phpt]
.....

However, I got the same parse error, as Rui mentioned, with SJIS source script.

sjis.php

<?php
echo '表';

(You cannot copy&paste, since this would be UTF-8. If you need
file, let me know. I'll directly mail you.)

表 is valid SJIS char code and has \ as second byte.
Since PHP complains with parse error, it seems zend.multibyte is not working on
PHP 5.4.

$ ./sapi/cli/php -i | grep multibyte
zend.multibyte => On => On
$ ./sapi/cli/php /usr/local/apache2.0/htdocs/sjis.php
PHP Parse error: syntax error, unexpected ''�';'
(T_ENCAPSED_AND_WHITESPACE) in /usr/local/apache2.0/htdocs/sjis.php on
line 2

It seems like working with zend.multibyte=Off.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

Hi Yasuo,

how did you see that "Zend Multibyte Support" weren't enabled?

$ sapi/cli/php -d zend.multibyte=1 -i | grep -i multibyte

Zend Multibyte Support => provided by mbstring
zend.multibyte => On => On ******
Multibyte Support => enabled
Multibyte string engine => libmbfl
Multibyte (japanese) regex support => enabled
Multibyte regex (oniguruma) backtrack check => On
Multibyte regex (oniguruma) version => 4.7.1

Thanks. Dmitry.

Hi all,

I noticed that "Zend Multibyte Support" won't be on with

./sapi/cli/php -d zend.multibyte=1
nor
zend.multibyte=on (in php.ini)

This happens both php-src and php-src-5.4.

According to php.ini-production from php-src:
; If enabled, scripts may be written in encodings that are incompatible
with
; the scanner. CP936, Big5, CP949 and Shift_JIS are the examples of such
; encodings. To use this feature, mbstring extension must be enabled.
; Default: Off
;zend.multibyte = Off

I thought it became runtime option.
Is this a bug or am I missing something?

--
Yasuo Ohgaki
yohgaki@ohgaki.net

13 years ago by Yasuo Ohgaki — view source

unread

Hi Dimity,

Now it seems working as it supposed. Thanks.

$ ./sapi/cli/php -d zend.multibyte=1 -d zend.script_encoding=SJIS sjis.php?
(? is due to my terminal encoding. It sets to UTF-8)

It seems LEAK problem was gone with PHP 5.4, too.

[yohgaki@dev php-src-5.4]$ TEST_PHP_EXECUTABLE=./sapi/cli/php
./run-tests.php -m -c ./php.ini

=====================================================================
PHP : ./sapi/cli/php
`PHP_SAPI` : cli
`PHP_VERSION` : 5.4.0RC1-dev
ZEND_VERSION: 2.4.0
`PHP_OS` : Linux - Linux dev.inter.es-i.jp 2.6.35.14-2m.mo7.x86_64
#1 SMP Mon Sep 12 11:09:50 JST 2011 x86_64
INI actual : /home/yohgaki/ext/svn/oss/php.net/php-src-5.4/php.ini
More .INIs :
CWD : /home/yohgaki/ext/svn/oss/php.net/php-src-5.4
Extra dirs :
VALGRIND : valgrind-3.6.1

TIME START 2011-11-03 18:23:53

PASS EXPECT [tests/run-test/test001.phpt]
PASS EXPECTF [tests/run-test/test002.phpt]
PASS EXPECTREGEX [tests/run-test/test003.phpt]
PASS INI section allows '=' [tests/run-test/test004.phpt]
PASS Error message handling (without ZendOptimizer)
[tests/run-test/test005.phpt]
PASS Error messages are shown [tests/run-test/test006.phpt]
PASS dirname test [tests/run-test/test007.phpt]

--
Yasuo Ohgaki
yohgaki@ohgaki.net

php -d zend.multibyte=1 -d zend.script_encoding=SJIS sjis.php

13 years ago by Yasuo Ohgaki — view source

unread

Oops, I thought "?" was due to terminal encoding, but I double checked with
redirecting to a file.

$ ./sapi/cli/php -d zend.multibyte=1 -d zend.script_encoding=SJIS sjis.php > tt

It became "?" instead of "表"..
It seems something wrong.

Thanks for you time.

--
Yasuo Ohgaki
yohgaki@ohgaki.net

Hi Dimity,

Now it seems working as it supposed. Thanks.

$ ./sapi/cli/php -d zend.multibyte=1 -d zend.script_encoding=SJIS sjis.php?
(? is due to my terminal encoding. It sets to UTF-8)

It seems LEAK problem was gone with PHP 5.4, too.

[yohgaki@dev php-src-5.4]$ TEST_PHP_EXECUTABLE=./sapi/cli/php
./run-tests.php -m -c ./php.ini

=====================================================================
PHP : ./sapi/cli/php
PHP_SAPI : cli
PHP_VERSION : 5.4.0RC1-dev
ZEND_VERSION: 2.4.0
PHP_OS : Linux - Linux dev.inter.es-i.jp 2.6.35.14-2m.mo7.x86_64
#1 SMP Mon Sep 12 11:09:50 JST 2011 x86_64
INI actual : /home/yohgaki/ext/svn/oss/php.net/php-src-5.4/php.ini
More .INIs :
CWD : /home/yohgaki/ext/svn/oss/php.net/php-src-5.4
Extra dirs :
VALGRIND : valgrind-3.6.1

TIME START 2011-11-03 18:23:53

PASS EXPECT [tests/run-test/test001.phpt]
PASS EXPECTF [tests/run-test/test002.phpt]
PASS EXPECTREGEX [tests/run-test/test003.phpt]
PASS INI section allows '=' [tests/run-test/test004.phpt]
PASS Error message handling (without ZendOptimizer)
[tests/run-test/test005.phpt]
PASS Error messages are shown [tests/run-test/test006.phpt]
PASS dirname test [tests/run-test/test007.phpt]

--
Yasuo Ohgaki
yohgaki@ohgaki.net

php -d zend.multibyte=1 -d zend.script_encoding=SJIS sjis.php

13 years ago by Gustavo Lopes — view source

unread

Em Thu, 03 Nov 2011 09:38:11 -0000, Yasuo Ohgaki yohgaki@ohgaki.net
escreveu:

Oops, I thought "?" was due to terminal encoding, but I double checked
with
redirecting to a file.

$ ./sapi/cli/php -d zend.multibyte=1 -d zend.script_encoding=SJIS
sjis.php > tt

It became "?" instead of "表"..
It seems something wrong.

Thanks for you time.

You also have to say what it's converted to (I think, but I'm not sure,
the default is ISO-8859-1). This works:

$echo 表 | uconv -f utf-8 -t sjis > sjis.php
$./php -d zend.multibyte=1 -d zend.script_encoding=SJIS -d
mbstring.internal_encoding=utf-8 sjis.php
表

--
Gustavo Lopes

13 years ago by Yasuo Ohgaki — view source

unread

Hi Gutavo,

Now I see what is going on. I thought string is treated as just a
bunch of data once
is was compiled and passed through.

$ ./sapi/cli/php -d zend.multibyte=1 -d zend.script_encoding=SJIS -d
mbstring.internal_encoding=utf-8 sjis.php 表

Thanks for your time.

May be I should document this, since there may be a confusion.

--
Yasuo Ohgaki
yohgaki@ohgaki.net

Em Thu, 03 Nov 2011 09:38:11 -0000, Yasuo Ohgaki yohgaki@ohgaki.net
escreveu:

Oops, I thought "?" was due to terminal encoding, but I double checked
with
redirecting to a file.

$ ./sapi/cli/php -d zend.multibyte=1 -d zend.script_encoding=SJIS sjis.php

tt

It became "?" instead of "表"..
It seems something wrong.

Thanks for you time.

You also have to say what it's converted to (I think, but I'm not sure,
the default is ISO-8859-1). This works:

$echo 表 | uconv -f utf-8 -t sjis > sjis.php
$./php -d zend.multibyte=1 -d zend.script_encoding=SJIS -d
mbstring.internal_encoding=utf-8 sjis.php
表

--
Gustavo Lopes

13 years ago by Yasuo Ohgaki — view source

unread

One last quick question.
Zend/tests/multibyte/multibyte_encoding_001.phpt sets
mbstring.internal_encoding=SJIS.

Does PHP 5.4+ suppose to work with SJIS(or other similar encoding)
internal_encoding?

--
Yasuo Ohgaki
yohgaki@ohgaki.net

13 years ago by Gustavo Lopes — view source

unread

Em Thu, 03 Nov 2011 10:31:47 -0000, Yasuo Ohgaki yohgaki@ohgaki.net
escreveu:

One last quick question.
Zend/tests/multibyte/multibyte_encoding_001.phpt sets
mbstring.internal_encoding=SJIS.

Does PHP 5.4+ suppose to work with SJIS(or other similar encoding)
internal_encoding?

No. What matters is that the parser generated by bison is able to
recognize the tokens. In an ASCII (as opposed to EBCDIC) machine, this
means the encoding must be ASCII compatible.

This is the table for SJIS:
http://icu-project.org/icu-bin/convexp?conv=ibm-943_P15A-2003&s=ALL

It would appear that it was ASCII compatible – \x20-\x7E represent
U+0020-U+007E, but if you take a closer look you'll see that these bytes
can also appear as part of larger sequences.

For instance, in this script:

<?php
function a漾() {}

the character 漾 is represented with \xE0\x40, where \x40 represents @ in
ASCII, so this would give an error, the same this would give an error:

<?php
function aà@() {}

would give an error. In fact, If I save the first script as UTF-8 and then
run PHP:

$ ./php -d zend.multibyte=1 -d zend.script_encoding=UTF-8 -d
mbstring.internal_encoding=SJIS sjis.php
php: Zend/zend_language_scanner.l:126: encoding_filter_script_to_internal:
Assertion `internal_encoding &&
zend_multibyte_check_lexer_compatibility(internal_encoding)' failed.
Aborted

it gives an assertion error.

--
Gustavo Lopes

13 years ago by Yasuo Ohgaki — view source

unread

Hi Gustavo,

Thanks for reply.

As long as bison didn't understand multibyte chars, parser would not
work well with them.
Your reply is exactly what I expected.

Thank you for clarification.

--
Yasuo Ohgaki
yohgaki@ohgaki.net

Em Thu, 03 Nov 2011 10:31:47 -0000, Yasuo Ohgaki yohgaki@ohgaki.net
escreveu:

One last quick question.
Zend/tests/multibyte/multibyte_encoding_001.phpt sets
mbstring.internal_encoding=SJIS.

Does PHP 5.4+ suppose to work with SJIS(or other similar encoding)
internal_encoding?

No. What matters is that the parser generated by bison is able to recognize
the tokens. In an ASCII (as opposed to EBCDIC) machine, this means the
encoding must be ASCII compatible.

This is the table for SJIS:
http://icu-project.org/icu-bin/convexp?conv=ibm-943_P15A-2003&s=ALL

It would appear that it was ASCII compatible – \x20-\x7E represent
U+0020-U+007E, but if you take a closer look you'll see that these bytes can
also appear as part of larger sequences.

For instance, in this script:

<?php
function a漾() {}

the character 漾 is represented with \xE0\x40, where \x40 represents @ in
ASCII, so this would give an error, the same this would give an error:

<?php
function aà@() {}

would give an error. In fact, If I save the first script as UTF-8 and then
run PHP:

$ ./php -d zend.multibyte=1 -d zend.script_encoding=UTF-8 -d
mbstring.internal_encoding=SJIS sjis.php
php: Zend/zend_language_scanner.l:126: encoding_filter_script_to_internal:
Assertion `internal_encoding &&
zend_multibyte_check_lexer_compatibility(internal_encoding)' failed.
Aborted

it gives an assertion error.

--
Gustavo Lopes

13 years ago by Dmitry Stogov — view source

unread

$ sapi/cli/php -d zend.multibyte=1 -d zend.script_encodinSJIS -d
mbstring.internal_encoding=UTF8 -d mbstring.output_encoding=UTF8 sjis.php

表

Too many different encodings :)

Thanks. Dmitry.

Oops, I thought "?" was due to terminal encoding, but I double checked with
redirecting to a file.

$ ./sapi/cli/php -d zend.multibyte=1 -d zend.script_encoding=SJIS sjis.php> tt

It became "?" instead of "表"..
It seems something wrong.

Thanks for you time.

--
Yasuo Ohgaki
yohgaki@ohgaki.net

Hi Dimity,

Now it seems working as it supposed. Thanks.

$ ./sapi/cli/php -d zend.multibyte=1 -d zend.script_encoding=SJIS sjis.php?
(? is due to my terminal encoding. It sets to UTF-8)

It seems LEAK problem was gone with PHP 5.4, too.

[yohgaki@dev php-src-5.4]$ TEST_PHP_EXECUTABLE=./sapi/cli/php
./run-tests.php -m -c ./php.ini

=====================================================================
PHP : ./sapi/cli/php
PHP_SAPI : cli
PHP_VERSION : 5.4.0RC1-dev
ZEND_VERSION: 2.4.0
PHP_OS : Linux - Linux dev.inter.es-i.jp 2.6.35.14-2m.mo7.x86_64
#1 SMP Mon Sep 12 11:09:50 JST 2011 x86_64
INI actual : /home/yohgaki/ext/svn/oss/php.net/php-src-5.4/php.ini
More .INIs :
CWD : /home/yohgaki/ext/svn/oss/php.net/php-src-5.4
Extra dirs :
VALGRIND : valgrind-3.6.1

TIME START 2011-11-03 18:23:53

PASS EXPECT [tests/run-test/test001.phpt]
PASS EXPECTF [tests/run-test/test002.phpt]
PASS EXPECTREGEX [tests/run-test/test003.phpt]
PASS INI section allows '=' [tests/run-test/test004.phpt]
PASS Error message handling (without ZendOptimizer)
[tests/run-test/test005.phpt]
PASS Error messages are shown [tests/run-test/test006.phpt]
PASS dirname test [tests/run-test/test007.phpt]

--
Yasuo Ohgaki
yohgaki@ohgaki.net

php -d zend.multibyte=1 -d zend.script_encoding=SJIS sjis.php

Zend Multibyte support

TIME START 2011-11-03 16:36:57

sjis.php

<?php echo '表';

TIME START 2011-11-03 16:36:57

sjis.php

<?php echo '表';

TIME START 2011-11-03 18:23:53

TIME START 2011-11-03 18:23:53

TIME START 2011-11-03 18:23:53

<?php
echo '表';

<?php
echo '表';