[Discussion] grapheme cluster for str_split function

1 year ago by youkidearitai — view source

unread

Hi, Internals

I noticed PHP does not have grapheme cluster for str_split function.,
Until now, you had to use the PCRE function's \X.

Therefore, I try create grapheme_str_split function.
https://github.com/php/php-src/pull/13580
It is possible to convert array per emoji and variation selectors using ICU.

If it's fine, I'll create an RFC.

Regards
Yuya

--

Yuya Hamada (tekimen)

1 year ago by Niels Dossche — view source

unread

Hi Yuya

This sounds useful.

I do have a question about the function signature:
function grapheme_str_split(string $string, int $length = 1): array {}

This always returns an array.
However, looking at your PR it seems you return NULL on failure, but the return type in the signature isn't nullable.
Also, from a quick look, it seems other functions return false instead of null on failure. So perhaps the return type should be array|false.

What do you think? :)

Kind regards
Niels

Hi, Internals

I noticed PHP does not have grapheme cluster for str_split function.,
Until now, you had to use the PCRE function's \X.

Therefore, I try create grapheme_str_split function.
https://github.com/php/php-src/pull/13580
It is possible to convert array per emoji and variation selectors using ICU.

If it's fine, I'll create an RFC.

Regards
Yuya

1 year ago by youkidearitai — view source

unread

2024年3月5日(火) 5:52 Niels Dossche dossche.niels@gmail.com:

Hi Yuya

This sounds useful.

I do have a question about the function signature:
function grapheme_str_split(string $string, int $length = 1): array {}

This always returns an array.
However, looking at your PR it seems you return NULL on failure, but the return type in the signature isn't nullable.
Also, from a quick look, it seems other functions return false instead of null on failure. So perhaps the return type should be array|false.

What do you think? :)

Kind regards
Niels

Hi, Internals

I noticed PHP does not have grapheme cluster for str_split function.,
Until now, you had to use the PCRE function's \X.

Therefore, I try create grapheme_str_split function.
https://github.com/php/php-src/pull/13580
It is possible to convert array per emoji and variation selectors using ICU.

If it's fine, I'll create an RFC.

Regards
Yuya

Hi, Niels

Thank you for your comment.
Indeed, returns false is make sense.

Therefore, I changed to returns false when invalid UTF-8 strings.

Regards
Yuya

--

Yuya Hamada (tekimen)

1 year ago by youkidearitai — view source

unread

Hi, Niels

Thank you for your comment.
Indeed, returns false is make sense.

Therefore, I changed to returns false when invalid UTF-8 strings.

Regards
Yuya

--

Yuya Hamada (tekimen)

https://tekitoh-memdhoi.info

https://github.com/youkidearitai

Sorry, again.
I checked behavior of mb_str_split function. So Illegal byte sequences
are returned as is.

sapi/cli/php -r 'var_dump(mb_str_split("あ\xc2\xf4\x80あ"));'
array(4) {
  [0]=>
  string(3) "あ"
  [1]=>
  string(2) "��"
  [2]=>
  string(1) "�"
  [3]=>
  string(3) "あ"
}

And, I reading ICU document about utext_openUTF8 (below is link):
https://unicode-org.github.io/icu-docs/apidoc/dev/icu4c/utext_8h.html#a130e7cba201c4b38799b432eb269f6d5

Any invalid UTF-8 in the input will be handled in this way: a sequence of bytes that has the form of a truncated, but otherwise valid, UTF-8 sequence will be replaced by a single unicode replacement character, \uFFFD. Any other illegal bytes will each be replaced by a \uFFFD.

Therefore, I think encoding check is not need.
Returns only arrays together with mb_str_split.

Regards
Yuya

--

Yuya Hamada (tekimen)

1 year ago by Larry Garfield — view source

unread

2024年3月5日(火) 5:52 Niels Dossche dossche.niels@gmail.com:

Hi Yuya

This sounds useful.

I do have a question about the function signature:
function grapheme_str_split(string $string, int $length = 1): array {}

This always returns an array.
However, looking at your PR it seems you return NULL on failure, but the return type in the signature isn't nullable.
Also, from a quick look, it seems other functions return false instead of null on failure. So perhaps the return type should be array|false.

What do you think? :)

Kind regards
Niels

Hi, Internals

I noticed PHP does not have grapheme cluster for str_split function.,
Until now, you had to use the PCRE function's \X.

Therefore, I try create grapheme_str_split function.
https://github.com/php/php-src/pull/13580
It is possible to convert array per emoji and variation selectors using ICU.

If it's fine, I'll create an RFC.

Regards
Yuya

Hi, Niels

Thank you for your comment.
Indeed, returns false is make sense.

Therefore, I changed to returns false when invalid UTF-8 strings.

Regards
Yuya

Many legacy functions return false on error, but that is widely regarded as bad design. Please do not continue bad design.

Right now, the best "standard" error handling mechanism available is exceptions. false (or null) can very easily lead to incorrectly using that value as though it were valid, when it's not, which will sometimes cause a fatal error and sometimes cause a security leak.

If the input value cannot be logically processed, that's an exception. (Or Error, perhaps.)

--Larry Garfield

1 year ago by Niels Dossche — view source

unread

Hi Larry
Hi Yuya

So first of all, I meant the error handling in cases like these: https://github.com/php/php-src/pull/13580/files#diff-b8fe038d9d7539694593978bea5605f38dde4bcb6a016865130590e45e23202eR852-R860
The implementation still returns NULL here, so the signature is still incorrect. Either it should return false to match the other functions, or throw something and not return a value.

2024年3月5日(火) 5:52 Niels Dossche dossche.niels@gmail.com:

Hi Yuya

This sounds useful.

I do have a question about the function signature:
function grapheme_str_split(string $string, int $length = 1): array {}

This always returns an array.
However, looking at your PR it seems you return NULL on failure, but the return type in the signature isn't nullable.
Also, from a quick look, it seems other functions return false instead of null on failure. So perhaps the return type should be array|false.

What do you think? :)

Kind regards
Niels

Hi, Internals

I noticed PHP does not have grapheme cluster for str_split function.,
Until now, you had to use the PCRE function's \X.

Therefore, I try create grapheme_str_split function.
https://github.com/php/php-src/pull/13580
It is possible to convert array per emoji and variation selectors using ICU.

If it's fine, I'll create an RFC.

Regards
Yuya

Hi, Niels

Thank you for your comment.
Indeed, returns false is make sense.

Therefore, I changed to returns false when invalid UTF-8 strings.

Regards
Yuya

Many legacy functions return false on error, but that is widely regarded as bad design. Please do not continue bad design.

I agree that returning false on error isn't ideal for exceptional cases, that's what exceptions are for.
Looking at the other grapheme functions makes me wonder though how consistent this would be, especially w.r.t. intl_get_error_*() and intl_error_name().

Right now, the best "standard" error handling mechanism available is exceptions. false (or null) can very easily lead to incorrectly using that value as though it were valid, when it's not, which will sometimes cause a fatal error and sometimes cause a security leak.

If the input value cannot be logically processed, that's an exception. (Or Error, perhaps.)

--Larry Garfield

Kind regards
Niels

1 year ago by youkidearitai — view source

unread

Hi, Larry
Hi, Niels

2024年3月6日(水) 6:47 Niels Dossche dossche.niels@gmail.com:

Hi Larry
Hi Yuya

So first of all, I meant the error handling in cases like these: https://github.com/php/php-src/pull/13580/files#diff-b8fe038d9d7539694593978bea5605f38dde4bcb6a016865130590e45e23202eR852-R860
The implementation still returns NULL here, so the signature is still incorrect. Either it should return false to match the other functions, or throw something and not return a value.

2024年3月5日(火) 5:52 Niels Dossche dossche.niels@gmail.com:

Hi Yuya

This sounds useful.

I do have a question about the function signature:
function grapheme_str_split(string $string, int $length = 1): array {}

This always returns an array.
However, looking at your PR it seems you return NULL on failure, but the return type in the signature isn't nullable.
Also, from a quick look, it seems other functions return false instead of null on failure. So perhaps the return type should be array|false.

What do you think? :)

Kind regards
Niels

Hi, Internals

I noticed PHP does not have grapheme cluster for str_split function.,
Until now, you had to use the PCRE function's \X.

Therefore, I try create grapheme_str_split function.
https://github.com/php/php-src/pull/13580
It is possible to convert array per emoji and variation selectors using ICU.

If it's fine, I'll create an RFC.

Regards
Yuya

Hi, Niels

Thank you for your comment.
Indeed, returns false is make sense.

Therefore, I changed to returns false when invalid UTF-8 strings.

Regards
Yuya

Many legacy functions return false on error, but that is widely regarded as bad design. Please do not continue bad design.

I agree that returning false on error isn't ideal for exceptional cases, that's what exceptions are for.
Looking at the other grapheme functions makes me wonder though how consistent this would be, especially w.r.t. intl_get_error_*() and intl_error_name().

Right now, the best "standard" error handling mechanism available is exceptions. false (or null) can very easily lead to incorrectly using that value as though it were valid, when it's not, which will sometimes cause a fatal error and sometimes cause a security leak.

If the input value cannot be logically processed, that's an exception. (Or Error, perhaps.)

--Larry Garfield

Kind regards
Niels

Thank you so much for advice.
Indeed, This current grapheme* functions seems inconsistent.

Therefore, it's one thing when returns null, throws any exception.
Shall we do so just for the grapheme_str_split function?

Regards
Yuya

--

Yuya Hamada (tekimen)

1 year ago by youkidearitai — view source

unread

2024年3月6日(水) 9:22 youkidearitai youkidearitai@gmail.com:

Hi, Larry
Hi, Niels

2024年3月6日(水) 6:47 Niels Dossche dossche.niels@gmail.com:

Hi Larry
Hi Yuya

So first of all, I meant the error handling in cases like these: https://github.com/php/php-src/pull/13580/files#diff-b8fe038d9d7539694593978bea5605f38dde4bcb6a016865130590e45e23202eR852-R860
The implementation still returns NULL here, so the signature is still incorrect. Either it should return false to match the other functions, or throw something and not return a value.

2024年3月5日(火) 5:52 Niels Dossche dossche.niels@gmail.com:

Hi Yuya

This sounds useful.

I do have a question about the function signature:
function grapheme_str_split(string $string, int $length = 1): array {}

This always returns an array.
However, looking at your PR it seems you return NULL on failure, but the return type in the signature isn't nullable.
Also, from a quick look, it seems other functions return false instead of null on failure. So perhaps the return type should be array|false.

What do you think? :)

Kind regards
Niels

Hi, Internals

I noticed PHP does not have grapheme cluster for str_split function.,
Until now, you had to use the PCRE function's \X.

Therefore, I try create grapheme_str_split function.
https://github.com/php/php-src/pull/13580
It is possible to convert array per emoji and variation selectors using ICU.

If it's fine, I'll create an RFC.

Regards
Yuya

Hi, Niels

Thank you for your comment.
Indeed, returns false is make sense.

Therefore, I changed to returns false when invalid UTF-8 strings.

Regards
Yuya

Many legacy functions return false on error, but that is widely regarded as bad design. Please do not continue bad design.

I agree that returning false on error isn't ideal for exceptional cases, that's what exceptions are for.
Looking at the other grapheme functions makes me wonder though how consistent this would be, especially w.r.t. intl_get_error_*() and intl_error_name().

Right now, the best "standard" error handling mechanism available is exceptions. false (or null) can very easily lead to incorrectly using that value as though it were valid, when it's not, which will sometimes cause a fatal error and sometimes cause a security leak.

If the input value cannot be logically processed, that's an exception. (Or Error, perhaps.)

--Larry Garfield

Kind regards
Niels

Thank you so much for advice.
Indeed, This current grapheme* functions seems inconsistent.

Therefore, it's one thing when returns null, throws any exception.
Shall we do so just for the grapheme_str_split function?

Regards
Yuya

--

Yuya Hamada (tekimen)

https://tekitoh-memdhoi.info

https://github.com/youkidearitai

Ah, If throws exception when intl_error*, is required other an RFC?
If we make grapheme_str_split's signature is below (include null):

function grapheme_str_split(string $string, int $length = 1): array|null {}

For now, I change signature to array|null.

Regards
Yuya

--

Yuya Hamada (tekimen)

1 year ago by Niels Dossche — view source

unread

2024年3月6日(水) 9:22 youkidearitai youkidearitai@gmail.com:

Hi, Larry
Hi, Niels

2024年3月6日(水) 6:47 Niels Dossche dossche.niels@gmail.com:

Hi Larry
Hi Yuya

So first of all, I meant the error handling in cases like these: https://github.com/php/php-src/pull/13580/files#diff-b8fe038d9d7539694593978bea5605f38dde4bcb6a016865130590e45e23202eR852-R860
The implementation still returns NULL here, so the signature is still incorrect. Either it should return false to match the other functions, or throw something and not return a value.

2024年3月5日(火) 5:52 Niels Dossche dossche.niels@gmail.com:

Hi Yuya

This sounds useful.

I do have a question about the function signature:
function grapheme_str_split(string $string, int $length = 1): array {}

This always returns an array.
However, looking at your PR it seems you return NULL on failure, but the return type in the signature isn't nullable.
Also, from a quick look, it seems other functions return false instead of null on failure. So perhaps the return type should be array|false.

What do you think? :)

Kind regards
Niels

Hi, Internals

I noticed PHP does not have grapheme cluster for str_split function.,
Until now, you had to use the PCRE function's \X.

Therefore, I try create grapheme_str_split function.
https://github.com/php/php-src/pull/13580
It is possible to convert array per emoji and variation selectors using ICU.

If it's fine, I'll create an RFC.

Regards
Yuya

Hi, Niels

Thank you for your comment.
Indeed, returns false is make sense.

Therefore, I changed to returns false when invalid UTF-8 strings.

Regards
Yuya

Many legacy functions return false on error, but that is widely regarded as bad design. Please do not continue bad design.

I agree that returning false on error isn't ideal for exceptional cases, that's what exceptions are for.
Looking at the other grapheme functions makes me wonder though how consistent this would be, especially w.r.t. intl_get_error_*() and intl_error_name().

Right now, the best "standard" error handling mechanism available is exceptions. false (or null) can very easily lead to incorrectly using that value as though it were valid, when it's not, which will sometimes cause a fatal error and sometimes cause a security leak.

If the input value cannot be logically processed, that's an exception. (Or Error, perhaps.)

--Larry Garfield

Kind regards
Niels

Thank you so much for advice.
Indeed, This current grapheme* functions seems inconsistent.

Therefore, it's one thing when returns null, throws any exception.
Shall we do so just for the grapheme_str_split function?

Regards
Yuya

--

Yuya Hamada (tekimen)

https://tekitoh-memdhoi.info

https://github.com/youkidearitai

Ah, If throws exception when intl_error*, is required other an RFC?
If we make grapheme_str_split's signature is below (include null):

Hi Yuya

If you want to change other grapheme functions with respect to error handling, then it requires a separate RFC.
I think consistency between the functions is important.

function grapheme_str_split(string $string, int $length = 1): array|null {}
For now, I change signature to array|null.

Most others functions return false on error, so I think it should be array|false, and the implementation should use RETURN_FALSE instead of RETURN_NULL.

Regards
Yuya

Kind regards
Niels

1 year ago by youkidearitai — view source

unread

2024年3月6日(水) 18:42 Niels Dossche dossche.niels@gmail.com:

2024年3月6日(水) 9:22 youkidearitai youkidearitai@gmail.com:

Hi, Larry
Hi, Niels

2024年3月6日(水) 6:47 Niels Dossche dossche.niels@gmail.com:

Hi Larry
Hi Yuya

So first of all, I meant the error handling in cases like these: https://github.com/php/php-src/pull/13580/files#diff-b8fe038d9d7539694593978bea5605f38dde4bcb6a016865130590e45e23202eR852-R860
The implementation still returns NULL here, so the signature is still incorrect. Either it should return false to match the other functions, or throw something and not return a value.

2024年3月5日(火) 5:52 Niels Dossche dossche.niels@gmail.com:

Hi Yuya

This sounds useful.

I do have a question about the function signature:
function grapheme_str_split(string $string, int $length = 1): array {}

This always returns an array.
However, looking at your PR it seems you return NULL on failure, but the return type in the signature isn't nullable.
Also, from a quick look, it seems other functions return false instead of null on failure. So perhaps the return type should be array|false.

What do you think? :)

Kind regards
Niels

Hi, Internals

I noticed PHP does not have grapheme cluster for str_split function.,
Until now, you had to use the PCRE function's \X.

Therefore, I try create grapheme_str_split function.
https://github.com/php/php-src/pull/13580
It is possible to convert array per emoji and variation selectors using ICU.

If it's fine, I'll create an RFC.

Regards
Yuya

Hi, Niels

Thank you for your comment.
Indeed, returns false is make sense.

Therefore, I changed to returns false when invalid UTF-8 strings.

Regards
Yuya

Many legacy functions return false on error, but that is widely regarded as bad design. Please do not continue bad design.

I agree that returning false on error isn't ideal for exceptional cases, that's what exceptions are for.
Looking at the other grapheme functions makes me wonder though how consistent this would be, especially w.r.t. intl_get_error_*() and intl_error_name().

Right now, the best "standard" error handling mechanism available is exceptions. false (or null) can very easily lead to incorrectly using that value as though it were valid, when it's not, which will sometimes cause a fatal error and sometimes cause a security leak.

If the input value cannot be logically processed, that's an exception. (Or Error, perhaps.)

--Larry Garfield

Kind regards
Niels

Thank you so much for advice.
Indeed, This current grapheme* functions seems inconsistent.

Therefore, it's one thing when returns null, throws any exception.
Shall we do so just for the grapheme_str_split function?

Regards
Yuya

--

Yuya Hamada (tekimen)

https://tekitoh-memdhoi.info

https://github.com/youkidearitai

Ah, If throws exception when intl_error*, is required other an RFC?
If we make grapheme_str_split's signature is below (include null):

Hi Yuya

If you want to change other grapheme functions with respect to error handling, then it requires a separate RFC.
I think consistency between the functions is important.
function grapheme_str_split(string $string, int $length = 1): array|null {}
For now, I change signature to array|null.
Most others functions return false on error, so I think it should be array|false, and the implementation should use RETURN_FALSE instead of RETURN_NULL.

Regards
Yuya

Kind regards
Niels

Hi, Niels
Thank you very much.

I got it. You're right. I couldn't understand well.
I use returns false when something wrong.

I updated Pull Request. Please see.

Regards
Yuya

--

Yuya Hamada (tekimen)