# [RFC] Multibyte char handling - Thread: https://externals.io/message/71162 - Original archive: https://news-web.php.net/php.internals/71162 - Started by Yasuo Ohgaki on 2014-01-15T23:50:18+00:00 - Messages: 31 - Page 1 of 2: messages 1 to 25 (25 per page, add `?page=N` to the URL) - Next page: https://externals.io/message/71162.md?page=2 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #71162 — Yasuo Ohgaki — 2014-01-15T23:50:18+00:00 URL: https://externals.io/message/71162#71162 Hi all, `addslashes()` could be vulnerable via char encoding based attacks. It is needed to decide what counter measure we adopt. This is RFC for this issue. https://wiki.php.net/multibyte_char_handling Please comment. Thank you. -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #71165 — Yasuo Ohgaki — 2014-01-16T00:12:44+00:00 URL: https://externals.io/message/71162#71165 · In reply to: #71162 Hi all, > `addslashes()` could be vulnerable via char encoding based attacks. > It is needed to decide what counter measure we adopt. > This is RFC for this issue. > > https://wiki.php.net/multibyte_char_handling > > Please comment. > Thank you. > I've copied line from "Array Of" RFC and URL was wrong. Correct URL is https://wiki.php.net/rfc/multibyte_char_handling Sorry for the confusion. Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #71168 — mails@thomasbley.de — 2014-01-16T02:36:16+00:00 URL: https://externals.io/message/71162#71168 · In reply to: #71165 Yasuo Ohgaki wrote on 16.01.2014 01:12: > Hi all, > > > > > […] > > I've copied line from "Array Of" RFC and URL was wrong. > Correct URL is > > https://wiki.php.net/rfc/multibyte_char_handling > > Sorry for the confusion. > > Regards, > > -- > Yasuo Ohgaki > yohgaki@ohgaki.net > Hello Yasuo, what about mb_trim? e.g. UTF-8: C2 A0, e2 80 82, e2 80 83, e2 80 af I currently have lots of untrimmed data in a database since php-trim() and mysql-trim() can't handle these characters. There are workarounds like trim($str, chr(0xC2).chr(0xA0)); but they are not really nice to code. Regards Thomas --- ## #71169 — Yasuo Ohgaki — 2014-01-16T02:58:05+00:00 URL: https://externals.io/message/71162#71169 · In reply to: #71168 > what about mb_trim? > e.g. UTF-8: C2 A0, e2 80 82, e2 80 83, e2 80 af > > I currently have lots of untrimmed data in a database since php-trim() and > mysql-trim() can't handle these characters. > There are workarounds like trim($str, chr(0xC2).chr(0xA0)); but they are > not really nice to code. > We need few more basic functions like `trim()`. This is different issue, but I may work on it. Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #71170 — Stas Malyshev — 2014-01-16T07:22:05+00:00 URL: https://externals.io/message/71162#71170 · In reply to: #71169 Hi! > We need few more basic functions like `trim()`. > This is different issue, but I may work on it. It feels like we will end up with creating clones of pretty every string function there is. Not sure it is the best approach... -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 --- ## #71177 — Yasuo Ohgaki — 2014-01-16T10:36:14+00:00 URL: https://externals.io/message/71162#71177 · In reply to: #71170 Hi Stas, On Thu, Jan 16, 2014 at 4:22 PM, Stas Malyshev wrote: > > […] > > It feels like we will end up with creating clones of pretty every string > function there is. Not sure it is the best approach... We could learn from Python 2.x and 3.x. There are Python users, who are serious about multilingual support, complain about 3.x. They insist 2.x behavior and even want to discontinue 3.x. I don't think mbstring is optimum multibyte string library neither. We may keep current structure until we have decent multibyte string library that could live long enough. By the time we have it, we may use compatibility switch for basic string functions to change multibyte awareness. If we are going to choose this way, it may be better to have byte_len() and other byte_*() function now for easier transition. I'm not sure which is the best having mb_*(), byte_*() or specifying binary encoding for standard string functions, though. It's out of this RFC scope. Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #71180 — Derick Rethans — 2014-01-16T11:38:59+00:00 URL: https://externals.io/message/71162#71180 · In reply to: #71170 > Hi! > > > […] > > It feels like we will end up with creating clones of pretty every string > function there is. Not sure it is the best approach... No, the best approach is to have "proper" unicode support in PHP, a la "PHP 6". Then at least the duplication happens for us devs and not users. cheers, Derick --- ## #71181 — Nikita Popov — 2014-01-16T12:18:36+00:00 URL: https://externals.io/message/71162#71181 · In reply to: #71162 > Hi all, > > `addslashes()` could be vulnerable via char encoding based attacks. > It is needed to decide what counter measure we adopt. > This is RFC for this issue. > > https://wiki.php.net/multibyte_char_handling > > Please comment. > Thank you. > Please do *not* add encoding parameters to our existing string functions - we have an mb extension and mb functionality should go there. Don't mix the things, it will only lead to a lot of confusion. Right now it's obvious which functions handle encoding how, no need to break that. Nikita --- ## #71196 — Yasuo Ohgaki — 2014-01-16T22:34:06+00:00 URL: https://externals.io/message/71162#71196 · In reply to: #71181 Hi Nikita, > > > > […] > > Please do *not* add encoding parameters to our existing string functions - > we have an mb extension and mb functionality should go there. Don't mix the > things, it will only lead to a lot of confusion. Right now it's obvious > which functions handle encoding how, no need to break that. > This discussion circulate discussion. At first, I proposed locale based solution using php_mblen(). This approach does not require additional encoding parameter since encoding is specified by locale. However, some people don't like the solution (in security ML) because it is locale based solution. It may have unwanted side effects. Locale is unreliable and most user just don't care about it. Therefore, I proposed this approach that introduce encoding parameter just like `htmlspecialchars()`/htmlentities(). Encoding parameter (or some way to specify encoding) for security related string function is mandatory. We should provide some way to specify encoding. Do you like locale based approach for now? Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #71197 — Nikita Popov — 2014-01-16T22:38:00+00:00 URL: https://externals.io/message/71162#71197 · In reply to: #71196 > Hi Nikita, > > On Thu, Jan 16, 2014 at 9:18 PM, Nikita Popov wrote: > > > […] > > This discussion circulate discussion. > > At first, I proposed locale based solution using php_mblen(). > This approach does not require additional encoding parameter > since encoding is specified by locale. > > However, some people don't like the solution (in security ML) > because it is locale based solution. It may have unwanted side > effects. Locale is unreliable and most user just don't care about it. > > Therefore, I proposed this approach that introduce encoding > parameter just like `htmlspecialchars()`/htmlentities(). > > Encoding parameter (or some way to specify encoding) for security > related string function is mandatory. We should provide some way > to specify encoding. > > Do you like locale based approach for now? > No, I don't want a locale-based approach. I want the string functions to stay as is. Multibyte variants of the functions can be added to the multibyte extension. Nikita --- ## #71198 — Yasuo Ohgaki — 2014-01-16T22:47:49+00:00 URL: https://externals.io/message/71162#71198 · In reply to: #71197 Hi Nikita, > No, I don't want a locale-based approach. I want the string functions to > stay as is. Multibyte variants of the functions can be added to the > multibyte extension. Creating mb_*() function would not solve security issues of multibyte char handling since multibyte aware functions are optional feature. However, it may work if PHP compiles mbstring by default and discourage use of `addslashes()`/var_export()/stripslashes() in favor of mb_*() variants. This could be voting option. Did I understand your opinion correctly? Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #71201 — Yasuo Ohgaki — 2014-01-17T03:19:21+00:00 URL: https://externals.io/message/71162#71201 · In reply to: #71180 Hi Derick, > > […] > > No, the best approach is to have "proper" unicode support in PHP, a la > "PHP 6". Then at least the duplication happens for us devs and not > users. I agree that we need to discuss encoding issue as a whole. This RFC is to eliminate known vulnerability from current implementation. Let's discuss this issue later. Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #71209 — Mike — 2014-01-17T16:37:58+00:00 URL: https://externals.io/message/71162#71209 · In reply to: #71196 > This discussion circulate discussion. I'm also in favor of putting those things in ext/mbstring. > > At first, I proposed locale based solution using php_mblen(). > This approach does not require additional encoding parameter > since encoding is specified by locale. Meh, but would be okay for me. > > However, some people don't like the solution (in security ML) > because it is locale based solution. It may have unwanted side > effects. Locale is unreliable and most user just don't care about it. > > Therefore, I proposed this approach that introduce encoding > parameter just like `htmlspecialchars()`/htmlentities(). > > Encoding parameter (or some way to specify encoding) for security > related string function is mandatory. We should provide some way > to specify encoding. > Many of us do not have access to that mailing list, so yould you shed some light on the acutal issue? -- Regards, Mike --- ## #71210 — Yasuo Ohgaki — 2014-01-17T19:53:58+00:00 URL: https://externals.io/message/71162#71210 · In reply to: #71209 Hi Mike, > > > > […] > > I'm also in favor of putting those things in ext/mbstring. > I'll make this a vote option. > > […] > > Meh, but would be okay for me. > It's feasible solution for older versions. I would like to remove locale based code for future releases, though. Functions that are using php_mblen() could be modified to use mbstring when PHP is built with mbstring. Functions may use internal_encoding. Use of internal_encoding requires user code modification in some cases. For instance, Japanese Windows command line uses Shift_JIS as terminal encoding while many users uses UTF-8 for script. Users has to add code that changes internal_encoding. e.g. `escapeshellarg()`/escapeshellcmd(). They could use simple wrapper for `escapeshellarg()`/escapeshellcmd(), though. Although users have to modify their code a little, `fgetcsv()` and like would be more usable because it's more reliable than locale. It may be better to add mb version of these functions and deprecate them like `addslashes()`, if we are not going to modify these functions. > > > […] > > Many of us do not have access to that mailing list, so yould you shed > some light on the acutal issue? > There are 2 classes of security issue in php_addslashes() First is PHP script execution. Suppose this is a script save app config script. then read it as PHP script. If '表' is SJIS, the char code is 0x955c (0x5c = \). Since `addslashes()` is not multibyte aware, it escapes the char as 0x95, 0x5c, 0x5c. This make possible that break out string quoting and write attack code. The contents of myconfig.php became Like everybody, I' absolutely against adding an "encoding" parameter > to ext/standard functions or relying on unreliable system locale. > Like Nikita says, every multibyte function should go to ext/mbstring , > and nowhere else, please , do not turn PHP into something even more > dirty as it is nowadays :-p > > I'm +1 to embed and activate mbstring by default in future PHP releases. > However, this has already been discussed (from what I remember) and I > dont remember why we ended with a "no" end-word, could we be refreshed > about this ? > > I'm not in favor of magic things. Magically replacing PHP strings by > mb_ implementation is a really bad idea. We should keep the INI > parameter about this alive though (mbstring.func_overload), so that > people that explicitely want to activate such a magic can do it if > they want to. > > I'm +1 also to start a "serious" (hum) discussion about multibyte/PHP6 > , we've been saying this for years : we need native support of unicode > in PHP :-p The actual problem we are facing once again shows this. > It seem this is the majority excluding INI usage. I updated the RFC to reflect this. https://wiki.php.net/rfc/multibyte_char_handling - Compile mbstring by default from 5.6 - Add mb_*() functions for 5.3 and up - Keep ext/standard function as it is now Open Issue - Use of INI for overriding single byte string functions by mbstring functions. I would like to consolidate code location 5.6 and up because of the history of mbstring function remain insecure. e.g. When `parse_str()`/mail() security issue was fixed, `mb_parse_str()`/mb_send_mail() didn't fixed. However, refactoring isn't mandatory. We could have 2 different codes that are mostly the same. We may postpone refactoring until PHP6. Please don't forget to update mbstring code when anyone update ext/standard. If there are opinions/open issues/questions, please reply. Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #71213 — Yasuo Ohgaki — 2014-01-17T21:41:31+00:00 URL: https://externals.io/message/71162#71213 · In reply to: #71212 Hi all, > - Keep ext/standard function as it is now SPL file object's fgetcsv method is using php_fgetcsv() which is using php_mblen(). Add mb_fget_csv method to it using #if HAVE_MBSTRING? I made new mb function names in the RFC a cording standard compliant. e.g. mb_add_slashes() Any comments for this? Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #71214 — Julien Pauli — 2014-01-17T21:50:04+00:00 URL: https://externals.io/message/71162#71214 · In reply to: #71212 > Hi all, > > > > […] > > > It seem this is the majority excluding INI usage. > I updated the RFC to reflect this. > > https://wiki.php.net/rfc/multibyte_char_handling > > - Compile mbstring by default from 5.6 > - Add mb_*() functions for 5.3 and up > - Keep ext/standard function as it is now > > Open Issue > - Use of INI for overriding single byte string functions by mbstring > functions. Why is that an issue ? We just leave it as-is , or ? > > > I would like to consolidate code location 5.6 and up because of the history > of > mbstring function remain insecure. e.g. When `parse_str()`/mail() security > issue > was fixed, `mb_parse_str()`/mb_send_mail() didn't fixed. > > However, refactoring isn't mandatory. We could have 2 different codes that > are > mostly the same. We may postpone refactoring until PHP6. Please don't forget > to update mbstring code when anyone update ext/standard. > > If there are opinions/open issues/questions, please reply. > > Regards, > > -- > Yasuo Ohgaki > yohgaki@ohgaki.net > Julien --- ## #71215 — Yasuo Ohgaki — 2014-01-17T22:10:27+00:00 URL: https://externals.io/message/71162#71215 · In reply to: #71214 Hi Julien, > > > […] > > Why is that an issue ? > We just leave it as-is , or ? Some users are annoyed by sloppy multilingual implementations using this option. There is feature request from user who want to remove mbstring.func_overload INI option. https://bugs.php.net/bug.php?id=65785 We may extend or drop this feature. I neutral for this. Regards. -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #71216 — Yasuo Ohgaki — 2014-01-17T22:31:05+00:00 URL: https://externals.io/message/71162#71216 · In reply to: #71215 Hi all, > Some users are annoyed by sloppy multilingual implementations using > this option. There is feature request from user who want to remove > mbstring.func_overload INI option. > > https://bugs.php.net/bug.php?id=65785 > > We may extend or drop this feature. I'm neutral for this. > Added this to the RFC. https://wiki.php.net/rfc/altmbstring is also added to related RFC. I'm not intended to propose/implement it anytime soon. Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #71217 — Julien Pauli — 2014-01-17T23:04:05+00:00 URL: https://externals.io/message/71162#71217 · In reply to: #71215 > Hi Julien, > > > > […] > > > Some users are annoyed by sloppy multilingual implementations using > this option. There is feature request from user who want to remove > mbstring.func_overload INI option. > > https://bugs.php.net/bug.php?id=65785 > > We may extend or drop this feature. I neutral for this. Yep, I admit that lots of people misconfigure it and that make PITA of developers that always have to test its value. Julien.P --- ## #71219 — Yasuo Ohgaki — 2014-01-18T00:20:06+00:00 URL: https://externals.io/message/71162#71219 · In reply to: #71215 Hi Julien, > Why is that an issue ? > > […] > > > Some users are annoyed by sloppy multilingual implementations using > this option. There is feature request from user who want to remove > mbstring.func_overload INI option. > > https://bugs.php.net/bug.php?id=65785 > > We may extend or drop this feature. I neutral for this. > Since we are better to concentrate on fixing the security issue, I've made it open issue for future releases. I'll just add mb new functions to mbstring.func_overload INI option. If func_overload is supported, we still needs php_mblen() for command line output since it's not good idea to detect locale and using internal_encoding opens new vulnerability. Functions that are better to use locale are - mb_escape_shell_arg() - mb_escape_shell_cmd() These function may override locale by encoding parameter. Since `fgetcsv()` uses locale now, we may do the same for `fgetcsv()` also. The RFC is updated. Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #71244 — Lester Caine — 2014-01-18T13:41:54+00:00 URL: https://externals.io/message/71162#71244 · In reply to: #71162 Yasuo Ohgaki wrote: > `addslashes()` could be vulnerable via char encoding based attacks. > It is needed to decide what counter measure we adopt. > This is RFC for this issue. > > https://wiki.php.net/multibyte_char_handling > > Please comment. Multibyte characters are still a contentious area, and the current compromise of supporting multibyte content, but being essentially 'single byte' for the programming structure as been a solution adopted in a few projects. Firebird is once again debating the same point that they and PHP last discussed 10 years ago, and was too difficult so PHP6 floundered and Firebird remained essentially single byte strings in the metadata. 10 years on isn't it time to re-open the debate on making the core unicode since 32 bit processors are more likely to be the norm these days. Certainly if everything internal is UTF8, then all of the encoding problems are moved to the client interface? (p.s rfc needs a little work via the spell checker and the link above is wrong) -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk --- ## #71247 — Yasuo Ohgaki — 2014-01-18T18:52:01+00:00 URL: https://externals.io/message/71162#71247 · In reply to: #71244 Hi Lester, > Multibyte characters are still a contentious area, and the current > compromise of supporting multibyte content, but being essentially 'single > byte' for the programming structure as been a solution adopted in a few > projects. Firebird is once again debating the same point that they and PHP > last discussed 10 years ago, and was too difficult so PHP6 floundered and > Firebird remained essentially single byte strings in the metadata. > Making a product only works for single byte char is completely OK. The issue is there is no proper function/method/feature that escapes PHP string with multibyte chars correctly. PHP needs to provide API that handles data properly/safely. It's awful that reading `var_export()`ed data could execute arbitrarily PHP script and/or terminate script execution, isn't it? It cannot be ignored. 10 years on isn't it time to re-open the debate on making the core unicode > since 32 bit processors are more likely to be the norm these days. > Certainly if everything internal is UTF8, then all of the encoding problems > are moved to the client interface? > I'm not proposing transition like Python 2.x to 3.x. The RFC is proposing required feature for proper/safe coding. Anyway, it seems Python's approach is not working well. We could learn from it. Server side should never expect clients are sending proper data, therefore proper encoding handling is mandatory on server side. Adoption of UTF-8 makes things easier, but there are ways to exploit UTF-8 encoding also. For example, recent Chrome may display blank page with malformed chars and it could be used for DoS attack, mixing systems that validate and un-validate encoding could be vulnerable DoS. New mb functions handle encoding properly not only SJIS like encoding but also any encoding supported by mbstiring. Did I make typo? My Chrome did not report spell error. I appreciate if you point it out. I sent correct URL right after first mail, but it wouldn't work. I also would not check second mail in long thread :( https://wiki.php.net/rfc/multibyte_char_handling Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #71255 — Lester Caine — 2014-01-18T21:19:43+00:00 URL: https://externals.io/message/71162#71255 · In reply to: #71247 Yasuo Ohgaki wrote: > I'm not proposing transition like Python 2.x to 3.x. The RFC is proposing > required feature for proper/safe coding. Anyway, it seems Python's approach is > not working well. We could learn from it. I am more than convinced that is a perfect example of how not to do it, but there surly is no reason to make such a mess of 'PHP6' over PHP5? I have to be able to program in Python for some of the tools, and they have no plans to move to 3.x any time soon :) -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk --- Next page: https://externals.io/message/71162.md?page=2