# Potential RFC: mb_rawurlencode() ? - Thread: https://externals.io/message/126828 - Original archive: https://news-web.php.net/php.internals/126828 - Started by Paul M. Jones on 2025-03-18T17:48:36+00:00 - Messages: 9 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #126828 — Paul M. Jones — 2025-03-18T17:48:36+00:00 URL: https://externals.io/message/126828#126828 Hi all, The discussion around WHATWG-URL on this list, as well as my work coordinating Uri-Interop , lead me to think PHP needs a multibyte equivalent of `rawurlencode()`. Broadly speaking, as far as I can tell: - For an RFC 3986 URI, delimiters need to be percent-encoded, as well as non-ASCII characters. - For an RFC 3987 IRI, delimiters need to be percent-encoded, but UCS characters do not. (There are other details but I think you get the idea.) The `rawurlencode()` function does fine for URIs, but not for IRIs. Using `rawurlencode()` for an IRI will encode multibyte characters when it should leave them alone. For example: ``` $val = 'fü bar'; $uriPath = '/heads/' . rawurlencode($val) . '/tails/'; assert($uriPath === '/heads/f%C3%BC%20bar/tails/'); // true $iriPath = '/heads/' . rawurlencode($val) . '/tails/'); assert($iriPath === '/heads/fü bar/tails/'; // false ``` (This might apply to WHATWG-URL component construction as well.) Have I missed something, either in the specs or in PHP itself? If not, how do we feel about an RFC for mb_rawurlencode()? A naive userland implementation might look something like the code below. Thoughts? * * * ```php function mb_rawurlencode(string $string) : string { $encoded = ''; foreach (mb_str_split($string) as $char) { $encoded .= match ($char) { chr(0) => "%00", chr(1) => "%01", chr(2) => "%02", chr(3) => "%03", chr(4) => "%04", chr(5) => "%05", chr(6) => "%06", chr(7) => "%07", chr(8) => "%08", chr(9) => "%09", chr(10) => "%0A", chr(11) => "%0B", chr(12) => "%0C", chr(13) => "%0D", chr(14) => "%0E", chr(15) => "%0F", chr(16) => "%10", chr(17) => "%11", chr(18) => "%12", chr(19) => "%13", chr(20) => "%14", chr(21) => "%15", chr(22) => "%16", chr(23) => "%17", chr(24) => "%18", chr(25) => "%19", chr(26) => "%1A", chr(27) => "%1B", chr(28) => "%1C", chr(29) => "%1D", chr(30) => "%1E", chr(31) => "%1F", chr(127) => "%7F", "!" => '%21', "#" => '%23', "$" => '%24', "%" => '%25', "&" => '%26', "'" => '%27', "(" => '%28', ")" => '%29', "*" => '%2A', "+" => '%2B', "," => '%2C', "/" => '%2F', ":" => '%3A', ";" => '%3B', "=" => '%3D', "?" => '%3F', "[" => '%5B', "]" => '%5D', default => $char, }; } return $encoded; } ``` * * * -- pmj --- ## #126853 — youkidearitai — 2025-03-20T06:31:38+00:00 URL: https://externals.io/message/126828#126853 ---------- Forwarded message --------- From: youkidearitai Date: 2025年3月20日(木) 14:41 Subject: Re: [PHP-DEV] Potential RFC: mb_rawurlencode() ? To: Paul M. Jones 2025年3月19日(水) 2:52 Paul M. Jones : > > Hi all, > > The discussion around WHATWG-URL on this list, as well as my work coordinating Uri-Interop , lead me to think PHP needs a multibyte equivalent of `rawurlencode()`. > > Broadly speaking, as far as I can tell: > > - For an RFC 3986 URI, delimiters need to be percent-encoded, as well as non-ASCII characters. > - For an RFC 3987 IRI, delimiters need to be percent-encoded, but UCS characters do not. > > (There are other details but I think you get the idea.) > > The `rawurlencode()` function does fine for URIs, but not for IRIs. Using `rawurlencode()` for an IRI will encode multibyte characters when it should leave them alone. For example: > > ``` > $val = 'fü bar'; > > $uriPath = '/heads/' . rawurlencode($val) . '/tails/'; > assert($uriPath === '/heads/f%C3%BC%20bar/tails/'); // true > > $iriPath = '/heads/' . rawurlencode($val) . '/tails/'); > assert($iriPath === '/heads/fü bar/tails/'; // false > ``` > > (This might apply to WHATWG-URL component construction as well.) > > Have I missed something, either in the specs or in PHP itself? > > If not, how do we feel about an RFC for mb_rawurlencode()? A naive userland implementation might look something like the code below. > > Thoughts? > > * * * > > ```php > function mb_rawurlencode(string $string) : string > { > $encoded = ''; > > foreach (mb_str_split($string) as $char) { > $encoded .= match ($char) { > chr(0) => "%00", > chr(1) => "%01", > chr(2) => "%02", > chr(3) => "%03", > chr(4) => "%04", > chr(5) => "%05", > chr(6) => "%06", > chr(7) => "%07", > chr(8) => "%08", > chr(9) => "%09", > chr(10) => "%0A", > chr(11) => "%0B", > chr(12) => "%0C", > chr(13) => "%0D", > chr(14) => "%0E", > chr(15) => "%0F", > chr(16) => "%10", > chr(17) => "%11", > chr(18) => "%12", > chr(19) => "%13", > chr(20) => "%14", > chr(21) => "%15", > chr(22) => "%16", > chr(23) => "%17", > chr(24) => "%18", > chr(25) => "%19", > chr(26) => "%1A", > chr(27) => "%1B", > chr(28) => "%1C", > chr(29) => "%1D", > chr(30) => "%1E", > chr(31) => "%1F", > chr(127) => "%7F", > "!" => '%21', > "#" => '%23', > "$" => '%24', > "%" => '%25', > "&" => '%26', > "'" => '%27', > "(" => '%28', > ")" => '%29', > "*" => '%2A', > "+" => '%2B', > "," => '%2C', > "/" => '%2F', > ":" => '%3A', > ";" => '%3B', > "=" => '%3D', > "?" => '%3F', > "[" => '%5B', > "]" => '%5D', > default => $char, > }; > } > > return $encoded; > } > ``` > > * * * > > > -- pmj Hi, Paul. I think signature is below: ```php function mb_rawurlencode(string $string, string $encode): string {} ``` Because the mbstring function is other than Unicode (ISO-8859-1 to ISO-8859-16, Shift_JIS, EUC-* etc). Other than that I don't know yet Oops, I missing to send to internals. Sorry resend this is. Yuya -- --------------------------- Yuya Hamada (tekimen) - https://tekitoh-memdhoi.info - https://github.com/youkidearitai --- ## #126870 — Paul M. Jones — 2025-03-20T16:46:46+00:00 URL: https://externals.io/message/126828#126870 · In reply to: #126853 Hi all, > > > ---------- Forwarded message --------- > From: youkidearitai > Date: 2025年3月20日(木) 14:41 > Subject: Re: [PHP-DEV] Potential RFC: mb_rawurlencode() ? > To: Paul M. Jones > > > 2025年3月19日(水) 2:52 Paul M. Jones : > > > […] > > Hi, Paul. > > I think signature is below: > > ```php > function mb_rawurlencode(string $string, string $encode): string {} > ``` Ah yes, you're right -- probably `?string $encode = null` to match with `mb_substr()`. > Oops, I missing to send to internals. > Sorry resend this is. Not to worry, thank you! -- pmj --- ## #126890 — tim@bastelstu.be — 2025-03-21T11:17:32+00:00 URL: https://externals.io/message/126828#126890 · In reply to: #126870 Hi Am 2025-03-20 17:46, schrieb Paul M. Jones: > > […] > > Ah yes, you're right -- probably `?string $encode = null` to match with > `mb_substr()`. I am not sure if that signature makes sense and if the proposed functionality fits into mbstring for that reason. IRIs are defined as UTF-8, any other encoding results in invalid output / results that are not interoperable. As one example paragraph from RFC 3987: > Conversions from URIs to IRIs MUST NOT use any character encoding > other than UTF-8 in steps 3 and 4, even if it might be possible to > guess from the context that another character encoding than UTF-8 was > used in the URI. The correct solution to me is to build a proper thought-through API as part of the proposed new Uri namespace and not adding new standalone functions without a clear vision. Best regards Tim Düsterhus --- ## #126892 — tim@bastelstu.be — 2025-03-21T11:22:50+00:00 URL: https://externals.io/message/126828#126892 · In reply to: #126828 Hi Am 2025-03-18 18:48, schrieb Paul M. Jones: > $iriPath = '/heads/' . rawurlencode($val) . '/tails/'); > assert($iriPath === '/heads/fü bar/tails/'; // false From my reading of RFC 3987 that result is incorrect. The space is neither listed as `iunreserved`, not as `sub-delims`, thus isn't a valid `ipchar`. Thus the space needs to be encoded as %20 for IRIs as well. The same mistake applies to the reference userland implementation below. Best regards Tim Düsterhus --- ## #126905 — Paul M. Jones — 2025-03-22T13:43:11+00:00 URL: https://externals.io/message/126828#126905 · In reply to: #126892 Hi Tim & all, > > > Am 2025-03-18 18:48, schrieb Paul M. Jones: > > […] > > From my reading of RFC 3987 that result is incorrect. The space is neither listed as `iunreserved`, not as `sub-delims`, thus isn't a valid `ipchar`. Thus the space needs to be encoded as %20 for IRIs as well. The same mistake applies to the reference userland implementation below. Agreed; the naive implementation would need to less naive and pay closer attention to the ABNF for `ucschar` and `ipchar` in the spec. Along those lines, I think there might need to be two additional changes/additions to help with encoding for RFC 3987 and WHATWG-URL component values: - `http_build_query()` would need PHP_QUERY_3987 and PHP_QUERY_WHATWG flags and corresponding logic (or entirely new functions); and - `parse_str()` would need a corresponding `mb_parse_str()`. -- pmj --- ## #126907 — Rowan Tommins [IMSoP] — 2025-03-22T15:20:15+00:00 URL: https://externals.io/message/126828#126907 · In reply to: #126890 > > I am not sure if that signature makes sense and if the proposed > functionality fits into mbstring for that reason. IRIs are defined as > UTF-8, any other encoding results in invalid output / results that are > not interoperable. This confirms a nagging feeling I had when I first saw the thread: the name "mb_rawurlencode" implies "do the same things as rawurlencode, but for multi-byte strings", but that's not what is being proposed. Notably, a similar feature is actually slated for removal; to quote https://www.php.net/manual/en/migration82.deprecated.php#migration82.deprecated.mbstring > Usage of the QPrint, Base64, Uuencode, and HTML-ENTITIES 'text encodings' is deprecated for all MBString functions. Unlike all the other text encodings supported by MBString, these do not encode a sequence of Unicode codepoints, but rather a sequence of raw bytes. It is not clear what the correct return values for most MBString functions should be when one of these non-encodings is specified. The same applies here: if you write mb_rawurlencode($my_string, 'SHIFT-JIS'), does that mean convert what you can to ASCII, and percent encode the rest for a URI; or does it mean convert to UTF-8, and percent encode as necessary for an IRI? If the input contains sequences which are not valid SHIFT-JIS, are those bytes treated as unencodable (producing errors or substitution characters), or are they directly percent encoded? > The correct solution to me is to build a proper thought-through API as > part of the proposed new Uri namespace and not adding new standalone > functions without a clear vision. I completely agree. For instance, the IRI standard does include an algorithm for converting a non-Unicode IRI representation to a URI - but it requires a Unicode Normalization step, which is a complex algorithm not included in ext/standard or ext/mbstring, only ext/intl. However, a function in the URI namespace that only handled the UTF-8 input case might still be useful. > Along those lines, I think there might need to be two additional changes/additions to help with encoding for RFC 3987 and WHATWG-URL component values: > > - `http_build_query()` would need PHP_QUERY_3987 and PHP_QUERY_WHATWG flags and corresponding logic (or entirely new functions); and > - `parse_str()` would need a corresponding `mb_parse_str()`. I haven't followed the other URI thread at all, but isn't replacing the scattered standard library functions with a consistent API the whole point of that effort? `parse_str()` in particular has a non-descriptive name, and a weird function signature because it used to directly overwrite variables by name. As a comparison, we didn't extend the `shuffle()` function with an algorithm parameter, we added a shuffleArray() method to the new Randomizer class. -- Rowan Tommins [IMSoP] --- ## #126908 — Paul M. Jones — 2025-03-22T16:08:54+00:00 URL: https://externals.io/message/126828#126908 · In reply to: #126907 Hi Rowan & all, > > > > > […] > > This confirms a nagging feeling I had when I first saw the thread: the name "mb_rawurlencode" implies "do the same things as rawurlencode, but for multi-byte strings", but that's not what is being proposed. [snip] No argument; my point is more "if we are going to do IRI and WHATWG-URL, we're going to need some additional support functionality around encoding component values for them." How that is achieved is up for grabs. If this discussion has revealed a tentative consensus that it needs to happen, I consider it a success. Next up: what exactly should the API around this functionality look like? I suggested functions but that's clearly a non-starter; what do we feel is a good alternative, and can it be achieved independently from (but in support of) the URI+WHATWG-URL proposal? -- pmj --- ## #126910 — Rowan Tommins [IMSoP] — 2025-03-23T12:04:10+00:00 URL: https://externals.io/message/126828#126910 · In reply to: #126908 >Next up: what exactly should the API around this functionality look like? I suggested functions but that's clearly a non-starter; what do we feel is a good alternative, and can it be achieved independently from (but in support of) the URI+WHATWG-URL proposal? As I say, I haven't followed the previous conversation at all, but from a glance at the RFC, it seems the proposed classes are called "Url"/"Uri", not "UrlParser"/"UriParser", so could maybe be expanded to creating *from* parts. I don't know where exactly IRIs should fit in, but maybe as a new object in the same hierarchy? There's also definitely a place for standalone functions for handling specific jobs on fragments of URIs. It would actually be really great to have a replacement for parse_str which didn't carry the baggage of old PHP versions - no by-reference output, no name mangling of keys (at least not by default). http_build_query isn't as urgently in need of replacement, but a clean start could default the separator to '&' rather than pulling from an INI setting. Whether each function should take an enum flag for encoding variants, or be split into a family of similar functions, I don't know. At the moment, http_build_query accepts constants, (raw)urlencode is split into two functions, and parse_str doesn't give any option. I don't want to have to memorise a bunch of RFC numbers in order to know whether spaces will be encoded as plus signs, but maybe we can find something more descriptive than "raw" to distinguish them. In short, I wouldn't start from the point of "how do we extend current functions to handle IRIs?", I'd start from the point of "what functions do we need for handling URI/URL/IRI parts, and what variations of each?" Rowan Tommins [IMSoP]