# [RFC] [Discussion] Add WHATWG compliant URL parsing API

- Thread: https://externals.io/message/123997
- Original archive: https://news-web.php.net/php.internals/123997
- Started by kocsismate90@gmail.com on 2024-06-28T20:06:14+00:00
- Messages: 149
- RFC: Throwable Hierarchy Policy for Extensions (Accepted) — https://externals.io/rfc/extension_exceptions.md
- RFC: Add RFC3986 and WHATWG URL compliant API (Implemented (PHP 8.5)) — https://externals.io/rfc/url_parsing_api.md
- Page 1 of 6: messages 1 to 25 (25 per page, add `?page=N` to the URL)
- Next page: https://externals.io/message/123997.md?page=2
- Reply by email to internals@lists.php.net
- More for agents: search with https://externals.io/search.md?q={query} · RFCs at https://externals.io/rfc.md · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt

---

## #123997 — kocsismate90@gmail.com — 2024-06-28T20:06:14+00:00

URL: https://externals.io/message/123997#123997

Hi Everyone,

I've been working on a new RFC for a while now, and time has come to
present it to a wider audience.

Last year, I learnt that PHP doesn't have built-in support for parsing URLs
according to any well established standards (RFC 1738 or the WHATWG URL
living standard), since the `parse_url()` function is optimized for
performance instead of correctness.

In order to improve compatibility with external tools consuming URLs (like
browsers), my new RFC would add a WHATWG compliant URL parser functionality
to the standard library. The API itself is not final by any means, the RFC
only represents how I imagined it first.

You can find the RFC at the following link:
https://wiki.php.net/rfc/url_parsing_api

Regards,
Máté

---

## #123998 — Marco Pivetta — 2024-06-28T20:21:33+00:00

URL: https://externals.io/message/123997#123998 · In reply to: #123997

Hey Máté,



> Hi Everyone,
>
> I've been working on a new RFC for a while now, and time has come to
> present it to a wider audience.
>
> Last year, I learnt that PHP doesn't have built-in support for parsing
> URLs according to any well established standards (RFC 1738 or the WHATWG
> URL living standard), since the `parse_url()` function is optimized for
> performance instead of correctness.
>
> In order to improve compatibility with external tools consuming URLs (like
> browsers), my new RFC would add a WHATWG compliant URL parser functionality
> to the standard library. The API itself is not final by any means, the RFC
> only represents how I imagined it first.
>
> You can find the RFC at the following link:
> https://wiki.php.net/rfc/url_parsing_api
>
> Regards,
> Máté
>

So far, amazing ! 👏

---

## #124000 — Lynn — 2024-06-28T21:02:08+00:00

URL: https://externals.io/message/123997#124000 · In reply to: #123997

> Hi Everyone,
>
> I've been working on a new RFC for a while now, and time has come to
> present it to a wider audience.
>
> Last year, I learnt that PHP doesn't have built-in support for parsing
> URLs according to any well established standards (RFC 1738 or the WHATWG
> URL living standard), since the `parse_url()` function is optimized for
> performance instead of correctness.
>
> In order to improve compatibility with external tools consuming URLs (like
> browsers), my new RFC would add a WHATWG compliant URL parser functionality
> to the standard library. The API itself is not final by any means, the RFC
> only represents how I imagined it first.
>
> You can find the RFC at the following link:
> https://wiki.php.net/rfc/url_parsing_api
>
> Regards,
> Máté
>
>
This is a great addition to have! I see there's nothing specifically about
`__toString` in the RFC, is this aiming to do the same as PSR-7?

---

## #124001 — Niels Dossche — 2024-06-28T21:35:36+00:00

URL: https://externals.io/message/123997#124001 · In reply to: #123997

> Hi Everyone,
> 
> I've been working on a new RFC for a while now, and time has come to present it to a wider audience.
> 
> Last year, I learnt that PHP doesn't have built-in support for parsing URLs according to any well established standards (RFC 1738 or the WHATWG URL living standard), since the `parse_url()` function is optimized for performance instead of correctness.
> 
> In order to improve compatibility with external tools consuming URLs (like browsers), my new RFC would add a WHATWG compliant URL parser functionality to the standard library. The API itself is not final by any means, the RFC only represents how I imagined it first.
> 
> You can find the RFC at the following link: https://wiki.php.net/rfc/url_parsing_api <https://wiki.php.net/rfc/url_parsing_api>
> 
> Regards,
> Máté
> 

Hi Máté

+1 from me, I'm all for modern web-related APIs as you know.

Some questions/remarks:
- Why did you choose UrlParser to be a "static" class? Right now it's just a fancy namespace.
  I can see the point of having a UrlParser class where you can e.g. configure it with which URL standard you want,
  but as it is now there is no such capability.
- It's a bit of a shame that the PSR interface treats queries as strings.
  In Javascript we have the URLSearchParams class that we can use as a key-value storage for query parameters.
  This Javascript class also handles escaping them nicely.
- Why is UrlComponent a backed enum?
- A nit: We didn't bundle the entire Lexbor engine, only select parts of it. Just thought I'd make it clear.
- About edge cases: e.g. what happens if I call the Url constructor and leave every string field empty?

Overall seems good.

Kind regards
Niels

---

## #124002 — Larry Garfield — 2024-06-28T22:14:19+00:00

URL: https://externals.io/message/123997#124002 · In reply to: #123997

> Hi Everyone,
>
> I've been working on a new RFC for a while now, and time has come to 
> present it to a wider audience.
>
> Last year, I learnt that PHP doesn't have built-in support for parsing 
> URLs according to any well established standards (RFC 1738 or the 
> WHATWG URL living standard), since the `parse_url()` function is 
> optimized for performance instead of correctness.
>
> In order to improve compatibility with external tools consuming URLs 
> (like browsers), my new RFC would add a WHATWG compliant URL parser 
> functionality to the standard library. The API itself is not final by 
> any means, the RFC only represents how I imagined it first.
>
> You can find the RFC at the following link: 
> https://wiki.php.net/rfc/url_parsing_api
>
> Regards,
> Máté

I am all for proper data modeling of all the things, so I support this effort.

Comments:

* There's no need for UrlComponent to be backed.

* I don't understand why UrlParser is a static class.  We just had a whole big debate about that. :-)

There's a couple of ways I could see it working, and I'm not sure which I prefer:

1. Better if we envision the parser getting options or configuration in the future.
$url = new UrlParser()->parseUrl(): Url;

2. The named-constructor pattern is quite common.
$url = Url::parseFromString()
$url = Url::parseToArray();

* I... do not understand the point of having public properties AND getters/withers.  A readonly class with withers, OK, a bit clunky to implement but it would be your problem in C, not mine, so I don't care. :-)  But why getters AND public properties?  If going that far, why not finish up clone-with and then we don't need the withers, either? :-)

* Making all the parameters to Url required except port makes little sense to me.  User/pass is more likely to be omitted 99% of the time than port.  In practice, most components are optional, in which case it would be inaccurate to not make them nullable.  Empty string wouldn't be quite the same, as that is still a value and code that knows to skip empty string when doing something is basically the same as code that knows to skip nulls.  We should assume people are going to instantiate this class themselves often, not just get it from the parser, so it should be designed to support that.

* I would not make Url final.  "OMG but then people can extend it!"  Exactly.  I can absolutely see a case for an HttpUrl subclass that enforces scheme as http/https, or an FtpUrl that enforces a scheme of ftp, etc.  Or even an InternalUrl that assumes the host is one particular company, or something.  (If this sounds like scope creep, it's because I am confident that people will want to creep this direction and we should plan ahead for it.)

* If the intent of the withers is to mimic PSR-7, I don't think it does so effectively.  Without the interface, it couldn't be a drop-in replacement for UriInterface anyway.  And we cannot extend it to add the interface if it's final.  Widening the parameters in PSR-7 interfaces to support both wouldn't work, as that would be a hard-BC break for any existing implementations.  So I don't really see what the goal is here.

* If we ever get "data classes", this would be a good candidate. :-)

* Crazy idea:

new UriParser(HttpUrl::class)->parse(string);

To allow a more restrictive set of rules.  Or even just to cast the object to that child class.

--Larry Garfield

---

## #124003 — Bilge — 2024-06-28T22:53:12+00:00

URL: https://externals.io/message/123997#124003 · In reply to: #124001

> - Why did you choose UrlParser to be a "static" class?

Because "static class" is the hip new cool ;)

Bilge

---

## #124004 — Ben Ramsey — 2024-06-28T23:28:36+00:00

URL: https://externals.io/message/123997#124004 · In reply to: #123997

> 
> 
> ﻿
> Hi Everyone,
> 
> I've been working on a new RFC for a while now, and time has come to present it to a wider audience.
> 
> Last year, I learnt that PHP doesn't have built-in support for parsing URLs according to any well established standards (RFC 1738 or the WHATWG URL living standard), since the `parse_url()` function is optimized for performance instead of correctness.
> 
> In order to improve compatibility with external tools consuming URLs (like browsers), my new RFC would add a WHATWG compliant URL parser functionality to the standard library. The API itself is not final by any means, the RFC only represents how I imagined it first.
> 
> You can find the RFC at the following link: https://wiki.php.net/rfc/url_parsing_api
> 
> Regards,
> Máté


The RFC states:

<snip>
The Url\Url class is intentionally compatible with the PSR-7 UriInterface.
</snip>

It mirrors the interface, but it can’t be swapped out for a UriInterface instance, especially since it can’t be extended, so I wouldn’t consider it compatible. I would still need to write a compatibility layer that composes Url\Url and implements UriInterface.

<snip>
This makes it possible for a next iteration of the PSR-7 standard to use Url\Url directly instead of requiring implementations to provide their own Psr\Http\Message\UriInterface implementation.
</snip>

Since PSRs are concerned with shared interfaces and this class is final and does not implement any interfaces, I’m not sure how you envision “a next iteration” of PSR-7 to use this directly, unless what you mean is that UriInterface would be deprecated and applications would type directly against Url\Url.

Cheers,
Ben

---

## #124013 — nyamsprod the funky webmaster — 2024-06-29T08:20:11+00:00

URL: https://externals.io/message/123997#124013 · In reply to: #123997

> Hi Everyone,
> 
> I've been working on a new RFC for a while now, and time has come to 
> present it to a wider audience.
> 
> Last year, I learnt that PHP doesn't have built-in support for parsing 
> URLs according to any well established standards (RFC 1738 or the WHATWG 
> URL living standard), since the `parse_url()` function is optimized for 
> performance instead of correctness.
> 
> In order to improve compatibility with external tools consuming URLs 
> (like browsers), my new RFC would add a WHATWG compliant URL parser 
> functionality to the standard library. The API itself is not final by 
> any means, the RFC only represents how I imagined it first.
> 
> You can find the RFC at the following link: 
> https://wiki.php.net/rfc/url_parsing_api 
> <https://wiki.php.net/rfc/url_parsing_api>
> 
> Regards,
> Máté
> 

As a maintainer of a PHP userland URI toolkit I have a couple of 
questioms/remarks on the proposal. Fist, I look forward for finally 
having a real Url parser AND validator in PHP core. Any effort on that 
direction is always a welcomed good news.

As far as I understand it, if this RFC were to pass as is it will model 
PHP URLs to the WHATWG specification. While this specification is 
getting a lot of traction lately I believe it will restrict URL usage in 
PHP instead of making developer life easier. While PHP started as a 
"web" language it is first and foremost a server side general purpose
language. The WHATWG spec on the other hand is created by browsers 
vendors and is geared toward browsers (client side) and because of 
browsers history it restricts by design a lot of what PHP developers can 
currently do using `parse_url`. In my view the `Url` class in
PHP should allow dealing with any IANA registered scheme, which is not 
the case for the WHATWG specification.

Therefore, I would rather suggest we ALSO include support for RFC3986 
and RFC3987 specification properly and give both specs a go (at the same 
time!) and a clear way to instantiate your `Url` with one or the other spec.
In clear, my ideal situation would be to add to the parser at least 2 
named constructors `UrlParser::fromRFC3986` and `UrlParser::fromWHATWG`
or something similar (name can be changed or improved).

While this is an old article by Daniel Stenberg 
(https://daniel.haxx.se/blog/2017/01/30/one-url-standard-please/), it 
conveys with more in depth analysis my issues with the WHATWG spec and 
its usage in PHP if it were to be use as the ONLY available parser in 
PHP core for URL.

the PSR-7 relation is also unfortunate from my POV: PSR-7 UriInterface 
is designed to be at its core an HTTP URI representation (so it shares 
the same type of issue as the WHATWG spec!) meaning in absence of a 
scheme it falls back to the HTTP scheme validation. This is why the 
interface can forgone any nullable component because the HTTP spec 
allows it, other schemes do not. For instance the FTP scheme prohibits 
the presence of the query and fragment components which means they MUST 
be `null` in that case.

By removing PSR-7 constraints we could add

-  the `Url::(get|to)Components` method: it would mimics `parse_url` 
returned value and as such ease migration from `parse_url`
-  the `Url::getUsername` and `Url::getPassword` to access the username 
and password component individually. You would still use
the `withUserInfo` method to update them but you give the developer the 
ability to access both components directly from the `Url` object.

These additions would remove the need for

- `UrlParser::parseUrlToArray`
- `UrlParser::parseUrlComponent`
- `UrlComponent` Enum

Cheers,
Ignace

---

## #124023 — Stephen Reay — 2024-06-29T09:57:17+00:00

URL: https://externals.io/message/123997#124023 · In reply to: #124001

> 
> 
> - It's a bit of a shame that the PSR interface treats queries as strings.
> In Javascript we have the URLSearchParams class that we can use as a key-value storage for query parameters.
> This Javascript class also handles escaping them nicely.

Agreed this is a weird choice to me, but I'm also not surprised by weird choices via
php-fig (log level constants I'm looking at you)

We hear all the time how userland is more flexible and can change quicker - and yet here we see a potential built in class having a worse api because it wants to be compatible with an existing userland interface with the same bad api....

Cheers

Stephen

---

## #124025 — Rob Landers — 2024-06-29T10:33:16+00:00

URL: https://externals.io/message/123997#124025 · In reply to: #124023

> 
> 
> > […]
> 
> Agreed this is a weird choice to me, but I'm also not surprised by weird choices via
> php-fig (log level constants I'm looking at you)
> 
> We hear all the time how userland is more flexible and can change quicker - and yet here we see a potential built in class having a worse api because it wants to be compatible with an existing userland interface with the same bad api....
> 
> Cheers
> 
> Stephen  

I personally ignore PSR when it doesn't make sense to use it. They're nice for library compatibility, but I will happily toss compatibility when it doesn't make sense to be compatible. This might be one of those cases as there is no reason it has to be PSR compliant. In fact, a wrapper may be written to make it compliant, if one so chooses. I suspect it is better to be realistic and learn from the short-comings of PSR and apply those learnings here, vs. reiterating them and "engraving them in stone" (so to speak).

— Rob

---

## #124047 — Juris Evertovskis — 2024-06-29T16:19:12+00:00

URL: https://externals.io/message/123997#124047 · In reply to: #123997

> Hi Everyone,
> 
> I've been working on a new RFC for a while now, and time has come to 
> present it to a wider audience.
> 
> Last year, I learnt that PHP doesn't have built-in support for parsing 
> URLs according to any well established standards (RFC 1738 or the 
> WHATWG URL living standard), since the `parse_url()` function is 
> optimized for performance instead of correctness.
> 
> In order to improve compatibility with external tools consuming URLs 
> (like browsers), my new RFC would add a WHATWG compliant URL parser 
> functionality to the standard library. The API itself is not final by 
> any means, the RFC only represents how I imagined it first.
> 
> You can find the RFC at the following link: 
> https://wiki.php.net/rfc/url_parsing_api
> 
> Regards,
> Máté

Hey,

That's great that you've made the Url class readonly. Immutability is 
realiable. And I fully agree that a better parser is needed.

I agree with the otters that

- the enum might be fine without the backing, if it's needed at all
- I'm not convinced a separate UrlParser is needed, 
Url::someFactory($str) should be enough
- getters seem unnecessary, they should only be added if you can be sure 
they are going to be used for compatibility with PSR-7
- treating $query as a single string is clumsy, having some kind of bag 
or at least an array to represent it would be cooler and easier to build 
and manipulate

I wanted to add that it might be more useful to make all the Url 
constructor arguments optional. Either nullable or with reasonable 
defaults. So you could `$url = new Url(path: 'robots.txt'); foreach 
($domains as $d) $r[] = file_get_contents($url->withHost($d))` and stuff 
like that.

Similar modifiers would be very useful for the query stuff, e.g. `$u = 
Url::current(); return $u->withQueryParam('page', $u->queryParam->page + 
1);`.

Sure, all of that can be done in the userland as long as you drop 
`final` :)

BR,
Juris

---

## #124054 — Ben Ramsey — 2024-06-29T17:35:46+00:00

URL: https://externals.io/message/123997#124054 · In reply to: #124013

> 
> 
> Therefore, I would rather suggest we ALSO include support for RFC3986 and RFC3987 specification properly and give both specs a go (at the same time!) and a clear way to instantiate your `Url` with one or the other spec.
> In clear, my ideal situation would be to add to the parser at least 2 named constructors `UrlParser::fromRFC3986` and `UrlParser::fromWHATWG`
> or something similar (name can be changed or improved).
> 
> While this is an old article by Daniel Stenberg (https://daniel.haxx.se/blog/2017/01/30/one-url-standard-please/), it conveys with more in depth analysis my issues with the WHATWG spec and its usage in PHP if it were to be use as the ONLY available parser in PHP core for URL.


I agree that I would love to see a more general IRI parser, with maybe a URI parser being a subtype of an IRI parser.

Cheers,
Ben

---

## #124061 — Krinkle — 2024-06-29T20:27:50+00:00

URL: https://externals.io/message/123997#124061 · In reply to: #123997

> […] add a WHATWG compliant URL parser functionality to the standard library. The API itself is not final by any means, the RFC only represents how I imagined it first.
> 
> You can find the RFC at the following link: https://wiki.php.net/rfc/url_parsing_api

First-pass comments/thoughts.

As others have mentioned, it seems the class would/could not actually satisfy PSR-7. Realistically, the PSR-7 interface package or someone else would need to create a new class that combines the two, potentially as part of a transition away from it to the built-in class, with future PSRs building directly on Url. If we take that as given, we might as well design for the end state, and accept that there will be a (minimal) transition. This end state would benefit from being designed with the logical constraints of PSR-7 (so that migration is possible without major surprises), but without restricting us to its exact API shape, since an intermediary class would come into existence either way.

For example, Url could be a value class with merely 8 public properties. Possibly with a UrlImmutable subclass, akin to DateTime, where the properties are read-only instead a clone method could return Url?).

It might be more ergonomic to leave the parser as implementation detail, allowing the API to be accessed from a single import rather than requiring two. This could look like Url::parse() or Url::parseFromString(). 

For the Url::parseComponent() method, did you consider accepting the existing PHP_URL_* constants? They appear to fit exactly, in naming, description, and associated return types.

Without UrlParser/UrlComponent, I'd adopt it direclty in applications and frameworks. WIthout it, further wrapping seems likely for improved usability. This is sometimes benefitial when exposing low-level APIs, but it seems like this is close to fitting in a single class, as demonstrated by the WHATWG URL API.

One thing I feel is missing, is a method to parse a (partial) URL relative to another. E.g. to expand or translate paths between two URLs. Consider expanding "/w/index.php", or "index.php" relative to "https://wikipedia.org/w/". Or expanding "//example.org" relative to either "https://wikipedia.org" vs "http://wikipedia.org". The WHATWG URL API does this in the form of a second optional string|Stringable parameter to Url::parse(). Implementing "expand URL"  with parsing of incomplete URLs is error-prone and hard to get right.  Including this would be valuable.

See also Net_URL2 and its `resolve()` method https://pear.php.net/package/Net_URL2 https://github.com/pear/Net_URL2 

--
Timo Tijhof
https://timotijhof.net/

---

## #124066 — kocsismate90@gmail.com — 2024-06-29T22:42:06+00:00

URL: https://externals.io/message/123997#124066 · In reply to: #124002

Hi Larry,

Thank you very much for your feedback! I think I have already partially
answered some of your questions in my previous email to Niels,
but let me answer your other questions below:

* I... do not understand the point of having public properties AND
> getters/withers.  A readonly class with withers, OK, a bit clunky to
> implement but it would be your problem in C, not mine, so I don't care.
> :-)  But why getters AND public properties?  If going that far, why not
> finish up clone-with and then we don't need the withers, either? :-)


I know it's disappointing, but the public modifiers are just a typo which
were forgotten there from the very first iteration of the API :) However,
I'm fine with having public readonly properties without getters as well, as
long as we declare this a policy that we are going to adopt... Withers are
indeed a must for now (and their implementation indeed requires some magic
in C...).

* Making all the parameters to Url required except port makes little sense
> to me. User/pass is more likely to be omitted 99% of the time than port.
> In practice, most components are optional, in which case it would be
> inaccurate to not make them nullable.  Empty string wouldn't be quite the
> same, as that is still a value and code that knows to skip empty string
> when doing something is basically the same as code that knows to skip
> nulls.  We should assume people are going to instantiate this class
> themselves often, not just get it from the parser, so it should be designed
> to support that.


I may have misunderstood what you wrote, but all the parameters - including
port - are required. If you really meant "nullable" instead of "required",
then you are right. Apart from this, I'm completely fine with making these
parameters optional, especially if we decide not to have the UrlParser (my
initial assumption was that the Url class is going to be instantiated via
UrlParser::parseUrl() calls).

* I would not make Url final.  "OMG but then people can extend it!"
> Exactly.  I can absolutely see a case for an HttpUrl subclass that enforces
> scheme as http/https, or an FtpUrl that enforces a scheme of ftp, etc.  Or
> even an InternalUrl that assumes the host is one particular company, or
> something.  (If this sounds like scope creep, it's because I am confident
> that people will want to creep this direction and we should plan ahead for
> it.)


Without having thought much about its consequences on the implementation,
I'm fine with removing the final modifier.

* If the intent of the withers is to mimic PSR-7, I don't think it does so
> effectively.  Without the interface, it couldn't be a drop-in replacement
> for UriInterface anyway.  And we cannot extend it to add the interface if
> it's final.  Widening the parameters in PSR-7 interfaces to support both
> wouldn't work, as that would be a hard-BC break for any existing
> implementations.  So I don't really see what the goal is here.


I've just answered this to Ben, but let me reiterate: PSR-7's UriInterface
is only needed because PHP doesn't have a Url internal class. :)

Máté

---

## #124071 — kocsismate90@gmail.com — 2024-06-30T06:00:00+00:00

URL: https://externals.io/message/123997#124071

>
> It mirrors the interface, but it can’t be swapped out for a UriInterface
> instance, especially since it can’t be extended, so I wouldn’t consider it
> compatible. I would still need to write a compatibility layer that composes
> Url\Url and implements UriInterface.
>

I guess my words were slightly misleading: what I should have written is
that the methods themselves are compatible.


> Since PSRs are concerned with shared interfaces and this class is final
> and does not implement any interfaces, I’m not sure how you envision “a
> next iteration” of PSR-7 to use this directly, unless what you mean is that
> UriInterface would be deprecated and applications would type directly
> against Url\Url.
>

Yes, I meant the latter exactly. If we had a well-usable URL object
representation in the standard library, then there would be no need to have
an userland interface as well (unless they have different behavior or
purpose). Analogically, we have DateTimeImmutable, and there is no PSR for
a date time interface. (I know there is Carbon and other libraries, but
they are for convenience, not for interoperability).

---

## #124072 — kocsismate90@gmail.com — 2024-06-30T06:00:00+00:00

URL: https://externals.io/message/123997#124072 · In reply to: #124001

Hi Niels,

First of all, thank you for your support!

Why did you choose UrlParser to be a "static" class? Right now it's just a
> fancy namespace.


That's a good question, let me explain the reason: one of my major design
goals was to make the UrlParser class to be
extendable and configurable (e.g. via an "engine" property similar to what
Random/Randomizer has). Of course, UrlParser
doesn't support any of this yet, but at least the possibility is there for
followup RFCs due to the class being final.

Since I knew it would be an overkill to require instantiating an UrlParser
instance for a task which is stateless (URL parsing),
finally I settled on using static methods for the purpose. Later, if the
need arises, the static methods could be converted to
non-static ones with minimal BC impact.

It's a bit of a shame that the PSR interface treats queries as strings.

In Javascript we have the URLSearchParams class that we can use as a
> key-value storage for query parameters.


Hm, yes, that's an observation I can agree with. However, this restriction
shouldn't limit followups to add key-value storage
support for query parameters. Although, as far as I could determine,
neither Lexbor is capable of such a thing currently.

Why is UrlComponent a backed enum?


To be honest, it has no specific reason apart from that's what I am used
to. I'm fine with whatever choice, even with getting rid of
UrlComponent completely. I added the UrlParser::parseUrlComponent() method
(and hence the UrlComponent enum) to the
proposal in order to have a direct replacement for `parse_url()` when it's
called with the $component parameter set, but I wasn't
really sure whether this is needed at all... So I'm eager to hear any
recommendations regarding this problem.

A nit: We didn't bundle the entire Lexbor engine, only select parts of it.
> Just thought I'd make it clear.


Yes, my wording was slightly misleading. I'll clarify this in the RFC.

About edge cases: e.g. what happens if I call the Url constructor and leave
> every string field empty?


Nothing :) The Url class in its current form can store invalid URLs. I know
that URLs are generally modeled as value objects (that's
also why the proposed class is immutable), and generally speaking, value
objects should protect their invariants. However, due to
separating the parser to its own class, I abandoned this "rule". So this is
one more downside of the current API.

Regards,
Máté

---

## #124077 — ignace nyamagana butera — 2024-06-30T06:51:52+00:00

URL: https://externals.io/message/123997#124077 · In reply to: #124023

> 
> 
> > […]
> 
> Agreed this is a weird choice to me, but I'm also not surprised by weird choices via
> php-fig (log level constants I'm looking at you)
> 
> We hear all the time how userland is more flexible and can change quicker - and yet here we see a potential built in class having a worse api because it wants to be compatible with an existing userland interface with the same bad api....
> 
> Cheers
> 
> Stephen

While I do not think the debate should be about compatibility with PSR-7 
some historical context shoyld be brought to light for a fair discussion:

- parse_url and parse_str predates RFC3986
- URLSearchParans was ratified before PSR-7 BUT the first implementation 
landed a year AFTER PSR-7 was released and already implemented.
- PHP historical query parser parse_str logic is so bad (mangled 
parameter name for instance) that PSR-7 was right not embedding that 
parsing algorithm in its specification.
- If you take aside URITemplate specification and now URLSearchParams 
there is no official, referenced and or agreed upon rules/document on
how a query string MUST or SHOULD be parsed.
- Last but not least URLSearchParans encoding/decoding rules DO NOT 
follow either RFC1738 nor RFC3986 (they follow the form data which is
kind of a mix between both RFC)

THis means that just adding a method or a class that mimic 100% 
URLSearchParans for instance will constitute a major departure in how
PHP trears query string you will no longer have a 1:1 relation between
the data you have inside your `_GET` array and the one in 
UrlSearchParams for better or for worse.

For all these arguments I would keep the proposed `Url` free of all
these concerns and lean toward a nullable string for the query string 
representation. And defer this debate to its own RFC regarding query 
string parsing handling in PHP.

---

## #124100 — Larry Garfield — 2024-06-30T14:41:02+00:00

URL: https://externals.io/message/123997#124100 · In reply to: #124071

> > […]
>
> I guess my words were slightly misleading: what I should have written 
> is that the methods themselves are compatible.
> 
> > […]
>
> Yes, I meant the latter exactly. If we had a well-usable URL object 
> representation in the standard library, then there would be no need to 
> have an userland interface as well (unless they have different behavior 
> or purpose). Analogically, we have DateTimeImmutable, and there is no 
> PSR for a date time interface. (I know there is Carbon and other 
> libraries, but they are for convenience, not for interoperability).

I cannot speak on behalf of FIG here, but as a long-time member of FIG and a member of the Core Committee, I would urge you to *not* try to make a core Url object compatible with UriInterface.  It's solving a slightly different problem, using a language that is very different (PHP 8.4 vs 5.5 or so), with somewhat different constraints.

Instead, let's make sure that any new Url object is *composable* by PSR-7 UriInterface.  A UriInterface implementation that is backed by a Url object internally should be an easy task to do if anyone wants, and we should make sure we don't do anything that makes that unnecessarily hard, but right now the language is simply not capable of making core Url a drop-in replacement for UriInterface, so let's not even try.

Whether that means Url should be readonly or not, have getters/setters/withers, split authority into user:pass, etc. are things we should discuss on their own merits, not based on "well PSR-7 did it this way over 10 years ago, so we'll just do that."

--Larry Garfield

---

## #124252 — kocsismate90@gmail.com — 2024-07-07T09:13:58+00:00

URL: https://externals.io/message/123997#124252 · In reply to: #124077

Hi Ignace,

As far as I understand it, if this RFC were to pass as is it will model
> PHP URLs to the WHATWG specification. While this specification is
> getting a lot of traction lately I believe it will restrict URL usage in
> PHP instead of making developer life easier. While PHP started as a
> "web" language it is first and foremost a server side general purpose
> language. The WHATWG spec on the other hand is created by browsers
> vendors and is geared toward browsers (client side) and because of
> browsers history it restricts by design a lot of what PHP developers can
> currently do using `parse_url`. In my view the `Url` class in
> PHP should allow dealing with any IANA registered scheme, which is not
> the case for the WHATWG specification.


Supporting IANA registered schemes is a valid request, and is definitely
useful.
However, I think this feature is not strictly required to have in the
current RFC.
Anyone we needs to support features that are not offered by the WHATWG
standard can still rely on `parse_url()`. And of course, we can (and should)
add
support for other standards later. If we wanted to do all these in the same
RFC, then the scope of the RFC would become way too large IMO. That's why I
opt for incremental improvements.

Besides, I fail to see why a WHATWG compliant parser wouldn't be useful in
PHP:
yes, PHP is server side, but it still interacts with browsers very heavily.
Among other
use-cases I cannot yet image, the major one is most likely validating
user-supplied URLs
for opening in the browser. As far as I see the situation, currently there
is no acceptably
reliable possibility to decide whether a URL can be opened in browsers or
not.

- parse_url and parse_str predates RFC3986
> - URLSearchParans was ratified before PSR-7 BUT the first implementation
> landed a year AFTER PSR-7 was released and already implemented.
>

Thank you for the historical context!

Based on your and others' feedback, it has now become clear for me that
 `parse_url()`
is still useful and ext/url needs quite some additional capabilities until
this function
really becomes superfluous. That's why it now seems to me that the behavior
of
 `parse_url()` could be leveraged in ext/url so that it would work with a
Url/Url class (e.g.
we had a PhpUrlParser class extending the Url/UrlParser, or a
Url\Url::fromPhpParser()
method, depending on which object model we choose. Of course the names are
TBD).

For all these arguments I would keep the proposed `Url` free of all
> these concerns and lean toward a nullable string for the query string
> representation. And defer this debate to its own RFC regarding query
> string parsing handling in PHP.
>

My WIP implementation still uses nullable properties and return types. I
only changed those
when I wrote the RFC. Since I see that PSR-7 compatibility is very low prio
for everyone
involved in the discussion, then I think making these types nullable is
fine. It was neither my
top prio, but somewhere I had to start the object design, so I went with
this.

Again, thank you for your constructive criticism.

Regards,
Máté

---

## #124255 — Rob Landers — 2024-07-07T10:40:02+00:00

URL: https://externals.io/message/123997#124255 · In reply to: #124252

> Hi Ignace,
> 
> > […]
> 
> Supporting IANA registered schemes is a valid request, and is definitely useful.
> However, I think this feature is not strictly required to have in the current RFC.
> Anyone we needs to support features that are not offered by the WHATWG
> standard can still rely on `parse_url()`. And of course, we can (and should) add
> support for other standards later. If we wanted to do all these in the same
> RFC, then the scope of the RFC would become way too large IMO. That's why I
> opt for incremental improvements.

It's also worth pointing out (as another reason not to do this) is that IANA may-or-may not be valid in the current network. For example, TOR, Handshake, IPFS, Freenet, etc. all have their own DNS schemes and do not (usually) use IANA registered schemes, and many people create sites that cater to those networks.

> 
> Besides, I fail to see why a WHATWG compliant parser wouldn't be useful in PHP:
> yes, PHP is server side, but it still interacts with browsers very heavily. Among other
> use-cases I cannot yet image, the major one is most likely validating user-supplied URLs
> for opening in the browser. As far as I see the situation, currently there is no acceptably
> reliable possibility to decide whether a URL can be opened in browsers or not.

Looking at the spec for WHATWG, it looks like `example%2Ecom` will be parsed as a valid URL, and transformed to `example.com`, while this doesn't currently happen in `parse_url()`:

https://3v4l.org/NtqQm

I don't know if that may be an issue, but might be if you are expecting the string to remain URL encoded.

> 
> > […]
> 
> Thank you for the historical context!
> 
> Based on your and others' feedback, it has now become clear for me that `parse_url()`
> is still useful and ext/url needs quite some additional capabilities until this function
> really becomes superfluous. That's why it now seems to me that the behavior of
> `parse_url()` could be leveraged in ext/url so that it would work with a Url/Url class (e.g.
> we had a PhpUrlParser class extending the Url/UrlParser, or a Url\Url::fromPhpParser()
> method, depending on which object model we choose. Of course the names are TBD).
> 
> > […]
> 
> My WIP implementation still uses nullable properties and return types. I only changed those
> when I wrote the RFC. Since I see that PSR-7 compatibility is very low prio for everyone
> involved in the discussion, then I think making these types nullable is fine. It was neither my
> top prio, but somewhere I had to start the object design, so I went with this.

The spec contains elements and their types. It would be good to adhere to the spec (simplifies documentation):

 1. scheme may be null or empty string
 2. port may be null
 3. path is never null, but may be empty string
 4. query may be null
 5. fragment may be null
 6. user/password may be null (to differentiate between an empty password or no password)
 7. host may be null (for relative URLs

> 
> Again, thank you for your constructive criticism.
> 
> Regards,
> Máté

— Rob

---

## #124256 — ignace nyamagana butera — 2024-07-07T10:55:18+00:00

URL: https://externals.io/message/123997#124256 · In reply to: #124252

Hi Máté,

 > Supporting IANA registered schemes is a valid request, and is 
definitely useful. However, I think this feature is not strictly 
required to have in the current RFC.

True. Having a WHATWG compliant parser in PHP source code is a big +1 
from me I have nothing against that inclusion.

 > Based on your and others' feedback, it has now become clear for me 
that `parse_url()` is still useful and ext/url needs quite some additional 
capabilities until this function really becomes superfluous.

`parse_url` can only be deprecated when a RFC3986 compliant parser is 
added to php-src, hence why I insist in having that parser being present 
too.

I will also add that everything up to now in PHP uses RFC3986 as basis 
for generating or representing URLs (cURL extension, streams, etc...). 
Having the first and only OOP representation of an URL in the language 
not following that same specification seems odd to me. It opens the door 
to inconcistencies that will only be resolved once an equivalent RFC3986 
URL object made its way into the source code.

On the public API side I would recommend the following:

- if you are to strictly follow the WHATWG specification no URI 
component can be null. They must all be strings. If we have to plan to 
use the same object for RFC3986 compliant parser, then all components 
should be nullable except for the path component which can never be null 
as it is always present.

- As other have mention we should add a method to resolve an URI against 
a base URI something like Url::resolve(string $url, Url|string|null 
$baseUrl) where the baseURL argument should be an absolute Url if 
present. If absent the url argument must be absolute otherwise an 
exception should be thrown

- last but not least the WHATWG specification is not only a URL parser 
but also a URL validator and can apply some "correction" to malformed 
URL and report them. The specification has a provision for a structure 
to report malformed URL errors. I failed to see this mechanism being 
mention anywhere the RFC. Will the URL only trigger exceptions or will 
it also triggers warnings ? For inspiration the excellent PHP userland 
WHATWG URL parser from Trevor Rowbotham 
https://github.com/TRowbotham/URL-Parser allow using a PSR-3 logger to 
record those errors.

Best regards,
Ignace

---

## #124257 — Rob Landers — 2024-07-07T10:59:45+00:00

URL: https://externals.io/message/123997#124257 · In reply to: #124255

> 
> > […]
> 
> It's also worth pointing out (as another reason not to do this) is that IANA may-or-may not be valid in the current network. For example, TOR, Handshake, IPFS, Freenet, etc. all have their own DNS schemes and do not (usually) use IANA registered schemes, and many people create sites that cater to those networks.
> 
> > […]
> 
> Looking at the spec for WHATWG, it looks like `example%2Ecom` will be parsed as a valid URL, and transformed to `example.com`, while this doesn't currently happen in `parse_url()`:
> 
> https://3v4l.org/NtqQm
> 
> I don't know if that may be an issue, but might be if you are expecting the string to remain URL encoded.
> 
> > […]
> 
> The spec contains elements and their types. It would be good to adhere to the spec (simplifies documentation):
> 
>  1. scheme may be null or empty string
>  2. port may be null
>  3. path is never null, but may be empty string
>  4. query may be null
>  5. fragment may be null
>  6. user/password may be null (to differentiate between an empty password or no password)
>  7. host may be null (for relative URLs
> 
> > […]
> 
> — Rob

Here's a list of examples worth adding to the RFC:

//example.com?
ftp://user@example.com/path/to/ffile
https://user:@example.com
https://user:pass@example%2Ecom/?something=other&bool#heading

etc.

— Rob

---

## #124258 — Rob Landers — 2024-07-07T11:10:11+00:00

URL: https://externals.io/message/123997#124258 · In reply to: #124256

> Hi Máté,
> 
> > […]
> definitely useful. However, I think this feature is not strictly 
> required to have in the current RFC.
> 
> True. Having a WHATWG compliant parser in PHP source code is a big +1 
> from me I have nothing against that inclusion.
> 
> > […]
> that `parse_url()` is still useful and ext/url needs quite some additional 
> capabilities until this function really becomes superfluous.
> 
> `parse_url` can only be deprecated when a RFC3986 compliant parser is 
> added to php-src, hence why I insist in having that parser being present 
> too.
> 
> I will also add that everything up to now in PHP uses RFC3986 as basis 
> for generating or representing URLs (cURL extension, streams, etc...). 
> Having the first and only OOP representation of an URL in the language 
> not following that same specification seems odd to me. It opens the door 
> to inconcistencies that will only be resolved once an equivalent RFC3986 
> URL object made its way into the source code.
> 
> On the public API side I would recommend the following:
> 
> - if you are to strictly follow the WHATWG specification no URI 
> component can be null. They must all be strings. If we have to plan to 
> use the same object for RFC3986 compliant parser, then all components 
> should be nullable except for the path component which can never be null 
> as it is always present.

This isn't true. It's just that in the language it is specified in, any element can be null (i.e., no nullable types). It specifies what may be null here: URL Standard (whatwg.org) <https://url.spec.whatwg.org/#url-representation>

— Rob

---

## #124280 — Nicolas Grekas — 2024-07-08T07:51:27+00:00

URL: https://externals.io/message/123997#124280 · In reply to: #124252

Hi Maté,

Fantastic RFC :)

Le dim. 7 juil. 2024 à 11:17, Máté Kocsis <kocsismate90@gmail.com> a écrit :

> Hi Ignace,
>
> As far as I understand it, if this RFC were to pass as is it will model
> > […]
>
>
> Supporting IANA registered schemes is a valid request, and is definitely
> useful.
> However, I think this feature is not strictly required to have in the
> current RFC.
> Anyone we needs to support features that are not offered by the WHATWG
> standard can still rely on `parse_url()`.
>

If I may, parse_url is showing its age and issues like
https://github.com/php/php-src/issues/12703 make it unreliable. We need an
escape plan from it.

FYI, we're discussing whether a Uri component should make it in Symfony
precisely to work around parse_url's issues in
https://github.com/php/php-src/issues/12703
Your RFC would be the perfect answer to this discussion but IRI would need
to be part of it.

I agree with everything Ignace said. Supporting RFC3986 from day-1 would be
absolutely great!

Note that we use parse_url for http-URLs, but also to parse DSNs like
redis://localhost and the likes.


> And of course, we can (and should) add
> support for other standards later. If we wanted to do all these in the same
> RFC, then the scope of the RFC would become way too large IMO. That's why I
> opt for incremental improvements.
>
> Besides, I fail to see why a WHATWG compliant parser wouldn't be useful in
> PHP:
> yes, PHP is server side, but it still interacts with browsers very
> heavily. Among other
> use-cases I cannot yet image, the major one is most likely validating
> user-supplied URLs
> for opening in the browser. As far as I see the situation, currently there
> is no acceptably
> reliable possibility to decide whether a URL can be opened in browsers or
> not.
>
> - parse_url and parse_str predates RFC3986
> > […]
>
> Thank you for the historical context!
>
> Based on your and others' feedback, it has now become clear for me that
> `parse_url()`
> is still useful and ext/url needs quite some additional capabilities until
> this function
> really becomes superfluous. That's why it now seems to me that the
> behavior of
> `parse_url()` could be leveraged in ext/url so that it would work with a
> Url/Url class (e.g.
> we had a PhpUrlParser class extending the Url/UrlParser, or a
> Url\Url::fromPhpParser()
> method, depending on which object model we choose. Of course the names are
> TBD).
>
> For all these arguments I would keep the proposed `Url` free of all
> > […]
>
> My WIP implementation still uses nullable properties and return types. I
> only changed those
> when I wrote the RFC. Since I see that PSR-7 compatibility is very low
> prio for everyone
> involved in the discussion, then I think making these types nullable is
> fine. It was neither my
> top prio, but somewhere I had to start the object design, so I went with
> this.
>
> Again, thank you for your constructive criticism.
>
> Regards,
> Máté

---

## #124292 — Lanre — 2024-07-08T17:24:09+00:00

URL: https://externals.io/message/123997#124292 · In reply to: #123997

> Hi Everyone,
>
> I've been working on a new RFC for a while now, and time has come to
> present it to a wider audience.
>
> Last year, I learnt that PHP doesn't have built-in support for parsing
> URLs according to any well established standards (RFC 1738 or the WHATWG
> URL living standard), since the `parse_url()` function is optimized for
> performance instead of correctness.
>
> In order to improve compatibility with external tools consuming URLs (like
> browsers), my new RFC would add a WHATWG compliant URL parser functionality
> to the standard library. The API itself is not final by any means, the RFC
> only represents how I imagined it first.
>
> You can find the RFC at the following link:
> https://wiki.php.net/rfc/url_parsing_api
>
> Regards,
> Máté
>
>
I was exploring wrapping ada_url for PHP (
https://github.com/lnear-dev/ada-url). It works, but it's a bit slower,
likely due to the implementation of the objects. I was planning to embed
the zvals directly in the object, similar to PhpToken, but I haven't had
the chance and don't really need it anymore. Shouldn't be too much work to
clean it up though

---

Next page: https://externals.io/message/123997.md?page=2
