Hi all,
A few years ago, I posted a message suggesting that PHP improve support
for HTTP/1.1 in its stream wrapper functions:
https://externals.io/message/96192
A quick summary of the current situation:
- HTTP/1.1 was officially standardised in January 1997, and most web
browsers had already implemented it by then - PHP has a very simple HTTP client implementation, used by the "http:"
and "https:" stream wrappers, and also by extensions which make HTTP
requests, such as ext/soap - The client implementation defaults to advertising HTTP/1.0 requests,
unless over-ridden by a stream context option - Since a lot of servers only actually talk HTTP/1.1, the client mostly
acts as an HTTP/1.1 client even when advertising HTTP/1.0
In my previous message, I identified four requirements in HTTP/1.1 but
not HTTP/1.0 that are relevant to a client:
a) Send a "Host" header with every request. (RFC 7230 Section 5.4)
b) Support persistent connections, or send "Connection: Close" with each
request. (RFC 7230 Section 6.1)
c) Ignore 1xx status lines (notably, "100 Continue") "even if the client
does not expect one" (RFC 7231 Section 6.2)
d) Support "chunked" transfer encoding (RFC 7230 Section 4.1)
The PHP client now supports all four regardless of protocol version
configured, i.e. it always sends "Host:" and "Connection: Close"
headers; and always handles "100 Continue" and "Transfer-Encoding:
Chunked" if returned by the server.
I would like to propose that the client advertises HTTP/1.1 in its
requests by default in PHP 8.0. Users can opt out of this behaviour in
a fully backwards- and forwards-compatible way if necessary using a
stream context option, e.g.:
https://gist.github.com/IMSoP/a685fed6589435530102d57138755511
What are people's opinions? Does this need an RFC, or should I just
submit a PR if nobody objects?
Regards,
--
Rowan Tommins (né Collins)
[IMSoP]
Hi all,
A few years ago, I posted a message suggesting that PHP improve support
for HTTP/1.1 in its stream wrapper functions:
https://externals.io/message/96192A quick summary of the current situation:
- HTTP/1.1 was officially standardised in January 1997, and most web
browsers had already implemented it by then- PHP has a very simple HTTP client implementation, used by the "http:"
and "https:" stream wrappers, and also by extensions which make HTTP
requests, such as ext/soap- The client implementation defaults to advertising HTTP/1.0 requests,
unless over-ridden by a stream context option- Since a lot of servers only actually talk HTTP/1.1, the client mostly
acts as an HTTP/1.1 client even when advertising HTTP/1.0In my previous message, I identified four requirements in HTTP/1.1 but
not HTTP/1.0 that are relevant to a client:a) Send a "Host" header with every request. (RFC 7230 Section 5.4)
b) Support persistent connections, or send "Connection: Close" with each
request. (RFC 7230 Section 6.1)
c) Ignore 1xx status lines (notably, "100 Continue") "even if the client
does not expect one" (RFC 7231 Section 6.2)
d) Support "chunked" transfer encoding (RFC 7230 Section 4.1)The PHP client now supports all four regardless of protocol version
configured, i.e. it always sends "Host:" and "Connection: Close"
headers; and always handles "100 Continue" and "Transfer-Encoding:
Chunked" if returned by the server.I would like to propose that the client advertises HTTP/1.1 in its
requests by default in PHP 8.0. Users can opt out of this behaviour in
a fully backwards- and forwards-compatible way if necessary using a
stream context option, e.g.:
https://gist.github.com/IMSoP/a685fed6589435530102d57138755511What are people's opinions? Does this need an RFC, or should I just
submit a PR if nobody objects?Regards,
--
Rowan Tommins (né Collins)
[IMSoP]
This makes complete sense to me. Whether it needs a formal RFC or not is above my pay grade, but I expect it would be just a formality if so. +1
--Larry Garfield
[2nd try, this time with proper etiquette]
Hi Rowan and Larry and others,
I would like to propose that the client advertises HTTP/1.1 in its
requests by default in PHP 8.0. Users can opt out of this behaviour
in a fully backwards- and forwards-compatible way if necessary using a
stream context option, e.g.:
https://gist.github.com/IMSoP/a685fed6589435530102d57138755511What are people's opinions? Does this need an RFC, or should I just
submit a PR if nobody objects?
To me the only question is whether you want HTTP1.1 or 2 to be the
default. version 2 is a bit more complex, to be sure, but quite
powerful in avoiding blocking.
Eliot
On Fri, May 22, 2020 at 12:54 AM Rowan Tommins rowan.collins@gmail.com
wrote:
- PHP has a very simple HTTP client implementation, used by the "http:"
and "https:" stream wrappers, and also by extensions which make HTTP
requests, such as ext/soap
Sorry, I'm an internals noob, but why does PHP implement this at all,
instead of using an external library like cURL which is already fairly
prominent in our ecosystem?
--
Best regards,
Max Semenik
On Thu, May 21, 2020 at 11:54 PM Rowan Tommins rowan.collins@gmail.com
wrote:
Hi all,
A few years ago, I posted a message suggesting that PHP improve support
for HTTP/1.1 in its stream wrapper functions:
https://externals.io/message/96192A quick summary of the current situation:
- HTTP/1.1 was officially standardised in January 1997, and most web
browsers had already implemented it by then- PHP has a very simple HTTP client implementation, used by the "http:"
and "https:" stream wrappers, and also by extensions which make HTTP
requests, such as ext/soap- The client implementation defaults to advertising HTTP/1.0 requests,
unless over-ridden by a stream context option- Since a lot of servers only actually talk HTTP/1.1, the client mostly
acts as an HTTP/1.1 client even when advertising HTTP/1.0In my previous message, I identified four requirements in HTTP/1.1 but
not HTTP/1.0 that are relevant to a client:a) Send a "Host" header with every request. (RFC 7230 Section 5.4)
b) Support persistent connections, or send "Connection: Close" with each
request. (RFC 7230 Section 6.1)
c) Ignore 1xx status lines (notably, "100 Continue") "even if the client
does not expect one" (RFC 7231 Section 6.2)
d) Support "chunked" transfer encoding (RFC 7230 Section 4.1)The PHP client now supports all four regardless of protocol version
configured, i.e. it always sends "Host:" and "Connection: Close"
headers; and always handles "100 Continue" and "Transfer-Encoding:
Chunked" if returned by the server.I would like to propose that the client advertises HTTP/1.1 in its
requests by default in PHP 8.0. Users can opt out of this behaviour in
a fully backwards- and forwards-compatible way if necessary using a
stream context option, e.g.:
https://gist.github.com/IMSoP/a685fed6589435530102d57138755511What are people's opinions? Does this need an RFC, or should I just
submit a PR if nobody objects?
Sounds good to me. Assuming there are no objections, feel free to just send
a PR.
@Eliot: Unfortunately we don't implement HTTP 2 in the http:// stream
wrapper, so HTTP 1.1 is the best we can do there right now. We only provide
HTTP 2 support through the curl extension. Implementing HTTP 2 support
would certainly be a possibility, but I don't think it's particularly easy
to do so without pulling in a dependency like nghttp2.
@Max: I'm only guessing here, because I'm not familiar with the historical
context, but I imagine part of the motivation is to support HTTP requests
in a minimal build of PHP, which does not have any dependencies (that we do
not bundle ourselves).
We did actually provide an implementation of the http:// stream wrapper via
curl for a long time, but dropped it at some point, because there were many
subtle behavior differences to our native implementation.
Regards,
Nikita
dopOn Fri, May 22, 2020 at 3:58 AM Nikita Popov nikita.ppv@gmail.com
wrote:
On Thu, May 21, 2020 at 11:54 PM Rowan Tommins rowan.collins@gmail.com
wrote:Hi all,
A few years ago, I posted a message suggesting that PHP improve support
for HTTP/1.1 in its stream wrapper functions:
https://externals.io/message/96192A quick summary of the current situation:
- HTTP/1.1 was officially standardised in January 1997, and most web
browsers had already implemented it by then- PHP has a very simple HTTP client implementation, used by the "http:"
and "https:" stream wrappers, and also by extensions which make HTTP
requests, such as ext/soap- The client implementation defaults to advertising HTTP/1.0 requests,
unless over-ridden by a stream context option- Since a lot of servers only actually talk HTTP/1.1, the client mostly
acts as an HTTP/1.1 client even when advertising HTTP/1.0In my previous message, I identified four requirements in HTTP/1.1 but
not HTTP/1.0 that are relevant to a client:a) Send a "Host" header with every request. (RFC 7230 Section 5.4)
b) Support persistent connections, or send "Connection: Close" with each
request. (RFC 7230 Section 6.1)
c) Ignore 1xx status lines (notably, "100 Continue") "even if the client
does not expect one" (RFC 7231 Section 6.2)
d) Support "chunked" transfer encoding (RFC 7230 Section 4.1)The PHP client now supports all four regardless of protocol version
configured, i.e. it always sends "Host:" and "Connection: Close"
headers; and always handles "100 Continue" and "Transfer-Encoding:
Chunked" if returned by the server.I would like to propose that the client advertises HTTP/1.1 in its
requests by default in PHP 8.0. Users can opt out of this behaviour in
a fully backwards- and forwards-compatible way if necessary using a
stream context option, e.g.:
https://gist.github.com/IMSoP/a685fed6589435530102d57138755511What are people's opinions? Does this need an RFC, or should I just
submit a PR if nobody objects?Sounds good to me. Assuming there are no objections, feel free to just send
a PR.@Eliot: Unfortunately we don't implement HTTP 2 in the http:// stream
wrapper, so HTTP 1.1 is the best we can do there right now. We only provide
HTTP 2 support through the curl extension. Implementing HTTP 2 support
would certainly be a possibility, but I don't think it's particularly easy
to do so without pulling in a dependency like nghttp2.@Max: I'm only guessing here, because I'm not familiar with the historical
context, but I imagine part of the motivation is to support HTTP requests
in a minimal build of PHP, which does not have any dependencies (that we do
not bundle ourselves).We did actually provide an implementation of the http:// stream wrapper
via
curl for a long time, but dropped it at some point, because there were many
subtle behavior differences to our native implementation.Regards,
Nikita
This is ridiculously timely as I've been spending my evening working on
HTTP/2 stuff in PHP.
This might be a good time to reopen discussion of adding support for HTTP/2
in PHP with the inclusion of libnghttp2. I posted a long time ago about
adding support for HTTP/2 to the CLI server and the http stream using
libnghttp2 [1].
I think PHP 8.0 would be a perfect opportunity to revisit this given that
HTTP/2 has now reached ~97% adoption[2] and HTTP/3 is on the horizon.
I believe that HTTP/2 has the potential to dramatically change how we serve
content on the web, and PHP should jump on the bandwagon.
- Davey
[1] https://externals.io/message/89932
[2] https://caniuse.com/#search=http%2F2
This is ridiculously timely as I've been spending my evening working
on HTTP/2 stuff in PHP.
[...]
I believe that HTTP/2 has the potential to dramatically change how we
serve content on the web, and PHP should jump on the bandwagon.
Hi Davey,
I'm glad someone's working on this, because PHP should definitely be
trying to keep up with such things.
My only hesitation is that other than as a proof-of-concept, just
allowing 'protocol_version'=>2 as a stream option isn't going to bring
much advantage, because it doesn't give access to any of the interesting
new features - it's still a single, one-way, synchronous request.
Given it's always been a web programming language, PHP has surprisingly
poor native support for HTTP. On the server side, we have the SAPI
layer, which delegates most of the protocol to external applications in
generally CGI-type ways. On the client side, we have the very minimal
stream wrapper (which doesn't even support pipelining with
Connection:Keep-Alive), or some painfully low-level bindings around libcurl.
To really make use of HTTP/2, I would imagine both would need quite
significant re-thinks: On the server side, can the SAPI and FastCGI
definitions be extended, or do they need to be replaced, as Python
defined ASGI [1] to replace WSGI? On the client side, should there be a
better built-in HTTP client, whether based on libcurl or lower-level
pieces like nghttp; or do we keep it low-level and expose nghttp2 itself
in some way?
[1] https://asgi.readthedocs.io/en/latest/introduction.html
Regards,
--
Rowan Tommins (né Collins)
[IMSoP]
Hi all,
A couple of months ago I revived my suggestion to make stream wrappers
advertise HTTP/1.1 by default (full message below), and then Real Life
intervened and I didn't raise the PR.
The implementation (which is a one-character change) is now here:
https://github.com/php/php-src/pull/5899
I realise it's now rather last-minute for 8.0, so if anyone thinks it
should have more careful thought, or an RFC, it will have to wait. If
there are no objections, though, it would be nice to have it merged
before the freeze.
Thanks,
Rowan
Hi all,
A few years ago, I posted a message suggesting that PHP improve
support for HTTP/1.1 in its stream wrapper functions:
https://externals.io/message/96192A quick summary of the current situation:
- HTTP/1.1 was officially standardised in January 1997, and most web
browsers had already implemented it by then- PHP has a very simple HTTP client implementation, used by the
"http:" and "https:" stream wrappers, and also by extensions which
make HTTP requests, such as ext/soap- The client implementation defaults to advertising HTTP/1.0 requests,
unless over-ridden by a stream context option- Since a lot of servers only actually talk HTTP/1.1, the client
mostly acts as an HTTP/1.1 client even when advertising HTTP/1.0In my previous message, I identified four requirements in HTTP/1.1 but
not HTTP/1.0 that are relevant to a client:a) Send a "Host" header with every request. (RFC 7230 Section 5.4)
b) Support persistent connections, or send "Connection: Close" with
each request. (RFC 7230 Section 6.1)
c) Ignore 1xx status lines (notably, "100 Continue") "even if the
client does not expect one" (RFC 7231 Section 6.2)
d) Support "chunked" transfer encoding (RFC 7230 Section 4.1)The PHP client now supports all four regardless of protocol version
configured, i.e. it always sends "Host:" and "Connection: Close"
headers; and always handles "100 Continue" and "Transfer-Encoding:
Chunked" if returned by the server.I would like to propose that the client advertises HTTP/1.1 in its
requests by default in PHP 8.0. Users can opt out of this behaviour
in a fully backwards- and forwards-compatible way if necessary using a
stream context option, e.g.:
https://gist.github.com/IMSoP/a685fed6589435530102d57138755511What are people's opinions? Does this need an RFC, or should I just
submit a PR if nobody objects?Regards,
--
Rowan Tommins (né Collins)
[IMSoP]
On Mon, Jul 27, 2020 at 5:25 PM Rowan Tommins rowan.collins@gmail.com
wrote:
A couple of months ago I revived my suggestion to make stream wrappers
advertise HTTP/1.1 by default (full message below), and then Real Life
intervened and I didn't raise the PR.The implementation (which is a one-character change) is now here:
https://github.com/php/php-src/pull/5899I realise it's now rather last-minute for 8.0, so if anyone thinks it
should have more careful thought, or an RFC, it will have to wait. If
there are no objections, though, it would be nice to have it merged
before the freeze.Given that it's a very small change, the RFC is probably not necessary, in
which case it's not too late, however I'd like some clarification about
what this actually offers over defaulting to 1.0. Specifically, are there
known servers which will refuse to serve 1.0 clients? Script authors have
had the option of opting into 1.1 for some significant time now.
Basically, what's the chance of something accidentally not working due to
advertising 1.0 versus something not working due to advertising 1.1? If
the changes are equal, I'd personally err in favor of status quo since
those needing 1.1 have already opted into it, those needing 1.0 may not
have.
-Sara
Hi Sara,
Given that it's a very small change, the RFC is probably not necessary, in
which case it's not too late, however I'd like some clarification about
what this actually offers over defaulting to 1.0.
That's a very reasonable question. The way I see it is this:
The risk of advertising 1.0 by default is that some software will have been
programmed to outright refuse that protocol version. I don't know of any
recent examples, but this bug report from 2007 was for a SOAP endpoint that
returned 505 Version Not Supported: https://bugs.php.net/bug.php?id=43069
Notably, Dmitry's patch for that bug made sure the protocol context option
is honoured by ext/soap, but also specifically defaulted it to HTTP/1.1 as
of PHP 5.3.
The risk of advertising 1.1 by default is that some software will respond
with a more complex response, and trigger some bug in our response parsing.
This was previously the case with detecting "Transfer-Encoding: chunked"
headers, for instance. By advertising 1.0, we may be benefitting from
servers "downgrading" their response.
In practice, a large amount of software appears to do neither, and simply
replies with an HTTP/1.1 response to a HTTP/1.0 request. This is why I talk
about "advertising" versions - in practice, the code is always acting as an
HTTP/1.1 client, e.g. sending "Connection: Close", and listening for
Transfer-Encoding, because it's increasingly rare for a server to actually
honour the 1.0 spec.
My main motivation for the change is that if someone was writing the
feature today, I don't think it would occur to them to default to 1.0, and
I think new users would be less surprised at needing to opt into 1.0 than
into 1.1.
Regards,
Rowan Tommins
[IMSoP]
Hi Rowan,
Den 2020-07-28 kl. 10:52, skrev Rowan Tommins:
Hi Sara,
Given that it's a very small change, the RFC is probably not necessary, in
which case it's not too late, however I'd like some clarification about
what this actually offers over defaulting to 1.0.That's a very reasonable question. The way I see it is this:
The risk of advertising 1.0 by default is that some software will have been
programmed to outright refuse that protocol version. I don't know of any
recent examples, but this bug report from 2007 was for a SOAP endpoint that
returned 505 Version Not Supported: https://bugs.php.net/bug.php?id=43069
Notably, Dmitry's patch for that bug made sure the protocol context option
is honoured by ext/soap, but also specifically defaulted it to HTTP/1.1 as
of PHP 5.3.The risk of advertising 1.1 by default is that some software will respond
with a more complex response, and trigger some bug in our response parsing.
This was previously the case with detecting "Transfer-Encoding: chunked"
headers, for instance. By advertising 1.0, we may be benefitting from
servers "downgrading" their response.In practice, a large amount of software appears to do neither, and simply
replies with an HTTP/1.1 response to a HTTP/1.0 request. This is why I talk
about "advertising" versions - in practice, the code is always acting as an
HTTP/1.1 client, e.g. sending "Connection: Close", and listening for
Transfer-Encoding, because it's increasingly rare for a server to actually
honour the 1.0 spec.My main motivation for the change is that if someone was writing the
feature today, I don't think it would occur to them to default to 1.0, and
I think new users would be less surprised at needing to opt into 1.0 than
into 1.1.Regards,
I think there is also a consistency argument to make here. If the code
behaves like HTTP 1.1, I think it also should advertise it as such. So go
for it!
Regards //Björn L
On Tue, Jul 28, 2020 at 3:52 AM Rowan Tommins rowan.collins@gmail.com
wrote:
The risk of advertising 1.0 by default is that some software will have been
programmed to outright refuse that protocol version. I don't know of any
recent examples, but this bug report from 2007 was for a SOAP endpoint that
returned 505 Version Not Supported: https://bugs.php.net/bug.php?id=43069
Notably, Dmitry's patch for that bug made sure the protocol context option
is honoured by ext/soap, but also specifically defaulted it to HTTP/1.1 as
of PHP 5.3.The risk of advertising 1.1 by default is that some software will respond
with a more complex response, and trigger some bug in our response parsing.
This was previously the case with detecting "Transfer-Encoding: chunked"
headers, for instance. By advertising 1.0, we may be benefitting from
servers "downgrading" their response.In practice, a large amount of software appears to do neither, and simply
replies with an HTTP/1.1 response to a HTTP/1.0 request. This is why I talk
about "advertising" versions - in practice, the code is always acting as an
HTTP/1.1 client, e.g. sending "Connection: Close", and listening for
Transfer-Encoding, because it's increasingly rare for a server to actually
honour the 1.0 spec.The "servers have a bad habit of ignoring this anyway" argument is
probably true, but it sort of takes the wind out of BOTH of the other two
points. :)
I'm not going to object to merging it, the worst case scenario is that a
few app developers will learn the protocol version option exists and have
to set "1.0" manually. Even that is a low probability. Probably better in
the long-term to inch our way forward and advertise a protocol that was
introduced... checks notes... 21 years ago.
-Sara
Hey all,
Given that it's a very small change, the RFC is probably not necessary,
in
which case it's not too late, however I'd like some clarification about
what this actually offers over defaulting to 1.0.
One thing it offers is detecting truncated responses. Servers will often
respond without transfer-encoding: chunked / content-length for an HTTP/1.0
request, which leaves connection closure as only indication where the
response ends.
As we progress to HTTP/3 already, newer implementations might not implement
1.0 at all. If we're compatible with 1.1 already, there's no reason IMO not
to advertise and use its features.
That's a very reasonable question. The way I see it is this:
The risk of advertising 1.0 by default is that some software will have been
programmed to outright refuse that protocol version. I don't know of any
recent examples, but this bug report from 2007 was for a SOAP endpoint that
returned 505 Version Not Supported: https://bugs.php.net/bug.php?id=43069
Notably, Dmitry's patch for that bug made sure the protocol context option
is honoured by ext/soap, but also specifically defaulted it to HTTP/1.1 as
of PHP 5.3.The risk of advertising 1.1 by default is that some software will respond
with a more complex response, and trigger some bug in our response parsing.
This was previously the case with detecting "Transfer-Encoding: chunked"
headers, for instance. By advertising 1.0, we may be benefitting from
servers "downgrading" their response.
Do we handle 1XX responses, yet?
https://tools.ietf.org/html/rfc7231#section-6.2
In practice, a large amount of software appears to do neither, and simply
replies with an HTTP/1.1 response to a HTTP/1.0 request. This is why I talk
about "advertising" versions - in practice, the code is always acting as an
HTTP/1.1 client, e.g. sending "Connection: Close", and listening for
Transfer-Encoding, because it's increasingly rare for a server to actually
honour the 1.0 spec.
This is what implementations should do, see
https://tools.ietf.org/html/rfc7230#section-2.6
That section also says we should send 1.1, as we support it.
Best,
Niklas
My main motivation for the change is that if someone was writing the
feature today, I don't think it would occur to them to default to 1.0, and
I think new users would be less surprised at needing to opt into 1.0 than
into 1.1.Regards,
Rowan Tommins
[IMSoP]
Do we handle 1XX responses, yet?
https://tools.ietf.org/html/rfc7231#section-6.2
Yes, as of this patch a few years back:
https://github.com/php/php-src/pull/2175/files
This is what implementations should do, see
https://tools.ietf.org/html/rfc7230#section-2.6
I guess that's part of the "generous in what you accept" philosophy,
since it makes no sense for a server to send a response chunked if the
client didn't advertise 1.1 capabilities.
That section also says we should send 1.1, as we support it.
Indeed, as far as I have been able to tell, the client code now complies
with the spec for HTTP 1.1, although I struggled to find a list of
minimum capabilities.
Regards,
--
Rowan Tommins (né Collins)
[IMSoP]
Hi Rowan,
d) Support "chunked" transfer encoding (RFC 7230 Section 4.1)
FWIW, I've just been hit by https://bugs.php.net/bug.php?id=47021 in a
SOAP response from a server using "Transfer-Encoding:<two
spaces>chunked". I see the bug is still open, although the comments to
the PR hint to the fact that it should have been fixed.
This was on PHP 7.3.17, I'll see if I can build a self-contained test
case and reproduce with the latest from git.
Cheers
Matteo Beccati
Development & Consulting - http://www.beccati.com/
Hi Rowan,
d) Support "chunked" transfer encoding (RFC 7230 Section 4.1)
FWIW, I've just been hit by https://bugs.php.net/bug.php?id=47021 in a
SOAP response from a server using "Transfer-Encoding:<two spaces>chunked". I see the bug is still open, although the comments to
the PR hint to the fact that it should have been fixed.This was on PHP 7.3.17, I'll see if I can build a self-contained test
case and reproduce with the latest from git.
Hi Matteo,
Thanks for the heads up; if you can manage to get a test case, I'll have
another look over it as soon as I have time.
Regards,
Rowan Tommins
[IMSoP]
Hi Rowan,
FWIW, I've just been hit by https://bugs.php.net/bug.php?id=47021 in a
SOAP response from a server using "Transfer-Encoding:<two spaces>chunked". I see the bug is still open, although the comments to
the PR hint to the fact that it should have been fixed.Thanks for the heads up; if you can manage to get a test case, I'll have
another look over it as soon as I have time.
I've just pushed what I think is the fix to the last remaining instance
of whitespace issues with HTTP headers, this time when parsing the SOAP
response. The fix is fairly similar to the one you made in the http
fopen wrapper. SOAP tests are passing and valgrind doesn't complain, but
an extra set of eyes on my commit would be appreciated.
The commit also close the bug report, which I think is good anyway.
Cheers
Matteo Beccati
Development & Consulting - http://www.beccati.com/