Hi internallers!
No, you haven't misread the subject line, I'm not talking about the
exciting new HTTP/2, but HTTP/1.1, 20 years old this year.
To my surprise, PHP's HTTP stream wrapper (e.g.
file_get_contents('http://example.com');) defaults to sending HTTP/1.0
requests. You can tell it to use HTTP/1.1 with an appropriate "stream
context", but it's fiddly [1], and the implementation is incomplete.
There's actually already an exception to this: some SOAP servers were
discovered to only talk HTTP/1.1, so ext/soap forces the wrapper to
upgrade by default [bug-43069]. This in turn revealed some of the bugs
in the wrapper's protocol support, one of which I have recently created
a patch for [bug-47021].
It's reasonable to suppose that 9 years on from that bug those SOAP
servers aren't alone in rejecting (or mishandling) HTTP/1.0 requests,
and moves to implement HTTP/2.0 will only make that more likely.
I think it would be good to get this support into a better state, make
it easier to switch on - e.g. with an INI setting, or some new stream
URL syntax - and possibly make HTTP/1.1 the default in PHP 8.0.
As I understand it, supporting HTTP/1.1 as a client requires the
following mandatory features on top of HTTP/1.0:
a) Send a "Host" header with every request. (RFC 7230 Section 5.4)
b) Support persistent connections, or send "Connection: Close" with each
request. (RFC 7230 Section 6.1)
c) Ignore 1xx status lines (notably, "100 Continue") "even if the client
does not expect one" (RFC 7231 Section 6.2)
d) Support "chunked" transfer encoding (RFC 7230 Section 4.1)
Let me know if there are any I've missed.
I believe this is the current state of the PHP implementation:
a) Host header always sent regardless of protocol version.
b) The SOAP wrapper explicitly generates a "Connection: Close" header as
well as forcing HTTP/1.1; users need to set it as a further option in
stream_context_create(). It would be better if this was implicit with
the protocol version.
c) I can't find a bug report for this, but 100 Continue appears to be
treated as an error status. This should be fixed.
d) Chunked encoding itself was implemented, but the code for parsing
headers was broken. If my PR [pr-1902] is approved, this will hopefully
work correctly.
What do people think? Would this be a worthwhile effort?
[1]: Sample HTTP/1.1 context
https://gist.github.com/IMSoP/8c62c97afc8765f418468b28a501409e
[bug-43069]: https://bugs.php.net/bug.php?id=43069
[bug-47021]: https://bugs.php.net/bug.php?id=47021
[pr-1902]: https://github.com/php/php-src/pull/1902
Regards,
--
Rowan Collins
[IMSoP]
Rowan Collins rowan.collins@gmail.com schrieb am So., 2. Okt. 2016, 23:12:
Hi internallers!
No, you haven't misread the subject line, I'm not talking about the
exciting new HTTP/2, but HTTP/1.1, 20 years old this year.To my surprise, PHP's HTTP stream wrapper (e.g.
file_get_contents('http://example.com');) defaults to sending HTTP/1.0
requests. You can tell it to use HTTP/1.1 with an appropriate "stream
context", but it's fiddly [1], and the implementation is incomplete.There's actually already an exception to this: some SOAP servers were
discovered to only talk HTTP/1.1, so ext/soap forces the wrapper to
upgrade by default [bug-43069]. This in turn revealed some of the bugs
in the wrapper's protocol support, one of which I have recently created
a patch for [bug-47021].It's reasonable to suppose that 9 years on from that bug those SOAP
servers aren't alone in rejecting (or mishandling) HTTP/1.0 requests,
and moves to implement HTTP/2.0 will only make that more likely.I think it would be good to get this support into a better state, make
it easier to switch on - e.g. with an INI setting, or some new stream
URL syntax - and possibly make HTTP/1.1 the default in PHP 8.0.As I understand it, supporting HTTP/1.1 as a client requires the
following mandatory features on top of HTTP/1.0:a) Send a "Host" header with every request. (RFC 7230 Section 5.4)
b) Support persistent connections, or send "Connection: Close" with each
request. (RFC 7230 Section 6.1)
c) Ignore 1xx status lines (notably, "100 Continue") "even if the client
does not expect one" (RFC 7231 Section 6.2)
d) Support "chunked" transfer encoding (RFC 7230 Section 4.1)Let me know if there are any I've missed.
I believe this is the current state of the PHP implementation:
a) Host header always sent regardless of protocol version.
b) The SOAP wrapper explicitly generates a "Connection: Close" header as
well as forcing HTTP/1.1; users need to set it as a further option in
stream_context_create(). It would be better if this was implicit with
the protocol version.
I think we already send it starting with 7.0, maybe just for 1.0, because
there are some servers that respond correctly with an HTTP/1.1 response to
an HTTP/1.0 request, but fail to close the connection and imply
"Connection: close" for those requests.
c) I can't find a bug report for this, but 100 Continue appears to be
treated as an error status. This should be fixed.
d) Chunked encoding itself was implemented, but the code for parsing
headers was broken. If my PR [pr-1902] is approved, this will hopefully
work correctly.What do people think? Would this be a worthwhile effort?
[1]: Sample HTTP/1.1 context
https://gist.github.com/IMSoP/8c62c97afc8765f418468b28a501409e
[bug-43069]: https://bugs.php.net/bug.php?id=43069
[bug-47021]: https://bugs.php.net/bug.php?id=47021
[pr-1902]: https://github.com/php/php-src/pull/1902Regards,
--
Rowan Collins
[IMSoP]
Hi all,
It sounds like the feeling so far is quite positive. I'll have another
look for a dupe, then file a bug for the missing 100-continue support,
and try to work on a patch. Once that's in place, and unless anyone
comes up with other major features we need to support, we can discuss
whether we should make it the default, or at least easier to opt into.
On 03/10/2016 09:43, Niklas Keller wrote ...:
I think we already send it starting with 7.0, maybe just for 1.0,
because there are some servers that respond correctly with an HTTP/1.1
response to an HTTP/1.0 request, but fail to close the connection and
imply "Connection: close" for those requests.
... and on 05/10/2016 17:14, Andrea Faulds wrote:
In order to connect to many modern websites, you need a Host: header,
and in order to use a Host: header, you need to use HTTP/1.1 (if you
don't, servers will sometimes send you back a 1.1 response anyway!).
Indeed, checking 7.0 suggests that we are unconditionally sending both a
Host: header and Connection: Close with every request, while still
claiming "HTTP/1.0". I believe both headers are technically backwards
compatible with servers (i.e. ignored by) that genuinely speak HTTP/1.0,
so this is not technically "wrong". However, the fact that it's
necessary does strengthen the case for implementing HTTP/1.1 in full,
rather than backporting features bit by bit.
Would fixing
this behaviour also be applicable for HTTPS?
Yes, I'd be very surprised if this code wasn't shared between the two
contexts. (It's a little more complicated than just piping an "http"
connection through a "tls" one, because of things like SNI, but there'd
be no reason to have the actual header processing code twice.)
Regards,
--
Rowan Collins
[IMSoP]
Hi Rowan
Hi all,
It sounds like the feeling so far is quite positive. I'll have another
look for a dupe, then file a bug for the missing 100-continue support, and
try to work on a patch. Once that's in place, and unless anyone comes up
with other major features we need to support, we can discuss whether we
should make it the default, or at least easier to opt into.On 03/10/2016 09:43, Niklas Keller wrote ...:
I think we already send it starting with 7.0, maybe just for 1.0,
because there are some servers that respond correctly with an HTTP/1.1
response to an HTTP/1.0 request, but fail to close the connection and imply
"Connection: close" for those requests.... and on 05/10/2016 17:14, Andrea Faulds wrote:
In order to connect to many modern websites, you need a Host: header,
and in order to use a Host: header, you need to use HTTP/1.1 (if you don't,
servers will sometimes send you back a 1.1 response anyway!).Indeed, checking 7.0 suggests that we are unconditionally sending both a
Host: header and Connection: Close with every request, while still claiming
"HTTP/1.0". I believe both headers are technically backwards compatible
with servers (i.e. ignored by) that genuinely speak HTTP/1.0, so this is
not technically "wrong". However, the fact that it's necessary does
strengthen the case for implementing HTTP/1.1 in full, rather than
backporting features bit by bit.Would fixing
this behaviour also be applicable for HTTPS?Yes, I'd be very surprised if this code wasn't shared between the two
contexts. (It's a little more complicated than just piping an "http"
connection through a "tls" one, because of things like SNI, but there'd be
no reason to have the actual header processing code twice.)
Afair then the code is separate in ext/standard/fopen_http_wrapper.c
Regards,
--
Rowan Collins
[IMSoP]
Den 2016-10-02 kl. 23:12, skrev Rowan Collins:
Hi internallers!
No, you haven't misread the subject line, I'm not talking about the
exciting new HTTP/2, but HTTP/1.1, 20 years old this year.To my surprise, PHP's HTTP stream wrapper (e.g.
file_get_contents('http://example.com');) defaults to sending HTTP/1.0
requests. You can tell it to use HTTP/1.1 with an appropriate "stream
context", but it's fiddly [1], and the implementation is incomplete.There's actually already an exception to this: some SOAP servers were
discovered to only talk HTTP/1.1, so ext/soap forces the wrapper to
upgrade by default [bug-43069]. This in turn revealed some of the bugs
in the wrapper's protocol support, one of which I have recently
created a patch for [bug-47021].It's reasonable to suppose that 9 years on from that bug those SOAP
servers aren't alone in rejecting (or mishandling) HTTP/1.0 requests,
and moves to implement HTTP/2.0 will only make that more likely.I think it would be good to get this support into a better state, make
it easier to switch on - e.g. with an INI setting, or some new stream
URL syntax - and possibly make HTTP/1.1 the default in PHP 8.0.As I understand it, supporting HTTP/1.1 as a client requires the
following mandatory features on top of HTTP/1.0:a) Send a "Host" header with every request. (RFC 7230 Section 5.4)
b) Support persistent connections, or send "Connection: Close" with
each request. (RFC 7230 Section 6.1)
c) Ignore 1xx status lines (notably, "100 Continue") "even if the
client does not expect one" (RFC 7231 Section 6.2)
d) Support "chunked" transfer encoding (RFC 7230 Section 4.1)Let me know if there are any I've missed.
I believe this is the current state of the PHP implementation:
a) Host header always sent regardless of protocol version.
b) The SOAP wrapper explicitly generates a "Connection: Close" header
as well as forcing HTTP/1.1; users need to set it as a further option
instream_context_create(). It would be better if this was implicit
with the protocol version.
c) I can't find a bug report for this, but 100 Continue appears to be
treated as an error status. This should be fixed.
d) Chunked encoding itself was implemented, but the code for parsing
headers was broken. If my PR [pr-1902] is approved, this will
hopefully work correctly.What do people think? Would this be a worthwhile effort?
[1]: Sample HTTP/1.1 context
https://gist.github.com/IMSoP/8c62c97afc8765f418468b28a501409e
[bug-43069]: https://bugs.php.net/bug.php?id=43069
[bug-47021]: https://bugs.php.net/bug.php?id=47021
[pr-1902]: https://github.com/php/php-src/pull/1902Regards,
Yup, personally I do find this a worthwhile effort! Would fixing
this behaviour also be applicable for HTTPS?
Regards //Björn Larsson
Hi Rowan,
Rowan Collins wrote:
I think it would be good to get this support into a better state, make
it easier to switch on - e.g. with an INI setting, or some new stream
URL syntax - and possibly make HTTP/1.1 the default in PHP 8.0.
HTTP/1.1 ought to be the default now, I think. The lack of the Host:
header in HTTP/1.0 is pretty serious given how much of the web runs on
vhosts now.
As I understand it, supporting HTTP/1.1 as a client requires the
following mandatory features on top of HTTP/1.0:a) Send a "Host" header with every request. (RFC 7230 Section 5.4)
b) Support persistent connections, or send "Connection: Close" with each
request. (RFC 7230 Section 6.1)
c) Ignore 1xx status lines (notably, "100 Continue") "even if the client
does not expect one" (RFC 7231 Section 6.2)
d) Support "chunked" transfer encoding (RFC 7230 Section 4.1)Let me know if there are any I've missed.
I don't know about ones you've missed, but the Host header, persistent
connections and chunked transfer were the main ones that caught me out
when I wrote my own minimal HTTP client a while ago. In order to connect
to many modern websites, you need a Host: header, and in order to use a
Host: header, you need to use HTTP/1.1 (if you don't, servers will
sometimes send you back a 1.1 response anyway!). In order to understand
the output of those servers, you may need chunked transfer encoding,
particularly if they're running dynamic code (say, PHP), and you need to
specify the connection is non-persistent, lest you wait forever.
What do people think? Would this be a worthwhile effort?
I think it's the absolute minimum we need to do to have it work properly
at all. Using HTTP/1.0 alone probably means PHP's HTTP stream wrapper
support is effectively broken at present, if my experiences are anything
to go by.
Thanks for bringing this up!
--
Andrea Faulds
https://ajf.me/