# [PATCH] Add option to disable POST data processing - Thread: https://externals.io/message/50885 - Original archive: https://news-web.php.net/php.internals/50885 - Started by Gustavo Lopes on 2010-12-07T07:08:34+00:00 - Messages: 45 - Page 1 of 2: messages 1 to 25 (25 per page, add `?page=N` to the URL) - Next page: https://externals.io/message/50885.md?page=2 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #50885 — Gustavo Lopes — 2010-12-07T07:08:34+00:00 URL: https://externals.io/message/50885#50885 The very simple attached patch adds an option to disable POST data processing, which implies the data can only be read in a stream fashion through php://input. As far as I know, PHP offers no way to inhibit processing RFC 1867 data and one has to use very hacky means to accomplish that. This is often required (or at least convenient) in order to, e.g., proxy requests or handle file uploads in memory. For other types of requests, the default processing of POST data may also be a problem. Take a non-application/x-www-form-urlencoded POST requests (say, some kind of RPC with a big XML payload) -- PHP is very memory inefficient as it will hold the whole POST data into memory and duplicate it twice (from SG(request_info).post_data to $HTTP_RAW_POST_DATA -- even if always_populate_raw_post_data=0 -- and SG(request_info).raw_post_data). This introduces a new ini setting, disable_post_data_processing, but it's a benign one. No incompatibilities between setups will arise because no one will enable it globally (it would be insane), only selectively to the scripts that require it. The reason for an ini setting is that it must be set early in the request life. Thoughts? -- Gustavo Lopes --- ## #50888 — Ferenc Kovacs — 2010-12-07T08:25:26+00:00 URL: https://externals.io/message/50885#50888 · In reply to: #50885 On Tue, Dec 7, 2010 at 8:08 AM, Gustavo Lopes wrote: > The very simple attached patch adds an option to disable POST data > processing, which implies the data can only be read in a stream fashion > through php://input. > > As far as I know, PHP offers no way to inhibit processing RFC 1867 data and > one has to use very hacky means to accomplish that. This is often required > (or at least convenient) in order to, e.g., proxy requests or handle file > uploads in memory. > > For other types of requests, the default processing of POST data may also > be a problem. Take a non-application/x-www-form-urlencoded POST requests > (say, some kind of RPC with a big XML payload) -- PHP is very memory > inefficient as it will hold the whole POST data into memory and duplicate it > twice (from SG(request_info).post_data to $HTTP_RAW_POST_DATA -- even if > always_populate_raw_post_data=0 -- and SG(request_info).raw_post_data). > > This introduces a new ini setting, disable_post_data_processing, but it's a > benign one. No incompatibilities between setups will arise because no one > will enable it globally (it would be insane), only selectively to the > scripts that require it. The reason for an ini setting is that it must be > set early in the request life. > > Thoughts? > > I like the idea. Tyrael --- ## #50889 — Michael Wallner — 2010-12-07T10:27:48+00:00 URL: https://externals.io/message/50885#50889 · In reply to: #50885 > The very simple attached patch adds an option to disable POST data > processing, which implies the data can only be read in a stream > fashion through php://input. > > This introduces a new ini setting, disable_post_data_processing, but > it's a benign one. No incompatibilities between setups will arise > because no one will enable it globally (it would be insane), only > selectively to the scripts that require it. The reason for an ini > setting is that it must be set early in the request life. > > Thoughts? > Definitely +1. --- ## #50890 — Richard Quadling — 2010-12-07T11:40:14+00:00 URL: https://externals.io/message/50885#50890 · In reply to: #50885 > The very simple attached patch adds an option to disable POST data > processing, which implies the data can only be read in a stream fashion > through php://input. > > As far as I know, PHP offers no way to inhibit processing RFC 1867 data and > one has to use very hacky means to accomplish that. This is often required > (or at least convenient) in order to, e.g., proxy requests or handle file > uploads in memory. > > For other types of requests, the default processing of POST data may also be > a problem. Take a non-application/x-www-form-urlencoded POST requests (say, > some kind of RPC with a big XML payload) -- PHP is very memory inefficient > as it will hold the whole POST data into memory and duplicate it twice (from > SG(request_info).post_data to $HTTP_RAW_POST_DATA -- even if > always_populate_raw_post_data=0 -- and SG(request_info).raw_post_data). > > This introduces a new ini setting, disable_post_data_processing, but it's a > benign one. No incompatibilities between setups will arise because no one > will enable it globally (it would be insane), only selectively to the > scripts that require it. The reason for an ini setting is that it must be > set early in the request life. > > Thoughts? > > -- > Gustavo Lopes > -- > > > As I understand things, the super globals are already populated by the time the script starts execution. So, `ini_set()` will have no impact. Can you set an ini option for a single script via some other method? -- Richard Quadling Twitter : EE : Zend @RQuadling : e-e.com/M_248814.html : bit.ly/9O8vFY --- ## #50892 — Derick Rethans — 2010-12-07T12:01:59+00:00 URL: https://externals.io/message/50885#50892 · In reply to: #50885 > The very simple attached patch adds an option to disable POST data > processing, which implies the data can only be read in > a stream fashion through php://input. Does without this patch, reading POST data from php://input work? > This introduces a new ini setting, disable_post_data_processing, but > it's a benign one. No incompatibilities between setups will arise > because no one will enable it globally (it would be insane), Because they can, people will do it though... I'm not fan of more ini settings. Derick --- ## #50894 — John Mertic — 2010-12-07T12:11:16+00:00 URL: https://externals.io/message/50885#50894 · In reply to: #50890 > > > […] > > As I understand things, the super globals are already populated by the > time the script starts execution. > > So, `ini_set()` will have no impact. > > Can you set an ini option for a single script via some other method? > Maybe thru an .htaccess file? That should work. Otherwise, +1 for the patch from me. John Mertic jmertic@gmail.com | http://jmertic.wordpress.com --- ## #50895 — Richard Quadling — 2010-12-07T12:37:11+00:00 URL: https://externals.io/message/50885#50895 · In reply to: #50894 > > > […] > > Maybe thru an .htaccess file? That should work. > > Otherwise, +1 for the patch from me. > > John Mertic > jmertic@gmail.com | http://jmertic.wordpress.com > Can you set an ini option via .htaccess for a single script? If the script has to reside in its own directory then that makes sense, but I don't know how to set an ini option for a single script. -- Richard Quadling Twitter : EE : Zend @RQuadling : e-e.com/M_248814.html : bit.ly/9O8vFY --- ## #50896 — Ferenc Kovacs — 2010-12-07T12:37:35+00:00 URL: https://externals.io/message/50885#50896 · In reply to: #50892 > > > > […] > > Does without this patch, reading POST data from php://input work? > yep. you will get the raw post data with this. same as $HTTP_RAW_POST_DATA the only difference, that the latter isn't always available. > > > […] > > Because they can, people will do it though... I'm not fan of more ini > settings. > so because in the past ini settings was abused,we won't add ini settings ever? Tyrael --- ## #50897 — John Mertic — 2010-12-07T12:42:06+00:00 URL: https://externals.io/message/50885#50897 > > > […] Wouldn't something like this work? php_value disable_post_data_processing On John Mertic jmertic@gmail.com | http://jmertic.wordpress.com --- ## #50898 — Richard Quadling — 2010-12-07T12:48:38+00:00 URL: https://externals.io/message/50885#50898 · In reply to: #50897 > > […] > > Wouldn't something like this work? > > > php_value disable_post_data_processing On > http://httpd.apache.org/docs/2.0/mod/core.html#files looks like it should. Now to find out how to do that for IIS. -- Richard Quadling Twitter : EE : Zend @RQuadling : e-e.com/M_248814.html : bit.ly/9O8vFY --- ## #50899 — Gustavo Lopes — 2010-12-07T13:07:42+00:00 URL: https://externals.io/message/50885#50899 · In reply to: #50892 > > > > […] > > Does without this patch, reading POST data from php://input work? Maybe. In case of a multipart/form-data request, it won't. For other requests, it will. However, php://input will be reading from memory, not from the SAPI, because the entire POST data will be in memory. The new option makes the behavior for a POST request similar to that of e.g. a PUT request, i.e., data can only be read with php://input and it will *not* be copied entirely into memory before the user code executes. Of course the user can use something like file_get_contents('php://input') to do that, but, depending on the application, he may take advantage of the low memory usage a stream provides (for the XML payload example, he could use a SAX-like parser, for proxying or saving the entire request into disk -- possibly filtered -- he could use stream_copy_to_stream, etc.). > > > […] > > Because they can, people will do it though... I'm not fan of more ini > settings. > I'm sure someone somewhere will enable it eventually without being aware of what it does (despite the obvious name...); my point is that programmers don't need to program against it (unlike say, magic quotes) because no working PHP setup will have it enabled, as it breaks every script that handles POST data. -- Gustavo Lopes --- ## #50900 — Pierre Joye — 2010-12-07T14:55:38+00:00 URL: https://externals.io/message/50885#50900 · In reply to: #50898 > > […] > > http://httpd.apache.org/docs/2.0/mod/core.html#files > > looks like it should. > > Now to find out how to do that for IIS. Add support for files in the .user.ini and [PATH=] support :) -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- ## #50901 — nicobn@gmail.com — 2010-12-07T16:05:42+00:00 URL: https://externals.io/message/50885#50901 · In reply to: #50885 Hi, On Tue, Dec 7, 2010 at 2:08 AM, Gustavo Lopes wrote: > The very simple attached patch adds an option to disable POST data > processing, which implies the data can only be read in a stream fashion > through php://input. > > As far as I know, PHP offers no way to inhibit processing RFC 1867 data and > one has to use very hacky means to accomplish that. This is often required > (or at least convenient) in order to, e.g., proxy requests or handle file > uploads in memory. > > For other types of requests, the default processing of POST data may also > be a problem. Take a non-application/x-www-form-urlencoded POST requests > (say, some kind of RPC with a big XML payload) -- PHP is very memory > inefficient as it will hold the whole POST data into memory and duplicate it > twice (from SG(request_info).post_data to $HTTP_RAW_POST_DATA -- even if > always_populate_raw_post_data=0 -- and SG(request_info).raw_post_data). > > This introduces a new ini setting, disable_post_data_processing, but it's a > benign one. No incompatibilities between setups will arise because no one > will enable it globally (it would be insane), only selectively to the > scripts that require it. The reason for an ini setting is that it must be > set early in the request life. > > Thoughts? > > This is a very, very good idea as more and more requests will be in non-urlencoded format (e.g.: JSON), especially in REST architectures. A little performance boost would not do any harm. +1. > -- > Gustavo Lopes > -- > > > -- Nicolas A. Bérard-Nault (nicobn@gmail.com) --- ## #50902 — Sergio A.Kessler — 2010-12-07T16:43:22+00:00 URL: https://externals.io/message/50885#50902 · In reply to: #50897 John Mertic php.net> writes: [snip] > Wouldn't something like this work? > > > php_value disable_post_data_processing On > maybe it should be: php_value post_data_processing Off so you don't have double negation in the case of php_value disable_post_data_processing Off regards, /sak --- ## #50903 — Julien Pauli — 2010-12-07T17:58:45+00:00 URL: https://externals.io/message/50885#50903 · In reply to: #50902 I don't remember the source, refresh my mind quickly please : What happens if variables_order doesn't content 'P' ? Is the data still duplicated into memory ? I know that $_POST is still there but is an empty array in that case. And yes, I'm +1 for such a patch :) J.Pauli On Tue, Dec 7, 2010 at 5:43 PM, Sergio A. Kessler wrote: > John Mertic php.net> writes: > > [snip] > > […] > > maybe it should be: > >  php_value post_data_processing Off > > so you don't have double negation in the case of > >  php_value disable_post_data_processing Off > > > regards, > /sak --- ## #50904 — Rasmus Lerdorf — 2010-12-07T18:18:30+00:00 URL: https://externals.io/message/50885#50904 · In reply to: #50885 > This introduces a new ini setting, disable_post_data_processing, but > it's a benign one. No incompatibilities between setups will arise > because no one will enable it globally (it would be insane), only > selectively to the scripts that require it. The reason for an ini > setting is that it must be set early in the request life. I'm ok with it. It's a bit awkward having disable=on to disable something though. Generally we prefer enable=off because it causes fewer brain injuries. -Rasmus --- ## #50905 — Gustavo Lopes — 2010-12-07T18:20:44+00:00 URL: https://externals.io/message/50885#50905 · In reply to: #50903 > I don't remember the source, refresh my mind quickly please : > What happens if variables_order doesn't content 'P' ? Is the data > still duplicated into memory ? I know that $_POST is still there but > is an empty array in that case. > Yes. Parsing the POST data occurs in a later phase (see post_reader vs post_handler/treat_data). This consideration doesn't apply to multipart, which combines the two steps. -- Gustavo Lopes --- ## #50906 — tigger.on@gmail.com — 2010-12-07T22:34:23+00:00 URL: https://externals.io/message/50885#50906 · In reply to: #50905 If the objective is to 'block' POST data from getting to PHP, (in apache) you can use: http://httpd.apache.org/docs/2.0/mod/core.html#limit No need to change / add anything to PHP. -Tig --- ## #50978 — Gustavo Lopes — 2010-12-09T20:37:45+00:00 URL: https://externals.io/message/50885#50978 · In reply to: #50885 On Tue, 07 Dec 2010 07:08:34 -0000, Gustavo Lopes wrote: > The very simple attached patch adds an option to disable POST data > processing, which implies the data can only be read in a stream fashion > through php://input. > I've committed to trunk the patch with the name of the ini option changed from disable_post_data_processing to enable_post_data_reading. -- Gustavo Lopes --- ## #51042 — Pierre Joye — 2010-12-16T12:32:15+00:00 URL: https://externals.io/message/50885#51042 · In reply to: #50978 hi, The more I look at this option the more I think it is confusing. I'm not sure the gain is worth this confusion either. However I would prefer to bring back a proposal we had a couple of years ago, to totally disable post data. When disabled, the POST data will be totally ignored, no matter if php://input, raw data or whatever other ways we may have to access it. The data given by the server/sapi will be ignored. This option has the benefit to be very simple and solves one known attack vector in a very clean way. Cheers, > On Tue, 07 Dec 2010 07:08:34 -0000, Gustavo Lopes > wrote: > > > […] > > I've committed to trunk the patch with the name of the ini option changed > from disable_post_data_processing to enable_post_data_reading. > > -- > Gustavo Lopes > > -- > > > > -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- ## #51043 — Ferenc Kovacs — 2010-12-16T12:42:08+00:00 URL: https://externals.io/message/50885#51043 · In reply to: #51042 > hi, > > The more I look at this option the more I think it is confusing. I'm > not sure the gain is worth this confusion either. However I would > prefer to bring back a proposal we had a couple of years ago, to > totally disable post data. When disabled, the POST data will be > totally ignored, no matter if php://input, raw data or whatever other > ways we may have to access it. The data given by the server/sapi will > be ignored. > > This option has the benefit to be very simple and solves one known > attack vector in a very clean way. > > Yeah, but that idea isn't solves the problem which the current one tries to address. So while I think that it would be a good security measure to alllow to disable POST altogether, but that should be discussed/voted in a different thread. So currently we didn't talk about security measures, but performance gains: If somebody wants to write a script, which handles big file uploads, but only writes it to somewhere (to file, or another stream), then currently you have to allocate the memory for the post data twice(see the first email), which is very inefficient if you don't need the $_POST at all. Tyrael --- ## #51044 — Pierre Joye — 2010-12-16T12:47:43+00:00 URL: https://externals.io/message/50885#51044 · In reply to: #51043 hi, > So currently we didn't talk about security measures, but performance gains: > If somebody wants to write a script, which handles big file uploads, but > only writes it to somewhere (to file, or another stream), then currently you > have to allocate the memory for the post data twice(see the first email), > which is very inefficient if you don't need the $_POST at all. yes, but that's something very confusing right now, the naming and the other ways to access POST data. The goal of this idea is a good thing to do, but the naming and its implementation are confusing (processing vs reading vs used at all). Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- ## #51045 — Ferenc Kovacs — 2010-12-16T12:54:53+00:00 URL: https://externals.io/message/50885#51045 · In reply to: #51044 > hi, > > > > > […] > gains: > > […] > you > > […] > > yes, but that's something very confusing right now, the naming and the > other ways to access POST data. The goal of this idea is a good thing > to do, but the naming and its implementation are confusing (processing > vs reading vs used at all). > > agree, but it's not helping, if we introduce another similar idea for a whole different purpose to the conversation. :) the best would be a nice and clean RFC with the current status, the known problems, the suggested solutions, and a common and well-understood consistent naming convention. (disable POST processing, disable POST population, etc.) Tyrael --- ## #51046 — Gustavo Lopes — 2010-12-16T13:09:57+00:00 URL: https://externals.io/message/50885#51046 · In reply to: #51044 On Thu, 16 Dec 2010 12:47:43 -0000, Pierre Joye wrote: As I said in IRC, I see no value in having an option that disables accessing POST data completely (i.e. the behavior of enable_post_data_reading=Off + disallowing reading it through php://input). With enable_post_data_reading=Off, the only way to access the data is through php://input; if the script uses the wrapper to read the data is because it presumably needs, or expects, such data. If the script doesn't need it, it won't read php://input. Activating a putative option that completely disabled reading POST data would break the first script and would have no effect on the second. And it's not like there would be performance benefits from introducing such as option. More importantly, the enable_post_data_reading is to allow efficient handling of POST data and access to raw POST data with multipart requests. Adding an option that disables accessing POST data completely would not help. > > > > […] > > […] but > > […] > > yes, but that's something very confusing right now, the naming and the > other ways to access POST data. The goal of this idea is a good thing > to do, but the naming and its implementation are confusing (processing > vs reading vs used at all). > I don't thinks it's that confusing. This new option just prevents PHP from doing any automatic reading and processing of the POST data, leaving it entirely on the hands of the programmer to that, if he wishes, through php://input. What is confusing is the different behavior of PHP with $HTTP_RAW_POST_DATA and the influence always_populate_post_data and the different handling of urlencoded vs multipart vs other content-type POST requests vs non-POST requests with a request body. -- Gustavo Lopes --- ## #51047 — Pierre Joye — 2010-12-16T13:16:34+00:00 URL: https://externals.io/message/50885#51047 · In reply to: #51046 There are a lot of values to disable POST completely. That's also why thinking the option you are proposing while keeping in mind the whole picture makes sense. There are different existing modes and one or two new modes (to be introduced, like disabling POST). Having a clean way to choose which mode should be used for a request makes sense and will improve the user experience. Right now and as shown the confusing discussions on IRC (or partially here) shows that the current situation is a mess. Again, I'm not saying that the idea behind this patch is bad, only that there is a better way to do it while being less confusing and making the whole thing cleaner to define. 2010/12/16 Gustavo Lopes : > On Thu, 16 Dec 2010 12:47:43 -0000, Pierre Joye > wrote: > > As I said in IRC, I see no value in having an option that disables > accessing POST data completely (i.e. the behavior of > enable_post_data_reading=Off + disallowing reading it through > php://input). > > With enable_post_data_reading=Off, the only way to access the data is > through php://input; if the script uses the wrapper to read the data is > because it presumably needs, or expects, such data. If the script doesn't > need it, it won't read php://input. Activating a putative option that > completely disabled reading POST data would break the first script and > would have no effect on the second. And it's not like there would be > performance benefits from introducing such as option. > > More importantly, the enable_post_data_reading is to allow efficient > handling of POST data and access to raw POST data with multipart requests. > Adding an option that disables accessing POST data completely would not > help. > > > […] > > > […] > but > > […] > > I don't thinks it's that confusing. This new option just prevents PHP from > doing any automatic reading and processing of the POST data, leaving it > entirely on the hands of the programmer to that, if he wishes, through > php://input. > > What is confusing is the different behavior of PHP with > $HTTP_RAW_POST_DATA and the influence always_populate_post_data and the > different handling of urlencoded vs multipart vs other content-type POST > requests vs non-POST requests with a request body. > > -- > Gustavo Lopes > -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- Next page: https://externals.io/message/50885.md?page=2