# Adding validate_var_array()/validate_input_array() to which version? - Thread: https://externals.io/message/94773 - Original archive: https://news-web.php.net/php.internals/94773 - Started by Yasuo Ohgaki on 2016-08-01T08:23:37+00:00 - Messages: 38 - Page 1 of 2: messages 1 to 25 (25 per page, add `?page=N` to the URL) - Next page: https://externals.io/message/94773.md?page=2 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #94773 — Yasuo Ohgaki — 2016-08-01T08:23:37+00:00 URL: https://externals.io/message/94773#94773 Hi all, We have `filter_var_array()`/filter_input_array() currently. They are designed as filter functions. i.e. They convert offending elements to NULL/FALSE. Therefore, it's difficult to validate and see if inputs are valid with specified specifications. https://github.com/php/php-src/pull/2048 This patch adds true validation functions - validate_var_array() - Almost the same as `filter_var_array()` except it returns scalar `FALSE` on validation failure(s), instead of filtered array. - validate_input_array() - Almost the same as `filter_input_array()` except it returns scalar `FALSE` on validation failure(s), instead of filtered array. These functions are handy for input validation that stops script execution upon invalid(attacker's) inputs. Question is which version should I target for? It's simple enough patch to be merged to 7.1. IMO. Comments are appreciated! Regards, P.S. It's possible to return array that contains offending values. It is not included since users can store whole offending input array. Whole input is more useful for attack analysis. -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #94776 — Christoph Becker — 2016-08-01T10:08:58+00:00 URL: https://externals.io/message/94773#94776 · In reply to: #94773 > We have `filter_var_array()`/filter_input_array() currently. They are > designed as filter functions. i.e. They convert offending elements to > NULL/FALSE. Therefore, it's difficult to validate and see if inputs > are valid with specified specifications. Maybe I'm missing something, but wouldn't a comparision of the original array with the filtered array already suffice, see ? -- Christoph M. Becker --- ## #94782 — Yasuo Ohgaki — 2016-08-01T21:28:55+00:00 URL: https://externals.io/message/94773#94782 · In reply to: #94776 Hi Christoph, > > […] > > Maybe I'm missing something, but wouldn't a comparision of the original > array with the filtered array already suffice, see ? It may work, but it changes data types when it's possible. Float value could be headache. https://3v4l.org/3DUmn - Comparing array by == is too slow for extensive use of validations. - "Validate and reject" is better than "Filter and accept" for most applications. Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #94783 — Yasuo Ohgaki — 2016-08-01T22:12:06+00:00 URL: https://externals.io/message/94773#94783 · In reply to: #94773 > We have `filter_var_array()`/filter_input_array() currently. They are > designed as filter functions. i.e. They convert offending elements to > NULL/FALSE. Therefore, it's difficult to validate and see if inputs > are valid with specified specifications. > > https://github.com/php/php-src/pull/2048 > > This patch adds true validation functions > - validate_var_array() - Almost the same as `filter_var_array()` except > it returns scalar `FALSE` on validation failure(s), instead of filtered > array. > - validate_input_array() - Almost the same as `filter_input_array()` > except it returns scalar `FALSE` on validation failure(s), instead of > filtered array. > > > These functions are handy for input validation that stops script > execution upon invalid(attacker's) inputs. > > Question is which version should I target for? > It's simple enough patch to be merged to 7.1. IMO. > > Comments are appreciated! > > Regards, Raising Exception would be prefered. Any comment raising exception? ExceptionFilterValidate wouldn't cause much BC, IMO. Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #94785 — Dan Ackroyd — 2016-08-01T22:29:50+00:00 URL: https://externals.io/message/94773#94785 · In reply to: #94773 > > Question is which version should I target for? Why does this need to be in PHP core? Why can't this just be a userland library? cheers Dan --- ## #94786 — Yasuo Ohgaki — 2016-08-01T22:51:29+00:00 URL: https://externals.io/message/94773#94786 · In reply to: #94785 Hi Dan, > > > […] > > Why does this need to be in PHP core? Input validation is the most important security measure. https://www.securecoding.cert.org/confluence/display/seccode/Top+10+Secure+Coding+Practices https://www.owasp.org/index.php/OWASP_Secure_Coding_Practices_-_Quick_Reference_Guide Input validation best practice is "Validate values by whitelist and Reject invalid", not "Filter values and Accept". PHP should have function follows best practices. (I'm not saying nobody should not "Filter values and Accept". It's okay if your security policy allows it.) > Why can't this just be a userland library? PHP must have input validation feature that achieves previously described validation. Basic feature like input validation must be able to perform quickly, so it should be provided as core feature like basic escaping functions. Regards, P.S. I'll add string validation filters (e.g. min/max, encoding check,etc) and validation function callback (i.e. Must return bool) soon. -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #94789 — Yasuo Ohgaki — 2016-08-01T23:59:14+00:00 URL: https://externals.io/message/94773#94789 · In reply to: #94783 Hi all, > > Raising Exception would be prefered. > Any comment raising exception? ExceptionFilterValidate wouldn't > cause much BC, IMO. Any comments on using SPL exception? http://php.net/manual/en/class.unexpectedvalueexception.php With Exception or UnexpedtedValueException, there is no BC. Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #94794 — Lester Caine — 2016-08-02T05:56:06+00:00 URL: https://externals.io/message/94773#94794 · In reply to: #94786 > PHP must have input validation feature that achieves previously > described validation. Basic feature like input validation must be able > to perform quickly, so it should be provided as core feature like > basic escaping functions. > > Regards, > > P.S. I'll add string validation filters (e.g. min/max, encoding > check,etc) and validation function callback (i.e. Must return bool) > soon. Once again lots of additional code is being added which only fixes HALF of the input validation problem. The same as 'strict typing'. All of these extras can simply be eliminated if you address the problem of adding a set of rules to the basic 'var' that allow proper validation of each individually ... and *I* include in those rules adding the correct escaping for that particular variable. Which is EXACTLY what oe does in the user land libraries that currently fill the gap. On one hand we are being pushed to add things like getter and setter and all that overhead to create proper objects, while this option is back with handling a raw set of variables as an array? -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk --- ## #94803 — Rowan Collins — 2016-08-02T14:37:21+00:00 URL: https://externals.io/message/94773#94803 · In reply to: #94783 > Raising Exception would be prefered. > Any comment raising exception? ExceptionFilterValidate wouldn't > cause much BC, IMO. I don't follow. Do you mean throwing an exception to the existing filter-and-continue functions? If so, that would surely break every single use of those functions. Or do you mean adding a new function which either silently succeeds or throws an exception? That doesn't sound good to me - an exception shouldn't be the expected result of something, and invalid input is an expected condition (just an undesirable one). It's easier to add a throw statement based on a boolean result than to catch an exception and continue with other checks. I may have misunderstood your question, though. Regards, -- Rowan Collins [IMSoP] --- ## #94819 — Christian Stadler — 2016-08-04T01:07:05+00:00 URL: https://externals.io/message/94773#94819 · In reply to: #94773 Am 01.08.2016 um 10:23 schrieb Yasuo Ohgaki: > P.S. It's possible to return array that contains offending values. It > is not included since users can store whole offending input array. > Whole input is more useful for attack analysis. Actually I wanted to suggest exactly that for ppl. who want to give Feedback to their users, what values failed to validate to the users. Probably with a fourth optional param, like `$return_invalid = false`? Of course logging is a different topic and should always use the whole offending input array. Regards, Christian Stadler --- ## #94826 — Yasuo Ohgaki — 2016-08-04T10:10:50+00:00 URL: https://externals.io/message/94773#94826 · In reply to: #94819 Hi Christian and all, > Am 01.08.2016 um 10:23 schrieb Yasuo Ohgaki: > > […] > > Actually I wanted to suggest exactly that for ppl. who want to give > Feedback to their users, what values failed to validate to the users. > Probably with a fourth optional param, like `$return_invalid = false`? > Of course logging is a different topic and should always use the whole > offending input array. I can set offending value to filter globals so that it can be retrieved later in catch block. I cannot return or modify referenced parameter because of raised exception. I don't mind adding this feature. It requires an API like validate_get_offending_value(). (The name should be nicer) How many of us are interested in this feature? Thank you for feedback! -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #94827 — Rowan Collins — 2016-08-04T10:28:45+00:00 URL: https://externals.io/message/94773#94827 · In reply to: #94826 > I cannot return or modify referenced > parameter because of raised exception. This is another reason why exceptions are not appropriate for this case. As soon as you make an exception the expected result of something, you end up tying yourself in knots trying to build useful control flow around it. Regards, -- Rowan Collins [IMSoP] --- ## #94828 — Yasuo Ohgaki — 2016-08-04T10:45:09+00:00 URL: https://externals.io/message/94773#94828 · In reply to: #94803 Hi Rowan, > > > […] > > > I don't follow. Do you mean throwing an exception to the existing > filter-and-continue functions? If so, that would surely break every single > use of those functions. Existing filters currently used are not changed at all. They behave exactly as they are now. If not, it's a bug. Exceptions are raised only when validate_*() functions are used. Input validations should pass almost always. So checking return value from validation functions are not prefered. > > Or do you mean adding a new function which either silently succeeds or > throws an exception? That doesn't sound good to me - an exception shouldn't > be the expected result of something, and invalid input is an expected > condition (just an undesirable one). It's easier to add a throw statement > based on a boolean result than to catch an exception and continue with other > checks. > > I may have misunderstood your question, though. Invalid inputs that violate input validation rules should not happen almost always because input validation is _not_ the same input mistakes by normal users. e.g. typo, user send a little too long password. These mistakes should be handled as input errors/mistakes, not input validation error. Example input validation errors are, - Broken UTF-8 encoding - NUL, etc control chars in string. - Too long or too short string. e.g. Client side validation inputs - JS validated and values set by server programs like , hash value, etc. - and so on. If there are these kind of inputs, they are intentionally broken inputs. It should result in Exception and program should be _terminated_ script execution after logging attack and sending nicely formatted page that warns attackers, for instance. validate_*() functions should be used to handle above input violations that should never happen under normal condition. Did I understand your concern correctly? I hope explained well. Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #94829 — Yasuo Ohgaki — 2016-08-04T10:56:24+00:00 URL: https://externals.io/message/94773#94829 · In reply to: #94794 Hi Lester, > > Once again lots of additional code is being added which only fixes HALF > of the input validation problem. The same as 'strict typing'. I'm not trying to solve all of input validation issues by this proposal. Large amount of responsibilities are left to programmers. These could be done by callback, regex, multiple filter definitions. If you feel there is missing critical feature, please let me know. I think basic features required by security best practices are provided by this RFC changes. > > All of these extras can simply be eliminated if you address the problem > of adding a set of rules to the basic 'var' that allow proper validation > of each individually ... and *I* include in those rules adding the > correct escaping for that particular variable. Which is EXACTLY what oe > does in the user land libraries that currently fill the gap. > > On one hand we are being pushed to add things like getter and setter and > all that overhead to create proper objects, while this option is back > with handling a raw set of variables as an array? Do you mean validation rule definition? If so, yes. It's an array. Array definition rules could be wrong/broken, e.g. typo, and consequence of a broken definition is severe. So I added definition validation function, validate_check_definition(). (Better name might be preferred "validate" and "check" sounds strange) validate_check_definition() could be called via `assert()` during development. Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #94830 — Christian Stadler — 2016-08-04T11:27:51+00:00 URL: https://externals.io/message/94773#94830 · In reply to: #94826 Am 04.08.2016 um 12:10 schrieb Yasuo Ohgaki: > Hi Christian and all, > > > > […] > I can set offending value to filter globals so that it can be > retrieved later in catch block. I cannot return or modify referenced > parameter because of raised exception. Well, since some people have objections about raising exceptions here, this should probably be either in a seperate vote or additional options in the main vote. Probably something, like: Yes, either | Yes, without the exception | Yes, with the exception | No Personally I would vote for 'Yes, either'. If I could, that is. > I don't mind adding this feature. It requires an API like > validate_get_offending_value(). (The name should be nicer) > How many of us are interested in this feature? Then this new function should have an offset param. With this I could check, if the array has any offending values and then continue with the rest ... mmh, now that I think of it, this isn't really necessary. Uhm, well anyway: I'd suggest, that the ind(ex/ices) should be returned rather, than the actual value names. Regards, Christian Stadler --- ## #94831 — Rowan Collins — 2016-08-04T11:30:14+00:00 URL: https://externals.io/message/94773#94831 · In reply to: #94828 > Invalid inputs that violate input validation rules should not happen > almost always because input validation is _not_ the same input > mistakes by normal users. e.g. typo, user send a little too long > password. These mistakes should be handled as input errors/mistakes, > not input validation error. To me, "validation" absolutely means "checking if the user input the right things". What you're talking about sounds more like assertions, which would fit with the "this must be OK else die" implication of exceptions. Perhaps the functions could be called filter_assert_var_array() etc? Or perhaps some other term, to make it clear that these functions are not intended for the common task of "form validation". Regards, -- Rowan Collins [IMSoP] --- ## #94835 — Yasuo Ohgaki — 2016-08-04T19:29:21+00:00 URL: https://externals.io/message/94773#94835 · In reply to: #94830 Hi Christian, > Am 04.08.2016 um 12:10 schrieb Yasuo Ohgaki: > > […] > > Well, since some people have objections about raising exceptions here, > this should probably be either in a seperate vote or additional options > in the main vote. Probably something, like: > Yes, either | Yes, without the exception | Yes, with the exception | No > Personally I would vote for 'Yes, either'. If I could, that is. One of my objective is following best practices. Prefer exception over error is one of them. Although, I strongly suggest to use exception for validation errors, I will have choices. (Exception should be used error cases that should not happen usually, but usual error handling would work. Error message could be more user friendly because php_error_docref() supports va arg) > > > […] > > Then this new function should have an offset param. With this I could > check, if the array has any offending values and then continue with the > rest ... mmh, now that I think of it, this isn't really necessary. > > Uhm, well anyway: I'd suggest, that the ind(ex/ices) should be returned > rather, than the actual value names. OK. Thank you. I'll add this. The reason why I said store "value" is the code. To get index, it has to store index somewhere or change many lines of code. Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #94836 — Rowan Collins — 2016-08-04T21:04:16+00:00 URL: https://externals.io/message/94773#94836 · In reply to: #94835 Hi Yasuo, > One of my objective is following best practices. > Prefer exception over error is one of them. Although, I strongly suggest > to use exception for validation errors, I will have choices. Best practice is to use exceptions to indicate an error *inside* the function, i.e. something going wrong with the function itself. If I run (as a simplified example) validate_var('abc', FILTER_NUMERIC) that's not something going wrong inside validate_var(), so validate_var() shouldn't be throwing an exception, IMO. Now, the *reason* I'm calling validate_var may be that the function I'm writing will go wrong if the validation doesn't pass, but that's up to *me* to throw an exception: if ( validate_var($foo, FILTER_NUMERIC) ) { throw new InvalidArgumentException; } That said, in another message you explained that this isn't intended for "validation" in the sense of "form validation", but in a more restricted sense of "dangerous data"; I think the name misled me, and may mislead others, if that is the sole purpose of the function. I'm still not entirely convinced, though, that a boolean-returning function wouldn't be enough to fill both purposes. Regards, -- Rowan Collins [IMSoP] --- ## #94838 — Yasuo Ohgaki — 2016-08-04T21:25:50+00:00 URL: https://externals.io/message/94773#94838 · In reply to: #94836 Hi Rowan, > > > […] > > > Best practice is to use exceptions to indicate an error *inside* the > function, i.e. something going wrong with the function itself. If I run (as > a simplified example) validate_var('abc', FILTER_NUMERIC) that's not > something going wrong inside validate_var(), so validate_var() shouldn't be > throwing an exception, IMO. > > Now, the *reason* I'm calling validate_var may be that the function I'm > writing will go wrong if the validation doesn't pass, but that's up to *me* > to throw an exception: > > if ( validate_var($foo, FILTER_NUMERIC) ) { > throw new InvalidArgumentException; > } Input validation is like assertion shouldn't fail. However, it's possible to design code like this. > > That said, in another message you explained that this isn't intended for > "validation" in the sense of "form validation", but in a more restricted > sense of "dangerous data"; I think the name misled me, and may mislead > others, if that is the sole purpose of the function. I'm still not entirely > convinced, though, that a boolean-returning function wouldn't be enough to > fill both purposes. I understands there are use cases for both with/without exception. In fact, I implemented it without exception at first. My preference is to raise exception to encourage users to handle validation error correctly, i.e. terminate script execution, but my intention may not be interpreted correctly with or without exception. How about to have switch like `assert()`? It has exception on/off switch. Anyone prefer parameter flag? Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #94839 — Rowan Collins — 2016-08-04T21:38:32+00:00 URL: https://externals.io/message/94773#94839 · In reply to: #94838 > My preference is to raise exception to encourage users to handle > validation error correctly, i.e. terminate script execution, but my > intention may not be interpreted correctly with or without exception. I know naming things is hard, but I really do think "validation" is not the correct name for what you are talking about. Search online for "form validation", and you will see what I mean. The correct response to a form validation error is to show a message to the user with as much detail as possible, not to simply terminate the script and assume they are trying to attack your application. Regards, -- Rowan Collins [IMSoP] --- ## #94840 — Yasuo Ohgaki — 2016-08-04T21:47:41+00:00 URL: https://externals.io/message/94773#94840 · In reply to: #94839 Hi Rowan, > > > […] > > > I know naming things is hard, but I really do think "validation" is not the > correct name for what you are talking about. Search online for "form > validation", and you will see what I mean. > > The correct response to a form validation error is to show a message to the > user with as much detail as possible, not to simply terminate the script and > assume they are trying to attack your application. We are talking about different things. I'll document it clearly in RFC. Thank you! -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #94841 — Yasuo Ohgaki — 2016-08-04T21:58:55+00:00 URL: https://externals.io/message/94773#94841 · In reply to: #94840 Hi Rowan, > > […] > > We are talking about different things. > I'll document it clearly in RFC. Different function name may help to avoid confusions. How about filter_assert_*() (I'm using filter_ prefix suggested by Pierre. It's few chars longer, but better consistency :) These functions could be used for input mistake/error checks, but I would like to users to use them only to assert external inputs, see if valid(acceptable) or invalid(unacceptable) for the software. The name may help. Hopefully. Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #94842 — Lester Caine — 2016-08-04T22:03:14+00:00 URL: https://externals.io/message/94773#94842 · In reply to: #94840 > > […] > We are talking about different things. > I'll document it clearly in RFC. But both need to be done at the same time ... Validating the input data requires that you have a set of rules for each variable, and if all are correct then one can process the 'array', but if an element fails validation then one needs to handle the error either as suspicious, or simply out of range. I repeat that handling the complexities of EACH variable of the validation is a package of work in it's own right, and trying to handle some aspects via an array function does not remove the more obvious need to decide what order to handle validation errors on each element, or to return error messages for each variable that fails validation. If the element is some attempt to create an injection of js or html tags then it should fail validation, but equally it may be a valid input as long as it is escaped and stored in a correct manor. The complexity of what you plan for your array validation elements start with all the same rules applied to a single variable. -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk --- ## #94844 — Yasuo Ohgaki — 2016-08-04T23:58:21+00:00 URL: https://externals.io/message/94773#94844 · In reply to: #94842 > > > […] > > But both need to be done at the same time ... Right and wrong? Both have to be done in a software. > > Validating the input data requires that you have a set of rules for each > variable, and if all are correct then one can process the 'array', but > if an element fails validation then one needs to handle the error either > as suspicious, or simply out of range. I repeat that handling the > complexities of EACH variable of the validation is a package of work in > it's own right, and trying to handle some aspects via an array function > does not remove the more obvious need to decide what order to handle > validation errors on each element, or to return error messages for each > variable that fails validation. If the element is some attempt to create > an injection of js or html tags then it should fail validation, but > equally it may be a valid input as long as it is escaped and stored in a > correct manor. The complexity of what you plan for your array validation > elements start with all the same rules applied to a single variable. Modern softwares are complex enough to handle inputs securely. Softwares should use "divide and conquer" principle. Input validation is like assertion that should never happen. It should be validated(asserted) according to the software input spec and it should be checked as soon as inputs arrive to the software, and leave behind mistakes, logical inconsistency, etc. Validation(assertion) errors are: - Broken char encoding - Broken input format. e.g. Broken int/float/bool, strings like hash/id/etc - Invalid chars in string. i.e. NUL and other control chars - Too large/small values. e.g. Too long/short string, too large/small int, float. - Too few/many inputs. ("Too many inputs" cannot be address by filter functions, but can be addressed input validation code) Some of above are need to be addressed by software business logic on occasions depending on client output, but input validation should make sure input values validity as much as possible to reduce software complexity. After input validation, business logic should take care of the rest. Input error(user mistake, logical inconsistency) check should be checked separately by the software as a part of the software business logic. This way, business logic code can concentrate on jobs that it should and reduce complexity of input handling. This is my view of what input validation should do. (and what business logic should do) Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #94845 — Yasuo Ohgaki — 2016-08-05T00:06:39+00:00 URL: https://externals.io/message/94773#94845 · In reply to: #94844 I should have been more precise. > > Input validation is like assertion that should never happen. > It should be validated(asserted) according to the software input spec and > it should be checked as soon as inputs arrive to the software, and leave > behind mistakes, logical inconsistency, etc. Input validation is like assertion that should never happen. Inputs should be validated(asserted) according to the software input spec and inputs should be checked as soon as inputs arrive to the software, and leave behind mistakes, logical inconsistency, etc. -- Yasuo Ohgaki yohgaki@ohgaki.net --- Next page: https://externals.io/message/94773.md?page=2