A validator module for PHP7

7 years ago by Yasuo Ohgaki — view source

unread

Hi all,

I spent a little time for a new input validation module. It's not totally
new module, but is based on Filter module's validation filter improvement
RFC in many ways. [1]

As all of us knew already, input validation is the most important practice
in secure coding. [2][3] Yet, we don't provide usable feature out of box.
Sadly, almost all apps do not have proper input validation at trust
boundary. Unless we improve filter's validation, we need usable basic
validator by default. IMO.

Since I didn't get much feedbacks during the RFC discussion, I cannot tell
what part is disliked. I guess too much features in filter is one reason.
Another is messed up codes/features by providing both "filter" and
"validation".

Validator for PHP7 (validate module) gets rid of unneeded features. It only
has features for basic PHP data type validations. Validation rule(spec)
array is flexible enough. Almost any types of inputs could be handled by
multiple and nested validation rules.

Except some minor features like overflow checks, most planned features are
implemented.

https://github.com/yohgaki/validate-php

Although the code is based on filter module's code, it's almost full
rewrite except validation logic came from filter. Please consider this as
under development module.
Feedbacks are appreciated.

Regards,

[1] https://wiki.php.net/rfc/add_validate_functions_to_filter
[2]
https://www.securecoding.cert.org/confluence/display/seccode/Top+10+Secure+Coding+Practices
[3]
https://www.owasp.org/index.php/OWASP_Secure_Coding_Practices_-_Quick_Reference_Guide

--
Yasuo Ohgaki
yohgaki@ohgaki.net

7 years ago by Rowan Collins — view source

unread

Hi all,

I spent a little time for a new input validation module. It's not
totally
new module, but is based on Filter module's validation filter
improvement
RFC in many ways. [1]

Hi Yasuo,

Thanks for tackling this. I do think the current filter module is user unfriendly and it would be great to have something better. A couple of quick thoughts based on your README:

The use of nested arrays keeps things simple in one sense, but the deep nesting can get confusing. I wonder if a ValidationRule class would make the distinction between parameters and references to existing rules clearer. This would also give you a natural home for validating the validation rules themselves (probably by throwing an exception in the constructor).
A minor point, but most style guides would suggest function names should be verbs, not adjectives, so "validate()" rather than "valid()".
Is there a use case for valid_id() or is it a temporary debugging thing that won't be in the final version?

In general, I like the idea, but would have to play around a bit to see if it felt easy to use in real world situations.

Regards,

--
Rowan Collins
[IMSoP]

7 years ago by Crocodile — view source

unread

In most cases users would like more than just valid/invalid, i. e. which
particular rule(s) failed and also human-readable error messages. As of
simple validation that is almost always at hand, filter_* functions do a
good job, although I agree that they could be better. I, for one, would
like to see a full-featured validation as part of PHP. However, this RFC
only looks like a slightly better version of filter_* functions, that is,
the new module will provide almost the same functionality but with a
different interface. I would vote against it.

On Mon, Sep 4, 2017 at 2:54 PM Rowan Collins rowan.collins@gmail.com
wrote:

Hi all,

I spent a little time for a new input validation module. It's not
totally
new module, but is based on Filter module's validation filter
improvement
RFC in many ways. [1]

Hi Yasuo,

Thanks for tackling this. I do think the current filter module is user
unfriendly and it would be great to have something better. A couple of
quick thoughts based on your README:

The use of nested arrays keeps things simple in one sense, but the deep
nesting can get confusing. I wonder if a ValidationRule class would make
the distinction between parameters and references to existing rules
clearer. This would also give you a natural home for validating the
validation rules themselves (probably by throwing an exception in the
constructor).

A minor point, but most style guides would suggest function names should
be verbs, not adjectives, so "validate()" rather than "valid()".

Is there a use case for valid_id() or is it a temporary debugging thing
that won't be in the final version?

In general, I like the idea, but would have to play around a bit to see if
it felt easy to use in real world situations.

Regards,

--
Rowan Collins
[IMSoP]

--

--
Best regards,
Victor Bolshov

7 years ago by Marco Pivetta — view source

unread

In most cases users would like more than just valid/invalid, i. e. which
particular rule(s) failed and also human-readable error messages. As of
simple validation that is almost always at hand, filter_* functions do a
good job, although I agree that they could be better. I, for one, would
like to see a full-featured validation as part of PHP. However, this RFC
only looks like a slightly better version of filter_* functions, that is,
the new module will provide almost the same functionality but with a
different interface. I would vote against it.

And also, it would need to be better than all of these to be worth writing
it in C:

https://packagist.org/search/?q=validator

Marco Pivetta

http://twitter.com/Ocramius

http://ocramius.github.com/

7 years ago by Paul Jones — view source

unread

In most cases users would like more than just valid/invalid, i. e. which
particular rule(s) failed and also human-readable error messages. As of
simple validation that is almost always at hand, filter_* functions do a
good job, although I agree that they could be better. I, for one, would
like to see a full-featured validation as part of PHP. However, this RFC
only looks like a slightly better version of filter_* functions, that is,
the new module will provide almost the same functionality but with a
different interface. I would vote against it.

And also, it would need to be better than all of these to be worth writing
it in C:

https://packagist.org/search/?q=validator

And these as well: https://packagist.org/search/?q=filter

(Or at least have some level of parity.)

--
Paul M. Jones
pmjones88@gmail.com
http://paul-m-jones.com

Modernizing Legacy Applications in PHP
https://leanpub.com/mlaphp

Solving the N+1 Problem in PHP
https://leanpub.com/sn1php

7 years ago by Yasuo Ohgaki — view source

unread

Hi all,

In most cases users would like more than just valid/invalid, i. e. which
particular rule(s) failed and also human-readable error messages. As of
simple validation that is almost always at hand, filter_* functions do a
good job, although I agree that they could be better. I, for one, would
like to see a full-featured validation as part of PHP. However, this RFC
only looks like a slightly better version of filter_* functions, that
is,
the new module will provide almost the same functionality but with a
different interface. I would vote against it.

And also, it would need to be better than all of these to be worth
writing
it in C:

https://packagist.org/search/?q=validator

And these as well: https://packagist.org/search/?q=filter

I cannot guess people's thought. I appreciated feedback!

Why do you think basic validation module should be better than full OO
implementation(s)?

Simple and basic type support NULL/BOOL/INT/FLOAT/STRING/ARRAY/OBJECT is
enough
for C written PHP module. IMHO, PHP modules are better of to provide basic
features
that may be extended by user scripts.

I picked 1st one on the list as an example.
This kind of rule construction is inefficient compare to simple array rules,
so I doubt this is the way for basic validator module written by C.

$validator = Validation::createValidator();
$violations = $validator->validate('Bernhard', array(
new Length(array('min' => 10)),
new NotBlank(),
));

However, this particular validation class seems it could be good one that
wraps
validate module for nicer OO API and faster execution.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

7 years ago by Lester Caine — view source

unread

I cannot guess people's thought. I appreciated feedback!

With a decent database layer a lot of the validation you are proposing
is already covered but PDO does not help in this area. Adding another
layer that does not integrate with a storage layer is just adding to the
current mess ...

--
Lester Caine - G8HFL

Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

7 years ago by Rowan Collins — view source

unread

I cannot guess people's thought. I appreciated feedback!

With a decent database layer a lot of the validation you are proposing
is already covered but PDO does not help in this area. Adding another
layer that does not integrate with a storage layer is just adding to
the
current mess ...

Validation should have nothing to do with the storage layer. Or at least, there should be a level of validation separate from the storage layer.

Inputs may come from all sorts of sources: the HTTP request, an API, an import file, etc; and they may be going to all sorts of destinations: the HTTP response, a database, an API, an export file, an e-mail, etc.

Regardless of where it came from, and where it's going to end up, the application knows what format that input data should be in to use as or populate appropriate domain models. For instance, "age_in_years is a non-negative integer" is an invariant fact about the domain being modelled, even if it's a value that goes nowhere near any form of database.

I actually agree that this module doesn't need to replace the existing userland libraries, only to act as a useful base for them, as well as a useful fallback when writing "raw PHP". The key problem is balancing flexibility and usability such that people will reach for this tool rather than brewing their own.

Regards,

--
Rowan Collins
[IMSoP]

7 years ago by lists@rhsoft.net — view source

unread

Am 05.09.2017 um 13:36 schrieb Lester Caine:

I cannot guess people's thought. I appreciated feedback!

With a decent database layer a lot of the validation you are proposing
is already covered but PDO does not help in this area. Adding another
layer that does not integrate with a storage layer is just adding to the
current mess ...

sorry, but you confuse "input validation" which this topic is about with
something different - input validation and reject bad requests belongs
some layers on top of any storage and should be done as soon as possible

that should even happen long before you open a database connection at
all because when you know the request is bad soon enough you won't talk
to any database, filesystem or whatever storage layer at all

the only question as applicaton developer is how you proceed in which cases

reject the whole request with a error-message
reset form-fields where you don't expect an array as input
reset from-fields with out-of-range input values

here you go:
https://en.wikipedia.org/wiki/Data_validation

7 years ago by Lester Caine — view source

unread

Am 05.09.2017 um 13:36 schrieb Lester Caine:

I cannot guess people's thought. I appreciated feedback!

With a decent database layer a lot of the validation you are proposing
is already covered but PDO does not help in this area. Adding another
layer that does not integrate with a storage layer is just adding to the
current mess ...

sorry, but you confuse "input validation" which this topic is about with
something different - input validation and reject bad requests belongs
some layers on top of any storage and should be done as soon as possible

that should even happen long before you open a database connection at
all because when you know the request is bad soon enough you won't talk
to any database, filesystem or whatever storage layer at all

the only question as applicaton developer is how you proceed in which cases

reject the whole request with a error-message

reset form-fields where you don't expect an array as input

reset from-fields with out-of-range input values

here you go:
https://en.wikipedia.org/wiki/Data_validation

When the database layer provides a complete list of fields and
validation rules as part of it's meta data, it is integral to any GOOD
process. Copying all that data and manually creating filter rules is
just unnecessary work. In addition much of the VALIDATION is best done
at the browser end, and building that code is a lot easier when there is
a standard validation base across all of the layers!

Rejecting crap from hackers that have no format matching the fields on
the browser page is something else and if the data set is corrupt then
yes you can simply skip out before doing anything with it! But the
problem these days is when hackers try injecting things like SQL into
fields they think may be able to get through to the database. Provided
that the validation layer can properly filter that injection requires
knowledge that a string has reason to be rejected. Just as simply type
casting a number to integer or float is only doing a small part of the job.

Typing and validating a field by the metadata constraints has to be the
right way forward?

--
Lester Caine - G8HFL

7 years ago by lists@rhsoft.net — view source

unread

Am 05.09.2017 um 15:44 schrieb Lester Caine:

the only question as applicaton developer is how you proceed in which cases

reject the whole request with a error-message

reset form-fields where you don't expect an array as input

reset from-fields with out-of-range input values

here you go:
https://en.wikipedia.org/wiki/Data_validation

When the database layer provides a complete list of fields and
validation rules as part of it's meta data, it is integral to any GOOD
process

your first error is thinking every input is related to databases at all

Copying all that data and manually creating filter rules is
just unnecessary work. In addition much of the VALIDATION is best done
at the browser end, and building that code is a lot easier when there is
a standard validation base across all of the layers!

NO VALIDATION is best done in the browser end because no attacker ever
will execute your clientside validation code or operate a browser at all

Rejecting crap from hackers that have no format matching the fields on
the browser page is something else and if the data set is corrupt then
yes you can simply skip out before doing anything with it!

and that's what the whole topic is about

7 years ago by Lester Caine — view source

unread

your first error is thinking every input is related to databases at all
So we end up with different code for different types of input?
An array that will work directly into a database save or some other
follow on process without having to think about where the input comes
from has to be the right way ...

Copying all that data and manually creating filter rules is
just unnecessary work. In addition much of the VALIDATION is best done
at the browser end, and building that code is a lot easier when there is
a standard validation base across all of the layers!

NO VALIDATION is best done in the browser end because no attacker ever
will execute your client side validation code or operate a browser at all
Again ... write different code for each area of checking?
My clients are complaining that the browser is not doing as good a job
as it could checking things that it CAN check before passing it back to
the server. YES the server needs to cross check no bugger has bypassed
the browser checks but if the set of data is EXPECTED to have a clean
format, then any corruption can be tagged as a failure, because we have
standard rules on what we are passing.

Rejecting crap from hackers that have no format matching the fields on
the browser page is something else and if the data set is corrupt then
yes you can simply skip out before doing anything with it!

and that's what the whole topic is about
But not at the cost of writing different sets of code to play to each
area where checking SHOULD be done. Stick to a single standard method of
defining the metadata and that already exists in the database layer.

That was what the topic was all about 15 years ago and nothing much has
changed since ... and annotating that data in the code ...

--
Lester Caine - G8HFL

7 years ago by lists@rhsoft.net — view source

unread

Am 05.09.2017 um 18:57 schrieb Lester Caine:

But not at the cost of writing different sets of code to play to each
area where checking SHOULD be done. Stick to a single standard method of
defining the metadata and that already exists in the database layer

ok, to make that point clear:

not every input which needs to be validated or sanitized is related to
a database at all and hence input validation can't be done in the
database-layer and only there by definition for everything

frankly since 3 weeks our core-application is at a level where the
database-layer get not loaded at all until inputs are not verfified
because database stuff is on-demand and 80% of all requests when there
is some traffic within 80% seconds don't load the database layer because
of smart caching

form-input is validated, checked against CSRF-tokens, captcha and
after all the validations are fine the database layer becomes part of
the game - that alone brought again 43% higher requests/second on a
already highly optimizied codebase

7 years ago by Yasuo Ohgaki — view source

unread

Hi Lester,

I always make some mistakes. s/are/is

There is one principle that developers is better to follow.
https://en.wikipedia.org/wiki/Fail-fast
If we follow this principle, validation at controller makes sense.

Anyway, thank you for pointer for PDO validation. I didn't notice the
project. We may cooperate so that there aren't unnecessary validaiton
rule incompatibilities.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

7 years ago by Lester Caine — view source

unread

There is one principle that developers is better to follow.
https://en.wikipedia.org/wiki/Fail-fast
If we follow this principle, validation at controller makes sense.
Since a large proportion of the data coming in is a result of input into
a previously generated form, the data can often be validated in terms of
basic structure before even needing to decide if the data set needs to
be iterated? If things like 'maximum data size' can be established when
the form is created, any data set larger than that can simply be killed
off.

Anyway, thank you for pointer for PDO validation. I didn't notice the
project. We may cooperate so that there aren't unnecessary validaiton
rule incompatibilities
I've been pushing the idea of a single method of managing metadata for a
long time. Most of the 'checking' loading down PHP now still misses the
point and the database style of rules has been around since long before
PDO and other abstractions messed it up. A single standard set of rules
that can be used across the board from variable creation to checking
data going out to forms and returns coming back and data between
operations such as database activity. This is NOT 'typing' since that
lacks the vast majority of checks that a decent validation will handle,
but the much finer details such as limits and value sets. There is a
vast discrepancy in how this is handled across databases, but the SQL
standard does provide a base which databases are slowly evolving towards.

--
Lester Caine - G8HFL

7 years ago by Rowan Collins — view source

unread

I've been pushing the idea of a single method of managing metadata for
a long time.
A single standard set of rules
that can be used across the board from variable creation to checking
data going out to forms and returns coming back and data between
operations such as database activity.

Sounds great, and sounds very much like you agree with Yasuo's aims, but have some ideas on how the rules could be structured or expressed differently. Can you give any examples of how you might express these rules, so we can compare with Yasuo's ext/filter based syntax, and with some of the other validation libraries which are out there?

Regards,

--
Rowan Collins
[IMSoP]

7 years ago by Lester Caine — view source

unread

I've been pushing the idea of a single method of managing metadata for
a long time.
A single standard set of rules
that can be used across the board from variable creation to checking
data going out to forms and returns coming back and data between
operations such as database activity.
Sounds great, and sounds very much like you agree with Yasuo's aims, but have some ideas on how the rules could be structured or expressed differently. Can you give any examples of how you might express these rules, so we can compare with Yasuo's ext/filter based syntax, and with some of the other validation libraries which are out there?

My only problem with Yasuo's latest offering is once again it adds a
whole new set of defines that have to be mapped to existing metadata
definitions ... That and it is a lot of longhand code using a different
style to existing arrays. We need yet another wrapper to build these
arrays from existing code ...

--
Lester Caine - G8HFL

7 years ago by Rowan Collins — view source

unread

My only problem with Yasuo's latest offering is once again it adds a
whole new set of defines that have to be mapped to existing metadata
definitions ... That and it is a lot of longhand code using a different
style to existing arrays. We need yet another wrapper to build these
arrays from existing code ...

The rules have to be defined somehow, and I'm not aware of a standard format that current code is likely to follow. Unless there is already such a standard, I can't see any way to avoid existing code having to be wrapped or amended.

Which is why Yasuo and I have both suggested we work together to come up with such a standard format that can be used or adapted for these different parts of the application. If you have suggestions for how the format should look, we are eager to hear them and see some examples.

Regards,

--
Rowan Collins
[IMSoP]

7 years ago by Dan Ackroyd — view source

unread

If you have suggestions for how the format should look

Don't use a format. Just write code - see below.

Which is why Yasuo and I have both suggested we work together

If you're going to work together and continue the conversation, please
can you move this conversation elsewhere?

It doesn't appear to be actually anything to do with PHP internals.

Since I didn't get much feedbacks during the RFC discussion, I cannot tell
what part is disliked.

Yes you did. You got feedback during the discussion and also during
the vote. For example: http://news.php.net/php.internals/95164

However you continually choose to ignore that feedback.

I will attempt once more, to get the main point through to you.
Perhaps a small amount of repetition, will get it through:

This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.

cheers
Dan
Ack

function validateOrderAmount($value) : int {
$count = preg_match("/[^0-9]*/", $value);

if ($count) {
    throw new InvalidOrderAmount("The order value must contain

only digits.");
}

$value = intval($value);

if ($value < 1) {
    throw new InvalidOrderAmount("The order value must be one or more.");
}

if ($value >= MAX_ORDER_AMOUNT) {
    throw new InvalidOrderAmount(
    "Order value to big. Maximum allowed value is ".MAX_ORDER_AMOUNT
    );
}

return $value;

}

(i'd probably recommend not using exceptions, but instead return
[$valid, $value] to allow validating multiple items without having to
use exceptions for flow control.)

7 years ago by Rowan Collins — view source

unread

On 6 September 2017 at 12:15, Rowan Collins rowan.collins@gmail.com
wrote:

If you have suggestions for how the format should look

Don't use a format. Just write code - see below.

I'm going to assume that the code you posted was something of a straw man, and you're not actually advocating people copy 20 lines of code for every variable they want to validate.

As soon as you have some shared functions, you have some design decisions about how those functions are named, what arguments they take, etc. When used for a lot of variables, that will resemble some kind of DSL, and that's all I meant by "format".

This type of library should be done in PHP, not in C.

I can certainly see the argument for that. In that case, how would you like to see the language evolve:

Should we deprecate ext/filter, to encourage people to use better implementations in userland?

Should the language include some set of basic primitives that these userland implementations can use, rather than having to endlessly reimplement things like a regex for detecting a valid integer?

Or do you view ext/filter's current scope as about right, and it should be neither extended nor reduced?

Regards,

--
Rowan Collins
[IMSoP]

7 years ago by Dan Ackroyd — view source

unread

I'm going to assume that the code you posted was something of a straw
man, and you're not actually advocating people copy 20 lines of code for
every variable they want to validate.

You assume wrong. No it's not, and yes I am.

I can point a junior developer at the function and they can understand it.

If I ask that junior developer to add an extra rule that doesn't
currently exist, they can without having to dive into a full library
of validation code.

If I need to modify the validation based on extra input (e.g whether
the user has already made several purchases, or whether they're a
brand new signup), it's trivial to add that to the function.

This is one of the times where code re-use through copying and pasting
is far superior to trying to make stuff "simple" by going through an
array based 'specification'. It turns out that that doesn't save much
time to begin with, and then becomes hard to manage when your
requirements get more complication.

I can certainly see the argument for that. In that case, how would
you like to see the language evolve:

I do not believe that trying to design features through an email
conversation, is a productive use of individual peoples time. But
also, this list is sent to thousands of people. The number of back and
forth messages required to (possibly) come up with a decent design
mulitplied thousands of people is a huge time sink.

I believe it is far more productive to come up with a good idea
off-list, and then present an almost finished version for discussion,
rather than design features from scratch via this list.

cheers
Dan
Ack

7 years ago by Rowan Collins — view source

unread

On 6 September 2017 at 13:31, Rowan Collins rowan.collins@gmail.com
wrote:

I'm going to assume that the code you posted was something of a straw
man, and you're not actually advocating people copy 20 lines of code
for
every variable they want to validate.

You assume wrong. No it's not, and yes I am.

I can point a junior developer at the function and they can understand
it.

If I ask that junior developer to add an extra rule that doesn't
currently exist, they can without having to dive into a full library
of validation code.

I can certainly agree that a complex DSL might be more pain than it's worth, but copying around a regex and all its scaffolding is surely worse than a clearly named function like validate_non_negative_int?

That's why I asked in my last email if you thought ext/filter should be removed, and perhaps replaced by some more straightforward primitives, or just left as it is. If a broad strategic question like that feels too much like "wasting time on the list designing features" to you, I'm not sure what you would find acceptable discussion.

Regards,

--
Rowan Collins
[IMSoP]

7 years ago by Tony Marston — view source

unread

"Dan Ackroyd" wrote in message
news:CA+kxMuSL1kEW60S7DFJb06+r2Q3rC1ueeWU1jAP78FY65aJoDg@mail.gmail.com...

On 6 September 2017 at 13:31, Rowan Collins rowan.collins@gmail.com
wrote:

I'm going to assume that the code you posted was something of a straw
man, and you're not actually advocating people copy 20 lines of code for
every variable they want to validate.

You assume wrong. No it's not, and yes I am.

I can point a junior developer at the function and they can understand it.

If I ask that junior developer to add an extra rule that doesn't
currently exist, they can without having to dive into a full library
of validation code.

If I need to modify the validation based on extra input (e.g whether
the user has already made several purchases, or whether they're a
brand new signup), it's trivial to add that to the function.

This is one of the times where code re-use through copying and pasting
is far superior to trying to make stuff "simple" by going through an
array based 'specification'. It turns out that that doesn't save much
time to begin with, and then becomes hard to manage when your
requirements get more complication.

As a person who has been developing database applications for several
decades and with PHP since 2003 I'd like to chip in with my 2 cent's worth.
Firstly I agree with Dan's statement:

This type of library should be done in PHP, not in C.

Secondly, there is absolutely no way that you can construct a standard
library which can execute all the possible validation rules that may exist.
In my not inconsiderable experience there are two types of validation:

Primary validation, where each field is validated against the column
specifications in the database to ensure that the value can be written to
that column without causing an error. For example this checks that a number
is a number, a data is a date, a required field is not null, etc.
Secondary validation, where additional validation/business rules are
applied such as comparing the values from several fields. For example, to
check that START_DATE is not later tyhan END_DATE.

Primary validation is easy to automate. I have a separate class for each
database table, and each class contains an array of field specifications.
This is never written by hand as it is produced by my Data Dictionary which
imports data from the database schema then exports that data in the form of
table class files and table structure files. When data is sent to a table
class for inserting or updating in the database I have written a standard
validation procedure which takes two arrays - an array of field=value pairs
and a array of field=specifications - and then checks that each field
conforms to its specifications. This validation procedure is built into the
framework and executed automatically before any data is written to the
database, so requires absolutely no intervention by the developer.

Secondary validation cannot be automated, so it requires additional code to
be inserted into the relevant validation method. There are several of these
which are defined in my abstract table class and which are executed
automatically at a predetermined point in the processing cycle. These
methods are defined in the abstract class but are empty. If specific code is
required then the empty class can be copied from the abstract class to the
concrete class where it can be filled with the necessary code.

If there are any developers out there who are still writing code to perform
primary validation then you may learn something from my implementation.

If there are any developers out there who think that secondary validation
can be automated I can only say "dream on".

--
Tony Marston

7 years ago by Yasuo Ohgaki — view source

unread

Hi Tony,

On Thu, Sep 7, 2017 at 5:40 PM, Tony Marston TonyMarston@hotmail.com
wrote:

"Dan Ackroyd" wrote in message news:CA+kxMuSL1kEW60S7DFJb06+r
2Q3rC1ueeWU1jAP78FY65aJoDg@mail.gmail.com...

On 6 September 2017 at 13:31, Rowan Collins rowan.collins@gmail.com
wrote:

I'm going to assume that the code you posted was something of a straw
man, and you're not actually advocating people copy 20 lines of code for
every variable they want to validate.

You assume wrong. No it's not, and yes I am.

I can point a junior developer at the function and they can understand it.

If I ask that junior developer to add an extra rule that doesn't
currently exist, they can without having to dive into a full library
of validation code.

If I need to modify the validation based on extra input (e.g whether
the user has already made several purchases, or whether they're a
brand new signup), it's trivial to add that to the function.

This is one of the times where code re-use through copying and pasting
is far superior to trying to make stuff "simple" by going through an
array based 'specification'. It turns out that that doesn't save much
time to begin with, and then becomes hard to manage when your
requirements get more complication.

As a person who has been developing database applications for several
decades and with PHP since 2003 I'd like to chip in with my 2 cent's worth.
Firstly I agree with Dan's statement:

This type of library should be done in PHP, not in C.

Secondly, there is absolutely no way that you can construct a standard
library which can execute all the possible validation rules that may exist.
In my not inconsiderable experience there are two types of validation:

Primary validation, where each field is validated against the column
specifications in the database to ensure that the value can be written to
that column without causing an error. For example this checks that a number
is a number, a data is a date, a required field is not null, etc.

Secondary validation, where additional validation/business rules are
applied such as comparing the values from several fields. For example, to
check that START_DATE is not later tyhan END_DATE.

Primary validation is easy to automate. I have a separate class for each
database table, and each class contains an array of field specifications.
This is never written by hand as it is produced by my Data Dictionary which
imports data from the database schema then exports that data in the form of
table class files and table structure files. When data is sent to a table
class for inserting or updating in the database I have written a standard
validation procedure which takes two arrays - an array of field=value pairs
and a array of field=specifications - and then checks that each field
conforms to its specifications. This validation procedure is built into the
framework and executed automatically before any data is written to the
database, so requires absolutely no intervention by the developer.

Secondary validation cannot be automated, so it requires additional code
to be inserted into the relevant validation method. There are several of
these which are defined in my abstract table class and which are executed
automatically at a predetermined point in the processing cycle. These
methods are defined in the abstract class but are empty. If specific code
is required then the empty class can be copied from the abstract class to
the concrete class where it can be filled with the necessary code.

If there are any developers out there who are still writing code to
perform primary validation then you may learn something from my
implementation.

If there are any developers out there who think that secondary validation
can be automated I can only say "dream on".

Please let me explain rationale behind input validation at outermost trust
boundary.
There are 3 reasons why I would like propose the validation. All of 3
requires
validation at outermost trust boundary.

Security reasons
Input validation should be done with Fail Fast manner.
Design by Contract (DbC or Contract Programming)
In order DbC to work, validations at outermost boundary is mandatory.
With DbC, all inputs are validated inside functions/methods to make sure
correct program executions.

However, almost all checks (in fact, all checks done by DbC support)
are disabled for production. How to make sure program works correctly?
All inputs data must be validated at outermost boundary when DbC is
disabled. Otherwise, DbC may not work. (DbC is supposed to achieve
both secure and efficient code execution.)

Native PHP Types
Although my validate module is designed not to do unwanted conversions,
but it converts basic types to PHP native types by default. (This can be
disabled)
With this conversion at outermost trust boundary, native PHP type works
fluently.

Although, my current primary goal is 1, but 2 and 3 is important as well.

2 is important especially. Providing DbC without proper basic validation
feature does not make much sense, and could be disaster.
Users may validate input with their own validation library, but my guess
is pessimistic. User wouldn't do proper validation due to too loose
validation libraries and rules. There are too few validators that do
true validations that meet requirements for 1 and 2. IMHO, even if
there are good enough validators, PHP should provide usable validator
for core features. (DbC is not implemented, though)

I hope you understand my intentions and accept the feature in core.
Feature for core should be in core. IMO.

Primary validation, where each field is validated against the column
specifications in the database to ensure that the value can be written to
that column without causing an error. For example this checks that a number
is a number, a data is a date, a required field is not null, etc.

Secondary validation, where additional validation/business rules are
applied such as comparing the values from several fields. For example, to
check that START_DATE is not later than END_DATE.

Validation rules for input, logic and database may differ.
Suppose you validate "user comment" data.
Input: 0 - 10240 bytes - Input might have to allow larger size
than logic. i.e. lacks client side validation.
Logic: 10 - 1024 bytes - Logic may require smaller range as
correct data.
Database: 0 - 102400 bytes - Database may allow much larger size for future
extension.

Under ideal situation, all of these may be the same but they are not in
real world.

I wouldn't aim to consolidate all validations, but I would like to avoid
unnecessary
incompatibilities so that different validations can cooperate if it is
possible.

I'm very interested in PDO level validation because SQLite3 could be very
dangerous.
(i.e. Type affinity allows store strings in int/float/date/etc) It may be
useful if PDO
can simply use "validate" module's rule or API.

BTW, Input validation should only validate format(used char, length, range,
encoding)
if we follow single responsibility principle. Logical correctness is upto
logic. i.e. Model in
MVC.

Anyway, goal is providing usable basic validator for core features and
security.
Required trade offs may be allowed.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

7 years ago by Tony Marston — view source

unread

"Yasuo Ohgaki" wrote in message
news:CAGa2bXa4UvkL-ZsLAB2bF05L4q_oduixSzVvYzu9nddkSVttXw@mail.gmail.com...

Hi Tony,

<snip> >> >> As a person who has been developing database applications for several >> decades and with PHP since 2003 I'd like to chip in with my 2 cent's >> worth. >> Firstly I agree with Dan's statement: >> >> This type of library should be done in PHP, not in C. >> >> Secondly, there is absolutely no way that you can construct a standard >> library which can execute all the possible validation rules that may >> exist. >> In my not inconsiderable experience there are two types of validation: >> 1) Primary validation, where each field is validated against the column >> specifications in the database to ensure that the value can be written to >> that column without causing an error. For example this checks that a >> number >> is a number, a data is a date, a required field is not null, etc. >> 2) Secondary validation, where additional validation/business rules are >> applied such as comparing the values from several fields. For example, to >> check that START_DATE is not later than END_DATE. >> >> Primary validation is easy to automate. I have a separate class for each >> database table, and each class contains an array of field specifications. >> This is never written by hand as it is produced by my Data Dictionary >> which >> imports data from the database schema then exports that data in the form >> of >> table class files and table structure files. When data is sent to a table >> class for inserting or updating in the database I have written a standard >> validation procedure which takes two arrays - an array of field=value >> pairs >> and a array of field=specifications - and then checks that each field >> conforms to its specifications. This validation procedure is built into >> the >> framework and executed automatically before any data is written to the >> database, so requires absolutely no intervention by the developer. >> >> Secondary validation cannot be automated, so it requires additional code >> to be inserted into the relevant validation method. There are several of >> these which are defined in my abstract table class and which are executed >> automatically at a predetermined point in the processing cycle. These >> methods are defined in the abstract class but are empty. If specific code >> is required then the empty class can be copied from the abstract class to >> the concrete class where it can be filled with the necessary code. >> >> If there are any developers out there who are still writing code to >> perform primary validation then you may learn something from my >> implementation. >> >> If there are any developers out there who think that secondary validation >> can be automated I can only say "dream on". >> > >Please let me explain rationale behind input validation at outermost trust >boundary. There are 3 reasons why I would like propose the validation. All >of 3 >requires validation at outermost trust boundary. > >1. Security reasons >Input validation should be done with Fail Fast manner.

The language should only provide the basic features which allow values to be
validated. That is what the filter functions are for. All that is necessary
is for user input to be validated before any attempt is made to write it to
the database.

Design by Contract (DbC or Contract Programming)
In order DbC to work, validations at outermost boundary is mandatory.
With DbC, all inputs are validated inside functions/methods to make sure
correct program executions.

Irrelevant. DbC is a methodology which PHP was never designed to support,
and I see no reason why it should. If you really want DbC then switch to a
language which supports it, or use a third-party extension which provides
supports.

However, almost all checks (in fact, all checks done by DbC support)
are disabled for production. How to make sure program works correctly?
All inputs data must be validated at outermost boundary when DbC is
disabled. Otherwise, DbC may not work. (DbC is supposed to achieve
both secure and efficient code execution.)

Native PHP Types
Although my validate module is designed not to do unwanted conversions,
but it converts basic types to PHP native types by default. (This can be
disabled) With this conversion at outermost trust boundary, native PHP type
works
fluently.

What is the difference between a basic type and a PHP native type?

Although, my current primary goal is 1, but 2 and 3 is important as well.

2 is important especially. Providing DbC without proper basic validation
feature does not make much sense, and could be disaster.
Users may validate input with their own validation library, but my guess
is pessimistic. User wouldn't do proper validation due to too loose
validation libraries and rules. There are too few validators that do
true validations that meet requirements for 1 and 2. IMHO, even if
there are good enough validators, PHP should provide usable validator
for core features. (DbC is not implemented, though)

It does, in the form of the filter functions.

I hope you understand my intentions and accept the feature in core.
Feature for core should be in core. IMO.

The filter functions are already in core. How these functions are used is
down to userland code.

Primary validation, where each field is validated against the column
specifications in the database to ensure that the value can be written to
that column without causing an error. For example this checks that a number
is a number, a data is a date, a required field is not null, etc.

Secondary validation, where additional validation/business rules are
applied such as comparing the values from several fields. For example, to
check that START_DATE is not later than END_DATE.

Validation rules for input, logic and database may differ.
Suppose you validate "user comment" data.
Input: 0 - 10240 bytes - Input might have to allow larger size
than logic. i.e. lacks client side validation.
Logic: 10 - 1024 bytes - Logic may require smaller range as
correct data.
Database: 0 - 102400 bytes - Database may allow much larger size for future
extension.

Under ideal situation, all of these may be the same but they are not in
real world.

I wouldn't aim to consolidate all validations, but I would like to avoid
unnecessary
incompatibilities so that different validations can cooperate if it is
possible.

What exactly are these "unnecessary incompatibilities"?

I'm very interested in PDO level validation because SQLite3 could be very
dangerous.

Anything which is misused can be dangerous. It is almost impossible to
provide a function and prevent stupid people from misusing it.

(i.e. Type affinity allows store strings in int/float/date/etc) It may be
useful if PDO
can simply use "validate" module's rule or API.

BTW, Input validation should only validate format(used char, length, range,
encoding)
if we follow single responsibility principle. Logical correctness is upto
logic. i.e. Model in
MVC.

Anyway, goal is providing usable basic validator for core features and
security.

If you wish to improve the filter functions ten go ahead. Anything more than
this would be a step too far.

Required trade offs may be allowed.

Do not waste time by trying to add into core what should be done in userland
code.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

--
Tony Marston

7 years ago by Yasuo Ohgaki — view source

unread

Hi Tony,

On Sat, Sep 9, 2017 at 6:26 PM, Tony Marston TonyMarston@hotmail.com
wrote:

"Yasuo Ohgaki" wrote in message news:CAGa2bXa4UvkL-ZsLAB2bF05L
4q_oduixSzVvYzu9nddkSVttXw@mail.gmail.com...

Hi Tony,
<snip>

As a person who has been developing database applications for several
decades and with PHP since 2003 I'd like to chip in with my 2 cent's
worth.
Firstly I agree with Dan's statement:

This type of library should be done in PHP, not in C.

Secondly, there is absolutely no way that you can construct a standard
library which can execute all the possible validation rules that may
exist.
In my not inconsiderable experience there are two types of validation:

Primary validation, where each field is validated against the column
specifications in the database to ensure that the value can be written to
that column without causing an error. For example this checks that a
number
is a number, a data is a date, a required field is not null, etc.

Secondary validation, where additional validation/business rules are
applied such as comparing the values from several fields. For example, to
check that START_DATE is not later than END_DATE.

Primary validation is easy to automate. I have a separate class for each
database table, and each class contains an array of field specifications.
This is never written by hand as it is produced by my Data Dictionary
which
imports data from the database schema then exports that data in the form
of
table class files and table structure files. When data is sent to a table
class for inserting or updating in the database I have written a standard
validation procedure which takes two arrays - an array of field=value
pairs
and a array of field=specifications - and then checks that each field
conforms to its specifications. This validation procedure is built into
the
framework and executed automatically before any data is written to the
database, so requires absolutely no intervention by the developer.

Secondary validation cannot be automated, so it requires additional code
to be inserted into the relevant validation method. There are several of
these which are defined in my abstract table class and which are executed
automatically at a predetermined point in the processing cycle. These
methods are defined in the abstract class but are empty. If specific code
is required then the empty class can be copied from the abstract class to
the concrete class where it can be filled with the necessary code.

If there are any developers out there who are still writing code to
perform primary validation then you may learn something from my
implementation.

If there are any developers out there who think that secondary validation
can be automated I can only say "dream on".

Please let me explain rationale behind input validation at outermost trust
boundary. There are 3 reasons why I would like propose the validation.
All of 3
requires validation at outermost trust boundary.

Security reasons
Input validation should be done with Fail Fast manner.

The language should only provide the basic features which allow values to
be validated. That is what the filter functions are for. All that is
necessary is for user input to be validated before any attempt is made to
write it to the database.

The reason why data should be validated at outermost trust boundary is
explained by me and other. Validation at database level is simply too late
for security purposes.

Input validations must be done at outermost boundary for the best security.
This is a secure coding best practice.

Design by Contract (DbC or Contract Programming)

In order DbC to work, validations at outermost boundary is mandatory.
With DbC, all inputs are validated inside functions/methods to make sure
correct program executions.

Irrelevant. DbC is a methodology which PHP was never designed to support,
and I see no reason why it should. If you really want DbC then switch to a
language which supports it, or use a third-party extension which provides
supports.

DbC is ad-hoc. No BC nor shortcomings.
All most all languages including PHP have support feature for it in some
forms.
If PHP is designed for DbC or not is irrelevant.

One can totally ignore DbC support just like some D users do, yet
DbC can achieve both better security and performance with proper design
and usage. DbC is extremely useful for building solid and faster app when
it is used properly.

However, almost all checks (in fact, all checks done by DbC support)

are disabled for production. How to make sure program works correctly?
All inputs data must be validated at outermost boundary when DbC is
disabled. Otherwise, DbC may not work. (DbC is supposed to achieve
both secure and efficient code execution.)

Native PHP Types

Although my validate module is designed not to do unwanted conversions,
but it converts basic types to PHP native types by default. (This can be
disabled) With this conversion at outermost trust boundary, native PHP
type works
fluently.

What is the difference between a basic type and a PHP native type?

PHP native types are NULL/BOOL/INT/FLOAT/STRING/ARRAY/OBJECT.

Almost all inputs are "text" in web apps. Data comes from clients is "text".
So they are "STRING", while PHP native types are zend_bool/zend_long/double/
zend_string/hash/object.

While basic type form string is almost the same as PHP native type, but
there is a little difference. e.g. 't' is TRUE, '999999999999999999999999'
is valid as
integer, but not for PHP int type.

Although, my current primary goal is 1, but 2 and 3 is important as well.

2 is important especially. Providing DbC without proper basic validation
feature does not make much sense, and could be disaster.
Users may validate input with their own validation library, but my guess
is pessimistic. User wouldn't do proper validation due to too loose
validation libraries and rules. There are too few validators that do
true validations that meet requirements for 1 and 2. IMHO, even if
there are good enough validators, PHP should provide usable validator
for core features. (DbC is not implemented, though)

It does, in the form of the filter functions.

It seems you haven't try to use filter module seriously.
It simply does not have enough feature for input validations.
e.g. You cannot validate "strings".

I hope you understand my intentions and accept the feature in core.

Feature for core should be in core. IMO.

The filter functions are already in core. How these functions are used is
down to userland code.

I suppose filter module is not used for validations much, since
it cannot validate string without my RFC for filter.

Primary validation, where each field is validated against the column

specifications in the database to ensure that the value can be written to
that column without causing an error. For example this checks that a
number
is a number, a data is a date, a required field is not null, etc.

Secondary validation, where additional validation/business rules are

applied such as comparing the values from several fields. For example, to
check that START_DATE is not later than END_DATE.

Validation rules for input, logic and database may differ.
Suppose you validate "user comment" data.
Input: 0 - 10240 bytes - Input might have to allow larger size
than logic. i.e. lacks client side validation.
Logic: 10 - 1024 bytes - Logic may require smaller range as
correct data.
Database: 0 - 102400 bytes - Database may allow much larger size for
future
extension.

Under ideal situation, all of these may be the same but they are not in
real world.

I wouldn't aim to consolidate all validations, but I would like to avoid
unnecessary
incompatibilities so that different validations can cooperate if it is
possible.

What exactly are these "unnecessary incompatibilities"?

I don't know either now, but there would be some.

I'm very interested in PDO level validation because SQLite3 could be very

dangerous.

Anything which is misused can be dangerous. It is almost impossible to
provide a function and prevent stupid people from misusing it.

Correct. However, too many users are ignoring the fact SQLite3 has type
affinity
that allows strings for any types. This is just an example for better
security.

(i.e. Type affinity allows store strings in int/float/date/etc) It may be

useful if PDO
can simply use "validate" module's rule or API.

BTW, Input validation should only validate format(used char, length,
range,
encoding)
if we follow single responsibility principle. Logical correctness is upto
logic. i.e. Model in
MVC.

Anyway, goal is providing usable basic validator for core features and
security.

If you wish to improve the filter functions ten go ahead. Anything more
than this would be a step too far.

I did it already by RFC with PoC patch.
https://wiki.php.net/rfc/add_validate_functions_to_filter

Required trade offs may be allowed.

Do not waste time by trying to add into core what should be done in
userland code.

Proper input validation is the most important task in secure coding.
https://www.securecoding.cert.org/confluence/display/seccode/Top+10+Secure+Coding+Practices
Nonetheless, I rarely see app that has proper input validations. It would
be
nice to have module for it with proper document.

"All that is necessary is for user input to be validated before any attempt
is made to write it to the database."

This fine for database, but not for app. There are too many codes that
don't even require database. Even when database is used, there are too many
cases database level validation is too late.

BTW, PHP script implemented validator cannot be faster than native C module
function. As you know, function call overhead is not cheap. We have number
of
array functions for this reason. Why not for validation which must be
called always?

Regards,

P.S. Many of us are confused what application level validation is.
Application level input validation is $_GET/$_POST/$_COOKIE/$_SERVER/$_FILES
validation before they are used by app codes. "Validate" module is
intended for this.
Logic(Model in MVC) or DB level validations are another input validations.
It cannot be replaced by others with proper design. i.e. Fail Fast, Single
Responsibility
principle.

--
Yasuo Ohgaki
yohgaki@ohgaki.net

7 years ago by Stephen Reay — view source

unread

It seems you haven't try to use filter module seriously.
It simply does not have enough feature for input validations.
e.g. You cannot validate "strings".

Yasuo,

I’ve asked previously what your proposal actually offers over the filter functions, and got no response, so please elaborate on this?

Can you show a concrete example that cannot be validated in user land currently, using the filter functions as a base?

Cheers

Stephen

7 years ago by Yasuo Ohgaki — view source

unread

Hi Stephen,

On Mon, Sep 11, 2017 at 6:37 PM, Stephen Reay php-lists@koalephant.com
wrote:

It seems you haven't try to use filter module seriously.
It simply does not have enough feature for input validations.
e.g. You cannot validate "strings".

Yasuo,

I’ve asked previously what your proposal actually offers over the filter
functions, and got no response, so please elaborate on this?

Can you show a concrete example that cannot be validated in user land
currently, using the filter functions as a base?

FILTER_VALIDATE_REGEXP is not good enough simply.
PCRE is known that it is vulnerable to regex DoS still. (as well as
Oniguruma)
Users should avoid regex validation whenever it is possible also to avoid
various
risks.

In addition, current filter module does not provide nested array validation
array key validation, etc. It's not true validation neither. It does not
provide
simple length, min/max validations. It does non explicit conversions (i.e.
trim), etc.
Length, min/max validation is mandatory validation if you would like to
follow
ISO 27000 requirement.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

7 years ago by Stephen Reay — view source

unread

Hi Stephen,

On Mon, Sep 11, 2017 at 6:37 PM, Stephen Reay <php-lists@koalephant.com mailto:php-lists@koalephant.com>
wrote:

It seems you haven't try to use filter module seriously.
It simply does not have enough feature for input validations.
e.g. You cannot validate "strings".

Yasuo,

I’ve asked previously what your proposal actually offers over the filter
functions, and got no response, so please elaborate on this?

Can you show a concrete example that cannot be validated in user land
currently, using the filter functions as a base?

FILTER_VALIDATE_REGEXP is not good enough simply.
PCRE is known that it is vulnerable to regex DoS still. (as well as
Oniguruma)
Users should avoid regex validation whenever it is possible also to avoid
various
risks.

In addition, current filter module does not provide nested array validation
array key validation, etc. It's not true validation neither. It does not
provide
simple length, min/max validations. It does non explicit conversions (i.e.
trim), etc.
Length, min/max validation is mandatory validation if you would like to
follow
ISO 27000 requirement.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net mailto:yohgaki@ohgaki.net

So, you still didn’t actually provide an example. I guess you’re talking about character class validation or something else equally “simple”, because I can’t imagine what else would be a common enough case that you’d want to have built-in rules for, and that you wouldn’t internally use RegExp to test anyway.

Ok so we can’t use filter_var() rules to validate that a string field is an Alpha or AlphaNum, between 4 and 8 characters long (technically you could pass mb_strlen() to the INT filter with {min,max}_range options set to get the length validation, but I’ll grant you that is kind of a crappy workaround right now)

Why not stop trying to re-invent every single feature already present in PHP (yes, I’ve been paying attention to all your other proposals), and just add the functionality that’s missing:

A FILTER_VALIDATE_STRING filter, with “Options” of min => ?int, max => ?int and “Flags” of FILTER_FLAG_ALPHA, FILTER_FLAG_NUMERIC (possibly a built in bit mask “FILTER_FLAG_ALPHANUMERIC” ?)

Lastly: it may not be the format you personally want, but the filter extension does have the filter_{input,var}_array functions. Claiming something doesn’t exist because it doesn’t work exactly how you would like it to, makes you seem immature and petty, IMO.

7 years ago by Yasuo Ohgaki — view source

unread

Stephen,

On Tue, Sep 12, 2017 at 12:22 AM, Stephen Reay php-lists@koalephant.com
wrote:

Hi Stephen,

On Mon, Sep 11, 2017 at 6:37 PM, Stephen Reay php-lists@koalephant.com
wrote:

It seems you haven't try to use filter module seriously.
It simply does not have enough feature for input validations.
e.g. You cannot validate "strings".

Yasuo,

I’ve asked previously what your proposal actually offers over the filter
functions, and got no response, so please elaborate on this?

Can you show a concrete example that cannot be validated in user land
currently, using the filter functions as a base?

FILTER_VALIDATE_REGEXP is not good enough simply.
PCRE is known that it is vulnerable to regex DoS still. (as well as
Oniguruma)
Users should avoid regex validation whenever it is possible also to avoid
various
risks.

In addition, current filter module does not provide nested array validation
array key validation, etc. It's not true validation neither. It does not
provide
simple length, min/max validations. It does non explicit conversions (i.e.
trim), etc.
Length, min/max validation is mandatory validation if you would like to
follow
ISO 27000 requirement.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

So, you still didn’t actually provide an example. I guess you’re talking
about character class validation or something else equally “simple”,
because I can’t imagine what else would be a common enough case that you’d
want to have built-in rules for, and that you wouldn’t internally use
RegExp to test anyway.

Your request is like "Devil's Proof". Example code that cannot do things
with existing API cannot exist with meaningful manner. It can be explained
why it cannot, though. Try what "validate" string validator can do,
Then you'll see.

$input = [
'defined_but_should_not_exist' => 'Developer should not allow unwanted
value',
'invalid_utf8_key_should_not_be_allowed' => 'Developer should validate
key value as well',
'utf8_text' => 'Validator should be able to allow UTF-8 and validate its
validity at least',
'default_must_be_safe' => 'Crackers send all kinds of chars. CNTRL chars
must not be allowed by default',
'array' => [
'complex' => 1,
'nested' => 'any validation rule should be able to be applied',
'array' => 1,
'key_should_be_validated_also' => 1,
'array' => [
'any_num_of_nesting' => 'is allowed',
],
],
'array_num_elements_must_be_validated' => [
"a", "b", "c", "d", "e", "f", "and so on", "values must be able to be
validated as user wants",
],
];

There is no STRING validation filter currently. This fact alone,
it could be said "filter cannot do string validation currently".

List of problems in current validation filter

no STRING validator currently
it allows any inputs by default
it does not allow multiple rules that allows complex validation rules
for string
it does not have callback validator
it does not have key value validation (note: PHP's key could be binary)
it does not validate num of elements in array.
it cannot forbids unwanted elements in array.
it cannot validate "char encoding".
it does not enforce white listing.
and so on

These are the list that "filter" cannot do.

Ok so we can’t use filter_var() rules to validate that a string field is an

Alpha or AlphaNum, between 4 and 8 characters long (technically you could
pass mb_strlen() to the INT filter with {min,max}_range options set to get
the length validation, but I’ll grant you that is kind of a crappy
workaround right now)

Why not stop trying to re-invent every single feature already present in
PHP (yes, I’ve been paying attention to all your other proposals), and just
add the functionality that’s missing:

https://wiki.php.net/rfc/add_validate_functions_to_filter
It's declined. You should have supported this RFC if you would like to
add features to filter.
(I'm glad there is a new RFC supporter regardless of occasion)

I don't mind this result much.
Adding features to "filter" has some of shortcomings mentioned above
even with my proposal.

A FILTER_VALIDATE_STRING filter, with “Options” of min => ?int, max

=> ?int and “Flags” of FILTER_FLAG_ALPHA, FILTER_FLAG_NUMERIC (possibly a
built in bit mask “FILTER_FLAG_ALPHANUMERIC” ?)

Simply adding these wouldn't work well as validator because

Filter is designed for black listing

As you may know, all of security standards/guidelines require

White listing for validation

We may change "filter", but it requires BC.

Lastly: it may not be the format you personally want, but the filter
extension does have the filter_{input,var}_array functions. Claiming
something doesn’t exist because it doesn’t work exactly how you would like
it to, makes you seem immature and petty, IMO.

Discussion is confusing because you ignore this RFC result.
https://wiki.php.net/rfc/add_validate_functions_to_filter
This RFC proposes filter module improvement while keeping compatibility.

I understand your point. This exactly the same reason why I proposed
"improvement" at first, not new extension.

I don't understand why you insist already failed attempt repeatedly.

Would you like me to propose previous RFC again?
and implement "ture validation" with filter?
I don't mind implementing it if you would like to update the RFC and it
passes.
I must use "white list" as much as possible.

Regards,

P.S. "Filter" module is black listing module. "Validate" is white listing
module.
Even with BC, mixing them would result in confusing FLAGs and codes.
Codes may be cleaned up later, but FLAGs cannot.
We should consider this also.

--
Yasuo Ohgaki
yohgaki@ohgaki.net

7 years ago by lists@rhsoft.net — view source

unread

Am 11.09.2017 um 23:07 schrieb Yasuo Ohgaki

On Tue, Sep 12, 2017 at 12:22 AM, Stephen Reay php-lists@koalephant.com

So, you still didn’t actually provide an example. I guess you’re talking
about character class validation or something else equally “simple”,
because I can’t imagine what else would be a common enough case that you’d
want to have built-in rules for, and that you wouldn’t internally use
RegExp to test anyway.

Your request is like "Devil's Proof". Example code that cannot do things
with existing API cannot exist with meaningful manner. It can be explained
why it cannot, though. Try what "validate" string validator can do,
Then you'll see.

There is no STRING validation filter currently. This fact alone,
it could be said "filter cannot do string validation currently".

List of problems in current validation filter
but you still fail to explain why in the world you don#t try to enhance
the existing filter functions instead invent a new beast leading finally
to have the existin filter functions and your new stuff which share the
same intention

7 years ago by Yasuo Ohgaki — view source

unread

Hi,

Am 11.09.2017 um 23:07 schrieb Yasuo Ohgaki

On Tue, Sep 12, 2017 at 12:22 AM, Stephen Reay php-lists@koalephant.com

So, you still didn’t actually provide an example. I guess you’re
talking
about character class validation or something else equally “simple”,
because I can’t imagine what else would be a common enough case that
you’d
want to have built-in rules for, and that you wouldn’t internally use
RegExp to test anyway.

Your request is like "Devil's Proof". Example code that cannot do things
with existing API cannot exist with meaningful manner. It can be explained
why it cannot, though. Try what "validate" string validator can do,
Then you'll see.

There is no STRING validation filter currently. This fact alone,
it could be said "filter cannot do string validation currently".

List of problems in current validation filter

but you still fail to explain why in the world you don#t try to enhance
the existing filter functions instead invent a new beast leading finally to
have the existin filter functions and your new stuff which share the same
intention

Why don't you read previous RFC and the vote result?
https://wiki.php.net/rfc/add_validate_functions_to_filter

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

7 years ago by Yasuo Ohgaki — view source

unread

Hi,

Hi,

On Tue, Sep 12, 2017 at 6:35 AM, lists@rhsoft.net lists@rhsoft.net
wrote:

Am 11.09.2017 um 23:07 schrieb Yasuo Ohgaki

On Tue, Sep 12, 2017 at 12:22 AM, Stephen Reay <php-lists@koalephant.com

So, you still didn’t actually provide an example. I guess you’re
talking
about character class validation or something else equally “simple”,
because I can’t imagine what else would be a common enough case that
you’d
want to have built-in rules for, and that you wouldn’t internally use
RegExp to test anyway.

Your request is like "Devil's Proof". Example code that cannot do things
with existing API cannot exist with meaningful manner. It can be
explained
why it cannot, though. Try what "validate" string validator can do,
Then you'll see.

There is no STRING validation filter currently. This fact alone,
it could be said "filter cannot do string validation currently".

List of problems in current validation filter

but you still fail to explain why in the world you don#t try to enhance
the existing filter functions instead invent a new beast leading finally to
have the existin filter functions and your new stuff which share the same
intention

Why don't you read previous RFC and the vote result?
https://wiki.php.net/rfc/add_validate_functions_to_filter

I'm a bit surprised by the fact there are "filter improvement" supporters.
You should have participated in the previous RFC discussion.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

7 years ago by lists@rhsoft.net — view source

unread

Am 11.09.2017 um 23:49 schrieb Yasuo Ohgaki:

    but you still fail to explain why in the world you don#t try to
    enhance the existing filter functions instead invent a new beast
    leading finally to have the existin filter functions and your
    new stuff which share the same intention


Why don't you read previous RFC and the vote result?
https://wiki.php.net/rfc/add_validate_functions_to_filter
<https://wiki.php.net/rfc/add_validate_functions_to_filter>

I'm a bit surprised by the fact there are "filter improvement" supporters.
You should have participated in the previous RFC discussion

and i am suprise that you act that stubborn and obviously think when
you give the bike a new name someone will buy it instead really consider
the contras of previous proposals

7 years ago by lists@rhsoft.net — view source

unread

Am 11.09.2017 um 23:39 schrieb Yasuo Ohgaki:

On Tue, Sep 12, 2017 at 6:35 AM, lists@rhsoft.net
but you still fail to explain why in the world you don#t try to
enhance the existing filter functions instead invent a new beast
leading finally to have the existin filter functions and your new
stuff which share the same intention

Why don't you read previous RFC and the vote result?
https://wiki.php.net/rfc/add_validate_functions_to_filter

and why do you not take the contra arguments against how do you think
things should be done into your considerations and believe bikeshed it
with a different name will achieve anything?

it's basially the same as your hash_hkdf() related stuff - you just
ignore everybody and cntinue to ride a dead horse up to a level where
even pure readers of the internals list just have enough and only think
"stop it guy"

7 years ago by Yasuo Ohgaki — view source

unread

Hi,

Am 11.09.2017 um 23:39 schrieb Yasuo Ohgaki:

On Tue, Sep 12, 2017 at 6:35 AM, lists@rhsoft.net but you still fail
to explain why in the world you don#t try to
enhance the existing filter functions instead invent a new beast
leading finally to have the existin filter functions and your new
stuff which share the same intention

Why don't you read previous RFC and the vote result?
https://wiki.php.net/rfc/add_validate_functions_to_filter

and why do you not take the contra arguments against how do you think
things should be done into your considerations and believe bikeshed it with
a different name will achieve anything?

If you understand the difference, there are huge different with respect to
behaviors.

Previous RFC was halfway finished "validation", it's far from "true
validation".

it's basially the same as your hash_hkdf() related stuff - you just ignore

everybody and cntinue to ride a dead horse up to a level where even pure
readers of the internals list just have enough and only think "stop it guy"

hash_hkdf() discussion comes to conclusion finally if you haven't noticed
it.
It is clear now that Nikita and Andrey does not understand the algorithm (
including underlying HMAC and cypto hash characteristics) and RFC.
See the relevant thread for conclusion. (The latest one)

In short, current hash_hkdf() is not only violates RFC 5869, but also
encourages
extremely insecure usage, has unnecessarily incompatible API with respect
to
other hash functions.

and i am suprise that you act that stubborn and obviously think when you
give the bike a new name someone will buy it instead really consider the
contras of previous proposals

"Validate" and "filter improvement" fundamentally different proposal in
fact.
i.e. Validate does true white list validation, while filter improvement is
halfway.
Almost all apps do not implement "proper application level input
validation" yet,
even if all of security guidelines/standards recommends/requires it.

What do you mean by "stubborn"?
Would you like me to try to remove "input validations" from security
guidelines or standards?
If you seriously think so, you're the one should try.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

7 years ago by Yasuo Ohgaki — view source

unread

Hi Stephen,

Would you like me to propose previous RFC again?
and implement "ture validation" with filter?
I don't mind implementing it if you would like to update the RFC and it
passes.
It must use "white list" as much as possible.

BTW, the patch mentioned here
https://wiki.php.net/rfc/add_validate_functions_to_filter
is seriously broken by merge and it does not work currently.
I didn't look into details, but it seems some codes are gone
by merge. Previous patch is written so that there are less
changes.

If you're serious about proposing "filter" improvement, I don't
mind make it work again. (And improve further. It's PoC in the
first place.)

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

7 years ago by Stephen Reay — view source

unread

Stephen,

Hi Stephen,

On Mon, Sep 11, 2017 at 6:37 PM, Stephen Reay <php-lists@koalephant.com mailto:php-lists@koalephant.com>
wrote:

It seems you haven't try to use filter module seriously.
It simply does not have enough feature for input validations.
e.g. You cannot validate "strings".

Yasuo,

I’ve asked previously what your proposal actually offers over the filter
functions, and got no response, so please elaborate on this?

Can you show a concrete example that cannot be validated in user land
currently, using the filter functions as a base?

FILTER_VALIDATE_REGEXP is not good enough simply.
PCRE is known that it is vulnerable to regex DoS still. (as well as
Oniguruma)
Users should avoid regex validation whenever it is possible also to avoid
various
risks.

In addition, current filter module does not provide nested array validation
array key validation, etc. It's not true validation neither. It does not
provide
simple length, min/max validations. It does non explicit conversions (i.e.
trim), etc.
Length, min/max validation is mandatory validation if you would like to
follow
ISO 27000 requirement.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net mailto:yohgaki@ohgaki.net

So, you still didn’t actually provide an example. I guess you’re talking about character class validation or something else equally “simple”, because I can’t imagine what else would be a common enough case that you’d want to have built-in rules for, and that you wouldn’t internally use RegExp to test anyway.

Your request is like "Devil's Proof". Example code that cannot do things
with existing API cannot exist with meaningful manner. It can be explained
why it cannot, though. Try what "validate" string validator can do,
Then you'll see.

$input = [
'defined_but_should_not_exist' => 'Developer should not allow unwanted value',
'invalid_utf8_key_should_not_be_allowed' => 'Developer should validate key value as well',
'utf8_text' => 'Validator should be able to allow UTF-8 and validate its validity at least',
'default_must_be_safe' => 'Crackers send all kinds of chars. CNTRL chars must not be allowed by default',
'array' => [
'complex' => 1,
'nested' => 'any validation rule should be able to be applied',
'array' => 1,
'key_should_be_validated_also' => 1,
'array' => [
'any_num_of_nesting' => 'is allowed',
],
],
'array_num_elements_must_be_validated' => [
"a", "b", "c", "d", "e", "f", "and so on", "values must be able to be validated as user wants",
],
];

There is no STRING validation filter currently. This fact alone,
it could be said "filter cannot do string validation currently".

List of problems in current validation filter

no STRING validator currently

it allows any inputs by default

it does not allow multiple rules that allows complex validation rules for string

it does not have callback validator

it does not have key value validation (note: PHP's key could be binary)

it does not validate num of elements in array.

it cannot forbids unwanted elements in array.

it cannot validate "char encoding".

it does not enforce white listing.

and so on

These are the list that "filter" cannot do.

Ok so we can’t use filter_var() rules to validate that a string field is an Alpha or AlphaNum, between 4 and 8 characters long (technically you could pass mb_strlen() to the INT filter with {min,max}_range options set to get the length validation, but I’ll grant you that is kind of a crappy workaround right now)

Why not stop trying to re-invent every single feature already present in PHP (yes, I’ve been paying attention to all your other proposals), and just add the functionality that’s missing:

https://wiki.php.net/rfc/add_validate_functions_to_filter https://wiki.php.net/rfc/add_validate_functions_to_filter
It's declined. You should have supported this RFC if you would like to add features to filter.
(I'm glad there is a new RFC supporter regardless of occasion)

I don't mind this result much.
Adding features to "filter" has some of shortcomings mentioned above
even with my proposal.

A FILTER_VALIDATE_STRING filter, with “Options” of min => ?int, max => ?int and “Flags” of FILTER_FLAG_ALPHA, FILTER_FLAG_NUMERIC (possibly a built in bit mask “FILTER_FLAG_ALPHANUMERIC” ?)

Simply adding these wouldn't work well as validator because

Filter is designed for black listing

As you may know, all of security standards/guidelines require

White listing for validation

We may change "filter", but it requires BC.

Lastly: it may not be the format you personally want, but the filter extension does have the filter_{input,var}_array functions. Claiming something doesn’t exist because it doesn’t work exactly how you would like it to, makes you seem immature and petty, IMO.

Discussion is confusing because you ignore this RFC result.
https://wiki.php.net/rfc/add_validate_functions_to_filter https://wiki.php.net/rfc/add_validate_functions_to_filter
This RFC proposes filter module improvement while keeping compatibility.

I understand your point. This exactly the same reason why I proposed
"improvement" at first, not new extension.

I don't understand why you insist already failed attempt repeatedly.

Would you like me to propose previous RFC again?
and implement "ture validation" with filter?
I don't mind implementing it if you would like to update the RFC and it passes.
I must use "white list" as much as possible.

Regards,

P.S. "Filter" module is black listing module. "Validate" is white listing module.
Even with BC, mixing them would result in confusing FLAGs and codes.
Codes may be cleaned up later, but FLAGs cannot.
We should consider this also.

--
Yasuo Ohgaki
yohgaki@ohgaki.net mailto:yohgaki@ohgaki.net

I was going to give a lot of detailed replies inline, but I’ve come to the realisation its pointless with you. You really respond to what people say, you just use their comments as jumping off points to re-post your same little rant, ad nauseam.

So here’s the summary. Don’t both replying, because I won’t be reading it.

I never asked for a working code example that is impossible with the current extension. I asked for a simple example of what you wanted to achieve.
More than half the “issues” you claim with the filter extension, are only “valid” if you agree that it needs to do complex array structure validation. I do not agree with this. Userland can iterate an array of rules/input and validate quite easily.
The actual issues with the filter extension could be solved by improving/adding filters.
I already agreed that a string based filter (to test character class and min/max length, etc) would be a good addition. Continuing to bring it up when others have acknowledged something doesn’t help your case, at all.
ACCEPTING that string validation is missing, number/bool/<complex string: url, email, etc> is still whitelisting. I don’t know how they’re implemented in C. Maybe that needs improvement. But a rule saying “validate that $X is a number between 10 and 100” or “validate that $Y is an email address) is whitelisting.

7 years ago by Yasuo Ohgaki — view source

unread

On Tue, Sep 12, 2017 at 1:04 PM, Stephen Reay php-lists@koalephant.com
wrote:

Stephen,

On Tue, Sep 12, 2017 at 12:22 AM, Stephen Reay php-lists@koalephant.com
wrote:

Hi Stephen,

On Mon, Sep 11, 2017 at 6:37 PM, Stephen Reay php-lists@koalephant.com
wrote:

It seems you haven't try to use filter module seriously.
It simply does not have enough feature for input validations.
e.g. You cannot validate "strings".

Yasuo,

I’ve asked previously what your proposal actually offers over the filter
functions, and got no response, so please elaborate on this?

Can you show a concrete example that cannot be validated in user land
currently, using the filter functions as a base?

FILTER_VALIDATE_REGEXP is not good enough simply.
PCRE is known that it is vulnerable to regex DoS still. (as well as
Oniguruma)
Users should avoid regex validation whenever it is possible also to avoid
various
risks.

In addition, current filter module does not provide nested array
validation
array key validation, etc. It's not true validation neither. It does not
provide
simple length, min/max validations. It does non explicit conversions (i.e.
trim), etc.
Length, min/max validation is mandatory validation if you would like to
follow
ISO 27000 requirement.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

So, you still didn’t actually provide an example. I guess you’re
talking about character class validation or something else equally
“simple”, because I can’t imagine what else would be a common enough case
that you’d want to have built-in rules for, and that you wouldn’t
internally use RegExp to test anyway.

Your request is like "Devil's Proof". Example code that cannot do things
with existing API cannot exist with meaningful manner. It can be explained
why it cannot, though. Try what "validate" string validator can do,
Then you'll see.

$input = [
'defined_but_should_not_exist' => 'Developer should not allow unwanted
value',
'invalid_utf8_key_should_not_be_allowed' => 'Developer should
validate key value as well',
'utf8_text' => 'Validator should be able to allow UTF-8 and validate its
validity at least',
'default_must_be_safe' => 'Crackers send all kinds of chars. CNTRL chars
must not be allowed by default',
'array' => [
'complex' => 1,
'nested' => 'any validation rule should be able to be applied',
'array' => 1,
'key_should_be_validated_also' => 1,
'array' => [
'any_num_of_nesting' => 'is allowed',
],
],
'array_num_elements_must_be_validated' => [
"a", "b", "c", "d", "e", "f", "and so on", "values must be able to
be validated as user wants",
],
];

There is no STRING validation filter currently. This fact alone,
it could be said "filter cannot do string validation currently".

List of problems in current validation filter

no STRING validator currently

it allows any inputs by default

it does not allow multiple rules that allows complex validation rules
for string

it does not have callback validator

it does not have key value validation (note: PHP's key could be binary)

it does not validate num of elements in array.

it cannot forbids unwanted elements in array.

it cannot validate "char encoding".

it does not enforce white listing.

and so on

These are the list that "filter" cannot do.

Ok so we can’t use filter_var() rules to validate that a string field is

an Alpha or AlphaNum, between 4 and 8 characters long (technically you
could pass mb_strlen() to the INT filter with {min,max}_range options set
to get the length validation, but I’ll grant you that is kind of a crappy
workaround right now)

Why not stop trying to re-invent every single feature already present in
PHP (yes, I’ve been paying attention to all your other proposals), and just
add the functionality that’s missing:

https://wiki.php.net/rfc/add_validate_functions_to_filter
It's declined. You should have supported this RFC if you would like to
add features to filter.
(I'm glad there is a new RFC supporter regardless of occasion)

I don't mind this result much.
Adding features to "filter" has some of shortcomings mentioned above
even with my proposal.

A FILTER_VALIDATE_STRING filter, with “Options” of min => ?int, max

=> ?int and “Flags” of FILTER_FLAG_ALPHA, FILTER_FLAG_NUMERIC (possibly a
built in bit mask “FILTER_FLAG_ALPHANUMERIC” ?)

Simply adding these wouldn't work well as validator because

Filter is designed for black listing

As you may know, all of security standards/guidelines require

White listing for validation

We may change "filter", but it requires BC.

Lastly: it may not be the format you personally want, but the filter
extension does have the filter_{input,var}_array functions. Claiming
something doesn’t exist because it doesn’t work exactly how you would like
it to, makes you seem immature and petty, IMO.

Discussion is confusing because you ignore this RFC result.
https://wiki.php.net/rfc/add_validate_functions_to_filter
This RFC proposes filter module improvement while keeping compatibility.

I understand your point. This exactly the same reason why I proposed
"improvement" at first, not new extension.

I don't understand why you insist already failed attempt repeatedly.

Would you like me to propose previous RFC again?
and implement "ture validation" with filter?
I don't mind implementing it if you would like to update the RFC and it
passes.
I must use "white list" as much as possible.

Regards,

P.S. "Filter" module is black listing module. "Validate" is white listing
module.
Even with BC, mixing them would result in confusing FLAGs and codes.
Codes may be cleaned up later, but FLAGs cannot.
We should consider this also.

--
Yasuo Ohgaki
yohgaki@ohgaki.net

I was going to give a lot of detailed replies inline, but I’ve come to the
realisation its pointless with you. You really respond to what people say,
you just use their comments as jumping off points to re-post your same
little rant, ad nauseam.

May be I shouldn't reply if a reply indicates previous mails aren't read.
I usually reply all regardless. As a result, I reply the basically the same
thing.

Since someone mentioned hash_hkdf() mess on this thread, short note for
this.
It's clearly Nikita and Andrey's fault. They don't read the internet RFC
fully. I had no
idea why they're acting like ignorant, kept insisting ridiculous/insecure
API clearly
violates the RFC, i.e. Salt as last optional param. Wrong is wrong. I
cannot stop point
it out problems in security feature(key derivation) unless it is fixed. If
one feels curious,
read RFC 5869, then you'll see why the API is so ridiculous/insecure
without salt.

So here’s the summary. Don’t both replying, because I won’t be reading it.

I never asked for a working code example that is impossible with the
current extension. I asked for a simple example of what you wanted to
achieve.

OK. My excuse for misunderstanding. Unit tests do not cover all features
yet,
but you can see them from working "validate" module's *.phpt.

More than half the “issues” you claim with the filter extension, are
only “valid” if you agree that it needs to do complex array structure
validation. I do not agree with this. Userland can iterate an array of
rules/input and validate quite easily.

I totally agree that "validate" and "filter" is similar. I also totally
agree that it's very easily
done by scripts. I thought "proper application level validation" would be
common sense
many years ago since it is easy, but it is not. As you can see from this
discussion, there
are many people that "database" and/or "model" level validation is good
enough for apps.
This is one of my motivation, another is performance. Validation should be
done always,
so module functions are suitable for both performance and documentation
purpose.

For example, developers provide escape API for security purpose even when
it is trivial
with string functions. It's good for documentation purpose, as well as
performance. It's the same.

The actual issues with the filter extension could be solved by

improving/adding filters.

Largest filter module issue as validator is "filter is made for filtering"
and "blacklisting
nature came from filtering architecture".

I realized following issues with my filter module improvement RFC, of
course. My
approach back then was "it's better than nothing". There are many issues.
I picked 2
most importants.

Although it can be used for whitelisting, but it's optional by design.
It does not enforce
whitelisting by default. Enforced whitelisting archives much better
results. Therefore,
security related features should use whitelisting. e.g. MAC(Mandatory
Access Control), SELinux
It applies extremely dangerous default filter which does nothing in case
user sets invalid
filter/validator. i.e. Simply pass inputs, let code use it. No security
check at all.

Making filter's validation a true whitelist validator is possible. However,
there are issues.

Making filter a true validator requires a lot of BC. We may have "strict
option switch",
but making filter a whitelist validator isn't so simple as it may seem.
"strict option switch"
will add many branches and code might be unmaintainable. Even without
"strict option
switch", the code is based on "filtering"/"blacklisting" and requires large
refactoring.

I've already tried both "filter validate improvement" and "new validation
module". From this
experience, MySQL to MySQLi like transition is the best choice, IMO. We can
use
whatever API/interface(e.g. spec array, flags) that is easy to
understand/maintain/expand.

But again, I don't mind implementing filter's validation improvement if
anyone would
like to spend time for RFC. Even if it would be far from true validation,
it's still better
than nothing.

If you would like to create filter improvement RFC and if it passes, I'll
write code for it.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

7 years ago by Yasuo Ohgaki — view source

unread

Hi Dan

On 6 September 2017 at 12:15, Rowan Collins rowan.collins@gmail.com
wrote:

If you have suggestions for how the format should look

Don't use a format. Just write code - see below.

Which is why Yasuo and I have both suggested we work together

If you're going to work together and continue the conversation, please
can you move this conversation elsewhere?

It doesn't appear to be actually anything to do with PHP internals.

Since I didn't get much feedbacks during the RFC discussion, I cannot
tell
what part is disliked.

Yes you did. You got feedback during the discussion and also during
the vote. For example: http://news.php.net/php.internals/95164

However you continually choose to ignore that feedback.

I will attempt once more, to get the main point through to you.
Perhaps a small amount of repetition, will get it through:

This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.
This type of library should be done in PHP, not in C.

cheers
Dan
Ack

function validateOrderAmount($value) : int {
$count = preg_match("/[^0-9]*/", $value);
if ($count) {
    throw new InvalidOrderAmount("The order value must contain
only digits.");
}
$value = intval($value);

if ($value < 1) {
    throw new InvalidOrderAmount("The order value must be one or
more.");
}
if ($value >= MAX_ORDER_AMOUNT) {
    throw new InvalidOrderAmount(
    "Order value to big. Maximum allowed value is ".MAX_ORDER_AMOUNT
    );
}

return $value;
}

You seems mixing up responsibility between

Input validation that should be input handling code's responsibility.
Logical validation that should be model code's responsibility .

Please stick to single responsibility principle that you should be well
aware of.
Input handling code should only accepts valid inputs and let other codes
use it.
Other responsibilities are not input handling code's responsibilities.

Your example code should be in logic, not input handling, that is written
by PHP script.

As I wrote in README.md, there are only 3 types of inputs.

Valid data should accepted.
Valid data should accepted, but user's mistake. e.g. Logical error like
your example above.
Invalid. Anything other than 1 and 2 (i.e. Client cannot send these
value)

"validate" module is supposed to take care 3 which is nothing to do with
models, etc.
It should validate against input data spec, not logical meaning of the
input. If
programmer did this, single responsibility principle is broken.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

7 years ago by Lester Caine — view source

unread

As I wrote in README.md, there are only 3 types of inputs.

Valid data should accepted.

Valid data should accepted, but user's mistake. e.g. Logical error like
your example above.

Invalid. Anything other than 1 and 2 (i.e. Client cannot send these
value)

"validate" module is supposed to take care 3 which is nothing to do with
models, etc.
It should validate against input data spec, not logical meaning of the
input. If programmer did this, single responsibility principle is broken.

BUT you require an accurate 'input data spec' in order to establish what
is not part of '3' and this is the same metadata that is needed to ALSO
define the 'logical checks'. Once you have established that the input
data has a valid set of data you need to VALIDATE that the data is
within the limits defined by the 'input data spec' and those checks ALSO
apply to any subsequent processing of the data set. The 'input data
spec' is important not only to your 'single validation process', but
also to further processing that data prior to producing some sort of
output. ( No mention of databases but in a lot of cases that is where
the key metadata resides? )

My point is that the 'input data spec' is not simply a stand alone array
of data only used by the validator. It is something either created by
other parts of the 'logic' or it is needed to give individual responses
to 'user's mistake' as per '2' ...

I understand that you want to return a 'fail' at the earliest possible
point, and a single step 'validate' meets that need, but the bulk of the
reasons validation should fail is because someone is trying to hack a
site by creating 'user's mistakes' that pass '3' that are not handled
correctly by '2'. I think where the latest offering fails is that it now
requires that any 'custom' validation needs to be written in 'C' while
that same code may be needed as a PHP version as in Dan's example. The
validation processing needs to be ABLE to be iterated through variable
by variable once one has established that there IS a valid set of
variables to work with.

--
Lester Caine - G8HFL

7 years ago by Lester Caine — view source

unread

My only problem with Yasuo's latest offering is once again it adds a
whole new set of defines that have to be mapped to existing metadata
definitions ... That and it is a lot of longhand code using a different
style to existing arrays. We need yet another wrapper to build these
arrays from existing code ...
The rules have to be defined somehow, and I'm not aware of a standard format that current code is likely to follow. Unless there is already such a standard, I can't see any way to avoid existing code having to be wrapped or amended.

Which is why Yasuo and I have both suggested we work together to come up with such a standard format that can be used or adapted for these different parts of the application. If you have suggestions for how the format should look, we are eager to hear them and see some examples.

The likes of ADOdb datadict are still used as a base for metadata in
projects, but PDO destroyed the standardisation that used to exist by
spawning a number of competing wrappers. https://github.com/ADOdb/ADOdb
has evolved from a private project to being supported by it's own
community and is worth reconsidering as a proper cross database standard
to build on. Validation rules simply build on that base.

--
Lester Caine - G8HFL

7 years ago by lists@rhsoft.net — view source

unread

Am 06.09.2017 um 13:52 schrieb Lester Caine:

The likes of ADOdb datadict are still used as a base for metadata in
projects, but PDO destroyed the standardisation that used to exist by
spawning a number of competing wrappers. https://github.com/ADOdb/ADOdb
has evolved from a private project to being supported by it's own
community and is worth reconsidering as a proper cross database standard
to build on. Validation rules simply build on that base

frankly - why don't you realize that input validation DOES NOT turn
around databases at all - databases and SQL injection are just one
subset of it and not every application works with databases at all

7 years ago by Lester Caine — view source

unread

Am 06.09.2017 um 13:52 schrieb Lester Caine:

The likes of ADOdb datadict are still used as a base for metadata in
projects, but PDO destroyed the standardisation that used to exist by
spawning a number of competing wrappers. https://github.com/ADOdb/ADOdb
has evolved from a private project to being supported by it's own
community and is worth reconsidering as a proper cross database standard
to build on. Validation rules simply build on that base

frankly - why don't you realize that input validation DOES NOT turn
around databases at all - databases and SQL injection are just one
subset of it and not every application works with databases at all

Metadata describing a data set should be standard across all interfaces.
PHP has had standards on the database interfaces for a long time and
writing different standards yet again for other interfaces is all I am
objecting to. If you build metadata for a form that can then simply be
dropped into the interface for a database or to pass into another
storage method then it will be a lot more usable.

--
Lester Caine - G8HFL

7 years ago by Rowan Collins — view source

unread

On 6 September 2017 09:29:37 BST, Lester Caine lester@lsces.co.uk
wrote:
Which is why Yasuo and I have both suggested we work together to come
up with such a standard format that can be used or adapted for these
different parts of the application. If you have suggestions for how the
format should look, we are eager to hear them and see some examples.

The likes of ADOdb datadict are still used as a base for metadata in
projects

I'm going to cut you off there, and say this one more time: please can you show us an example of how you would like it to look. I have no idea what an "ADOdb datadict" is, so if that's what you're advocating, you'll need to show us what it looks like.

Spare the commentary on what decisions we could have made differently 15 years ago, and concentrate on what we can do right now, specifically, for this particular piece of functionality.

Thank you,

--
Rowan Collins
[IMSoP]

7 years ago by Stephen Reay — view source

unread

My only problem with Yasuo's latest offering is once again it adds a
whole new set of defines that have to be mapped to existing metadata
definitions ... That and it is a lot of longhand code using a different
style to existing arrays. We need yet another wrapper to build these
arrays from existing code ...

The rules have to be defined somehow, and I'm not aware of a standard format that current code is likely to follow. Unless there is already such a standard, I can't see any way to avoid existing code having to be wrapped or amended.

Which is why Yasuo and I have both suggested we work together to come up with such a standard format that can be used or adapted for these different parts of the application. If you have suggestions for how the format should look, we are eager to hear them and see some examples.

Regards,

--
Rowan Collins
[IMSoP]

--

Does this proposal actually offer any improvement over what's available with filter_var/etc and a userland wrapper class?

If there are deficiencies in how the existing filters work, maybe work on detailing the issues and then fixing/improving them?

That way, all the userland validation currently built around them, needs possibly minor changes to make use of new flags/options, as opposed to a completely new API to use.

Cheers
Stephen

7 years ago by Yasuo Ohgaki — view source

unread

Hi Rowan, Crocodile and Lester,

Rowan,
Thank you for helpful suggestions. I think it may better be named
'validate()'/etc.
Sole reason I named functions as 'valid*()' is name collision.
'validate*()' seems
too common, but users should use namespace to avoid collisions. I'll take
your
suggestions into the module.

Crocodile,
I agree your comment. However, my last filter module improvement RFC
with PoC patch was failed. Upside having new module is we can enforce more
strict validations and can have much simpler/extensible API with cleaner
codes.
(Having both 'filter' and 'validator' in one code is mess)
Since the last attempt was failed, new module would be the next reasonable
attempt.

There is one principle that developers is better to follow.
https://en.wikipedia.org/wiki/Fail-fast
If we follow this principle, validation at controller makes sense.

Oops. A 'is' should be 'are'.

Since a large proportion of the data coming in is a result of input into
a previously generated form, the data can often be validated in terms of
basic structure before even needing to decide if the data set needs to
be iterated? If things like 'maximum data size' can be established when
the form is created, any data set larger than that can simply be killed
off.

Anyway, thank you for pointer for PDO validation. I didn't notice the

project. We may cooperate so that there aren't unnecessary validaiton
rule incompatibilities
I've been pushing the idea of a single method of managing metadata for a
long time. Most of the 'checking' loading down PHP now still misses the
point and the database style of rules has been around since long before
PDO and other abstractions messed it up. A single standard set of rules
that can be used across the board from variable creation to checking
data going out to forms and returns coming back and data between
operations such as database activity. This is NOT 'typing' since that
lacks the vast majority of checks that a decent validation will handle,
but the much finer details such as limits and value sets. There is a
vast discrepancy in how this is handled across databases, but the SQL
standard does provide a base which databases are slowly evolving towards.

It's nice to have central input type (Not only data type, but length,
chars, format, range and char encoding, as you mentioned) repository for an
app. That's the reason why I would like to cooperate with other validator
implementation(s) to avoid unnecessary incompatibilities.

(For security perspective, it can be said different types of input
validations is
better because if one failed, another may validate data correctly. However,
managing multiple validation rules is burden. Some people try to validate
inputs with WAF. However, I suggest input validation is better to be
implemented
at web apps, not WAF.
It's a lot easier and maintainable if input validation is done by app. App
developers know what the input should be exactly. In addition, app must
validate inputs regardless of WAF. I'm not saying WAF is useless. WAF
is still useful as network layer protection. I'm saying multiple layer
validations
are recommended, but it can be unmanageable easily unless some trade
off is taken)

I agree that PDO level validation is almost mandatory, especially for
SQLite3.
SQLite3 data type is pseudo type and allows any "characters". e.g. Int type
can store '<script>alert("XSS")</script>', etc.

My validate module allows users to have central input type repository by
simple native PHP array. Array could be stored anywhere users want.
I think it will work as you wish if 'validate' module is compatible with
PDO
validator.

For the time being, I'm not sure what it should be. Data specifications for
database may be stored as additional info in 'validate' spec array, or PDO
validator may simply assume data types specified in 'validate' spec array
should be enforced, and use 'string' data spec as required format,
or 'validate' module may expose API so that PDO validator can use it for
basic PHP data type validation.

Regards,

--
Yasuo Ohgaki

7 years ago by Yasuo Ohgaki — view source

unread

Hi Lester,

I cannot guess people's thought. I appreciated feedback!

With a decent database layer a lot of the validation you are proposing
is already covered but PDO does not help in this area. Adding another
layer that does not integrate with a storage layer is just adding to the
current mess ...

I'm fun of multiple tier and multiple layer of protections.
For instance, Microsoft's SQL injection security page states as follows.

Never build Transact-SQL statements directly from user input; use stored
procedures to validate user input.
Validate user input by testing type, length, format, and range. Use the
Transact-SQL QUOTENAME() function to escape system names or the REPLACE()
function to escape any character in a string.
Implement multiple layers of validation in each tier of your application.

https://docs.microsoft.com/en-us/dotnet/framework/data/adonet/sql/writing-secure-dynamic-sql-in-sql-server

This is what secure coding practice recommends, too.
It may seem mess, but it's not. Outermost trust boundary that can be
controlled
is the most important trust boundary. For server side web app developers,
outermost
trust boundary is controller in MVC model. Input validations at model is a
bit too late
to mitigate risks involved with invalid(attacker) inputs.

Both model and controller layer Input validations (as well as in the
database, too) are
good/important to have.

There are one principle that developers are better to follow.
https://en.wikipedia.org/wiki/Fail-fast
If we follow this principle, validation at controller makes sense.

Regards,

P.S. For database administrators or web app developers who maintain
application
Models, outermost trust boundary is "database system" and "the Model layer"
respectively.
Outermost trust boundary is changed by what they can control.

This kind of discussion could result in mess. I hope I explained well
enough.

--
Yasuo Ohgaki
yohgaki@ohgaki.net

6 years ago by Yasuo Ohgaki — view source

unread

Hi all,

I spent a little time for a new input validation module. It's not totally
new module, but is based on Filter module's validation filter improvement
RFC in many ways. [1]

As all of us knew already, input validation is the most important practice
in secure coding. [2][3] Yet, we don't provide usable feature out of box.
Sadly, almost all apps do not have proper input validation at trust
boundary. Unless we improve filter's validation, we need usable basic
validator by default. IMO.

Since I didn't get much feedbacks during the RFC discussion, I cannot tell
what part is disliked. I guess too much features in filter is one reason.
Another is messed up codes/features by providing both "filter" and
"validation".

Validator for PHP7 (validate module) gets rid of unneeded features. It
only has features for basic PHP data type validations. Validation
rule(spec) array is flexible enough. Almost any types of inputs could be
handled by multiple and nested validation rules.

Except some minor features like overflow checks, most planned features are
implemented.

https://github.com/yohgaki/validate-php

Although the code is based on filter module's code, it's almost full
rewrite except validation logic came from filter. Please consider this as
under development module.
Feedbacks are appreciated.

Regards,

[1] https://wiki.php.net/rfc/add_validate_functions_to_filter
[2] https://www.securecoding.cert.org/confluence/display/
seccode/Top+10+Secure+Coding+Practices
[3] https://www.owasp.org/index.php/OWASP_Secure_Coding_
Practices_-_Quick_Reference_Guide

--
Yasuo Ohgaki
yohgaki@ohgaki.net

I thought it would be nice to have PHP script version for
Validate PHP. It a lot easier to modify API as needed. So
I spend few hours last weekend.

https://github.com/yohgaki/validate-php-scr

Caution, I just wrote it and didn't debug it yet.
However, it is good enough to play with, I suppose.

API differs a little. This has more simplified parameter
structure. Suggestions and comments are appreciated.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

6 years ago by Crocodile — view source

unread

It's almost always the case that you need to provide a meaningful feedback
about what exactly went wrong, rather then to just say "Failed!" While
simplicity is nice and you cannot overrate value of validation, this whole
thing is pretty much useless to me personally without this ability. Also, I
don't think it's a good idea to mix validation of scalar values, arrays and
even multiple arrays, in a single function.

Hi all,

I spent a little time for a new input validation module. It's not totally
new module, but is based on Filter module's validation filter improvement
RFC in many ways. [1]

As all of us knew already, input validation is the most important
practice
in secure coding. [2][3] Yet, we don't provide usable feature out of box.
Sadly, almost all apps do not have proper input validation at trust
boundary. Unless we improve filter's validation, we need usable basic
validator by default. IMO.

Since I didn't get much feedbacks during the RFC discussion, I cannot
tell
what part is disliked. I guess too much features in filter is one reason.
Another is messed up codes/features by providing both "filter" and
"validation".

Validator for PHP7 (validate module) gets rid of unneeded features. It
only has features for basic PHP data type validations. Validation
rule(spec) array is flexible enough. Almost any types of inputs could be
handled by multiple and nested validation rules.

Except some minor features like overflow checks, most planned features
are
implemented.

https://github.com/yohgaki/validate-php

Although the code is based on filter module's code, it's almost full
rewrite except validation logic came from filter. Please consider this as
under development module.
Feedbacks are appreciated.

Regards,

[1] https://wiki.php.net/rfc/add_validate_functions_to_filter
[2] https://www.securecoding.cert.org/confluence/display/
seccode/Top+10+Secure+Coding+Practices
[3] https://www.owasp.org/index.php/OWASP_Secure_Coding_
Practices_-_Quick_Reference_Guide

--
Yasuo Ohgaki
yohgaki@ohgaki.net

I thought it would be nice to have PHP script version for
Validate PHP. It a lot easier to modify API as needed. So
I spend few hours last weekend.

https://github.com/yohgaki/validate-php-scr

Caution, I just wrote it and didn't debug it yet.
However, it is good enough to play with, I suppose.

API differs a little. This has more simplified parameter
structure. Suggestions and comments are appreciated.

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

--
Best regards,
Victor Bolshov

6 years ago by Yasuo Ohgaki — view source

unread

It's almost always the case that you need to provide a meaningful feedback
about what exactly went wrong, rather then to just say "Failed!" While
simplicity is nice and you cannot overrate value of validation, this whole
thing is pretty much useless to me personally without this ability. Also, I
don't think it's a good idea to mix validation of scalar values, arrays and
even multiple arrays, in a single function.

It seems you underrate validations.
It's a fundamental requirement for programs to work correctly.

Fundamental requirement: Programs can only work correctly with valid inputs.

There are countless reason why one should validate ALL inputs other than
this.
Almost nobody does this now, even if it is "fundamental" requirement.

How you would solve this?

--
Yasuo Ohgaki
yohgaki@ohgaki.net

6 years ago by Yasuo Ohgaki — view source

unread

It's almost always the case that you need to provide a meaningful feedback
about what exactly went wrong, rather then to just say "Failed!" While
simplicity is nice and you cannot overrate value of validation, this whole
thing is pretty much useless to me personally without this ability. Also, I
don't think it's a good idea to mix validation of scalar values, arrays and
even multiple arrays, in a single function.

There are 3 types of validations in general.
There are only 3 types of inputs.
Explaining them need lots of word as well as for other
fundamentals/principles related to this.

https://blog.ohgaki.net/computer-programming-fundamentals-and-basics-input-validation

You are referring to "Business Logic Validation" which needs feedback to
users in case
they send incorrect values.

"App Level Input Validation" does not require feedback other than "You sent
really bad
request. This has been reported to admins." Therefore, App Level Input
Validation can
validate ALL inputs at once and there is no good reason not to do this.

Btw, you can use the validator for a single value validation as well. It
also able to
disable Exception so that one can use it for "Business Logic Validations"
as well.

https://github.com/yohgaki/validate-php-scr/tree/master/src/examples

Regards,

--
Yasuo Ohgaki
yohgaki@ohgaki.net

A validator module for PHP7

-- Lester Caine - G8HFL

-- Lester Caine - G8HFL

-- Lester Caine - G8HFL

-- Lester Caine - G8HFL

-- Lester Caine - G8HFL

-- Lester Caine - G8HFL

-- Lester Caine - G8HFL

-- Lester Caine - G8HFL

--
Lester Caine - G8HFL

--
Lester Caine - G8HFL

--
Lester Caine - G8HFL

--
Lester Caine - G8HFL

--
Lester Caine - G8HFL

--
Lester Caine - G8HFL

--
Lester Caine - G8HFL

--
Lester Caine - G8HFL