# Better RFC conformance for FILTER_VALIDATE_URL - Thread: https://externals.io/message/77968 - Original archive: https://news-web.php.net/php.internals/77968 - Started by dunglas@gmail.com on 2014-10-14T11:41:51+00:00 - Messages: 15 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #77968 — dunglas@gmail.com — 2014-10-14T11:41:51+00:00 URL: https://externals.io/message/77968#77968 Hi, I opened a PR making `FILTER_VALIDATE_URL` more strict and more compliant with standards: https://github.com/php/php-src/pull/826 Can anyone review (and merge) this patch? Thanks! -- Kévin Dunglas Consultant et développeur freelance http://dunglas.fr Tél. : 06 60 91 20 20 --- ## #77969 — Daniel Ribeiro — 2014-10-14T11:48:11+00:00 URL: https://externals.io/message/77968#77969 · In reply to: #77968 Nice work man, it looks really good. Daniel Ribeiro http://danielribeiro.org > Hi, > > I opened a PR making `FILTER_VALIDATE_URL` more strict and more compliant > with standards: https://github.com/php/php-src/pull/826 > > Can anyone review (and merge) this patch? > > Thanks! > -- > Kévin Dunglas > Consultant et développeur freelance > > http://dunglas.fr > Tél. : 06 60 91 20 20 --- ## #78750 — dunglas@gmail.com — 2014-11-05T21:57:02+00:00 URL: https://externals.io/message/77968#78750 · In reply to: #77969 Hi, According to the discussion on GitHub, I've made some changes on this PR: - Added a new `FILTER_VALIDATE_DOMAIN` filter validating domain names - Added a `FILTER_FLAG_HOSTNAME` flag to allow checking hostnames (_ are forbidden in hostname but not in domains) - Changed `FILTER_VALIDATE_URL` to use this new validator When https://github.com/php/php-src/pull/890 will be merged, it will be easy to add IDN support to this new domain validator. 2014-10-14 13:48 GMT+02:00 Daniel Ribeiro : > > Nice work man, it looks really good. > > > Daniel Ribeiro > http://danielribeiro.org > > > > […] > > -- Kévin Dunglas Consultant et développeur freelance http://dunglas.fr Tél. : 06 60 91 20 20 --- ## #78759 — Andrey Andreev — 2014-11-05T23:32:06+00:00 URL: https://externals.io/message/77968#78759 · In reply to: #78750 Hi, > > - Added a new `FILTER_VALIDATE_DOMAIN` filter validating domain names > - Added a `FILTER_FLAG_HOSTNAME` flag to allow checking hostnames (_ are > forbidden in hostname but not in domains) This doesn't make any sense. A domain *is* a hostname and underscores are forbidden. Cheers, Andrey. --- ## #78782 — dunglas@gmail.com — 2014-11-06T06:19:50+00:00 URL: https://externals.io/message/77968#78782 · In reply to: #78759 Hi Andrey, Sorry but I think you're wrong. Domain != hostname. Underscore are allowed in domains (RFC 2181) but not in hostnames (RFC 1123 and next). To quote Wikipedia: "While a hostname may not contain other characters, such as the underscore character (_), other DNS names may contain the underscore. Systems such asDomainKeys and service records use the underscore as a means to assure that their special character is not confused with hostnames. For example,_http._sctp.www.example.com specifies a service pointer for an SCTP capable webserver host (www) in the domain example.com." http://en.wikipedia.org/wiki/Hostname#Restrictions_on_valid_host_names You can also see this StackOverflow answer http://stackoverflow.com/a/2183140/1352334 2014-11-06 0:32 GMT+01:00 Andrey Andreev : > Hi, > > > > […] > > This doesn't make any sense. A domain *is* a hostname and underscores > are forbidden. > > Cheers, > Andrey. > -- Kévin Dunglas Consultant et développeur freelance http://dunglas.fr Tél. : 06 60 91 20 20 --- ## #78814 — Andrey Andreev — 2014-11-06T12:55:06+00:00 URL: https://externals.io/message/77968#78814 · In reply to: #78782 Hi, > Hi Andrey, > > Sorry but I think you're wrong. Domain != hostname. Underscore are allowed > in domains (RFC 2181) but not in hostnames (RFC 1123 and next). To quote > Wikipedia: > > "While a hostname may not contain other characters, such as the underscore > character (_), other DNS names may contain the underscore. Systems such > asDomainKeys and service records use the underscore as a means to assure > that their special character is not confused with hostnames. For > example,_http._sctp.www.example.com specifies a service pointer for an SCTP > capable webserver host (www) in the domain example.com." > http://en.wikipedia.org/wiki/Hostname#Restrictions_on_valid_host_names > > You can also see this StackOverflow answer > http://stackoverflow.com/a/2183140/1352334 I agree to an extent, but that is highly contextual. Who said that 'domain' === 'DNS record' (which is a very broad term anyway)? And IF we assume this, why do you need `FILTER_VALIDATE_DOMAIN` for it if it's only going to check length? Cheers, Andrey. --- ## #78816 — dunglas@gmail.com — 2014-11-06T13:39:46+00:00 URL: https://externals.io/message/77968#78816 · In reply to: #78814 FILTER_VALIDATE_DOMAIN checks conformance with DNS RFCs : total length, label length and allowed characters (_ are allowed in domain names but many other characters are forbidden such as ~/+...). I'll add IDN support too when IDN support for streams will be merged. `FILTER_VALIDATE_URL` checks conformance with URL RFCs (and not URI, as discussed on GitHub). URL's host part RFCs conformance implies DNS RFCs conformance, IPv4 and IPv6 RFCs conformance + some additional checks (no underscore allowed in hostnames and IPv6 enclosed with brackets for instance). It's why I've added the convenience flag FILTER_FLAG_HOSTNAME. Btw, there is many use case for validating that a string is a valid domain (or a valid hostname): hoster and registar apps, mail server management apps and anything else DNS related. Maybe be can we find a better name for FILTER_VALIDATE_DOMAIN. Such as FILTER_VALIDATE_DOMAIN_NAME or FILTER_VALIDATE_DNS_DOMAIN (a bit redundant, DNS = Domain Name System) but please not something related "DNS Record" because a valid DNS record can have the following format: les-tilleuls.coop. 3600 IN SOA monsite.nnx.com .root.monsite.nnx.com. ( 2014092300 ; serial 21600 ; refresh (6 hours) 3600 ; retry (1 hour) 604800 ; expire (1 week) 86400 ; minimum (1 day) ) 2014-11-06 13:55 GMT+01:00 Andrey Andreev : > Hi, > > > > […] > allowed > > […] > underscore > > […] > SCTP > > […] > > I agree to an extent, but that is highly contextual. > > Who said that 'domain' === 'DNS record' (which is a very broad term > anyway)? And IF we assume this, why do you need `FILTER_VALIDATE_DOMAIN` > for it if it's only going to check length? > > Cheers, > Andrey. > -- Kévin Dunglas Consultant et développeur freelance http://dunglas.fr Tél. : 06 60 91 20 20 --- ## #78817 — Andrey Andreev — 2014-11-06T13:45:03+00:00 URL: https://externals.io/message/77968#78817 · In reply to: #78816 Hi, > `FILTER_VALIDATE_DOMAIN` checks conformance with DNS RFCs : total length, > label length and allowed characters (_ are allowed in domain names but many > other characters are forbidden such as ~/+...). I'll add IDN support too > when IDN support for streams will be merged. I am not trying to argue, but where does it say that ~/+ are disallowed, yet an underscore is? The only rule *allowing* underscores that I've seen is the one that says any binary string is a valid DNS record value. Cheers, Andrey. --- ## #78823 — Sean Coates — 2014-11-06T17:56:14+00:00 URL: https://externals.io/message/77968#78823 · In reply to: #78817 > > > […] > > I am not trying to argue, but where does it say that ~/+ are > disallowed, yet an underscore is? The only rule *allowing* underscores > that I've seen is the one that says any binary string is a valid DNS > record value. FWIW, there *is* a practical in-use (de facto if nothing else) convention of using _ in hosts for DKIM: $ dig +short -ttxt mandrill._domainkey.mailchimp.com "k=rsa\; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCrLHiExVd55zd/IQ/J/mRwSRMAocV/hMB3jXwaHH36d9NaVynQFYV8NaWi69c1veUtRzGt7yAioXqLj7Z4TeEUoOLgrKsn8YnckGs9i3B3tVFB+Ch/4mPhXWiNfNdynHWBcPcbJ8kjEQ2U8y78dHZj1YeRXXVvWob2OaKynO8/lQIDAQAB\;” S --- ## #78825 — Sanford Whiteman — 2014-11-06T21:48:08+00:00 URL: https://externals.io/message/77968#78825 · In reply to: #78823 > FWIW, there *is* a practical in-use (de facto if nothing else) convention of using _ in hosts for DKIM: _domainkey is actually in all the DKIM RFCs and in the formal STD 76, see § 3.6.2.1. Namespace, so it's more than a convention! --- ## #78875 — Yasuo Ohgaki — 2014-11-12T06:38:00+00:00 URL: https://externals.io/message/77968#78875 · In reply to: #78825 Hi all, On Fri, Nov 7, 2014 at 6:48 AM, Sanford Whiteman wrote: > > […] > convention of using _ in hosts for DKIM: > > _domainkey is actually in all the DKIM RFCs and in the formal STD 76, > see § 3.6.2.1. Namespace, so it's more than a convention! "_" is used for service name. Active Directory uses "_" a lot, for example. e.g. _tcp, _sites, _ldap, etc. https://tools.ietf.org/html/rfc2782 Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #78876 — dunglas@gmail.com — 2014-11-12T07:09:23+00:00 URL: https://externals.io/message/77968#78876 · In reply to: #78875 Hi, I'll change my PR according to the RFC I've quoted earlier: - check for valid characters (excluding underscore) only when `FILTER_FLAG_HOSTNAME` is set - allow any character but check lengths by default - use `FILTER_FLAG_HOSTNAME` to validate URLs What do you think about that? Best regards, 2014-11-12 7:38 GMT+01:00 Yasuo Ohgaki : > Hi all, > > On Fri, Nov 7, 2014 at 6:48 AM, Sanford Whiteman > wrote: > > > […] > > > "_" is used for service name. Active Directory uses "_" a lot, for > example. e.g. _tcp, _sites, _ldap, etc. > > https://tools.ietf.org/html/rfc2782 > > Regards, > > -- > Yasuo Ohgaki > yohgaki@ohgaki.net > -- Kévin Dunglas http://dunglas.fr --- ## #78879 — Yasuo Ohgaki — 2014-11-12T09:41:14+00:00 URL: https://externals.io/message/77968#78879 · In reply to: #78876 Hi Kevin, > I'll change my PR according to the RFC I've quoted earlier: > - check for valid characters (excluding underscore) only when > `FILTER_FLAG_HOSTNAME` is set > - allow any character but check lengths by default > - use `FILTER_FLAG_HOSTNAME` to validate URLs > > What do you think about that? > I haven't read diff closely, but it seems ok to me. How email domain is checked? I cannot see changes for it from the diff. Validating host correctly is difficult, I would like to have your PR. Regards, -- Yasuo Ohgaki yohgaki@ohgaki.net --- ## #78886 — dunglas@gmail.com — 2014-11-12T18:10:16+00:00 URL: https://externals.io/message/77968#78886 · In reply to: #78879 Hi Yasuo, I've not changed (and even read) the email validator. I'll take a look at it. 2014-11-12 10:41 GMT+01:00 Yasuo Ohgaki : > Hi Kevin, > > > > > […] > > I haven't read diff closely, but it seems ok to me. > How email domain is checked? I cannot see changes for it from the diff. > > Validating host correctly is difficult, I would like to have your PR. > > Regards, > > -- > Yasuo Ohgaki > yohgaki@ohgaki.net > -- Kévin Dunglas http://dunglas.fr --- ## #78903 — dunglas@gmail.com — 2014-11-14T09:44:53+00:00 URL: https://externals.io/message/77968#78903 · In reply to: #78886 I've just pushed some changes in the PR. `FILTER_VALIDATE_DOMAIN` now checks characters validity only if `FILTER_FLAG_HOSTNAME` is set. I've also rebased and fixed some issues detailed on GitHub. Yasuo, it's not trivial to use this new validator in FILTER_VALIDATE_EMAIL. Its current implementation use a big regex that doesn't extract the domain part. Anyway, having a good RFC compliant email validator cannot be done with a regex. See https://github.com/egulias/EmailValidator for instance. I think it's a work for another PR. I'll keep the email validator in it's current state for now. Do you guys are OK to get the current PR merged? 2014-11-12 19:10 GMT+01:00 Kévin Dunglas : > Hi Yasuo, > > I've not changed (and even read) the email validator. I'll take a look at > it. > > 2014-11-12 10:41 GMT+01:00 Yasuo Ohgaki : > > > […] > > > > -- > Kévin Dunglas > > http://dunglas.fr > -- Kévin Dunglas Consultant et développeur freelance http://dunglas.fr Tél. : 06 60 91 20 20