# [RFC] is_literal

- Thread: https://externals.io/message/114835
- Original archive: https://news-web.php.net/php.internals/114835
- Started by Craig Francis on 2021-06-12T17:00:12+00:00
- Messages: 68
- RFC: Is literal (Declined) — https://externals.io/rfc/is_literal.md
- RFC: LiteralString (Inactive) — https://externals.io/rfc/literal_string.md
- Page 1 of 3: messages 1 to 25 (25 per page, add `?page=N` to the URL)
- Next page: https://externals.io/message/114835.md?page=2
- Reply by email to internals@lists.php.net
- More for agents: search with https://externals.io/search.md?q={query} · RFCs at https://externals.io/rfc.md · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt

---

## #114835 — Craig Francis — 2021-06-12T17:00:12+00:00

URL: https://externals.io/message/114835#114835

Hi Internals,

I'd like to start the discussion on the is_literal() RFC:

https://wiki.php.net/rfc/is_literal

is_literal() brings a proven way to identify Injection Vulnerabilities to
PHP, already used by Google in their Java and Go projects, and is currently
being added to JavaScript. It's a lightweight and simple approach:
"Distinguishing strings from a trusted developer from strings that may be
attacker controlled", allowing Libraries to identify the mistakes made by
the thousands of developers using them incorrectly.

When Libraries use is_literal() to protect against these mistakes, we can
trust their output, and (as covered in the Future scope section) PHP can
then raise warnings with certain native functions like PDO::query,
mysqli_query, exec, preg_match, etc. (we would only consider warnings,
anything stricter like exceptions would be in many years time, if at all -
the intention is to alert and inform people, not break things).

The length is due to the FAQ Section, on why it's needed, how it can be
used by Libraries, and the important differences of using this flag versus
the flawed Taint Checking approach with its false sense of security
(error-prone escaping).

Thanks,
Craig Francis


PS: If anyone wants to discuss face-to-face on Zoom, I'll be available (UK
Time/BST/UTC+1) at:

https://chat.craigfrancis.co.uk/

Saturday 12th June, 6pm to 8pm;
Sunday 13th June, 10am to Midday;
Monday 14th June, 5pm to 7pm;
Tuesday 15th June, 9pm to 11pm;
Thursday 16th June, 10am to Midday;
(other times on request)

---

## #114836 — Joe Watkins — 2021-06-12T17:45:53+00:00

URL: https://externals.io/message/114835#114836 · In reply to: #114835

Afternoon all,

While this is not at all my idea, I wrote the patch, so my words may seem
bias/hollow. Still, here are some words ...

In the past this kind of feature would have been extremely invasive, it
would have had so many edges because of the way we handled strings that it
was never really feasible.

Today the implementation of this is not particularly invasive or
complicated, and it lends a very useful tool to the developer.

This starting place, of second class support seems like a reasonable
starting place, but there is a possible future where we have first class
support.

This may or may not be appealing and we'll be in a better position to judge
when we see how this may be used in the wild.

Cheers
Joe


On Sat, 12 Jun 2021 at 19:00, Craig Francis <craig@craigfrancis.co.uk>
wrote:

> Hi Internals,
>
> I'd like to start the discussion on the is_literal() RFC:
>
> https://wiki.php.net/rfc/is_literal
>
> is_literal() brings a proven way to identify Injection Vulnerabilities to
> PHP, already used by Google in their Java and Go projects, and is currently
> being added to JavaScript. It's a lightweight and simple approach:
> "Distinguishing strings from a trusted developer from strings that may be
> attacker controlled", allowing Libraries to identify the mistakes made by
> the thousands of developers using them incorrectly.
>
> When Libraries use is_literal() to protect against these mistakes, we can
> trust their output, and (as covered in the Future scope section) PHP can
> then raise warnings with certain native functions like PDO::query,
> mysqli_query, exec, preg_match, etc. (we would only consider warnings,
> anything stricter like exceptions would be in many years time, if at all -
> the intention is to alert and inform people, not break things).
>
> The length is due to the FAQ Section, on why it's needed, how it can be
> used by Libraries, and the important differences of using this flag versus
> the flawed Taint Checking approach with its false sense of security
> (error-prone escaping).
>
> Thanks,
> Craig Francis
>
>
> PS: If anyone wants to discuss face-to-face on Zoom, I'll be available (UK
> Time/BST/UTC+1) at:
>
> https://chat.craigfrancis.co.uk/
>
> Saturday 12th June, 6pm to 8pm;
> Sunday 13th June, 10am to Midday;
> Monday 14th June, 5pm to 7pm;
> Tuesday 15th June, 9pm to 11pm;
> Thursday 16th June, 10am to Midday;
> (other times on request)

---

## #114837 — lauri.kentta@gmail.com — 2021-06-12T18:59:23+00:00

URL: https://externals.io/message/114835#114837 · In reply to: #114835

Hi,

I wrote the untaint() / make_literal() function, just in case.

implode("", array_map(fn($c) => $chars[ord($c)], str_split($s, 1)))

https://3v4l.org/EaN9Z#focus=rfc.literals

Sorry and bye.

-- 
Lauri Kenttä

---

## #114838 — Craig Francis — 2021-06-12T19:21:19+00:00

URL: https://externals.io/message/114835#114838 · In reply to: #114837

> Hi,
>
> I wrote the untaint() / make_literal() function, just in case.
>
> implode("", array_map(fn($c) => $chars[ord($c)], str_split($s, 1)))
>
> https://3v4l.org/EaN9Z#focus=rfc.literals
>
> Sorry and bye.
>
>

Yes, I have a similar example in the RFC (eval).

The is_literal() function provides a guard rail (for everyone, but
especially for less knowledge developers), but as with most security
things, there’s nothing stopping you from deliberately climbing up and
jumping off, but at that point it’s very clearly your choice to do so.

Thanks for looking it over, and I hope it will be useful.




> --
> Lauri Kenttä
>
> --
> 
> To unsubscribe, visit: https://www.php.net/unsub.php

---

## #114839 — lauri.kentta@gmail.com — 2021-06-12T20:45:55+00:00

URL: https://externals.io/message/114835#114839 · In reply to: #114838

> On Sat, 12 Jun 2021 at 19:59, Lauri Kenttä <lauri.kentta@gmail.com> 
> wrote:
> 
> > […]
> 
> Yes, I have a similar example in the RFC (eval).

Oh, the irresponsible use of eval was so overwhelming that I missed the 
new string literal there. You could add var_export to make it more 
realistic.

Anyway, the RFC is well motivated and thoroughly thought out. The 
approach is very simple but fits many use cases, compared to previous 
alternatives which were complex but more limited in the end. When 
libraries and PDO start to use this, we can finally get rid of SQL 
injections and a number of self-made input handling tricks, if only 
people have learned to read warnings...

So thanks for writing this RFC. I hope it passes.

-- 
Lauri Kenttä

---

## #114840 — Mike Schinkel — 2021-06-13T06:46:35+00:00

URL: https://externals.io/message/114835#114840 · In reply to: #114835

Hi Craig,

> 
> 
> Hi Internals,
> 
> I'd like to start the discussion on the is_literal() RFC:
> 
> https://wiki.php.net/rfc/is_literal

Nice!  There is an awful lot to like here.

And few bits of concern.

What's to like?
-------------------

1. Adding a proactive method for guarding against injection vulnerabilities would be a huge step forward for PHP. This should not be underemphasized.

2. That you added the section "Previous Work" and covered what other languages are doing in this regard.

3. The "Thanks" section and all the external references and prior comments that provides insight into how this RFC was developed.

4. And I love that you publicly stated these points in an RFC:

	a. Why not use static analysis? It will never be used by most developers.
	b. Why not educate everyone? You can't - developer training simply does not scale, and mistakes still happen.


What's of concern? 
-------------------------
(Note: the last point could address concerns #2 and #4.)

1. Bike-shedding a bit, but I find a bit of cognitive dissonance over the semantic incorrectness of is_literal(), similar to Jakob Givoni's concern. 

I also fear it will be confusing for developers who know what a literal is, and thus I agree with Jakob. 

Minimally I'd ask for a separate vote on is_literal() vs. is_from_literal(). 

2. Regarding the sections "Non-Parameterised Values" and "Future Scope," if I understand correctly, your vision is to ultimately require all SQL and other vulnerable literals to be written as literals in the source code for them to be used or otherwise disallowed by future library code and/or PHP internal functionality?

If I did not misunderstand I believe you would disallow an important class of functionality from being implemented in PHP, including several existing and widely used applications such as but not limited to PhpMyAdmin and Adminer that open and interact with arbitrary SQL databases.  It would also disallow using any data returned by APIs where the PHP application cannot know in advance what data will be acquired, even if that data has been properly escaped and/or sanitized.

3. I notice your RFC does not grant `sprintf()` the ability to return a string with the "literal" flag even though `sprintf()` can frequently return strings composed from known literals and safe numeric value. And I have frequently see it used for this purpose, e.g.:

$fields = ['id','name'];
$table = 'my_table';

$sql = sprintf( 'SELECT %s FROM %s WHERE ID=%d', 
	implode( ',', $fields ),	
	$my_table,
	intval($_GET['id']));

Of course your section "WHERE IN" "addresses" this, but please see #4 below. 

I argue `sprintf()` should be able to pass thru literal flags and validate integer values to determine that the result is literal.  Am I missing something here?

4. Regarding the section "WHERE IN" you argue that we should push everyone to use parameters properly — which is a great idea in theory — but that would have a significant breakage in existing sites online.  While many here in Internals seem to look down on WordPress if does power over 40%[1] of the web at this point and this would break most if not all those sites if the site owners or their hosts were to upgrade to a version of PHP that was militant about requiring is_literal() for `mysqli_query()`[2].  

And yes, WordPress could upgrade their core code[3] to address this, but that would not also fix the 60k plugins and 10k themes on WordPress.org <http://wordpress.org/>, or likely the million+ combined custom plugins and themes in use on the web. 

Is it really reasonable to change PHP to just break of all those sites so that we can force all the site owners to beg and plead with their plugin and theme developers to better protect them from injection errors?  Seems that medicine might be worse than the cure, at least for all those site owners. 

5. Given #2 and #4, I think your section "Support Functions" is missing what would allow WordPress, PhpMyAdmin, Adminer and all other PHP applications to address the concerns in #2 and #4 and it is neither literal_concat() nor literal_implode(). 

The function we would need is the one Lauri Kenttä wrote (in jest?), but we'd need one that is performant, not a userland version. Name it as_literal() or make_literal() or even as_unsafe_literal() — whichever chosen is unimportant IMO — but applications like the ones I mentioned need to be able to say "I absolutely know what I am explicitly doing so do not throw a warning or an error telling me my value is `not a literal`, dammit!"  

Certainly most people would not use such a method by accident, especially if called something like `as_unsafe_literal()`.  There might even be ways to validate that this was not used by a copy-and-paste artist, but I will leave that validation to other people who are smarter than me.

6. Regarding the section "Reflection API," shouldn't their be an is_literal()/is_from_literal() method added to the ReflectionParameter() and ReflectionProperty() classes?

So assuming you address at least #5 — and thus #2 and #4 indirectly — I would really like to see this RFC pass.

-Mike


[1] https://w3techs.com/blog/entry/40_percent_of_the_web_uses_wordpress <https://w3techs.com/blog/entry/40_percent_of_the_web_uses_wordpress> 
[2] https://www.php.net/manual/en/mysqli.query.php#refsect1-mysqli.query-parameters <https://www.php.net/manual/en/mysqli.query.php#refsect1-mysqli.query-parameters> 
[3] https://github.com/WordPress/WordPress/blob/master/wp-includes/wp-db.php#L2056 <https://github.com/WordPress/WordPress/blob/master/wp-includes/wp-db.php#L2056

---

## #114842 — Craig Francis — 2021-06-13T16:51:36+00:00

URL: https://externals.io/message/114835#114842 · In reply to: #114840

> Nice!  There is an awful lot to like here.
>


Thank you.



> 1. [...] Minimally I'd ask for a separate vote on is_literal() vs.
> is_from_literal().
>


It would definitely be possible to have a vote on the name. I only went
with is_literal() because it's shorter (I'm a slow typer), and to match the
current convention of existing "is_*" functions (which all use a single
word suffix).


2. Regarding the sections "Non-Parameterised Values" and "Future Scope," if
> I understand correctly, your vision is to ultimately require all SQL and
> other vulnerable literals to be written as literals in the source code for
> them to be used or otherwise disallowed by future library code and/or PHP
> internal functionality?
>


No, fortunately that's not the case, as you're right, we do need to support
PhpMyAdmin, Adminer, systems like Drupal (which take certain values, like
the table name, from user input), and (as noted in your 4th point), there
is a lot of existing code that would break if we were militant about
requiring is_literal() for `mysqli_query()` and other functions.

Native functions like `mysqli_query()` should be covered in a future RFC, but
still only use warnings. The exact details would need to be confirmed, but
should provide a simple way to mark certain non-literal strings as trusted,
and still have a way to switch off those warnings entirely (e.g. legacy
systems).

Ideally as future scope, we would also provide a way for certain projects
to opt-in to having exceptions raised (the militant option), which would
not be appropriate for the vast majority of developers, but it's the level
I'd find useful for some of my projects (dealing with medical records and
other sensitive information).

As to how we could (in a future RFC) mark certain non-literal strings as
trusted, I've briefly touched on how this can work in the "Future Scope"
section (it's an idea I copied from Trusted Types in JavaScript, which
follows the same principle). I only did a summary because the exact
implementation would need to be discussed later (there are a few options,
but they all rely on is_literal() being implemented first).

First, we need libraries to use is_literal() to check inputs from
developers (i.e. the code they didn't write), which is their main source of
injection vulnerabilities. What goes on inside the library is entirely up
to them (they know that they are doing). For their output, they create a
stringable value-object, which (as future scope) could be marked as trusted
for certain functions (like mysqli_query).

These value-objects could be as simple as a single private property
(value), a __construct() method that takes the libraries output, and a
__toString() method to return it; or the library might want to be extra
sure of its output, and use something like the Query Builder example linked
in the "Non-Literal Values" section.

As future scope, this allows functions like `mysqli_query()` to simply check
for a literal, or one of these trusted value-objects (or have warnings
switched off entirely).


3. I notice your RFC does not grant `sprintf()` the ability to return a
> string with the "literal" flag even though `sprintf()` can frequently return
> strings composed from known literals and safe numeric value.
>


Joe has created a `literal_sprintf()` function for testing, I think it does
exactly what you're suggesting, and I believe that logic can be applied
directly to `sprintf()`.

The only reason I've been hesitant is to keep this implementation of
is_literal() limited to identifying literals and supporting concatenation
(to keep it simple, and easy to use).

It's like the "String Splitting" part in the RFC, it's probably fine, and
we can do it if there's interest; but we also need to consider what issues
it might cause (if any). From a security point of view, it's always best to
keep something as simple as possible (and that may well include supporting
sprintf), because "You cannot prove security. You can only prove
insecurity".



> 4. Regarding the section "WHERE IN" you argue that we should push everyone
> to use parameters properly — which is a great idea in theory — but that
> would have a significant breakage in existing sites online.
>


Hopefully my answer to #2 addresses this, as you're right, that would
definitely cause more problems than it solves. Certainly for this
implementation I support libraries providing warnings and think those would
be the most appropriate, rather than exceptions. (Though obviously
libraries know their user bases best of course).

In summary of my future scope answer above, libraries could be able to mark
their output as trusted for functions like `mysqli_query()`. For SQL it would
ideally still be a literal string, but that's not always possible
(especially with HTML), so that's where the trusted stringable
value-objects would be used. For legacy systems, developers could simply
switch off the warnings entirely (but at least that's a choice they are
actively choosing to make).



> 5. Given #2 and #4, I think your section "Support Functions" is missing
> [...] as_literal() or make_literal() or even as_unsafe_literal()
>


I'm not against it, but I don't think it's needed (and with it raising
warnings not exceptions it won’t be necessary to prevent things breaking).

In the Example Library I made under the "Try It" section in the RFC:

https://github.com/craigfrancis/php-is-literal-rfc/blob/main/justification/example.php?ts=4

It checks for literals but also accepts a value-object allowing you to
bypass that check (which I think is probably the best way to implement this
at the moment). It shows how an `unsafe_value` value-object could be used
and accepted by the example library, in the same way DB::raw() works in
Laravel. In the future a similar setup in PHP’s native functions could
accept literals or value-objects which can be trusted, so we shouldn’t need
a way to lie to PHP and tell it to treat one as the other.


6. Regarding the section "Reflection API," shouldn't their be an
> is_literal()/is_from_literal() method added to the ReflectionParameter()
> and ReflectionProperty() classes?
>


If I can just check we are talking about the same thing, so where we have:

  class my_class {
    private string|int $my_property = 0;
    private function my_method(string|int $my_parameter) {
    }
  }

  $reflector = new ReflectionClass('my_class');
  $property = $reflector->getProperty('my_property');
  $method = $reflector->getMethod('my_method');
  $parameter = $method->getParameters()[0];

  var_dump(
      $property->getType()->__toString(),
      $parameter->getType()->__toString(),
    );

The ReflectionParameter/ReflectionProperty allows us to return their
respective types (ReflectionType). In this case, it's saying these two are
"string|int", which isn't the current value, just what they can accept.

I think this is touching on an area that Joe mentioned yesterday, "a
possible future where we have first class support" in a more intricate way.
I think there is value in this, as it would allow PHP itself to reject
non-literal values, but that should be a separate RFC, as that's a bit more
of a language change, and will need its own discussion.


So assuming you address at least #5 — and thus #2 and #4 indirectly — I
> would really like to see this RFC pass.
>
> -Mike
>


Thank you Mike

Craig

---

## #114846 — someniatko — 2021-06-13T19:46:47+00:00

URL: https://externals.io/message/114835#114846 · In reply to: #114835

Hi!

From what I understood reading the RFC, this looks like a new type,
"literal string", specifically a subtype of "string". Have you
considered instead of forcing library authors to use `is_literal()`
check everywhere on their input boundary and making sure no place
where input is passed, is forgotten, to let them type their whole
codebase, including internal, against the new `literal-string` type,
which would be enforced by PHP all the time? From my perspective as a
PHP developer, it would serve clearer intent than only checking in a
few places which presumably may take user input, i.e. you could
explicitly state "this function does only work with literal strings".

Regards,
someniatko

---

## #114847 — Craig Francis — 2021-06-13T20:12:07+00:00

URL: https://externals.io/message/114835#114847 · In reply to: #114846

[...] Have you considered [a] new `literal-string` type, which would be
enforced by PHP all the time?


Thanks someniatko, a "clearer intent" is a really nice way of putting it.

It is something that Joe is considering, but for a future version,
simply because it will require more discussion (which I'm happy to help
with).

And I agree, there would be value in having a `literal-string` type,
especially when working with IDE's and Static Analysers; but it brings a
couple of issues that the is_literal() function solves:

1) Libraries need to support older versions of PHP, so they wouldn't be
able to use the `literal-string` for many years; whereas they can use
`function_exists('is_literal')` today, probably to provide their own
function that simply returns true on pre PHP 8.1.

2) The is_literal() function gives the libraries a choice on how to handle
the situation. At the moment they just receive a string of unknown origin,
so they can trigger a warning, while also supporting legacy projects who
want to switch off the warnings. And some developers might want to use a
library that has an option that can raise exceptions (so they can be
certain there are no issues).

Craig

---

## #114848 — Matthew Brown — 2021-06-13T21:30:06+00:00

URL: https://externals.io/message/114835#114848 · In reply to: #114835

On Sat, 12 Jun 2021 at 13:00, Craig Francis <craig@craigfrancis.co.uk>
wrote:

> Hi Internals,
>
> I'd like to start the discussion on the is_literal() RFC:
>
> https://wiki.php.net/rfc/is_literal
>
> is_literal() brings a proven way to identify Injection Vulnerabilities to
> PHP, already used by Google in their Java and Go projects, and is currently
> being added to JavaScript. It's a lightweight and simple approach:
> "Distinguishing strings from a trusted developer from strings that may be
> attacker controlled", allowing Libraries to identify the mistakes made by
> the thousands of developers using them incorrectly.
>
> When Libraries use is_literal() to protect against these mistakes, we can
> trust their output, and (as covered in the Future scope section) PHP can
> then raise warnings with certain native functions like PDO::query,
> mysqli_query, exec, preg_match, etc. (we would only consider warnings,
> anything stricter like exceptions would be in many years time, if at all -
> the intention is to alert and inform people, not break things).
>
> The length is due to the FAQ Section, on why it's needed, how it can be
> used by Libraries, and the important differences of using this flag versus
> the flawed Taint Checking approach with its false sense of security
> (error-prone escaping).
>
> Thanks,
> Craig Francis
>
>
> PS: If anyone wants to discuss face-to-face on Zoom, I'll be available (UK
> Time/BST/UTC+1) at:
>
> https://chat.craigfrancis.co.uk/
>
> Saturday 12th June, 6pm to 8pm;
> Sunday 13th June, 10am to Midday;
> Monday 14th June, 5pm to 7pm;
> Tuesday 15th June, 9pm to 11pm;
> Thursday 16th June, 10am to Midday;
> (other times on request)
>


Hi Craig and Dan,

I was skeptical about the first draft of this RFC when I saw it last
month, but now I see the light (especially with the concat changes). This
looks like a very solid solution for any library authors wanting to add a
layer of protection against SQL injection attacks. Sorry for any
unnecessary grief I caused.

The only remaining issue I have is performance — Psalm and other static
analysis tools perform quite a lot of concatenation (and never have to
worry about user input). What sorts of slowdowns do you see when running
those tools?

Best wishes,

Matt

---

## #114849 — Craig Francis — 2021-06-14T00:20:41+00:00

URL: https://externals.io/message/114835#114849 · In reply to: #114848

On Sun, 13 Jun 2021 at 22:30, Matthew Brown <matthewmatthew@gmail.com>
wrote:

> I was skeptical about the first draft of this RFC when I saw it last month
> [...] Sorry for any unnecessary grief I caused.
>


Hi Matthew, actually I'd like to thank you for that early feedback, the
first draft did need work doing to it, and your feedback, along
with others, helped me focus on the main concerns.



> The only remaining issue I have is performance — Psalm and other static
> analysis tools perform quite a lot of concatenation (and never have to
> worry about user input). What sorts of slowdowns do you see when running
> those tools?
>


For Psalm in particular, I’ve just done a quick run of 10 alternating tests
on a website with 90,000 lines of code, at error level 2 (the default
level), using `--no-cache`, using PHP 8.1.0-dev from a couple of weeks ago.
Without `is_literal` it averaged 8.355s, and with is_literal, 8.373s - so
that’s a +0.018s difference, or +0.22% (full numbers below).

It’s not full lab-condition testing, but hopefully more than representative
enough.

(And repeating the RFC, just for a quick reference: Máté, who does
stress-tests on servers, found that the Laravel Demo was +0.124%,
and Symfony Demo was +0.161%. There was also a stress test, which simply
concatenated 4 strings, 5 million times, with no other actions, so not
exactly representative of a typical PHP script, which was +3.719%. None of
these tests involved connecting to the database, as the variability
introduced made it impossible to measure that level of difference).


Full Psalm-check Stats:

A = Without is_literal
B = With is_literal

A = Checks took 8.36 seconds and used 229.894MB of memory
B = Checks took 8.46 seconds and used 229.894MB of memory
A = Checks took 8.43 seconds and used 229.894MB of memory
B = Checks took 8.57 seconds and used 229.894MB of memory
A = Checks took 8.50 seconds and used 229.894MB of memory
B = Checks took 8.40 seconds and used 229.894MB of memory
A = Checks took 8.36 seconds and used 229.894MB of memory
B = Checks took 8.32 seconds and used 229.894MB of memory
A = Checks took 8.31 seconds and used 229.894MB of memory
B = Checks took 8.38 seconds and used 229.894MB of memory
A = Checks took 8.30 seconds and used 229.894MB of memory
B = Checks took 8.29 seconds and used 229.894MB of memory
A = Checks took 8.31 seconds and used 229.894MB of memory
B = Checks took 8.37 seconds and used 229.894MB of memory
A = Checks took 8.37 seconds and used 229.894MB of memory
B = Checks took 8.32 seconds and used 229.894MB of memory
A = Checks took 8.31 seconds and used 229.894MB of memory
B = Checks took 8.29 seconds and used 229.894MB of memory
A = Checks took 8.30 seconds and used 229.894MB of memory
B = Checks took 8.33 seconds and used 229.894MB of memory

---

## #114850 — Mike Schinkel — 2021-06-14T00:41:09+00:00

URL: https://externals.io/message/114835#114850

> 
> 
> 
>  
> 1. [...] Minimally I'd ask for a separate vote on is_literal() vs. is_from_literal().
> 
> It would definitely be possible to have a vote on the name.

Cool.

> 2. Regarding the sections "Non-Parameterised Values" and "Future Scope," if I understand correctly...
> 
> ...you're right, we do need to support PhpMyAdmin, Adminer, systems like Drupal (...)

That is good to know.

> Native functions like `mysqli_query()` should be covered in a future RFC, but still only use warnings. The exact details would need to be confirmed, but should provide a simple way to mark certain non-literal strings as trusted, and still have a way to switch off those warnings entirely (e.g. legacy systems).

Having code that is valid for a given application throw warnings is a huge non-starter to me. 

During development I believe is is a best practice to develop with display of errors *and* warnings to eliminate all warnings and all errors before moving to testing, staging and production. If warnings are not eliminated it is to easy for a developer to miss warnings that are important vs. ones that are "OK to ignore."

To illustrate, I just added a single `trigger_error()` to the project I am currently working on.  Take a look at what it does to my project:

https://www.evernote.com/l/AAV2_-KSz2RA9pcwtWo8AQJseoZYHtnD-aw

It would be impossible to program with any library that throws a warning and also develop with a "zero warnings" policy.

So again, having legitimate code display warnings is a full-stop "no" in my book.

More on this below.

> Ideally as future scope, we would also provide a way for certain projects to opt-in to having exceptions raised (the militant option), which would not be appropriate for the vast majority of developers, but it's the level I'd find useful for some of my projects (dealing with medical records and other sensitive information).
 
Yes, when you are in full control of your source code, being able to use is_literal() liberally would be great. 

BUT.  There needs to be a way to mark certain non-literal strings as trusted for when using Other People's Code(tm), and as I will argue in the rest of this reply, that way needs to be part of the RFC that also introduces is_literal().  Doing otherwise is like manufacturing a poison but not also manufacturing its antidote.

> As to how we could (in a future RFC) mark certain non-literal strings as trusted, I've briefly touched on how this can work in the "Future Scope" section (it's an idea I copied from Trusted Types in JavaScript, which follows the same principle). I only did a summary because the exact implementation would need to be discussed later (there are a few options, but they all rely on is_literal() being implemented first).
>  
> First, we need libraries to use is_literal() to check inputs from developers (i.e. the code they didn't write), which is their main source of injection vulnerabilities. 

And that is where the first problem lies.  Assuming libraries do start using is_literal() and there is not yet an existing way to mark a string as trusted, then at best-case the developer using that library will have to jump through hoops when they need a non-literal use-case, and worse-case this will mean the developer will be unable to use said library for their use-case. 

And it might be the only library available, or only one "approved" by their company (which would be worse than the above worst case.)

Introducing is_literal() without make_literal() is like manufacturing a gun without a safety catch.  

I assume that implementing make_literal() would not be a technical challenge, or would it?

Would there be any reason *not* to include make_literal() as part of your RFC?

> What goes on inside the library is entirely up to them (they know that they are doing). For their output, they create a stringable value-object, which (as future scope) could be marked as trusted for certain functions (like mysqli_query).
> 
> These value-objects could be as simple as a single private property (value), a __construct() method that takes the libraries output, and a __toString() method to return it; or the library might want to be extra sure of its output, and use something like the Query Builder example linked in the "Non-Literal Values" section.

Having library developers create a stringable value-object is a nice idea, but you are assuming all library vendors will be enlightened enough to do so. 

On one end you will have library developers who are not sufficiently educated on this topic (and I will remind you of the RFC's comment regarding education) and on the other end you will have library developers who are dogmatic in their belief that requiring actual literals is the only one true way to program (and here I will remind you of the RFC's comment regarding static analysis.)  

In either extreme you will have the library vendor who has taken control away from the developer to use the library and still be able to implement their use-case.

And if you think I am being unrealistic I point you to the developers of the 60k plugins on WordPress.org.  You have lots of less than fully educated developers publishing plugins that are used by unsuspecting users there, and a handful of super dogmatic developers too.

> As future scope, this allows functions like `mysqli_query()` to simply check for a literal, or one of these trusted value-objects (or have warnings switched off entirely).

How might `mysqli_query()` check for a "trusted value-object" and who could declare a class of that nature? 

Elaborating on ways this could be done really needs to be added to the RFC otherwise how are voters to know what is likely and/or even possible in the future?

> 3. I notice your RFC does not grant `sprintf()`...
> 
> <snip>
> 
> It's like the "String Splitting" part in the RFC, it's probably fine, and we can do it if there's interest; but we also need to consider what issues it might cause (if any). From a security point of view, it's always best to keep something as simple as possible (and that may well include supporting sprintf), because "You cannot prove security. You can only prove insecurity".

If the RFC provides make_literal() then having `sprintf()` be able to return a literal flag would just become a nice-to-have instead of a must-have.

> 4. Regarding the section "WHERE IN" you argue that we should push everyone to use parameters properly — which is a great idea in theory — but that would have a significant breakage in existing sites online.
> 
> Hopefully my answer to #2 addresses this, as you're right, that would definitely cause more problems than it solves. Certainly for this implementation I support libraries providing warnings and think those would be the most appropriate, rather than exceptions. (Though obviously libraries know their user bases best of course).

A big *NO* on warnings. Full stop.

An alternate could be (something like):

declare(relax_literal_checks=1).  

While I am not a big fan of adding declare() directives, this seems like a good use because it would only be needed for special cases, and a developer could structure their PHP files such that the files needing it could be limited to just the code that needs to relax literal checking.

BTW, I am assuming that setting this one would have the affect of adding the literal flag to all strings used in the file.

> In summary of my future scope answer above, libraries could be able to mark their output as trusted for functions like `mysqli_query()`. For SQL it would ideally still be a literal string, but that's not always possible (especially with HTML), so that's where the trusted stringable value-objects would be used. 

If the library vendor must develop the trusted value object then you are depending on an unreliable actor. 

Further, if every library vendor gets to define their own value object it could make life very confusing for application developers who have to learn which different value objects for every different library will solve the "Sorry, you can't use a non literal here" warning or exception.  

I can already see the avalanche of StackOverflow questions that requires lots of questions for the questioner, and the answer is always "It depends..."

So ideally there would either be "trusted value-objects" provided by PHP core, or they would be able to be easily identified by some consistent trait, such as a "as_literal()" method, or similar.

> 5. Given #2 and #4, I think your section "Support Functions" is missing [...] as_literal() or make_literal() or even as_unsafe_literal()
> 
> I'm not against it, but I don't think it's needed (and with it raising warnings not exceptions it won’t be necessary to prevent things breaking).

Count me as one who thinks that it is absolutely, unequivocally essential because warnings are not a valid solution.

See the result of this simple `trigger_error()` again?  https://www.evernote.com/l/AAV2_-KSz2RA9pcwtWo8AQJseoZYHtnD-aw

> In the Example Library I made under the "Try It" section in the RFC:
> 
> https://github.com/craigfrancis/php-is-literal-rfc/blob/main/justification/example.php?ts=4
> 
> It checks for literals but also accepts a value-object allowing you to bypass that check (which I think is probably the best way to implement this at the moment). It shows how an `unsafe_value` value-object could be used and accepted by the example library, in the same way DB::raw() works in Laravel. In the future a similar setup in PHP’s native functions could accept literals or value-objects which can be trusted, so we shouldn’t need a way to lie to PHP and tell it to treat one as the other.

That is a great example showing how a qualified developer would attack the requirement.

It is also significantly more elaborate than many developers working on libraries will go to the trouble to implement.  

And I don't mean that most popular libraries, I mean the long tail of libraries, including internal-use libraries developed within an organization that have to be used by others in that organization where internal politics do not allow the developer in need to modify the library written by their developer in control of the library. 

With great power comes great responsibility, but we both know that in the real world power corrupts.  In my experience, anyway.

> 
> 6. Regarding the section "Reflection API," shouldn't their be an is_literal()/is_from_literal() method added to the ReflectionParameter() and ReflectionProperty() classes?
> 
> <snip>
> The ReflectionParameter/ReflectionProperty allows us to return their respective types (ReflectionType). In this case, it's saying these two are "string|int", which isn't the current value, just what they can accept.
> 
> I think this is touching on an area that Joe mentioned yesterday, "a possible future where we have first class support" in a more intricate way. I think there is value in this, as it would allow PHP itself to reject non-literal values, but that should be a separate RFC, as that's a bit more of a language change, and will need its own discussion.

Lack of reflection support is not a deal-killer in my eyes, just a missing aspect to make the feature fully fleshed out.  But I think it could come later without causing major issues for developers in the mean time.

-Mike

---

## #114851 — someniatko — 2021-06-14T07:26:21+00:00

URL: https://externals.io/message/114835#114851 · In reply to: #114847

> And I agree, there would be value in having a `literal-string` type, especially when working with IDE's and Static Analysers; but it brings a couple of issues that the is_literal() function solves:

Yeah, those are valid arguments why we should have a dedicated
`is_literal()` function, and in fact we could have both the type and
the function at the same time, the same as we have `string` type and
`is_string()` function. My worry is, if type is not added in the
present RFC, adding it later may look as "too low value" and probably
will have a lower chance of being accepted.

> 1) Libraries need to support older versions of PHP, so they wouldn't be able to use the `literal-string` for many years; whereas they can use `function_exists('is_literal')` today, probably to provide their own function that simply returns true on pre PHP 8.1.

If the type is added in say 8.2, they wouldn't be able to use it even
longer. Of course this can be re-modelled using a value object which
performs the is_literal() check in the constructor, however I think a
native PHP type would be better, for example in the case of variance
checks (being a subtype of "string").

> 2) The is_literal() function gives the libraries a choice on how to handle the situation. At the moment they just receive a string of unknown origin, so they can trigger a warning, while also supporting legacy projects who want to switch off the warnings. And some developers might want to use a library that has an option that can raise exceptions (so they can be certain there are no issues).

If we have both the fn and the type, libraries would still have that
choice, either enforce it solidly and throw TypeError (PHP 8.1), or
polyfill the `is_literal()` function to always return `true` and be
able to both take advantage of PHP 8.1 and work on the earlier
versions.

I hope these points will be really considered, and PHP's type system
will evolve into more robust and strong one. (offtopic: hoping we will
have typed vector and dict types someday too)

> Craig

Regards,
someniatko

---

## #114852 — someniatko — 2021-06-14T07:34:11+00:00

URL: https://externals.io/message/114835#114852 · In reply to: #114851

Nevertheless, I am sure even without a dedicated type it is still a
valuable addition to the engine. Just to clarify, I understand that if
this would require too much additional work to implement the type, IMO
it's okay to ship the feature without it for now. I was just replying
to other arguments, which are not related to the work amount :P

Thank you for your work!

Regards,
someniatko

---

## #114853 — Pierre — 2021-06-14T08:07:04+00:00

URL: https://externals.io/message/114835#114853 · In reply to: #114850

Le 14/06/2021 à 02:41, Mike Schinkel a écrit :
> A big*NO*  on warnings. Full stop.

Hello,

I agree. A lot of DBAL / ORM / query builders tools exist and use the 
mysqli_, pgsql_ or PDO libraries as simple arbitrary SQL executor, and 
handle the security aspect at a higher level. Any warning raised by the 
low level functions would be too restrictive, and in almost all cases 
always false positives.

Regards,

--

Pierre

---

## #114855 — Craig Francis — 2021-06-14T09:36:24+00:00

URL: https://externals.io/message/114835#114855 · In reply to: #114852

> I am sure even without a dedicated type it is still a valuable addition to
> the engine [...] IMO it's okay to ship the feature without it for now.



Thanks someniatko, it's certainly something I'm interested in (8.2?), as I
think a dedicated type would be useful, I'm just cautious of trying to
cover too many things before the 8.1 deadline - especially because
types will be part of the native functions discussion.

---

## #114856 — Craig Francis — 2021-06-14T09:36:30+00:00

URL: https://externals.io/message/114835#114856 · In reply to: #114853

> Le 14/06/2021 à 02:41, Mike Schinkel a écrit :
> > […]
>
> [...] Any warning raised by the low level functions would be too
> restrictive.
>


Hi Pierre, I'll be talking to Mike on Zoom later today (5pm UK time), which
you're welcome to join.

Just to confirm, this RFC is about the literal flag, and giving libraries
the ability to use it via the is_literal() function (which I'm glad to hear
peoples support for).

It does not raise any warnings, or affect native / low-level functions. How
the flag can be used in the future, like the dedicated type that someniatko
and Joe have suggested, is part of a larger conversation which I'm looking
forward to having in the near future. And just to put your mind at ease,
I'm very much aware that systems that causes too much noise (false
positives) will simply be switched off, which is no use to anyone.

Craig

---

## #114857 — Dik Takken — 2021-06-14T10:53:24+00:00

URL: https://externals.io/message/114835#114857 · In reply to: #114846

> Hi!
> 
> From what I understood reading the RFC, this looks like a new type,
> "literal string", specifically a subtype of "string". Have you
> considered instead of forcing library authors to use `is_literal()`
> check everywhere on their input boundary and making sure no place
> where input is passed, is forgotten, to let them type their whole
> codebase, including internal, against the new `literal-string` type,
> which would be enforced by PHP all the time? From my perspective as a
> PHP developer, it would serve clearer intent than only checking in a
> few places which presumably may take user input, i.e. you could
> explicitly state "this function does only work with literal strings".

That sounds like a great future extension. Then it would be nice to have
the name of the new type match the name of the function that checks for
it. In case the existing 'string' type would be paired with a new
'lstring' type the existing `is_string()` would pair nicely with a new
is_lstring() function.

Regards,
Dik Takken

---

## #114858 — Dan Ackroyd — 2021-06-14T12:29:53+00:00

URL: https://externals.io/message/114835#114858 · In reply to: #114836

Hi Craig, Joe,

While I think the idea behind this RFC is great, the current
implementation is terrible, as it will cause some fatal errors in
production.

# The problem

Imagine we have this code:

function getInfoPanel(UserPreferences $up): string
{
    $html = "<div style='color: " . $up->getColor() . "'>";
    $html .= // more html here
    $html .= "</div>"

    return $html;
}

class UserPreferences {
    private DB $db;

    function getColor(): string {
        $preferredColor = $this->db->getColor();

        if ($preferredColor === 'light') {
            return '#fff';
        }

        if ($preferredColor === 'dark') {
            return '#000';
        }
        return '#fff'; // default is light
    }
}

And then the string returned by getInfoPanel() is used elsewhere that
does a check for is_literal. That check will pass.

Now imagine a feature request comes in, to allow people to customise
the color of their UI, so the UserPreferences code is changed to:

class UserPreferences {
    private DB $db;

    function getColor(): string {
        $preferredColor = $this->db->getColor();

        if ($preferredColor === 'light') {
            return '#fff';
        }

        if ($preferredColor === 'dark') {
            return '#000';
        }
        return $preferredColor; // user has set custom color.
    }
}

Assume that both UserPreferences and getInfoPanel code is covered by
unit tests. The developer who made that change would run the tests,
see the tests pass and then push their code live. At which point, it
would cause an error in production, as the string returned would not
pass the is_literal check.

This would be a complete pain in the butt to fix.

The only information would be that the html string is not
literal....but there wouldn't be any info about which bit was the
problem. Someone would have to manually check which code had been
called to produce the non-literal string.

That could mean  the site would be unavailable for users that had set
a custom color. Or more likely, the developers would just turn off the
is_literal check or at least downgrade it to a mere logging error aka
something to be ignored.


# How to fix this in the RFC

By not carrying the literal flag on string concatenation, but
requiring developers to use functions like literal_concat() and
literal_implode(), the 'false affordance'* goes away:

function getInfoPanel(UserPreferences $up): string
{
    $html[] = "<div style='color: " . $up->getColor() . "'>";
    $html[] = // more html here
    $html[] = "</div>"

    return literal_concat(...$html);
}


Now, when the feature request comes in to allow users to set custom
colors, the code will fail on the literal_concat. When that function
checks internally that all the strings passed to it are literal, it
will throw an exception from the function that has made the bad
assumption about whether a string is user input or not.

Yes, it's slightly ironic that making errors be louder and happen
earlier in the program is better than making them happen later.

# Wouldn't forcing people to use literal_concat() and
literal_implode() be annoying

I don't think so.

The code change required is pretty small, and can be done either by a
junior developer, or could be automated by things like
https://github.com/rectorphp/rector. Even if there are 100 places
where an existing series of string concatenations needs to be
converted to chucking in an array, and then using literal_concat() and
each of those takes 10 minutes... that's 2 days to do the conversion.
In return for which the application becomes a lot more secure than it
was previously.

But additionally, I don't think many people are actually going to be
using literal_concat() in many places. My guess is that the vast
majority of places where you are combining bits of strings together,
you would realise that you can switch to using type-specific escaping,
as you're going to need to do the escaping anyway.


# Is this a problem in other languages?

The similar implementation used in Google is used for Java. As Java is
statically compiled, any attempt to pass a non-literal string to a
function that expects a literal string fails to compile. And if it
won't compile, it won't run, and so won't have errors in production.

The only thing that needs to be changed in my opinion is not carrying
the literal flag through string concatenation. Although that would be
slightly more work to use the feature, avoiding having inevitable and
hard to debug errors in production is the trade-off.

cheers
Dan
Ack


* false affordance - something that works, except when it doesn't.

---

## #114859 — Matthew Brown — 2021-06-14T13:06:08+00:00

URL: https://externals.io/message/114835#114859 · In reply to: #114858

> Hi Craig, Joe,
>
> While I think the idea behind this RFC is great, the current
> implementation is terrible, as it will cause some fatal errors in
> production.
>
>
I'm not sure it's productive to call the implementation terrible. With
Someniatko's suggestion of a literal-string type, that issue (like so many
others) could be caught by a type-checker:

    /** @return literal-string */
    function getColor(): string {
        return $_GET["color"]; // this would trigger a typechecker error
    }

In fact I think there's a solid case for adding support for literal-string
to type checkers like mine today — mainly to prevent patterns that might
later lead to SQL injection.

---

## #114860 — Thomas Nunninger — 2021-06-14T13:27:47+00:00

URL: https://externals.io/message/114835#114860 · In reply to: #114858

Hi!

> class UserPreferences {
>      private DB $db;
> 
>      function getColor(): string {
>          $preferredColor = $this->db->getColor();
> 
>          if ($preferredColor === 'light') {
>              return '#fff';
>          }
> 
>          if ($preferredColor === 'dark') {
>              return '#000';
>          }
>          return $preferredColor; // user has set custom color.
>      }
> }
> 
> Assume that both UserPreferences and getInfoPanel code is covered by
> unit tests. The developer who made that change would run the tests,
> see the tests pass and then push their code live. At which point, it
> would cause an error in production, as the string returned would not
> pass the is_literal check.
> 
> This would be a complete pain in the butt to fix.

When you write about testing in this database scenario, I find some 
additional issue: When you try to create pure unit tests you would 
somehow mock/stubb or replace the database with some test double. In 
that situation your tests will still pass even if testing the variant 
that the value comes from the database. Only some (infrastructure or 
end-to-end) test that covers the business logic plus the corresponding 
infrastructure by accident would uncover an error.

I wonder if we would need some method to mark a value/variable as 
non-literal. Or perhaps mark some return value (or input parameters?) in 
an interface/class as non-literal using some annotation?

That still puts the burden on the developer to think about that issue. 
But it's probably better than nothing.

Perhaps someone has a better idea?

Cheers
Thomas

---

## #114861 — Marco Pivetta — 2021-06-14T13:29:59+00:00

URL: https://externals.io/message/114835#114861 · In reply to: #114860

Hey Thomas,



> Hi!
>
> > […]
>
> When you write about testing in this database scenario, I find some
> additional issue: When you try to create pure unit tests you would
> somehow mock/stubb or replace the database with some test double. In
> that situation your tests will still pass even if testing the variant
> that the value comes from the database. Only some (infrastructure or
> end-to-end) test that covers the business logic plus the corresponding
> infrastructure by accident would uncover an error.
>
> I wonder if we would need some method to mark a value/variable as
> non-literal. Or perhaps mark some return value (or input parameters?) in
> an interface/class as non-literal using some annotation?
>
> That still puts the burden on the developer to think about that issue.
> But it's probably better than nothing.
>
> Perhaps someone has a better idea?
>

A small userland utility that writes the data to an in-memory stream, then
reads from it, is sufficient.

---

## #114862 — Craig Francis — 2021-06-14T13:30:19+00:00

URL: https://externals.io/message/114835#114862 · In reply to: #114858

> [...] it will cause some fatal errors in production.
>


No, the only way fatal errors can occur is if you choose to make them fatal
(in userland code).

I have included examples of the literal_concat() and literal_implode()
functions in the RFC, showing how they can be implemented in userland
instead, as those functions can be useful for some projects.

However requiring developers to rewrite all of their code to use
literal_concat() and literal_implode(), with those functions triggering an
exception whenever anything goes wrong, that's going to make adoption very
difficult.

Craig

---

## #114866 — Craig Francis — 2021-06-14T15:06:50+00:00

URL: https://externals.io/message/114835#114866 · In reply to: #114857

> That sounds like a great future extension. Then it would be nice to have
> the name of the new type match the name of the function that checks for
> it. In case the existing 'string' type would be paired with a new
> 'lstring' type the existing `is_string()` would pair nicely with a new
> is_lstring() function.
>


That's a good suggestion for a name as well, especially when thinking about
its use as a type.

I've put up a basic MarkDown file on GitHub (the repo has been my general
dumping ground for files related to this).

https://github.com/craigfrancis/php-is-literal-rfc/blob/main/name.md

I'm happy to make additions and/or edits (can also accept pull requests if
you prefer).

Then maybe next week we can have a simple poll somewhere? Not sure if
anyone has any suggestions, but maybe Google Forms, where we can simply
enter you name and choose your preference (just going on a basic honour
system).

Craig

---

## #114868 — Rowan Tommins — 2021-06-14T18:43:52+00:00

URL: https://externals.io/message/114835#114868 · In reply to: #114858

Hi Dan,



> While I think the idea behind this RFC is great, the current
> implementation is terrible, as it will cause some fatal errors in
> production.


I think you have raised a legitimate concern here, but agree with 
Matthew Brown that calling the implementation "terrible" is neither 
warranted nor helpful.

It's also a bit bizarre to criticise the RFC for causing fatal errors 
even though it doesn't introduce any, then propose a different version 
which *does* introduce fatal errors.


> The only information would be that the html string is not
> literal....but there wouldn't be any info about which bit was the
> problem. Someone would have to manually check which code had been
> called to produce the non-literal string.


The difficulty of tracing why something is not in the expected state is 
indeed a tricky one, but I don't think mandating literal_implode solves 
it well enough to be worth the pain. Essentially, changing getInfoPanel 
to use literal_implode is equivalent to this:



> function getInfoPanel(UserPreferences $up): string
> {
>      $html = "<div style='color: " . $up->getColor() . "'>";
>      $html .= // more html here
>      $html .= "</div>"
>
>      assert( is_literal($html) );
>      return $html;
> }


However, there could still be an arbitrarily long code path between the 
user input and that assertion (or call to literal_implode), so tracking 
the root cause could be just as difficult.

The actual root cause is in getColor(), so the actual assertion needed 
is there:


> class UserPreferences {
>      private DB $db;
>
>      function getColor(): string {
>          $preferredColor = $this->db->getColor();
>
>          if ($preferredColor === 'light') {
>              return '#fff';
>          }
>
>          if ($preferredColor === 'dark') {
>              return '#000';
>          }
>          assert( is_literal($preferredColor) );
>          return $preferredColor; // user has set custom color.
>      }
> }

Adding that assertion would be easier and more elegant if it was 
possible to set a "literal_string" return type, but it would still be up 
to the user to add it. That leads to the original problem, but in 
reverse: when writing the getColor() function, the author needs to know 
where it will eventually be used to know that an assertion is relevant.

It would also not solve the problem of the assertion only triggering in 
production without a careful integration test. That can only be solved 
with a static analyser, which would also have enough information to 
isolate the cause.

Regards,

-- 
Rowan Tommins
[IMSoP]

---

Next page: https://externals.io/message/114835.md?page=2
