Adding the OpenSSF Scorecards GitHub Action

2 years ago by Pedro Nacht via internals — view source

unread

I've made this suggestion as issue #9778 (
https://github.com/php/php-src/issues/9778) and PR # 9789 (
https://github.com/php/php-src/pull/9789), but have been invited by
@damianwadley to bring it to the mailing list.

The Scorecards GitHub Action basically keeps an eye on a repo's security
posture and makes simple, objective suggestions for possible improvements.

For PHP's current Scorecard results, see here:
https://api.securityscorecards.dev/projects/github.com/php/php-src. At the
moment it's a raw json dump, but it contains information on the results of
all the individual checks as well as comments on how to improve the scores.
When the Action is installed, this is cleanly added to the project's GitHub
Security Panel with step-by-step instructions.

@iluuu1994 raised the issue that Scorecards suggests maximal branch
protection and code review (prefer all contributions come via PRs with some
form of code review prior to being added to the repo), which is quite
distinct from the current PHP workflow which allows core maintainers to
simply push directly. The reasons for this are entirely understandable. The
Scorecard simply serves to indicate that other, more secure workflows
exist. Whether their costs (in terms of agility and especially maintainer
time) are worth it is a determination only the core team can make.

I'm happy to answer any questions anyone might have, and am also happy to
help PHP in other ways if I can!

Thanks,
Pedro

P.S. First time contribution to the mailing-list, apologies for any
missteps!

2 years ago by Dan Ackroyd — view source

unread

Hello Pedro.

On Thu, 20 Oct 2022 at 23:26, Pedro Nacht via internals
internals@lists.php.net wrote:

I'm happy to answer any questions anyone might have

What is the morality of open source?

What is the morality of encouraging people to contribute to open source?

When people see open source projects being abused by multi-billion
dollar companies, should they stay quiet or should they ask really
annoying questions on the mailing list?

First time contribution to the mailing-list, apologies
for any missteps!

Apologies in advance for being difficult, but a lot of people who have
contributed to Open Source for years have quite strong feeling about
events that have occurred, and manipulation by large companies.

cheers
Dan
Ack

2 years ago by Jordan LeDoux — view source

unread

On Thu, Oct 20, 2022 at 2:26 PM Pedro Nacht via internals <
internals@lists.php.net> wrote:

I've made this suggestion as issue #9778 (
https://github.com/php/php-src/issues/9778) and PR # 9789 (
https://github.com/php/php-src/pull/9789), but have been invited by
@damianwadley to bring it to the mailing list.

The Scorecards GitHub Action basically keeps an eye on a repo's security
posture and makes simple, objective suggestions for possible improvements.

For PHP's current Scorecard results, see here:
https://api.securityscorecards.dev/projects/github.com/php/php-src. At the
moment it's a raw json dump, but it contains information on the results of
all the individual checks as well as comments on how to improve the scores.
When the Action is installed, this is cleanly added to the project's GitHub
Security Panel with step-by-step instructions.

What actionable benefit could this provide the project? Letting
contributors know that some issues weren't responded to within 24 hours or
something? I mean... none of us are paid to do that. Of course Google OSS
projects have that kind of response, there's a trillion dollar company
paying people to do that. But it's not like seeing a notification about
something like that would provide php-src with actionable information. If
we miss a metric that it's concerned about... who is responsible for doing
something about it? Obviously the project is very active, and we have a lot
of people who contribute, and we often do have quick responses between all
the volunteers for things that need to be done quickly.

But metrics and guarantees? If Christoph goes on vacation and Nikita is
busy with work and Dmitry hasn't checked for notifications for a few days
and everyone else thinks that one of them should weigh in first, then what
would such a score actually tell us about the project? That we don't have
employees responsible for those tasks? We already know that. What would
forcing maintainers to go through a PR and review process for the types of
changes that normally get pushed directly to master provide? A way for
third parties to weigh in? Can't they already do that through the mailing
list, issues, and PRs?

If Google wants to help the PHP project, helping the project is probably
better than supplying a tool that makes volunteers feel obligated in ways
that employees do. Joe, Christoph, Dmitry, Nikita, Dan, etc.... all of
these people with deep knowledge of the project and its history are
critical to the project, but none of them are beholden to it. We were all
sad to hear that Nikita's focus would shift with his new professional
opportunities, but that doesn't mean he was wrong to take those
opportunities or that he owed anything more than he was willing to give to
the project.

I just don't see what tangible benefit or actionable information something
like this could provide. It's neat, and interesting, and maybe a bit of a
novelty. But as part of an organizational workflow for the PHP project...
why?

Jordan

2 years ago by Pedro Nacht via internals — view source

unread

Hey Jordan,

The tool is only meant to be informative regarding the project's
supply-chain security posture and gives actionable suggestions on how it
can be improved. However, it doesn't create issues, bug maintainers and
volunteers with notifications/emails, or make any assumptions regarding
maintainer/volunteer responsiveness. Maintainers decide when they wish to
check the GitHub Security Dashboard to check the project's security posture.

What actionable benefit could this provide the project?

Some of the suggestions it gives for PHP are:

add a security policy to the repo where security vulnerabilities can
be reported privately (the file could simply point users to the
instructions at https://wiki.php.net/security, for example).
set GitHub workflow dependencies to adopt hash-pinning instead of
major-version pinning. This is to eliminate the risk of tag-hijacking
attacks where a malicious commit is published to an Action you rely on and
the tag is then recreated to point to that malicious commit. Dependabot or
Renovatebot (which the tool also recommends adding to the project) can then
be used to keep your dependencies up-to-date.
set all GitHub workflows with top-level read-only permissions. It
recognizes that almost all PHP's workflows have top-level read-only
permissions, but points out that github/workflows/labeler.yml doesn't
(though it does have job-level contents read-only permissions).
and yes, enforce Code-Review and maximal Branch-Protection. I
understand this would be quite impactful on the project's current workflow,
but it's the sort of thing that would mitigate the sort of
attack @ricardoboss mentioned in the linked GH issue. Whether the costs are
worth the benefit is a question you are all certainly better equipped to
determine than me.

Some of these tips are one-offs: for example, once you have a security
policy, there's likely nothing else to do in that regard.

However, others offer more continuous monitoring in case there's an
accidental slip-up: if any new workflows are added to the project in the
future without minimal permissions or without pinning dependencies, for
example, the Action will update the Security Dashboard with an alert.

I'd be happy to submit a few PRs to implement the changes I can.

What would forcing maintainers to go through a PR and review process for

the types of changes that normally get pushed directly to master provide? A
way for third parties to weigh in? Can't they already do that through the
mailing list, issues, and PRs?

Code review is simply a means of getting an extra pair of eyes on any new
code. Be that to perhaps identify bugs, typos, missed edge-cases or actual
malicious activity as seen last year. It does, however, undoubtedly have
associated costs (crucially, volunteer time). As I stated above, the
cost-benefit analysis is best done by you all who have much more experience
with the project.

Thank you,
Pedro

2 years ago by Dan Ackroyd — view source

unread

Hi Pedro,

On Mon, 24 Oct 2022 at 19:06, Pedro Nacht via internals
internals@lists.php.net wrote:

Hey Jordan,

You seem to have responded to Jordan's tech questions, but I was
really hoping for an answer to my more fundamental questions:

Danack wrote:

What is the morality of open source?

What is the morality of encouraging people to contribute to open source?

Though....perhaps we should start you off with an easier question.

What is the morality of OpenSSF asking projects to add these
scorecards to their projects?

I'll even give you a hint; try describing the motivations of the
OpenSSF (and it's backers) and how they differ from contributors to
open source projects.

cheers
Dan
Ack

2 years ago by tim@bastelstu.be — view source

unread

I'm also highly sceptical of wasting CPU time on such an automated scan.

Once the few universally useful low-hanging fruits are picked, only the
non-low-hanging fruits remain by definition. If they are not implemented
reasonably timely then there is likely a reason for that, be that
technical or processual. Then there is also no need for the reports to
sit within the Security tab forever and also no need for them to be
rechecked by maintainers regularly. Then there's also the issue of
what's effectively false-positives (see below) which further distract
from whatever benefit to the automated scan there might be in the first
place.

- set GitHub workflow dependencies to adopt hash-pinning instead of
major-version pinning. This is to eliminate the risk of tag-hijacking
attacks where a malicious commit is published to an Action you rely on and
the tag is then recreated to point to that malicious commit. Dependabot or
Renovatebot (which the tool also recommends adding to the project) can then
be used to keep your dependencies up-to-date.

Is there any practical benefit to this when all the workflows are
read-only with regard to the repository contents?

On the contrary I believe that hashes make it much harder to verify
which major version of an action is used, e.g. to check the changelog
for any relevant breaking changes before upgrading the action.

set all GitHub workflows with top-level read-only permissions. It
recognizes that almost all PHP's workflows have top-level read-only
permissions, but points out that github/workflows/labeler.yml doesn't
(though it does have job-level contents read-only permissions).

I consider that a false-positive then, because the workflows is secure
in practice. No need to waste maintainers time with the busy-work of
moving the 'permissions' section around.

I can see the small benefit of consistency across workflows, though and
will likely send a PR to unify this if no one beats me to it.

and yes, enforce Code-Review and maximal Branch-Protection. I
understand this would be quite impactful on the project's current workflow,
but it's the sort of thing that would mitigate the sort of
attack @ricardoboss mentioned in the linked GH issue. Whether the costs are
worth the benefit is a question you are all certainly better equipped to
determine than me.

I would hope none of the core contributers disputes the benefit of code
review, so … there is no need for a tool to tell them what they already
know.

However, others offer more continuous monitoring in case there's an
accidental slip-up: if any new workflows are added to the project in the
future without minimal permissions or without pinning dependencies, for
example, the Action will update the Security Dashboard with an alert.

The repository is already configured to only grant "read" permissions to
the workflow by default using this setting:

https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository#configuring-the-default-github_token-permissions

I believe this is a much more reliable solution than expecting the
maintainers to regularly check the Security Tab and noticing that a new
warning popped up.

The proposed automated scanner does not appear to detect that this
setting is enabled, thus effectively making the labeler.yml report a
double false-positive.

Best regards
Tim Düsterhus

2 years ago by Pedro Nacht via internals — view source

unread

Tim,

Thank you for the feedback, I'll be sure to pass it on to the Scorecards
team.

And if there's interest, I'll happily fix that small inconsistency in the
labeler.yml permissions.

And with this I believe I'll go ahead and close the GH issue and PR. I'll
stay in the mailing list for a few more days if anyone has any further
questions or comments.

Truly thank you everyone for your time,
Pedro

Hi

I'm also highly sceptical of wasting CPU time on such an automated scan.

Once the few universally useful low-hanging fruits are picked, only the
non-low-hanging fruits remain by definition. If they are not implemented
reasonably timely then there is likely a reason for that, be that
technical or processual. Then there is also no need for the reports to
sit within the Security tab forever and also no need for them to be
rechecked by maintainers regularly. Then there's also the issue of
what's effectively false-positives (see below) which further distract
from whatever benefit to the automated scan there might be in the first
place.
- set GitHub workflow dependencies to adopt hash-pinning instead of
major-version pinning. This is to eliminate the risk of tag-hijacking
attacks where a malicious commit is published to an Action you rely
on and
the tag is then recreated to point to that malicious commit.
Dependabot or
Renovatebot (which the tool also recommends adding to the project)
can then
be used to keep your dependencies up-to-date.
Is there any practical benefit to this when all the workflows are
read-only with regard to the repository contents?

On the contrary I believe that hashes make it much harder to verify
which major version of an action is used, e.g. to check the changelog
for any relevant breaking changes before upgrading the action.

set all GitHub workflows with top-level read-only permissions. It
recognizes that almost all PHP's workflows have top-level read-only
permissions, but points out that github/workflows/labeler.yml doesn't
(though it does have job-level contents read-only permissions).

I consider that a false-positive then, because the workflows is secure
in practice. No need to waste maintainers time with the busy-work of
moving the 'permissions' section around.

I can see the small benefit of consistency across workflows, though and
will likely send a PR to unify this if no one beats me to it.

and yes, enforce Code-Review and maximal Branch-Protection. I
understand this would be quite impactful on the project's current
workflow,
but it's the sort of thing that would mitigate the sort of
attack @ricardoboss mentioned in the linked GH issue. Whether the
costs are
worth the benefit is a question you are all certainly better equipped
to
determine than me.

I would hope none of the core contributers disputes the benefit of code
review, so … there is no need for a tool to tell them what they already
know.

However, others offer more continuous monitoring in case there's an
accidental slip-up: if any new workflows are added to the project in the
future without minimal permissions or without pinning dependencies, for
example, the Action will update the Security Dashboard with an alert.

The repository is already configured to only grant "read" permissions to
the workflow by default using this setting:

https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository#configuring-the-default-github_token-permissions

I believe this is a much more reliable solution than expecting the
maintainers to regularly check the Security Tab and noticing that a new
warning popped up.

The proposed automated scanner does not appear to detect that this
setting is enabled, thus effectively making the labeler.yml report a
double false-positive.

Best regards
Tim Düsterhus

2 years ago by tim@bastelstu.be — view source

unread

as a heads up: Please do not top post (as per
https://github.com/php/php-src/blob/master/docs/mailinglist-rules.md).
Your previous replies were already correct.

And if there's interest, I'll happily fix that small inconsistency in the
labeler.yml permissions.

I obviously can't speak for the others, but I am interested for as I
said in my first reply. Consistency is good, if it makes some automated
scanner happy to prevent users from asking about the results in the
future, even better. So yes, please send a PR. I'd also say: Feel free
to also send PRs for other stuff in the future. A PR that actually
fixes some obvious issue or improves something without a drawback
should generally be well-received. In fact that's how the action
permissions config was introduced in the first place - by an non-core
contributor:

https://github.com/php/php-src/pull/9440

Best regards
Tim Düsterhus

2 years ago by tyson andre — view source

unread

Hi Pedro Nacht,

Hello, I'm working on behalf of Google and the Open Source Security Foundation to help essential open-source projects improve their supply-chain security.

Could you expand on that? It isn't obvious from your comment, and I'm curious about this initiative at Google.

How many hours a week do you spend working for Google/Alphabet, roughly? (e.g., averaged over the last month)
How many hours a week do you spend working for the Open Source Security Foundation, roughly? Is that work part of your job role at Google?
What is your job title, team, and department in those organizations?
What is the team size?

I also had a few other questions:

How many of the top N security-critical open-source projects does the OSSF plan to propose this badge to this year?
What studies have been published or are being conducted by Google/OSSF on the impact of the badge on open-source organizations (or being conducted externally, e.g., by universities) (e.g. comparing organizations where it is proposed to vs not proposed to)? If so, where can I find them?

E.g., I saw https://news.ycombinator.com/item?id=33309969 recently and wanted to learn more about what is known about the impact on metrics of projects short-term and long-term. (e.g. on developers that strongly focus on scorecards, or perfectionists, or averaged)

I'm interested in learning more about what is being done to ensure the overall security, stability, and ongoing improvements of open source software in general as an end user, contributor, maintainer, and user of the companies that use open source software.

This would be useful to know when an organization considers adopting a badge or change to process.
6. Is creating PRs to add this badge part of your job role (If so, the job role of which organization)? Is this done in your free time?

Sorry, it isn't clear - From https://opensource.google/documentation/reference/patching, I see that the use of @google.com emails is required for all open-source contributions, so I was initially confused.
7. Are there recent posts by Google clarifying their involvement in the Open Source Security Foundation (funding provided, team size, shared employees/contractors, etc)?
I wanted to know more.

https://security.googleblog.com/2022/10/announcing-guac-great-pairing-with-slsa.html mentions that the foundation exists,
but doesn't mention any details about how Google is involved in it.

> An open source organization like the Open Source Security Foundation wants to identify critical libraries to maintain and secure....
8. What is the roadmap/timeline for this tool? https://github.com/ossf/scorecard/issues has a lot of open issues.
E.g., avoiding false positives in some contexts seems to be a TODO,
the preview is a one-line JSON dump (https://stedolan.github.io/jq/ is a fantastic tool), and there are a lot of open tickets for the website.

What other practices are planned for inclusion in this badge?

Best regards,
Tyson

2 years ago by Pedro Nacht via internals — view source

unread

Tyson,

Could you expand on that? It isn't obvious from your comment, and I'm

curious about this initiative at Google.

How many hours a week do you spend working for Google/Alphabet,
roughly? (e.g., averaged over the last month)

How many hours a week do you spend working for the Open Source Security
Foundation, roughly? Is that work part of your job role at Google?

What is your job title, team, and department in those organizations?

What is the team size?

[...]

Is creating PRs to add this badge part of your job role (If so, the job

role of which organization)? Is this done in your free time?

Sorry, it isn't clear - From
https://opensource.google/documentation/reference/patching, I see that
the use of @google.com emails is required for all open-source
contributions, so I was initially confused.

I'm a full-time, run-of-the-mill Software Engineer at Google. Specifically,
I'm a member of the Google Open Source Security Team (GOSST). We don't have
a public-facing website I can point you towards, but you can read this
interview with our tech-lead for some background:
https://reproducible-builds.org/news/2022/04/26/supporter-spotlight-google-open-source-security-team
.

GOSST and the Linux Foundation's Open Source Security Foundation (OpenSSF)
were both created in 2020 after the SolarWinds attack. GOSST is a part of
the commitment Google has made to improve the supply-chain security of the
open-source community (
https://www.cnbc.com/2021/08/25/google-microsoft-plan-to-spend-billions-on-cybersecurity-after-meeting-with-biden.html
).

While most GOSST teams work to develop OpenSSF tooling open-source projects
can adopt to improve their supply-chain security (i.e. Scorecards, AllStar,
SLSA), my team is focused on actually trying to improve the supply-chain
security of external projects (which includes suggesting the Action to
projects where we deem it relevant). So yes, offering to help open-source
projects improve their supply-chain security is my full-time job. We've
been referred to as the "Open Source Maintenance Crew" (
https://therecord.media/google-open-source-security-team-openssf/) and are
currently a team of four (started just a few months ago, still ramping up).

I also had a few other questions:

How many of the top N security-critical open-source projects does the
OSSF plan to propose this badge to this year?

What studies have been published or are being conducted by Google/OSSF
on the impact of the badge on open-source organizations (or being conducted
externally, e.g., by universities) (e.g. comparing organizations where it
is proposed to vs not proposed to)? If so, where can I find them?

E.g., I saw https://news.ycombinator.com/item?id=33309969 recently and
wanted to learn more about what is known about the impact on metrics of
projects short-term and long-term. (e.g. on developers that strongly focus
on scorecards, or perfectionists, or averaged)

I'm interested in learning more about what is being done to ensure the
overall security, stability, and ongoing improvements of open source
software in general as an end user, contributor, maintainer, and user of
the companies that use open source software.

This would be useful to know when an organization considers adopting a
badge or change to process.

I'd first like to emphasize that this isn't about a badge, but including
the Scorecards workflow. A project may choose to include a badge in their
README so that consumers of the project can have a better understanding of
its security posture, but the badge is strictly optional. In fact, the PR I
submitted doesn't include the badge.

As for the impact of the Scorecards system, the timing is quite fortuitous:
Sonatype (also a member of the OpenSSF) released their 8th Annual State of
the Software Supply Chain Report a few days ago (
https://www.sonatype.com/state-of-the-software-supply-chain/introduction).
According to their analysis in the "Project Quality Metrics" section, the
Scorecards system is the best single predictor of a project supply-chain
security. Now, to be clear, that analysis was regarding the Scorecards
results, not whether a project did or did not have the Action installed
(the scores can be calculated by anyone via a CLI tool). So not precisely
what you asked, but I hope this demonstrates the signals the workflow tries
to collect are significant. The report also analyses other tools such as
Libraries.io's SourceRank and a bunch of metrics (public and proprietary).

Are there recent posts by Google clarifying their involvement in the
Open Source Security Foundation (funding provided, team size, shared
employees/contractors, etc)?
I wanted to know more.

https://security.googleblog.com/2022/10/announcing-guac-great-pairing-with-slsa.html
mentions that the foundation exists,
but doesn't mention any details about how Google is involved in it.

An open source organization like the Open Source Security Foundation
wants to identify critical libraries to maintain and secure....

Google is a founding member of the OpenSSF. See here for the full list of
member organizations: https://openssf.org/about/members/, and here for the
founding press release:
https://openssf.org/press-release/2020/08/03/technology-and-enterprise-leaders-combine-efforts-to-improve-open-source-security/.
I honestly have no idea or information regarding funding. I honestly don't
know anything about the OpenSSF's team size or whether there are any shared
employees/contractors (I'm not one... just a Google employee doing work
that's aligned with OpenSSF objectives).

What is the roadmap/timeline for this tool?
https://github.com/ossf/scorecard/issues has a lot of open issues.
E.g., avoiding false positives in some contexts seems to be a TODO,
the preview is a one-line JSON dump (https://stedolan.github.io/jq/ is
a fantastic tool), and there are a lot of open tickets for the website.

The tool is indeed still in active development, but I'm not aware of a
specific roadmap. There are many open issues and there is certainly room
for improvement. The current state of the report is one we're keenly aware
of (I actually have to look through those on a daily basis, trust me!) and
has been raised in https://github.com/ossf/scorecard-webapp/issues/206.

What other practices are planned for inclusion in this badge?

Do you mean what new checks will be included? Again, I'm not aware of a
specific roadmap, but there are plenty of ideas being discussed or actively
developed in their issues:
https://github.com/ossf/scorecard/issues?q=is%3Aopen+is%3Aissue+label%3Aenhancement

Thank you and let me know if you have any further questions,
Pedro