Docker Image Scanning: Answering Customer Vulnerability Report from a Vendor Perspective
Containers are ubiquitous in modern production environments within Kubernetes clusters hosted on various cloud platforms such as AWS EKS, Azure AKS, or GCP GKE. By consequence securing Docker images is a necessity for all companies.
Both customers and vendors must ensure that the images they use are free from vulnerabilities that could compromise their systems and their data or, respectively, tarnish their reputation if they ship vulnerable images. Nowadays, it's well known that there are vulnerabilities in Docker images that can be exploited to escape the Docker environment and achieve privilege escalation on the host or just leak confidential data.
A Plethora of Docker Vulnerability Scanning Tools
From my experience as a Security Analyst answering customers vulnerability reports, I have seen many different tools used to do vulnerability scanning on Docker images: Trivy from AquaSec, Clair from Quay , Grype from Anchore, Xray from Jfrog, AWS ECR, Black Duck from Synopsys, and Snyk.
These Docker image scanners generate reports with a list of vulnerabilities affecting every single package embedded in the Docker image. These reports can contain various types of vulnerability IDs: CVEs, GHSA, Snyk, Trivy and can be treated in different formats such as table, json, sarif, and more.
The reality is that it's rare to see reports coming from actual engineers concerned about security, it's more often coming from security compliance departments to comply with whatever certification (ISO27001, SOC2, etc).
This compliance requirements have pushed all customers to analyze Docker images deployed on their infrastructure. This is a quality control that development teams have to pass in order to satisfy their security governance teams, who work in spreadsheets and don't always understand the technicality of a vulnerability. And thus, making it a world of pain for software engineers to deal with these vulnerability reports that are most of the time just noise.
A Vulnerability Scan of the Grafana Docker Image
Let's first analyze the SBOM with Syft of the Grafana docker Image:
When we look at the Grafana 10.3.5 docker image Dockerfile we can see it's based on Alpine Linux version 3.19.1 and uses Node.js version 20 as well as Golang version 1.21.8.
The file consists of multiple stages:
- js-builder: Builds JavaScript assets using Node.js.
- go-builder: Builds Go code.
- tgz-builder: Extracts Grafana binaries from a compressed tarball.
- Final stage: Assembles the Grafana image, installing necessary dependencies, setting up configurations, and copying built assets from previous stages.
From this, we can expect vulnerabilities coming from Grafana itself but also from the Alpine base OS image and the js and go builders only used at the build stage of the Grafana application.
Most of the time, it's a lot of noise with transitive dependencies or packages flagged as vulnerable while the actual vulnerable function in the package is not even used in the application.
Don't take me wrong, it's possible for unused packages to still pose a security risk. This is why you need to ensure that all packages in your Docker image are kept up-to-date and secure, even if they're not directly used by the application.
For example, the Trivy scan can output only 2 CVEs, both affecting Go libraries:
While, the Grype scan mainly outputs CVEs affecting Busybox and glibc which are coming from the underying layers in the docker image:
The Difficulty of Handling Vulnerability Reports
As the number of Docker vulnerability scanning tools has increased, so too has the complexity of managing the resulting vulnerability reports as a vendor. Customers often send these reports to vendors with lists of CVEs, expecting a detailed response for each one. However, there are several challenges vendors face when dealing with these reports.
False Positives
One significant challenge is the presence of false positives. Docker images can inherit vulnerabilities from base images or libraries that are not actively used by the application. Customers may report these vulnerabilities, unaware that they do not pose a risk in their specific context. Vendors must invest time and effort in investigating each reported vulnerability to determine its relevance and potential impact.
Multiple Scanning Tools
Another challenge arises from the multitude of Docker vulnerability scanning tools in use. Customers may employ various tools to scan the same images, each potentially producing different results. And many of these Docker vulnerability scanners are commercial and not free to use. For vendors, automating scanning and patching for every tool can be impractical. This necessitates the adoption of a single source of truth and trust, often leading vendors to rely on one preferred solution for vulnerability scanning.
Docker Hub Vulnerability Scan
Adding to the complexity, Docker Hub now displays vulnerability reports for Docker official images. While this is a positive step towards transparency, it places additional responsibility on vendors to monitor and address these vulnerabilities promptly.
Leveraging SCA Tools
To address this effectively, vendors often turn to Software Composition Analysis (SCA) tools. These tools help identify vulnerable libraries within Docker images and provide insights into their usage. Vendors can prioritize patching and mitigation efforts, reducing false positives and improving overall security.
Regarding the false positives you'll find, depending on how noisy is your scanner. Either you choose to ignore them and have to track why you ignored them for the next versions of your product while verifying if the reason is still valid. Or if possible, you upgrade the package that triggers the false positives to get rid of them.
Furthermore, some vendors, like Canonical (Ubuntu), have taken a proactive steps offering their customers to self-consume their database of CVEs with their CVE search tool.
Customers can freely browse this database and find answers to critical questions regarding the vulnerabilities, such as:
Which version of the software is affected?
Which is the version to upgrade to get the patch?
Is the CVE triaged?
Is there a fix available yet?
By providing this level of transparency and accessibility, Canonical not only reduces the number of vulnerability reports from customers but also empowers them to take proactive measures.
To sum up, because securing Docker images is hard, you need a vulnerability scanning workflow that maximizes detection of true positives while minimizing false positives. And this workflow should be easy to manage for developers and should facilitate the addressing of customer vulnerability reports.
References
[1]: https://www.docker.com/blog/container-security-and-why-it-matters/
[2]: https://www.docker.com/blog/automating-your-containers-security-scanning/