I'm calling this problem "silent (upstream) rebuilds"
Eric Smalling of Chainguard and I have been working for weeks on an idea for how to explain this growing problem, and how we can automate our way through a solution that you can quickly implement
Many of us already auto-build our images based on specific factors, such as new code PRs or new version tags of upstream base images.
More advanced teams even automate dependency updates via Dependabot or Renovate that will kick off a PR. These tools can also update Dockerfiles, Compose, Charts, and K8s YAML when a new semver tag is released.
But what about when tags don't change? What if you need to stay on a specific tagged version for months or longer? The underlying image distro will get worse over time (more CVEs discovered).
There's a BIG gap that I need you to be aware of. Those upstream "base" container images you're pinning to in the FROM line, yeah, they are changing too. Open source official images have mutable tags, and they are frequently rebuilt without changing the tag.
How would we even know that?
Easy, we don't, unless we're tracking the current digest (sha hash) and detecting if it's changing over time for that tag. This means if you pin to python:3.13.8 and deploy it today, it could have been built last week. But if you pull that same image later, it could be a different, rebuilt image.
Pinning to the digest with FROM python:3.13.8@sha256:14adcc7a0... doesn't fix this problem, because that 3.13.8 tag will eventually get rebuilt with an updated version of the Debian distro it's built on. By pinning (a good thing) to the digest, you're now using a fixed point in time, and CVEs are being found that make that image worse as it ages.
Chainguard, the home of the world's first Zero CVE image catalog, rebuilds their base images daily, but we don't have to do that level of brute force rebuilding and redeploying.
Chainguard created a GitHub Action called digestabot, which we'll show you how to use for detecting when that image tag you're using gets updated with a new build digest (likely to remove a CVE in the underlying image OS like Debian, Alpine, or Wolfi). This will work on any upstream image, including Docker Hub, GHCR, Chainguard Images, or base images your team builds.
Only by implementing digestabot + dependabot/renovate + CI automation to rebuild images and redeploy, can you feel confident that your automation will catch all dependency CVE fixes and keep your production image deployments fresh with the most secure images you could build.
The goal is Zero CVEs in production containers. We're going to present a strategy that'll get you closer to that goal.
I hope to see you there, we'll show off our open source tools and GitHub Actions to help you implement this solution in your code!