Investing in Our Foundation of Trust
How we scaled secure container images without slowing down development
At Wealthsimple, we view security as an investment, not just an expense. That's why our Infrastructure Security Team took on a challenge: reduce developer toil while significantly strengthening our container security posture. Our goal? To unlock secure velocity at scale, allowing us to deliver innovative features and unwavering reliability to our clients.
In this post, I'll share our journey migrating from traditional container base images to hardened, minimal container images. We'll explore the 'why' behind our decision, the obstacles we overcame, the solutions we built, and most importantly, how you can approach container security at your organization–without overwhelming your developers or hindering innovation. Think of it as a critical upgrade to your security infrastructure, designed to protect clients and empower your engineering teams.
Why Hardened Base Images?
Containers are the fundamental units of the modern applications that power Wealthsimple. Each container originates from a base image–the operating system and core libraries that serve as its foundation. Relying on popular, off-the-shelf images like Debian, Ubuntu, or even language-specific images like `node:alpine` means inheriting everything baked into those images, including the vulnerabilities. Even newly published images often contain known vulnerabilities, some with critical severity ratings. This presents a few fundamental challenges:
- Security Risks: Containers might ship with outdated libraries, misconfigurations, or unnecessary packages for which there are exploits that a threat actor may leverage to breach our network. This isn't just a technical challenge; it's an opportunity to further strengthen our commitment to protecting our clients' financial information.
- Noise Over Signal: Clearing the backlog of benign or low impact vulnerabilities obscures the vulnerabilities in base images that require immediate attention.
- Developer Friction: Triaging alerts takes up time, diverting developers from building features and the work they want to be doing.
We needed a more secure, reliable foundation. Just as we encourage our clients to build diversified investment portfolios, we required a risk-managed approach to our containers.
The Solution: Chainguard Hardened Images – Minimal, Secure, and Always Up-to-Date
We built new Wealthsimple base images using Chainguard's hardened images. Chainguard provides minimal, zero-CVE container images. They're signed for integrity, continuously rebuilt, and rigorously monitored for vulnerabilities. This significantly reduces our attack surface by minimizing the potential for vulnerabilities.
What the Migration Looked Like: A Cross-Functional Partnership
Success demanded collaboration across teams. We adopted a phased approach involving our Vulnerability Management Team, Developer Tools team, Platform Infrastructure team, and every engineering team shipping code at Wealthsimple across hundreds of repositories.
- Inventory: We meticulously cataloged and mapped all services and their current base images and versions. Before we began the migration, we assigned engineering management stakeholders and directly responsible individuals (DRIs) to over one hundred Wealthsimple services.
- Automation: We automated patching our base images across all supported versions. This meant our production images were continuously being patched.
- Hardening: We removed unnecessary dependencies and locked down permissions within our Dockerfiles.
- Validation: We implemented a methodical validation process, scanning each service both before and after deploying Chainguard images. This allowed us to directly measure and confirm the reduction of CVEs, providing quantifiable, data-driven metrics of our security posture.
- Developer Engagement: We hosted office hours, created project channels and delivered weekly updates about how the migration roll out was going. We partnered with DRIs from each service to test in non-production environments.
- Rollout: The Security team owned all pull requests and local testing. One by one, we partnered with teams to ship to staging, test, and deploy to production without impacting our customers.
Challenges We Faced (And How We Overcame Them)
We encountered several challenges along the way. Because Chainguard builds a minimal OS, we needed to explicitly install the necessary packages, rather than relying on pre-bundled libraries. The differing OS environments also meant adjusting our Dockerfiles to ensure compatibility when transitioning from one operating system to another. To support these changes, we invested heavily in education, workshops, regular office hours, and comprehensive documentation to make this project and container security both approachable and understandable for our teams.
The Payoff: Secure Velocity & Culture Shift
The migration reduced OS Level CVEs by 97% across all Wealthsimple services. And most importantly, we didn't slow down development. The response from our engineering teams has been overwhelmingly positive. Since completion a few months ago, we've seen increased security awareness, with teams proactively reaching out to discuss infrastructure security considerations for new services. New developers report faster onboarding and feeling more confident building secure services with clear, hardened defaults in place.
We've also experienced reduced alert fatigue. As one software engineer noted, "It was so easy to migrate our service and amazing to instantly see the reduction of vulnerabilities." They couldn't wait to share the results with their team. Additionally, the project has sparked cross-team collaboration with increased dialogue between security and product engineering teams. This shift in how teams think about and integrate security represents one of the project's most valuable outcomes. Security is increasingly viewed as an enabler rather than a blocker.
Lessons Learned: Principles for Secure Containerization
Our experience highlighted a few key takeaways:
- Make the secure path the default path.
- Smaller images lead to fewer vulnerabilities.
- Close collaboration between security and development teams is essential.
- Security is an ongoing journey requiring continuous improvement.
What’s Next: Image Signing with Cosign and Kubernetes Policy Enforcement
Migrating to hardened base images was a significant step, but only the first. In the coming weeks, we'll be implementing image signing using Cosign.
Hardened images ensure security at build time. However, we must also ensure that the container running in production hasn't been tampered with.
Image signing allows us to cryptographically sign container images, verify signatures automatically, and ensure image provenance. We're integrating Cosign directly into our CI/CD pipelines. We'll be enforcing signature verification inside Kubernetes using Kyverno, a Kubernetes-native policy engine that continuously validates image integrity. This will result in a comprehensive security approach: hardened images reduce vulnerabilities, signed images ensure the right images are deployed, and policy enforcement guarantees it stays that way. Together, we're building a complete supply chain security model.
Final Thoughts: Invest in a Secure Future
Container security can feel daunting, but it's essential. By starting with secure, hardened base images and following up with signing and policy enforcement, you can dramatically reduce risk without slowing down your teams.
If you're considering a similar journey:
- Start small: Pick a non-critical service.
- Stay focused: Don't try to solve everything at once.
- Work closely with your developers: Their expertise is crucial.
You’d be amazed at how much more secure and productive your organization can become. At Wealthsimple, we believe that investing in security is investing in our clients, our team, and our future.
...
Written by Alexa Yiouroukis, Senior Infrastructure Security Developer
Interested in working at Wealthsimple? Check out the open roles on our team today.