Ahmed: And what I see is the trend is continuing now going beyond just the DevOps to the DevSecOps. How could we do the same for security as well? Because yes, I want to move fast and push things, but I cannot compromise security. Security cannot be an afterthought. It's a prerequisite. And I think this is what Security as Code is all about is, it's enabling this where you can move fast and still have guarantees about your security.
Mo: Welcome to the SaC podcast. Here will discuss different cloud security topics and how to streamline your security from code to cloud. We will talk to experts and leaders about the most pressing challenges and solutions to balance security with operational agility. You will learn something new in each episode to build secure cloud-based systems, using the right combination of technology, people, and culture. Today, we are talking to Ahmed Badran, the Co-founder and CTO of Magalix about Security as Code and why it matters to build the right DevSecOps culture in your team.
Ahmed: I’m currently the CTO and Co-founder at Magalix, which is a startup that is focused on streamlining security from code to cloud. Before that I used to work at… I actually started my career back at Amazon. I worked for AWS for a few years, so I've seen the cloud kind of as it emerged. And I've been around some big companies, some small companies. And just before I joined Magalix I used to work for Blue Origin, which is a company thinking about the actual cloud and going through the real cloud. So, it's kind of a fun journey from a metaphorical cloud and cloud services to an actual company that's trying to launch rockets into space.
But the common theme has been the power and the convenience of cloud and how it's changing the landscape for startups and big companies alike, and really introducing some new models that offer some new really business opportunities and opportunities that's really kind of unheard of before in terms of infrastructure and agility and moving and elasticity and cost-effectiveness. So, it has been a great journey. And here at Magalix we've been focusing on kind of addressing some of those challenges that we've seen ourselves in our own career as we move between companies’ who are trying to adopt the cloud and now the cloud native movement, and really tried to gain some of those benefits from this new paradigm.
Mo: Google returns at the time of this recording around 120,000 results when you search for Security as Code. The term is not new, but we noticed that Security as Code started to gain a lot of momentum since 2019. And by the way, it seems that there is a correlation between Security as Code and Infrastructure as Code. I also noticed a significant increase on Google search after each major data breach event, such as last year's Capital One incident. So, what exactly is Security as Code?
Ahmed: Right. To give it a bit of a context, I mean, when you think about why do companies move to the cloud? One of the biggest benefits to agility, the idea of moving fast, shortening your time to market, being able to be more, adaptive and more agile. And this agility, which is the ability to move fast, deploy things, I mean, we all remember kind of the old way of doing things where the product is launched like once a quarter. That's like the big bang release. And now you hear about places like Amazon and others, even smaller companies who have adopted kind of the cloud models, you know tens of deployments a week, even hundreds of deployments a week when you go to something like Amazon scale. And this agility obviously is a very attractive luring thing for many companies and it's the biggest promise for them and why they want to move to the cloud.
But there is no free lunch. There's no panacea, of course. And with that comes some responsibilities. When you move fast, you deploy things off and it means things get broken often as well. Your stability might suffer, you know, because you are making a lot of changes. So how do you balance the two? How do you move fast yet not break things? And this is really where the rise of this problem comes. And when we think about it as a multiple dimension. It happened in the DevOps world where the developers are moving fast, but the ops team is trying to push back because they are trying to kind of manage all these changes. The process's kind of legacy was manual. People have to review things, make sure or put a barrier between the development team and the operation team. And the DevOps culture eventually evolved to address this problem and make it more streamlined. And what I see is the trend is continuing. But now going beyond just the DevOps to the DevSecOps, how could we do the same for security as well? Because yes, I want to move fast and push things, but I cannot compromise security. Security cannot be an afterthought. It's a prerequisite. And I think this is what Security as Code is all about, is enabling this, what you use can move fast and still have guarantees about your security.
Mo: Infrastructure as Code was introduced around 2006. It was crucial to enable developers to become more involved in defining configurations and for the ops teams to get involved earlier in the development process. The 2020 cloud threats report released by Unit 42, which is by the way, the threat intelligence unit of Palo Alto Networks, identified around 200,000 potential vulnerabilities in Infrastructure as Code templates. So, this Security as Code is about making DevOps more security conscious, or is it about helping security teams adopt the DevOps practices to secure the cloud infrastructure and applications?
Ahmed: Teams, kind of have been operating in silos. So, in the past it was the dev team and the operation team and kind of would bring them together. So, the Devs could do some of the ops work and the ops are involved in that other side. And it's the same thing. And we want the security to be part of this, where we bring some of the DevOps mindset and ideas and agility to the security. And then bringing kind of security into basically the pipeline from the development from code to cloud. So, security is not an afterthought because that has been the challenge. And security also does not come in a way that is still intrusive and manual that stops your flow and your pipeline from progressing, which kind of ended up slowing it down. So, the challenge here and kind of this is what the DevSecOps promises, how can we integrate these things so we break these silos between those teams, so your ops team and development team is more security conscious, and your security team is more integrated into your DevOp processes and tools, so everyone can work together as opposed to work against each other.
Mo: DevOps aims to shorten the systems development life cycle and provide continuous delivery with high quality software. Security as Code aims to balance that agility with security, so that teams keep accelerating their software development life cycle while building air tight secure applications and infrastructure. Securing valuable information isn't just a modern-day challenge. A few days ago, the Egyptian government celebrated the Mummies Parade. They moved 22 mummies to their new home in Cairo. That made us think about how the Pharaoh secured their tombs. We realized that they used sophisticated techniques to keep tomb robbers away. Keep listening to learn a few interesting security tactics that the pharaohs used to protect their valuables. Discovering vulnerabilities is important, but being able to estimate the associated risk to the business is just as important. Codified security goes beyond the identification and provides new levels of controls.
Ahmed: I think it all starts with this idea of Security as Code, codifying, your security, codifying your policies. Whatever you call it, it has to start by codification and we can see the pattern from before. When we look at the Infrastructure as Code that's, how the DevOp culture kind of, and the DevOp movement kind of started. And this is an essential element, if you need to codify your infrastructure so it's reproducible and then it's automatable as a result and you can track it easily and we can all collaborate. And then, because it's codified, you can automate. You can think of unit testing even as a similar idea from just purely from a development perspective, right? You don't want to do manual testing. So, what do you do? Well, you codify your tests so you can run them more often.
Infrastructure is good. Well, you codify it, right? You codify your infrastructure and then you can automate it. So, the same thing with security and those policies, you want to start by codifying those. And once you codify it, you can start automating that process. Now, who does the codification? There's a lot of ways. Even the ops team, they have their own policies and their own rules about reliability. Let's just take examples who are not talking in the abstract. Here's a simple policy you might have from an operation side. Every service must have two replicates, right? For reliability reasons, for full tolerance reasons. That's a rule you can enforce, especially in the cloud native, something like Kubernetes, where everything is declarative or you have an Infrastructure as Code basically built in. This is an easy idea. Now I can write a policy that says the replica count must be two or more. Same thing for security. Pods should not run as root. That's an anti-pattern. That's not a good practice. Pod numbers must not be below a certain nano 1024. Privilege mod must be off. Whatever it may be, let's codify that. And therefore, we can then automate that process. And that's how you streamline this whole DevSecOps.
Mo: In DevOps security must fit into the way engineers think and work. More iterative and incremental and automated in ways that are efficient, repeatable, and easy to use. So where do we ideally apply Security as Code?
Ahmed: So, what do you want to do, obviously is integrate within the development life cycle. You don't want to be intrusive, but you want to be integrated into a process. Now, traditionally, once you codify your policy and you can automate it, obviously it can automate the detection, something in like Kubernetes, and that's something actually we do at Magalix. We can monitor your production system and we can look at what is running and we can basically audit what you have running and look at the violations of those policies. But that tends to be after the fact. That's me telling you, hey, you have something that is really violating and that's certainly better than nothing. But what you want to do and that's where just shift lift movement, is all about, well, if things are codified, like in my Infrastructure as Code is already codified and it's tracked in Git or some source control, why don't I also move my policy checking and validation to the left so I can prevent issues from happening in the first place?
So, integrating, for example, why don't we just, when you commit your changes to your Helm Charts, your Terraforms, why don't we also run the policies, that we just codify those Security as Code policies that we just codified? Why don't we also run that check as early as possible? And the beauty of this, the prevention here, this is the developer or the ops, or whoever is making the change, getting the feedback right away when things are still fresh and they are more able to do the changes quicker to fix the issue, as opposed to detecting something a week or two later in production. And then having to contact switch again, and having to find the right person and put the time and the sprint or in the queue for that thing to be fixed. And this is really the power of automating your Security as Code policies, so you can streamline the whole plot process and create that basically DevSecOps culture.
Mo: A lot of startup ideas come out of a real-life experience. We had a few interesting incidents that made us think using Security as Code at Magalix. I know that at Magalix, we have our own stories building Magalix as a Security as Code platform. Can you share two stories? One of them is how did we reach, you know, how did we get the idea and whether we also try to implement our own system on our own infrastructure?
Ahmed: Yeah. I mean, one of the stories I love to tell, early in the days at Magalix, you know, we had this incident, so my lead SRE was doing some changes and he noticed a workload and we didn't know who the owner was. And there were some problems with it and we needed to fix it. But it was hard to figure out who deployed this workload. So, he reached out to the team in Slack, but it was the end of the day. And it took a while until we figured out who was the owner and what was this new service doing and how we fixed the problem we were facing. And a question came to our mind, what if this was a malicious thing? What if this was a really urgent issue and we needed to get hold of the owner of the service so we figured out...? There's a lot of ideas. We can have a list of who owns what or we can have a checklist and a manual checklist that we approve every change that goes to production to make sure we know, like maybe add a label to every workload. In Kubernetes you can add the labels to the spec, who's the owner.
But then we thought about it. It was like, this is a perfect example. That's a policy, that's like a governance policy. And maybe we should have a rule that says every workload in the spec, we should have a label that says, who's the owner. That could be; the person’s name, or it could be an email alias for the team that owns that service or that system. And we should enforce that. And we certainly did that. So, we used our own policy engine which we were just kind of developing as we were getting this, but it was kind of live and fresh to see us falling into the problem and seeing the kind of pain our customers probably are going through. And we wrote our own policy and we started getting kind of… we detected all the violations that existed already in production, and we tried to fix them. And then the question comes is like, how do you maintain this? So, let's say I fixed it. I fixed all those workloads in production that are violating this policy and we added the owner label to everyone. What about our new service that somebody just developed and they're trying to push? Am I going to find out after the fact? And that's why we started shifting left. So why don't we enforce this policy at GET? At the time of commit, the person making that change gets the feedback immediately. And that's also what we did, and that's where we can start preventing now issues from going out. And now it's more streamlined and we know exactly who owns what, and we know who to call when there is a big problem with any of the services we have.
Mo: According to the 2020 IDC survey, 67% of breaches are due to misconfigurations that could be easily prevented. Security as Code will play a vital role to tighten system security. Other than these large trends, what do I lose as a DevOps engineer or leader if I do not adopt Security as Code?
Ahmed: So, the question is, does that scale? And what happened is, was they moved to the cloud, especially the cloud native. I mean, the basic tendency as you move to cloud native, it's the de-centralization. You move into microservices architecture. You have more workloads running than before. The ownership is even as not clear who owns what. You have multiple teams. Your whole organization is decentralized, which means also the decision-making in your organization is decentralized. What that means is different teams could be establishing their own standards and you lose control very quickly about your infrastructure and any standards you have, you know security reliability, things are not decentralized concepts. These are not like containers you deploy like security equal true.
These things have to be baked into the system as a whole. These are system properties, quality of the whole system. So, you want a way to manage your standards. Something you want to enforce then because policies that are not enforceable or not really policies or standards, and security is not like a nice to have. This is something that could make your business or your company a headline in the news, and we've seen this especially recently when people take security lightly. And it's about the weakest link, right? It's not how much is secured. It's how much is in-secured. And that's where the attackers will go, is after the weakest link. So, you are as secure as your weakest link.
Mo: The old Egyptians adopted what we call in modern security practices as the reduced attack surface. They used to locate their tombs in locations that are hard to predict or find. Even when the robbers find the tomb location, the Egyptian architects created a series of chambers with low value artifacts to make robbers think that they found the buried mummy and its treasures. Keep listening to learn how they apply something similar to IT firewalls to protect The Great Pyramid of Giza.
Ahmed: The thing that we know that is going to continue to happen or take place is, the complexity of the cloud infrastructure is going to keep increasing. And if you even just take a look at the landscape of the cloud technologies, we have servers, we have containers, we have serverless, we have a Platform as a Service, Infrastructure as a Service. We have so many assets and so many applications or so many services that the single company can use even if they're just a small shop of maybe 10 engineers, 15 engineers. And all these services are now expressed as code. And it's not really possible to manage all the non-provisioning aspects of that infrastructure otherwise. So, security needs to be codified, as you mentioned. Also, the other aspects like the operational stuff, and that's what the DevOps guys excelled at and really did a good job throughout the last decade.
They codified everything. They codify how you provision that infrastructure, or how you monitor it, how you upgrade it, how you take it down, how you scale it up and scale it down. Now it's time for the security to do that job. And also, from a business's perspective, businesses are always calling for agility. And some sectors are being disrupted more than others and they want to really innovate quickly. For example, in FinTech, if we take this as an example industry, they are forced to innovate new ways to manage money, new ways for people to access the bank accounts and to transfer money and do financial transactions. So, they have to innovate. And yet this is one of the most regulated industries. So, you want to move fast and you want at the same time to make sure that you're secure because that involves money, other people's money. And if you cannot really, the only solution that we see right now is to codify it in a way or that you integrate the security in the software development lifecycle.
Mo: DevOps, automation, and codification practices reduce the code to cloud lead time by almost 80%. When it comes to Security as Code, there are a few important building blocks that must be there when any team wants to achieve similar improvements to their security practices. What are these building blocks to codify security and build DevSecOps practices in any organization?
Ahmed: So, I believe there's three key tenants here that you need to build your DevSecOps and your Security as Code to do that paradigm shift. One is, let’s call them, you need a set of rich policies, you need integrated workflows and you need insightful analytics and visibility. So, let's think of them one by one. With policies, you need to be able to… it's basically what we said, just codified policies, but they're going to grow over time. So, you need a way to manage them, distribute them, review them. They're just like code, right? They have their own version control. You need to be able to manage the changes to them. And you need to be able to distribute them to whatever clusters you have, how many different cloud providers you have. So, you need a way to manage this across your multi-cloud or hybrid infrastructure.
And then you want to be able to integrate those, the enforcement of those policies throughout your development lifecycle. We talked about the commit time as an example but also sometimes not everything necessarily, especially organization early in their maturity levels, they probably don't have Infrastructure as Code fully for everything. So, you might also want to do prevention and Kubernetes, for example, you want to maybe do prevention at admission control time. So, you want those policies applied at the admission control to prevent bad things from happening. The least is to have obvious detection by just doing audit checks of your infrastructure, your Kubernetes, maybe your cloud native components, to make sure everything is compliant. This is kind of one side of it, but also in those integration, sometimes you want also in the end of that integration is being notified.
You want to be proactive. You want to be notified when things happen so you can maybe take other action. You want to integrate it with your dev tools, you're Slack, your JIRA, your pager duty, what have you. So, this is why integration is very critical and it applies kind of in terms of the development life cycle, and also the development tools you have in your organization.
The last thing I say is this idea of visibility and insights. Like I said, key challenges really in cloud native and cloud agility is this decentralization and you start losing control. And you have this plethora of technologies and deployments and teams, and it's easy to lose control and lose sight of what is happening and lose that visibility. And then now we have the multi-cloud and the hyper models. So, it's becoming more and more difficult. So, it is essential to have this centralized place where you can have visibility across your infrastructure footprint. Be able to see the force really instead of just the individual trees. Be able to slice and dice and see your basically security posture across your cloud deployments, so you are able to take actions. So, you have a prioritized list of where to go, where to improve your posture. So, these things are very critical to enable this DevSecOps culture and the shift left, and this new movement really to streamline your dev, your ops and your security teams to work together to achieve your goal.
If you take a look at any cloud infrastructure, you will realize they consist of three types of assets. There’s compute, storage and network. And when it comes to computation, it could be your VM, it could be your containers, it could be your functions, but let's take one example. In the case of containers, there are lots and lots of possible ways for your containers to go south or to do something that is not really safe for your application infrastructure. The very famous example that we have been using is running your container as root on your VM. That's the default mode for any container to run on a VM. And if you do not properly configure it, and if you're using open-source libraries, then you're basically giving your container full access to everything on that VM. It could be customer data, it could be other application data, and that's dangerous by itself. Writing a policy and enforcing your containers to run as non-root in a way that would just give them what they need, and if they are doing anything malicious will not jeopardize the security of the rest of the application is very critical.
Another example from the storage is the simplest stuff like enabling encryption of your S3 buckets or of your EBS volumes. Those are now dynamic and you create lots of EBS volumes all the time. You cannot really depend only on your infrastructure’s code or your scripts to just enable the encryption by default. You want to have those policies that would always check whether those EBS volumes in your AWS accounts are encrypted by default or not. And for networks, if you're using Kubernetes, it's the same thing. You want to make sure, for example, that the developers are not creating ingress rules and exposing your application to the internet. And this is something, given the complexity of running a containerized application and given the dynamism of the applications right now, more than before the containerized application is really hard to manage those endpoints. So, it's tough to do that. So, it's interesting that you can apply it even at those granular building blocks of your cloud infrastructure.
Mo: Pharaohs protected the great pyramids with mechanisms similar to the modern IT firewalls we use to protect our systems. Archeologists have been exploring an ancient security structure consisting of an intricate system of locks, grooves, and chasse that was installed in The Great Pyramid of Giza over 4,500 years ago. The goal of course, was to protect the mummified body of Pharaoh, Khufu and his treasures from grave robbers. A recent permits scan suggested this unique architecture is used to hide a yet to be discovered king's chamber. What we discovered so far could be rather a decoy to deter grave robbers from looting his treasures or damaging his mummy. Dozens of Security as Code startup companies were founded in the last two years.
Developers and dev ops are hungry to learn and build secure cloud-based applications. On the other hand, according to Palo Alto Networks an enterprise uses on average 130 security solutions. Security as Code will continue to evolve and make the shared responsibility model an everyday reality, and also automate software security in efficient and scalable ways. It will take us some time to be there though. So, where should companies invest when it comes to Security as Code?
Ahmed: I see two key dimensions. I mean, there is the depth and the breadth. In terms of being comprehensive with the type of policies you have, I can envision now, even at the application level, I can see this kind of platforms or queues and databases maybe as they publish the source code and binaries for people to use, they also publish a set of good practices. But instead of writing it as a, here's a document of just best practices on some FAC, why don't they produce it as a Policy as Code as well so you can actually enforce it and automate the process and make sure that your team is following the best practices recommended by that vendor. This I think, would be a great development that moves the whole industry forward because this is hard, right?
People have to read a document to kind of know what are the best practices, but then how do you enforce it? You can't just keep repeating yourself in meetings or doing code reviews. And then the depth. So, you know, going deeper in the stack from the infrastructure all the way up to the applications like we're talking about. Because again, security compliance is all about your weakest link and you want to make sure you have the comprehension, but also you want to have the depth in the whole stack from your infrastructure, from your platform level, your application and services, and applying all those policies at all those levels is really critical.
And then the other aspect is definitely making it seamless, making it seamlessly integrated because otherwise anything that's going to get into the way of development and agility and speed, people will always find work arounds and that's how you break everything you've been doing. You're basically creating back doors to your security posture if you don't make those integration seamless into kind of the existing tools that the dev teams are using. This is how you really, I think, create more success and help the whole industry really move forward with Security as Code.
Mo: 67% of breaches in cloud infrastructure took place due to misconfigurations. These are either due complex core relations that engineers were not able to capture or sometimes because security is taking the backseat in a lot of DevOps cultures. What's your recommendations for engineers and also for technical leadership to get ready for that next wave of innovation, especially that sometimes when I talk to engineers, they would say we were just done understanding Infrastructure as Code. And there are new services everyday coming out. What should they be looking for to keep up with Security as Code?
Ahmed: Right. This is a fast-moving target and that's why I think it's a collaborative initiative really where we need to all collaborate and build this kind of Security as Code in every application and in kind of your major vendors. And as a developer, you need to be thinking about these early on. It's better to prevent things and it's better to think about these aspects early and introduce them. It's just like doing unit testing and linting of your code early. It keeps the hygiene better. It keeps this, you know, the code smells out of your code and out of your infrastructure. Helps improve the quality of the overall system and security must be treated as such. It's something that has to be addressed early. It cannot be an afterthought. And as an industry, we need to kind of start adopting these things early so we make it easy for the newcomers and people who need to manage a lot of different things and end up having to deal with a lot of different technologies and a lot of different standards. We need to come together to move forward.
Mo: Thanks for listening to this episode of The SaC. Please pass it along to someone you believe can apply what we discussed here with their team or company. For updates on upcoming episodes, or to find out how to be my next guest, please visit Magalix.com/thesac.