Tony Chong: I think Policy as Code solves a lot of the problems. But when it comes to the big picture, I think what Policy as Code is implementing like you said build time, runtime checks, using Policy as Code, I think it takes care of a lot of the DevOps responsibilities. I'm not talking about the team, but I'm just talking about the processes. But I think in regard to the big picture, what it really does, it reinforces this message of security.
Mo: Welcome to the SaC podcast. Here we discuss different topics to streamline security from code to cloud. We will talk to experts and leaders in cloud security about the most pressing challenges to balance security and agility. You will get tangible advice to secure your infrastructure using the right combination of technology, people, and culture. I'm your host, Mo joined today by Tony Chong, a solution architect at Magalix to talk about what is Policy as Code or codify policies as a way to scale security and compliance in cloud infrastructure.
Tony Chong: I started out many, many years ago, over two decades doing IT system administration, supporting NT4, Active Directory Exchange, corporate IT, and then over the years, I transferred to more enterprise computing, application support, production Ops before there was like a DevOps team. And then as the cloud became the predominant way, at least in my role that people were provisioning infrastructure, I moved to the cloud, got into DevOps. I've been a data engineer, I've been a CTO, a co-founder, worn many different hats. And now, I am a solution architect at Magalix helping people understand their compliance needs and helping them implement a Policy as Code solution.
Mo: This podcast is sponsored by Magalix. Magalix streamlines security from code to cloud using codified policies, tight integration with your software development lifecycle and with the right insights. For more details, visit our website magalix.com. In my previous episode with Ahmed Badran, we discussed what is Security-as-Code. Security as Code is about codifying policies and standards and automating their enforcement. Since we're focusing today on codified policies, what does Policy as Code mean, in simple terms?
Tony Chong: To me, having codified policies really accomplishes a couple of things? I think the first thing that comes to mind is security. I think if we're in a world where your infrastructure is codified, obviously your microservices are codified, I think when everything is orchestrated, you want to make sure that maybe certain flags are enabled, certain security flags are enabled to configurations. And so, to me, one aspect of Policy as Code is ensuring that not only is your code, you have the right configurations in place to make sure that you're secure, but also applying those same policies to a running environment to make sure you're constantly staying secure, but you're doing it all through code. Instead of using things like a UI, or a console, or alerting, you can codify the things which you want to secure and then enforce those against either your code itself or your environment.
Mo: Most of the major cloud providers offer dozens of services and products. AWS alone has more than 200 products and services at the time of this episode. As a matter of fact, a company uses on average 20 to 30 cloud services and products. With all the possible ways things can go wrong with these services, the operational and security complexity is exponentially increasing. So, is there a correlation between the increased complexity of cloud infrastructure and the rising popularity of codified policies?
Tony Chong: Sure. This is a bit anecdotal, but from my own experience, I think the enterprise, well, I think cutting edge folks, you know, I've been in startups, we leverage DevOps, we've been leveraging DevOps as a culture for a long time. It's been proven that we're able to increase agility, flexibility. You know, couple that with agile methodology back in maybe a decade ago, and you have this nice, harmonious way of developing code and integrating systems and software. And I think now as the enterprise is starting to adopt more of these DevOps principles, you know, they have a need. Big enterprises have probably much more regulation that they have to deal with on top of other things.
And so, I think as the enterprise starts adopting these DevOps principles like cloud native, Kubernetes, things of this nature, security has always been something that they need to take care of, especially in areas like FinTech. Even social media, we need to secure the data, right? And so, as enterprises start adopting this DevOps methodology, they have to think about security. And I think because of recent, I guess, breakthroughs in Policy as Code, a lot of these initiatives are getting open sourced to the cloud native, it's becoming much more prevalent, there's much more eyes on it. And much like other open-source projects, the more eyes are on it, the more people that adopt it, the faster it becomes better. And I think it's been a couple of years since OPA and these other solutions have been out, and I think now is just kind of the right timing for those to gain traction.
Mo: Enforcing policies and standards is not a new concept. It has been there for decades. Many existing policy or ACL systems do not practice Policy as Code however. Many policies are set by clicking an agile, which is not easily repeatable nor versionable. They usually don't provide any system for testing policies other than testing an action that would violate the policy. This makes it actually difficult for automated testing and the policy language itself varies by product. It seems that the new concept of codifying policies is a natural evolution of Infrastructure as Code. So, what problems does Policy as Code solve for engineering teams?
Tony Chong: To me, it solves a couple of problems. I think, first, when you apply Policy as Code to a running environment, you have an idea in real time what is compliance and what is not? What I mean by compliance, I use these terms interchangeably, it's probably wrong, but just for the sake of the ease of conversation, when I talk about compliance, governance, security in this context, I'm really talking about standards in your organization, simply put. And so, understanding what is in compliant with your standards and what is not is the first problem that this solves. I mean, nobody is going to check every single pod to make sure every single configuration is aligned. It doesn't make sense, it doesn't scale, it's probably impossible to catch up and maintain that if you have a very, very large cloud native deployment. So first off, knowing what's going on.
You know, you also keep a running tab of what's compliance from a holistic level. Instead of just looking at one deployment, you could look at your fleet of clusters across your multi-cloud environment. So having the same policy scale across a multi-cloud even on prem, allows you to have this uniformity regardless of what's under the hood. So, in terms of cloud native, I think it is one of the beauties of Kubernetes, it's kind of this lift and shift idea that if I deploy Kubernetes anywhere, I can deploy the same artefacts, the same architecture, and whatnot. And policy really helps define uniformity across all of my cloud systems.
And I think thirdly, which is not just in the running environment, but more importantly, as organizations adopt this more SecDevOps or DevSecOps approach, you want to start incorporating the security left. And what I mean by this is just like how developers are using CI/CD today for unit testing to ensure they're not introducing any bugs into the system. You could do the same thing with policies. Just to take that a step deeper, let's say, for example, we're in an organization where the developers have much more autonomy over their micro-services. Meaning regarding specific Kubernetes, let's say the developer creates a deployment YAML file, or service file, maybe an Ingress YAML, and within those they configure everything. And so, unbeknownst to them, they have the ability to deploy these things but they might be deploying configurations that are very lax in security, maybe the containers are running as root or some configuration that's not adhering to the org standards. You don't want to introduce those into your environment. The people that are maintaining this are going to be chasing your tail all day trying to chase people down and get this fixed. But instead, what they do is they codify the policies, incorporate that into the CI/CD jobs and "shift left". So, when the developer is building, they're deploying YAML or their Helm chart or whatnot, and if they're doing something there that doesn't adhere to the organization, they can flag it and the developer can then remedy that even before the Ops person, whoever is in charge of the environment has to even know about it. So really trying to catch these violations as early in the SDLC as possible and then because of DevOps now reusing their CI/CD jobs to incorporate this.
Mo: We see companies approaching Policy as Code in different ways. Some would see it as a new way for security engineers to secure infrastructure without slowing down operations, while others see it as a tool to help DevOps secure their infrastructure and their applications without the need for manual security review. So, who Policy as Code is meant for?
Tony Chong: That's a great question. I think, first and foremost, I think that maybe it's going to vary between organization to organization. In my opinion, this is probably a DevOps responsibility. I think the reason is because you have the software development team that's focusing on solving the business challenges or the business logic. You're the security folks who understand security better than anybody, right? And so, they understand the different algorithms, when it comes to security, the different ways that the hacker is trying to compromise the system. They understand this much more. And then kind of in the middle you have a DevOps engineer who understands what the software engineer is working on, how that gets deployed, and how to manage this application. And so to me, because they kind of, you know, the DevOps engineer sits under the hood and because a security engineer is not necessarily a Kubernetes administrator, neither is a software engineer, and I think because especially in cloud native and even infrastructure, because the DevOps person is most likely developing that Infrastructure as Code, managing the infrastructure, managing Kubernetes, provisioning Kubernetes, it makes sense that, at least in this use case they have their hands in the CI/CD as well. So, it makes sense that a DevOps engineer is probably going to manage the policies as well now. Mind you, who's going to write them? Again, that's organization’s specific, excuse me. But in my personal experience, that to me most likely will fall on the on the shoulders of the DevOps since because they're managing the infrastructure anyway and because they understand Kubernetes the most, most likely it just makes sense that they would be developing or at least applying those policies because they understand the impact of what those policies can or cannot do.
Mo: If DevOps should own the codification of policies, we should expect a lot of automation and innovative ways of using it. But first, what are the main characteristics of policy codification programming language? Any policy language needs to have three main qualities. First, they need to be non-programmer friendly. Non-programmers need the ability to enforce certain rules within a system without complex structures. Second, they also need to be programmer friendly. At the same time, the language needs to support programmer friendly constructs such as conditionals, loops, and functions for complex policies that the programmer might be writing. And finally, they need of course to be safe. Most policy languages are used in security sensitive environments. They must not be able to crash the whole system or access the system resources without an explicit approval. I asked Tony to share one to two examples to clarify further why the DevOps engineers are the best to codify policies.
Tony Chong: Well, I think TLS [inaudible 00:14:41] has been deemed maybe not as secure. But maybe a DevOps engineer doesn't quite know that. And so, if a security engineer says, "Look, these protocols have been defined as insecure, you need to use the latest." And so, if they could communicate that to the DevOps team, when anything gets provisioned that tries to use that insecure protocol, the DevOps team can then say, "Hey, the security team says this is insecure, I trust them. Let me write a policy that prevents this protocol for being used within the domain of my control, which is probably Kubernetes, and my cloud or my on prem footprint." And then if I take this codified policy and apply that back to my CI/CD jobs, the developer, mind you, they may not necessarily care about insecure protocols, but when they try to build their artefacts, they're going to get some feedback that says, "You're trying to use this insecure protocol part of the configuration. You can't use it for XYZ reasons. Contact me for more information. And I think in a real-world case, this is how I see the three different stakeholders of deploying modern day secure DevOps applications like how this will work. Now, mind you, every organization is different but I think this is where everyone's expertise really plays a role in this kind of trinity of teams that really can strive for deploying and maintaining secure applications. Because you have three experts in three different domains that need to come together to form a secure application and secure infrastructure.
Mo: Policy codification should bring Ops, security and developers together to secure their infrastructure and move fast. Security teams want to protect the current value of company products and data. Developers want to increase the value of these products however, which requires changes and disruptions of the status quo. Operations teams on the other hand want to keep the system stable and available all the time for their customers. Friction is highly possible given these conflicting changes. So, how can codified policies help these teams work closer together?
Tony Chong: I think Policy as Code solves a lot of the problems. But when it comes to the big picture, I think what Policy as Code is implementing like you said, commit time, build time, runtime checks, using Policy as Code, I think it takes care of a lot of the DevOps responsibilities. I'm not talking about the team but I'm just talking about the processes. But I think in regards to the big picture, what it really does, it reinforces this message of security. And the reason why I say that is because imagine a developer is building a new micro-service, they build the job, they run a build job for the first time and one of their proposed configurations is in violation. And again, going back to the previous example, maybe security isn't necessarily on top of their mind. But what this does is it actually forces in a way the developer to start considering, "Hey, I have to build a secure configuration in order for my build jobs to pass in order for me to bundle this package, in this case, build my container image and be able to deploy it.
I think in terms of the big picture, Policy as Code really reinforces the message that security is really everyone's in terms of the development, the tech side of things, really everyone's responsibility. But when you think about it piece by piece, it plays a huge role in the SDLC in pretty much every phase of the software development lifecycle. Because you can check your policies at the commit time using GET pre-commit hooks. If you have a CI/CD system, you can check it there. So, every time you open a pull request, you can have a trigger there if you wish. And then also, you could do it at deploy time in Kubernetes. You can use a mission controller so if someone wants to bypass all of that or you don't have it, and someone wants to deploy a policy violating configuration, it gets blocked there and then for those entities that are already deployed, you can catch it at runtime. So, it plays a huge role in the SDLC but it also reinforces the message that security is something that everyone needs to pay attention to.
Mo: Policy as Code decouples decision logic from business logic and services. And we can even go as far as to make the policy decisions based on the application or the environment context. I asked Tony about some of his personal experiences that made him realize the value of codified policies with different applications in cloud environments.
Tony Chong: Really in my own experience, it's really about the scale. I'll tell you a quick story. The last couple, I would say, not as a solution architect but in previous roles, I've worked in smaller shops and startups. And so, what I've noticed is that security isn't really a consideration, people are trying to move fast. And so, when I tried to play with Policy as Code initially, what I saw was it was kind of an eye opener. Because as the developers are building micro-services as quickly as possible, the minute that I turned on the Policy as Code within my production Kubernetes cluster, I was able to see a lot of violations. Developers are running containers as roots, that's a very popular one. A lot of times I think that's what triggers people to look at the Policy as Code solution because every time a new micro-service is deployed or actually anytime there's a change, that's something that I need to check. And so, what Policy as Code allows me to do, it allows me to scale those checks.
And what's really eye opening for me is the more policies that I apply, the more violations I'm discovering. So it's really eye opening just how insecure default Kubernetes really is. And so just from those experiences Policy as Code really, I guess the word I'm looking for, it kind of wakes people up in terms of security. I can talk for days about security initiatives in different places. But it really wakes people up because there's proof. Policy as Code kicks in, they get violated and you can show people look, this is exactly what we're building, this is why it's violated, this is why it's a problem. We need to fix it. And if you catch that early on especially during your PLC phases in your dev environments, or in the staging environment, you can really, really be as proactive as possible and not introduce these exploits or these vulnerabilities or violations in prod. And if you do, I'm going to be spending all day trying to fix those while the developer continues to make changes, which they should. But I just don't want to spend all day chasing the same violations over and over again for either edited artefacts or new ones. So Policy as Code can really help scale that challenge of trying to maintain compliance, well actually, see that you're compliant and maintain it.
Mo: A simple Google search for Policy as Code gives you dozens of tools and startups in this area. Technology is important, of course, but it should not be your starting point when you think about the adoption of codified policies. There are four key pillars that must be considered when looking to shift to the DevOps culture and use Policy as Code in your organization. First, people. Restructuring DevOps and security teams to establish efficient cooperation between them as well as offering good quality and targeted training will ensure that security becomes a frame of mind rather than a hindrance. Second, processes. Aim to further improve collaboration between people to achieve more secure development processes as a whole. Third, technologies. When the right technology is in place, you will enable the employees to execute these processes as well as automate them. Finally, governance, which measures the performance of other elements that can point out where more focus is needed to ensure all parts of your culture form together. I wanted to have Tony's inputs on what is the ideal culture that would make codified policies work at their best.
Tony Chong: The best way to get started, I would say, do it. Build a PLC and try to get a single policy running in a non-production environment and see what happens. I think there's no substitution for experience, I think, in this case and every environment is different. But I think getting familiar with the terminology and just getting one policy applied to your Kubernetes cluster is a great way to get started because you're going to understand what's at stake in terms of how to install it, how to configure it, how to troubleshoot it, and then after you have the system in place, then you can start actually working on the policy. So, understanding the whole Policy as Code ecosystem is great. And I think there's no substitution for experience so try it. Do it in a lab whether using mini cube kind or whatnot. Just get it going.
That's on the technical side but I think kind of on the non-technical side, company adoption, I think it helps having it and coming in from two different angles. Obviously, if it comes from the top down and your industry is regulated, excuse me, the business you're in is regulated by industry, coming from the top down or from the outside in, that's a driver, no doubt. You have to be compliant with this regulation or else you're going to get fined or you can't do business or you can't even really launch at that matter if you're a startup. I think from that aspect, that's a driver. But I think from another angle there's kind of this inside out pathway that I think starts from your champions.
I feel like in every organization there is somebody that always seems to be on the forefront of this cutting-edge stuff. And I mean stuff because it could be all types of things but let's say security there's probably your DevOps engineer or maybe your security engineer. There's somebody in an organization that knows about security or cognitive security who's an advocate and so leveraging their enthusiasm, their knowledge, their... what's the word I'm looking for? Their constant feedback of that, "Hey, we have to be secure, we have to be secure." I think putting those folks in a team coupled with maybe security engineer if you have it or software engineer, maybe even a lead or a senior person. Putting them all together in a small mini project to get it going I think is great. I know those as Tiger teams. I stole that term from NASA, which is getting specialists from different domains and putting them together to PLC it out. I think letting people do their job, what they love, what they're most knowledgeable about, I think is a great way to get the ball rolling. I've done this in many, many other organizations for different projects. I remember doing this over 10 years ago when we were rolling out agile and sprints with Scrum in particular. I remember we started with one new project, one new initiative, and we tried to figure it out, work all the kinks, built the process, and then deployed it out to the rest of the organization. So, I think in terms of adoption it really helps. Again, experience is no substitute so having a small team if you can build out the experience, understand what it's like to implement Policy as Code, because there's going to be feedback for the developer. You just don't want to turn on a policy [inaudible 00:29:40] the developer and say, "Hey, what's this rule? Why is this failing? What is this?" You don't want to do that. You want to educate your team and the best way to do that is to get the right folks involved at the right time. Have them build it out, have them experience it, document it the best way possible, and make adjustments. And then when you're ready to go release that out to the wider organization.
Mo: Thanks for listening to this episode of the SaC. Mo was your host today with Tony Chong. We discussed what is Policy as Code and how you can make best use of it to bring security operations and developers together. Please pass it along to someone you believe can apply some of what we discussed here. For updates on upcoming episodes or to find out how to be our next guest please visit magalix.com/thesac.