<img src="https://ws.zoominfo.com/pixel/JHVDdRXH2uangmUMQBZd" width="1" height="1" style="display: none;">

Magalix Introduces the Cloud-Native Application Policy Pack

Exit icon Learn More

A Tale of Two Woes: Security and DevOps - The SaC Podcast

Policy as Code Security as Code The SaC PodCast
A Tale of Two Woes: Security and DevOps - The SaC Podcast
Policy as Code Security as Code The SaC PodCast

Can Security and DevOps Work Together? 

“Organizational learning and change are key to allowing DevOps to flourish. In other words, people-related factors tend to be the greatest challenges — not technology,” -  George Spafford, Senior Director Analyst, Gartner.  

In recent years, the risk landscape has changed drastically. Cyberattacks are rising in frequency, complexity, and impact as attackers take advantage of security risks to infiltrate enterprise infrastructure. As such, more organizations are amending their business priorities to include cybersecurity strategies. 

That pushed companies to think differently about the role of security and their engagement with the rest of the organization. Some have specialized ops and security teams. Others have roles combined in the same team. The number one challenge is helping teams working together from different disciplines.

In this episode of the SaC, Mo, Magalix CEO and co-founder, talks to Peter Samaan, an Infrastructure Engineering Manager at Remitly about building and maintaining a secure infrastructure, in the DevOps world. 

We tackle different topics, such as: 

  • Should DevOps be security-aware? 
  • How people-related factors pose the greatest challenges to DevOps initiatives
  • Peter’s experience in leading the infrastructure Operations team at Remitly
  • What should be the security team's priority and focus to enable a healthy DevSecOps culture? 

About our Guest:

Peter is a DevOps leader with 10+ years of experience across the IT service lifecycle and in various companies and projects around the world. He specializes in Web Application Infrastructure and enjoys working in the space between software development and SRE operations.

Listen to More Podcasts From the SaC

Full Podcast Transcription

 

Peter: ...so many to take people on the journey with us and knowledge is to me still the biggest gap. Security is still a very specialized knowledge. Security teams are the holders of that knowledge and that developer teams are stakeholders for those teams. How do we make security more amenable to this iterative improvement approach rather than what security looks like to me most of the time, which is, here's a bunch of, like here's a kill list?

Mo: Welcome to the SaC podcast. Here we discuss different topics to streamline security from code to cloud. We will talk to experts and leaders in cloud security about the most pressing challenges to balance security and agility. You will get the tangible advice to secure your infrastructure using the right combination of technology, people, and culture. I'm your host Mo and today we're talking to Peter Samaan, an infrastructure engineering manager at Remitly. A Seattle based FinTech unicorn. Today we're going to talk about Peter's experience building and maintaining a secure infrastructure while applying the best DevOps practices and agility.

Peter: I'm responsible for the infrastructure engineering team, which is a product development team for our own consuming purposes within Remitly. I'll talk a little bit more about how it's different from other product development teams a little bit later, because I think that's very relevant to the conversation about security as code and DevOps in general as a direction for teams to take to manage their infrastructure as well as streamline development. I've also managed security for some time at Remitly, so I have some experience with how security thinks about approaching these types of problems. Most recently, I've started getting involved with IT as well. It's an interesting space how these things overlap.

Before Remitly, I've worked in a number of other places. I've worked in HP Enterprise Services, I've worked in really, really small, profitable companies and much larger startups. I've worked in multiple countries, I've worked in Egypt, in the United Arab Emirates, I've worked in the UK, I've worked in projects literally all over the world before I came to the US. And I think I've learned something a little bit from each of them on the journey here. DevOps itself, I kind of got introduced to the term and the practice about seven or eight years ago. At the time, I was what you would call a deployment engineer or production operations engineer, so much more focused on the ops side of things. But I was very interested in software development in general, it's kind of what my formal training is in. And I rediscovered a love of coding a little late in my career. And then this DevOps thing was coming up and it seemed that the distance between development and technical operations was closing and I found my opportunity in that closing distance to act as a bridge in a way through process and tooling between the two previously distinct practices. 

Mo: This podcast is sponsored by Magalix. Magalix streamlines security from code to cloud using codified policies, tight integration with your software development lifecycle, and with the right insights. For more details, visit our website magalix.com. According to the Gartner 2018 DevOps report, organizational learning and change are key to allowing DevOps to flourish. In other words, people related factors tend to be the greatest challenges, not the technology. That's why Gartner in the same report predicts that through 2022, 75% of DevOps initiatives will fail to meet the expectations due to issues around organizational learning and change. DevOps engineers and leaders face a lot of challenges as they push for more automation and tighter operations. Remitly is one of the advanced cloud-based companies, so I asked Peter about his first challenge when he joined Remitly.

Peter: It's a really interesting question. Let me see if I can come up with a thoughtful answer. I think one thing that will never stop surprising me whenever I enter a new endeavor or get introduced to a new company, is how there isn't a linear path of maturity. Meaning you get in and you start talking to your stakeholders, to developers, engineering managers, to like technical operations across the company, and you start realizing, hey, some things here are like, you guys are way ahead of the competition on specific things or in specific areas. But it's not a linear progression. As in, that doesn't necessarily mean that you've solved other problems that some of the competition have or have actually solved. That some of the other companies I've worked at have solved. 

For example, ironically, I came in and I saw that one metric that everybody likes to use to describe the health of a DevOps organization is how many deployments you're doing per day. Actually, Remitly was doing dozens of deployments a day across the stack. That was really impressive to me, just out of the gate. Wow, we seem to have pretty quick iteration speed. We're capable of rolling out low risk deployments multiple times per day, like up to sometimes 30 or 40 or 50 times per day. And everybody has visibility of what those changes are. They could be traced through the life cycle, they could be traced through the development flow from products running in production all the way back to Git commit. And that's amazing. On the other hand, you immediately start running into some things like, well, the local development experience is a little bit like lacking in certain areas, some of the technologies we're using are a little out of date The product is well architected but teams are autonomous in many ways and so working together and collaborating on things that span multiple domains isn't that straightforward and there's a bit of a challenge in collaboration there. 

So, teams are very fast when they're working on their own services. When you're working across different services, the story changes a little bit. It depends how long have you been here and how much of the institutional knowledge have you already absorbed? There's a lot of non-obvious things that I also came to realize I think, like maybe a year later that illuminated some of the earlier observations. I guess my point is, I don't think there's like a clear track of maturity that every company ends up going through. And so, you kind of need to fit the solutions a little bit to where people are in their journey with that in mind. That there's like parallel paths to maturity.

Mo: In recent years, the risk landscape has changed drastically. Cyber-attacks are rising in frequency, complexity, and impact, as attackers take advantage of security risks to infiltrate enterprise infrastructure. As such, more organizations are amending their business priorities to include cybersecurity strategies that push companies to think differently about the role of security and their engagement with the rest of the organization. Some have specialized ops and security teams, others have roles combined in the same team. Number one challenge though is helping teams working together from different disciplines. Remitly wanted to find effective ways to maintain compliance. They wanted the security and operations teams to work closer all the time. So, they pushed them to have both teams to work under the same leadership. 

Peter: Remitly has a security team that is its own staff team. We also have an infrastructure team. When I came in, there was a single developer on that team. We're at five now but it's still a fairly, I think, lean team by most standards. Aside from those two teams, we have autonomous developer teams. These are spread across different business units. For example, we might have a pricing team, we might have a treasury engineering team, we might have a marketing team. These are engineering teams that are paired with their business counterparts in order to build technology solutions for business problems each in their own area. In a world like that, it's a little easy to end up with a very difficult problem coming out of time, for example. 

Remitly is a FinTech company and so we have an important compliance obligation as well as what I would call a higher bar for security than probably a lot of tech companies. And so, there's a little bit of a challenge here, like how do you square the circle between, hey, every team needs to be autonomous because we don't want to slow them down, we don't want to gate keep things. We don't want to gate keep production specifically through people. We're comfortable gatekeeping production via tooling, that's fine because that doesn't really take away from autonomy of the teams and we need a way to effectively show compliance, I want to say, even though teams are more or less like on their own in their development journeys. So, it's really tricky. 

And I think one of the really interesting ways that Remitly has tackled this is, very early in its lifetime, Remitly built a product internally. We call it Forge. That's how you deploy to production. It's not really unique. I think many other companies have something of this sort. I hear that Amazon has a system that does something like that, Netflix also does. Basically, you cannot deploy an application to production without going through the system. And this is distinct from our CI/CD pipelines. This is basically a highly opinionated definition of what constitutes a production application. And that allows us to insert all sorts of things at the deployment stage for any given application. And then it allows us to turn around and take evidence produced by that system, show it to auditors, show it to internal teams who want to understand more about how security works at Remitly, for example. So, we basically solved the DevOps problem by tooling. That said, I mean, it's a really complex problem and there are other facets of it still to cover and trying to solve for. I'm not pitching tooling as the silver bullets here.

Mo: The top challenging cloud security according to many reports is the lack of security skill set. It is hard to wait for the market to balance itself with the traditional ways to fill in the security skills gap. We need to secure our systems now and very quickly. That's why I always ask these questions: should DevOps be security aware to fill in this gap or should we bring more automations to security and adopt DevOps methods to democratize security? In all cases, healthy collaboration between both roles is necessary. Otherwise, there are a lot of possible overlaps and conflicts. That's why I was wondering how Remitly managed to let security and DevOps work together. 

Peter: The infrastructure team at Remitly, which I think is the closest thing to what in other companies will be called a DevOps team, was actually very close to security from the get go. They started out on the same team, which I think was one of the best things that could have happened in how security teams get started. Because we ended up for a long time working together on the same project, just bringing different lenses and different skill sets to the same problems. So, CI/CD was not seen as a DevOps responsibility, it was seen as a shared responsibility between security and the infrastructure team. And for sure, also product development teams. But for the large part, we tried to fit the use cases that developer teams came to us with and factoring their feedback. But largely, we saw it as our problem to solve. Like, hey, we need to figure out how to solve this. And security, because the team considered this their project, there was no, to be honest, clear distinction between, hey, this is an infra concern and this is security concern. A lot of the time they got intermingled. 

On a relatively small scale, I think that's kind of where we were about three years ago. This was a blessing. At larger scales, that doesn't work very well, I don't think and you need to take a more structured approach to program managing something like that. But more concretely, to give a couple of examples, we'd approach a certain problem, let's say, hey vulnerability management and patch management, we need to be able to roll out patches to all of our infrastructure in a short amount of time, and I have an existing system for that. And then security infrastructure would do a design session together, and they'd look at the system and they'd say, well, you know, it takes us days to roll out a new image for people to use. And then it takes us weeks to get everybody to adopt that image and update their infrastructure. How do we solve this? Like this isn't very good from a responsiveness standpoint and it's also taking a lot of time and effort from what is essentially a small team, which again, I think it's one of the interesting decisions on how we structure teams. We deliberately tried to run lean teams. That created resource constraints. And when you have resource constraints, you're forced to come up with creative solutions and adopt technologies that would fit these resource constraints or try to solve them. 

I think that was also another early blessing of the team. I've seen many teams who just because of the sheer size of the team, you can kind of swallow a lot of pain in the process just by saying, well, it's fine. We'll just throw more people at it. You're forced to come up with more creative solutions when you're running against those resource constraints I was talking about. I guess what I'm trying to say is, security and infrastructure being treated as more or less two parts of the same team for a significant amount of time really helped in bringing a security first mindset, to continuous integration and deployment, to infrastructure management, and to the whole developer experience question as well of how do we make the developer experience less friction, or ensure that the developer experience has less friction without sacrificing security. Because security is kind of embedded into infrastructure in a way.

Mo: To successfully adopt DevSecOps methodologies, organizations will need to create a culture that fosters cross team collaboration and innovation. As with new processes, organizations will need to introduce outcomes and matrices that are focused on security to ensure all teams are aligned. These could be number one, security flow, which is improving the velocity to fix problems faster. The second is resiliency, which is about improving your capacity to respond and recover to security incidents. And finally, risk reduction, which focuses on reducing the risk that matters at the source. So, I asked Peter's point of view about what should security teams focus on to enable healthy DevSecOps culture? And when should they get involved with the DevOps teams to secure the cloud infrastructure?

Peter: These are great questions. I'll try to tackle the first one first, which is, if I remember correctly, you're asking how did the two teams figure out what to engage with or where to start engaging? Like focus on what should security focus on or where should they get involved when DevOps is concerned? A little bit maybe driven by pain points that security had, as well as compliance requirements. It kind of forced us to focus on Forge, the tool that I mentioned we built internally. Forge was really the site of many security conversations, improvements, requirements. It's the gate to production and so automatically security gave it a lot of attention. So that was helpful. And then earlier on in the SDLC, like Jenkins or CI/CD tooling in general also automatically came into play like, hey, this is an area security needs to focus on. 

Besides that, there was also the edge, we call it the edge, so all services that are outside on the perimeter of our infrastructure. So, proxies, routers, devices or services, software services that are exposed to the public. Because of that, it was also a site of security deployments. And the way security worked, to be honest, was a lot similar to how infrastructure worked in that security-built services. Some of them were open-source tools that were integrated into the flow, some of them were tools created from scratch built by engineers, software engineers on the security team to secure the perimeter, to secure software development practices into Forge and into our CI/CD pipelines. I'm not able to go too far into the details, but broadly speaking, these are the three areas that I think security wanted to focus on the most. So, the edge, the gate to production, so the deployment tool, and then earlier on the build and testing tooling.

Mo: Got it. And now when it comes to infrastructure, I'll tell you a couple of interesting conversations that I had at AWS when I was working there. Now, every time we deploy to production, the security team needed to review basically what was going there. And there are two things that they can review. There's the application set that is going out and there is the culture of your general infrastructure. Of course, the application part makes sense and they work with the developers and there was some sort of an overlap because they were also checking the configuration of that application. Then when it comes to infrastructure, that makes it really interesting because they did not understand the terraform files that we were using. But they can understand the final outcome. And the problem here is, they do not want to wait for the after the fact. They do not want to wait for you to run those terraform files and provision the infrastructure and oops, there's something really bad happening there in terms of security. That definitely creates a contention. That makes it really hard because you want to move fast and they do not want to wait until after the fact. Did you face any situation similar to that? And how were you able to bridge the gap between the knowledge that they have in yours and the speed between your team and the security team? 

Peter: This is I think kind of the core of the whole DevOps culture, which is treating infrastructure as well security problems as software problems. And what you described to me sounds a lot like the principle of failing fast or the need to fail fast. Because development time is expensive, and so the faster you're able to fail in software, the faster you'll be able to move overall. Because you want to shorten the cycle between failure and fix, failure and fix. So that means you want to fail as quickly as possible. And essentially to me it's very typical of any security team or infrastructure team, which is developers build something and then they submit it for review. And then we come back and we say, "Well, you know, that's not going to work for this reason, or whatever." It could be because we ran a bunch of tests and it failed, it could be because an expert looked at it and said, "Ah, no, that's not going to fit in with how we do things." This is a slow and relatively expensive process.

And so, the right solution here in my mind, which isn't always easy or in some cases feasible, but it's to try and fail as quickly as possible, which means frontload the tests that are most likely to fail. If it's something like open ports are going to be on automatic red flag when you get to a security review, then makes sure early on in the process to scan for or detect or just eliminate the possibility entirely of having open ports in your service, unless they're within a very strictly defined interface that's been pre-approved by security and everybody's comfortable with it and there's a WAF in front of it and like all sorts of other security measures have been taken. So first of all, yes, they fail fast as a concept in software development, bringing that into the security review process and the infrastructure review process of that applies. 

Two, I think it's reducing surface area. This is the harder one I feel for companies because it requires upfront investment. But basically, think of the number of things that developers can change like the surface area, the addressable surface area when they're releasing a new service or a new feature, and then deliberately reduce that as much as possible. Make it so that there's a relatively narrow road that a service needs to take in order to go to production, that there's as little variability as possible in that process. Because that reduces the number of things that can change, number of things that require review, number of things that later on can be exploited, or number of things that can be buggy. I had a security engineer once make the point that like, every bug is a potential security vulnerability. And so, actually, we want to reduce security vulnerabilities by reducing the possibility of bugs too. That means, yeah, reduce the number of things that can change. This is easier said than done, but I find it to be good directional advice.

Mo: Throughout the last decade, developers were able to adopt a lot of the operational practices. And that's why we had the DevOps role. As the importance of security is on the rise, there are many possible ways to bring everyone up to speed in terms of securing their applications and infrastructure. The questions that I had in mind are, do security teams need to catch up with the DevOps methodologies to scale quickly and efficiently? What kind of security automation is needed at this point? And why is security still considered a niche area that developers find it hard to understand?

Peter: Honestly, I still see the gap primarily to be knowledge-based. I think as an industry generally, we've managed to find multiple points in which to insert security tooling. We're able to insert at the runtime, we're able to insert at deploy time, build time. I haven't seen many things that try to insert themselves up like the development time tool like linters and so on. But I think that's a very, very interesting area to invest in, again, back to the whole fail fast thing. Linters are better than deployment gates, for example. Whatever you can catch earlier on gives your team overall more velocity. So that's a good area to invest in, in my mind, but also more broadly, I think that because we're technologists at the end of the day, we think of problems and then we say, "Yeah, I'll build something to solve this problem," which I think is a good instinct to have. But we sometimes forget that we still need to take people on the journey with us and knowledge is, to me, still the biggest gap. Security is still a very specialized knowledge. Security teams are the holders of that knowledge and that developer teams are stakeholders for those teams. 

The shift to making security as much of a developer knowledge area, as say, infrastructure has become or operations has become, that's still a big flag. Most developers now don't cringe so much when you tell them, hey, you need to really know what the operational metrics are for your service. You need to monitor those and then you need to adjust both your application and the infrastructure on which it runs for performance. That used to be a thing that a developer would say, none of you do it. And then we've come a long way and that's become part of the core responsibility of a developer to think of the performance of their application and how it responds to and reacts. Honestly it responds to and integrates with the platform in which it runs. Security is further back, I think there. Security is still treated as more of a niche knowledge area. I guess we still want security reviews, but the security reviews would be a lot easier if security knowledge was a bigger investment for developers. And if it was more approachable, to be honest, it's a difficult discipline to master.

Mo: According to a survey conducted by Palo Alto Networks, 45% of highly prepared companies have embedded security into the DevOps processes and 41% integrate security in at least four stages of the development lifecycle. Peter mentioned that security should shift left. So, I asked him to elaborate more on that concept with one or two examples.

Peter: I'm not going to have a great answer for this, to be honest. I'm probably going to have more questions, which I think might be interesting for your audience anyway. I think one of the distinct differences between security and Ops or TechOps in general is that operations tend to be comfortable on a continuum. Meaning, my application is kind of slow or its capacity isn't very well calculated and planned for. And then there's a little better and then it's a lot better and then we have dynamic auto scaling and like we never have a performance issue. It kind of works well on a scale like that. Security doesn't work quite as well on a scale because it's like, well, we have a number of vulnerabilities. Sure. How many of these are really critical? They could be called critical right now but what is our actual exposure to a breach at this point? And the answers for this question usually are not great to put on a scale. It's either, yeah, we're really bad. We need to lock things down immediately. Like, pick a month, everybody, don't do anything, we're just going to secure the perimeter. Or no, we're okay, we can just ignore this for another year or something. Or we don't have a problem at all, actually, we're very secure. 

It's a much more of a one versus zero situation than it is a scale. And at least in my experience, it has been. The problem with that then is, the cost of failure is really high. Basically, in my mind, the transformation for DevOps was to give developers a pager, "Hey, everybody you're on call or duty." When we have an outage or when we have a higher latency or reduced availability or whatever it is, you're going to be called into a war room or on a conference call, or you're going to figure out what the problem is and we're going to solve it. And then we're going to mitigate the problem immediately and then you're going to almost immediately go work on these high priority items on your backlog in order to make sure that this doesn't happen again. This process of making performance and availability a developer concern feels to me to be a little easier than making security developer concern because the cost of failure is a lot higher in the security world. That always makes teams and leaders nervous about shifting that responsibility entirely and they're always like, "No, I want the number," subject matter experts here to tell us what to do all the time, which is typically any security team. 

I don't necessarily think of that as a bad thing. I don't know if there's a way to transform security into a more forgiving realm or invest a lot more in early detection. But again, a lot of it is theoretical in my mind. As I said, it's not exactly an answer, it's more of a question, which is how do we make security more amenable to this iterative improvement approach rather than what security looks like to me most of the time which is, like here's a kill list, high priority items to solve for security, to handle and these are identified by security. And then they're mostly action by security. It's kind of hard to shift left. That mentality or approach to detecting and solving security issues is harder than the performance or latency one.

Mo: Yeah. Now, I think to your point, you can apply maybe parts of that pattern that you mentioned about DevOps. So, DevOps is still responsible for the availability and the reliability of the infrastructure. But when it comes to, for example, to cloud native, you give the developers the ability to control how much resources they're going to give to each part or container. But at the end of the day, you have to review it. You give them the criteria and then you have to review it. But you give them somehow the keys to that part of the infrastructure to maintain it. Maybe for security, the same thing can be done. For example, hey guys, we all agree that your containers should not run as root or with privileged rights. So, here's how you can configure it. But by the way, I can capture it if something wrong happens. Because the problem sometimes may look simple or I would say yes, the security guy or engineer can point to it, but fixing it requires a team effort. Because how many containers do you have that you need to configure properly? And how many new containers are being introduced every day? It's not about the existing ones because maybe the existing ones you can automate, updating their rights. But the new ones, you do not want to be in that endless loop of herding cats, if you will, right? You're trying to get them on board. So, there could be a hybrid approach. But your point, of course, there are some critical pieces that you cannot really just hand over to your developers. Like, for example, your terraform files you have to be the maintainer of that, right? but you cannot, you cannot hand them over completely to the developers.

Peter: That's a great point. It kind of ties a little bit to what I was saying about reducing the surface area. And that's good that you could do it through tooling, because like I said, we're technologists and we like solving problems via tooling. It just appeals to us. And yeah, you can definitely use tools to say, here's a very strict definition of what constitutes a production application. Yeah, your container cannot run as root. Or your container needs to expose these specific ports, or whatever other things, these are the images that you could use. For example, we've taken a stance at Remitly to run containers that don't have a shell. It's not an option to open a shell on an existing container and we use images for all our production applications. So, you could define this narrow road and then enforce it via tooling. And that would take away a lot of your exposure. I guess a good way to approach the security problem is, take risk production as a primary metric and then find ways to measure it. So, say, here's how much risk we have or how much exposure we have and then have teams continually work towards driving that metric down. And then separately from that, there's many other security practices that you still need to run, security reviews, red team exercises, pentesting. You still need to do all of these things. You're not just going to stop because you're saying, hey, I've mastered my pipeline tool production, its layers, right? You still need to do all of these things, but at least you've shifted left some of the security concerns. Maybe not. 

Mo: Yep, absolutely. Now, my last question about the future of security, DevOps, and developers. Now, we have that trio, all those roles are working together. You mentioned some of the gaps. How do you envision tooling systems in general? I think we all agree that things are going to get more complex, they're going to get more crazy. We need to move faster. Now, what do you envision that is going to happen the next, let's say 18 to 24 months that would help those teams to work together? Especially the security aspect, the security teams, because those guys are the ones as you said, they have niche knowledge, nice experience, but they impact everyone. They have the detour right. So, how do you envision them working together so that things become smoother and faster.

Peter: This is a hard one because extending like kind of a straight line and extrapolating from where we are right now, you're right, there's more complexity. There's also just a sheer mass of services being produced in every company that I see. I think historically, we hate to let anything like fully get decommissioned as an industry. And so, we have multiple systems running for long, long periods of time. We created a lot of new services, there's also a real explosion on the market in terms of demand for technology. So, we're just going to have more things and they're going to be more complex. It's not going to be easy for all of them to fit together and security will have a harder landscape to deal with over time. I love technologies like containers, Linux containers, and Kubernetes for example, and Istio but these are complicated things. And their complexity doesn't seem to be going down over time if anything is going up. So, we're going to have a much more complex landscape in front of us to secure and to deal with and to reason about after all, like make decisions and understand what's happening.

Mo: A few trends taking place right now that are shaping DevSecOps. The top notable ongoing changes are as follows. Number one; monolithic to microservices architecture. This trend, which is becoming the new normal, is a great catalyst to DevOps adoption. And with secure infrastructure such as containers, is indeed upgrading work cycles to DevSecOps. Cloud-first is the second one. Cloud-first architecture is directly integrating a part of the DevOps work cycle into the company, making it promotive of DevSecOps adoption in technology organizations. And finally, Kubernetes plus DevOps.  Kubernetes sets an infrastructure backed on logging, continuous detection, consistency, and self-healing, making it extremely easy and consistently really faster for companies to make sense of data and potentially eliminating human interaction. So, I asked Peter how he envisions DevSecOps tooling and systems in the next 18 to 24 months.

Peter: To be completely honest, I think security as a practice needs to shift a lot more towards engaging with developers rather than engaging with systems as a way to try and deal with that. So, scale out the security model by, we talked about this a little bit earlier and like I said, I don't know if that it's entirely achievable. But it sounds to me like something we should at least try very seriously, which is, what does it take to make every software engineer a security expert to some degree? Now, that might sound completely crazy, but I'm sure it sounded equally crazy when we told every software developer, you need to also kind of be a system admin, in a way to some degree. That means security as a practice needs to shift a lot more to education, to training, to engaging in building systems directly with software developers. The embedding of security engineers in individual business units might be something that I honestly haven't seen done anywhere but it would be very interesting to see how that plays out. Like security teams being central is very powerful. As you said, somebody needs to own those terraform files and somebody needs to be able to do those security reviews as well as run those trainings I was just referring to. But also, do we need security to be embedded kind of like QA is embedded or DevOps is embedded in some teams? I don't know, maybe it's something that's very suitable for some teams that are building user facing services, for example. Basically, we need to create more security awareness and knowledge in different parts of the organization in different parts of the technology world. And I don't think of security as a silo producing tooling and policies and working through a strict interface of reviews and tickets and that sort of thing, I don't think it's going to scale given the level of complexity and just the sheer size of technology, a sheer number of technology changes.

Mo: Yep. And if you don't mind following up on that. So, what do you think the security team should know more about? You said that they need to work more with the developers. What do you think they should know about what actually happened, in that direction?

Peter: They need to talk the same language, I think. I think security teams need to develop a product mentality a lot more, and understand who the users are. Not your users as a security team but the ultimate customer. Who are we serving as a company and why? What matters to that user persona? They need to get much more involved with product managers than they currently are in my experience. Right now, the engagement between product management as a function of company security is usually limited to things like logins and authentication. Security is still thought of, I think, as a back-end function. But we kind of need to learn more about the product, learn more about the customer, and learn more about what are the priorities of these software developers that we interact with on a day-to-day basis?  Again, it's not very different from the shift I've seen system admins take in order to become DevOps engineers. They needed to worry and understand and interact a lot more with the product than they did in the past. Basically, there was a wall between tech operations and developers and we had to tear it down for DevOps to work. And I see that same wall between security and development as well as product development or product management. That same wall kind of needs to go down for security to be able to scale out.

Mo: Yep, I 100% agree. And actually, to your point, the developers, DevOps, they're all about increasing the value of the company in the future. They add more product features, as you said. The security teams are all about protecting the current value. This is the point where they both have that friction, because for you to increase the value, you have to change what exists, right? And they don't like that because they want to protect it. That's our main goal. That's usually where the friction comes from. So, to your point, when the security team starts to understand why the operations teams and the developers are now changing the status quo and trying to move forward because there's a user need, there's a business need, then they become more and more speaking the same language.

Peter: Yeah, I absolutely agree.

Mo: Thanks for listening to this episode of the SaC. Mo was your host today with Peter Samaan, who works as an infrastructure engineering manager at Remitly. We discussed his experiences working with security initiatives and how Remitly prioritizes security without impacting its operational agility. Please pass it along to someone you believe can apply some of what we discussed here. For updates on upcoming episodes or to find out how to be our next guest, please visit magalix.com/thesac.

Comments and Responses

Related Articles

What Is Zero Trust Architecture and How Does It Work?

In an enterprise environment with containers and micro-segmentation, zero-trust architecture helps enhance security protocols. Learn more.

Read more
Cloud Asset Management and Protection: Storage Assets

Learn useful strategies to manage cloud storage assets and get the most value for your investment.

Read more
4 Reasons Why Companies should Codify their Security

The move to the cloud has significantly increased the operational and security complexity. Codifying security policies can help mitigate the potential risk

Read more