A CISO’s Take on How to Build a Security-First Culture

Intro: Our daily operation is elevating security in terms of mindshare, in terms of prioritization. It's been a consistent theme and challenge, except for when I started at WebLogic know. There was a culture that believed in this and sort of understood it implicitly. After that, every other organization I've been in either didn't care, wasn't a priority, thought they could manage, and it's always been a challenge to get traction.

Mo: Welcome to the SaC or Security as Code podcast. Here we discuss different topics to streamline security from code to cloud. We will talk to experts and leaders in cloud security about the most pressing challenges to balance security and agility. You will get tangible advice to secure your infrastructure using the right combination of technology, people, and culture. I'm your host Mo, and I have with me today, Kevin Eberman, a Senior Director of Information Security at MineralTree. Today we're going to discuss CISO's point of view about how the security teams, Ops, and developers should work together to secure cloud-based infrastructure and applications. Kevin will share his experience at MineralTree to align teams and build security-first culture. This podcast is sponsored by Magalix. Magalix streamlines security from code to cloud using codified policies, tight integrations with your software development life cycles with the right insights. For more details, visit our website magalix.com. Welcome, Kevin.

Kevin: It's great to be here. Thank you.

Mo: All right, so I'm excited about having you here today with us. Can you introduce yourself and just give us a bit of a background about yourself and about also MineralTree and your role there?

Kevin: Absolutely. I've been doing IT, information security, operations for more than 20 years now. Currently, I am the Chief Compliance Officer and Senior Director of Information Security at MineralTree. MineralTree is an account payable and payment automation solution for businesses of all sizes. We operate a SaaS operation. And we're doing about $12 billion annually in payments through our platform. 

Mo: Sounds great. Now, when we were chatting offline, you mentioned some interesting stories and how you came to the DevOps world and how then you moved into security from that. Can you give us some background about this and maybe share your story with our audience here?

Kevin: Yes. My DevOps experience actually goes back up to about 1995, long before the actual word was coined. But I had moved to San Francisco at the height of the .com boom and got a job working at a company called WebLogic. WebLogic was developing an application server actually even before the term application server was coined. But anyway, they were writing server-side Java applications to enable large infrastructure and large businesses to do sort of... I'm getting lost in the words here. Anyway, I got a job at WebLogic. WebLogic was building an application server. And the folks there were very, very talented with deep experience and in Silicon Valley. And by that time developed really effective patterns for writing quality software. And those patterns are many of the things that we now call DevOps. So, it started with version control, and then it was a continuous build. And then it was also unit testing on each build cycle, automated integration testing, automated system integration testing, and then even taking it to the next level with respect to automated load testing, automated performance testing. And so, they were engaged in this feedback loop that was very effective in terms of writing effective and good code. 

And that model is really what DevOps in part became to represent. DevOps sort of took it to the next level as operations became a part of businesses, where they weren't just sort of writing packaged software and throwing it over to the fence to other businesses, but they were responsible for the operations as well. And applying those same techniques to operations is something that was very interesting to me. And I kind of go back to my history about how I got connected to all this. After I was at WebLogic, and I didn't want to share. WebLogic team was very, very talented. And there was a number of people that were young people at the time that went on to very distinguished careers being the CTO at places like Twitter, Pandora, Sigma Computing. But they all had that culture of DevOps long before the term DevOps existed. And in terms of operations, it was sort of too early in the market in terms of time to market because it was intended to be a platform as a service. But the market wasn't there yet because this was in like the 2000, 2001 timeframe. But the way we sort of envisioned this platform as a service, incorporated all the ideas that exist now as DevOps. And that was like server configuration management. And so, in terms of server configuration management, I used CFEngine version 1.0. And that was the tool that we used for server configuration management. 

When you fast forward 15 years, well, I don't know more like eight years to like 2003, 2004 when the term DevOps was coined. Yeah, and I was at a conference. It was a LISA conference in Boston. LISA is the Large Information Systems Administration conference. And DevOps was sort of the topic of the day. And it just sorts of fit perfectly into how I always envisioned operations and development should work together. And it felt like a real home and that these ideas about developing really effective code were being sorted of elevated into the Ops range. And it was all very meaningful.

Mo: I personally saw a lot of successful security engineers coming from backgrounds in computer networks, operating systems, or DevOps. It is interesting to see how DevOps were able to make this transition and introduce some agility to security practices. I asked Kevin, what motivated him to switch careers from DevOps to security and how he managed to do this transition, given the greater responsibilities he will be holding.

Kevin: One of the things that always drew me to Ops, it was where the rubber hits the road with respect to taking what development writes in terms of product and making it available or accessible to customers. And I appreciated being in a profit center part of the business rather than in IT, which is a cost center. And I liked and I thought it was exciting to be on call to see things as they were happening in real-time and to have your hand on the controls for operating a large-scale operation. Now, when I was working at a company here in the Boston area that was an Internet of Things company that had very large operations, and at the time, I was building up data centers for them. One was in the United States, one was in Europe, they have regulations because they were selling to medical equipment manufacturers. And one of them was based in Europe and so for privacy purposes, it was over there. But anyway, and so very involved in operations, large scale building. And as part of that, working with these medical equipment manufacturers, they were very interested in security. And I got exposed to the sort of formal information security practices. And that really appealed to me in terms of it being a lexicon of how to talk about security and how to manage and mitigate security. 

And I could see that security, which was a critical part of not just our business and what we were trying to do with that company, but in terms of everybody's experience with technology, and everybody has a visceral experience with that. And that goes at a personal level from identity theft or leakage of your personal information all the way up to multinational organizations, banks being hacked, Google being hacked, Amazon being hacked, Target being hacked, the CIA being hacked, and elections being hacked. Or even sort of the next iteration of that where information and how information is attacked, whether it'll be stored in Facebook and how it's sort of communicated through Facebook, that's causing other security issues that are more nuanced and sophisticated. And all that tells me is like security is this thing that's super-duper important that affects all of us. We don't have our arms around it and I want to be a part of that conversation and try to make an impact.

Mo: Any successful company with online systems is a target for hackers. That makes the career of security engineers and leadership in security extra stressful. Starting a new job as well as its own stresses, regardless of your role that stems from flipped priorities, lack of resources, or some cultural adjustments that leadership needs to sponsor and push for. I naturally asked Kevin about his first set of challenges when he joined MineralTree.

Kevin: I was actually hired as the Director of Operations and made this transition to security while at MineralTree. And in some ways, it sort of speaks to my individual challenges but sort of collectively a challenge with respect to security and that is that for many companies, that security is not a priority. And that many companies think that they can take care of security without dedicating resources to it. But you know, by that time, I was very clear about where I wanted my career to go. And so, even though I was hired as the Director of Operations, I had already attained certification in CISSP. So that's Information System Security Professional because I wanted to be credible to talk about my security experience. And MineralTree needed an Ops guy. And it was a plus that I also knew about security. 

So anyway, I wanted to make that clear that I was hired to be the Ops guy. Now, I would say that with the respect to the challenges at MineralTree were the same or similar to what I've seen at other companies and other experiences where their processes were amateur. One of the things that I did early on was implement server configuration management. So even though our environment was up in AWS and we were operating physical hardware, essentially, all the systems were being managed as islands, individually manually. And so, I implemented Puppet as one of the first things I did so that we would have centralized server configuration management. And obviously, that has real impacts with respect to security and in particular, security expectations. That was amongst the first things that I did. And I want to say in terms of challenges, sort of elevating operations, elevating security in terms of mindshare, in terms of prioritization, it has been a consistent theme of a challenge except for when I had started at WebLogic and the job after that, where there was a culture that believed in this and sort of understood it implicitly. And there was never any question about how to do things. After that, every other organization I've been to either for different reasons didn't care, wasn't a priority, though they could manage at last, or whatever. And it's always been a challenge to get traction.

Mo: Threat modeling is a process by which potential threats, such as structural vulnerabilities or the absence of appropriate safeguards can be identified, enumerated, and mitigations can be prioritized. It is a systematic analysis of what controls our defenses need to be included. Threat modeling answers questions like, where am I most vulnerable to attack? What are the most relevant threats? And what do I need to do to safeguard against these threats? There are three known threat modeling methodologies, STRIDE, PASTA, and Trike. Let's talk about STRIDE. The STIDE approach to threat modeling was introduced in 1999 at Microsoft, providing a framework for developers to find threats to their products, which includes spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege. I'll talk about the other two approaches later in this episode. I asked Kevin, at this point, how he started his new role at MineralTree and how did he model possible security threats?

Kevin: It can be overwhelming. And I think that there are some well-developed patterns for managing that. And the first one that sort of springs to mind is about risk assessment and identify where you're vulnerable. And risk assessment actually starts with an inventory. In terms of inventorying your systems and applications, you know, what it is you're actually trying to protect. And then from there, you know, so you have an inventory of your environment and your systems and then you can look at like, well, what are the vulnerabilities of those systems? And then what are the threats to those systems? And then based upon those threats and vulnerabilities, you can evaluate what you think your risks are and then prioritize. 

The other thing I would say with regard to this is that there are other frameworks that are out there. And the one that I find very useful is PCI. And that's for the payment card industry. And whether or not you're actually storing credit cards, the PCI framework is a framework that can be adapted to any information system. And in terms of the 12 requirements within PCI, most of them are just plain old common sense. Requirement number two is don't use default passwords. If you are, this is low-hanging fruit to sort of clean things up and make your security posture better. Because it's not that hard to do. But it's a mistake that lots of places still make where they just don't update those things. And so those frameworks can help you identify what kinds of things should you be doing in a comprehensive way and how do we prioritize.

Mo: Based on the US Bureau of Labor Statistics, the employment of information security analysts is projected to grow 31% from 2019 to 2029, much faster than the average of all occupations. Demand for information security analysts is expected to be very high as these analysts will be needed to create innovative solutions to prevent hackers from stealing critical information or causing problems for computer networks. It is certainly a challenge for all CISOs to build a strong team fast enough. But you can get creative building such a critical function. Let's listen to Kevin's point of view about what kind of engineers you should hire and where you can source for the best minds, even if they're not yet classified as security engineers.

Kevin: This actually kind of speaks to one of the things that have happened in my career. Well, I mentioned WebLogic and I was the helpdesk guy or the IT guy. And I was setting up laptops and workstations for developers and also IT systems, mail servers, and file servers, and all that physical hardware and infrastructure. And where I'm getting to is, all of that physical hardware and infrastructure is not a job requirement that exists for many IT people anymore because everything's being hosted in the cloud. And so, the skill sets, job requirements have changed over the course of my career. And someone who knows scuzzy drives, RAID formats, and all that stuff, it's not just relevant anymore. What is relevant now is scripted. And so, the key requirement for me for hiring people now in a DevOps or Ops or sort of the dev security Ops role is, can they actually code? Can they write scripts? Can they write a script that will sort of extract information from an API? Because in a DevOps role, in a DevSecOps role, you're often patching together different systems with scripts, with Python, or other things. It's not programming in terms of a product necessarily, although it can evolve into that. But that's fundamentally a job requirement these days.

The other thing that I've always felt was really important is sort of like, do they have basic sort of fundamental skills? In my mind, that's like, do they understand how the internet works? You know, what is an IP address? What is DNS? How are packets routed? What does a firewall actually do when it does filtering? You know, these kinds of things that well, oftentimes today, I still find developers that don't necessarily understand how Ops actually works and how things on the internet actually work. So that's also a key thing that I look to make sure people really get at a very fundamental way. I would say there's a third thing that's really important and is whether or not they understand and buy into a DevOps methodology. Because I've certainly talked to people that are like, "Having training is a great thing." Like, we should be able to do all different kinds of things and make it work in all these different places, whatever. And you know, it's just a different approach. So like, do they get it? Do they buy into it? That's key because ultimately, it's people that drive DevOps.

Mo: Security is a journey, not a destination. Therefore, building the right culture to continuously enforce the best security practices is a critical factor in this journey. You can take a few very practical and actionable steps to develop and nurture a strong security culture across seven distinct dimensions. Number one, employees’ feelings and beliefs about security protocols and issues. Second, employee actions that impact security directly or indirectly. The third is the employees’ understanding, knowledge, and awareness of security issues and activities. Fourth, how will communication channels promote a sense of belonging and offer support related to security issues and incident reporting? Fifth, employee knowledge and support of security policies. Six, employee knowledge and adherence to unwritten rules of conduct related to security. And finally, how employees perceive the role as a critical factor in helping or harming security. I asked Kevin, how they were able to build a continuously improving security culture at MineralTree.

Kevin: The first thing to remember is that security is a journey, not a destination. An audit is a point in time, not where you're going. And that depth of landscape and profile of the threats and DOS attack space change, and that you have to continue to work out. It's a perpetual work in progress. And I think that's the most important thing to recognize. I think in terms of sustainability, that often it means sort of developing, enhancing good tools that are sustainable and are adaptable and that are a platform for you to build on the sort of to the extent that you're doing things manually or you have weak or bad processes. Where the infrastructure doesn't do what you need, you can sort of grind yourself into very specific things that aren't repeatable. And so, I think that sort of emphasizing repeatability is a critical part of sustainability.

Mo: The second type of threat modeling is PASTA, which stands for the process for the attack, simulation, and threat analysis. And this is a seven-step risk-centric methodology. It provides a seven-step process for aligning business objectives and technical requirements, taking into account compliance issues and business analysis. The intent of this method is to provide a dynamic threat identification, enumeration, and scoring process. At this stage, I decided to ask Kevin about team dynamics and how he was able to make the security teamwork with the rest of the engineering teams. I was curious, where did they start their efforts to secure their systems as well.

Kevin: One of the things that I feel has been a positive experience in this role is to move security, and not just security also Ops further left into the design process. Because particularly with Ops, you often find out about things at the end of the project cycle. So the product team and development team go and build something. And then with three weeks left to get to their deadline, they say, "Hey Ops, deploy it." And Ops is like, "What do you mean deploy it? We're completely underwater. There was an outage yesterday. Everybody's tired. We have to do a rollback. We have eight other projects and now this is the most important thing?" And that's a pattern that happens all the time, that Ops gets jammed. And then also the security gets jammed as well where very late in the process we find out that something's been done. And at that point, the product and developers are looking for, well, give us a thumbs up or thumbs down. Does it meet our security requirements or not? And if it doesn't, then tell us the three things we need to do to fix it so we can move on. And so, moving that process to the left, I think is the biggest challenge. 

And that really requires change in how the development leadership thinks and also how about the product leadership changes. And I think in my role as a security person, I'm in a better position to advocate for Ops and make sure the Ops teams needs are moved left. Because all of those things have to work together to be effective in terms of when it actually gets delivered to the customer. And so, everybody is participating in the design. There is buying into what we're doing. There is a clear plan of how we get there. There is an understanding of the work that's involved, and there's a higher probability of getting to where we want to be. And I think in terms of security, it's been easier for me to move Ops and security left rather than when I was just in the Ops role.

Mo: The last type of threat modeling Trike. The focus of the Trike methodology is using threat models as a risk management tool. Within this framework, threat models are used to satisfy security auditing process. Threat models are based on the requirements model. The requirements model establishes a stakeholder-defined acceptable level of risk assigned to each asset class. Analysis of the requirements uses a threat model from which threats are enumerated and assigned risk values. The completed threat model is used to construct a risk model based on assets, rules, actions, and calculated risk exposure. I decided here to talk with Kevin about tackling security issues early in the software development process. Everyone now is talking about shifting left everything since infrastructure is now fully qualified.

Kevin: Fundamentally, like having all these stakeholders participating in the design process also means that they're bought in. They're a part of the process. And I think that's key because, throughout the whole software development lifecycle, there are a number of areas where security needs to be addressed. And often it happens within certain groups. And so, if a software developer is writing unit tests, for example, that is part of security and how they write unit tests and what sort of tests are they performing? Are they doing input validation? Checking for data types? Checking for values? Checking for range? Because these are all security issues with respect to, and then also with respect to what you mentioned, peer review. Are they doing peer review? And so, part of my role now in terms of the security person is compliance. So, are we opening up tickets? And that change management part of it and making it audible and repeatable and observable in terms of what the process was. And so, for each change? Is it approved? Is it documented? What's the impact? Who owned the ticket? Who was the peer reviewer? Who did the QA? Did all those things happen for each change? And then as it moves from development into Ops, what actually got built? What changes? We're in that build. Was that build approved for deployment? Was it validated post-deployment? You know, all those things are sort of inherent through the whole software development lifecycle. And then on an ongoing basis in terms of doing vulnerability scans and other things to sort of make sure that we're not regressing or leaking issues of why we're operating here are a critical part. But you know, the most difficult part, and I'm coming back to this is getting everybody on the same page from the beginning of the process.

Mo: Got it. Now, since you mentioned that, maybe we shift a bit to the culture of peace. One argument or one point of view about developers versus operations versus security is as follows. Developers are all about disrupting the status quo because they're shipping new features and they're making changes. Operation is all about maintaining the system and making sure that it's available, reliable, etc. Security is all about preserving the current value of the company. You protect the data, you protect your assets, so any disruption, makes them feel agitated, right? Unless security and developers and operations need to work or think of the others differently. What challenges have you faced at MineralTree and maybe probably at other companies, from a cultural perspective to make sure that everyone is on the same page and they understand the other party? They have that connection and empathy.

Kevin: There are two things that I want to say. First of all, one thing that sometimes happens is that developers and Ops people, and even product people don't always understand the regulations and requirements for a compliance program with respect to HIPAA or PCI and what our obligations are for that. And that's one of the critical reasons why security needs to participate in the beginning of a process, to make sure that those requirements are identified and that the product is built with those in mind. Okay, so I wanted to mention that. And then the other thing in terms of a challenge is that a lot of times, and this is true for developers, it's true for product people, they don't always really understand what security needs in terms of, and when I say security, I'm talking about security in terms of compliance, in terms of it being a well-defined term that has a whole vast base of knowledge behind it that really sort of developed out of World War II and protecting secrets and emerging in large systems. And applying those methodologies to the emerging computer systems, right? 

So, security means something very specific. And that's confidentiality, integrity, and availability. And not a lot of people that are developers or product people necessarily understand that it's those three things and can sort of describe it succinctly. I have been in situations in meetings with developers where we were talking about operation, infrastructure, and design, how that was going to be built. And a developer asked me, "Kevin, why are you at this meeting? What does security have to do with this?" I'm like, "Well, the availability of the infrastructure is sort of a key element of security." I had to explain to him like, "How secure is your bank if you can't withdraw your money out of it?" That's what we're talking about. The system's got to be up for it to be secure. And he's like, "Oh, okay, that makes sense." But yeah, I see that regularly. People don't necessarily get it in terms of what my role is and what my concern is.

Mo: Got it. Now, let me play the devil's advocate here. Some argument that you may hear from the developer is, okay, we were separate from operations or Ops at a certain point, but with the cloud, we were able to understand some of the ops and we were able to participate in that and sometimes do a lot of the operational stuff, right? The operations become, the Ops part becomes more democratized and become more available for developers to do or stick their head into. Now, isn't that possible also for security? And if you agree to that, if you think that this is achievable one way or another, how can we do that? And just also to be clear about this, now, developers are sticking their hands into operations, but we still have now SREs, right? And we still have specialized Ops people. It doesn't mean that the function needs to go away. But it's a matter of, I think the DevOps and the developers being engaged with the Ops, made them more empathetic with us and make things faster than before. Do you think that same pattern could take place between security and the rest of the organization? Or do you think, no, it's actually so niche and so deep in terms of knowledge and experience that it still needs to be the way it is right now?

Kevin: I think what DevOps do was raise the bar for when you need a dedicated Ops team. And so, if you're a one-person shop, CEO, CTO, chief salesperson, whatever, you can do a lot on your own in terms of developing a product and doing the operations on your own. Now, I think for larger organizations with complex infrastructures, with no multifaceted integrations with third parties, that there's a lot of work that is required that is outside of building, developing, and deploying the product. And so, in terms of build, develop, deploy, yes, developers are more involved and have more control. And in often cases are writing and engaging in the tools that do deployments. But you know, the Ops has got a lot of other work to do in terms of managing the integrations, monitoring the system, and reacting to issues and problems. And so, Ops doesn't go away, not for a large environment. And I would say the same as similar for security. And obviously, there are certain things that need to be banked in to what the development team does and we mentioned already with regard to unit testing, and testing in general, and doing sort of security testing, and best coding practice, and peer reviews, and all of this Those are all inherently within the development organization and should stay there. 

I think that there are a few areas where security becomes more important. And again, there's a threshold in terms of the size of the business when developers can't do it all. And one of them has to do with requirements, regulatory and legal requirements. Developers should be focused on software development, you know, how to build the product. Requirements in terms of security, how data is stored, how data is encrypted, who has access to it? What are the roles and controls to get there? Now, those are all things that the security people bring to the table to help inform what the requirements should be. And then here I think there's a big chunk of compliance activities that happens when you get into larger organizations. And again, that's a role that needs to be separated. And this comes back to security in terms of the segregation of duties, that you don't want your developers also doing all your Ops functions and vice versa. And similarly, you don't want your Ops people doing all your security functions because Ops people have a lot of control and access to information, sensitive information. And having segregation of duties between the people who are doing the work and having watchers of the people do their work becomes more important, particularly with the sensitivity of the information and the size of your organization. And so, I think that journey definitely has a role even as we become more effective at automation.

Mo: Got it. Now, in terms of information flow, you know, that's fair, where you mentioned that for the scale and--

Kevin: Oh, I want to say one other thing. Security also goes across borders. They go all borders, you know, country borders, but the thought I had when I started the sentence was borders between development and operations. And so, security is able to see the whole lifecycle where developers may not.

Mo: Yeah, absolutely. And that's to your point because developers, the unit when you talk with the Ops, it's all about the infrastructure and securing it. Again, the surface is now much bigger than before that you need to secure. You're now in the cloud, there are a lot of things that can go wrong with your infrastructure. The same thing with your application, especially if you're using open source and microservices. Definitely, architecture is even--

Kevin: Everything is more porous than when we were on-premise and were protected data centers.

Mo: Exactly. Thanks for listening to this episode of the SaC. Mo was your host today with Kevin Eberman. We discussed today, Kevin's experience as the Chief Information Officer at MineralTree. We chatted also about how to get started building a security program at a FinTech company. We also discussed how he managed to switch careers from DevOps to security. Please pass it along to someone you believe can apply some of what we discussed here. For updates on upcoming episodes or to find out how to be my next guest, please visit magalix.com/thesac.