As introduced in the last post, a successful security program is made up of two distinct elements:
A series of episodic big bets that yield transformational improvements.
A set of management practices and approaches applied relentlessly, iteratively and subject to constant incremental improvement.
This post looks at the second part: the overall program management and practices that are necessary for ongoing and effective risk mitigation. This is not intended to be exhaustive or totally in-depth, so, consider this a launch pad for further research. Similarly, these are mostly management practices not technical practices - this isn’t a security engineering post.
Establish Sources of Funding
As you begin, or enhance, the security program you will either have been assigned or will have to develop lines of funding to build an operating budget. Every organization does this differently and it also depends in many cases on where you are in the organization. I would recommend you aim to drive at least 4 lines of funding:
Base operating funding. This is enough to keep your operations going and to maintain adequate oversight over all the major risks across the corporation. This is likely dictated by mandate and approved by the CFO with funding either coming centrally or from a required allocation from revenue / business units.
Episodic / major investments. Specific funding for the big bets we talked about in the first post. This may not be under the security team’s management depending on the nature of the investment. If it is under the management of the security team then some of the funding will need to be allocated to other areas responsible for their part of the work.
Activity based / flexible funding. This is incremental funding from revenue / business units according to the activities they conduct or the inherent risk they are running. For example, if a business unit is radically expanding its digital presence then they may need to fund an increase in the application security team’s work. Ultimately this may become a permanent item and this expense would be incorporated into the base operating funding and proportionally charged out to the business unit(s) driving the most need. I’ve seen similar approaches work by assigning funding for various activities according to the risks a business unit is running. For example, if a business unit has a high level of residual risk then it may need a disproportionate amount of security operations / incident response coverage and support, and so might need to pick up a bigger share of that cost as the Security Operations Center grows. Be careful with this though, while it correctly sets up the incentives (among many) for business units to decrease residual risk so as to reduce cost you can potentially disappoint them if you don’t pre-communicate that there is a budgetary floor to cover the base operating costs.
Preventative maintenance. This is the budget required of each team - revenue and support areas - to reserve resources (people and money) to preemptively update systems. This is to avoid systems becoming stagnant and unmaintained, to accommodate the costs of vulnerability remediation, control improvement, risk issue resolution and likely also audit findings and possibly compliance / regulatory upgrades. Without an established preventative maintenance budget then it is inevitable there will be a build-up of technical debt across various dimensions and an increase in residual risk. You will likely not control this budget but should work with the finance (CFO) and business teams to establish what the percentage of other area's budgets are assigned to this. This is another possible place to apply incentives akin to SRE error budgets where superbly maintained and streamlined businesses can run with less preventative maintenance (as a percentage) and hence either become more profitable or direct more investment to business development. Similarly, if a business has overly economized here and has had a build-up of technical debt, risk issues, or avoidable incidents then that is trigger to mandate an increase in the percentage of the budget required for preventative maintenance going forward.
In determining the budget levels and justifying the required expenditure it is worth using a supply and demand approach.
In simple terms (described in more detail here), you determine what is the demand on your operations or the demand for resources for risk mitigation activities. Then you look at the supply of people, money and other resources - there will be an inevitable mismatch in supply and demand. The naive approach is to simply ask for more budget to make this balance. A more sophisticated approach is to work both sides of supply and demand by seeking to reduce demand through inherent risk reduction or risk avoidance (1 and 2). Additionally, think of different means of meeting supply requirements through efficiency and productivity improvements or structural changes (4). If that isn't enough then appeal to add resources (3) and if that isn't granted then proceed to have executive level (perhaps Board level) sign-off of the residual risk associated with mismatched supply and demand (5). I've often found that across many organizations that apply this technique you get the benefits of realizing efficiencies but you also show a level of commercialism that makes it likelier you'll get the same or higher budget than if you just asked for a big increase without much analysis.
As part of budget justifications you may also be tempted to develop an approach for Return-on-Investment (ROI). This is dangerous to do simplistically and I’ve seen many security teams get tripped up by savvy CFOs or business unit finance teams who can easily pick apart tenuous numbers and soft ROI justifications. But, tangible ROI can be found and can be taken seriously. Such tangible ROI is more likely to do with investing in tools or services that actually save money on people / other expense or otherwise assuredly avoid future cost. Loss avoidance ROI or fractional time savings across large numbers of people usually doesn’t work because it’s so easy to refute and also so hard to actually track and show the real cost savings ("show me the actual money"). The best approach is usually to treat security as a fixed cost justified by corporate need with a focus on ROI to show attention to efficiency, leverage and effectiveness. A more thorough discussion on this is here.
Finally, you might want to or be asked to develop some budget benchmarks to compare yourself to other organizations. Try and avoid this and instead shift the narrative to benchmarking on control performance and effectiveness not budgets. The reason for this, explained in more depth here, is that you want to benchmark results not inputs - budget is mere input. Someone else having a bigger budget than you doesn't mean they necessarily have a better security posture.
Build and Maintain an Effective Team
You will either inherit or have to build a team. In either case you will need to pay constant attention to whether the team is effective vs. its mission. The mission changes all the time and so should the team. This can be through periodic adjustments, realignment and constant professional development and growth of people at all levels. Particular attention should be paid to diversity (in all its forms), not just because this is the right thing to do but also because diversity and consequent diversity of thought is a prerequisite for effective risk management. Diversity is also crucial to be representative of your customer base that for many organizations are global and inherently diverse. Basing risk decisions, implementation choices and operational approaches from a singular mindset is likely to put you at odds with large numbers of people, teams and customers if you don’t also intrinsically take their perspective.
It is also vital to establish a talent development program from entry level upwards so that you are building a solid leadership pipeline for the future. You might need to hire a lot of people as you’re building the organization, many of which might be senior lateral hires, but in the long run you need to be developing and promoting from within. It’s never too early to start this pipeline and take risk with getting people into roles who may not be a perfect fit but have the right attributes and can be trained. Your success as a leader, in the industry, not just in your own organization is measured by how many people you or your team have developed that have risen through your or other’s ranks to their own level of success.
As you build the team you will need to look at the balance of role types. Teams that are solely technical can struggle to communicate in business terms and win support for necessary change. But, conversely, teams that are full of risk/program managers with no technical depth can create an illusion of effectiveness while crucial issues remain undiscovered, and hence unresolved. Even if you have a good balance of technical and risk professionals then you also need a cadre of operational experts to shift your program from artisanal to industrial - to scale and sustain. I like to think of this as a “rule of thirds” and while each role or sub-team doesn’t need to be perfectly balanced - your overall organization in summation does. This post goes into a lot more detail on how to do this.
No matter how much budget you have you will always be constrained by staffing - whether it is an inability to find the right talent, develop your existing talent quickly enough or to adapt to a specific mission or business challenge. Here is another place to use the supply and demand mental model.
In other words, don't just look at the raw supply of how many people you have, also look at the means of enhancing their productivity to match demand as well as exploring ways to reduce that demand. See this post for more detail.
As you develop the organization you will need to consider whether to be centralized or decentralized, and if a combination of the two is right then which functions are centralized or not. This could be a whole post in itself and there is no right answer here as it depends on the organization’s size, scope and culture. As that changes often then the centralized vs. decentralized approach will need to change. For many large companies the approach I’ve usually seen people gravitate to is a combination of centralized teams (under the CISO) for policy, strategy, security operations, tooling / framework development, global risk assessment and so on, with the business units having a specific CISO for their own specific risk mitigation programs.
Finally, you will also now (or before) be starting to contemplate the positioning and reporting line of your team. People over-think this. Also, the people who seem to have the strongest views on the reporting line of the CISO have never been a CISO. People that are or have been tend to be more pragmatic about the issue. When you are a CISO you often realize quickly that the most important things are to be in a position of influence to get what you need to protect the organization, to have sufficient authority (directly or through your governance apparatus) to hold the line on crucial security matters and to be in the part of the organization where you can embed security by design. If you find yourself thinking that you can’t solve for this without a specific reporting line then you might want to first question your options and what flaws in your governance model you first need to fix. I talk about some of this in this post.
Persistent and Useful Engagement
The most effective security leaders do these three things:
1. Seek adjacent benefits. They realize it’s not enough for security (or other risk mitigations) to just avoid incidents and losses. The real success comes from finding and realizing adjacent benefits that deliver business (or mission, if you’re not a business) value. This can come in many forms and even if they’re not dramatic benefits then you will still get significant kudos and commitment from business leaders for simply trying to be commercial and mission aligned - and this will increase the chances of support for all the other required work. There is more detail on this here and here. Finally, while there may be situations where you just have to say no to something, most of the time the most commercial and mission oriented security programs are more likely to say, “Yes, but do it this way."
2. Build relationships. They build relationships in structured ways according to the objectives of the program. Also, they don’t wait to be given a, so called, “seat at the table”. Rather, they go work their way into seats at many tables in deliberate ways.
3. Persistent Engagement. They stick around and are relentless over long periods of time. Leaders with a long term mindset realize that most major changes take time (multiple years) and they prepare their leadership that this will be a long haul.
Governance, Risk Committees and Board Engagement
You will need to establish an approach whereby the Board, the Executives and other leadership have appropriate oversight and visibility into the security program. This is so they have a say in prioritization and are informed about the current risk profile. But, it is also so they can provide strong and visible support ("air cover") for the security leadership and program in what will likely be a sequence of difficult and perhaps even culture-changing programs of work. I have seen, in many organizations, the most useful pattern is to focus on:
1. Board Relationship. Establish a relationship with the Board and not just inform them of your risks and status but also educate them on how to oversee security risk so they can be an ally as well as a source of appropriate challenge. There is a lot of advise on how to do this here and here. 2. Risk Register. Establish a risk register to focus attention on which risks you should to go after first. Actively manage the risk register as if it were a maintained financial ledger and as part of curating risks in the register also implement an effective issue/risk escalation process as described here.
3. Risk Committee. Set up an executive risk committee to operate below the Board to more regularly and operationally oversee risk identification and mitigation. Tips for running this are here. As part of this, use scenario planning as a risk communication technique. Regularly pre-warn leadership that as risk management matures then the situation across many risk types can look progressively worse before it looks better. This uncanny valley of security is a frequent cause of premature adjustment of security program direction and leadership (which is why I call a variant of this graphic the "don't fire me" chart). Getting this right permits the persistent engagement we talked about before. Also, as part of your risk committee consider methods to quantify risk but don’t obsess on perfection.
Slipstreaming and Business/Process Integration
While building relationships with business units or other functions also keep a constant eye out for, and integrate with, major business or other initiatives to help make them successful - not just secure them. A big part of getting things done in organizations is to not just champion your own initiatives but, rather, to slip-stream improvements in as part of other initiatives.
Keep watch for moments of change or even crises/incidents as opportunities to advance an agenda you might have prepared earlier. Be constantly looking to improve the speed of your own operations. It is especially important to pay close attention to the speed of your threat and risk OODA (observe, orient, decide, act) loop.
Manage Your Attitude
Establishing, enhancing and running a security program is one of the hardest jobs there is. It requires a unique balance of protecting and enabling business and missions. Too much protection can strangle the organization just as too little slices it up. There are 4 key attributes a security leader, or indeed any security professional should follow to keep a positive attitude so as to be able to drive the persistent and positive engagement needed for long term success:
When something goes wrong assume it is always your fault. Even if it isn’t. What I mean by this is that in any situation where it hasn’t turned out well you should adopt an attitude to consider what you could have done differently, what lessons do you or your team have to learn and then pick yourself up and go after it again. This also is psychologically healthy as you retain a sense of agency. A corollary of this is the leader’s dictum that success is always due to your team and failure is on you. If that seems tough then remind yourself that this is the life you chose.
Take care of yourself. Not just taking overall care of yourself but paying particular attention to managing stress. "It's not the load that breaks you down, it's the way that you carry it."
Handle conflict well. In most roles there is inevitable disagreement and conflict. Most leader’s success is dictated by how they handle this and especially how to deal with disagreement in a non-combative way.
Focus on career success over the long haul. You might, sometimes, have to take one step back to then go two-steps forward - and you might need to temporarily take on responsibilities you don't want to for the good of your organization. But, remember, that while you might not be able to choose the work you want you can mostly choose the way in which you do it to make it more enjoyable (don't only do what you love, learn to love what you do). I remember when I was first "asked" to take on business continuity / disaster recovery work many years ago in a prior role. I positively didn't want to do it but quickly realized that I could do it in a way that made it interesting. The approach we (and it's always a team effort) invented was, in the end, transformational and was one of the most rewarding things I've done so far.
Establish a Strong Security Baseline : The Essential 20
Now, the obvious nuts and bolts of your work are to establish a strong security baseline - not just of technical controls but of management processes, operational practices and risk assessment routines. This is the essential list:
1. Identity and Access. Develop a rigorous and pervasive identity and access management program. This should cover the full range of organizational privileges across internal and external (supply chain) as well as human and non-human. Establish strong governance over the program and create processes to sustain controls. See here for more detail.
2. Critical Controls. Implement the CIS Critical Controls for your end points and production systems with special focus on adherence to configuration benchmarks, the use of strong authentication and overall adoption of zero trust principles for user access and production segmentation. 3. Metrics. Establish a metrics and measurement program to report on the effectiveness of your control implementation, continuous controls monitoring and other relevant Key Risk Indicators, Key Control Indicators and Key Performance Indicators. See here for the principles to adopt in selecting and managing these metrics. Look at all metrics and constantly ask, "So what?" to gauge their utility.
4. Vulnerability Management. Operate a vulnerability management program that goes beyond basic patch management into configuration and architectural vulnerability discovery as discussed here.
5. Raise the Baseline. Work aggressively to raise the baseline by reducing the cost of control so that expensive to procure, integrate or operate controls are progressively commoditized so they can be economically deployed to more places whether the current risk justifies it or not. See here for detail.
6. Inventories. Establish continuously assured inventories of key assets, but most importantly, actively use and embed those inventories into operational processes so you can triangulate between them to detect discrepancies.
7. Workforce Development. Pursue the goal of pervasive workforce development as opposed to focusing on simple security training and awareness programs. More detail is here. 8. Insider Risk Management. Threats from previously trusted insiders are increasing either as a result of disgruntlement or, more likely now, from coercion by bad actors to harm the organization. Therefore, don’t just think about background checks and surveillance, rather focus on event blast radius reduction as your primary goal for insider risk mitigation. 9. Compliance. While compliance doesn’t solve for all security issues it is nevertheless useful to sustain a big part of your baseline and is, of course, crucial if you are in a regulated industry. In such cases, seeking maximum opportunity from necessary compliance activities is important, which is explored here.
10. Intelligence Driven. Make strategic use of threat intelligence to forward position your priorities according to the threats you face. Distinguish between macro threat intelligence (directional cues for wide-scale mitigation or defensive architecture changes) and micro threat intelligence (specific indicators to prime or tune intrusion prevention and detection systems or other preventative controls). This distinction is explored in more detail here. 11. Security Operations. Establish and enhance your security operations capability and in particular focus on making it not just automated but autonomic (PDF).
12. Resilience. Augment (or even displace) scenario specific plans for disaster recovery and resilience in favor of building capabilities that can be adapted or seamlessly used across many known or even unknown disaster scenarios. This is explored here. 13. Chesterton’s Fence. Watch out for Chesterton’s fence, in other words, when you are taking over an existing security program or practice and you encounter controls that seem wasteful, intrusive or inefficient and you wish to replace them or eliminate them, first pause and consider whether you fully understand what they do and why they were put there.
14. Prioritize Uplifts. Some new control implementations may need different tactics in their deployment. Having some tactical flexibility here is key to get the right mix of controls deployed in the right time frame with the least operational risk. There is more detail on this concept here.
15. Supply Chain Risk Management. Set up governance, risk assessment and remediation for third parties and perhaps even fourth or fifth parties. Over time develop this program into an extended enterprise risk assessment that works not just down stream to vendors but also upstream to customers. This is because as customers more routinely electronically connect to you then your environment or the provision of services to them may be affected by their controls (or lack of them).
16. Software Security. Set up a comprehensive, tool driven, approach to increasing assurance levels of the software you produce or consume including establishing a vulnerability rewards / bug bounty program. Focus on seamless integration with the SDLC and reducing toil for developers by providing frameworks and tools for common classes of issue.
17. Solutions not Policies. You will inevitably need policies, standards, guidance, and procedures in various forms to encode your security objectives. You will especially need these if you need to evidence controls to some outside auditors or regulators. But really focus on making sure policies and standards are implemented automatically as much as possible through solutions, toolkits, process integration and other approaches that make the secure path the easiest path. Remember, if you expect risk mitigation from people doing things based only on reading policies then you're on a path to disappointment.
18. Data Protection. Identify the most critical data (some organizations use the phrase “Crown Jewels” to describe this) and then make sure that the levels of protection for such critical data is commensurate with the risk. 19. Blameless Post-mortems. Establish (this takes time) a culture of incident and close-call (a.k.a. near-miss) post-mortem analysis to look for root causes and to go down the 5Y’s path of looking for root causes of root causes. Do this in a way that avoids assigning blame so as to ensure as open and positive a process as possible - which will enable better identification of issues. And, remember, human error is not an explanation but rather is something to be explained.
20. Red Team. Set up a red team / offensive security program to push the boundaries and get in between the seams of controls to effectively simulate what a real top tier attacker might do. Remember, an effective red team program is much more than penetration testing.
__________________________
Bottom line: sustaining the effectiveness of your security program requires driving a relentless set of management, technical and operational practices. It is vital to keep them in balance and to keep developing the team in such a way that it can effectively sustain this. Adjust this constantly according to the business and mission needs of the organization.