How much of your work that you would like to describe as a “grand” challenge is really more of a “grind”?
As an industry we like to talk of grand challenges, moonshots and other grandiose terms. At one level that is great, we do have grand challenges and we need moonshots. When these are successful, as they sometimes are, they are truly transformational in defeating whole classes of attacks or enabling new approaches that were previously too risky. Even when they fail, they can still improve our collective situation.
However, much work, not just security, but especially security or broader technology risk mitigation is tough. The work can require careful planning, meticulous execution, relentless influencing and prioritization. It also needs a large dose of grit and determination on the part of leaders or teams to maintain progress in the face of difficulties. It can require diligence, care and persistence in unraveling dependencies so that progress can be made without spiking other operational risks. Even the implementation of grand challenges can require such long term focus to realize that vision.
So, a major success factor of many large scale, or even smaller scale security programs, is the ability of the teams to grind through the work. But the grind doesn’t have to be negative. Teams can find glory in the grind. Leaders should establish techniques to make this a success - the grind can be grand.
The inspiration of this was this immensely useful post about grind challenges overall.
Unlike the grand challenges that transfix communities with big, bold targets that push the frontiers of achievement, grind challenges are the myriad interlocking tasks that keep our highly engineered world functioning.
Let’s look at some approaches.
1. Confront the Grind
I have nothing to offer but blood, toil, tears and sweat. - Winston Churchill
Be up front in the beginning about the nature of the challenge. Accept, if not outright embrace, that the work will be a grind but that it is important and fulfills a grand mission. Teams are more likely to press-on with the work if they know the nature of the challenge in advance.
Try and map out the expected complexity. This will likely have been done as part of the planning stage of the program that is underway. At this stage it is important to widely consult some of the organization’s more experienced and long-tenured people to get their input and, almost in a pre-mortem like experience, to get their view on what aspects of the program are likely to be difficult. The veterans, likely battle-scarred from past similar programs, will help map the difficulties so you can try and avoid them and acknowledge them to the broader team so they know what is coming. I remember multiple examples of where lessons from SAN (Storage Area Network) migrations informed DMZ upgrades in terms of the unexpected application dependency management grind. Similarly, pretty much every middleware or database upgrade has lessons for security upgrades and every prior modification to end-user or customer behavior from product upgrades informs adjustments in endpoint security policy. The list could go on.
It’s even possible to look to business programs for inspiration and motivation for the execution of complex security changes. I remember in a prior organization some vast business process adjustments in response to new regulations made even some of our bigger security programs look straightforward by comparison.
Even more broadly, there is inspiration in sharing experiences with or seeking counsel from peer organizations. One of the bigger values of industry ISACs is not actually the sharing of threat and vulnerability information but the sharing of practical experiences on how to tackle major upgrades or even just the psychological comfort of sharing the experience of the grind.
I’ve even known one organization literally develop an analogous topographical map showing the epic journey they were about to undertake to upgrade their corporate desktop environment to a higher default security state. It had named valleys and mountains, known good paths and places marked as “Here be Dragons” and the “Land Beyond the Wall”. When you map the grind it becomes more tractable.
2. Select People for the Grind - The Obstacle is the Way
The impediment to action advances action. What stands in the way becomes the way. - Marcus Aurelius
Some programs need a special type of person, not just the skills and personalities, but the experience of program management and the intrinsic knowledge of how to make progress, one step at a time, one day at a time. But in most cases you shouldn’t rely on having special people who can manage the grind but, rather, organize so that everyone can tackle this.
Establishing a solid program office, staffed with one or more (depending on the scale and nature of the work) program managers is important to maintain focus. For staffing the wider work it is, of course, useful to select people with experience of similar programs, or programs of comparable complexity. It is likely such challenges will not be exactly like the work that has been done before. It might be necessary to use outside consultants to fill in gaps of experience or even just raw capability to get this done. But be really careful to make sure the consultants, the specific individuals not just the company, actually have the right experience and grit. I’ve seen many examples of consultants being ground up by such programs if they’re the type of people who’ve only worked on new builds with less complexity.
Finally, it is vital to put some of your best people on this work. Most of these grind challenges are absolutely crucial for the success of the organization. Having your best people work on these issues is important not just to get the work done but also because it signals it is a worthy challenge. There are plenty of organizations that make such grind challenges a right of passage for people’s next promotion or career rotation: “go take that hill!”
3. Make the Grind Grand
We’re surrounded, that simplifies our problem - Chesty Puller
Many of these large and complex programs that constitute grind challenges can take many quarters, if not years, to fully complete. Maintaining consistent progress is key but so is maintaining consistent motivation. This is not just for the teams but also for organizational leadership to assure continued funding, perhaps even increased funding when the inevitable unknown unknowns surface along the way.
There are many ways to do this, some obvious and some less so:
Show and celebrate constant progress (see Grind Away, below) and in particular celebrate milestones along the way.
Plan for surges of special activity, for example, many organizations have “fix-it” days, internal “hackathons” or other moments when teams are encouraged to stop regular work and spend time burning down some technical debt, building new tools or otherwise innovating. You have to be careful here as some of these activities are deliberately shielded from being used for regular (no matter how important) work. As a result you need to frame some activities that are appropriate for such work, like developing some tooling that makes migrations easier or building libraries that might better abstract some dependencies that also help your program.
Establish some bounties and rewards, not just for hitting the milestones, but for tools and techniques to leapfrog the milestones in various ways.
Connect the progress to the risk that has been reduced already, perhaps incidents avoided, some adjacent business enablement now possible because of the progress made or even just showing how this work benefits customers or the organization’s overall mission.
Celebrate the technical challenges along the way, write internal articles or technical papers on what was uncovered and how it was tackled in heroic and interesting ways - this might include writing up remaining technical challenges to encourage submissions of ideas from other teams to help out on the mission.
Finally, make executive leadership, perhaps even the Board, visible to the program team(s) to show them how important this work is.
4. Avoid the Grind
The oak fought the wind and was broken, the willow bent when it must and survived. - Robert Jordan, The Fires of Heaven
You might think I’d have put this one up first and I could have since one of the best ways of dealing with the grind is to just avoid it. Maybe there is a way to avoid the work, biding your time and hoping for or letting the problem go away. This could be by actively deprecating some legacy technology, isolating or otherwise containing the risk and then letting it whither. Conversely, it could be that some grind challenges really are in need of a grand challenge and it might be the organization should sponsor some R&D internally or with one or more vendors to do something that truly bypasses the grind challenge they are facing. All of this is worth searching for.
But, in the event there is no avoiding the wider grind there could be ways of avoiding some of the grind in parts of the program. This could all add up to a substantial benefit over time. One of the best approaches I’ve seen is to avoid seeking perfection in managing all operating risks of changes driven by the program. For example, when migrating or upgrading applications you, naturally, want to make sure nothing breaks as a result. Having that as a goal with a 100% success objective creates a huge grind. One other approach, I’ve seen used many times, is to simply carve up the application base into critical vs. other. Then rigorously test a migration or change for the critical applications so there is high assurance they won’t fail. Then for the remainder (often this can be 20/80 but even if 50/50 you get huge benefit) do some rudimentary changes and roll with the change knowing you can roll back if something breaks. In many cases nothing breaks and you can move on having saved a ton of work and collectively the grind is reduced.
Another common practice is to evolve-out changes where you establish a new infrastructure baseline or framework and then have the wider organization adopt the changes as they evolve and change their environment - this is especially useful if there is a natural pace of frequent change in the environment. Even if there isn’t such change then you can slip-stream this where there is and then chip away at the more static parts separately.
5. Grind Away
Courage doesn’t always roar. Sometimes courage is the quiet voice at the end of the day saying ‘I will try again tomorrow’. - Mary Anne Radmacher
Whether you’re really able to make the grind grand or not it’s still important to track and show progress so there is a sense of the step by step work getting you and everyone else closer to the end state. There’s no rocket science here but it’s worth enumerating the practices:
Develop scorecards of progress and make it granular enough so that it is transparent which teams are making what progress to the goals.
Show league tables or other comparisons to create some degree of competition or other incentivization to sustain activity without constant chasing from the program team.
Link such transparency to existing executive management reporting. This could be Board reporting, risk or executive committee reporting or anything that will create “pull” from the leaders whose teams need to do the work. Having teams do the work and pull help from you is, clearly, much better than you have to constantly push.
Make scorecards and progress reporting granular enough that there is a regular sense of achievement and that progress in the scorecards is easily linked to actions people can take. Keep reducing the activation energy required of managers to take corrective steps to improve their progress score.
Create some granularity of progress so you get an early warning of program goals being at risk. There are so many examples of programs or projects where the status reporting was “Green, Green, Green, Green, RED!!!!!” In reality you want some quantitative or qualitative reporting to flag project risk, or create “canary” milestones to warn of things going awry before dealing with broader failure.
In the program reporting, use your team to do some thematic analysis of successful patterns that result in a short-cut to achieve goals in some way or to highlight issues some teams are facing that others can get ahead of. Your program office is as much an intelligence team as a drum beat for progress.
I find stagger charts are a useful means of weaving together progress transparency, creating pull from teams (because they get to assert what they predict as their next result) and revealing themes or concentrations of issues for the program team to address.
Finally, make sure when you are reporting progress you are reporting progress to the full risk mitigation goal not just the immediate project objectives. There are plenty of examples where teams will declare we are 100% done on X, but X is only a mitigation of 20% of the overall risk or scope. So you’re really only 20% done. Having paired metrics of percentage done for implementation with percentage scope addressed is a means of not just getting the job done but also avoiding the false hope when intra-program milestone completion is over-heralded.
6. Connect the Grind
If you’re going through hell, keep going. - Winston Churchill
In many respects this is a part of the grind away of program progress reporting and other measurement. I include it separately as it is so often overlooked. There are many points to integrate your program into the teams who you need to do work. Sometimes the scale of your program and its visibility causes you to neglect the micro-integrations that are still useful. These could include aligning with OKR processes, existing management reporting, linking to performance review objectives all the way through to periodic reviews in management / team meetings and other venues.
7. Give up the Grind
You may have to fight a battle more than once to win it. - Margaret Thatcher
Finally, let’s not forget that sometimes it is not only fine, but actually necessary, to cancel a program or project that is just not making the right progress in the right way. This is often because of unexpected difficulties, changing factors in the wider organization, or mostly because the team(s) didn’t do a good enough job to plan the work or did it in such a way that there were too many overly optimistic assumptions for the work to be a success.
There are ways of countering this by conducting pre-mortems. You can also break down the project goals into all the major activities and then develop a probability range or distribution for each element. Once you have this you can then run a small Monte-Carlo simulation to look at the likelihood of success of the whole program. This is a good article summarizing that approach.
In any case, if you’re in the mode of a program not succeeding it may be crucial to simply fail it, pick up the pieces and restart, having done a post-mortem. Sometimes the restart might be quick in other cases it might need a longer pause. However, unless the risks have truly changed it will still be necessary to tackle the work and ask, “are the conditions that caused this failure still true?” One example I worked on a few jobs ago was the implementation of a software allow-listing package to achieve the goal that all endpoints should only execute explicitly allow-listed software. This was a true grind of a program that eventually was successfully completed and delivered immense risk mitigation. But, we had a number of false starts and restarts due to a range of factors from overly optimistic planning, underestimation of dependency impact and a bunch of over-promising and under-delivery from the particular technology vendor we were relying on. A true grind.
Bottom line: you should aspire for grand challenges to make radical changes in what you do, but recognize that many challenges are more of a grind. When this happens get your playbook out to do all you can to find glory in the grind - to make the grind grand.