Do You Really Know What’s Going On?

At some point every leader needs to ask themselves: Do I really know what is going on in my company? Do I even know what is really going on in my own organization?

Most leaders do not know the actual truth of what is happening. This is not because people are overtly hiding things or that leaders are ineffective, although sometimes it is both of those, but rather this is because of the “thermocline of truth” that I covered in this post. Organizations are full of cultural, structural, process, and other barriers that stop reality making its way to leadership. 

When you started out in your career you probably felt constant frustration that the “higher ups” had no clue about reality on the ground. Even today, you probably feel frustrated that peers in other organizations are clueless to the reality you know, or even in many cases that you know their teams know but are failing to push up to their leader. 

When you become progressively more senior it’s easy to forget this experience and get in a position where you believe you know everything because you sit in the project update meetings, the risk committees, or read the status reports and other summaries that have been carefully curated to please you. 

There is also a related problem where every leader has a set of beliefs frozen in time from when they were on the front line. They still filter the world through that mental model despite things having moved on dramatically since then. I’ve been in countless situations debating a senior technology executive about some need for change in approach on, say, software delivery security that they said would never work based on the premise of “back in their day” the manual controls wouldn’t scale. This confidence is without their realizing that how it was all working in their day has been replaced and automated many times over. I’m sure I’ve been constantly guilty of the same pattern of behavior despite my trying not to.

More comically, in one of my first board meetings at a grand old Merchant Bank in London in the 1990’s a rather crusty old gentleman (he might have literally been a Lord) who was a Board director asked me whether “we all still wore white lab coats while tending to the computer machines?” In a similar situation, in the 2000’s I was presenting to an Executive Committee at another Bank where we were upgrading the strength of customer authentication to our call centers. A senior person exclaimed why we would need to do this because “my personal banker can recognize my voice when I call him up, and besides I do all this with him on the golf course”, oblivious that pretty much everyone else had to call an unknown person in some call center in rural America. 

Anyway, divergence aside, what are the ways to get to ground truth? 

Walk the Halls and Factory Floors 

One of the most important ways to know what is going on is to see for yourself. Visiting your various sites and facilities, production lines, data centers, call centers or anywhere your security touches workers, customers, partners, and suppliers. This is especially important when you are new to an organization but it’s also important to do regularly. I’ve always done this when new and I tried to do this often thereafter, but I admit it can take discipline to carve out the time for this. When I lapsed and let too long go by between such visits to key locations I always regretted leaving it so long because when I did visit again I always found out things my other information flows weren’t getting me. 

Some of the things you discover can be excess friction in people’s work due to your processes, people doing things you want them to do but ineffectively because they don’t understand why they are doing it, stalled projects due to some unexpected conditions your project teams aren’t aware of, problematic customer interactions, and many other issues. 

Seeing all of this first hand equips you with the knowledge that you need to make the changes and it gives you fresh motivation to do so having seen the issues you are creating, or not fixing, for people who are otherwise trying their best to fulfill the goals of the organization. 

Another major benefit beyond simply finding out things is to actually engage with and motivate the people who work in all these places. On many occasions when I’ve done this, I’d been one of the few “headquarters” people many of them had met, and often the first senior person on the security team they'd interacted with. Having conversations, individual, group or town-hall style meetings let them all hear my perspective on why security was important. It was usually a revelation to them what the extent of the threat was, the commercial importance of security, and many other factors that they were unaware of despite them being prime players in implementing and sustaining many of our most important controls. When doing all of this you need to be curious about what people’s roles and activities are. It helps if you are genuinely curious, but even if you aren’t then making yourself be is fine as doing this will inevitably trigger your curiosity. The act of finding out as much as you can about the people themselves and their work will in the process of doing that reveal much that you need to know about the situation on the ground. All of these conversations create renewed enthusiasm and engagement and provide a surprisingly large source of great ideas of what could be improved or adjusted. 

In many cases I also got to see lots of other issues that benefited other executives and their work across reliability, efficiency, customer support, and more. 

A Day in the Life Of

Another good way to achieve this is to spend a day doing a job shadow or even doing a role, as best you can, for a day. This might be staffing a few call center calls, helping with some installations, working on a factory floor, supporting customer interactions, and so on. It could be spending a day just being alongside someone else doing their job. I’ve done this occasionally, but not as much as I really should have. I have encouraged my teams in the past to do this and when they have they’ve come back with tons of great ideas. Doing this gets you similar information to site visits but, of course, at a much deeper and visceral level. For example, being on a customer support call and experiencing the glorious least privilege access you’ve put in place and how it interferes with the ability to find the more wide-ranging information needed to deal with their issue, all while watching the clock of expected resolution time, is tough. Not that such access control is wrong, but experiencing situations like this helps you understand the need for processes that more deftly respond to authorized elevations of access for certain use cases. 

Meet Customers 

Many of you, not just in product organizations, are likely compelled to meet customers, so it’s not a huge effort to carve out the time for this. However, these are often meeting customer CISOs and other executives to deal with issues or more usually to represent your company at a more strategic level. For all the reasons just discussed they are also likely living in their own bubble and so most of these interactions don’t give you enough information. So, it’s important to seek out and occasionally meet other people in customer organizations. This might be as simple as staying in the meeting at the back of the room after you’ve done your “executive opener” so you get to hear how the customer’s engineer interacts with and raises issues with your sales engineers. Incidentally, this is one of the important reasons for an effective “Field CISO” organization, to experience customer’s issues and advocate for them back to the product, service and support teams. I’ve covered more about this in a recent post. 

Test and Experience Your Processes and Products 

Another approach is to actually use your own products, especially pre-release or beta products and services. For many organizations with a plethora of products this might be difficult but even just using a few can be a revelation. Try to experience your product's equivalent of “time to hello world” or do some other tasks.

It’s also instructive to exercise support processes like trying to reset your access, reporting a fraud or other issues, filing a vulnerability disclosure (if you can), or experience support and call centers. If you haven’t done this you will inevitably be shocked when you do. If you doubt this then just think about all the terrible experiences you deal with every single day with all the products and services you interact with at a consumer and business level from every other company. Then think, is my organization unique in having none of those issues. Of course it isn’t. 

It’s also important to test out internal products and services to see what your peer teams are experiencing courtesy of you. Push some code, complete a risk assessment, do an access review, ask for help, simulate the experience of being subjected to a vendor assessment etc. Ideally experience these before you inflict them on the wider organization. As above, think of all the painful processes other functions like finance, HR, and compliance inflict on you and then remind yourself that if you’re not careful people might be bucketing your teams’ processes among those. 

Set the Example

Be visible and intentional about all of this to encourage your teams at all levels to do their own version of this. The same goes for your peers and other leaders who might not be doing as much as they should. Sharing trip reports or discussing what you’ve learnt in team or leadership meetings are useful ways of doing this. 

Create Pull Moments 

All this gets you more clarity over the organizational ground-truth to compensate for the natural tendency of your management systems to not get you that. But that doesn’t mean you should try to improve those “systems” to overcome the thermocline of truth. This includes putting in place processes and practices to automatically flag issues and drive their escalation by process rather than relying on the moral courage or diligence of individuals. Some examples of this are:

• Canary milestones. To counter the Green, Green, Green,... .RED pathology where projects are declared good until they’ve totally failed, it’s important to have interim milestones, perhaps unnaturally constructed warning milestones (canaries) that will fail well before the risk of a critical project failure. This can make it unavoidable to spot things going wrong before it's too late. 

• Executive pull questions. Executives and other leaders in the relevant reviews, team discussions, program and product meetings, or just on some other regular cadence, can establish a fixed protocol of questions to pull the truth from the environment, for example: “What has to continue to go right for us to keep to the schedule?”, “Let’s go around the room, everyone needs to name one thing they are most concerned about.” 

• “Go Flight”. The best one, building on this, that I’ve seen in a few organizations is a “Go Flight” process modeled off the NASA Mission Control process. That is, an executive on the lead up to a launch goes around the room (or virtual room) and requires explicit go approvals from people. A similar construct is, thankfully, quite common in design and launch documents in many organizations where explicit approval is required from a selection of leaders and technical domain specific authorities. 

• Rule Driven Risk Acceptance & Re-triggers. To counter the tendency for risk acceptance to be done at too low a level in the organization, establish a formal rule set that frames different levels of risk need to be reviewed and endorsed at specific levels of the organization. You know when you’ve got this right if your rule set has some potential things needing CEO sign-off. It’s also important to manage this collection of risk acceptance as part of a risk inventory or risk register (more on that here). Revisit previously accepted, possibly long term, risks when the executive who accepted them changes role. If a risk is realized you don’t want the current risk owner to be able to legitimately say, “I never personally accepted that risk.”

• Run toward problems. Some organizations are culturally inclined to cultivate leaders at all levels who “run toward problems”. I worked at one organization that was supremely good at this. In fact sometimes leaders were almost competitive to highlight and fix issues even beyond their regular span of control. This was something that was a natural outcrop of its heritage as a partnership where there is an intrinsic shared accountability among leaders. But it also helped that in promotion processes and other rewards people were explicitly graded on whether they did this and whether they “reached for accountability”. 

• Escalation as a service. People are naturally nervous to escalate risks or issues. This is usually because people have cultivated close working relationships with teams and to escalate feels like something done to that team, a bypassing of them or otherwise trying to get them into trouble. I’ve found that if you keep framing escalation as a “service” to leaders and teams then it helps people get over this i.e. I’m going to partner with you to frame this risk issue so we can both go together up your management chain to get some attention to this important issue. Now, as a leader who might be in receipt of escalations it’s massively important to not be shooting messengers. Most thermoclines of truth are created by leaders who exhibit negative behaviors in the face of bad news. 

AI as Truth Seeker 

I’m starting to see many organizations use AI to help with all this. For example, upload all your project status data coupled with your internal service tickets, internal message boards and other channels - as much as can be safely permitted. Then ask if there are discrepancies of reality vs. expectations. Don’t expect a perfectly curated analysis but in some of the output noise is usually 2 or 3 nuggets that can be further explored that reveal insights and issues you would have only found by your own exhaustive analysis or extensive interaction with teams that you most likely would not have had the time to do.  

Bottom line: it’s vital for security, and risk management overall, that leaders ensure they are getting accurate and timely information about projects, issues, risks and that they create a culture, through deliberate action, that can bust through the thermocline of truth. A great way to do this is the personal experience of hitting the road and walking the halls of internal facilities, suppliers and customers. Personally experience your products and services pre- and post-release, not just external but all the things you inflict on your organization as well.