Everyone is deluged with approaches from product and service vendors, small and large. Even vendors struggle to keep track of who their competitors are in an ever crowded market place. There is no way you can evaluate them all, take a meeting with many of them or even read their product white papers. You have to have some way to screen vendors. Here are some criteria you can use that might help with screening - great vendors can pass many, probably not all, of these :
No B.S. Statistics. If the vendor pushes some clearly made up or over-inflated “study” that, for example, cyber-crime or some other thing is now exceeding the GDP of the planet then they can be scrubbed from the list right away.
Investor Expertise. If it is a private company check that the investors (venture capital, private equity) have the expertise to not only help the company grow but are able to stand behind the products and provide in-depth guidance. Note : some people use a negative signal that if the investors don’t use the product then steer clear. Sometimes this is valid, but often not, as the product might be great but just not a fit for the nature of the investing company. For public (and private companies), look for similar expertise and product centricity from multiple of their Board members - a good flag here is whether some Board members have a track record of having been in actual security roles.
Grass Roots Demand. There is demonstrated product demand from the grass roots of the customers (end users, engineers, security team) - genuine buzz, love, or zeal demonstrated by organic take up that then led to an enterprise deal is a great signal. The graveyard of hopes and dreams of vendors and customers alike is full of imposed product use from a top-down executive sale with no team buy-in.
Adjacent Benefits. There are demonstrated adjacent benefits, examples : the security monitoring product can enlighten application teams as to performance or other issues, the authentication product delights customers or reduces friction as well as risk. A great vendor can show real case studies of these.
No Hypothetical Savings. Saving 10 minutes of end-user time, saving 100 help desk calls a day, or reducing false positive alerts by 10% do not add up to actual monetary savings unless you are paying a service provide at piece work rates, or the savings are so extensive that they will trigger actual staff redeployment or future cost avoidance. Such efficiencies are great and should be touted but if the marketing material equates these to actual $ by some casual calculation then the vendor is giving a big signal they don’t think you know how the world really works.
Disdain for Ratings & Awards. The leadership / sales team (I’m assuming the engineers have this as default mode) have a disdain for analyst ratings and industry awards. I’m not naive, I know you need some of these things to pass the RFP filter of some large organizations and so the game needs to be played - but just use them for that.
Engineering Respect. The engineers in your organization and those at the vendor respect each other because of known expertise, research, (good) conference presentations, blogs, past work relationships, or other reputed caliber.
Product Displacement. It can demonstrably displace at least 1 (ideally 2) other products which (see Adjacent Benefits) might not just be security products.
80/20. The product can work - perhaps not perfectly, but good enough - with existing infrastructure and processes, for example: a monitoring product can run ok on existing logs, even though to be awesome it might need some extra collection, or a software analysis product can plug into your existing tooling and work ok even though to be fantastic if might need developers to occasionally step into the product itself.
API Accessible. It is API accessible, amenable to automation and can have its configuration managed like code. They are honest that the GUI (unless the purpose of the product is the GUI) is just for sales demos and for organizations early on the automation journey.
Security Assurance. The product is continuously and rigorously security reviewed by a named (and reputable) independent testing organization as well as their own testing team.
Executive Knowledge. The CEO knows the product, can demo it reasonably well, and knows the key product roadmap decisions that have been made and are in pipeline to be made.
Self-Use. They use the product for themselves and their own CISO can demo their own use - the CISO isn’t purely a customer advocate.
No FUD. The vendor never uses phrases, when initially rebuffed, like….. “well if you don’t care about your security then I can see you wouldn’t want to pilot us”, or “we’ve heard other customers are very unhappy with [competitor]".
Metrics. They are visibly obsessed (almost to the point of being really annoying) by actual customer success metrics and data derived (appropriately) from product instrumentation.
More than a Feature. If the product screams loudly that it is just a feature waiting to be built into products or services you already have, or you can easily write the potential AWS, GCP or Azure “press release” for this feature then you might want to tread carefully.
Technical White Papers. When you ask for a technical white paper you actually get one. (oh, and don’t get me started about filling in forms to get access to the white papers on vendor web sites, hint, if you ever see gjho@gjhrogtoh.com or something like that in your sales lead database, that was me).
But, remember that we are all on the same side. Vendors are full of professional people from engineering to product and sales doing their best to build and deliver a great product, are passionate about their goals and want to help you get the best outcome. Like all of us, sometimes, they are working with imperfect processes in systems that conspire against them and are doing their best to keep improving. People deserve respect and courtesy and if you can find the time to politely coach vendors on why some or all of the above points are important to you then we'll all likely end up better. Bottom line : even applying just a few of these filters will get you to a pre-shortlist of vendors to explore. Although, there’s nothing better than having a clear view of your actual requirements before talking to vendors and using that to guide where you focus. Doing that may make you realize you already have the capability as some feature in some place you already have. It’s in all our interest (investors, customers, vendors) to get better at doing these things to keep improving the market. We are all someone's vendor.