The S4x25 Vulnerability Management Pavilion
Challenges Comparing Products
If you’ve ever been in the market for an OT vulnerability management product you likely have seen many vendor demos. Not surprisingly every demo shows an accurate asset inventory and list of important vulnerabilities. They have fine tuned the demo to show the product in the best light, like they should. How would the product perform in the real world?
The other challenge is each vendor has their own ICS, or simulated ICS, they are using for the demo. There is no way to compare products apples to apples in demos. You would need to either bring the products in for bake-off on one of your systems or send the vendors pcap files. (I’ve seen a lot of expert massaging and other shenanigans with vendor’s running pcap files to show how their product would work).
Vulnerability Detection Product Comparison At S4x25
We tried to provide this comparison in the S4x24 Vulnerability Management Pavilion with partial success. We’ve taken the lessons learned and are bringing the Pavilion back at S4x25. Here’s the differences and how it will work:
- The ICS Village will bring and run a single, highly realistic ICS in the Vulnerability Management Pavilion.
The great team at the ICS Village brought down a large number of devices and applications to S4x24. For example, there were about 40 different types of PLC / Level 1 devices. The goal was breadth. Let’s see how many different systems the asset inventory / vulnerability management products could identify and analyze.
At S4x25 we are tasking the ICS Village to bring down an ICS that is similar to what you would see at an asset owner’s single site or system … a water treatment plant, manufacturing plant, pipeline, etc. The reasons for this will be clear below.
- The ICS Village will have a complete asset inventory.
The ICS Village will have a variety of notecards available for Pavilion visitors that have a few assets with asset detail that the visitor can carry around and see how each solution did at identifying that asset.
Vendors will be able to submit their full asset inventory for comparison to actual asset inventory (prior to it being revealed). The ICS Village will also simulate maintenance and small changes during a “planned outage”.
- The ICS Village will have a “complete” list of vulnerabilities in the asset inventory.
This information will also be on the notecards for Pavilion visitors. As you walk around the Pavilion you can see: did the vendor identify the asset? did the vendor identify vulnerabilities in the asset? how did the vendor risk rate / categorize the vulnerability? what remediation recommendation did the vendor make?
Vendors will be able to submit their vulnerability list with the full asset inventory for comparison to the ICS Village’s master list. Of course I expect that the vendors may find some additional vulnerabilities.
- The Vulnerability Management Pavilion will allow you to compare how the full set of asset inventory and vulnerability information is presented.
The previous two bullets are about accuracy. This bullet is about how useful the product is and what it tells you to do.
My expectation is the vendors will be close in creating asset inventories and identifying vulnerabilities, but I could be surprised. It’s the presentation of this information and the recommendations that I find to be the key decision criteria for this product category. With all the products running off the same ICS you should be able to pick what you like best.