The ICS Detection Challenge took place at S4x18 and S4x19. You can see the on-stage videos below and links to articles describing the contest and the analysis. Also take a look at Dale Peterson’s page analysing the ICS detection market.
Ron Brash of Deloitte
Dale Peterson of S4 Events
Dale Peterson of S4 Events
Ron Brash of Deloitte
Patrick Feeley of Rockwell Automation
Steven Miller of FireEye
David Smith of Schneider Electric
Corey Thuen of Gravwell
We will have additional volunteers on the day of the Challenge to insure the event runs smooth and is fair.
1. Competitors will be emailed the Excel scoresheets at 6AM EST. This is to allow for configuration of the tools to auto-populate the scoresheets. The Asset Inventory spreadsheet will include all of the IP addresses that will be scored.
2. Competitors will be provided the SIEM detail to configure export of information to the SIEM as part of the Detection phase.
3. Competitors will be encouraged to visit the Competition Room to set up their table and test the ability to properly ingest the packet captures (pcaps).
4. Each Competitor will be allowed two people in the Competition Room (no exceptions).
5. The Challenge will take place for eight hours on Tuesday, January 15th at the Shelborne Hotel.
6. There will be no Internet access or external access allowed in the Competition Room. Phones and any other computing devices that are not part of the competitor’s solution cannot be brought into the Competition Room. The two Competitor participants are not allowed to contact anyone to discuss the Challenge. Time outside the Competition Room will restricted to bio-breaks and emergencies. S4 will provide lunch and refreshments throughout the 8-hours of the Challenge on Tuesday. Competitors can bring in their own food / drink as well.
7. The Competitor must identify all software that will be on the computer used in the Challenge prior to the Challenge beginning. The price of any non-Competitor software (external, non-open source software) must be provided by the Competitor. These software lists will be made publicly available when the results are announced.
8. There will be a 15-minute inspection time for each Competitor during the 8-hour competition to validate items 6 & 7.
9. There will be no bonus points in the S4x19 Challenge. Specific questions will be asked, and specific answers expected and scored.
10. There will not be separate competition periods for Asset Inventory and Detection. The multi-hour pcaps will be used for both.
11. The scoresheet format for the Asset Inventory will be provided on or before 15 December 2018 with the points that will be awarded for each answer. We will not be asking for a full inventory. We will be asking for competitors to fill out the Asset Inventory for computers and devices at specific IP addresses. There will be multiple information fields requested, with each field well defined. Examples are device type (select from a list), vendor, model, firmware, cards, function or process supported, …
12. The Detection scoring will consist of three parts:
a. An Excel spreadsheet asking for specific information. The Excel scoresheet format for Detection will be provided on or before 31 December 2018.
b. The ability to send specific information to a SIEM.
c. The identification, visualization and explanation of a specific cyber incident. This will be the one part of the Challenge that will have subjective scoring, and it will comprise only 5% of the Detection score. Each team will have 15 minutes to describe the cyber incident and show how their solution identified the issue and provided information to an analyst that would assist in detection and response. We will video the 15 minutes for possible use on the S4 Main Stage and the S4 Events YouTube Channel.
13. The Judges will score the contest on Wednesday, January 16th.
14. Each competitor will receive a score on Asset Inventory, Detection and Overall, with Overall simply be an addition of Asset Inventory and Detection scores.
15. The Judges will create three tiers of Competitors (Top, Middle, Bottom) in Asset Inventory and in Detection, see diagram below. This chart will be the primary way that the Challenge will present the results. This is to recognize that the scoring between Competitors may be small or large, and small point differences should be viewed as a tie given the limitations of a Challenge like this. Each tier could have 0 to 8 members, and our expectation is the tiering decision will be quite easy. Think of it as a different approach, but similar goal to a “magic quadrant”.
16. The Judges will set a 15-minute appointment with each Competitor on Wednesday between 11AM and 1PM to allow the Competitor to address any Judge’s questions on the Competitor’s answers.
17. The four top performing teams that will be interviewed for 10 minutes on the S4 Main Stage will be notified on or before 8PM. This notification will include the primary topics that will be covered in the brief interview.
18. Competitors will be provided access to their scoresheets at least one hour prior to the S4 Main Stage ICS Detection Challenge session, currently scheduled for 11AM. The scoresheets will not be allowed to be taken from the Competition Room for data protection reasons.
19. There will be a one-hour ICS Detection Challenge session on the S4 Main Stage on Thursday. Four of the teams will be invited on stage for an interview. This will be the top two in Asset Inventory and the top two in Detection. If this results in less than four teams due to finishing top two in more than one category, then the remaining spot(s) will be filled by the teams with the highest combined score. Each of the four teams will be interviewed on the area where they performed the best or had some unique insights. We will likely show some video or screen shots from the cyber incident in 12.c.
The pcap files are being anonymized, but even with this precaution the asset owner contributor insists that all packet captures (pcaps) be deleted by the competitors prior to leaving the Competition Room. All work product and all details related to the pcaps must be deleted by 5PM EST on Thursday, January 17th. Competitors will be required to sign a Confidentiality Agreement with these terms and other terms to protect the asset owner contributor.
We have been able to get larger pcaps, 100G+, from multiple ICS at a very large site. Each ICS has many hours of real-world data. There are a much wider variety of systems than the S4x18 Challenge. We also have a corresponding detailed asset inventory for those systems, which we did not have last year. Most of the packets are from large, well known and widely deployed ICS vendors and are using popular, widely used protocols. There are some less well-known software, devices and protocols that will be a test of the breadth of support in the Competitor solutions.