6 Results and Discussion
6.1 Results
All blue teams participated in seven engagements throughout the STAC program.
- Engagement 1 (E1) was a live engagement that took place in February 2016 at DARPA.
- Engagement 2 (E2) was a take home version of Engagement 1 which ended in July 2016.
- Engagement 3 (E3) was a live engagement that took place in January 2017.
- Engagement 4 (E4) was a take home version of Engagement 3 which ended in June 2017.
- Engagement 5 (E5) had a home component that ended in January 2018 and a live collaborative engagement at DARPA in February 2018.
- Engagement 6 (E6) had a home component and a live collaborative engagement at DARPA in August 2018.
- Engagement 7 (E7) had a home component and a live collaborative engagement at DARPA in February 2019.
A summary of the ISU-ENSOFT team results is shown in the figure below. Results are shown for the end engagement for each set of challenge applications. For example, E1 and E2 were based on the same challenge applications; and so were E3 and E4. E5, E6 and E7 each had a home component, followed by the live one and the results after the live engagement are shown below.
6.2 Engagement Observations
Figure 14 shows that our accuracy continued to increase for each engagement (except the last one). Up to the last engagement E7, teams could choose to analyze only a subset of the applications questions. (The attempted column reflects that). As of E7, all teams were required to answer all the questions. That is the reason the accuracy has dipped a little for E7.
7. Conclusions
The STAC program was quite challenging. While the program was restricted to algorithmic complexity (AC) and side channel (SC) as the two classes of vulnerabilities, they involve almost endless variations of triggers and malicious payloads. Coming up with analyses to detect these vulnerabilities is extremely challenging because of these variations. It often entails complex analyses involving loops. Existing techniques from the large body of research on loop analyses were not particularly useful for addressing problems from the engagements, and a new approach was needed.
The STAC program enabled significant advances. The foremost advance is the intelligence amplifying (IA) technology for detecting AC and SC vulnerabilities. Fredrick Brooks, recipient of Turing Award and the US National Medal of Technology, writes “If indeed our objective is to build computer systems that solve very challenging problems, my thesis is that IA > AI, that is, that intelligence amplifying systems can, at any given level of available systems technology, beat AI systems. That is, a machine and a mind can beat a mind-imitating machine working by itself.” Complex software engineering problems cannot be solved by AI, it calls for IA technology. An IA automation technology for addressing complex software problems is presented in the report.
Several challenges were encountered in coming up with an effective way to combine automated analysis with human ingenuity. This report discussed these challenges and how to address them. The advances in the IA technology lead to steady improvement in the performance on the engagements. The program manager recommended the IA technology for potential adoption.
The take-home and on-site engagements very useful to drive the research. The engagements enabled identification of the weaknesses in the approach and provided concrete directions to refine it. The collaborative part added to the engagements was especially useful. Understanding the approaches taken by other teams was helpful and all teams benefitted by learning from each other.