Evaluating Security Techniques using Mutation Testing

This project evaluates the practical implications of the decades-old tradeoff between soundness and precision. The analysis is sound if it identifies every instance of the target behavior (e.g., SSL use vulnerabilities or data leaks) in the application being analyzed. Further, the analysis is precise if every instance of the target behavior that it has identified is true (i.e., if there are no false positives).

Intuitively, there is a conflict between these two properties: sound analysis requires the technique to over-approximate (i.e., consider instances of target behaviors that may not be executed in reality, which deteriorates precision). Static security analysis techniques used for analyzing applications are not exempt from this paradox, as while they may be theoretically sound , they often make unsound choices in practice. Such techniques can be termed as soundy, i.e., consisting of a core set of sound security decisions accompanied by unsound choices with unknown implications.

To address this fundamental problem, we propose mutation-based Soundness Evaluation (μSE, read as ”muse”) of Android static analysis techniques, which aims at enabling evaluation of Android static analysis tools by relying on the technique of mutation analysis in software engineering.

Grants: SaTC: CORE: Small: Enabling Systematic Evaluation of the Soundness of Android Security Analysis Techniques


Why Crypto-detectors Fail: A Systematic Evaluation of Cryptographic Misuse Detection Techniques
Amit Seal Ami, Nathan Cooper, Kaushal Kafle, Kevin Moran, Denys Poshyvanyk, and Adwait Nadkarni
IEEE S&P 2022

Demo: Mutation-based Evaluation of Security-focused Static Analysis Tools for Android.
Amit Seal Ami, Kaushal Kafle, Kevin Moran, Adwait Nadkarni, and Denys Poshyvanyk
ICSE 2021, Demonstrations track

Systematic Mutation-based Evaluation of the Soundness of Security-focused Android Static Analysis Techniques
Amit Seal Ami, Kaushal Kafle, Kevin Moran, Adwait Nadkarni, and Denys Poshyvanyk

Discovering Flaws in Security-Focused Static Analysis Tools for Android using Systematic Mutation
Richard Bonett, Kaushal Kafle, Kevin Moran, Adwait Nadkarni, and Denys Poshyvanyk
USENIX Security 2018

Practical Characterization and Security Analysis of Home Automation

Home automation platforms such as Samsung SmartThings or Google Nest represent the next step in the evolution of operating systems. Their general security architectures adopt core design elements from smartphone OSes, such as providing APIs to integrate third-party applications or apps, and using permissions to constrain the damage from malicious/misconfigured apps.

However, three central characteristics of home automation result in novel challenges for secure OS design, as well as security, safety, and privacy analyses:

  1. user-driven routines, i.e., automation directly configured by end-users, without the help of software developers,
  2. device-type characteristics, i.e., the non-obvious physical limitations and capabilities of individual devices that must be accounted for in security analyses or defenses (e.g., fire sprinklers cannot be turned ON via apps, as they are triggered through heat), and,
  3. runtime limitations, i.e., the conventions and limitations imposed by smart home platform runtime environments, which may impact the validity of assumptions made for analysis.

Our goal is to develop a realistic and long-lasting characterization of the automated home by demystifying each of the three factors, and hence lay the groundwork for building practical analysis frameworks and defenses for home automation. We expect such a characterization to be instrumental for security analyses in years to come, as operating systems evolve further to increasingly interact with users their physical environments in smart homes, communities, and cities.

Grants: Collaborative Research: CPS: Medium: Enabling Data-Driven Security and Safety Analyses for Cyber-Physical Systems


Security in Centralized Data Store-based Home Automation Platforms: A Systematic Analysis of Nest and Hue
Kaushal Kafle, Kevin Moran, Sunil Manandhar, Adwait Nadkarni, and Denys Poshyvanyk

Towards a Natural Perspective of Smart Homes for Practical Security and Safety Analyses
Sunil Manandhar, Kevin Moran, Kaushal Kafle, Ruhao Tang, Denys Poshyvanyk, and Adwait Nadkarni
IEEE S&P (Oakland) 2020

A Study of Data Store-based Home Automation
Kaushal Kafle, Kevin Moran, Sunil Manandhar, Adwait Nadkarni, and Denys Poshyvanyk
ACM CODASPY 2019 (Best Paper Award)