Task guidelines and the CDF questionnaire instrument of security API usability study

Abstract

Research in the field of usable security and privacy suffered in recent times due to the lack of systematic approach for evaluating the usability of security APIs. As a solution to this problem, we proposed a Cognitive Dimensions Framework (CDF) based methodology to evaluate the usability of security APIs [1,2]. Thereafter, we conducted an empirical investigation to evaluate the proposed methodology. In this evaluation, we employed the proposed methodology to evaluate the usability of four security APIs (Java Secure Socket Extension (JSSE) API, Bouncy Castle lightweight crypto API, OWASP Enterprise Security API, Google authentication API). We expected to evaluate the proposed methodology by measuring its thoroughness, validity, effectiveness, and reliability. This data set includes the study material used in this experiment, which includes the CDF based questionnaire and the task guidelines (including code artifacts) used for each component of the study. Results revealed that the proposed methodology identified 80% of the usability issues of each APIs with only seven participant programmers [3]. It was also observed that the validity of the issues identified was between 60% - 80% in all four evaluations [3]. In addition to these findings about the proposed usability evaluation methodology, the results of this study revealed interesting details about usability issues that exist in the four security APIs that were mentioned above [4-7]. 1. Chamila Wijayarathna, Nalin Asanka Gamagedara Arachchilage, and Jill Slay. ``A generic cognitive dimensions questionnaire to evaluate the usability of security APIs." International Conference on Human Aspects of Information Security, Privacy, and Trust. Springer, Cham, 2017. 2. Chamila Wijayarathna, and Nalin Asanka Gamagedara Arachchilage. ``A methodology to evaluate the usability of security APIs." Proceedings of the 9th IEEE International Conference on Information and Automation for Sustainability, 2018. 3. Chamila Wijayarathna, and Nalin Asanka Gamagedara Arachchilage. ``Using cognitive dimensions to evaluate the usability of security APIs: An empirical investigation." Information and Software Technology 115 (2019): 5-19. 4. Chamila Wijayarathna, and Nalin Asanka Gamagedara Arachchilage. ``Why Johnny can’t develop a secure application? A usability analysis of Java Secure Socket Extension API." Computers \& Security 80 (2019): 54-73. 5. Chamila Wijayarathna, and Nalin Asanka Gamagedara Arachchilage. ``Why Johnny can't store passwords securely?: A usability evaluation of Bouncy Castle password hashing." Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018. ACM (2018). 6. Chamila Wijayarathna, and Nalin Asanka Gamagedara Arachchilage. ``Fighting against XSS attacks: A usability evaluation of OWASP ESAPI output encoding." Proceedings of the 52nd Hawaii International Conference on System Sciences (HICSS) (2019). 7. Chamila Wijayarathna, and Nalin Asanka Gamagedara Arachchilage. ``An empirical usability analysis of the Google authentication API." Proceedings of the Evaluation and Assessment on Software Engineering. ACM (2019).