Data Assurance for AI Test and Evaluation
About | Education & Experience | Publications & Presentations | Service | Students & Teaching

Publications

Book Chapters

  1. A. T. Sherman, E. Lanus, M. Liskov, E. Zieglar, R. Chang, E. Golaszewski, R. Wnuk-Fink, C. J. Bonyadi, M. Yaksetig, and I. Blumenfeld, “Formal Methods Analysis of the Secure Remote Password Protocol,” in Logic, Language, and Security, Lecture Notes in Computer Science, Cham: Springer International Publishing, vol. 12300, 2020, pp. 103–126.

Refereed Journal Publications

  1. D. Kauffman, S. Travis, E. Lanus, D. McWilliams, and E. Gurian, "Encouraging Diversity in AI Test and Evaluation," in The ITEA Journal of Test and Evaluation, vol. 46, no. 2, 2025.
  2. E. Lanus, C. J. Colbourn, and Gail-Joon Ahn, "Guaranteeing Anonymity in Attribute-Based Authorization," Journal of Information Security and Applications, vol. 87, 2024.
  3. J. Chandrasekaran, E. Lanus, T. Cody, L. Freeman, R. N. Kacker, M S Raunak, and D. R. Kuhn, "Leveraging Combinatorial Coverage in ML Product Lifecycle," Computer, vol. 57, no. 7, pp. 16-26, 2024.
  4. D. R. Kuhn, M S Raunak, R. N. Kacker, J. Chandrasekaran, E. Lanus, T. Cody, L. Freeman, "Assured Autonomy through Combinatorial Methods," Computer, vol. 57, no. 5, pp. 86-90, 2024.
  5. N. McCarthy, T. Cody, J. Chandrasekaran, E. Lanus, L. Freeman, K. Alexander, and S. Hobson, "Key steps to Fielding Combat Credible AI-Enabled Systems," Naval Engineers Journal, vol. 136, no. 1-2, pp. 237-248, 2024.
  6. J. Chandrasekaran, T. Cody, N. McCarthy, E. Lanus, L. Freeman, and K. Alexander, "Testing Machine Learning: Best Practices for the Life Cycle," Naval Engineers Journal, vol. 136, no. 1-2, pp. 249-264, 2024. *2024 ITEA Publications Award
  7. C. J. Colbourn and E. Lanus, Subspace Restrictions and Affine Composition for Covering Perfect Hash Families, Art of Discrete and Applied Mathematics vol. 1, pp. 1-19, 2018.
  8. C. J. Colbourn, E. Lanus, and K. Sarkar, Asymptotic and Constructive Methods for Covering Perfect Hash Families and Covering Arrays, Designs, Codes and Cryptography vol. 86, pp.907-937, 2018.
  9. E. F. Lanus and E. V. Zieglar, Analysis of a Forced-Latency Defense Against Man-in-the-Middle Attacks, Journal of Information Warfare vol. 16, pp. 66-78, 2017.

Refereed Conference Publications

  1. E. Lanus, D. Wolodkin, and L. J. Freeman, "Hierarchical Scoring for Machine Learning Classifier Error Impact Evaluation," in 2025 International Conference on Machine Learning and Applications (ICMLA), 2025, to appear.
  2. E. Lanus, B. Lee, D. Steberg, J. Chandrasekaran, and L. J. Freeman, "CODEX: Testing Machine Learning with the Coverage of Data Explorer Tool," in 2025 IEEE Conference on Artificial Intelligence Testing (AITest), 2025, pp. 94-101.
  3. J. Chandrasekaran, B. Mayer, H. Frase, E. Lanus, P. Butler, S. Adams, J. Gregersen, N. Ramakrishnan, and L. J. Freeman, "Test and Evaluation of Large Language Models to Support Informed Government Acquisition," in 22nd Annual Acquisition Research Symposium and Innovation Summit, 2025, pp. 1-22.
  4. E. Lanus and L. J. Freeman, "Combinatorial Testing Applications for Artificial Intelligence Metacognition," in 2nd Workshop on Metacognitive Prediction of AI Behavior at SDM (METACOG-25), 2025, Non-archival proceedings.
  5. S. Mhatre, B. Lee, E. Lanus*, and L. J. Freeman, "Coverage Metrics for Detecting Bias in Facial Recognition Datasets," in 2025 IEEE Conference on Artificial Intelligence (CAI), 2025, pp.258-264. (*Corresponding author.)
  6. E. Lanus, B. Lee, J. Chandrasekaran, L. J. Freeman, M S Raunak, R. N. Kacker, and D. R. Kuhn, "Data Frequency Coverage Impact on AI Performance," in 2025 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 2025, pp. 258-267.
  7. J. Chandrasekaran, A. R. Patel, E. Lanus, and L. J. Freeman, "Evaluating Large Language Model Robustness using Combinatorial Testing," in 2025 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 2025, pp. 300-309.
  8. E. Lanus, B. Lee, L. Pol, S. Sobien, J. Kauffman, and L. J. Freeman, "Coverage for Identifying Critical Metadata in Machine Learning Operating Envelopes," in 2024 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 2024, pp. 217- 226.
  9. S. Travis, D. Gracanin, and E. Lanus, "Why Cyber Threat Modeling Needs Human Factors Expansion: A Position Paper," in 2023 Third Intelligent Cybersecurity Conference (ICSC), 2023, pp. 110-118. *Best Paper Award
  10. A. Molina-Markham, S. G. Ionescu, E. Lanus, D. Ng, S. Sommerer, J. J. Rushanan, “Evaluating the robustness of automated driving planners against adversarial influence,” in Workshop on Deception Against Planning Systems and Planning in Adversarial Conditions (DAPSPAC) at the 2022 International Conference on Automated Planning and Scheduling (ICAPS 2022).
  11. T. Cody, E. Lanus, D. D. Doyle, and L. J. Freeman, "Systematic training and testing for machine learning using combinatorial interaction testing," in 2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 102-109.
  12. S. F. Ahamed, P. Aggarwal, S. Shetty, E. Lanus and L. J. Freeman, "ATTL: An Automated Targeted Transfer Learning with Deep Neural Networks," in 2021 IEEE Global Communications Conference (GLOBECOM), pp. 1-7.
  13. S. Wood, E. Lanus, D. D. Doyle, J. Ogorzalek, C. T. Franck and L. J. Freeman, "Developing Hierarchies for Image Classification Model Evaluation," in 2021 4th International Conference on Artificial Intelligence for Industries (AI4I), pp. 34-37.
  14. E. Lanus, I. Hernandez, A. Dachowicz, L. J. Freeman, M. Grande, A. Lang, J. H. Panchal, A. Patrick, and S. Welch, "Test and Evaluation Framework for Multi-Agent Systems of Autonomous Intelligent Agents," in 2021 16th International Conference of System of Systems Engineering (SoSE), pp. 203-209.
  15. E. Lanus, L. J. Freeman, D. R. Kuhn and R. N. Kacker, "Combinatorial Testing Metrics for Machine Learning," in 2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) pp. 81-84.
  16. E. Lanus and C. J. Colbourn, "Algorithms for Constructing Anonymizing Arrays," Proc. of the International Workshop on Combinatorial Algorithms," Bordeaux, France, Lecture Notes in Computer Science, vol. 12126, 2020, pp. 382-394.
  17. R. E. Dougherty, E. Lanus, C. J. Colbourn, and S. Forrest, "Genetic Algorithms for Affine Transformations to Existential t-Restrictions," in Proc. of GECCO, 2019, pp. 1707-1708.
  18. E. Lanus, C. J. Colbourn and D. C. Montgomery, "Partitioned Search with Column Resampling for Locating Array Construction," in 2019 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 214-223.

Abstracts

  1. R. Kuhn, M S Raunak, R. Kacker, J. Chandrasekaran, E. Lanus, T. Cody, and L. Freeman, “Measurements to Improve AI/ML Training Data Sets,” in High Confidence Software and Systems Conference, 2024.
  2. E. Lanus, A. T. Sherman, M. Liskov, E. Zieglar, R. Chang, E. Golaszewski, R. Wnuk-Fink, C. J. Bonyadi, M. Yaksetig, and I. Blumenfeld, “Analysis of the Secure Remote Password Protocol Using CPSA,” in High Confidence Software and Systems Conference, 2020.

Posters

  1. S. Travis, D. Gracanin, and E. Lanus, “Why Cyber Threat Modeling Needs Human Factors,” at Hume National Security Colloquium, Blacksburg, VA, 2023.
  2. L. Pol, B. Lee, A. Thatavarthy, E. Lanus, J. Kauffman, and J. Chandrasekaran, “Combinatorial Testing to Measure Machine Learning Dataset Differences,” at Hume National Security Colloquium, Blacksburg, VA, 2023.
  3. S. Osborn, T. Morrison, and E. Lanus, “To Perturb or not to Perturb: Striking the Balance Between Data Utility and Privacy,” at Hume National Security Colloquium, virtual, 2021.
  4. M. Gonley, M. Tran, M. Lionetti, A. Min, A. Owens, A. Shevalier, A. Vadicherla, J. Jiang, R. Murudkar, S. Kahn, S. Osborn, E. Lanus, and T. Morrison, “Utility-Privacy Tradeoff & the Netflix Challenge,” at Hume National Security Colloquium, virtual, 2021.
  5. E. Lanus and C.J. Colbourn, “Efficient Construction of Intermediate-size Covering Arrays,” at Computing Research Association Committee on the Status of Women in Computing Research (CRA-W) Grad Cohort, San Francisco, CA, 2015.

Other Publications

  1. B. Lee, L. Eisenbeiser, and E. Lanus*, “Training Generative Adversarial Networks on Small Datasets by way of Transfer Learning,” in The ITEA Journal of Test and Evaluation, vol. 44, no. 2, 2023. (*Corresponding author.)
  2. L.J. Freeman, J. Kauffman, D. Sobien, T. Cody, E. Lanus, “Best practices for addressing new challenges in testing and evaluating artificial intelligence-enabled systems,” ITEA Journal of Test and Evaluation, vol. 43, no. 3, pp. 174–180, 2022.
  3. E. Lanus, “Interaction Testing, Fault Location, and Anonymous Attribute-Based Authorization,” Ph.D. dissertation, Computer Science, ASU, Tempe, AZ, 2019.

In Submission

  1. P. Roy, J. Chandrasekaran, E. Lanus, L. Freeman, J. Werner, "A Survey of Data Security: Practices from Cybersecurity and Challenges of Machine Learning."

Talks, Conferences, and Workshops

  1. International Test and Evaluation Association (ITEA) Symposium. Talk: Dataset evaluation for operational testing of artificial intelligence-enabled systems. Destin, FL, 11/2025 (Presented by Laura Freeman).
  2. Carnegie Mellon Software Engineering Institute (CMU SEI) Data Quality Workshop. Talk: Combinatorial Testing for Data Assurance. Virtual, 10/2025.
  3. Systems Engineering Research Center (SERC) Artificial Intelligence for Systems Engineering and Systems Engineering for Artificial Intelligence Workshop. Talk: Tailorable Risk-Informed AI Test and Evaluation Strategy (TRAITES) Framework. Washington DC, 9/2025.
  4. National Defense Industrial Association (NDIA) Emerging Technologies for Defense Conference. Talk: Tailorable Risk-Informed AI Test and Evaluation Strategy (TRAITES) Framework. Washington, DC, 8/2025.
  5. 7th IEEE AITest Conference. Talk: CODEX: Testing Machine Learning with the Coverage of Data Explorer Tool.
  6. NPS 22nd Annual Acquisition Research Symposium and Innovation Summit. Talk: Test and Evaluation of Large Language Models to Support Informed Government Acquisition.
  7. SDM Workshop on Metacognitive Prediction of AI Behavior (METACOG-25). Talk: Combinatorial Testing Applications for Artificial Intelligence Metacognition. Alexandria, VA, 5/1/2025.
  8. Combinatorial Testing for AI-Enabled Systems (CT4AIES) Short Course at Defense and Aerospace Test and Analysis Workshop (DATAWorks). Organizer, instructor. Alexandria, VA 4/2025. Slides and materials available.
  9. IEEE International Workshop on Combinatorial Testing (IWCT). Talk: Data Frequency Coverage Impact on AI Performance. Naples, Italy, 3/2025.
  10. 3rd Annual IEEE Workshop on Assured Autonomy, AI and Machine Learning (WAAM). Invited panelist. Washington, DC, 10/2024.Panel discussions captured in P. Laplante, "Assured Autonomy, Artificial Intelligence, and Machine Learning" in Computer, vol. 58, no. 06, pp. 17-28, June 2025.
  11. Combinatorial Testing for AI-Enabled Systems (CT4AIES). Organizer, presenter. Arlington, VA 9/2024. Slides and materials available.
  12. International Workshop on Combinatorial Testing (IWCT). Talk: Coverage for Identifying Critical Metadata in Machine Learning Operating Envelopes. Toronto, VA, 5/2024.
  13. Graduate Research in AI T&E (GRAITE) Women Workshop . Organizer (PI), presenter. Arlington, VA 1/2024.
  14. 2nd Annual IEEE Workshop on Assured Autonomy, AI and Machine Learning (WAAM). Invited panelist. Atlanta, GA, 11/2023. Panel discussions captured in P. Laplante, J. F. DeFranco and R. Kuhn, "Assured Autonomy, Artificial Intelligence, and Machine Learning: A Roundtable Discussion" in Computer, vol. 57, no. 03, pp. 14-21, March 2024.
  15. Assured Autonomy: The Way Forward. Lightning talk: Data Assurance for Machine Learning. Virtual, 11/2022.
  16. Success Through Rewarding and Inclusive Virtual Experience Mathematics - Opportunities in Research and Education (STRIVE for MORE). Panelist. Virtual, 9/2021
  17. High Confidence Software and Systems Conference (HCSS). Talk: Analysis of the Secure Remote Password Protocol Using CPSA. Maryland, USA, 9/2020. Virtual due to SARS-CoV-2.
  18. 31st International Workshop on Combinatorial Algorithms (IWOCA 2020). Talk: Algorithms for Constructing Anonymizing Arrays. Bordeaux, France, 6/2020. Virtual due to SARS-CoV-2.
  19. Defense and Aerospace Test and Analysis Workshop (DATAWorks). Short Course: Software Testing. Virginia, USA, 3/2020. Cancelled due to SARS-CoV-2.
  20. Fall Technical Conference (FTC). Talk: Tuning Algorithmic Parameters for Locating Array Construction. Maryland, USA, 9/2019.
  21. UMBC Cyber Defense Lab invited talk. Talk: Analysis of the Secure Remote Password (SRP) Protocol Using CPSA. Maryland, USA, 9/2019.
  22. International Workshop on Combinatorial Testing (IWCT). Talk: Partitioned Search with Column Resampling for Locating Array Construction. Xian, China, 4/2019.
  23. Defense and Aerospace Test and Analysis Workshop (DATAWorks). Talk: Constructing Designs for Fault Location. Virginia, USA, 4/2019.
  24. Conference on Combinatorics and its Applications in Celebration of Charlie Colbourn's 65th Birthday. Talk: Covering Perfect Hash Families and Covering Arrays. Singapore, 7/2018.
  25. Protocol eXchange. Talk: Analysis of a Forced-Latency Defense Against Man-in-the-Middle. Maryland, USA, 6/2015.
  26. CRA-W Grad Cohort 2015. Poster: Efficient Construction of Intermediate-size Covering Arrays, San Francisco, USA, 4/2015.