* Knowledge of computer networking concepts and protocols, and network security methodologies.

* Knowledge of risk management processes (e.g., methods for assessing and mitigating risk).

Skill in determining an appropriate level of test rigor for a given system.

Determine level of assurance of developed capabilities based on test results.

Develop test plans to address specifications and requirements.

Make recommendations based on test results.

Test, evaluate, and verify hardware and/or software to determine compliance with defined specifications and requirements.

Record and manage test data.

* Knowledge of national and international laws, regulations, policies, and ethics as they relate to cybersecurity.

* Knowledge of cybersecurity principles.

* Knowledge of cyber threats and vulnerabilities.

Conduct hypothesis testing using statistical processes.

Assess technical risks and limitations of planned tests on AI systems.

Build assurance cases for AI systems that support the needs of different stakeholders (e.g., acquisition community, commanders, and operators).

Conduct AI risk assessments to ensure models and/or other solutions are performing as designed.

Create or customize existing Test and Evaluation Master Plans (TEMPs) for AI systems.

Determine methods and metrics for quantitative and qualitative measurement of AI risks so that sensitivity, specificity, likelihood, confidence levels, and other metrics are identified, documented, and applied.

Develop machine learning code testing and validation procedures.

Develop possible solutions for technical risks and limitations of planned tests on AI solutions.

Maintain current knowledge of advancements in DoD AI Ethical Principles and Responsible AI.

Measure the effectiveness, security, robustness, and trustworthiness of AI tools.

Provide quality assurance of AI products throughout their lifecycle.

Report test and evaluation deficiencies and possible solutions to appropriate personnel.

Select and use the appropriate models and prediction methods for evaluating AI performance.

Test AI tools against adversarial attacks in operationally realistic environments.

Test components to ensure they work as intended in a variety of scenarios for all aspects of the AI application.

Test how users interact with AI solutions.

Test the reliability, functionality, security, and compatibility of AI tools within systems.

Test the trustworthiness of AI solutions.

Use models and other methods for evaluating AI performance.

Ability to collect, verify, and validate test data.

Ability to translate data and test results into evaluative conclusions.

Knowledge of machine learning theory and principles.

Skill in assessing the predictive power and subsequent generalizability of a model.

Skill in preparing Test & Evaluation reports.

Skill in providing Test & Evaluation resource estimate.

* Knowledge of specific operational impacts of cybersecurity lapses.

* Knowledge of cloud computing service models Software as a Service (SaaS), Infrastructure as a Service (IaaS), and Platform as a Service (PaaS).

* Knowledge of cloud computing deployment models in private, public, and hybrid environment and the difference between on-premises and off-premises environments.

Knowledge of AI security risks, threats, and vulnerabilities and potential risk mitigation solutions.

Knowledge of AI Test & Evaluation frameworks.

Knowledge of best practices from industry and academia in test design activities for verification and validation of AI and machine learning systems.

Knowledge of coding and scripting in languages that support AI development and use.

Knowledge of DoD AI Ethical Principles (e.g., responsible, equitable, traceable, reliable, and governable).

Knowledge of how AI is developed and operated.

Knowledge of how AI solutions integrate with cloud or other IT infrastructure.

Knowledge of how to automate development, testing, security, and deployment of AI/machine learning-enabled software to the DoD.

Knowledge of how to collect, store, and monitor data.

Knowledge of how to deploy test infrastructures with AI systems.

Knowledge of interactions and integration of DataOps, MLOps, and DevSecOps in AI.

Knowledge of laws, regulations, and policies related to AI, data security/privacy, and use of publicly procured data for government.

Knowledge of machine learning operations (MLOps) processes and best practices.

Knowledge of metrics to evaluate the effectiveness of machine learning models.

Knowledge of remedies against unintended bias in AI solutions.

Knowledge of testing, evaluation, validation, and verification (T&E V&V) tools and procedures to ensure systems are working as intended.

Knowledge of the AI lifecycle.

Knowledge of the benefits and limitations of AI capabilities.

Knowledge of the possible impacts of machine learning blind spots and edge cases.

Knowledge of the user experience (e.g., decision making, user design, and human-computer interaction) as it relates to AI systems.

Knowledge of tools for testing the robustness and resilience of AI products and solutions.

Skill in explaining AI concepts and terminology.

Skill in identifying low-probability, high-impact risks in machine learning training data sets.

Skill in identifying risk over the lifespan of an AI solution.

Skill in integrating AI Test & Evaluation frameworks into test strategies for specific projects.

Skill in testing and evaluating machine learning algorithms or AI solutions.

Skill in testing for bias in data sets and AI system outputs as well as determining historically or often underrepresented and marginalized groups are properly represented in the training, testing, and validation data sets and AI system outputs.

Skill in translating operation requirements for AI systems into testing requirements.

Knowledge of organization’s evaluation and validation requirements.

Perform AI architecture security reviews, identify gaps, and develop a risk management plan to address issues.

Knowledge of the organization’s core business/mission processes.

Knowledge of service management concepts for networks and related standards (e.g., Information Technology Infrastructure Library, current version [ITIL]).

Assist integrated project teams to identify, curate, and manage data.

Identify and submit exemplary AI use cases, best practices, failure modes, and risk mitigation strategies, including after-action reports.

Knowledge of current test standards and safety standards that are applicable to AI (e.g. MIL-STD 882E, DO-178C, ISO26262).

Knowledge of Personal Health Information (PHI), Personally Identifiable Information (PII), and other data privacy and data reusability considerations for AI solutions.