ISTQB AI Testing Certification

1. Introduction to artificial intelligence

The certification explains fundamental AI concepts and how AI-based systems differ from conventional software.

Topics include:

• Artificial intelligence
• Narrow AI and other classifications of AI
• Machine learning
• Deep learning
• Generative AI
• Large language models
• AI technologies and system components
• Hardware and infrastructure considerations
• AI frameworks
• Regulations and standards affecting AI systems

This section provides the terminology needed to understand the remainder of the syllabus.

It also establishes an important testing principle: the behavior of an AI-based system may depend on data and statistical relationships rather than only on explicitly programmed rules.

2. Quality characteristics for AI-based systems

AI systems create quality concerns that may require different test objectives and acceptance criteria.

The syllabus addresses AI-specific quality characteristics and how they affect testing.

The official CT-AI v2.0 quality characteristics are:

• AI functional correctness
• Functional adaptability
• User controllability
• Transparency
• AI robustness
• Intervenability
• Societal and ethical risk mitigation

The syllabus also addresses the special considerations involved when AI is used in safety-related systems. Fairness, privacy, security, accountability and human control are considered within these characteristics and the broader testing and regulatory context.

Not every characteristic applies equally to every AI system. The relevant characteristics depend on the purpose, risk, users and operating environment of the system.

The certification also covers acceptance criteria for AI-based systems. These criteria may need to include statistical thresholds, acceptable error rates and limits on model behavior rather than a simple expectation that every input will always produce one exact output.

3. Machine learning concepts

Machine learning systems learn patterns from data instead of relying only on manually programmed rules.

The syllabus introduces:

• Different forms of machine learning
• The machine learning workflow
• Training, validation and test data
• Data preparation
• Pretrained models
• Fine-tuning
• Retrieval-augmented generation
• Machine learning model development
• Model deployment

Testers need to understand these concepts because defects can enter an AI system through its data, model, configuration, integration or operating environment.

A system can appear technically correct while still producing poor results because the training data was incomplete, biased, mislabeled or unrepresentative.

4. Data for machine learning

Data is a central part of an AI system and therefore a central part of AI testing.

The certification covers:

• Training data
• Validation data
• Test data
• Data quality
• Data preparation
• Data representativeness
• Data bias
• Data imbalance
• Missing or incorrect data
• Data labels
• Data constraints
• Data pipelines
• Data privacy and security

The test data used to assess a model should be separate from the data used to train it. Otherwise, the evaluation may give an overly favorable view of model performance.

Testers also need to determine whether the data represents the users, situations and conditions the system will encounter after deployment.

5. Machine learning performance metrics

A traditional test may have a clear pass or fail result. Machine learning systems often require statistical measures.

The syllabus includes functional performance metrics for classification models and the use of a confusion matrix.

Relevant measures include:

• True positives
• True negatives
• False positives
• False negatives
• Accuracy
• Precision
• Recall
• Other measures derived from classification results

The right metric depends on the business and safety consequences of different errors.

For example, a false negative may be more serious than a false positive in one system, while the opposite may be true in another. A high overall accuracy figure can therefore be misleading when considered by itself.

6. Neural networks

The syllabus introduces simple neural networks and the testing issues associated with them.

Testers are not expected to become neural network researchers. They do need enough understanding to recognize how neural network behavior is created and why traditional source-code coverage alone may not adequately describe test thoroughness.

The certification includes concepts related to:

• Neural network structure
• Inputs and outputs
• Nodes and layers
• Training
• Testing neural network behavior
• Coverage approaches for neural networks

7. Testing AI-based systems

The certification explains why AI-based systems create different testing challenges.

These include:

• Probabilistic behavior
• Non-deterministic output
• Dependence on data
• Difficulty defining a test oracle
• Model changes
• Environmental changes
• Statistical acceptance criteria
• Locked and adaptive systems
• The interaction between AI and conventional components

A test oracle determines the expected result of a test. Defining one can be difficult when several outputs may be acceptable or when a precise correct answer is not available.

Testing may therefore require:

• Statistical evaluation
• Multiple executions
• Tolerance ranges
• Comparison against another system or model
• Metamorphic testing
• Expert review
• User evaluation
• Monitoring after deployment

8. Testing generative AI and large language models

The current AI Testing syllabus includes testing generative AI systems and large language models.

Testing may need to evaluate:

• Accuracy
• Relevance
• Completeness
• Hallucinations
• Bias
• Safety
• Security
• Privacy
• Robustness
• Consistency
• The handling of unsuitable or unexpected inputs
• The effectiveness of system controls
• Human oversight

The certification distinguishes testing a generative AI system from using generative AI as a testing assistant.

Testing the system itself belongs primarily to AI Testing. Using generative AI to help perform software testing belongs primarily to Testing with Generative AI.

9. Input data testing

Input data testing examines the information used to train, validate, test or operate a machine learning system.

The syllabus addresses risks such as:

• Unrepresentative data
• Missing data
• Incorrect values
• Duplicate data
• Labeling problems
• Bias
• Imbalanced classes
• Data leakage caused by unintended overlap or insufficient independence between training, validation and test datasets
• Data pipeline defects
• Values outside expected constraints
• Differences between development data and production data

Input data testing is important because a model cannot be evaluated separately from the data on which it depends.

10. Model testing

Model testing evaluates whether the trained model performs appropriately for its intended purpose.

The syllabus includes topics such as:

• Model performance
• Probabilistic results
• Robustness
• Adversarial inputs
• Metamorphic testing
• Model drift
• Overfitting
• Underfitting
• A/B testing
• Back-to-back testing
• Model documentation
• Performance thresholds

Overfitting occurs when a model performs well on the data used during development but does not generalize well to new data.

Underfitting occurs when the model has not learned the relevant patterns sufficiently and performs poorly even on the intended problem.

Model drift occurs when model performance changes because the data, users, environment or relationships in the real world have changed.

11. Machine learning development testing

The syllabus also addresses testing during the development and deployment of a machine learning system.

Testing should not be delayed until the model is fully developed.

Relevant activities may include:

• Testing data pipelines
• Testing feature engineering
• Testing model training
• Testing model interfaces
• Testing integration with conventional software
• Testing deployment
• Testing monitoring mechanisms
• Checking configuration and infrastructure
• Evaluating changes to models or data
• Detecting performance degradation after release

This lifecycle approach helps teams identify problems in the data, model and surrounding system before they reach production.