ISTQB Testing with Generative AI Certification

1. Generative AI foundations for software testing

The certification begins by explaining the technologies and terminology behind generative AI.

Topics include:

• The relationship between symbolic AI, machine learning, deep learning and generative AI
• Large language models
• Foundation models
• Instruction-tuned models
• Reasoning models
• Tokens and tokenization
• Context windows
• Multimodal models
• Vision-language models
• The capabilities and limitations of generative AI
• The difference between a general AI chatbot and an LLM-powered testing application

This foundation helps testers understand why an AI system may produce useful answers in one situation but unreliable or incomplete answers in another.

It also helps testers avoid treating generative AI as a source that is automatically correct.

2. Prompt engineering for software testing

Prompt engineering is a major part of the certification.

The syllabus covers how to create prompts that provide the model with enough information to produce a useful result. A structured prompt may include:

• The role the AI should perform
• Relevant context
• A specific instruction
• Input data
• Constraints
• The required output format

The certification also addresses prompting techniques and the differences between system prompts and user prompts.

Testers learn how prompting can support activities such as:

• Reviewing requirements
• Identifying ambiguities and missing information
• Generating test conditions
• Creating test cases
• Suggesting test data
• Supporting test automation
• Reviewing automated test results
• Summarizing test information
• Assisting with test monitoring and reporting

The purpose is not simply to produce more content. The tester must determine whether the output is accurate, relevant, complete and appropriate for the testing objective.

3. Evaluating AI-generated results

A generative AI response should not be accepted only because it sounds convincing.

The certification covers ways to evaluate AI-generated results and refine prompts when the initial output is weak.

A tester may need to check whether the response:

• Addresses the actual testing objective
• Uses the supplied context correctly
• Includes unsupported assumptions
• Omits important test conditions
• Produces duplicate or low-value test cases
• Conflicts with the requirements
• Contains technically incorrect information
• Uses the requested output format
• Can be reproduced consistently enough for the intended use

Prompt refinement is an iterative activity. A tester may change the context, constraints, examples or requested format and then evaluate the revised result.

4. Managing the risks of generative AI in testing

Generative AI can help testers, but it also introduces risks that require review and oversight.

The syllabus addresses:

• Hallucinations
• Reasoning errors
• Bias
• Non-deterministic behavior
• Data privacy
• Information security
• Inappropriate disclosure of confidential information
• Regulatory requirements
• Standards and responsible-use frameworks
• Energy use and environmental impact

These topics matter because a polished response can still be wrong. An AI system can invent facts, misunderstand requirements, repeat bias found in its data or produce different answers when given similar prompts.

Testers and organizations therefore need controls for:

• Deciding which information may be entered into an AI system
• Reviewing AI-generated output
• Protecting confidential and personal information
• Tracking how generative AI is used
• Maintaining human responsibility for testing decisions
• Determining when generative AI should not be used

The certification does not teach testers to rely on AI without review. It teaches them to use it with appropriate evaluation, risk management and human oversight.

5. LLM-powered testing solutions

The syllabus goes beyond entering prompts into a general chatbot.

It also covers concepts used to develop more structured LLM-powered testing solutions, including:

• Architectural approaches for LLM-powered testing
• Retrieval-augmented generation
• LLM-powered agents
• Fine-tuning
• LLMOps
• Connecting an LLM to approved organizational information
• Operating and maintaining generative AI solutions

Retrieval-augmented generation can allow a solution to retrieve relevant information from an approved source before generating its response. This can improve relevance, but the resulting output still requires testing and evaluation.

LLM-powered agents may perform sequences of actions or interact with other systems. Their additional autonomy also creates additional testing, security and oversight concerns.

6. Introducing generative AI into a test organization

The certification also addresses organizational adoption.

Topics include:

• Shadow AI
• Defining a generative AI strategy
• Selecting appropriate models
• Comparing large and smaller language models
• Creating an adoption roadmap
• Building employee skills
• Managing organizational change
• Updating test processes
• Establishing appropriate controls and responsibilities

Shadow AI occurs when employees use AI tools without the organization’s formal approval or oversight. This can create privacy, security, quality and compliance risks.

A planned approach helps an organization decide where generative AI provides value, what information may be used, how output will be reviewed and who remains accountable for the work.