AI Test Automation

This feature enables you to efficiently test one or several configurations of your generative AI setup.

Instead of manually typing individual questions into the chat, you can upload a list of test questions (as a CSV or XLSX file). For each question, an answer will be automatically generated by the AI process as configured in your process model. This includes the GPT Gateway, all GPT knowledge sources, and other relevant modules. After a processing time (typically around 30 minutes, depending on the number of questions), you can download the generated answers as a CSV file. This allows you to easily evaluate your AI's performance, especially after making changes to its configuration (e.g., updated modules, prompts, system messages, knowledge sources, or LLMs).

Since the AI tests are specific to the process, you can only test the AI of a particular process/agent. To upload test questions and test the AI, navigate to the Knowledge tab of the process and click on the AI Test Automation button.

Upload file format

The file for uploading questions should be a CSV or XLSX file with the following columns:

• question: The question that should be answered by the AI.
• locale: The locale (language) in which the question should be answered. This is important for the language model to generate the correct answer. The locale should be in the format en for English, de for German, etc.
  • Please note: Only locales that are configured and active in the agent (process) are supported.

Example file content:

question	locale
What is the capital of France?	en
Was ist die Hauptstadt von Deutschland?	de
Suche ein Produkt mit einer Bewertung über 4 Sternen.	de
¿Cuál es el horario de atención al cliente?	es