Testing Studio Structure
Testing Studio has the following directory structure:
/tests
: directory where test suites are saved/tests/.omilia-ci.yml
: Testing Studio configuration file/tests/example_test_suite/
: directory where logically grouped test cases are saved./tests/example_test_suite/test_cases.yml
: file containing test cases./tests/example_test_suite/scenarios.yml
: file containing user defined test cases/tests/example_test_suite/golden_dialogs/
: directory where golden dialogs are saved/tests/example_test_suite/golden_dialogs/dialog.json
: golden dialog file in JSON format downloaded from OmDRTViewer/tests/example_test_suite/global_properties.yml
": file containing property values that are set globally, for all test cases
View the following .zip file to check the Testing Studio structure and download it as a template:
Testing Studio Configuration File
The omilia-ci.yml configuration file is a readable self-explanatory YML file with the following content:
.omilia-ci.yml
diamant:
app: abc_uat@SND_ABC_Root
miniapp_id: 034a0a36-6df3-489c-b99a-2fb92942c5f5.SND_ABC_Root.Flow.abc_uat@SND_ABC_Root.abc_uat
assertions:
prompt: true
intent: true
end_type: true
grammar: true
annotation: true
action_name: true
target_name: true
dialog_events: true
interpretation: true
destination_uri: true
outbound_attached_data: true
prompt_threshold: 0.8
Below you can find the properties description of omilia-ci.yml:
Section | Property | Data Type | Value | Description |
---|---|---|---|---|
| The property is mandatory and must be configured | |||
| string |
| For example if the group name is abc_uat and the application name is ABC-Root the property value will be | |
| Information related to the application under test | |||
| string |
| ID of the miniApp application/flow that we want to test. This information can be found under Console →miniApps → General, and is available for copying. | |
| This section contains all the available assertions. Depending on the testing level, other assertion strategies can be configured | |||
| boolean | true/false | The prompt of the dialog step | |
| boolean | true/false | The end type of the dialog | |
| boolean | true/false | Events of the dialog | |
| boolean | true/false | Transfer line of the dialog | |
| boolean | true/false | Destination URI of the dialog | |
| boolean | true/false | Intent of the dialog step | |
| boolean | true/false | Grammar of the dialog step | |
| boolean | false | Annotation of the dialog step | |
| boolean | false | Interpretation of the dialog step | |
| boolean | true/false | Action name of the dialog step | |
| boolean | true/false | Target name of the dialog step | |
| boolean | true/false | Attached data of the dialog step | |
| integer | 0 to 1 | The lowest limit for the similarity of prompts |
Test Suite Folder
The test suite directory contains the following file types:
Below you can find a detailed description of each file type.
dialog.json File
The dialog.json file is the actual dialog that the Testing Studio is going to reproduce and validate.
In order to be used, the dialog.json files should be placed under golden_dialogs
folder in every Test suite
folder.
Make sure that all fields are snake_case. Any other format (for example, camelCase) is incompatible, and the tool will fail on startup.
dialog.json
{
"total_steps": 1,
"duration": ,
"dialog_id": "",
"app_name": "",
"origin_uri": "",
"destination_uri": "",
"inbound_attached_data": "",
"dialog_events":,
"end_type": "",
"transfer_line": "",
"dialog_date": "",
"dialog_steps": [
{
"step_num": 1,
"confidence":,
"action_name": "",
"target_name": "",
"utterance": "",
"interpretation": "[]",
"annotation": "[]",
"grammar": "",
"locale": "",
"prompt": "",
"comment": "",
"intent": "",
"outbound_attached_data": "",
"encryption": "",
"step_time": ""
}
],
"audios": [],
"metadata": {
"app_version": "",
"diamant_version": "9.6.0",
"encryption": ""
}
}
test_case.json/yml File
Testing Studio parses test case files in order to execute an actual dialog. Test case file has a one-to-one reference with a dialog.json file along with any test/simulation data required for the specific scenario.
Testing Studio supports test case files of two types: JSON and YML.
test_case.json
{
"id":"101",
"description":"Sample test case",
"group": "a group",
"simulation_data": {
"initial":{
"Ani":"124",
"Dnis":"12345"
}
},
"golden_dialog": "dialog_1.json",
"tries": 5,
"utterances": [
{"utterance": "hello"},
{"utterance": "what is my balance?","modality": "speech"},
{"utterance": "one two three"}
],
"dialog_events_rules": {
"assert": {
"exp": [
"Intent%",
"OQ_Welcome"
],
"operator": "and"
}
},
"session_channel": "web-chat",
"handle_delays": false
}
test_case.yml
id: 101
description: Sample test case
group: a group
simulation_data:
initial:
Ani: 124
Dnis: 12345
golden_dialog: dialog_1.json
tries: 5
utterances:
- utterance: hello
- utterance: what is my balance?
modality: speech
simulation_data:
bio_data:
BioScore: TRUE_HIGH
- utterance: one two three
dialog_events_rules:
assert:
exp:
- Intent%
- OQ_Welcome
operator: and
session_channel: web-chat
handle_delays: false
Both JSON and YML have the same properties:
Mandatory properties are marked with an asterisk (*)
Property | Description | Example | Comment |
---|---|---|---|
| Test case unique identifier | ||
| Test case description | ||
| Named tag to distinguish test cases. Used for filtering execution of test cases. | ||
| Referenced dialog.json ID number | ||
| Simulation data. The following simulation data categories are available:
These nodes should in turn contain field-value pairs that simulate the output parameters of that action. | Let’s simulate the following case:
Here’s the YML syntax:
CODE
| This parameter is mandated supporting service. Testing Studio does not filter or operate on any simulation data. They are just optionally sent to supporting service when a dialog starts in order to facilitate testing needs. Include only type and values you want to use in the simulation data, otherwise the test will fail. For example, if you don’t want to send Make sure that you use the correct case for the fields you are simulating. For example |
| How many times a failed test case is to be executed (0_to_20, default: 1) | A test case might fail due to various reasons. A common reason is that a dialog flow in an Omilia application is not deterministic, thus the flow might have a different dialog step order every time it is executed. For this reason, a test case is executed multiple times (as determined by | |
| Actual utterances of the dialog to be tested. If omitted,the tool will use the utterances from the dialog.json file. Utterances consist of the following fields:
| If modality is not present, the test case modality will be used. The number of utterances in the test case should not exceed the number of actual steps of the referenced dialog, otherwise the test case will fail. | |
| A set of rules by which the Testing Studio will assert or ignore specific dialog events. It may comprise the
| YML syntax:
CODE
CODE
| Since the |
| Represents the channel on which Testing Studio will start the dialog for the specific Test Case. | Possible values: | |
| An option to let Testing Studio handle delays (such as ‘firstWait’, ‘repeadedWait’) The default value is |
|
Test case files can contain special character in their file names. Testing studio will replace them when running test cases to avoid any issues. Avoid using % as a part of a test case file name, since it is used as a wild card when filtering. In general, it is considered a good practice to avoid using special characters in file names or folder names.
scenarios.yml File
The scenarios.yml file contains a list of test cases used as user-defined testing scenarios. The purpose of this file is to support Test Driven Development.
If golden dialog files are unavailable, to run a test, create a scenarios.yml file with test cases. Test cases will run normally with assertions disabled.
scenarios.yml files are supplementary to any existing test cases.
scenarios.yml
scenarios:
- id: 101
golden_dialog: dialog1.json
utterances:
- utterance: ""
- utterance: hello
- utterance: card balance
- id: 102
golden_dialog: dialog2.json
group: a group
utterances:
- utterance: ""
- utterance: good morning
- utterance: i lost my card
- utterance: one two three four
dialog_events_rules:
assert:
exp:
- Intent=Balance%
- OQ_Welcome
operator: or
The golden_dialog values must be unique within a test suite, otherwise the process will fail.
User-defined test cases should include an empty utterance as the first utterance of the test case.
test_cases.yml File
The test_cases.yml is a file created by Testing Studio during test case generation which contains a list of test cases. Testing Studio parses test_cases.yml to execute an actual dialog.
test_cases.yml
---
- id: 101
golden_dialog: dialog1.json
utterances:
- utterance: hello
- utterance: card balance
- id: 102
golden_dialog: dialog2.json
group: a group
utterances:
- utterance: ""
- utterance: good morning
- utterance: i lost my card
- utterance: one two three four
The golden_dialog values must be unique within a test suite, otherwise the process will fail.
global_properties.yml File
This file allows you to set a number of test case properties globally for all test cases inside a test suite folder. The file is optional and has a lower precedence than the properties set in the test case file. The supported properties are the following:
tries
simulation_data
modality
dialog_events_rules
global_properties.yml
tries: 5
simulation_data:
initial:
Ani: 123
Dnis: 12345
dialog_events_rules:
assert:
exp:
- Intent%
- OQ_Welcome
operator: and