Testing Studio Structure

Testing Studio has the following directory structure:

/tests : directory where test suites are saved
/tests/.omilia-ci.yml: Testing Studio configuration file
/tests/example_test_suite/: directory where logically grouped test cases are saved.
/tests/example_test_suite/test_cases.yml: file containing test cases.
/tests/example_test_suite/scenarios.yml: file containing user defined test cases
/tests/example_test_suite/golden_dialogs/: directory where golden dialogs are saved
/tests/example_test_suite/golden_dialogs/dialog.json: golden dialog file in JSON format downloaded from OmDRTViewer
/tests/example_test_suite/global_properties.yml": file containing property values that are set globally, for all test cases

View the following .zip file to check the Testing Studio structure and download it as a template:
testing_studio_template.zip

Testing Studio Configuration File

The omilia-ci.yml configuration file is a readable self-explanatory YML file with the following content:

.omilia-ci.yml

YAML

assertions:
  prompt: true
  intent: true
  end_type: true
  grammar: true
  annotation: true
  action_name: true
  target_name: true
  dialog_events: true
  interpretation: true
  destination_uri: true
  outbound_attached_data: true
  prompt_threshold: 0.8

Below you can find the properties description of omilia-ci.yml:

Section	Property	Data Type	Value	Description
`assertions`				This section contains all the available assertions. Depending on the testing level, other assertion strategies can be configured
	`prompt`	boolean	true/false	The prompt of the dialog step
	`end_type`	boolean	true/false	The end type of the dialog
	`dialog_events`	boolean	true/false	Events of the dialog
	`transfer_line`	boolean	true/false	Transfer line of the dialog
	`destination_uri`	boolean	true/false	Destination URI of the dialog
	`intent`	boolean	true/false	Intent of the dialog step
	`grammar`	boolean	true/false	Grammar of the dialog step
	`annotation`	boolean	true/false	Annotation of the dialog step
	`interpretation`	boolean	true/false	Interpretation of the dialog step
	`action_name`	boolean	true/false	Action name of the dialog step
	`target_name`	boolean	true/false	Target name of the dialog step
	`outbound_attached_data`	boolean	true/false	Attached data of the dialog step
	`prompt_threshold`	integer	0 to 1 default set to 0.8	The lowest limit for the similarity of prompts

Test Suite Folder

The test suite directory contains the following file types:

Below you can find a detailed description of each file type.

dialog.json File

The dialog.json file is the actual dialog that the Testing Studio is going to reproduce and validate.

In order to be used, the dialog.json files should be placed under golden_dialogs folder in every Test suite folder.

Make sure that all fields are snake_case. Any other format (for example, camelCase) is incompatible, and the tool will fail on startup.

dialog.json

JSON

{
 "total_steps": 1,
 "duration": ,
 "dialog_id": "",
 "app_name": "",
 "origin_uri": "",
 "destination_uri": "",
 "inbound_attached_data": "",
 "dialog_events":,
 "end_type": "",
 "transfer_line": "",
 "dialog_date": "",
 "dialog_steps": [
   {
     "step_num": 1,
     "confidence":,
     "action_name": "",
     "target_name": "",
     "utterance": "",
     "interpretation": "[]",
     "annotation": "[]",
     "grammar": "",
     "locale": "",
     "prompt": "",
     "comment": "",
     "intent": "",
     "outbound_attached_data": "",
     "encryption": "",
     "step_time": ""
   }
 ],
 "audios": [],
 "metadata": {
   "app_version": "",
   "diamant_version": "9.6.0",
   "encryption": ""
 }
}

test_case.json/yml File

Testing Studio parses test case files in order to execute an actual dialog. Test case file has a one-to-one reference with a dialog.json file along with any test/simulation data required for the specific scenario.

Testing Studio supports test case files of two types: JSON and YML.

test_case.json

JSON

{
    "id":"101",
    "description":"Sample test case",
    "group": "a group",
    "simulation_data": {
        "initial":{
            "Ani":"124",
            "Dnis":"12345"
          }
        },
    "golden_dialog": "dialog_1.json",
    "tries": 5,
    "utterances": [
        {"utterance": "hello"},
        {"utterance": "what is my balance?","modality": "speech"},
        {"utterance": "one two three"}
      ],
  "dialog_events_rules": {
    "assert": {
      "exp": [
        "Intent%",
        "OQ_Welcome"
      ],
      "operator": "and"
    }
  },
  "session_channel": "web-chat",
  "handle_delays": false
}

test_case.yml

YAML

id: 101
description: Sample test case
group: a group
simulation_data:
  initial:
    Ani: 124
    Dnis: 12345
golden_dialog: dialog_1.json
tries: 5
utterances:
- utterance: hello
- utterance: what is my balance?
  modality: speech
  simulation_data:
    bio_data:
      BioScore: TRUE_HIGH
- utterance: one two three
dialog_events_rules:
  assert:
    exp:
      - Intent%
      - OQ_Welcome
    operator: and
session_channel: web-chat
handle_delays: false

Both JSON and YML have the same properties:

Mandatory properties are marked with an asterisk (*)

Property	Description	Example	Comment
`id` *	Test case unique identifier
`description` *	Test case description
`group`	Named tag to distinguish test cases. Used for filtering execution of test cases.
`golden_dialog` *	Referenced dialog.json ID number
`simulation_data`	Simulation data. The following simulation data categories are available: `initial`: contains application field-value pairs that you want to send upon dialog initiation (e.g. to simulate the value of `Ani`, `Dnis` or other fields right when the dialog starts). `bio_data`: contains voice biometrics field-value pairs that you want to simulate (e.g. BioScore or BioSearch) These nodes should in turn contain field-value pairs that simulate the output parameters of that action.	Let’s simulate the following case: `Ani` gets the value 124 and `Dnis` gets the value 12345 on dialog start. `Ani`, `Dnis`, `balance` and `card` are the parameters defined either by the application in test. Here’s the YML syntax: CODE `simulation_data: initial: Ani: 124 Dnis: 12345`	This parameter is mandated supporting service. Testing Studio does not filter or operate on any simulation data. They are just optionally sent to supporting service when a dialog starts in order to facilitate testing needs. Include only type and values you want to use in the simulation data, otherwise the test will fail. For example, if you don’t want to send `Dnis`, don’t include it at all. Make sure that you use the correct case for the fields you are simulating. For example `Ani` for the origin number and `Dnis` for the destination number.
`tries` *	How many times a failed test case is to be executed (0_to_20, default: 1)		A test case might fail due to various reasons. A common reason is that a dialog flow in an Omilia application is not deterministic, thus the flow might have a different dialog step order every time it is executed. For this reason, a test case is executed multiple times (as determined by `tries` value) and considered failed only if assertions failed every time. If the test case passed at least once, it is considered as successful.
`utterances`	Actual utterances of the dialog to be tested. If omitted,the tool will use the utterances from the dialog.json file. Utterances consist of the following fields: `utterance`: The text representation of the utterance. This field is mandatory. `modality`: Whether this utterance will be sent as a chat, DTMF or with ASR support. Possible values are `text`, `dtmf` and `speech` respectively.		If modality is not present, the test case modality will be used. The number of utterances in the test case should not exceed the number of actual steps of the referenced dialog, otherwise the test case will fail.
`dialog_events_rules` NOTE: these rules work only if the `dialog_events` assertion is enabled in the .omilia-ci.yml file	A set of rules by which the Testing Studio will assert or ignore specific dialog events. It may comprise the `assert` or `ignore` nodes: when using the `assert`, then only what you have put under the assert node will be validated. any other dialog event will not (same goes for `ignore`), when using `ignore`, it will check all the dialog events except those declared under the ignore node	YML syntax: Assert node CODE `dialog_events_rules: assert: exp: - Intent% - OQ_Welcome operator: and` `Intent%` means that the golden dialog must contain the same dialog events starting with `Intent` as the ones in the actual running case. A successful scenario example: `Golden: OQ_Welcome, Intent-Balance, HUP` `Actual: OQ_Welcome, AgentRequest, Intent-Balance, TRANSFER` A failed scenario example: `Golden: OQ_Welcome, Intent-Card_Activate, HUP` `Actual: OQ_Welcome, AgentRequest, Intent-Balance, TRANSFER` Ignore node CODE `dialog_events_rules: ignore: exp: - Ani:% operator: or` `Ani=%` means that all dialog events starting with `Ani` can be ignored. The rest of the dialog events must all be the same. A successful scenario example: `Golden: Ani=12345, OQ_Welcome, Intent-Balance, HUP` `Actual: Ani=99999, OQ_Welcome, Intent-Balance, HUP` A failed scenario example: `Golden: Ani=12345, OQ_Welcome, Intent-Balance, HUP` `Actual: Ani=99999, OQ_Welcome, AgentRequest, Intent-Balance, HUP`	Since the `Assert` node only compares the declared expressions and implicitly ignores all the other ones, defining both type of nodes under `dialog_events_rules` is redundant. The `Ignore` node will be ignored.
`session_channel`	Represents the channel on which Testing Studio will start the dialog for the specific Test Case.	Possible values: `web-chat, chat, ivr, miniapps.`
`handle_delays`	An option to let Testing Studio handle delays (such as ‘firstWait’, ‘repeadedWait’) The default value is `true`	`false`

Test case files can contain special character in their file names. Testing studio will replace them when running test cases to avoid any issues. Avoid using % as a part of a test case file name, since it is used as a wild card when filtering. In general, it is considered a good practice to avoid using special characters in file names or folder names.

scenarios.yml File

The scenarios.yml file contains a list of test cases used as user-defined testing scenarios. The purpose of this file is to support Test Driven Development.
If golden dialog files are unavailable, to run a test, create a scenarios.yml file with test cases. Test cases will run normally with assertions disabled.
scenarios.yml files are supplementary to any existing test cases.

scenarios.yml

YAML

scenarios:
  - id: 101
    golden_dialog: dialog1.json
    utterances:
      - utterance: ""
      - utterance: hello
      - utterance: card balance
  - id: 102
    golden_dialog: dialog2.json
    group: a group
    utterances:
      - utterance: ""
      - utterance: good morning
      - utterance: i lost my card
      - utterance: one two three four
    dialog_events_rules:
      assert:
        exp:
          - Intent=Balance%
          - OQ_Welcome
        operator: or

The golden_dialog values must be unique within a test suite, otherwise the process will fail.

User-defined test cases should include an empty utterance as the first utterance of the test case.

test_cases.yml File

The test_cases.yml is a file created by Testing Studio during test case generation which contains a list of test cases. Testing Studio parses test_cases.yml to execute an actual dialog.

test_cases.yml

YAML

---
  - id: 101
    golden_dialog: dialog1.json
    utterances:
      - utterance: hello
      - utterance: card balance
  - id: 102
    golden_dialog: dialog2.json
    group: a group
    utterances:
      - utterance: ""
      - utterance: good morning
      - utterance: i lost my card
      - utterance: one two three four

The golden_dialog values must be unique within a test suite, otherwise the process will fail.

global_properties.yml File

This file allows you to set a number of test case properties globally for all test cases inside a test suite folder. The file is optional and has a lower precedence than the properties set in the test case file. The supported properties are the following:

tries
simulation_data
modality
dialog_events_rules

global_properties.yml

YAML

tries: 5
simulation_data:
    initial:
      Ani: 123
      Dnis: 12345
dialog_events_rules:
  assert:
    exp:
      - Intent%
      - OQ_Welcome
    operator: and