Testing Studio API
Overview
Testing Studio® is Omilia’s automation testing framework engineered to address the development and testing needs of all Omilia’s conversation solutions at every stage of the development lifecycle.
This codeless testing framework is designed to support any chosen development type as well as different levels of functional and non-functional testing, from unit to system testing. It also offers visualization results and test analytics for monitoring the quality progress.
Testing Studio API allows you to interact, access and use Testing Studio, Omilia’s Automation Testing framework engineered for Omilia Conversational applications, able to reproduce and validate conversational scenarios.
Authentication
Navigate to the API Authentication to learn more about Authorization Token Endpoint and Authentication process.
Errors Code
Omilia's HTTP response codes can be found on the HTTP Code Responses page.
Guides
Overview
Here you will find some useful guides on how to use Testing Studio API and combine it as part of your work.
Use through CI/CD
In case you have a pipeline running and you want to add Testing Studio as part of your CI/CD, you will need to follow these steps:
Create a Testing Studio project
You could Create Project only once, and reuse it anytime to run your tests. So you may create it manually instead of creating and deleting a project as part of your pipeline.
Pass your Test Suites
Keep in mind that there two ways of passing your test suites in a project:
use a git provider (GitLab, Github)
use internal cloud storage (Testing Studio’s S3 service)
In case you want to use a Git repo, you don’t need to make any extra step here. When creating the project, you will pass your git values and Testing Studio will load them and make a clone of your repo. In case you want to use the provided S3 service, you will need to make a call to Upload Project Test Suites.
Run your tests
Since you have uploaded your tests, you may now make a run call.
Aggregate tests results
Your tests are running asynchronously. As a result, you will need to make periodic calls to Check the status of your run job.
If your status is in pending or starting, then your test suites are still running, and you will need to recheck later.
If your status is successful, it means that all your tests run and pass.
If your status is failed, then some tests didn’t pass. To check which ones are the failure ones, you will need to Get Run Test Suits (given a test suite) and parse the details. You may also use a call to Get Run, in case you don’t remember them.
API
Testing Studio API
Here you can find out more details about Testing Studio API endpoints.
Authorization requests
If you want to include an authorization token with an API request, you shall pass the Authorization header following by the token:
Header: Authorization
Value: Bearer lhdCI6MTYxMjgx...kNGZjIiwiYW
Models
Here you can find a list of the models we are using in Testing Studio API.
Project
Name | Type | Default | Description |
---|---|---|---|
| uuid | - | project’s id |
| string | - | project’s name |
| string | - | project’s group |
| uuid | - | the created by user’s id |
| strong | null | project’s git-repo url [git storage enabled] |
| string | null | project’s git token [git storage enabled] |
| null | project’s git provider [git storage enabled] | |
| null | latest project’s run job | |
| boolean | - | true if project is git storage enabled |
| boolean | - | true if project is S3 storage enabled |
GitProvider (enum)
Available values for this 'string' enum:
"gitlab"
"github"
Example
JSON{ "data": { "name": "ia_iandronis_app_1", "gitUrl": "https://gitlab.omilia.com/iandronis/my_testing_studio_repo.git", "gitToken": "jDdEdsfFRGr4Gmd3R47r", "gitProvider": "gitlab", "id": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f", "userId": "f7e3d168-053c-47ff-b592-eea2071ff794", "group": "iandronis", "latestRun": { "projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f", "parallelTestCases": 1, "group": null, "utterance": null, "description": null, "runId": "e6fa9e9a-5055-4773-9652-309b05e88c49", "status": "SUCCESS", "createdAt": "2023-04-11T15:22:50.193226", "finishedAt": "2023-04-11T15:22:53.568724", "tasks": [], "artifact": { "artifactId": "b96f6c0d-c08f-4454-a624-99564dbf8fb4", "name": "e6fa9e9a-5055-4773-9652-309b05e88c49_artifacts.zip", "createdAt": "2023-04-11T15:22:52.916354", "deletedAt": null, "ttl": 604800, "isExpired": false, "runId": "e6fa9e9a-5055-4773-9652-309b05e88c49" }, "isRevocable": false, "executionMeta": null, "duration": "0:00:03" }, "gitStorage": true, "s3Storage": false }, "meta": { "count": 1 } }
Run
Name | Type | Default | Description |
---|---|---|---|
| uuid | - | run’s project id |
| int | 1 | number of simultaneous test cases |
| string | null | group that has been used as a filter for the test cases |
| string | null | utterance that has been used as a filter for the test cases |
| string | null | description that has been used as a filter for the test cases |
| uuid | - | run’s id |
| - | run’s execution status | |
| timestamp | null | run’s created at timestamp |
| timestamp | null | run’s finished at timestamp |
| list | [] | run’s tasks list |
| null | run’s artifact | |
| boolean | false | true if run can be revoked |
| null | run’s meta info | |
| timestamp | - | run’s duration time |
RunStatus (enum)
Available values for this 'string' enum:
"PENDING"
"STARTED"
"SUCCESS"
"FAILURE"
"RETRY"
"REVOKED"
Example
JSON{ "data": [ { "projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f", "parallelTestCases": 1, "group": null, "utterance": null, "description": null, "runId": "e6fa9e9a-5055-4773-9652-309b05e88c49", "status": "SUCCESS", "createdAt": "2023-04-11T15:22:50.193226", "finishedAt": "2023-04-11T15:22:53.568724", "tasks": [], "artifact": { "artifactId": "b96f6c0d-c08f-4454-a624-99564dbf8fb4", "name": "e6fa9e9a-5055-4773-9652-309b05e88c49_artifacts.zip", "createdAt": "2023-04-11T15:22:52.916354", "deletedAt": null, "ttl": 604800, "isExpired": false, "runId": "e6fa9e9a-5055-4773-9652-309b05e88c49" }, "isRevocable": false, "executionMeta": null, "duration": "0:00:03" } ], "meta": { "count": 1 } }
Task Artifact
Name | Type | Default | Description |
---|---|---|---|
| uuid | - | artifact’s id |
| string | - | artifact’s name |
| timestamp | null | artifact’s created at timestamp |
| timestamp | null | artifact’s finished at timestamp |
| int | - | artifact’s time to live (in seconds) |
| boolean | - | true if artifact has been expired |
| uuid | - | artifact’s run id |
Task Execution Meta
Name | Type | Default | Description |
---|---|---|---|
| int | - | task’s completed jobs |
| int | - | task’s total jobs |
Generated Test Case
Name | Type | Default | Description |
---|---|---|---|
| uuid | - | test case’s project id |
| string | null | group to entered to newly generated test cases |
| string | null | comma separated test suite names for which test cases will be be generated |
| boolean | false | true if we want to overwrite existing test cases |
| string | null | comma separated simulation data to be added during generation |
| uuid | - | test case’s id |
| - | generation job’s execution status | |
| timestamp | null | generation job’s created at timestamp |
| timestamp | null | generation job’s finished at timestamp |
| list | [] | generation job’s tasks list |
| null | generation job’s artifact | |
| boolean | false | true if generation job can be revoked |
| null | generation job’s meta info | |
| timestamp | - | generation job’s duration time |
Example
{
"data": {
{
"projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"group": null,
"testSuite": null,
"overwrite": true,
"simulationData": null,
"runId": "ee6a56f0-1d2b-4266-bbe8-d4374156c400",
"status": "SUCCESS",
"createdAt": "2023-04-10T14:24:39.345269",
"finishedAt": "2023-04-10T14:24:41.812109",
"tasks": [],
"artifact": {
"artifactId": "86828048-9e69-4f3f-a053-80daad7ecd66",
"name": "ee6a56f0-1d2b-4266-bbe8-d4374156c400_artifacts.zip",
"createdAt": "2023-04-10T14:24:41.144837",
"deletedAt": null,
"ttl": 604800,
"isExpired": true,
"runId": "ee6a56f0-1d2b-4266-bbe8-d4374156c400"
},
"isRevocable": false,
"executionMeta": null
}
},
"meta": {
"count": 1
}
}
Run Stats
Name | Type | Default | Description |
---|---|---|---|
| uuid | - | run stats’s id |
| uuid | - | run’s id |
| int | 0 | run’s success test cases |
| int | 0 | run’s fail test cases |
| int | 0 | run’s unavailable test cases |
Example
{
"data": {
"statsId": "9fc2cc39-f533-4557-a844-bc3ee0fd5d0e",
"runId": "e6fa9e9a-5055-4773-9652-309b05e88c49",
"pass": 2,
"fail": 0,
"unavailable": 0
}
}
Run Test Suites
Name | Type | Default | Description |
---|---|---|---|
| string | - | test suite’s name |
Example
{
"data": [
{
"testSuiteName": "test_suite_1"
},
{
"testSuiteName": "test_suite_2"
}
]
}
Run Test Case Stats
Name | Type | Default | Description |
---|---|---|---|
| int | - | number of tries |
| float | - | execution time (in seconds) |
| string | - | test case’s name |
| string | - | the dialog id, on which the test case has been run |
| string | - | the golden dialog id |
| uuid | - | test case’s id |
| string | null | test case’s id description |
| string | - | test case’s status (one of 'pass', 'fail', 'unavailable', 'None') |
| list Test Case Error | [] | test case’s assertion errors |
| list Test Case Warning | [] | test case’s general errors |
| list Test Case Warning | [] | test case’s warnings |
| boolean | - | true if test case has delays |
Test Case Error
Name | Type | Default | Description |
---|---|---|---|
| int | - | test case’s step index |
| string | - | assertion field |
| string | - | what value is to be expected |
| string | - | what is the actual result’s value |
Test Case Warning
Name | Type | Default | Description |
---|---|---|---|
| string | - | warning’s text |
Example
{
"data": {
"numberOfTries": 1,
"executionTime": 889.0,
"testCaseName": "scenario_1",
"dialogId": "iandronis_SND_ia_iandronis_app_1.1681812519019.7db27172adc94a3cbc648c8456f6f796",
"goldenDialogId": "c0bf3e121907d0b2ca2beb219c38926858201a18.1681143584265.83db1d064122481b848adad10fb5c0f3",
"testCaseId": "c43143e9-1ec5-45be-8565-539382a3fd01",
"testCaseDescription": null,
"status": "success",
"assertionErrors": [],
"generalErrors": [],
"warnings": [],
"hadDelays": false
}
}
Run Log
Name | Type | Default | Description |
---|---|---|---|
| string | - | section name |
| string | - | log’s id |
| list Run Log Content | - | log’s content |
| int | - | log’s offset |
| boolean | - | run’s completed status |
Run Log Content
Name | Type | Default | Description |
---|---|---|---|
| string | - | log content text |
Example
{
"sectionName": "Clone Repository",
"id": "580edef7-667b-44b2-9f29-a05ec254e300",
"content": [
{
"text": "---- Billing sdk is not enabled ----"
},
{
"text": "---- Cloning repository ----"
},
{
"text": "Cloning into '/workspace/f7e3d168-053c-47ff-b592-eea2071ff794_c2a9ec59-0c38-4372-bff5-83cd7f9c721f_cd933ffa-189b-40fc-857c-5f60071f842f'...\n"
},
{
"text": "Task completed successfully"
},
{
"text": "---- Billing sdk is not enabled ----"
}
],
"offset": 5,
"completed":
}
Health Check
Endpoint to get service’s health and connectivity with other services.
GET /api/healthcheck/
Response (Success)
HTTP/1.1 200 OK
API’s Status
Parameter | Type | Description |
---|---|---|
| string | API’s status |
Example
{
"data":
{
"status": "up"
}
}
Projects
Get Projects
Return the list of Project the user has access to.
GET /api/projects/
Response (Success)
HTTP/1.1 200 OK
Example
{
"data": [
{
"name": "ia_iandronis_app_1",
"gitUrl": "https://gitlab.omilia.com/iandronis/my_testing_studio_repo.git",
"gitToken": "jDdEdsfFRGr4Gmd3R47r",
"gitProvider": "gitlab",
"id": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"userId": "f7e3d168-053c-47ff-b592-eea2071ff794",
"group": "iandronis",
"latestRun": {
"projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"parallelTestCases": 1,
"group": null,
"utterance": null,
"description": null,
"runId": "e6fa9e9a-5055-4773-9652-309b05e88c49",
"status": "SUCCESS",
"createdAt": "2023-04-11T15:22:50.193226",
"finishedAt": "2023-04-11T15:22:53.568724",
"tasks": [],
"artifact": {
"artifactId": "b96f6c0d-c08f-4454-a624-99564dbf8fb4",
"name": "e6fa9e9a-5055-4773-9652-309b05e88c49_artifacts.zip",
"createdAt": "2023-04-11T15:22:52.916354",
"deletedAt": null,
"ttl": 604800,
"isExpired": false,
"runId": "e6fa9e9a-5055-4773-9652-309b05e88c49"
},
"isRevocable": false,
"executionMeta": null,
"duration": "0:00:03"
},
"gitStorage": true,
"s3Storage": false
},
{
"name": "ia_iandronis_app_2",
"gitUrl": null,
"gitToken": null,
"gitProvider": null,
"id": "2b20af9e-2761-43b7-9ff9-e6b0addf14fe",
"userId": "f7e3d168-053c-47ff-b592-eea2071ff794",
"group": "iandronis",
"latestRun": null,
"gitStorage": false,
"s3Storage": true
}
],
"meta": {
"count": 2
}
}
Create Project
A project has two storage options of how we shall provide our test suites to it:
through a Git provider
through Testing Studio S3 service
To select the Git provider approach, you shall give values to all git-related variables (url, git_token, git_provider). To select the S3 approach, you shall skip all the git-related variables.
POST /api/projects/
Request Payload
Parameter | Type | Default | Description |
---|---|---|---|
| string | - | project’s name |
| string | - | project’s group |
| string | null | project’s url |
| string | null | project’s git token |
| null | project’s git provider |
Example
{
"name": "my_project_1",
"group": "my_group",
"url": "gitlab_url.git",
"gitToken": "my_gitlab_token",
"gitProvider": "gitlab"
}
Response (Success)
HTTP/1.1 200 OK
The created Project
{
"data": {
"name": "ia_iandronis_app_1",
"gitUrl": "https://gitlab.omilia.com/iandronis/my_testing_studio_repo.git",
"gitToken": "jDdEdsfFRGr4Gmd3R47r",
"gitProvider": "gitlab",
"id": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"userId": "f7e3d168-053c-47ff-b592-eea2071ff794",
"group": "iandronis",
"latestRun": {
"projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"parallelTestCases": 1,
"group": null,
"utterance": null,
"description": null,
"runId": "e6fa9e9a-5055-4773-9652-309b05e88c49",
"status": "SUCCESS",
"createdAt": "2023-04-11T15:22:50.193226",
"finishedAt": "2023-04-11T15:22:53.568724",
"tasks": [],
"artifact": {
"artifactId": "b96f6c0d-c08f-4454-a624-99564dbf8fb4",
"name": "e6fa9e9a-5055-4773-9652-309b05e88c49_artifacts.zip",
"createdAt": "2023-04-11T15:22:52.916354",
"deletedAt": null,
"ttl": 604800,
"isExpired": false,
"runId": "e6fa9e9a-5055-4773-9652-309b05e88c49"
},
"isRevocable": false,
"executionMeta": null,
"duration": "0:00:03"
},
"gitStorage": true,
"s3Storage": false
},
"meta": {
"count": 1
}
}
Response (Failure)
HTTP/1.1 400 BAD_REQUEST
when user has no access to the payload’s group
Response Message
{
"detail": "Group: my_groupde invalid"
}
when you pass some of git-related parameters (meaning it’s not clear what storage-approach you want to follow)
Response Message
{
"detail": "Request contains an error. Field: __root__, Error: In case of git storage option, you should initialize all git variables."
}
Edit Project
PUT /api/projects/<project_id>
URL Parameters
Parameter | Type | Description | Example |
---|---|---|---|
| uuid | the project’s id | "9b399774-3a0a-40c1-8a1c-4a124170b6f3" |
Request Payload
Project field values
Parameter | Type | Default | Description |
---|---|---|---|
| string | - | project’s name |
| string | - | project’s group |
| string | null | project’s url |
| string | null | project’s git token |
| null | project’s git provider |
Example
{
"name": "my_project_1",
"group": "my_group_1",
"url": "gitlab_url.git",
"gitToken": "my_gitlab_token",
"gitProvider": "gitlab"
}
Response (Success)
HTTP/1.1 200 OK
The updated Project.
Example
{
"data": {
"name": "ia_iandronis_app_1",
"gitUrl": "https://gitlab.omilia.com/iandronis/my_testing_studio_repo.git",
"gitToken": "jDdEdsfFRGr4Gmd3R47r",
"gitProvider": "gitlab",
"id": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"userId": "f7e3d168-053c-47ff-b592-eea2071ff794",
"group": "iandronis",
"latestRun": {
"projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"parallelTestCases": 1,
"group": null,
"utterance": null,
"description": null,
"runId": "e6fa9e9a-5055-4773-9652-309b05e88c49",
"status": "SUCCESS",
"createdAt": "2023-04-11T15:22:50.193226",
"finishedAt": "2023-04-11T15:22:53.568724",
"tasks": [],
"artifact": {
"artifactId": "b96f6c0d-c08f-4454-a624-99564dbf8fb4",
"name": "e6fa9e9a-5055-4773-9652-309b05e88c49_artifacts.zip",
"createdAt": "2023-04-11T15:22:52.916354",
"deletedAt": null,
"ttl": 604800,
"isExpired": false,
"runId": "e6fa9e9a-5055-4773-9652-309b05e88c49"
},
"isRevocable": false,
"executionMeta": null,
"duration": "0:00:03"
},
"gitStorage": true,
"s3Storage": false
},
"meta": {
"count": 1
}
}
Delete Project
DELETE /api/projects/<project_id>
$body
URL parameters
Parameter | Type | Description | Example |
---|---|---|---|
| uuid | the project’s id | "9b399774-3a0a-40c1-8a1c-4a124170b6f3" |
Response (Success)
HTTP/1.1 204 NO_CONTENT
Get Project
GET /api/projects/<project_id>
URL parameters
Parameter | Type | Description | Example |
---|---|---|---|
| uuid | the project’s id | "9b399774-3a0a-40c1-8a1c-4a124170b6f3" |
Response (Success)
HTTP/1.1 200 OK
Return the Project.
Example
{
"data": {
"name": "ia_iandronis_app_1",
"gitUrl": "https://gitlab.omilia.com/iandronis/my_testing_studio_repo.git",
"gitToken": "jDdEdsfFRGr4Gmd3R47r",
"gitProvider": "gitlab",
"id": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"userId": "f7e3d168-053c-47ff-b592-eea2071ff794",
"group": "iandronis",
"latestRun": {
"projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"parallelTestCases": 1,
"group": null,
"utterance": null,
"description": null,
"runId": "e6fa9e9a-5055-4773-9652-309b05e88c49",
"status": "SUCCESS",
"createdAt": "2023-04-11T15:22:50.193226",
"finishedAt": "2023-04-11T15:22:53.568724",
"tasks": [],
"artifact": {
"artifactId": "b96f6c0d-c08f-4454-a624-99564dbf8fb4",
"name": "e6fa9e9a-5055-4773-9652-309b05e88c49_artifacts.zip",
"createdAt": "2023-04-11T15:22:52.916354",
"deletedAt": null,
"ttl": 604800,
"isExpired": false,
"runId": "e6fa9e9a-5055-4773-9652-309b05e88c49"
},
"isRevocable": false,
"executionMeta": null,
"duration": "0:00:03"
},
"gitStorage": true,
"s3Storage": false
},
"meta": {
"count": 1
}
}
Get Project Runs
GET /api/projects/<project_id>/runs/
URL parameters
Parameter | Type | Description | Example |
---|---|---|---|
| uuid | the project’s id | "9b399774-3a0a-40c1-8a1c-4a124170b6f3" |
Response (Success)
HTTP/1.1 200 OK
Return the Project’s Run.
Example
{
"data": [
{
"projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"parallelTestCases": 1,
"group": null,
"utterance": null,
"description": null,
"runId": "e6fa9e9a-5055-4773-9652-309b05e88c49",
"status": "SUCCESS",
"createdAt": "2023-04-11T15:22:50.193226",
"finishedAt": "2023-04-11T15:22:53.568724",
"tasks": [],
"artifact": {
"artifactId": "b96f6c0d-c08f-4454-a624-99564dbf8fb4",
"name": "e6fa9e9a-5055-4773-9652-309b05e88c49_artifacts.zip",
"createdAt": "2023-04-11T15:22:52.916354",
"deletedAt": null,
"ttl": 604800,
"isExpired": false,
"runId": "e6fa9e9a-5055-4773-9652-309b05e88c49"
},
"isRevocable": false,
"executionMeta": null,
"duration": "0:00:03"
},
{
"projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"parallelTestCases": 1,
"group": null,
"utterance": null,
"description": null,
"runId": "cce602f1-80af-437a-883d-015ace310802",
"status": "SUCCESS",
"createdAt": "2023-04-11T10:28:31.863561",
"finishedAt": "2023-04-11T10:28:35.356189",
"tasks": [],
"artifact": {
"artifactId": "f2e834bb-fa0c-46de-98dd-cb5b1f4f0711",
"name": "cce602f1-80af-437a-883d-015ace310802_artifacts.zip",
"createdAt": "2023-04-11T10:28:34.712515",
"deletedAt": null,
"ttl": 604800,
"isExpired": false,
"runId": "cce602f1-80af-437a-883d-015ace310802"
},
"isRevocable": false,
"executionMeta": null,
"duration": "0:00:03""projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"parallelTestCases": 1,
"group": null,
"utterance": null,
"description": null,
"runId": "49b1e745-4dab-429c-a691-0e7d35144d5d",
"status": "FAILURE",
"createdAt": "2023-04-11T10:27:47.938815",
"finishedAt": "2023-04-11T10:27:51.456288",
"tasks": [],
"artifact": {
"artifactId": "ab9bedc0-992f-412b-9e2e-f43902cc464c",
"name": "49b1e745-4dab-429c-a691-0e7d35144d5d_artifacts.zip",
"createdAt": "2023-04-11T10:27:50.778791",
"deletedAt": null,
"ttl": 604800,
"isExpired": false,
"runId": "49b1e745-4dab-429c-a691-0e7d35144d5d"
},
"isRevocable": false,
"executionMeta": null,
"duration": "0:00:03""projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"parallelTestCases": 1,
"group": null,
"utterance": null,
"description": null,
"runId": "cdeda41a-ca21-4dda-aa27-91d4de8b1091",
"status": "FAILURE",
"createdAt": "2023-04-11T10:25:37.465533",
"finishedAt": "2023-04-11T10:25:41.072522",
"tasks": [],
"artifact": {
"artifactId": "20127f7d-30ee-4334-a61d-5f4defbf2685",
"name": "cdeda41a-ca21-4dda-aa27-91d4de8b1091_artifacts.zip",
"createdAt": "2023-04-11T10:25:40.423027",
"deletedAt": null,
"ttl": 604800,
"isExpired": false,
"runId": "cdeda41a-ca21-4dda-aa27-91d4de8b1091"
},
"isRevocable": false,
"executionMeta": null,
"duration": "0:00:03"
},
{
"projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"parallelTestCases": 1,
"group": null,
"utterance": null,
"description": null,
"runId": "12dc3118-77e7-4ac7-8dc9-9221348c5378",
"status": "SUCCESS",
"createdAt": "2023-04-10T14:30:43.468499",
"finishedAt": "2023-04-10T14:30:46.725014",
"tasks": [],
"artifact": {
"artifactId": "e96293b2-1bdd-4bb4-b5e4-ec4d3d0e5523",
"name": "12dc3118-77e7-4ac7-8dc9-9221348c5378_artifacts.zip",
"createdAt": "2023-04-10T14:30:46.120939",
"deletedAt": null,
"ttl": 604800,
"isExpired": false,
"runId": "12dc3118-77e7-4ac7-8dc9-9221348c5378"
},
"isRevocable": false,
"executionMeta": null,
"duration": "0:00:03"
}
],
"meta": {
"count": 5
}
}
Get Project Generated Test Cases
GET /api/projects/<project_id>/generations/
URL parameters
Parameter | Type | Description | Example |
---|---|---|---|
| uuid | the project’s id | "9b399774-3a0a-40c1-8a1c-4a124170b6f3" |
Response (Success)
HTTP/1.1 200 OK
Return the Project’s Generated Test Case.
Example
{
"data": [
{
"projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"group": null,
"testSuite": null,
"overwrite": true,
"simulationData": null,
"runId": "ee6a56f0-1d2b-4266-bbe8-d4374156c400",
"status": "SUCCESS",
"createdAt": "2023-04-10T14:24:39.345269",
"finishedAt": "2023-04-10T14:24:41.812109",
"tasks": [],
"artifact": {
"artifactId": "86828048-9e69-4f3f-a053-80daad7ecd66",
"name": "ee6a56f0-1d2b-4266-bbe8-d4374156c400_artifacts.zip",
"createdAt": "2023-04-10T14:24:41.144837",
"deletedAt": null,
"ttl": 604800,
"isExpired": true,
"runId": "ee6a56f0-1d2b-4266-bbe8-d4374156c400"
},
"isRevocable": false,
"executionMeta": null
},
{
"projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"group": null,
"testSuite": null,
"overwrite": false,
"simulationData": null,
"runId": "5d9dca1a-f873-4355-bbe0-1083f809f26a",
"status": "SUCCESS",
"createdAt": "2023-03-20T14:23:08.805446",
"finishedAt": "2023-03-20T14:23:11.940246",
"tasks": [],
"artifact": {
"artifactId": "48fcd89a-cfc7-4fe7-a386-3fd55cb8fe5a",
"name": "5d9dca1a-f873-4355-bbe0-1083f809f26a_artifacts.zip",
"createdAt": "2023-03-20T14:23:11.136530",
"deletedAt": null,
"ttl": 604800,
"isExpired": true,
"runId": "5d9dca1a-f873-4355-bbe0-1083f809f26a"
},
"isRevocable": false,
"executionMeta": null
},
{
"projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"group": null,
"testSuite": null,
"overwrite": false,
"simulationData": "Ani,Dnis",
"runId": "6ea08453-04b4-425c-81d5-582c311e47db",
"status": "SUCCESS",
"createdAt": "2023-03-14T14:36:39.768927",
"finishedAt": "2023-03-14T14:36:42.480968",
"tasks": [],
"artifact": {
"artifactId": "9a7f04d2-d31a-40ca-a9f9-1684162c7445",
"name": "6ea08453-04b4-425c-81d5-582c311e47db_artifacts.zip",
"createdAt": "2023-03-14T14:36:41.810116",
"deletedAt": null,
"ttl": 604800,
"isExpired": true,
"runId": "6ea08453-04b4-425c-81d5-582c311e47db"
},
"isRevocable": false,
"executionMeta": null
},
{
"projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"group": "my_group1",
"testSuite": null,
"overwrite": true,
"simulationData": "Ani,Dnis",
"runId": "b19a8219-66cf-4193-9248-87e9b86294ce",
"status": "SUCCESS",
"createdAt": "2023-03-14T14:35:44.135703",
"finishedAt": "2023-03-14T14:35:47.417652",
"tasks": [],
"artifact": {
"artifactId": "087ef50b-2a59-43de-a297-7ccb2acc12e9",
"name": "b19a8219-66cf-4193-9248-87e9b86294ce_artifacts.zip",
"createdAt": "2023-03-14T14:35:46.786142",
"deletedAt": null,
"ttl": 604800,
"isExpired": true,
"runId": "b19a8219-66cf-4193-9248-87e9b86294ce"
},
"isRevocable": false,
"executionMeta": null
},
{
"projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"group": "my_group",
"testSuite": null,
"overwrite": false,
"simulationData": null,
"runId": "41d90bf2-80b0-4afc-89de-29a79b7a0c7e",
"status": "SUCCESS",
"createdAt": "2023-03-14T14:34:49.206265",
"finishedAt": "2023-03-14T14:34:52.143211",
"tasks": [],
"artifact": {
"artifactId": "c0422160-daaa-422a-bbd7-efc0c02a3e6e",
"name": "41d90bf2-80b0-4afc-89de-29a79b7a0c7e_artifacts.zip",
"createdAt": "2023-03-14T14:34:51.459441",
"deletedAt": null,
"ttl": 604800,
"isExpired": true,
"runId": "41d90bf2-80b0-4afc-89de-29a79b7a0c7e"
},
"isRevocable": false,
"executionMeta": null
},
{
"projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"group": null,
"testSuite": null,
"overwrite": false,
"simulationData": null,
"runId": "3e85b428-e017-4f8b-9a22-2531c751331c",
"status": "SUCCESS",
"createdAt": "2023-03-14T10:04:59.356590",
"finishedAt": "2023-03-14T10:05:02.802061",
"tasks": [],
"artifact": {
"artifactId": "08867172-1fd1-4952-bc8d-2d53455842ee",
"name": "3e85b428-e017-4f8b-9a22-2531c751331c_artifacts.zip",
"createdAt": "2023-03-14T10:05:01.888551",
"deletedAt": null,
"ttl": 604800,
"isExpired": true,
"runId": "3e85b428-e017-4f8b-9a22-2531c751331c"
},
"isRevocable": false,
"executionMeta": null
},
{
"projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"group": null,
"testSuite": null,
"overwrite": false,
"simulationData": null,
"runId": "905cc4fd-733b-4f20-94e8-d1fe2269e44d",
"status": "SUCCESS",
"createdAt": "2023-03-07T12:53:37.077502",
"finishedAt": "2023-03-07T12:53:40.272056",
"tasks": [],
"artifact": {
"artifactId": "7c6a9cd6-476d-461e-a44b-040b60ffe772",
"name": "905cc4fd-733b-4f20-94e8-d1fe2269e44d_artifacts.zip",
"createdAt": "2023-03-07T12:53:39.512556",
"deletedAt": null,
"ttl": 604800,
"isExpired": true,
"runId": "905cc4fd-733b-4f20-94e8-d1fe2269e44d"
},
"isRevocable": false,
"executionMeta": null
},
{
"projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"group": null,
"testSuite": null,
"overwrite": false,
"simulationData": null,
"runId": "d08f41a2-9722-4212-b9c0-cb915401ee8a",
"status": "SUCCESS",
"createdAt": "2023-03-01T09:31:26.503398",
"finishedAt": "2023-03-01T09:31:30.353180",
"tasks": [],
"artifact": {
"artifactId": "5907af5a-96de-4bd6-a416-65a732d3d329",
"name": "d08f41a2-9722-4212-b9c0-cb915401ee8a_artifacts.zip",
"createdAt": "2023-03-01T09:31:29.539810",
"deletedAt": null,
"ttl": 604800,
"isExpired": true,
"runId": "d08f41a2-9722-4212-b9c0-cb915401ee8a"
},
"isRevocable": false,
"executionMeta": null
}
],
"meta": {
"count": 8
}
}
Get Project Run Stats
GET /api/projects/<project_id>/runs_stats/
URL parameters
Parameter | Type | Description | Example |
---|---|---|---|
| uuid | the project’s id | "9b399774-3a0a-40c1-8a1c-4a124170b6f3" |
Response (Success)
HTTP/1.1 200 OK
Return the Project’s Run Stats.
Example
{
"data": [
{
"statsId": "9fc2cc39-f533-4557-a844-bc3ee0fd5d0e",
"runId": "86a1848d-3836-45d8-a41d-bc36654bd7d6",
"pass": 2,
"fail": 0,
"unavailable": 0
},
{
"statsId": "ac1044a0-bdfa-4364-8d0f-0f8718b802c7",
"runId": "ecb906bc-bd2c-4c29-98f5-ad175a8e4c4d",
"pass": 2,
"fail": 0,
"unavailable": 0
},
{
"statsId": "9df7c2e0-b911-4901-97b8-2d703a0054f2",
"runId": "663d2750-72dd-4c71-af9a-956650b18f14",
"pass": 0,
"fail": 1,
"unavailable": 0
},
{
"statsId": "0af32693-127a-4e18-a539-c50341dd6e4d",
"runId": "c896e1c4-e91d-4af2-a488-b875224b35a5",
"pass": 0,
"fail": 1,
"unavailable": 0
},
{
"statsId": "6ef2eb29-949e-4ad2-aa7a-939b41ad7876",
"runId": "12dc3118-77e7-4ac7-8dc9-9221348c5378",
"pass": 1,
"fail": 0,
"unavailable": 0
}
]
}
Upload Project Test Suites
Upload your test suites to your project in a ZIP file. This ZIP has to follow the structure as any test suite (for example, tests, .omilia-ci.yml, golden dialogs).
Available only for Projects that have Testing Studio S3 storage enabled.
POST /api/projects/<project_id>/suites/
URL parameters
Parameter | Type | Description | Example |
---|---|---|---|
| uuid | the project’s id | "9b399774-3a0a-40c1-8a1c-4a124170b6f3" |
Request payload
The zip file of your test suites.
Response (Success)
HTTP/1.1 204 NO_CONTENT
Download Project Test Suites
Download your project’s test suites. With this request, you will download the ones that you have already uploaded in a previous step.
Available only for Projects that have Testing Studio S3 storage enabled.
GET /api/projects/<project_id>/suites/download/
URL parameters
Parameter | Type | Description | Example |
---|---|---|---|
| uuid | the project’s id | "9b399774-3a0a-40c1-8a1c-4a124170b6f3" |
Response (Success)
HTTP/1.1 200 OK
Your test suites in a ZIP file.
Runs
Create Run
Create and start a Run job, for example, start running your test suites.
POST /api/runs/
Request Payload
name | Type | Default | Description |
---|---|---|---|
| uuid | - | run’s project id |
| string | null | filter test cases based on their group |
| string | null | filter test cases based on their utterances |
| string | null | filter test cases based on their description |
Example
{
"projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"group": null,
"utterance": null,
"description": null
}
Response (Success)
HTTP/1.1 200 OK
The created Run.
Example
{
"data": [
{
"projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"parallelTestCases": 1,
"group": null,
"utterance": null,
"description": null,
"runId": "e6fa9e9a-5055-4773-9652-309b05e88c49",
"status": "SUCCESS",
"createdAt": "2023-04-11T15:22:50.193226",
"finishedAt": "2023-04-11T15:22:53.568724",
"tasks": [],
"artifact": {
"artifactId": "b96f6c0d-c08f-4454-a624-99564dbf8fb4",
"name": "e6fa9e9a-5055-4773-9652-309b05e88c49_artifacts.zip",
"createdAt": "2023-04-11T15:22:52.916354",
"deletedAt": null,
"ttl": 604800,
"isExpired": false,
"runId": "e6fa9e9a-5055-4773-9652-309b05e88c49"
},
"isRevocable": false,
"executionMeta": null,
"duration": "0:00:03"
}
],
"meta": {
"count": 1
}
}
Get Run
GET /api/runs/<run_id>
URL Parameters
Name | Type | Description | Example |
---|---|---|---|
| uuid | the run’s id | "e6fa9e9a-5055-4773-9652-309b05e88c49" |
Response (Success)
HTTP/1.1 200 OK
Return the Run.
Example
{
"data": [
{
"projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"parallelTestCases": 1,
"group": null,
"utterance": null,
"description": null,
"runId": "e6fa9e9a-5055-4773-9652-309b05e88c49",
"status": "SUCCESS",
"createdAt": "2023-04-11T15:22:50.193226",
"finishedAt": "2023-04-11T15:22:53.568724",
"tasks": [],
"artifact": {
"artifactId": "b96f6c0d-c08f-4454-a624-99564dbf8fb4",
"name": "e6fa9e9a-5055-4773-9652-309b05e88c49_artifacts.zip",
"createdAt": "2023-04-11T15:22:52.916354",
"deletedAt": null,
"ttl": 604800,
"isExpired": false,
"runId": "e6fa9e9a-5055-4773-9652-309b05e88c49"
},
"isRevocable": false,
"executionMeta": null,
"duration": "0:00:03"
}
],
"meta": {
"count": 1
}
}
Get Run Artifact
GET /api/runs/<run_id>/artifacts/
URL Parameters
Name | Type | Description | Example |
---|---|---|---|
| uuid | the run’s id | "e6fa9e9a-5055-4773-9652-309b05e88c49" |
Response (Success)
HTTP/1.1 200 OK
Return the Run’s artifact ZIP file.
Revoke Run
POST /api/runs/<run_id>/revoke/
URL Parameters
Name | Type | Description | Example |
---|---|---|---|
| uuid | the run’s id | "e6fa9e9a-5055-4773-9652-309b05e88c49" |
Response (Success)
HTTP/1.1 204 NO_CONTENT
Get Run Test Suites
GET /api/runs/<run_id>/suites/
URL Parameters
Name | Type | Description | Example |
---|---|---|---|
| uuid | the run’s id | "e6fa9e9a-5055-4773-9652-309b05e88c49" |
Response (Success)
HTTP/1.1 200 OK
Return the list Run Test Suites.
Example
{
"data": [
{
"testSuiteName": "test_suite_1"
},
{
"testSuiteName": "test_suite_2"
}
]
}
Get Run Stats
GET /api/runs/<run_id>/run_stats/
URL Parameters
Name | Type | Description | Example |
---|---|---|---|
| uuid | the run’s id | "e6fa9e9a-5055-4773-9652-309b05e88c49" |
Response (Success)
HTTP/1.1 200 OK
Return the Run Stats.
Example
{
"data": {
"statsId": "9fc2cc39-f533-4557-a844-bc3ee0fd5d0e",
"runId": "e6fa9e9a-5055-4773-9652-309b05e88c49",
"pass": 2,
"fail": 0,
"unavailable": 0
}
}
Get Run Test Cases Stats
GET /api/runs/<run_id>/test_cases/
URL Parameters
Name | Type | Description | Example |
---|---|---|---|
| uuid | the run’s id | "e6fa9e9a-5055-4773-9652-309b05e88c49" |
Response (Success)
HTTP/1.1 200 OK
Return the Run Test Cases.
Example
{
"data": [
{
"numberOfTries": 1,
"executionTime": 889.0,
"testCaseName": "scenario_1",
"dialogId": "iandronis_SND_ia_iandronis_app_1.1681812519019.7db27172adc94a3cbc648c8456f6f796",
"goldenDialogId": "c0bf3e121907d0b2ca2beb219c38926858201a18.1681143584265.83db1d064122481b848adad10fb5c0f3",
"testCaseId": "c43143e9-1ec5-45be-8565-539382a3fd01",
"testCaseDescription": null,
"status": "success",
"assertionErrors": [],
"generalErrors": [],
"warnings": [],
"hadDelays": false
}
]
}
Get Run Logs
GET /api/runs/<run_id>/logs/
$body
URL Parameters
Name | Type | Description | Example |
---|---|---|---|
| uuid | the run’s id | "e6fa9e9a-5055-4773-9652-309b05e88c49" |
Response (Success)
HTTP/1.1 200 OK
Return the Run Log.
Example
{
"data": [
{
"sectionName": "Clone Repository",
"id": "580edef7-667b-44b2-9f29-a05ec254e300",
"content": [
{
"text": "---- Billing sdk is not enabled ----"
},
{
"text": "---- Cloning repository ----"
},
{
"text": "Cloning into '/workspace/f7e3d168-053c-47ff-b592-eea2071ff794_c2a9ec59-0c38-4372-
bff5-83cd7f9c721f_cd933ffa-189b-40fc-857c-5f60071f842f'...\n"
},
{
"text": "Task completed successfully"
},
{
"text": "---- Billing sdk is not enabled ----"
}
],
"offset": 5,
"completed": true
},
{
"sectionName": "Load Configuration",
"id": "66a67222-a620-4dda-981b-dab7ecc182b4",
"content": [
{
"text": "---- Billing sdk is not enabled ----"
},
{
"text": "---- Loading configuration ----"
},
{
"text": "Task completed successfully"
},
{
"text": "---- Billing sdk is not enabled ----"
}
],
"offset": 4,
"completed": true
},
{}
],
"offset": 5,
"completed": true
},
{
"sectionName": "Run Test Suites",
"id": "acfab7f2-244a-480c-9665-367fe5d1e3fa",
"content": [
{
"text": "---- Billing sdk is not enabled ----"
},
{
"text": "---- Running test suites ----"
},
{
"text": "Starting run of suite with name: test_suite_1"
},
{
"text": "\n------------------------------------------"
},
{
"text": "test_suite_1"
},
{
"text": "------------------------------------------"
},
{
"text": "scenario_1 : success [ 889.000 ms ]"
},
{
"sectionName": "Load Test Suites",
"id": "df18ea2a-ebcf-4dcd-8ded-3d5734f973af",
"content": [
{
"text": "---- Billing sdk is not enabled ----"
},
{
"text": "---- Loading test suites ----"
},
{
"text": "Test suites found: 1"
},
{
"text": "Task completed successfully"
},
{
"text": "---- Billing sdk is not enabled ----""text": "Test cases run: 1, passed: 1, failed: 0, unavailable: 0"
},
{
"text": "Task completed successfully"
},
{
"text": "---- Billing sdk is not enabled ----"
},
{
"text": "Sending billing event"
},
{
"text": "Saving test case results"
}
],
"offset": 12,
"completed": true
},
{
"sectionName": "Run Test Suites",
"id": "f1b74254-5d73-466b-84df-40f7b4c50a94",
"content": [
{
"text": "---- Billing sdk is not enabled ----"
},
{
"text": "---- Running test suites ----"
},
{
"text": "Task completed successfully"
},
{
"text": "---- Billing sdk is not enabled ----"
},
{
"text": "Sending billing event"
},
{
"text": "Saving test case results"
}
],
"offset": 6,
"completed": true
},
{},
{},
{
"sectionName": "Run Test Suites",
"id": "0be2004e-2d2c-44db-af07-12659d4df386",
"content": [
{
"text": "---- Billing sdk is not enabled ----"
},
{
"text": "---- Running test suites ----"
},
{
"text": "Task completed successfully"
},
{
"text": "---- Billing sdk is not enabled ----"
},
{
"text": "Sending billing event"
},
{
"text": "Saving test case results"
}
],
"offset": 6,
"completed": true
"sectionName": "Run Test Suites",
"id": "f4f20d08-25fd-4b6c-b016-a5fd528ec263",
"content": [
{
"text": "---- Billing sdk is not enabled ----"
},
{
"text": "---- Running test suites ----"
},
{
"text": "Task completed successfully"
},
{
"text": "---- Billing sdk is not enabled ----"
},
{
"text": "Sending billing event"
},
{
"text": "Saving test case results"
}
],
"offset": 6,
"completed": true
"sectionName": "Aggregate Results",
"id": "7a976057-cff8-4002-8f1b-8acad4712584",
"content": [
{
"text": "---- Billing sdk is not enabled ----"
},
{
"text": "---- Handling artifacts -----"
},
{
"text": "Reporting to csv"
},
{
"text": "Finished reporting to csv"
},
{
"text": "Creating artifacts .zip file"
},
{
"text": "Saving artifacts .zip file"
},
{
"text": "Task completed successfully"
},
{
"text": "---- Billing sdk is not enabled ----"
},
{
"text": "Performing final steps"
},
{
"text": "Performing clean up"
},
{
"text": "Deleting /workspace/f7e3d168-053c-47ff-b592-eea2071ff794_c2a9ec59-0c38-4372-bff5-83cd7f9c721f_cd933ffa-189b-40fc-857c-5f60071f842f"
}
],
"offset": 11,
"completed": true
}
],
"completed": true
}
Generation Runs
Create Test Case Generation Run
POST /api/generations/
Request Payload
Parameter | Type | Default | Description |
---|---|---|---|
| string | - | project’s id to create run for |
| string | null | group to entered to newly generated test cases |
| string | null | comma separated test suite names for which test cases will be be generated |
| string | false | whether to overwrite existing test cases or not |
| string | null | comma separated simulation data to be added during generation |
Example
{
"overwrite": false,
"projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f", "simulationData": "Ani,Dnis"
}
Response (Success)
HTTP/1.1 200 OK
The created Generated Test Case.
Example
{
"data": {
{
"projectId": "c2a9ec59-0c38-4372-bff5-83cd7f9c721f",
"group": null,
"testSuite": null,
"overwrite": true,
"simulationData": null,
"runId": "ee6a56f0-1d2b-4266-bbe8-d4374156c400",
"status": "SUCCESS",
"createdAt": "2023-04-10T14:24:39.345269",
"finishedAt": "2023-04-10T14:24:41.812109",
"tasks": [],
"artifact": {
"artifactId": "86828048-9e69-4f3f-a053-80daad7ecd66",
"name": "ee6a56f0-1d2b-4266-bbe8-d4374156c400_artifacts.zip",
"createdAt": "2023-04-10T14:24:41.144837",
"deletedAt": null,
"ttl": 604800,
"isExpired": true,
"runId": "ee6a56f0-1d2b-4266-bbe8-d4374156c400"
},
"isRevocable": false,
"executionMeta": null
}
},
"meta": {
"count": 1
}
}
Get Test Case Generation Run Artifact
GET /api/generations/<generation_id>/artifacts/
URL Parameters
Name | Type | Description | Example |
---|---|---|---|
| uuid | the run’s id | "e6fa9e9a-5055-4773-9652-309b05e88c49" |
Response (Success)
HTTP/1.1 200 OK
Return the Generated Test Cases artifact ZIP file.
Dialogs
Generate Golden Dialog
GET /api/dialogs/<dialog_id>/golden/
URL Parameters
Name | Type | Description | Example |
---|---|---|---|
| uuid | the dialog’s id | "76a070229d046b5d2bdb0bd5cc36b70f37d760de. 1681819138546.b94bd85ea868402eb82cb42446a473e3" |
Response (Success)
HTTP/1.1 200 OK
Return the generated golden dialog json file.