Exports API
Data Compliance
Omilia OCP® Data Warehouse and Export API comply with General Data Protection Regulation (GDPR) requirements. For any data type exported from OCP, through Export API, the OCP users are responsible to ensure that their exported data are handled accordingly and meet necessary regulations.
Overview
Export Service is a dedicated service that allows OCP users to gain access to their raw application data.
Export API Includes two different functionalities: Batch Export and Stream Export. For billing information about Streaming Export, please coordinate with your Account Manager
Read more about the Export Service features in the Export Service User Guide.
Export API is crucial for OCP users who want to retrieve, export, and manage their raw application data within the specified constraints and authentication mechanisms. Here is a brief overview of what Export API could assist with:
Access to Raw Application Data: The Export Service allows users to access their raw application data.
Exporting Call Detail Records (CDRs): The API supports the export of Call Detail Records (CDRs) for a specific time range. The API allows users to request CDRs for specific types of data, such as "Dialog Data," which includes Dialog Step, Dialog Start, and Dialog End.
Export Creation: Users can create exports using the API by making a request to the specific endpoint.
Export Status and Expiry: After a successful export creation, the exported data is stored for one day. The data is deleted from the API after this period, and the status of the export is updated to expired.
Export Metadata and Data Retrieval: Users can retrieve metadata of a specific export and list export data using the respective GET endpoints. The export data retrieval process is paginated, involving multiple requests to access all files.
Rate Limiting: The API has a rate limiting mechanism to control the number of requests per group within a specific time window. Rate limiting can be configured using parameters such as the number of requests permitted, the length of the fixed window, and the TTL for cache entries.
Exports API is organized around HTTP REST. Our API allows you to request CDRs as exports for specific time range. The maximum date range supported is two days. The smallest requested date range is one hour (for example, 2023-01-12T12:00:00.000Z to 2023-01-12T13:00:00.000Z).
Minutes/seconds/milliseconds will not change the export (example, 2023-01-12T12:00:00.000Z to 2023-01-12T13:00:00.000Z is equal to 2023-01-12T12:10:00.000Z to 2023-01-12T13:20:00.000Z).
Here are the key reasons for exporting data:
Retention Policy: Omilia enforces a 60-day retention policy; to prevent data loss, it is imperative to export and store the data in your individual database.
Flexible Data Transfer: Seamlessly transfer data to a library of your choice, ensuring immediate and unrestricted access for your specific requirements.
Custom Metric Development: Empower your analytics by creating personalized metrics tailored to your unique business needs, leveraging the exported data for comprehensive insights and reporting capabilities.
Each group can run one export at a time. After the successful creation of the export, the exported data is stored for one day, after this period the data is deleted from the API and the status of the export is updated to expired.
We use built-in HTTP authentication to validate users based on groups/roles stored in IAM.
Sequence Diagram
Authentication
Navigate to the API Authentication to learn more about Authorization Token Endpoint and Authentication process.
HTTP Code Responses
Omilia's HTTP response codes can be found on the HTTP Code Responses page.
API Reference
{{baseUrl}} depends on the user's environment which could be the following:
EU1 - https://eu1-a.ocp.ai
US1 - https://us1-a.ocp.ai
US2 - https://us2-a.ocp.ai
Create Export
Request
POST {{baseUrl}}/exports-api/v1/groups/{{group}}/jobs
{
"from_date": "2023-01-12T13:00:00.000Z",
"to_date": "2023-01-12T14:00:00.000Z",
"types": [
"Dialog Data"
],
"export_name": "Jan-12-2023-1300-Export"
}
Headers
Parameter | Type | Description |
---|---|---|
| String | Set it to |
Path Parameters
Property | Type | Required | Description |
---|---|---|---|
| string | Yes | The group (organization) the export requested belong to. |
Body Parameters
Property | Type | Required | Description |
---|---|---|---|
| string | Yes | Name of specific export (^[a-zA-Z0-9-][a-zA-Z0-9-\\s]+$). The export name is selected by the exporter. |
| DateTime(UTC) | Yes | Timestamp before |
| DateTime(UTC) | Yes |
|
| array(string) | Yes | The types of data needed for export
|
Response
A successful request returns the HTTP 200 status code and a JSON response body that shows the export id created (UUID) and a description field (String) with success.
Example
{
"description": "Task Submitted",
"export_id": "73bd903e-e5c5-4b87-a9a9-6a35e45af48f"
}
List Exports
Request
GET {{baseUrl}}/exports-api/v1/groups/{{group}}/jobs
Path Parameters
Property | Type | Required | Description |
---|---|---|---|
| string | Yes | The group (organization) that the list export refers to. |
Query Parameters
Property | Type | Required | Description |
---|---|---|---|
| Integer | No | Integer > 0 , default value = 1 |
| Integer | No | 100 >= Integer > 0 , default value = 50 |
| string | No | The types of data needed for export
|
Response
A successful request returns the HTTP 200 status code and a JSON response body that shows a list of created exports from the specific group.
Example
{
"pagination": {
"pages": 1,
"page_number": 1,
"page_size": 2,
"total_results": 2
},
"exports": [
{
"name": "KT_dw_qa_2_AwsDev",
"status": "EXPIRED",
"group": "dw_qa_2_AwsDev",
"export_id": "3342d2f8-2279-43a3-b4e9-082ceed19a68",
"time_submitted": "2023-01-13T09:54:58.192Z",
"from_date": "2023-01-13T05:00:00Z",
"finished_date": "2023-01-13T09:55:50.57Z",
"to_date": "2023-01-13T06:00:00Z",
"types": [
"Dialog Step",
"Dialog Start",
"Dialog End"
]
},
{
"name": "tests",
"status": "EXPIRED",
"group": "dw_qa_2_AwsDev",
"export_id": "b52d8a2c-358f-486d-a464-f89dbf2eaa9b",
"time_submitted": "2023-01-05T12:19:54.061727Z",
"from_date": "2022-12-05T00:00:00Z",
"finished_date": "2023-01-08T18:00:01.784466Z",
"to_date": "2022-12-05T23:00:00Z",
"types": [
"Dialog Step"
]
}
]
}
GET Export Metadata
In order to GET export data, exports must be in status READY.
Request
GET {{baseUrl}}/exports-api/v1/groups/{{group}}/jobs/{{export_id}}
Path Parameters
Property | Type | Required | Description |
---|---|---|---|
| string | Yes | The group (organization) that the list export refers to. |
| UUID string | Yes | A valid UUID string referencing a created export. |
Response
A successful request returns the HTTP 200 status code and a JSON response body that shows metadata of the specific export.
Example
{
"export_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"num_of_files": 0,
"size_of_export": 0
}
List Export Data
In order to GET export data, exports must be in status READY.
Request
GET {{baseUrl}}/exports-api/v1/groups/{{group}}/jobs/{{export_id}}/data/compressed
Path Parameters
Property | Type | Required | Description |
---|---|---|---|
| string | Yes | The group (organization) that the list export refers to. |
| UUID string | Yes | A valid UUID string referencing a created export. |
Query Parameters
Property | Type | Required | Description |
---|---|---|---|
| Integer | No | Integer > 0, default value 1 |
| Integer | No | 50 >= Integer > 0, default value 20 |
To acquire the data from an export containing several files (information on the number can be found in the "GET Export Metadata" endpoint response body, or the response headers of the "LIST response data" endpoint), there are several approaches.
For example:
For an export containing 160 files, you can either access the specific endpoint 8 times, utilizing the default page size of 20 and incrementing the page number by 1 with each request (ranging from 1 to 8), or you can access the endpoint 4 times, using a page size of 50 and advancing sequentially from page numbers 1 to 4.
Essentially, what this means, is that the export process is paginated and consequently, it involves making multiple requests.
Response
A successful request returns the HTTP 200 status code with pagination HTTP headers and initiates a streaming response that when finished, a zip file is created and downloaded.
The HTTP headers you will be dealing with involve the following:
Pagination-pages
: This refers to the total number of accessible pages.Pagination-page_number
: Indicates the specific page you're getting.Pagination-page_size
: This represents the quantity of files downloaded with this request.
Example
More info regarding the ZIP file and the information contained can be found in Export Service User Guide and CDR Models Schema.
Exports Supported
Currently the API supports only CDR exports. The CDRs have 3 different types: dialog_start
, dialog_step
, dialog_end
. For more information regarding CDRs please read the CDR Models Schema page.
Stream Export (Subscriptions)
Subscription functionality within the Export Service of Omilia OCP® empowers users to subscribe to message channels, facilitating the receipt of near real-time data streams. Read more about how to use subscriptions in the Export Service User Guide.
For Export Subscriptions please contact your Account Manager.
Streaming Exports API allows clients to subscribe to message channels and receive streams of data in a near real-time manner. The message channels are organised by data type. Currently, the following data types are supported:
Dialog data
dialog_start
dialog_end
dialog_step
Audit data
audit_events
Streaming Exports API is organized around HTTP REST for the management of the subscriptions. Although it aims to support multiple delivery methods, currently the streams are delivered via HTTP requests sent to the client’s public API gateway.
Each group is allowed to have only one running streaming export at a time for each message type (for example, dialog_step
, dialog_start
). After the successful creation of the export, the data start streaming in the clients provided callback path.
We use built-in HTTP authentication to validate users based on groups/roles stored in IAM.
Currently Audit_events
may be available on particular environment. Reach out to your Account Manager for more details.
Exports access token creation
The access token is a JWT generated by Omilias IAM server and exports-api will use to validate the user against the same server and check user’s whitelist of groups.
Whitelisted groups should be added as attributes assigned to the user or to a group which the user belongs to.
After creating the user, the client should obtain access token from IAM server via its API.
cURL Example Request
$ curl -v -X POST '{iam-url}/auth/realms/master/protocol/openid-connect/token' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {access_token}' \
--data-raw '{
"username": "api_test_user1",
"password": "omilia",
"grant_type": "password",
"client_id": "ocp"
}'
Response
A successful request returns the HTTP 200 status code and a JSON response body that contains the access token to use with exports-api.
Sequence Diagram
API Reference
Create Streaming Export
Request
POST {{baseUrl}}/exports-api/v1/groups/{{group}}/subscriptions
{
"data_type": "dialog_start",
"schema_name":"my_ocp_start_cdrs",
"delivery":{
"type": "http",
"path": "http://wiremock_wiremock:8080/wiremock/data",
"auth_type": "oauth2",
"auth_id": "test-*****",
"auth_secret": "BLdnUUDbtgy0YdFdN*****",
"token_path": "https://aws-dev-m.ocp.ai/auth/realms/master/protocol/openid-connect/token"
}
}
Path Parameters
Property | Type | Required | Description |
---|---|---|---|
| string | Yes | The group (organization) the export requested belong to. Possible values: |
Body Parameters
Property | Type | Required | Description |
---|---|---|---|
| string | Yes | type of specific subscription (for example, dialog_start,dialog_step, dialog_end, audit) |
| string | Yes | The schema name used for outbound calls (for example, ocp_cdr_start, ocp_cdr_end, my_ocp_start_cdrs) |
| object | Yes | The information delivery |
| string | Yes | http,https |
| string | Yes | The callback path, the customer will get the data to |
| string | Yes | oauth2 |
| string | Yes | The user id used to retrieve oauth2 jwt token from client |
| string | Yes | The user secret used to retrieve token from client |
| string | Yes | The user’s uri to use in order to get the token |
Response
A successful request returns the HTTP 200 status code and a JSON response body that shows the streaming export id created (UUID) and a description field (String) with success.
Example
{
"description": "Streaming subscription Submitted",
"export_id": "c545eaac-dad5-44fc-8d6c-77db67cc5235"
}
Create Subscription cURL Example Request
curl --location '{{baseUrl}}/exports-api/v1/groups/{group}/subscriptions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {access_token}' \
--data '{
"data_type": "dialog_start",
"schema_name":"my_ocp_start_cdrs",
"delivery":{
"type": "https",
"path": "https://randomclientsapipath.com/wewaitfordata",
"auth_type": "oauth2",
"auth_id": "dummyid",
"auth_secret": : "my_auth_secret",
"token_path": "get_authentication"
}
}'
List streaming Exports subscriptions
Request
GET {{baseUrl}}/exports-api/v1/groups/{{group}}/subscriptions
Path Parameters
Property | Type | Required | Description |
---|---|---|---|
| string | Yes | The group (organization) that the list export refers to. Possible values:
|
Query Parameters
Property | Type | Required | Description |
---|---|---|---|
| Integer | No | Integer > 0 , default value = 1 |
| Integer | No | 100 >= Integer > 0 , default value = 50 |
Response
A successful request returns the HTTP 200 status code and a JSON response body that shows a list of created streaming exports subscriptions from the specific group.
Example
{
"pagination": {
"pages": 1,
"page_number": 1,
"page_size": 3,
"total_results": 3
},
"exports": [
{
"data_type": "dialog_step",
"schema_name": "my_ocp_dialog_step",
"created_at": "2024-01-08T12:47:12.268328Z",
"subscription_id": "c545eaac-dad5-44fc-8d6c-77db67cc5235",
"delivery": {
"type": "http",
"path": "http://wiremock_wiremock:8080/dw_qa_1_AwsDev/data"
}
},
{
"data_type": "dialog_end",
"schema_name": "my_ocp_dialog_end",
"created_at": "2024-01-08T12:47:12.268328Z",
"subscription_id": "6e0dc618-01e5-4f7e-a91a-8ad01eb26355",
"delivery": {
"type": "http",
"path": "http://wiremock_wiremock:8080/dw_qa_1_AwsDev/data"
}
},
{
"data_type": "dialog_start",
"schema_name": "my_ocp_dialog_start_random_name",
"created_at": "2024-01-08T12:47:12.268328Z",
"subscription_id": "a59a3b0f-4d20-432c-9766-d87ab77b1410",
"delivery": {
"type": "http",
"path": "http://wiremock_wiremock:8080/dw_qa_1_AwsDev/data"
}
}
]
}
Export Streams Example
The events have the corresponding subscription type inside messages. In this example, the type is the start of the CDRs JSON files whilst "my_ocp_start_cdrs"
is the name selected for the export subscription.
{
"schemaName": "my_ocp_start_cdrs",
"sessionId": "5c2b9dc1-899a-4124-97f5-b7e3e02a0660",
"sessionStartingSequenceNumber": "0",
"messages": [
{
"app":
...
"message_type": "dialog_start",
...
},
{
...
},...
]
}
Delete subscriptions by ID
Request
DELETE{{baseUrl}}/exports-api/v1/groups/{{group}}/subscriptions/{{export_id}}
Path Parameters
Property | Type | Required | Description |
---|---|---|---|
| string | Yes | The group (organization) that the list export refers to. Possible values:
|
| uuid | Yes | The export_id to be deleted. Possible values:
|
Response
A successful request returns the HTTP 200 status code and a JSON response body with a message success.
Example
{
"message": "success"
}
Reconciliation Process
Exports-api offers a reconciliation process for messages that were dropped due to customers API unavailability or other kind of failures.
Reconciliation process can fetch up to 24 hours of data, meaning it can run even if the client’s API is down for up to 24 hours. The reconciliation mechanism is being triggered automatically, approximately within 10 minutes, after the clients API is back online. The reconciliation process will run on the background to fetch older data, while the stream continues to send newly generated data. Newly generated data are handled with priority over older data, but the streams of old and new data run in parallel. The reconciliation time may vary depending on both current stream load, and amount of older data that needs to process, bounded by client’s API rate limiting requirements.
For cases that the clients' API was down more than 24 hours, we recommend using the batch process to fetch older data.
Rate Limiting
Exports-api has a rate limiting mechanism configured by its configuration file which applies a threshold of requests per group for a specific time window. The algorithm used is a token-bucket applied over a fixed window of time. This means that for a specific window a certain amount of tokens are produced for each group and they’re consumed as requests are landing on each endpoint. One limit per group applies for all endpoints. If the limit is exceeded, the API will respond with an error response of Status Code 429
(Too many requests). The API will again allow requests for this group after the bucket will refill with tokens in the next fixed request window. The status of the buckets are kept in a memory cache. In order to prevent the cache from growing indefinitely, there is a Time To Live (TTL) for each entry. Entries will be evicted if the TTL of the entry is exceeded (The TTL value configured should be more than the request window).
After eviction, a new cache entry will be created if a new request arrives.