OCP® Reporting User Guide

Overview

OCP® Reporting is an out-of-the-box reporting and analytics platform available for all conversations through OCP® applications. These involve Orchestrator Application dialogs, Flows, and miniApps innovations originating from calls or web chats.

OCP Reporting® provides rich, graphical dashboards with key performance metrics in order to pinpoint opportunities for continuously improving the Virtual Assistant and gaining a deeper insight into what customers are saying. The reports are unsupervised as they come directly from the system without any analysis from a human.

To access OCP® Reporting, go to Insights → Reporting in the side panel.

Reporting Components

The OCP® Reporting is divided into two main components:

Real-time Data reports put emphasis on near real-time reporting.
Conversational Insights offers more analytical historical metrics.

Real-time Data

Real-time Data shows the necessary information about the current status of your miniApp. They are created from a specific selection of existing batch metrics, targeted to ensure the user is informed about the current state of their system, without being overwhelmed with information.

To produce near real-time reports on top of the underlying data, the Data Warehouse infrastructure is used. Reporting metrics are broken down into 5-minute intervals and updated near real-time, with an additional watermark of 5 minutes. This means any data delay (due to external services) can be captured even with up to a five-minute delay. These metrics are publicly available through the Metrics API.

The Real-time Data is updated automatically every 60 seconds.

You can refresh the results at any time by clicking the Refresh button as shown below.

Real-time Data graphs

The following information is available in the Real-time Data:

Sessions

The Sessions report shows the number of total sessions the application received.

Sessions refer to either the number of unique IVR calls or chat sessions (depending on the application), no matter how many miniApps, or Orchestrator Flows were triggered. For the session counter to increase, at least one miniApp must be triggered within OCP®.

Dialogs

The Dialogs report is applicable for Orchestrator applications and refers to the number of dialogs per Orchestrator application. A dialog is a unique conversation (call or chat) within an OCP® application.

The following metrics are available:

Completed: Number of completed dialogs. Includes both terminated and transferred dialogs, as well as edge cases, when the dialog end is unknown.
Transferred: Total number of dialogs that got transferred to a live agent.
Terminated: Total number of dialogs that got terminated by the user or the system.

miniApp Invocations

The miniApp Invocations report is applicable for OCP® miniApp applications. Each counter is increased after the compilation of each miniApp task.

The following metrics are available:

Total: The number of total invocations (tasks) originating from miniApps. This includes successful miniApps, miniApps that ended with system errors, as well as miniApps that ended with any other failed reason like user termination or invalid input.
Successful: The number of total successful miniApps invocations.
System Errors: The number of invocations ended with a system error, critical error, or timeout error only.

miniApp Events

The miniApp Events report displays a total number of No Matches and No Inputs per miniApp.

No Matches and No Inputs are applicable for those miniApps only that expect a user input.

No Matches: Total number of steps with no match. A single miniApp may have multiple steps with no matches. This is a per-step metric.
No Inputs: Total number of steps with no input. A single miniApp may have multiple steps with no inputs. This is a per-step metric.
Critical Error: Number of Critical Error events for each application. A Critical Error event may affect one or multiple Applications within the same call.

End type reports

The End type reports show Dialog Termination and Transfer Rates over Total Calls.

The following metrics are available:

Terminated: The percentage rate of the terminated calls over the total number of calls.
Transferred: The percentage rate of the transferred calls over the total number of calls.

miniApp Success rate

The miniApp Success rate displays the percentage of miniApps Successful Invocations and System Errors over Total miniApp Invocations.

The following metrics are available:

Success: The percentage rate of successfull miniApp invocations over the total number of miniApp invocations.
System Errors: The percentage rate of miniApp invocations containing system errors over the total number of miniApp invocations.

Real-time Data filters

The data results for real-time data are based on the filters set in the Filters menu. The search filter allows you to easily target specific applications or miniApps within OCP®.

Not all filters are applicable for all reports!
For example, Session reports may be selected by Language. However, this filter is not applicable for miniApp Invocations report, as the same session may reach multiple OCP® applications.

To open the search filters, proceed as follows:

Click the Filters button.

Fill in the search filter fields as per your needs.

Group: The OCP group that the metrics refer to. Only one group may be selected each time.
Name: The name of the miniApp for the miniApp Invocations report or Orchestrator application for the Dialogs report.
Type: The type of the miniApp shown in the reports. For Orchestrator applications, the type is always Flow. Not applicable for Session reports.
Channel: The channel of the user system interaction (for example, Voice or Chat). Applicable for all reports.
Language: The selected language of the user system interaction (for example, English US).
Test Flag: Select true for fetching only test-flagged calls (or char sessions). To exclude test-flagged calls, select false. By default, all calls are reported. Applicable for all reports.
Time: Select the time range visible in reports.

Click the View button to get the real-time data results.

Downloading data

You can download each report locally as a CSV file by clicking the Export Chart data button next to each graph.

Selecting metrics

You can exclude / include certain metrics from the report by checking / unchecking the checkbox next to each metric name as shown below.

Viewing metrics details

Click the Expand button to see a chart in full-screen mode.

To view the metrics details, proceed as follows:

Hover over the graph to see the metric’s time.

2. Click on a dot with a specific time.

3. Click the Show Actual Dialogs button. You will be redirected to OCP® Monitor to review the dialog IDs corresponding to the reported metric.

Conversational Insights

OCP Conversational Insights® provides rich, graphical dashboards with key performance metrics and come directly from the system without any analysis from a human.

There are several dashboards that help you identify how your applications are performing and point out possible improvements:

Total Sessions

The Total Sessions interactive visualization shows the number of sessions that a customer is receiving for the selected period. Sessions refer to unique calls or chat sessions, regardless of how many applications were triggered in between.

Avg Invocations per Session Daily

The Avg Invocations per Session Daily graph connects sessions to miniApp invocations. Presents how many miniApps are triggered during an average call or chat session.

Conversational Outcome

The Conversational Outcome dashboard allows observing number of total dialogs per day.

The data is presented both in numbers and percentages.

Completed - how many dialogs have been completed by the user or system. Includes both terminated and transferred dialogs, as well as edge cases, when the dialog end is unknown.
Terminated - how many dialogs have been terminated by the user or system without getting transferred and no subflow (self-service) was completed successfully.
Transferred - how many dialogs have been transferred out of the system, which means, were not terminated.
Errors - how many dialogs have been completed with an error.
Other - dialogs which are not classified in any of the above cases. These are not necessary errors, but accounts for possible edge cases of dialogs for which there wasn’t enough information received to determine the exact termination type.

miniApps Invocations

A graphical representation of the total invocations during the selected period of time as well as the percentage of the successful against the unsuccessful due to fail or error.

The OCP miniApp Success Rate graph shows the success and failure rates of the selected miniApps. This is the task completion rate of each miniApp, which breaks down in the Overall miniApp Success Rate graph to:

Successful Invocations - the percentage of successful invocations over the total number of invocations on a daily basis.
Failed Invocations - the percentage of failed invocations over the total number of invocations on a daily basis.
Terminated - the percentage of terminated (either by the user or by the system) invocations over the total number of invocations on a daily basis.
DTMF Escape Key - the total number of invocations within which the user opted out using the DTMF escape key over the total number of invocations on a daily basis.
Errors - the total number of invocations with at least one system error over the total number of invocations on a per-day basis.
Other - a final categorization that captures all invocations not falling in the above categories. These may include yet unclassified errors or edge cases.

miniApps Results

A graphical representation of the the miniApp results of all the invocations within the selected period. The Multi-line graph is customizable and is possible to select to view only the desired results. Click on the checkboxes to remove or add back a task category.

Conversational Insights Filters

The data results for Conversational Insights are based on the filters set in the corresponding Filters menu. The search filter allows you to easily target specific applications or miniApps within OCP®.

To open the search filters, click the corresponding Filters button.
Fill in the search filter fields as per your needs.
- Date range: Select the date range visible in reports. You can select +Add comparison to add a 2nd date range to compare.
- Group: The OCP group that the metrics refer to. Only one group may be selected each time.
- Name: The name of the miniApp for the miniApp Invocations report or Orchestrator application for the Dialogs report.
- Type: The type of the miniApp shown in the reports. For Orchestrator applications, the type is always Flow. Not applicable for Session reports.
- Channel: The channel of the user system interaction (for example, Voice or Chat). Applicable for all reports.
- Language: The selected language of the user system interaction (for example, English US).
- Test Flag: Select true for fetching only test-flagged calls (or char sessions). To exclude test-flagged calls, select false. By default, all calls are reported. Applicable for all reports.
Click the View button to view the updated reporting results.
Change the granularity of the results to be daily or hourly with the + and - buttons. (+ increases interval to hourly results)

Advanced Conversational Insights

More detailed advanced metrics for the full application history may be found in OCP Conversational Insights®.

Metrics API

Insights Report Metrics, with corresponding filters, are also available through the Metrics API.