Advanced OCP Conversational Insights®

To access Advanced OCP Conversational Insights®, contact the Customer Support team for additional access rights.

In this guide, we are going to take a look at the default analytics that OCP® offers.

OCP Conversational Insights® provides rich, graphical dashboards with key performance metrics and come directly from the system without any analysis from a human.

There are several dashboards that help you identify how your applications are performing and point out possible improvements.

Below you can find information on each dashboard:

Accessing the Advanced Conversational Insights®

To access the Advanced Conversational Insights in OCP, follow these steps:

Navigate to Insights on the OCP menu.
Select Reporting.
Within the Reporting section, click on Conversational Insights.
Within the Conversational Insights section, click on Advanced Conversational Insights®.

Sessions Metrics

The Sessions Metrics dashboard contains the following graphs:

The number of received Sessions Per Day for the selected period.
The 10sec Increments per Day graph breaks down the duration of the sessions into 10-sec increments and computes the average of all sessions on a daily basis.
The Avg Invocations per Session Daily graph connects sessions to miniApp invocations.
The Grand Totals board on the bottom right sums up in numbers the results presented in the previous graphs.

The filters on the top of the view can be used to adjust the visualizations by:

Date - the date when the session took place.
Weekday - the day of the week when the session took place.
Organization - the group that the miniApps and Flow resources are assigned to when created in an OCP® instance.

Conversational Outcome

The Conversational Outcome dashboard allows observing the following key concepts:

Number of total Dialogs Per Day.
Number of dialogs Contained/Abandoned/Transferred/Error/Other (End Type Distribution Per Day).
Average Avg No Match & No Inputs per Dialog per day.

The Dialogs per Day graph shows the total number of calls per day. As you can see from the example above, the traffic on Sundays is much lower than on any other day.

The Traffic Peak Hours graph in the upper right corner displays when the system is getting the highest call volume.

The Overall Performance graph below it reveals the most important factors that influence the overall performance of the miniApps. The data is presented both in numbers and percentages.

Contained - how many dialogs have been terminated by the user or system without getting transferred, and at least one subflow (self-service) was completed successfully.
Abandoned - how many dialogs have been terminated by the user or system without getting transferred, and no subflow (self-service) was completed successfully.
Transferred - how many dialogs have been transferred out of the system, meaning they were not terminated.
Errors - how many dialogs have been completed with an error.
Other - dialogs not classified in any of the above cases (should be zero). If not zero, these cases need further investigation.

The Avg No Match/No Inputs per Dialog graph contains the data on when and how many corresponding incidents took place.

The End Type Distribution per Day report provides the percentage of calls that have been either transferred, contained, abandoned, or ended due to an error within 24 hours.

You can also apply filters located above the graphs to view more precise data:

The available filters include the following:

Date - the date when the calls took place.
Weekday - the day of the week when the call took place.
OCP miniApp Name - the name of the miniApp as defined when the resource was created in OCP®. One miniApp can be invoked in several different conversations in an OCP® application. This filtering will adjust the view to show exactly these. This name differs from the id (the unique identifier) and is always unique per organization.
Organization - the group that the miniApp is assigned to when created in an OCP® instance.
Channel - the channel through which the conversation occurred.
Language - the language of the miniApps invoked by the dialogs under review. A miniApp is multilingual, so this filter can work by itself without adjusting the OCP miniApp Name at the same time.

Self Services Success Rate

This report shows the total success rate of the application’s self-services:

Self Services Requests Per Day - a number of total self-services. A self-service may invoke several underlying miniApps.
Self Services Success Rates Per Day - a number of successful self-services over a total number of flows in a day. By default, a self-service is successful if all its underlying miniApps are successful. Custom logic on top of this can be applied in the Orchestrator.
Self Services Distribution - for which flows users reached self-services.
Self Services Success Rates - a number of successful self-services over a total number of flows.
Self Services Fail Reasons - reasons why self-services have failed, including but not limited to MaxAgentRequests, MaxContinuousError, CriticalError, MaxNoInputs, MaxNoMatches, and Custom Fail Reason, among others.
Services Fail Origin:miniApp Name - the name of the miniApp that wasn’t successful.
Services Fail Origin:miniApp Type - the type of the miniApp that wasn’t successful.

Filters available to adjust the views include the following:

Day of Dialogdate - the date range for which the data is displayed.
WeekDay - the day of the week when the self-services took place.
SelfServices IDs - the ID(s) of the self-services in question.
SelfServices Names - the name(s) of the self-services in question.
Application - the application(s) being reviewed.
Language - the language of the invoked self-services under review.
Channel - the channel through which the conversation occurred.
Organization - the group that the self-services belong to.
Reporting Type - the type of reporting (selfServe/authentication).

Task Completion Rate

The Tasks per Day graph shows the total miniApp invocations on a daily basis. The embedded black line corresponds to the successful invocations on each day.

The OCP Task Success Rate graph shows the success and failure rates of the selected miniApps. This is the task completion rate of each miniApp, which breaks down in the Overall Task Success Rate visualization located to the right of the mentioned graphs:

Success - the percentage of successful invocations over the total number of invocations on a daily basis.
Fail - the percentage of failed invocations over the total number of invocations on a daily basis.
Terminated - the percentage of terminated (either by the user or by the system) invocations over the total number of invocations on a daily basis.
DTMF Escape Key - the total number of invocations, within which the user opted for using the DTMF escape key over the total number of invocations on a daily basis.
Errors - the total number of invocations with at least one system error over the total number of invocations on a daily basis.
Other - a final categorization that captures all invocations not falling in the above categories. These may include yet unclassified errors or edge cases.

The Fail Reasons daily dashboard shows the fail reasons for all the failed miniApps invocations distributed on a daily basis together with the cumulative percentages spanning over the selected time period.

Above all the graphs, you can see the number of Total Invocations of miniApps, as well as an average duration Avg Duration of each miniApp invocation. A table on the right of the dashboard shows a summary of all the metrics with filters available for more precise analysis.

Date - the date when the invocations took place.
Weekday - the day of the week when the invocations took place.
Organization - the group that the miniApp is assigned to when created in an OCP® instance.
miniApp ID - the unique ID of a miniApp.
MiniAppCategories - the category to which a miniApp belongs.

You can also filter the displayed data according to specific parameters available above the graphs:

miniApp Name - the names of the miniApps to be reviewed.
miniApp Type - the type of a miniApp: Alphanumeric, Numeric, Intent, and so on.
Language - the language of the miniApps.
Channel - the channel through which the conversation occurred.
Date Range Slider - the date range for which the data is displayed.

Intents & Utterances

This dashboard analyzes the application’s Intents distribution as they are recognized by the corresponding miniApps on a daily basis. Most frequent utterances are bound with each intent.

The metrics shown in the report are as follows:

Intents - total Intents distribution cumulated over the selected time period.
Intent Daily Distribution - the distribution percentage of each intent on a daily basis.
Top 10 Utterances per Intent - top 10 utterances per intent together with the distribution for the last 31 days.
Top Utterances - top utterances in total for the last 31 days.
Top 10 Utterance Daily Distribution - the distribution of the top 10 utterances on a daily basis.

Above the graphs, you can find the following filtering options:

Date - the date range for which the data is displayed.
Organization - the group that the miniApp is assigned to when created in an OCP® instance.
OCP miniApp Name - the names of the miniApps to be reviewed.
Weekday - the day of the week when the conversation took place.
Channel - the channel through which the conversation occurred.
Language - the language of the miniApps.
OCP miniApp Type - the type of a miniApp: Alphanumeric, Numeric, Intent, and so on.
OCP miniApp ID - the unique ID of a miniApp.
Intent Names - the names of the intents to filter by if needed.

Custom Metrics

The Custom Metrics reporting shows custom loggings of an Orchestrator application as they are set through the OCP Console® UI.

The goal is to enable the tracking of specific custom events that are necessary for the monitoring of the application. These events are key-value pairs and are logged through the OCP Console® UI. Their various combinations are aggregated in the underlying data warehouse engine on a daily basis and are served in the visualization layer.

The Metrics Distribution graph displays the frequency of logging specific metrics. The Values Distribution visualization shows the corresponding values related to these metrics.

The Metrics Frequency Per Day report report displays the correlation between the frequency of specific custom loggings on a daily basis. The Values Daily Distribution graph shows the occurrence of logged values per day.

The Number of Attempts graph in the right bottom corner shows how many times the same custom logging has appeared within the same miniApp during the same invocation.

You can adjust the view using the following filters:

Day of dialogdate (at the top) - the date range for which the data is displayed.
Metrics (to the right) - can be used to focus on specific metrics or their combinations.
Channel - the channel through which the conversation occurred.
Language - the language of the miniApps.
Organization - the name of the organization.
Flow Name - the name(s) of the Flows to be reviewed.
Flow ID - the ID(s) of the flows to be reviewed.
Attempts - how many times the same custom logging has appeared within the same miniApp during the same invocation.

Self-Services Contribution

The Self-Services Contribution dashboard displays the call level success rate of the self-services of the application. Also, it shows the contribution “impact” of each self-service in the conversation outcome (Transferred vs. Terminated). The same self-service may be triggered multiple times, so this Dashboard focuses on calls that contain self-services at least once.

This dashboard contains the following graphs:

Contribution per Self Service - the contribution percentage of the self-services reviewed.
Avg Contribution Rate - a pie chart for average self-services contribution and non contribution compared to the number of total calls.
Avg Daily Contribution Rate - the contribution of self-services on a daily basis.
SelfService End Type Distribution - the distribution of Terminated (contributed) and Transferred (non contributed) calls with at least one self-service.
Avg Completion Rate Call Level - the average number of completed calls (with at least one successful self-service) compared to the number of total calls.
Self Service Completion Rate Call Level - a table displaying results of each self-service sorted by name, including Fail, Success, and Grand Total result percentages.

You can apply the filters to the data displayed according to the criteria below:

Day of dialogdate - the date range for which the data is displayed.
Weekday - filtering according to the day of the week.
Channel - the channel through which the conversation occurred.
Language - the language of the miniApps.
Self-Service Names - the name of the self-service of the application.
End Types - the outcome of the conversation, either Terminated or Transferred.
Organization - the group that the miniApp is assigned to when created in an OCP® instance.

Transfer Reasons

The Transfer Reasons dashboard provides an overview of all the calls that were transferred out from the OCP IVR and reached a call center agent. You can use this dashboard to:

prioritize enhancements based on the transfer reasons;
provide possible new announcements or self-services based on the last active intent;
gain an understanding of which transfer lines receive higher call volumes.

The metrics shown in the report are the following:

Transfer Reasons - a compilation of transfer reasons and how often they were applied within the selected time period. For example, AgentRequest, MaxNoInputs, MaxContinuousNoMatches, and so on.
Transfer Reasons per Day - a daily and overall distribution of transfer reasons.
Exit (Transfer) Lines - a distribution of calls per each different transfer line that the Omilia IVR transferred the call to.
Last Active Intent - a compilation of the last intents extracted in the calls before they were transferred out.
Last Active Intents per Day - a daily distribution of the call level intents, corresponding to the last intent extracted in these calls before they were transferred out. If no intent was requested by the caller within the call, the last active intent would be Undefined.

You can adjust the filters according to the criteria below:

Date - the date range for which the data is displayed.
Language - the language of the miniApps.
Channel - the channel through which the conversation occurred.
OCP Application Name - the name of the application reviewed.
Organization - the group that the miniApp is assigned to when created in an OCP® instance.
Last Active Intent - the last intent extracted from the call before it was transferred out.
Transfer Reason - the reason for the call transfer.
Exit Line - the transfer line to which the call was transferred.

Caller Classification Reporting

The Caller Classification dashboard provides an overview of the caller's engagements inside an OCP IVR. All the calls that reach the OCP IVR are classified based on how the callers behave and how much they engage within the application.

The metrics shown in the report are as follows:

Power Users - number of calls that entered the IVR application, where the caller successfully requested an Intent without encountering any issues, such as no inputs, no matching results, and no agent requests during the entire call.
Engaged Users - number of calls that entered the IVR application, and the caller did request an Intent but might have faced a challenge at a certain step, like having difficulty interacting with the application or remaining silent during the call.
Not Engaged - number of calls that entered the IVR application, but the callers did not express any specific Intent while attempting to engage with the system.
Instant Calls - number of calls that entered the IVR application but were swiftly terminated or transferred right away, either upon the first system request or before the initial system query.
Silent Calls - number of calls that entered the IVR application, but throughout the call, there was no input or interaction from the user.
Agent Requests - number of calls that entered the IVR application, where only agent was requested and no intent was provided, typically due to the caller’s refusal to cooperate with the application.

The dashboard also contains a pie chart for Callers Category Distribution, as well as the Callers Classification Daily Distribution graph to show metrics described above on a daily basis.

You can adjust the filters according to the criteria below:

Date - the date range when the calls took place.
Organization - the group that the miniApp is assigned to when created in an OCP instance.

Recaller Overview

The Recaller Overview visualization offers insights for OCP IVR users, specifically focusing on call recall patterns. Calls reaching the OCP IVR are analyzed based on the ANI initiating the callback. The recall timeframe spans a 24-hour window, aligned with the UTC calendar date (00:00 UTC - 00:00 UTC).

Notably, Anonymous or Unknown ANI instances are excluded from the recall analysis, ensuring only valid ANI (phone numbers) are taken into account. The visualization also delves into distinct behavioral patterns exhibited by recallers in comparison to one-time callers, aiming to uncover motivations prompting repeated interactions with the application.

The metrics shown in the report are as follows:

Once Off Callers - displays the number and percentage of calls originating from an ANI that called only once within a 24-hour period (for a given calendar date).
Recallers - presents the number and percentage of calls from an ANI that made multiple calls within a 24-hour period (for a given calendar date).
Unknown - indicates the number of calls initiated by an Unknown/Anonymous ANI (origin URI) to the application.
Call Back Intervals - illustrates the time interval from the initial call to the callback.
Distribution of Calls - highlights the number of calls made by each ANI, consistently one for once-off callers and varying from 2 to more than 10 for recallers.
User Classification - categorizes callers, with filter options available for different call types. Additional details on caller classification reporting can be referenced for further information.
Call Outcome - depicts the distribution of different call outcomes (User HUP for user termination, System HUP for system termination, Transfer, or error). The distribution of different call outcomes can be filtered between different callers types (recallers or once off callers) by clicking on the corresponding pie chart section of the Once Off Callers vs Recallers Overall graph.

You can also adjust available filters according to the criteria below:

Date - the date range when the calls took place.
Organization - the group that the miniApp is assigned to when created in an OCP instance.