OCP® Monitor User Guide

Overview

OCP® Monitor is the portal to understanding every single low detail information about a system-user interaction. This tool allows the user to drill down into any conversation occurring within an OCP® application. This involves Orchestrator application dialogs, Flows, and miniApp invocations originating from calls or web chats.

To access OCP® Monitor, go to Insights → Monitor in the side panel.

Monitor Components

The Monitor tab is divided into three main components:

• Search filter

• Dialog list

• Dialog Review

In the next sections, you can find detailed explanations for each of the components.

Search Filter

The search filter allows you to restrict and refine the dialog search according to the selected criteria.

For a quick search, select one of the Search by criteria:

• Dialog ID

• Session ID

• Connection ID

It is possible to search for all the dialog sessions that match the selected criteria or to select one particular dialog session:

• To search for all the dialog sessions that match the selected criteria, choose the criteria and click the Search icon.

• To search for the particular dialog session, apply the criteria and paste its ID into the ID search field.

Searching by Dialog ID, Session ID and Connection ID cancels all other filters if selected.

Switch on the Show Live toggle to get new session data every 15 seconds and monitor ongoing calls.

Application Layer

Enabling the Application Layer allows you to view full end-to-end conversations only without the subcomponents, providing a clear high-level overview of each dialog in OCP.

Once the Application Layer is activated, you can expand each Dialog Application and view its components by clicking the dropdown icon. However, if you apply the Search by filter using either Dialog ID, Connection ID, or Session ID, the hierarchical structure will not be displayed.

By default, the Application Layer is enabled.

Advanced Search

To open the Advanced Search filters, proceed as follows:

It is recommended to disable the Application Layer for Advanced Search, especially if you are going to search for specific miniApps patterns.

1. Click on the Funnel Filter icon.

2. The Advanced Search area opens. Click More Filters to additionally extend the search options.

3. Fill in the search filter fields. In the table below, you can find detailed information on each field.

4. Click the Count button near the View button to find out the number of dialogs retrieved according to the applied criteria. It can be useful to decide if the search criteria meet your expectations. This step is optional.

5. To submit your search request, click the View button. Every time you add or remove filters to refine your search, click the View button to refresh the filter status.

6. The dialogs that match your selected criteria are shown in the Dialog list.

The Funnel filter icon with a blue circle indicates that filters are applied.

To clear the search filter fields, click the Clear all button.

Search Filters Description

Initial filters available in the Advanced Search area include:

Filter Name	Description
Date and time	Select the specific time period you are interested in by indicating start/end dates and start/end times. The time corresponds to your local time zone.
Group	The resource group that miniApps or Flows belong to.
Name	The application name. This value can be multiple.
Region	The OCP Environment origin region. This filter is available on specific environment configuration setups only.
Type	The type of the miniApp. Select the type Flow from the dropdown list to review Flow dialogs only.
ANI	The caller’s phone number.
Dialog Group ID	Alternative to Session ID to fetch all Dialog IDs requested by the same Orchestrator Application.

The following filters are available upon clicking the More Filters button:

Dialog List

The Dialog List presents all the dialog events that meet the criteria of the Search Filter section. Every column contains information regarding the dialog.

• TIME: The time when the dialog started.

• NAME: The name of the application.

• TYPE: The type of the application.

• GROUP: The group that miniApps or Flows belong to.

• ANI: The caller’s phone number if available.

• DURATION: The length of the dialog session.

• STEPS: The number of steps in the session. This is the total number of system prompts and the user utterances, as well as one step added per the dialog ending, such as [hup], meaning hang-up, or transfer to an agent. The [no_input] values are not counted.

• END: The way the dialog ended. Hover over the icon to see the tooltip with details.

By default, 10 dialogs per page are presented. However, you can change this setting by clicking the 10/page button and selecting the desired value. To navigate through the dialog list, click the Next button.

Click on a particular dialog to get detailed information in the Dialog Review area.

Dialog Review

The Dialog Review contains information about a selected dialog.

• Date time: The date and time the dialog took place.

• miniApp Name: The name of the miniApp used for the selected dialog.

• Application Version: The publishing version of the Application.
  Navigate to the Orchestrator Application Options Menu guide to learn more about publishing.

• Channel: The channel the user used for the session, such as voice or web chat.

• Session ID: The identification number of the session. You can copy the Session ID value by clicking on the Copy icon next to it.

• Connection ID: The unique identification number of the connection. You can copy the Connection value by clicking on the Copy icon next to it.

• Duration: The length of the dialog session.

• miniApp Type: The type of the miniApp used for the selected dialog.

• Group: The group this Application belongs to.

• ANI: The caller’s phone number if available.

• Dialog ID: The identification number of the dialog. You can copy the Dialog ID value by clicking on the Copy icon next to it.

• Dialog Group ID: The identification number of a Flow application that consists of multiple miniApps with their own specific dialog IDs. You can copy the Dialog Group ID value by clicking on the Copy icon next to it.

The miniApp bot icon indicates the prompt generated by the miniApp. The numbering on the left represents the step number.

Correspondingly, the utterances produced by the caller are indicated by the User icon.

If this was a call, you can click the Play button next to the User icon to listen to the audio and check whether the system captured it correctly.

Click the Save button to download the utterance in WAV format.

You can expand the user’s utterance by clicking the arrow to view the Intent interpretation. When expanded, the arrow will be pointing up:

If there is sensitive data in the utterance, the details are not shown, and the message **encrypted content** appears instead.

You can tag an utterance as correct or false, acceptance or rejection, or as a misinterpretation. To do so, click the # symbol and select the corresponding options in the pop-up window.

• CISR: Concept Identification Success Rate. Select a value from the drop-down list.

  • correctAcc: The system correctly accepted the caller's input, which means that either the correct intent was extracted, or the expected input was correctly identified (for example, the caller answering yes or no when asked to confirm payment).

  • correctRej: The system correctly rejected something that is not within the application’s scope (for example, the customer utterance: I could call back if this is a bad time).

  • falseAcc: The system mistakenly confused something out of the application’s scope with something within scope.

  • falseRej: Mistakenly rejected an in-scope caller input (for example, I want to know my balance is not understood and is considered by the system to be irrelevant).

  • misinterpretation: Accepted with wrong understanding. A caller said I want the balance on card, and the system understood I want the balance on my loan. The system understood the utterance as related to the application, but it misinterpreted it by assigning the wrong intent.

• Custom Field: Add a keyword to categorize the utterance.

The Show events button allows retrieving system and/or custom loggings from the application. For more information, refer to the Events review section.

To open the DiaManT log file of a selected dialog, click the Log File button.

To get a JSON file with all the dialog session info, click the JSON file icon.

To get an RTF file with all the dialog session info, click the TXT file icon.

Text Matching

We use a Standard Text Matching analysis to process step’s prompts and utterances and an Events Matching Approach for dialog events. Both of them use a list of terms parsed from the text instead of the whole text, but in a different way to ensure the best accuracy.

The Standard Text Matching analysis:

• matches all the words of the given text;

• does not care about the words’ order;

• matches all the given words anywhere in the text;

• case insensitive.

Example

This text:

Welcome to our new voice-enabled service line. I'll be your Virtual Assistant.

will have the following matches:

• “welcome” (single word match)

• “welcome to our new“ (all words match)

• “Welcome” (case insensitive)

• “Welcome to our...” (ignores trailing punctuation)

• “assistant service Welcome” (ignores word order)

and will not match the following:

• “Hello welcome to our” (not all words match)

• “come” (any random substring won’t match)

The Events analysis:

• matches all the words of the given text;

• cares about the words’ order (to avoid wrong matches, due to events string format);

• matches consecutive words (phrases);

• case-insensitive.

Example

This text:

“DialogGroupID:13188be3907c4f340d69479834b07b0a5fb64b6d.1712755163180.49a7bca9167e4d1eb1c2aceb8f3592d1,FlowName:MiniApps,appId:c958d295-9348-4899-9783-e09a5e8abcec.Card_GetAccessToken_WebService.WebService.MiniApps.ocp-qa,requestId:undefined,envMode:DEV,testMode:true,Locale:en-US,globalResult:fail:apiException,HUP:Custom FailExitReason,HUP:NEAR,endType:NEAR_HUP,billable:false”

will have the following matches:

• “DialogGroupID” (single word match)

• “dialoggroupid” (case insensitive)

• “globalResult fail apiException HUP Custom FailExitReason” (all words match)

• “globalResult:fail:apiException,HUP:Custom FailExitReason” (ignores punctuation)

• “globalResult:fail:apiException” (fail with reason)

• “globalResult:fail” (fail with no reason)

• “13188be3907c4f340d69479834b07b0a5fb64b6d.1712755163180.49a7bca9167e4d1eb1c2aceb8f3592d1” (group ID match)

• “Webservice” (other word from the list of tokens)

• other IDs or app name/org which could be found in event string

and will not match the following:

• “globalResult fail HUP FailExitReason” (words in order but not consecutive)

• “FailExitReason DialogGroupID HUP” (random word order)

• “group” (any random substring won’t match)

Path View

The Path View is a sequence of dialog session components (miniApps, Flows and so on), which represents the order of dialog steps that were called within the Application in a tree-like hierarchical view.

To access the Path View, proceed as follows:

1. Navigate to Insights → Monitor and select a Dialog. In the Dialog Review section, click the Dialog Tree button.

2. Once clicked, the Path View opens with all the dialog steps components in the corresponding order.

It is possible to drag the Path View window for convenience to any part of the window. It is also possible to resize the Path View window by dragging the bottom right corner of the pop-up.

Every branch of the Path View window is expanded by default but collapsable, and it is possible to click each item of the tree to search for dialogs in the particular miniApp.

When the particular item is clicked, you’ll be redirected to the page with the more detailed information about this item.

Monitor Voice Biometrics

You can view the report on each step of the dialog in terms of Voice Biometrics. This allows the user to assess the caller's identification level based on the following parameters:

Only applicable for IVR applications with Voice Biometrics enabled.

Read more about Voice Biometrics in the Voice Biometrics section.