OCP® Monitor User Guide

Overview

OCP® Monitor is the portal to understanding every detail of system-user interaction. This tool allows the user to drill down into any conversation occurring within an OCP® application. This involves Orchestrator application dialogs, Flows, and miniApp invocations originating from calls or web chats.

To access OCP® Monitor, go to Insights → Monitor in the side panel.

Monitor Components

The Monitor tab is divided into three main components:

• Search filter

• Dialog list

• Dialog Review

In the next sections, you can find detailed explanations for each of the components.

Search Filter

The search filter allows you to restrict and refine the dialog search according to the selected criteria.

For a quick search, select one of the Search by criteria:

• Dialog ID

• Session ID (Available only for Virtual Agent)

• Connection ID

It is possible to search for all the dialog sessions that match the selected criteria or to select one particular dialog session:

• To search for all the dialog sessions that match the selected criteria, choose the criteria and click the Search icon.

• To search for the particular dialog session, apply the criteria and paste its ID into the ID search field.

Application Layer

Application Layer allows you to view full end-to-end conversations only without the subcomponents, providing a clear high-level overview of each dialog in OCP.

Once the Application Layer is activated, you can expand each Dialog Application and view its components by clicking the dropdown icon. However, if you apply the Search by filter using either Dialog ID, Connection ID, or Session ID, the hierarchical structure will not be displayed.

Default Filters

You can refine the results using default filters, for example, Date Range, Time Range, Agent Type, and Group.

Advanced Search

To open the Advanced Search filters, proceed as follows:

1. Select the Agent Type.
   

   

   
   

   
   

2. Click on Filters.
   

   

   
   

3. The Advanced Search area opens.
   

   

   
   

4. Fill in the search filter fields. In the table below, you can find detailed information on each field.

5. To submit your search request, click the Apply button.
   

   

   
   

6. The dialogs that match your selected criteria are shown in the Dialog list.
   
   Applied filters are indicated below the Filters bar.

Search Filters Description

Initial filters available in the Default Filters area include:

The following filters are available upon clicking the Filters and selecting type of agent, not all filters button:

Dialog List

The Dialog List presents all the dialog events that meet the criteria of the Search Filter section. Every column contains information regarding the dialog.

• TIME: The time when the dialog started.

• ORIGIN: The origin of the voice of the dialog (Virtual / Live).

• NAME: The name of the application.

• TYPE: The type of the application.

• GROUP: The group that miniApps or Flows belong to.

• ANI: The caller’s phone number if available.

• DURATION: The length of the dialog session.

• STEPS: The number of steps in the session. This is the total number of system prompts and the user utterances, as well as one step added per the dialog ending, such as [hup], meaning hang-up, or transfer to an agent. The [no_input] values are not counted.

• END: The way the dialog ended. Hover over the icon to see the tooltip with details.

By default, 50 dialogs per page are presented. However, you can change this setting by clicking the 50/page button and selecting the desired value. To navigate through the dialog list, click the Next button.

Click on a particular dialog to get detailed information in the Dialog Review area.

You can click the dropdown arrow on the far left of a main application's row to expand it. This will reveal all the component miniApps that were run as part of that single conversation, showing you each sub-dialog and its corresponding details.

Dialog Review

The Dialog Review component contains all the detailed information about a single, selected conversation. When you click on a specific dialog from the Dialog List, the Dialog Review pane will open.

The content of the pane is collapsed by default; click the expand icon (a + symbol) next to the Dialog Review title to see the details.

Next to the title, you'll see a tag indicating the agent type for the session, such as VIRTUAL AGENT or LIVE AGENT. The information and options available will change depending on this type.

The pane is split into several key areas:

Dialog Header (Metadata)

This top section displays key metadata about the conversation. You can click the Copy icon next to any ID to copy its value to your clipboard.

The fields displayed change depending on whether you are viewing a VIRTUAL AGENT or a LIVE AGENT session.

Virtual Agent View Fields

• Date time: The date and time the dialog took place.

• Duration: The length of the dialog session.

• miniApp Name: The name of the miniApp used for the selected dialog.

• miniApp Type: The type of the miniApp used for the selected dialog.

• Application Version: The publishing version of the Application.
  Navigate to the Orchestrator Application Options Menu guide to learn more about publishing.

• Group: The group this Application belongs to.

• Channel: The channel the user used for the session, such as voice or web chat.

• ANI: The caller's phone number, if available.

• Session ID: The identification number of the session.

• Dialog ID: The identification number of the dialog.

• Connection ID: The unique identification number of the connection.

• Dialog Group ID: The identification number of a Flow application that consists of multiple miniApps.

Live Agent View Fields

• Date time: The date and time the dialog took place.

• Duration: The length of the dialog session.

• Connection ID: The unique identification number of the connection.

• Dialog ID: The identification number of the dialog.

• Master ID: The master ID for the Agent Assist session.

• CCAAS Provider: The Contact Center as a Service (CCaaS) provider handling the session (e.g., niceInContact).

• Agent ID: The unique ID of the human agent (e.g., agentId).

• Agent Skill: The skill group assigned to the human agent (e.g., vb_skill).

• AA Profile ID: The unique identifier for the Agent Assist (AA) profile being used (e.g., 870..a1f9).

• AA Profile Name: The name of the associated agent profile (e.g., konan_with_vb_profile).

The miniApp bot icon indicates the prompt generated by the miniApp. The Agent icon indicates the prompt written by the Agent. The numbering on the left represents the step number.

Correspondingly, the utterances produced by the caller are indicated by the User icon.

If this was a call, you can click the Play button next to the User icon to listen to the audio and check whether the system captured it correctly.

Click the Save button to download the utterance in WAV format.

You can expand the user’s utterance by clicking the arrow to view the Intent interpretation. When expanded, the arrow will be pointing up:

You can tag an utterance as correct or false, acceptance or rejection, or as a misinterpretation. To do so, click the # symbol and select the corresponding options in the pop-up window.

• CISR: Concept Identification Success Rate. Select a value from the drop-down list.

  • correctAcc: The system correctly accepted the caller's input, which means that either the correct intent was extracted, or the expected input was correctly identified (for example, the caller answering yes or no when asked to confirm payment).

  • correctRej: The system correctly rejected something that is not within the application’s scope (for example, the customer utterance: I could call back if this is a bad time).

  • falseAcc: The system mistakenly confused something out of the application’s scope with something within scope.

  • falseRej: Mistakenly rejected an in-scope caller input (for example, I want to know my balance is not understood and is considered by the system to be irrelevant).

  • misinterpretation: Accepted with wrong understanding. A caller said I want the balance on card, and the system understood I want the balance on my loan. The system understood the utterance as related to the application, but it misinterpreted it by assigning the wrong intent.

• Custom Field: Add a keyword to categorize the utterance.

Dialog Options and Toolbar

At the top of the Dialog Review pane, you have several buttons to download data and analyze the dialog.

• Dialog Tree: (Available in VIRTUAL AGENT view)
  This window shows a tree-like hierarchical view of all the dialog components (miniApps, Flows, etc.) that were called during the session, in the order they were executed. You can click on any item in this tree to search for dialogs in that specific miniApp.

• Options menu: Clicking this icon opens a dropdown menu with more actions:

  • Show events: Retrieves system and/or custom loggings from the application. For more information, refer to the Events review section.

  • Open DiaManT Log File: Opens the DiaManT log file for the selected dialog.

  • Download JSON: Downloads a JSON file containing all the information for that dialog session.

  • Download RTF: Downloads an RTF file (plain text) with the dialog session info.

  • Download Log File: Downloads the dialog's log file directly to your device in a .txt format.

  • Create Test Case: This option allows you to save the current dialog as a new test case in OCP Testing Studio®.

    1. Clicking this opens the Create Test Case window.
       

       

    2. Enter a name in the Name Test Case field.

    3. Select a destination project from the Select Project dropdown list. These projects are pulled from Testing Studio®.

    4. Click the Create Test Case button.

    5. A test case creation notification will appear, with a link to View in Testing Studio.

Post Call Summary

For interactions that include a LIVE AGENT, you may see a POST CALL SUMMARY. This section provides a brief, AI-generated summary of the dialogue, outlining the caller's intent, the agent's actions, and the call's outcome.

Text Matching

We use a Standard Text Matching analysis to process step’s prompts and utterances and an Events Matching Approach for dialog events. Both of them use a list of terms parsed from the text instead of the whole text, but in a different way to ensure the best accuracy.

The Standard Text Matching analysis:

• matches all the words of the given text;

• does not care about the words’ order;

• matches all the given words anywhere in the text;

• case insensitive.

Example

This text:

Welcome to our new voice-enabled service line. I'll be your Virtual Assistant.

will have the following matches:

• “welcome” (single word match)

• “welcome to our new“ (all words match)

• “Welcome” (case insensitive)

• “Welcome to our...” (ignores trailing punctuation)

• “assistant service Welcome” (ignores word order)

and will not match the following:

• “Hello welcome to our” (not all words match)

• “come” (any random substring won’t match)

The Events analysis:

• matches all the words of the given text;

• cares about the words’ order (to avoid wrong matches, due to events string format);

• matches consecutive words (phrases);

• case-insensitive.

Example

This text:

“DialogGroupID:13188be3907c4f340d69479834b07b0a5fb64b6d.1712755163180.49a7bca9167e4d1eb1c2aceb8f3592d1,FlowName:MiniApps,appId:c958d295-9348-4899-9783-e09a5e8abcec.Card_GetAccessToken_WebService.WebService.MiniApps.ocp-qa,requestId:undefined,envMode:DEV,testMode:true,Locale:en-US,globalResult:fail:apiException,HUP:Custom FailExitReason,HUP:NEAR,endType:NEAR_HUP,billable:false”

will have the following matches:

• “DialogGroupID” (single word match)

• “dialoggroupid” (case insensitive)

• “globalResult fail apiException HUP Custom FailExitReason” (all words match)

• “globalResult:fail:apiException,HUP:Custom FailExitReason” (ignores punctuation)

• “globalResult:fail:apiException” (fail with reason)

• “globalResult:fail” (fail with no reason)

• “13188be3907c4f340d69479834b07b0a5fb64b6d.1712755163180.49a7bca9167e4d1eb1c2aceb8f3592d1” (group ID match)

• “Webservice” (other word from the list of tokens)

• other IDs or app name/org which could be found in event string

and will not match the following:

• “globalResult fail HUP FailExitReason” (words in order but not consecutive)

• “FailExitReason DialogGroupID HUP” (random word order)

• “group” (any random substring won’t match)

Path View

The Path View is a sequence of dialog session components (miniApps, Flows and so on), which represents the order of dialog steps that were called within the Application in a tree-like hierarchical view.

To access the Path View, proceed as follows:

1. Navigate to Insights → Monitor and select a Dialog. In the Dialog Review section, click the Dialog Tree button.

2. Once clicked, the Path View opens with all the dialog steps components in the corresponding order.

Every branch of the Path View window is expanded by default but collapsable, and it is possible to click each item of the tree to search for dialogs in the particular miniApp.

When the particular item is clicked, you’ll be redirected to the page with the more detailed information about this item.

Monitor Voice Biometrics

You can view the report on each step of the dialog in terms of Voice Biometrics. This allows the user to assess the caller's identification level based on the following parameters:

Voice Food Ordering (VFO) View

When VFO functionality is activated for the particular environment, there are some additional features available in the Monitor tab, including an additional Order Summary component near the Dialog Review:

The standard and advanced Filters options are the same as for standard environments, with Agent Type filter excluded. The Advanced Search options contains VFO-specific filter dropdown in addition to the standard options:

• Step result (success/fail)

• Fail area (ASR, VAD, NLU, menu, integration)

• NLU breakdown (entity_identification_menu_item, entity_identification_assistive_annotation, entity_identification_faq, reasoner_correction, reasoner_repetition, reasoner_disambiguation, reasoner_other)

• ASR breakdown (misrecognition, missed_word, low_confidence, background_speech, background_noise, volume)

• VAD events (interruption, speak_over, no_trigger, interruption_hes_ending, false_trigger, late_end trigger)

• Order output (success/fail)

• Sentiment (negative)

• Turn detection breakdown (correct_take_over, correct_pause_handling, incorrect_take_over, incorrect_pause_handling, late_take_over, unclear, previous_turn_answer)

• Comments (applicable comments if available)

The Dialog List component contains the same elements as standard environment.

Besides the default elements within the Dialog Review component, you can also access the Full Call Recording feature under the Show events button. You can listen to the whole dialog at once and download the full call recording by clicking the Save button as shown on the screenshot below.

To the right to the Dialog Review, you can also see the VFO-specific Order Summary component containing all the items ordered by the user within the dialog you’re viewing.

You can adjust the components' width for better visibility by dragging horizontal borders to the left or to the right as shown on the screenshot below.

To collapse the currently open dialog, click on the > icon near the Dialog Review heading. This will return you to the Select Dialog component being shown on the full screen.

Order Summary

The Order Summary component contains the following information:

• Store ID - the ID of the store where the order was processed. You can copy the ID by clicking on the Copy icon to the right.

• Number of items - the total number of items ordered.

• Order list - a list of all the items ordered. If there’s any addition to the item (for example, a dip), it will be displayed as a subitem with the ^ icon showing the expanded view.

• Total cost - the total cost of the order in the currency you set.

• Download Order - a button to download the order details in a JSON format.

If a customer asked to add/remove some ingredients for a specific item, this information will be visible in the Order Summary under the modified item.