Adding custom data to an NLU model

Intents and entities

Intent is a goal or purpose that the user expresses in their utterance, it represents an action that the user wants to perform. The same intent can be expressed by the user in many different ways, or phrases. These phrases are called utterances.

Entity represents values collected from the user in a conversation, so this is basically a keyword that you want to extract from the user’s utterance to resolve their query.

When you are creating a model with a specific domain, the out-of-the-box intents and entities are loaded automatically, if available. Your model will use these elements to understand the natural language input and direct the conversation flow.

The out-of-the-box data are labeled as RB-XP and cannot be deleted. If needed, you can disable selected out-of-the-box intents by switching off the toggle as shown below.

Double intent recognition

Omilia’s out-of-the-box offering also includes double intent recognition. It handles scenarios where a single user input might involve more than one intention.

For example, a customer might state, "I want to check my balance and pay minimum payment". In this situation, the system will identify and process the two intents: Balance-Inquiry and Payment-Minimum_Payment. The intelligent double intent recognition algorithm ensures efficient and accurate processing of user intentions, enhancing the conversational experience.

Adding custom data

Besides using the existing resources, you may also want to extend your model’s understanding with your custom intents and entities. Also, if you are creating a custom domain model, you will have to add your own data from scratch. Either way, you can find the step-by-step instructions below.

Uploading custom data

You can upload your Custom Training Data using TXT, CSV, or TSV file format making sure the encoding is UTF-8 and that the line breaks are Unix/Linux style. Below in the text you can find an example on how the file’s content should look like.

To upload your custom data, proceed as follows:

1. Navigate to NLU → NLU Models tab.

2. Select a model and click on it. The model drill-down page opens.

3. Click the Context menu icon → Upload intents.

4. In the opened dialog box, upload a file in TXT, CSV, or TSV by dropping it directly or clicking Browse Files to select one.

Click the Sample File button to download a sample set of intents in CSV. Also you can see an example of the training set structure below:

I don't need my account anymore;Close.Account
i have a question;Inquire
When is the new model coming out;Inquire.Availability
I want to change my username.;Update.Account
My credit card was declined. Can you rerun it;Trouble.Payment

5. Click Upload. The dataset is uploaded, and the intents and utterances are now added to your model.

Adding intents

The intent is the goal of the user's request to the application that describes what the end-user wants. For example:

Airline Virtual Assistant: Welcome to ABC Airlines. What can I do for you today?
User: I'd like to get a ticket from New York City to Jamaica for next Friday.

In the example above, the user wants to buy a ticket, so the application will analyze the request and select an intent from the list of intents. In this case, it would be something like Flight_Ticket-Purchase.

To add an intent to your model, proceed as follows:

1. Navigate to NLU → NLU Models section.

2. Select a model and click on it. The model drill-down page opens.

3. Go to the Intents tab and click the Plus icon.

4. In the opened dialog box, enter an intent name.

The intent name must start with a letter and contain no spaces. It can consist only of letters, dots, hyphens, or underscores. Symbols must be followed by a letter.

5. Click Create to confirm. The intent has been added.

The created intent is highlighted in red. The red Exclamation sign next to the Utterances list informs you that no or not enough utterances have been added to the intent.
Models with intents that are highlighted in red cannot be trained! Add utterances to your intent to be able to train your mode.

• To edit an intent’s name, click the Pencil icon.

• To delete an intent, click the Delete icon.

You can only delete custom intents. The out-of-the-box intents cannot be deleted or edited.

Merging intents

There may be cases where it is desirable to combine several intents into one singular intent. You have the capacity to do this with both standard out-of-the-box intents and those that are custom-created.

To merge intents, proceed as follows:

1. Click the Context menu icon in the Intents section and choose Merge intents.

A window will appear where you should complete the fields as illustrated below:

• Intent name: Input a name for your newly merged intent.

• Description: Give a brief explanation of the intent. This field is optional.

• Intents: Choose the intents to merge; you have the option to select multiple intents.

As an illustration, we are consolidating a new intent named ATM that includes several out-of-the-box intents as depicted below:

3. Click Merge to confirm. The newly merged intent will appear under Intents, marked with a Merged icon. The intents that have been merged will no longer be visible in the Intents section. In our ATM example, all the ATM-related intents are no longer showcased in the list.

4. The merged intent can be updated at any point. To revise the intent, select the Edit icon and modify the necessary fields. All fields are editable.

For example, in our ATM intent, we decided to remove ATM-Problem and ATM-Retained.Card intents from being merged.

After clicking the Update button, the excluded intents, in our case, ATM-Problem and ATM-Retained.Card, will reappear in the list of Intents.

Deleting merged intents

If you wish to delete a merged intent, simply select the Delete icon and confirm the deletion.

Take note that deleting a merged intent will permanently erase any custom intents that were part of the merger. However, the standard out-of-the-box intents that were earlier involved in the merger will be restored in the Intents list.

Adding utterances

The same intent can be expressed by the user in many different ways, or phrases. These phrases are called utterances.

You need to add at least five training utterances for each intent. Ideally, ten or more utterances should be added. The more utterances you add, the better the result will be.

To add utterances to an intent, follow the steps below:

1. Select an intent and click on it to activate the utterance input field.

2. Enter an utterance into the highlighted input field.

3. Press <Enter> on your keyboard to add the utterance to the corresponding intent. The Exclamation sign next to Utterances will change its color from red to yellow depending on their number:

   • Red: Less than five utterances have been added

   • Yellow: Less than ten utterances have been added.

As soon as ten or more utterances are added, the exclamation sign will disappear and the intent will not be highlighted anymore.

4. When adding utterances, you can refer to the existing entity by inserting the @-sign. As you start typing the entity name, you will be presented with a dynamic drop-down list from which you can select your desired entity.

5. If you need to refer to a specific value and not to the whole entity, add a dot after the entity name and select an entity value from the existing entity dictionary.

6. If an entity holds no values, you will be alerted via a corresponding message.

7. After selecting an entity or an entity value, you will see the Information icon next to Utterances that urges you to pay extra attention to this utterance. When hovering over this icon, you will see the following message: This intent includes entity references that might impact its effectiveness. Consider reviewing them for quality and relevance.

Deleting utterances

To delete an utterance, hover over it and click the Delete icon.

Adding entities

Entities allow an application to understand the meaning of natural language text and extract relevant information for further processing.

An entity is a specific piece of information or data within a sentence that represents a real-world object, such as a person, place, thing, event, action, concept, and so on. It’s a label we use to shelter words or small phrases under the same umbrella. For example, green, orange, blue, purple, yellow, red are colors. Visa, MasterCard, and American Express are card networks.

Custom domain models do not support entities.

To add an entity, follow the steps below:

1. Navigate to NLU → NLU Models section.

2. Select a model and click on it. The model drill-down page opens.

3. Go to the Entities tab and click the Plus icon.

4. In the opened dialog box, fill in the fields as described below:

• Entity Name: Enter the entity name.

The entity name may only consist of letters and contain no spaces. Please note that entity names are treated as case insensitive.

• Dictionary: Upload an entity dictionary file. Omit this field if you are going add values and alternatives manually.

5. Click Create to confirm. The entity has been added. The added entities are labeled as CUSTOM.

Uploading an entity dictionary

The upload entity dictionary feature allows you to quickly and easily create or update a custom entity by uploading a dictionary file. Instead of manually adding values and alternatives to your entity, this feature significantly reduces the time and effort involved in the process.

To upload a dictionary file, follow the guidelines below:

1. Navigate to the Entities tab and click the Plus icon.

2. In the dialog box that appears, fill in the fields as described below:

• Entity Name: Enter a name for your entity.

• Dictionary: Upload a dictionary file for your custom entity. This file can either be dragged and dropped directly, or selected by clicking Browse Files. Ensure your file is in CSV format and does not exceed 1 MB in size. Each line in your dictionary file should contain one reference value and one global alternative, as illustrated below:

  CODE

  testValue;testAlternative1
  testValue;testAlternative2
  testValue;testAlternative3
  testValue2;testAlternative4
  testValue2;testAlternative5
  testValue2;testAlternative6

There is also an option to use a sample file to experience this feature. Simply click Sample file to download the file.

3. Click Create once everything is set. The uploaded dictionary is now visible and should populate your custom entity with the respective values and global alternatives.

4. To update the dictionary, click the Edit icon in front of the entity name and repeat steps 2-3 described above.

Uploading a new dictionary file will overwrite the existing values and global alternatives!

In case the dictionary that is being replaced contains context alternatives, these will be maintained in the new dictionary only if the reference value to which they belong is also maintained.

Adding values and alternatives

Sometimes, we can use different words for the same meaning. This means that several words under the same entity can have the same value. For example, cat, kitten, dog, canine, puppy, hen, duck, chicken are animals. As you can see below, dog, canine and puppy are different alternatives of the same value.

For each entity, you can add multiple values and multiple alternatives for each value. All alternatives (global and context ones) are mapped to the reference value that you’ve defined. If you do not want to declare a reference value and you just want the entity to be triggered based on your alternatives, just leave it blank.

To add a reference value and global alternatives to the entity, follow the steps below:

1. Select an entity and click on it.

2. Click the + Add Reference value and Alternatives button.

3. Enter the reference value.

4. Enter a global alternative and press <Enter> on the keyboard. You can use the smart dictionary commands to refer to the existing entities and streamline the process of adding alternatives.

You can add multiple alternatives to one value. For alternatives, you can use single words and short phrases.

5. When you are finished, press <Enter> on the keyboard again. The value and alternatives are added.

Unmarking an alternative

There may be instances where it becomes necessary to exclude a specific entity from the annotation
if its alternative contains a specific word or phrase. In this case, you can unmark such an alternative to exclude it from being annotated.

You can unmark both global alternative and context alternative.

To unmark an alternative, proceed as described below:

1. Insert an exclamation mark before the alternative.

2. Press <Enter>. The unmarked alternative will be highlighted in yellow and excluded from the annotation.

You can also left-click a selected alternative to unmark it.

Deleting values and alternatives

To delete a reference value, hover over it and click the Delete icon.

To delete an alternative, double-click it and then click the Delete icon.

You can only delete your custom values and alternatives. The out-of-the-box elements cannot be deleted, however, they can be excluded from being annotated.

Smart dictionary

The smart dictionary feature has been designed to streamline the process of adding global alternatives and context alternatives. You can effortlessly refer to pre-existing entities, or formulate new alternatives by leveraging the existing entity dictionaries. With the smart dictionary feature, incorporating or excluding entities is now quicker, simpler and more efficient.

Start by hovering over the Keyboard icon. This will present you with a comprehensive overview of all the operations supported by the smart dictionary.

Referring to an entity

To refer to the entity's global alternatives, add the @-sign before the entity name. As you start typing the entity name, you will be presented with a dynamic drop-down list from which you can select your desired entity. This entity will then be incorporated as an alternative for annotation.

Referring to a specific entity value

If you need to refer to a specific value and not to the whole entity, add a dot after the entity name. You will now be able to select an entity value from the existing entity dictionary.

If an entity holds no values, you will be alerted via a message as demonstrated below:

Excluding an entity from annotation

If needed, you can exclude an entity from being annotated or even discard an out-of-the-box annotation entirely. This functionality facilitates a more accurate annotation process, allowing for the creation of more precise and effective dialogues.

To exclude an entity from the annotation process, proceed as described below:

1. Insert an exclamation mark before its reference.

2. Press <Enter>. The entity excluded will be highlighted in yellow.

You can also left-click the existing entity reference to exclude it from the annotation.

Removing entity reference

To remove a reference to an entity or entity value, just click the Delete icon next to it.

Adding contexts

Context makes a conversation more informal and human-like. It adds specific details to the dialog that help clarify the meaning.

For example, “credit” is a generic term that means debt. However, if the question is “Is it a debit or a credit card?”, then the meaning of the word “credit” is a “card type”. In the same context, if a user says “the second”, this would again mean a card type. However, without context, this is just an ordinal number.

To add a context, proceed as described below.

1. Click the Contexts button.

2. Click the Plus icon.

3. In the opened dialog box, fill in the fields as described below.

• Context Name: Enter the context name. The name must start with a letter and contain no spaces. It can consist only of letters, dots, hyphens, or underscores. This is a mandatory field.

  • miniApp Name: Enter the miniApp name.

  • miniApp Type: Enter the miniApp type.

  • Tag: Enter a tag.

At least one of these three fields must be completed. The remaining two fields are optional and can be left blank if desired.

You can add a custom NLU tag in the corresponding text box in your miniApp. In this case, if a user uses one of the alternative words added under the context that has this tag, these words will get the related value.

4. Confirm by clicking the Create button. The context has been added.

Adding context alternatives

Context alternatives are words or short phrases that a customer may use under a specific context. These context alternatives are mapped to the declared reference value or just trigger the entity if no reference value exists, ONLY when the selected context is applicable.

To add a context alternative, proceed as described below:

1. Select a value and click the Arrow button.

2. Click the Add Context Alternatives button.

3. Add a context alternative and press <Enter> on the keyboard. You can use the smart dictionary commands to refer to the existing entities and streamline the process of adding alternatives.

You can add multiple alternatives to one value. For context alternatives, you can use single words, short phrases and regular expressions.

4. Select a context from the drop-down list of previously created contexts.

5. The context alternative has been added. To add another context alternative, click the Plus icon.

Unmarking a context alternative

There may be instances where it becomes necessary to exclude a specific entity from the annotation
if its alternative contains a specific word or phrase. In this case, you can unmark such an alternative to exclude it from being annotated.

You can unmark both context alternative and global alternative.

To unmark a context alternative, proceed as described below:

1. Insert an exclamation mark before the alternative.

2. Press <Enter>. The unmarked alternative will be highlighted in yellow and excluded from the annotation.

You can also left-click an existing context alternative to unmark it.

Deleting a context alternative

To delete a context alternative, click the Delete icon next to it.

Deleting contexts

To delete a context, you should use the search function to first locate where this context is being used. You can search for a context in the Entities list or in the Values list per entity.

Before deleting a context, you should delete all context alternatives that are linked to it or map them to a different context.

To find and deleted a context, proceed as described below:

1. Navigate to the Entities tab and click the Search icon.

2. Enter the context name into the search field and press <Enter> on the keyboard. You will see all the entities that use this context.

3. Select an entity and use the search function in the Values list for the entity. The search result is highlighted.

4. Click the Delete icon to delete the context alternatives linked to the context you searched for. Alternatively, you can map them to a different context from the drop-down list of previously created contexts.

Deleting an entity

To delete an entity, hover over it and click the Delete icon.

You cannot delete the entity that is being used as a reference. To be able to delete it, you have to remove all references to this entity.

Adding custom NLU Logic

The Custom domain is not loaded with a pre-set understanding, hence it allows you complete flexibility to customize it. You can either incorporate your own data or integrate your own NLU Logic. The NLU Logic can be added by uploading a custom NLU file. For this, you can use the NLU file from an exported OCP NLU model.

To integrate your NLU Logic into your Custom domain model, follow these instructions:

1. Navigate to NLU → NLU Models section.

2. Select a model and click on it. The model drill-down page opens.

3. Open the Upload Resources tab.

4. Upload your NLU file. This can be done by either clicking on Browse Files to locate the file on your device or by dragging and dropping the file directly into the upload section.

5. Once your NLU file has been uploaded, it will appear in the Upload History list.

6. Check the current status of the custom RB model. The custom RB model can have the following statuses:

   • Running: The model is up.

   • Working: Temporary status from the model creation up until the model up and its status is changed to Running.

   • Stopped: The model is not running.

7. In case the custom RB model stops for any reason, you can re-activate it manually by clicking on the Menu icon and then selecting Start.