How to handle punctuation and symbol in rasa? - supervised-learning

Question

Rasa version - 1.3.7
pipeline: “supervised_embeddings”
I have trained the bot with no punctuation in intent like.
intent: ask_holiday_in_a_year
How many holidays do we have in a year?
If i ask below question to bot
How many holidays do we have in a year? - ( NLU is able to recognize
it correctly).
How, many ()? Holidays!!,do!##we have$%^ in a %^& year. - (NLU is
able to recognize it correctly.)
How many ###################### holidays do we have in a year? .(NLU
is not able to recognize it correctly.)
How many ####### holidays %%^&*$$% do we have in a year? .(NLU is
not able to recognize it correctly.)
For cases 1 and 2, it worked but for case 3 and 4, it didn’t work? Is there any way(adding some settings in the pipeline)i can handle these symbols and punctuation and give the expected result?

Answer 1

i'm not sure weather this is the correct approach. but, you can try creating random data using a tool like Chatito while including symbols in your train data. again, i'm not sure if this is correct or not

Answer 2

First of all rasa recognize examples based on what you give to it. If you have examples as you 3 and 4 sentences rasa will recognize it. If you think outside of the box there can be multiple issues like this and there is no way rasa can recognize wthat is this and so on. So you want to give examples which are somewhat related to probable questions that can be asked from bot.

Answer 3

This can be handled using a custom nlu component.
In the third parameter of the maketrans function, add the symbols you want to delete.
This custom pipeline will remove all defined keywords and send the filtered text to nlu.
from rasa.nlu.components import Component
import typing
from typing import Any, Optional, Text, Dict
if typing.TYPE_CHECKING:
from rasa.nlu.model import Metadata
class DeleteSymbols(Component):
provides = ["text"]
#requires = []
defaults = {}
language_list = None
def __init__(self, component_config=None):
super(DeleteSymbols, self).__init__(component_config)
def train(self, training_data, cfg, **kwargs):
pass
def process(self, message, **kwargs):
mt = message.text
message.text = mt.translate(mt.maketrans('', '', '$%&(){}^'))
def persist(self, file_name: Text, model_dir: Text) -> Optional[Dict[Text, Any]]:
pass
#classmethod
def load(
cls,
meta: Dict[Text, Any],
model_dir: Optional[Text] = None,
model_metadata: Optional["Metadata"] = None,
cached_component: Optional["Component"] = None,
**kwargs: Any
) -> "Component":
"""Load this component from file."""
if cached_component:
return cached_component
else:
return cls(meta)
Add the pipeline in config.yml
# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/
language: en
pipeline:
- name: "Pipelines.TextParsing.TextParsingPipeline"
- name: "WhitespaceTokenizer"
- name: "RegexFeaturizer"
- name: "CRFEntityExtractor"
- name: "EntitySynonymMapper"
- name: "CountVectorsFeaturizer"
- name: "EmbeddingIntentClassifier"
Source - https://forum.rasa.com/t/how-to-handle-punctuation-and-symbol-in-rasa/19454