Allow statements to be filtered by their tags

[gunthercox]

This issue is blocked by #924 and can't be worked on until those changes are made.

---

Changes need to be made to ChatterBot's storage adapters so that statements can be filtered by their tags attribute.

This is an important change that will allow more efficient response generation. After this change is made it will be possible to add a processing step where new input statements are classified based on the labels of categories of the dialog that was used to train the chat bot (adding this additional processing step will have a separate issue and pull request created for it).

[vkosuri]

@gunthercox Master, If you didn't started working on this, I am happy to work on this feature, Is there any ETA to do this?

[vkosuri]

I am planning to write a utility method to find categories something like below

Question: If same/similar statement present in multiple categories are we going considering those?

def find_categories(corpus_paths, input_statement):
    data_file_paths = self.list_corpus_files(corpus_paths)
    categories = [list]
    for file_path in data_file_paths:
        data = read_corpus(file_path):
        for k, v in data.items():
            if input_statement in v:
                categories.append(k)
                
    return categories

[gunthercox]

@vkosuri Thank you, that would be greatly appreciated.

Yes, multiple categories for a statement should be supported.

Also, if you check the corpora variable on #L119 of trainers.py you should be able to get the categories from it. They are being added in the load_corpus function of chatterbot-corpus.

corpora = self.corpus.load_corpus(corpus_path)
for category in corpora.categories:
    # ...

Thank you again, let me know if anything comes up that I can assist with. 👍

[vkosuri]

Thanks Master, Another question

1. is None is the default category?
2. If user didn't specified any category, do we need search whole corpora? If not what's your thoughts?

[gunthercox]

1. is None is the default category?

Since a statement can have multiple categories, the default should be an empty list.

2. If user didn't specified any category, do we need search whole corpora? If not what's your thoughts?

Yes, but that is going to be a different pull request. #925 describes this a bit more.

[Issen007]

@gunthercox @vkosuri in what area should we start using tags?
I'm playing around with this at the moment and could I add tags example in to the Corpus training data files?

Thanks

[gunthercox]

@Issen007 Tags will be added to the statements based on the category field from their corpus data (at least when training with the chatterbot-corpus).

[jxfruit]

@vkosuri hi,bro. you are really great. I am caught in problem with efficiency, even i use mongodb to be storage adapter, the response is still too slow. I found you are very active. So can you share some experience of improving efficiency or using methods. thank u very much!!!

[calmzealA]

Great work! This is a very important feature.
It's slow for now while corpus over 10000 records.

[ghost]

Might want to consider porting the logic adapters over to C. Even with tags, you will be crunching too much data for python to handle within any reasonable time limit.

[gunthercox]

I'm going to close this off now that the ability to filter statement results by tags has been added.