Rasa CALM Demo

This demo showcases a chatbot built with Rasa's LLM-native approach: CALM.

Caution

Please note that the demo bot is an evolving platform. The flows currently implemented in the demo bot are designed to showcase different features and capabilities of the CALM bot. The functionality of each flow may vary, reflecting CALM's current stage of development.

Note

This demo bot is currently compatible with 3.10.1.

Terms of Use

This project is released under the Rasa's Early Release Software Access Terms.

Demo Bot

The demo bot's business logic is implemented as a set of flows, rules, and stories, which are organized into three main skill groups: Contacts, Transactions, and Others/Misc.

The skill groups Contacts and Transactions are implemented using CALM, e.g. are defined in flows. The skill group Others/Misc is implemented via the nlu-based approach. Coexistence allows you to run a single assistant that uses both the Conversational AI with Language Models (CALM) approach and an NLU-based system in parallel.

Each flow consists of a yaml file and a domain definition, which includes actions, slots, and bot ressponses.

Additionaly, the bot can showcase the enterprise search capability based on the SQUAD dataset.

The table below shows all the skills implemented in the bot:

Skill Group	Flow Name	Description	Link to flow	Link to domain
Contacts	Add new contact	Adds a new contact to the user's list.	Link	Link
	Remove contact	Removes selected contact from the user's list.	Link	Link
	List contacts	List all of user's saved contacts.	Link	Link
Transactions	Check account balance	Allows users to check their current account balance.	Link	Link
	Transfer money	Facilitates the transfer of funds to user's contacts.	Link	Link
	Setup recurrent payment	Sets up recurring payments which can either be a direct debit or a standing order.	Link	Link
	List transactions	List the last user's transactions.	Link	Link
	Replace card	Replace the user's card.	Link	Link
	Replace eligible card	Replace the user's card that meets specific eligibility criteria. This is a flow link exclusively accessed by replace_card flow	Link	N/A
	Verify account	Verify an account for higher transfer limits.	Link	Link
Ordering Pizza	Order Pizza	Allows users to order a pizza.	Link	Link
	Fill pizza order details	User is asked to fill out pizza order details.	Link	Link
	Use membership points	User asks to use membership or loyalty points.	Link	Link
	Correct Order	Allows users to correct order details.	Link	Link
	Correct Address	Allows users to correct the delivery address.	Link	Link
	Job vacancies	Allows users to ask for job vacancies.	Link	Link

Skill Group	Name	Description	Link to story, rules, nlu data	Link to domain
Others / Misc	Book Restaurant	Make a reservation at a restaurant.	Link	Link
	Health Advice	Detects an out-of-scope topic: health advice.	Link	Link
	Hotel search	Search for a hotel and show hotel rating.	Link	Link

Skill Group	Name	Description	Link to loading script
Enterprise Search	Q&A based on SQUAD Dataset	Load and search the https://huggingface.co/datasets/rajpurkar/squad dataset.	Link

Rasa ships with a default behavior in CALM for every conversation repair case which is handled through a default pattern flow. In addition to its core functionality, the demo bot also includes an examples of pattern overriding in data/flows/patterns.yml.

Running the project

This section guides you through the steps to get your Rasa bot up and running. We've provided simple make commands for a quick setup, as well as the underlying Rasa commands for a deeper understanding. Follow these steps to set up the environment, train your bot, launch the action server, start interactive sessions, and run end-to-end tests.

Installation

Important

To build, run, and explore the bot's features, you need Rasa Pro license. You also need access to the rasa-pro Python package. For installation instructions please refer our documentation here.

Note

If you want to check out the state of the demo bot compatible with Rasa 3.9, please check out the branch 3.9.x.

Prerequisites:

• rasa pro license
• python (3.10.12), e.g. using pyenv pyenv install 3.10.12
• Some flows require to set up and run Duckling server The easiest option is to spin up a docker container using docker run -p 8000:8000 rasa/duckling. Alternatively, you can use the make run-duckling command locally. This runs automatically only when you use the make run command, before it launches the Inspector app.

After you cloned the repository, follow these installation steps:

1. Locate to the cloned repo:

   cd rasa-calm-demo
   

2. Set the python environment with pyenv or any other tool that gets you the right python version

   pyenv local 3.10.12
   

3. Install the dependencies with pip

   pip install uv
   uv pip install rasa-pro --extra-index-url=https://europe-west3-python.pkg.dev/rasa-releases/rasa-pro-python/simple/
   

4. Create an environment file .env in the root of the project with the following content:

   RASA_PRO_LICENSE=<your rasa pro license key>
   OPENAI_API_KEY=<your openai api key>
   RASA_DUCKLING_HTTP_URL=<url to the duckling server>

5. If using qdrant for extractive search:

   • Setup a local docker instance of Qdrant

     docker pull qdrant/qdrant
     docker run -p 6333:6333 -p 6334:6334 \
        -v $(pwd)/qdrant_storage:/qdrant/storage:z \
        qdrant/qdrant
     

   • Upload data to Qdrant
     • In your virtual environment where Rasa Pro is installed, also install these dependencies:

       pip install uv
       uv pip install -r qdrant-requirements.txt
       

     • Ingest documents from SQUAD dataset (modify the script if qdrant isn't running locally!)

       python scripts/load-data-to-qdrant.py
       

   You can toggle parameter use_generative_llm in config.yml to change the behavior. The answer is selected from the first search result -> metadata -> answer key

Custom Information Retriever

You can use a custom component for Information Retrieval by defining the custom component class name in the config as follows:

policies:
- name: FlowPolicy
- name: EnterpriseSearchPolicy
vector_store:
   type: "addons.qdrant.Qdrant_Store"

This configuration refers to addons/qdrant.py file and the class Qdrant_Store. This class is also an example that information retrievers can use a custom query, note that in search() function the query is rewritten using the chat transcript by prepare_search_query function.

Configuration

Check config/config.yml to make sure the configuration is appropriate before you train and run the bot. There are some alternative configurations available in the config folder. Theses can be used via the appropriate make command during training.

Training the bot

To train a model use make command for simplicity:

make rasa-train

which is a shortcut for:

rasa train -c config/config.yml -d domain --data data

Alternative configurations can be accessed for Multistep command generation:

make rasa-train-multistep

or for Enterprise search with qdrant for extractive search:

make rasa-train-qdrant

The trained model is stored in models directory located in the project root.

Starting the assistant

Before interacting with your assistant, start the action server to enable the assistant to perform custom actions located in the actions directory. Start the action server with the make command:

make rasa-actions

which is a shortcut for:

rasa run actions

Once the action server is started, you have two options to interact with your trained assistant:

1. GUI-based interaction using rasa inspector:

rasa inspect --debug

2. CLI-based interaction using rasa shell:

rasa shell --debug

Running e2e tests

The demo bot comes with a set of e2e tests, categorized into two primary groups: failing, and passing. These tests are organized not per individual flow but according to CALM functionalities.

Note

The passing and failing statuses are relative to the performance of the GPT-4, which is enabled by default. The use of different models may yield varying results.

You have the flexibility to run either all tests, only the passing tests, only the failing tests, or a single specific test.

---

To run all the tests you can use the make command:

make rasa-test

or

rasa test e2e e2e_tests

---

To run passing/failing/flaky tests you can use the make command:

make rasa-test-passing

make rasa-test-failing

make rasa-test-flaky

or

run rasa test e2e e2e_tests/passing

run rasa test e2e e2e_tests/failing

run rasa test e2e e2e_tests/flaky

---

To run a single test with make command, you need to provide the path to a target test in an environment variable target:

export target=e2e_tests/path/to/a/target/test.yml

and then run:

make rasa-test-one

or

rasa test e2e e2e/tests/path/to/a/target/test.yml

---

To run only the tests which are relevant to multistep command generation, you can use the make command:

make rasa-test-multistep

or

run rasa test e2e e2e_tests/multistep

E2E tests with assertions

To enable the feature, please set the environment variable RASA_PRO_BETA_E2E_ASSERTIONS to true in your testing environment.

export RASA_PRO_BETA_E2E_ASSERTIONS=true

To run all the tests:

rasa test e2e e2e_tests_with_assertions

---

To run passing/failing/flaky tests:

rasa test e2e e2e_tests_with_assertions/passing

rasa test e2e e2e_tests_with_assertions/failing

rasa test e2e e2e_tests_with_assertions/flaky

---

To run a single test , provide the path to a target test:

rasa test e2e e2e_tests_with_assertions/tests/path/to/a/target/test.yml

---