This project contains example tests for a Kafka cluster that can be run on the Testable platform. There are two basic tests:
e2e-test.js: Publish messages on a topic and then consume them in a new consumer group while capturing custom metrics for the number of messages published, consumed, and the E2E latency.publish-and-wait-test.js: Publish messages on a topic, wait for the lag for the specified consumer group id to go to zero, and then perform a follow up step. In this case we capture number of messages published and lag as custom metrics.
To run locally you must have Node.js 12.x or later. Locally only 1 concurrent user will be executed.
./run-e2e-local.sh [kafka-bootstrap-url]
./run-publish-and-wait-local.sh [kafka-bootstrap-url] [consumer-group-id]
Any metrics captured will be output to the console as JSON.
To run on Testable you must have curl installed and have created a Testable account. The number of concurrent users in our example is setup to fit within the limits of the free account.
Step 1: Get a Testable API Key
Login to Testable and go to Organization => API Keys. Copy an API key an set it locally as the TESTABLE_KEY environment variable (export TESTABLE_KEY=xxxx).
Step 2: Start the load test
Check and update the parameters set in run-[...]-testable.sh before starting the test. This includes number of concurrent users, topic, message frequency, regions to generate load, instance type, etc.
./run-e2e-testable.sh [kafka-bootstrap-url]
./run-publish-and-wait-testable.sh [kafka-bootstrap-url] [consumer-group-id]
Use the test run URL to follow progress in real-time. This example project includes custom views (kafka-e2e-view.json and kafka-publish-and-wait-view.json) that will also get uploaded to Testable and set as the default for this test case. The custom view features the key metrics this test captures.
