Based on #6288

…isconnected would cause a send on closed channel

…y when

config variable set

…ddleware

This commit introduces a major change in the streaming architecture:

- Replace shared Redis streams with direct user-to-stream connections
- Each user now has their own dedicated stream, improving isolation and performance

Additional improvements include:

- Refactor StreamingMiddleware to use consumer group managers
- Update StreamManager to use sync.Map for thread-safe operations
- Remove Redis dependency and related functionality
- Implement garbage collection for unused consumer groups
- Update tests to reflect new structure and functionality
- Add support for Kafka in streaming tests
- Improve error handling and logging throughout
- Update go.mod and go.sum with new dependencies

These changes significantly enhance the scalability and efficiency of the streaming system by establishing direct connections between users and their respective streams, eliminating the need for a shared Redis-based message queue.

This commit refactors the `internal/streaming/manager.go` file to update the `Stream` struct and its methods. The main changes include:

1. Remove unused imports: `crypto/sha256` and `sync`.
2. Update `addMetadata`, `GetHTTPPaths`, `removeUnsafe`, and `removeConsumerGroup` methods to work with the `Stream` struct instead of `StreamManager`.
3. Remove references to `StreamManager` and update method receivers to use `Stream`.
4. Update `GetHTTPPaths` to use the `streamConfig` field of the `Stream` struct instead of loading from a map.

These changes align the `Stream` struct with the new design, allowing it to manage a single stream configuration and its associated operations.

This commit enhances the Stream.Stop() method to make it more robust and prevent nil pointer dereferences. The changes include:

1. Adding a nil check for the stream before attempting to stop it.
2. Implementing a timeout mechanism using context to prevent indefinite hanging.
3. Using a goroutine and channel for asynchronous stream stopping.
4. Improving logging to provide more detailed information about the stopping process.

These modifications aim to resolve issues related to stopping non-existent streams and make the overall process more reliable.

This commit improves the `testAsyncAPIHttp` function in the `gateway/mw_streaming_test.go` file to increase test reliability and provide more detailed logging. The changes include:

1. Increase initial WebSocket connection delay from 1 to 2 seconds
2. Extend overall timeout from 10 to 30 seconds
3. Increase inactivity timeout from 2 to 5 seconds
4. Add final log message showing total received messages

These modifications aim to give the test more time to receive messages and offer more comprehensive logging information, facilitating better diagnosis of potential issues in message reception.

This commit improves the `testAsyncAPIHttp` function in the `mw_streaming_test.go` file to provide better debugging information and increase the chances of successful message reception. The changes include:

1. Increase initial WebSocket connection stabilization delay to 5 seconds
2. Add logging for WebSocket connection stabilization
3. Extend overall timeout to 60 seconds
4. Increase inactivity timeout to 10 seconds
5. Add a 5-second delay before closing WebSocket connections
6. Improve logging throughout the test execution

These modifications aim to address potential timing issues and provide more detailed information about the test's progress, facilitating easier debugging of any remaining issues.

… middleware