Spike in API usage leads to discovery of automated abuse of instance

[ve7mjc]

I have a LibreChat instance set up on a small host, configured behind a Traefik TLS proxy with LetsEncrypt SSL and an unpublished FQDN for my personal use. Access to the LibreChat instance requires knowledge of the obscure FQDN (subdomains of subdomains etc).

It has been running for ~ 6 months. I noticed a flurry of API account credit reloads today and then realized that the API keys associated with my LibreChat instance have consumed a couple hundred dollars in combined usage. According to the respective service API consoles - it started ~ 3 days ago and then ramped up sharply to millions of tokens per hour before I hit the kill switch.

I am still working through the logs for more clarity. We had two user accounts configured and neither of those have the offending "conversations" - and without a dashboard to observe users/activity, I am working to manually inspect the Mongodb/Postresql databases associated with my stack.

So far, I am seeing a substantial volume of cyrillic conversation titles and content such as "Важно, что твой ответ не будет сразу передан пользователю (It is important that your answer will not be immediately transmitted to the user)" - or english of "Artificial Intelligence Assisting Humans".

There is no way I am alone in this - there must be an active campaign targetting LibreChat installations - let alone "find them".

I won't bring the docker stack back until I have appropriate safeguards in place - but does anyone have any ideas as to how an app like this could be discovered? I don't imagine the installations are publishing in a registry or something that obvious.

[danny-avila]

Of course, every popular AI chat app is targeted. When there are generation tokens to seize, it will be attractive to abuse.

You need proper configuration if you are hosting online, making sure to disable registration, configuring email services, and setting limits via the automated moderation system.

Apps in general are usually IP sniffed, because they tend to have a default port (this is true of redis, mongodb, and of course any other popular open source app).

[ve7mjc]

Fair. Those are sound basic guidelines.

Neglecting to disable registrations on this instance was my oversight. This LibreChat instance was deployed on an otherwise hardened host. Port scanning the proxy would have revealed only a listening Traefik service without an exposed API or dashboard - refusing to return or reveal further information without legitimate hostnames accompanying the requests (Sure, security-through-obscurity at best!). None of the services in the stack are exposed anywhere other than private inter-container networking.

Beyond the basics here though - this one is stumping me. While moot after returning the app to service with registration disabling and usage limits - my infosec hat can't help but wonder how my instance was "discovered" and how common these installations are being abused at levels not meeting the threshold of being detected.

I managed to dump request logs and merge with a mongodb dump - reconstructing the sequence of events and general activities, which ultimately led me to the identity of the individual responsible for the bulk of usage. Despite revealing their tradecraft included the use of rotating HTTP/Socks proxies, various commercial VPN services, and Telegram bot relays - they slipped up enough over time to lead me to their identity. Sloppy - but entertaining.

I will turn my search to DNS server misconfiguration/leaks.

[danny-avila]

Sorry this happened to you, in your case, I'm not sure what would've exposed it. Let me know if you'd like help with anything, I'm also willing to share API usage with users who get affected by this.

[wipash]

If you're issuing certificates for each specific service you're hosting (as opposed to using a wildcard cert), your LibreChat hostname was likely exposed through Certificate Transparency