-
-
Notifications
You must be signed in to change notification settings - Fork 121
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Connection blocking due to too many requests #347
Comments
In addition, a spam of these messages is sent to the console, which refers to the problem mentioned |
This means that the plugin shutdown. Is it possible to search for the first error? |
It seems to me that the first error are these |
Do you also see the first error about Hikari? |
What's this Hikari? |
It's a database connection pooling library. It was closed. The reason for it should be in the server log. |
I searched for that word and found nothing about it. I was trying to paste the entire log file on hastebin but being very large it is taking a long time to save it |
It should, because the first log you posted contains it. |
Where can I send you the log file? Do you have discord? So check |
Let me know |
Here is the first mistake, the one you asked me for. I managed to find it: https://hastebin.com/kutakazuwa.pl I hope you know how to solve because even at this moment the server is blocked again due to the bot attack that is causing too many requests to be sent to the database with FastLogin and consequently lags everything |
https://hastebin.com/hixifenema.xml This is what happens now when trying to restart Bungeecord. This causes me to block connection to the server machine for more than 10 minutes |
https://hastebin.com/nayiyuyele.pl Can you solve this problem? It causes me to block all the other databases and the machine itself, it happened again today. It really creates a lot of problems for me. Can you solve it? Thanks in advance |
This comment has been minimized.
This comment has been minimized.
I am using this version on bungeecord 1.11-SNAPSHOT-f40e787 |
Preventing bot attacks is not really in scope of this project. It's best to drop them before they reach the Minecraft server part. Another solution is to install an AntiBot plugin that will drop connections if there are too many. The performance impact would be higher than something low level like The error messages you posted show, that your MySQL/MariaDB server is overloaded. Therefore you should check the CPU usage. Is your database server running on the same host? Then it could be affected by a high load of the Minecraft server handling the connections. Do you verified the load? If you identified Java, you can now attach a Profiler and identify the load there. Finding a good solution requires time. There is a lot of other stuff to do besides developing a free Minecraft plugin. Sorry. Nevertheless I added a very basic rate limiter, which will ignore incoming connections if there are too many. |
I have already installed a good anti bot plugin. MySql is overloaded for the number requested by the plugin, however I thought of a hypothetical solution. Soon I will try to implement it hoping that it will be fine. As for the limiter you added, is it available in the latest version? So I unload and install it in place of the current one |
Well FastLogin still sees the incoming connection. Anti bot plugins should prevent connections before plugins could see them. As pointed out in #304, Nevertheless I found that Furthermore I don't know if performing an an extra network request for every joining player in order to check if the IP is blacklisted like many anti bot plugins do, is really useful. It would require a good connection pooling implementation, because connection initialization could be very expensive and has some overhead in performing the request too. The optimal solution would be to different behavior depending on the incoming load. An order like this:
Yes |
I have a feeling that even many paid plugins have this misunderstanding. |
right now at the bungeecord post I use FlameCord which is even better than Waterfall on some points. Anyway okay, thank you for your availability. Later I will try to install the latest version of FastLogin and make changes that I thought of, which, I hope, should solve the problem. Eventually I'll let you know |
Even using the version of the plugin where you said you entered the option that ignores incoming requests when there are too many, I always have the same problem |
You can always adjust the rate limit in the config.
Furthermore please check the usage (CPU, RAM, Query Performance, ...) of the database server to verify that it's really crashing, refusing the connection or just being overloaded. |
What exactly does that number indicate? How many connections can there be before I start ignoring them? What number do you advise me to set? While "expire" indicates after how long (in minutes, seconds) the premium check is reactivated? The main problem should be database overload, which occurs due to the many requests at the same time |
You are right, I'm going to add a better description. For now you can image it like a bucket. Connections are the maximum amount that can be filled. While expire means how long it takes in minutes for every entry to be removed.
FastLogin have to ask the database if any data is available and then on successful force login to save the data. How do you think the saved data is retrieved otherwise? Even caching doesn't work here, because those names will be always new players. For the database FastLogin is already leveraging query caching a lot if possible in order to reduce the load on parsing the query. Retrieving the data also always uses FastLogin isn't likely the only plugin hitting your database. Nevertheless I could also add an option to reduce the pool size of the database connector. That would reduce the amount of work that hits the server concurrently. Current value is the |
I just noticed that the "AntiBot" entry is not present in my FastLogin config. When I updated the version I didn't know I had to clear the config to get it generated again, now I will. Probably, not being present in the config, that option was not active yet, I think. Furthermore, I installed FastLogin on both the Bungeecord server and the lobby server both connected to the same database of course. Do I have to make changes in the lobby one too? Even if the main problem is on bungeecord. As for the database, on the same machine I have also installed other databases but they never give me problems. The car in question is also quite powerful so I shouldn't have any performance issues. The only time I have problems is when this thing I told you about happens. As I said, when this happens all the other databases and the entire server machine also crash (and I can't connect to it for a few minutes, until it recovers). |
BTW: Could you post your configuration.
No it then picks up the default.
The database connection is deactivated on Spigot if you use BungeeCord.
The error means that the database didn't respond for 30 seconds. This could be caused by an overloaded system.
If your database is on the same system, then an high usage for other system also effects the database. If your CPU (example, you could also saturate memory, io bandwidth or others) is already saturated with other workload, the database cannot process the database queries. So in this case, don't assume that there too many requests going to the database. This high saturation for example be caused by too many spawned threads. Thread creation and context switch can be expensive in some situations. So my recommendation is to definitely to some benchmark tests and observe system usages (CPU, memory, etc.). Then check the source of the high usages. Is it your database, a specific Java process or something else? For Java you could attach a Profiler (like Java Mission Control, VisualVM, etc.) to it to analyze it deeper. Is it a specific Thread or sum of specific threads. For example is another plugin doing HTTP calls for every single requests and that caused your system to go down under load. |
So FastLogin which is in the lobby server should I disconnect it from the database? I currently have the one in the lobby and bungecoord connected on the same database. In the end I am attaching the current configuration of both bungeecord and spigot (server lobby) of the plugin. As far as performance is concerned, this problem exists only when there are bot attacks and many requests are made via FastLogin. At other times there are never these problems. Even in the past, when I received bot attacks and I didn't have FastLogin I never had this type of problem so I doubt there could be other causes. Below I attach the configuration of FastLogin on Bungeecoord and Spigot (this is the one not updated with the new option, as I said before I should have the file generated again to get it). Bungeecord: https://hastebin.com/evadakuhic.makefile Spigot (Lobby): https://hastebin.com/ojuyabataf.makefile Let me know if I have to change something and if I have to remove the database connection from the plugin in the spigot server (lobby) |
There is no need to. It's already off. FastLogin will all the work BungeeCord and forward the data down using plugin messages.
That's why you should benchmark it... Do these tests to reproduce it and observe the situation under a controlled environment.
You could disable nameChangeCheck. This makes a Mojang request even if you disabled autoRegister. |
If disabled nameChangeCheck, does something change for users? And with this thing disabled there should be a minor load right? I suppose this option nameChangeCheck is used to check if a premium user has changed nickname, if so, so if it is disabled and a player changes nickname, is it not updated on the server? |
It needs to create HTTP call for every user. That could cause a higher load.
It asks Mojang for the UUID and then checks if the UUID is already in the database. Then it assumes that player is same person and requests an onlinemode connection. |
So if this option is disabled and a registered premium user changes his nickname, will he still be able to enter the server while always keeping his uuid? |
The person needs to activate the premium command again and then re-login, because this decision needs to be done on some factor. |
Doing the premium command will return to its premium uuid, right? |
yes |
Okay, I'll try to disable this option hoping it's better |
Hi, I have a big problem with FastLogin.
In this period my server is receiving numerous bot attacks and in these moments, when the bots try to enter FastLogin starts sending all the requests to the mojang (so I understand) and to the sql database, this causes a block of the server and lag. The server is no longer reachable and no user can enter. This also managed to cause me a whole block of the server machine given the numerous requests to the sql database.
Is there a way to solve this problem?
If it is not clear I use FastLogin through Bungeecord (obviously it is also installed on spigot servers)
The text was updated successfully, but these errors were encountered: