Freenode would urgently like us to kick out idle users

[ara4n]

As discussed briefly last week, we really should be purging bridged users (and users in general, tbh) who appear to not have logged into their accounts for months. I suggest we just use normal presence to determine this, and have the bridge kick them from the room if their presence has been idle for more than (say) 3 months, on the basis that the remote network doesn't want them lurking. We should make an exception if they are the frontier user.

[TJuberg]

3 months is too long in my opinion, but you should probably try to get some feedback from the channels affected.

Guest users should also be treated differently as they are a lot less likely to return, yet they stay in the channels indefinitely. Thankfully these are easy to spot and clean up for channel operators.

[kegsay]

Is presence reliable enough for us to kill connections based on this? I don't think it is. How will this work for bots? Whilst some bots need to hit /sync (like the github command bot), other bots don't and hence won't ever set presence.

I would absolutely love to have a way to purge idle connections because I don't like having unbounded growth of TCP connections, but I'm not convinced using presence is the right way of approaching this. Sadly, I have no other viable alternative.

[kegsay]

I feel this extends beyond the IRC bridge. Matrix in general has this problem, and I'd rather we solve it for Matrix and then let the bridge's normal membership list semantics remove idle users. This may take the form of an 'idle time' state event which lets people know that they will be kicked if they are idle for more than N minutes in this room, etc. The bridge can then make sure all its bridged rooms have this idle state event set.

[Mikaela]

Would it be possible for one way Matrix integrations to have their ghost killed when they aren't actively doing anything (listening to events on the room)?

I have 15 Twitter ghosts + RSS bot + twitter-bot in one bridged room and I don't think they have any purpouse being in the room when they aren't saying/doing anything, especially those Twitter users who are results of retweets and possibly never say anything again. Twitter-bot again just joins to see that the user adding integration has power and never says anything.

I expressed concern about this earlier at #irc:matrix.org and @Half-Shot reacted by opening Half-Shot/matrix-appservice-twitter#45 .

[kegsay]

See my earlier comment: I feel this is a Matrix-wide problem and should be solved via Matrix rather than IRC.

It's currently not possible for the IRC bridge to work out who is actively doing anything (listening to events on the room).

[Mikaela]

See my earlier comment: I feel this is a Matrix-wide problem and should be solved via Matrix rather than IRC.

I agree, but shouldn't this issue be moved to some more appropiate issue tracker and link there rather than being here?

[kegsay]

We currently have:

• https://github.com/matrix-org/matrix-doc/issues/524
• https://github.com/matrix-org/matrix-doc/issues/488

[kegsay]

I spoke to @leonerd about this, and whilst presence isn't necessarily super-reliable, the last_active_ago time should be fairly accurate, in that if it says they were last active 2 months ago, you can be reasonably sure that isn't a lie.

I'd like to conduct a few tests to experiment and see how many users would be affected by such a hypothetical threshold, and then we can decide if it's feasible to do.

[ara4n]

What a mess. :( Freenode is now formally complaining to me about Matrix (PMs from IRCops) and we've been banned from ##linux and ##ubuntu due to the amount of join/part spam from stale users when the bridge restarts :(

The reason for suggesting to do this in a bridge rather than solve it across all of Matrix is because a) it's urgent and we need a fix that doesn't drag on for months, and b) the idler threshold is surely a per-room policy, and this feels like it's most easily enforced with a bot/bridge (especially when we already have one one in the effected rooms). In other words, if ##ubuntu's FN ops want to kick idle Matrix users after 3 days of no-presence, they should be able to. It shouldn't matter what HS they arrive from, or what client they're using. And the place to enforce that is then the bridge that is linking into FN. If people on the Matrix side dislike that idler policy, then they shouldn't be bridging into the IRC channel anyway, given they need to respect the rules of that channel.

I also proposed using normal presence for this because these days it is indeed persisted, and should be reliable enough (as @leonerd says).

Please can we just do this before we burn further bridges (hah) with FN?

I'm not sure that we even necessarily need a matrix-wide solution to this; if a room gets big and bad enough to merit kicking idlers, then they should go find a bot/bridge that can do that with whatever threshold logic is required, rather than trying to bolt it into the federation consensus protocol itself...

[ara4n]

Anonymized IRC Chanop (IRC)
Hi, i'm one of the ops of a channel here on freenode. We were getting annoyed with the disconnects. I understand its an issue that is in the process of getting resolved but for now we will forward all gateway/shell/matrix.org masks to ##fix_your_connection.
Sorry for the maybe harsh treatment, but otherwise i will get the harsh treatment from our users.

Shell
Anonymized IRC Chanop: note that the Matrix bridge doesn't understand forwards, so you might as well ban them.
Anonymized IRC Chanop: what ##ubuntu has done is ban all Matrix users with default nicknames (ending in [M]), I think.

Anonymized IRC Chanop (IRC)
Ah ok. I'll make it a ban only.
If this hickup is sorted out, feel free to contact me. I will remove the ban.

:(

[eternaleye]

@ara4n @Mikaela

One caveat with using presence and/or doing this in the bridge is that it resolves one issue (idle people cluttering the IRC user list) by worsening another, which Freenode is also unhappy about: people being able to "invisilurk" and see history without being seen.

As a result, I think @kegsay is correct that this requires a more thorough solution at the Matrix level - otherwise, we're just trading one problem for another, when both are already known to be issues Freenode has strong opinions on.

[kegsay]

I spoke to @leonerd about this, and whilst presence isn't necessarily super-reliable, the last_active_ago time should be fairly accurate, in that if it says they were last active 2 months ago, you can be reasonably sure that isn't a lie.

I'd like to conduct a few tests to experiment and see how many users would be affected by such a hypothetical threshold, and then we can decide if it's feasible to do.

Tests were done and this seems like a reasonable solution, so I've rigged together a script which will purge idlers. The current magic threshold for kicking from the bridge is currently set to 30 days. I'll be running this script over the next few days (>2.5k rooms to query and then subsequently kick). Sample done on some test rooms:

#ipfs Kicked 187 users, IRC membership count now sits at 661
#nixos Kicked 15 users, IRC membership count now sits at 478
#matrix-irc Kicked 60 users, IRC membership count now sits at 192

Importantly, this has nothing to do with the state of these users IRC connections: They will remain until the bridge gets restarted, which is intentional. It's a compromise between making it impossible for IRC users to PM Matrix users if they are in no channels, and reaping vital resources on the bridge.

[kegsay]

Going forwards, we can periodically fire this script (e.g. once every fortnight) to keep the growth of idle connections down, and hence the join/part spam if/when the bridge restarts. We can't eagerly reap connections as they tick over the 30 day mark because the bridge doesn't get told when this happens, hence the polling solution (batched once per fortnight) feels like an acceptable compromise.

[ara4n]

@eternaleye the 'invisilurk' problem is separate to this one - it's bug matrix-org/matrix-spec-proposals#212 and friends. i don't think we're making that any worse (or better) by kicking idle users.

[ara4n]

@kegsay: can we do the #ubuntu and ##linux rooms (carefully) first, to try to mend relations with the chanops there atm?

[kegsay]

@kegsay: can we do the #ubuntu and ##linux rooms (carefully) first, to try to mend relations with the chanops there atm?

Will do that first thing tomorrow morning, when Freenode traffic is typically at its lowest.

[kegsay]

#ubuntu and ##linux were done this morning but there were few users to kick because they had already applied banmasks which removed most of the Matrix users. #ubuntu had a slightly more generous mask which only removed users ending in [m], so a handful of users were removed from that channel.

I've now set the script going to enumerate all bridged rooms. Will close this issue after this has been run, as all idle users (where idle is not being active on Matrix for >30 days) will have been removed from IRC.

[christeld]

Please accept my apologies for butting in but I felt I had better clarify that freenode has NOT requested that Matrix kick idle users, urgently or not.

I assume there must be some confusion on your part as none of the freenode volunteers have been able to recall discussing this matter with you @ara4n -- it is also not a topic that has been discussed by freenode. A great deal of users will idle for extensive periods of time, whether connecting over a Matrix bridge or leaving their screen/tmux detached for weeks and even months and it is not our place to set down 'idling rules' -- although individual projects using freenode infrastructure as part of their everyday communications with their communities may of course choose to do in line with their specific community guidelines/rules/codes of conduct.

In the event that freenode wishes to address a matter relating to Matrix and freenode we would do so by reopening the support ticket where we have previously discussed matters relating to your projects -- much in the same way you do when contacting us.

With that in mind I am somewhat baffled and confused as to 'freenode is now formally complaining to me about Matrix (PMs from IRCops)' and would appreciate it if you could clarify which freenode staff has contacted you with formal complaints as that would be an issue we would need to address internally considering the above.

Thank you!

[kegsay]

I've now set the script going to enumerate all bridged rooms. Will close this issue after this has been run, as all idle users (where idle is not being active on Matrix for >30 days) will have been removed from IRC.

This script has now finished. I'm in the process of setting up a cronjob to periodically do this so we don't build up idlers in the future.

[ara4n]

For those still getting updates on this bug: kicking idlers who have not touched their matrix account for > 1 month has reduced the number of connections to Freenode from 6940 to 4498; this hopefully counts as a win, although does mean that there's still quite a lot of join/part spam on busy rooms when the Matrix bridge bounces. Meanwhile matrix-org/matrix-spec-proposals#38 and matrix-org/matrix-spec-proposals#347 can help further with that.

@christeld: sorry for the delay in response to your comment above. in terms of FN feedback, i've honestly not been tracking which convos have been with ircops (e.g. e) and which with chanops (e.g. elky), and which have been about join/part spam from Matrix. Irrespective of where the feedback has come from, I personally agree that it's a horrible situation that when the matrix.org bridge to freenode restarts, it floods rooms with hundreds of joins & parts. I've seen complaints from ##linux, #ubuntu, #arduino, #prometheus and many more. Hence the urgency of wanting to at least get rid of the stale users (otherwise they'll literally stack up forever on freenode!).

Just to be clear, we are NOT timing out connections where users do not speak. We are timing out connections where the user has not logged into their Matrix account whatsoever for over a month, and so is neither reading nor writing - i.e. is simply not using the system. Obviously if we get complaints from these stale users then we'll reconsider this, but I'd be amazed if anyone objects seriously to that.

Meanwhile, we have matrix-org/matrix-spec-proposals#38 and matrix-org/matrix-spec-proposals#347 to help reduce join/part spam further. Hope this makes sense and resolves any concern about what's going on here. Please come hunt me down on #matrix to discuss further if needed (or use the FN bugtracker; up to you :)

[fuzzy76]

Is there any way you can share this script, or (ideally) have this as part of the bridge behind a configuration? My own bridge leaves ghosts from time to time, and it is very irritating. Particulary in small channels.

[Mikaela]

@fuzzy76 In case you didn't already find it, I think it's https://github.com/matrix-org/matrix-appservice-irc/blob/master/scripts/remove-idle-users.py .

[supagu]

I have a matrix channel where users do not always log out nicely so remain "idle". Can I apply this script to my room to run periodically or must I run it on my own server like a cron job?

[Mikaela]

I have a matrix channel where users do not always log out nicely so remain "idle". Can I apply this script to my room to run periodically or must I run it on my own server like a cron job?

I believe that the script has to be run on the server running the specific IRC bridge.