Mobile push notifications are no longer working

[Drangleic]

Description:

With default Push settings, mobile push notifications have stopped working for all of our users. By "stopped working", I mean that no one is receiving them, even when @ mentioned. In testing myself, I have enabled notifications for all messages on desktop and mobile, and it didn't fix the problem. This is especially odd since they were working last week.

I've tried toggling the options under Push to no avail. Resetting to defaults and saving has no effect either.

Our current settings:

FYI, everyone accessing our chat is using the native Rocketchat + mobile app. Most of my testing has been done using the "Send a Test Push Notification"

Server Setup Information:

• Version of Rocket.Chat Server: 0.62.1
• Operating System: Ubuntu 16.04.4 LTS
• Deployment Method(snap/docker/tar/etc): tar
• Number of Running Instances: 6
• DB Replicaset Oplog: Enabled
• Node Version: 8.9.4 LTS
• mongoDB Version: 3.6

Expected behavior:

Mobile notifications should be pushed to mobile devices.

Actual behavior:

No notifications are sent.

Relevant logs:

No errors are seen in the browser console, and I see no logs from today's testing. If I come across relevant logs, I will attach them in a comment.

[lucianlazar1983]

I confirm the issue, no mobile push notifications and also "send a push to my user" button produces the following error: "There are no tokens for this user"

[theorenck]

Thanks for reporting it to us @Drangleic. Hey @sampaiodiego, can you take a look in to this one?

[sampaiodiego]

we have changed push gateway DNS recently. can you guys please confirm gateway.rocket.chat is resolving to 104.196.189.8 on your serves?

[Drangleic]

@sampaiodiego I checked and verified that gateway.rocket.chat is resolving to the address you listed in your comment.

[Drangleic]

@sampaiodiego @theorenck Wanted to let you know that rebooting each of the servers in our rocketchat setup fixed the problem. After rebooting the first server, everyone received a slew of notifications for past messages, almost like RocketChat had been saving them up.

Does RocketChat resolve gateway.rocket.chat on the only on the start of the app? The only thing I can think of is that it was storing the old IP and never re-resolved the Push Gateway DNS.

[sampaiodiego]

Does RocketChat resolve gateway.rocket.chat on the only on the start of the app?

That might be a NodeJS DNS cache issue, maybe.

[sampaiodiego]

I'll close the issue since it seems solved 😃

[megamaced]

Please reopen, this is not a new issue, it has been happening for months. After about a week of uptime, push notifications seem to stop getting sent out, and then when RC is restarted, the backlog is spat out to all users

[sampaiodiego]

@megamaced do you ming opening a new issue describing what you are facing? it does not seem related to this one, thx

[tjommie]

I know this is already closed, but we have been experiencing the exact same issue for months now. A restart solves the issue for a brief period of time, where all old push notifications come through at once. After a day or two push notifications simply just stop again.

gateway.rocket.chat resolves to 104.196.189.8

There are no obvious errors in the logs, and push notification info in the logs look successful as far as I can see.

I do not mind creating a new issue, if this is required, what information would you need to know?

Thanks!

[easy-admin]

Just installed the Rocket Chat on my CentOS 7 Box, did a lot of testing, no luck puch notification for mobile IOS/Android still not working ;-( any news about the fix ?

Regards

[bayareacoder]

Any update on this? Just installed and now testing Rocket.Chat to consider moving from Slack and using it for LiveChat also, but if push notifications are not working or not working reliable, it's basically useless (sorry if that statement ruffles feathers...). Using deployment on Heroku and standard IOS app. Not receiving any notifications for chatroom msgs with either All or Mentions setting. The 'send test push to my user' from the Admin panel does work with gateway set to https://gateway.rocket.chat so clearly not a DNS issue. Also what is the logic for the '5 minute' window of not sending notifications after you close web app? Especially for live chat requests it should send immediate notifications I believe.

[magicbelette]

Hi,

@bayareacoder could you try "Administration > General > Notifications > Always notify mobile" please ?

Cheers

[bayareacoder]

HI, yes I got it to work by changing this general setting, rather than the room setting. There are still issue though with notifications.

1. The above issue: per-room override does not seem to affect general setting
2. Why is there a 5 minute timeout between logging off from desktop app to receiving push on mobile client as has been reported? I believe if you have it configured to receive notifications for all messages, it should be for all message as soon as you're not logged in on any client.
3. Not getting push notifications on IOS app for live chat, neither for 'least routing' nor for 'guest routing' for a logged in user who is an agent. (desktop notifications work, but not mobile). Pretty crucial that a support agent receives a notification of a customer chat waiting on whichever client he/she is connected.

[sunnythaper]

I guess I just want to ditto on the issue everyone is having. I have tried turning the gateway off to use my own server as the gateway and tried multiple reboots, rebuilding the container, everything mentioned as a solution on this thread, etc. Still no notifications.

[NOBLES5E]

still having this issue on a fresh installed rocket chat

[sampaiodiego]

guys, we've fixed a few issues regarding push notifications on the current release candidate 0.69.0-rc.0.. can you please give it a try?

if still not receiving push notifications can you please describe exactly what are you doing?

you can do the following steps to test:

• spin up a server (or make sure you're using the default values)
• make sure the server has access to internet
• create the admin user via web
• download one of mobile apps: Android or iOS
• add the server that you've just spun up to the mobile app
• log in with the same user you created before
• close the app
• go to web again, on the admin panel, under push notification section click on "send a push test to my user"
• you should receive a success toast message saying a push was sent to 1 device.

if you didn't receive the notification, please check server logs if there is any useful info.

[sunnythaper]

@sampaiodiego my team has since moved to Mattermost to test that product out but I can say before we switched, the steps you have listed are what I followed. I would get the test notification no problem but none of my triggers would result in a notification. No matter my logged in/out status, avail, away, dnd, etc.

[uncledent]

We are experiencing the same issue. Pushes work for like a week or more and then just stop. No errors in logs. They just stack in DB with sent: false, sending: 0.

Rebooting server sends all old notifications and helps for a while. We run multiple instances on Kubernetes.

Version: 0.66.3
Linux 4.15.0-1013-azure
Node v8.11.3
Running Instances: 3

[sampaiodiego]

@sunnythaper thanks for input.. we have have two kind of issues so far:

• issues with the server not sending push notifications
• issues with people thinking they have received a push notification but they didn't

the main problem in my opinion is that people doesn't have a way to find out what are their case. so I'm working with the whole team to make it easier to identify what issues people having and letting them know what to do to solve it.

[sampaiodiego]

@uncledent the issue you're facing was fixed on version 0.69.0 released this week.. pls make sure to give it a try.

the issue was not even on our source base, it was on a package we use. I've submitted a pull request for them Meteor-Community-Packages/raix-push#344 since it was not merged yet, we decided to fork it and release 0.69.0 with the fix.

[uncledent]

@sampaiodiego thank you, we will give a try

[nanocat-net]

I'm running 0.69.1 (from docker rocketchat/rocket.chat) and I have the same issue - can you help me tell you how to help to debug it?

Symptoms: Notifications seem to just stop, after a few days of the container running.

[sampaiodiego]

hi @grooverboy .. is there anything that might help on server logs?

the collection _raix_push_notifications is the queue of push notifications to be sent. if there are stuck notifications, you run a find command on it:
db.getCollection('_raix_push_notifications').find({})

[phoenix-bjoern]

I get the following error:

rocketchat_logger rocketchat_logger.js:278 �[31mSystem ➔ error Error sending push to gateway (4 try) -> { Error: tunneling socket could not be established, statusCode=400 at ClientRequest.onConnect (/app/bundle/programs/server/npm/node_modules/meteor/http/node_modules/tunnel-agent/index.js:166:19) at Object.onceWrapper (events.js:319:30) at emitThree (events.js:136:13) at ClientRequest.emit (events.js:217:7) at Socket.socketOnData (_http_client.js:474:11) at emitOne (events.js:116:13) at Socket.emit (events.js:211:7) at addChunk (_stream_readable.js:263:12) at readableAddChunk (_stream_readable.js:250:11) at Socket.Readable.push (_stream_readable.js:208:10)

[sampaiodiego]

Error: tunneling socket could not be established

@phoenix-bjoern based on error message looks like you're running rocket.chat server behind a proxy, is that the case? if so, are you setting env vars HTTP_PROXY and HTTPS_PROXY?

[phoenix-bjoern]

@sampaiodiego we have a firewall which allows https/443 connections to wan and an nginx proxy to offload ssl for Rocketchat. But we don’t run a classic http/https proxy to handle wan requests, no.

[sampaiodiego]

@phoenix-bjoern when push gateway is enabled, Rocket.Chat will send the push notification to https://gateway.rocket.chat that will then deliver the content to correct push service (Apple or Google). so you'll need to make sure your Rocket.Chat instance can make HTTP requests to that address.

you can also turn the gateway off and send pushes directly to Apple/Google, but in this case you'll need to fill in your own app certificates (you won't be able to use our official apps).

[nova168168]

have same issue here !!! mobile can't receive any notification !

Rocket.Chat version 0.73.2

OS Type | Linux
linux
x64
4.4.0-87-generic
v8.11.3

[sampaiodiego]

@nova168168 can you please open a new issue describing exactly in what scenarios you're missing push notifications and providing as much details as you can? thanks

[vinhnh3918]

@phoenix-bjoern when push gateway is enabled, Rocket.Chat will send the push notification to https://gateway.rocket.chat that will then deliver the content to correct push service (Apple or Google). so you'll need to make sure your Rocket.Chat instance can make HTTP requests to that address.

you can also turn the gateway off and send pushes directly to Apple/Google, but in this case you'll need to fill in your own app certificates (you won't be able to use our official apps).

this is my config and I get the same phoenix-bjoern's result

[GoetheG]

I can confirm that this issue is not resolved. The push notification topic is still really complicated and does not work properly. I don't know why.

[KayzerCode]

The same with me.
Looks like nobody cares.

[sunnythaper]

Damn, this post just reminded me how long ago I tried out RocketChat and I'm surprised this still isn't fixed. I'm not joking when I say the only reason we switched to MatterMost (and honestly love it now) is solely because this one feature didn't work on RocketChat. Had this issue not been a thing for us, I honestly think I'd be on RocketChat to this day.

[sampaiodiego]

@GoetheG what do you mean by notification topic? what exactly is not working for you?

@KayzerCode what do you mean by anybody cares? I'm here from the very beginning trying to fix multiple issues you guys are facing.. I haven't heard from you before that you're having issues btw

@sunnythaper I'm not sure what exactly you mean by this is not fixed, as many different subjects were discussed in this issue.. just so you know, our push gateway is now sending over a million push notifications per day, so I think most people don't have issues with it.. sadly you moved before I could try help you. hopefully you can give rocket.chat a try again in the future 😉

[KayzerCode]

"what do you mean by notification topic? what exactly is not working for you?"

Me and my system administrator were trying to make RocketChat possible to push notification about new messages to mobile. I use "Send Test Notification" button and only once it worked.
Never ever later.
So, the point is: we are still on RocketChat, but this problem make us think about alternatives.
We created channels, invited friends and clients, believing that this problem is what we could manage later. Unfortunately.
What ever happened in web interface - no single notification on mobiles (smartphones, tablets), just silence.
"I haven't heard from you before that you're having issues btw"

Sorry for that. I googled and find many questions about this, and no one solution.
Any way, excuse me for my speech. I never mind to make you feel bad. =)

[sampaiodiego]

@KayzerCode no problem, I'll need more information in order to help you, we definitely don't want to have a software that doesn't work as expected.

Can you please open a new issue describing (again) the issue you're facing? Saying what how you expect things to work and they're actually working (or not working). Please make sure to include in the issue:

• the Rocket.Chat version you're running
• if you're using a custom mobile app or our official apps from stores
• turn the logs verbose (under Admin > Logs) and attach any logs the server have emitted.

Please mention me on the new issue so I can have a look on it in a timely manner.

[Salmankhan033]

In IOS mobile notifications did not send
have I put the .pem files keys correctly?