2.6.5 not sending pushes, went back to 2.4.2, pushes are ok

[dongalor]

Issue Description

Updated to 2.6.5, pushes not working, i'm having problems with pushes:

Went back to 2.4.2:

npm install [email protected] --save

everything is working well

I've seen #4256, and I've seen this comment, which might be related to my problems:

hey @flovilmart are you saying this will be updated in 2.6.4?
@jeffreyjackson yes!

Expected Results

I'd like to have most recent version, cause I feel like lack of performance issues updates in last versions.

Environment Setup

• Server
  • parse-server version: 2.6.5
  • Operating System: ubuntu16.04-x86_64

Logs/Trace

0|parse-wr | verbose: _PushStatus n59yItvDR0: sending push to 1 installations
0|parse-wr | verbose: Sending push to 1
0|parse-wr | verbose: _PushStatus n59yItvDR0: sent push! 0 success, 1 failures
0|parse-wr | verbose: _PushStatus n59yItvDR0: needs cleanup devicesToRemove=[]

[flovilmart]

Can you provide your push configuration please?

[brianyyz]

Over on SO there is lots of chatter about APNS arbitrarily throttling delivery of data pushes and a new process DuetActivitySchedulerDaemon that can intercept a push notification that has been successfully delivered to your device and prevent it from being delivered to your appDelegate based on things like frequency of the silent notifications and battery state.

I spent two days trying to figure out why my server's silent push notifications would mysteriously stop being received by my test devices until I restarted them. I haven't 100% verified that this is the cause but if you are using silent notifications to trigger activity you need to handle the use case that they could be throttled by APNS or blocked at the device which means your appDelegate may never see them.

[flovilmart]

What's bugging me is that the push adapter is not reporting/logging the error when sending the push. As you're running with VERBOSE=1, and the failure was related to APNS not being able to transmit, it should be logged. There isn't any other logs related to the parse-server-push-adapter? Can you try the same running the sever locally and targeting the push with your test device? (also, provide your push configuration please).

[dongalor]

I'm running with VERBOSE=1

My push configuration is:

push: {
ios: {
pfx: "/path/to/certs/file.p12",
passphrase: "",
bundleId: "com.company.AppName",
production: true
}
},

I'll try running it locally, and report my results. Thank you in advance!

[brianyyz]

Small point not directly related bundleId is deprecated and replaced with topic.

[dongalor]

brianyyz, thanks for reply, yeah, I have had deprecation warning, but I already tried using topic instead of bundleId, same story...

[flovilmart]

@dongalor do you have some more logs for us?

[brianyyz]

@dongalor would you be able to share the code that is creating the push payload as well ?

[dongalor]

Here's my code, which I use to test my pushes:
https://gist.github.com/dongalor/96d13ee1b0344d222f4a4caf899bd8ed

this exact code is working well on [email protected]

[flovilmart]

If you run the query on itself does it work? ie: does it return installations?

[paulovitin]

I have the same problem here @dongalor. What is the database you are using?

My problem is because Im using Postgres. The 2.6.5 version support "expiration_interval" now. When we update the version, the column in _PushStatus is missing then, the push notification stops. I added the column manually and works now.

When schema change, we need a better option to solve problems like this.

The log trace:

2017-11-10T18:53:32.413657765Z �[31merror�[39m: _PushStatus undefined: error while sending push error: column "expiration_interval" of relation "_PushStatus" does not exist
2017-11-10T18:53:32.413692926Z     at Connection.parseE (/parse-server/node_modules/pg/lib/connection.js:546:11)
2017-11-10T18:53:32.413700050Z     at Connection.parseMessage (/parse-server/node_modules/pg/lib/connection.js:371:19)
2017-11-10T18:53:32.413705701Z     at TLSSocket.<anonymous> (/parse-server/node_modules/pg/lib/connection.js:114:22)
2017-11-10T18:53:32.413711165Z     at emitOne (events.js:96:13)
2017-11-10T18:53:32.413715592Z     at TLSSocket.emit (events.js:188:7)
2017-11-10T18:53:32.413720755Z     at readableAddChunk (_stream_readable.js:176:18)
2017-11-10T18:53:32.413725436Z     at TLSSocket.Readable.push (_stream_readable.js:134:10)
2017-11-10T18:53:32.413733407Z     at TLSWrap.onread (net.js:547:20)

[montymxb]

@paulovitin If that's the case using 2.6.3 would work, as the expiration_interval was introduced in 2.6.5 (2.6.4 has a CLI bug in it but you could use that too). I would think that the schema would have been auto-updated on start though.

[paulovitin]

@montymxb we have a docker auto-redeploy, and the auto-updated for schema doesnt happened.

[montymxb]

I'll take a look and see if I can replicate this upgrade issue regarding expiration_interval.

[montymxb]

@paulovitin I can confirm that there is a Schema upgrade issue, but only in postgres. Mongodb looks to handle it properly. I'm going to open up a separate issue with further details on that specifically, including reproducability.

@dongalor if you're using postgres as your database it's possible that you're experiencing the same issue post-upgrade.

::EDIT:: Postgres upgrade issue opened under #4338

[dongalor]

@flovilmart > If you run the query on itself does it work? ie: does it return installations?
yes

@montymxb, @paulovitin
no, I'm using mongodb :( that's sad

at this moment i'm still using 2.4.2, i wish I could come back to 2.6.5 :)

[flovilmart]

Ok, I may have an idea, to be clear, your users hold the deviceTokens as well as the installation right?

[dongalor]

installationId? yes!

[flovilmart]

@dongalor the gist you provided don’t mention installationId at all, just matching on device token.

Also, just a note on architecture, usually a user can have many installation, so I would recommend you set the user on the installation objects, instead of storing the installationId’s on the users.

[dongalor]

@flovilmart sorry, now I understand the question, do my users table hold installationId? No, but I have system table Installation, which is being filled automatically.

So, my users table doesn't hold installationId, but my users table hold deviceToken field.

[flovilmart]

Ok so the gist is correct, I'll have a look.

[flovilmart]

So far, I can pin point the issue in the push adapter, as everything in parse-server is going smooth. It is an issue with all your push’s, or just some?

[dongalor]

@flovilmart actually i've provided my gist (https://gist.github.com/dongalor/96d13ee1b0344d222f4a4caf899bd8ed), all other pushes use the same Push class, so none of them work (

also when I downgrade to 2.4.2, everything works fine..

[flovilmart]

Did you try sending a push through the dashboard?

[dongalor]

@flovilmart yeah, same result :( is there anything i can show you in my configuration to check?

[flovilmart]

Ok, the issue seem to lie in the push adapter itself, I need more time to investigate the issue.

[hopye]

if i go back to 2.4.2 , do i have to downgrade parse-dashboard as well to work well sending pushes throught it ?

npm install [email protected] --save

[dongalor]

@hopye actually this is the only command I used to downgrade parse without downgrading dashboard, I guess there's no need in additional actions, also I think there are dependencies between dashboard and parse-server

[promisenxu]

We are also having problems with push for 2.6.5. There are constant error messages from parse-push-adapter and push are sometime delayed rather significantly (a few hours). However, our users can still receive push notifications, both iOS and Android devices.

The errosr we get look like:

node-pre-gyp ERR! parse-server-push-adapter APNS VError: apn write failed: Not receiving Ping response after 150000 ms
node-pre-gyp ERR! parse-server-push-adapter APNS     at Stream.stream.on.err (/app/node_modules/parse-server/node_modules/apn/lib/client.js:105:21)
node-pre-gyp ERR! parse-server-push-adapter APNS     at emitOne (events.js:96:13)
node-pre-gyp ERR! parse-server-push-adapter APNS     at Stream.emit (events.js:188:7)
node-pre-gyp ERR! parse-server-push-adapter APNS     at _connection._streamIds.forEach (/app/node_modules/parse-server/node_modules/apn/lib/protocol/endpoint.js:155:16)
node-pre-gyp ERR! parse-server-push-adapter APNS     at Array.forEach (native)
node-pre-gyp ERR! parse-server-push-adapter APNS     at EventEmitter.close [as _close] (/app/node_modules/parse-server/node_modules/apn/lib/protocol/endpoint.js:139:33)
node-pre-gyp ERR! parse-server-push-adapter APNS     at emitOne (events.js:101:20)
node-pre-gyp ERR! parse-server-push-adapter APNS     at TLSSocket.emit (events.js:188:7)
node-pre-gyp ERR! parse-server-push-adapter APNS     at _handle.close (net.js:497:12)
node-pre-gyp ERR! parse-server-push-adapter APNS     at TCP.done [as _onclose] (_tls_wrap.js:332:7)

and

node-pre-gyp ERR! parse-server-push-adapter APNS  APNS error transmitting to device %s with error %s the_device_token { VError: apn write failed: Not receiving Ping response after 150000 ms
node-pre-gyp ERR! parse-server-push-adapter APNS     at Stream.stream.on.err (/app/node_modules/parse-server/node_modules/apn/lib/client.js:105:21)
node-pre-gyp ERR! parse-server-push-adapter APNS     at emitOne (events.js:96:13)
node-pre-gyp ERR! parse-server-push-adapter APNS     at Stream.emit (events.js:188:7)
node-pre-gyp ERR! parse-server-push-adapter APNS     at _connection._streamIds.forEach (/app/node_modules/parse-server/node_modules/apn/lib/protocol/endpoint.js:155:16)
node-pre-gyp ERR! parse-server-push-adapter APNS     at Array.forEach (native)
node-pre-gyp ERR! parse-server-push-adapter APNS     at EventEmitter.close [as _close] (/app/node_modules/parse-server/node_modules/apn/lib/protocol/endpoint.js:139:33)
node-pre-gyp ERR! parse-server-push-adapter APNS     at emitOne (events.js:101:20)
node-pre-gyp ERR! parse-server-push-adapter APNS     at TLSSocket.emit (events.js:188:7)
node-pre-gyp ERR! parse-server-push-adapter APNS     at _handle.close (net.js:497:12)
node-pre-gyp ERR! parse-server-push-adapter APNS     at TCP.done [as _onclose] (_tls_wrap.js:332:7)

[dongalor]

@flovimart Damn it🥇, Bingo! Now I got it, we don't have appIdentifier in our Installation table, so, with parse 2.4.2 (which is sending pushes right now) we don't need apIdentifier, I guess, right? :)

Is it possible to assign apIdentifier manually without IOS SDK?

[mman]

@dongalor Sure you can just fire up mongo cli and modify _Installation documents in your database to manually set appIdentifier.

I'm deliberately not copy pasting any command to do it as this will be a massive (harmless) update, so I'd like you to understand what are you doing...

[dongalor]

@mman First of all, thank you so much for the support in solving my problem!

I have a development server, where I can do any dangerous stuff :)

So, I've made a small experiment, I've entered my appId from my parse's index.js file to every installation record and tried doing the same test push sending as here, what I've got in logs is:

verbose: _PushStatus UgmAPHkeyG: sending push to installations with 1 batches verbose: Sending push to 1 verbose: _PushStatus UgmAPHkeyG: sent push! 0 success, 1 failures verbose: _PushStatus UgmAPHkeyG: needs cleanup devicesToRemove=[]

By the way, am I getting right this value, which is able to be in appIdentifier field in Installation table?

[mman]

@dongalor Please try once again with VERBOSE_PARSE_SERVER_PUSH_ADAPTER=1 environment variable set, you should get more detailed diagnostics in the log.

[dongalor]

@mman so, I run my parse 3.1.1 instance this way, without pm2 for test purposes:

VERBOSE=1 VERBOSE_PARSE_SERVER_PUSH_ADAPTER=1 node index.js

and I get this in my log:

verbose: RESPONSE from [POST] /parse/functions/testPush: {
  "response": {
    "result": true
  }
} result=true
verbose: REQUEST for [POST] /parse/push: {
  "where": {
    "deviceToken": {
      "$select": {
        "key": "deviceToken",
        "query": {
          "where": {
            "objectId": "3ePnw29d9C"
          },
          "className": "_User"
        }
      }
    }
  },
  "data": {
    "data": {
      "subscriberId": "xku6EHzzwD",
      "subscriberName": "Test Subscriber",
      "code": "3"
    },
    "alert": "Free hotdogs at the Parse concession stand!"
  }
} method=POST, url=/parse/push, user-agent=node-XMLHttpRequest, Parse/js2.1.0 (NodeJS 11.2.0), accept=*/*, content-type=text/plain, host=localhost:1339, content-length=404, connection=close, key=deviceToken, objectId=3ePnw29d9C, className=_User, subscriberId=xku6EHzzwD, subscriberName=Test Subscriber, code=3, alert=Free hotdogs at the Parse concession stand!
verbose: RESPONSE from [POST] /parse/push: {
  "headers": {
    "X-Parse-Push-Status-Id": "GsEETYJSAd"
  },
  "response": {
    "result": true
  }
} X-Parse-Push-Status-Id=GsEETYJSAd, result=true
verbose: _PushStatus GsEETYJSAd: sending push to installations with 1 batches
verbose: Sending push to 1
verbose: _PushStatus GsEETYJSAd: sent push! 0 success, 1 failures
verbose: _PushStatus GsEETYJSAd: needs cleanup devicesToRemove=[]

[antonfire]

Add a column in your installation class with the name appIdentifier and the type string. Then populate all your existing installation objects with your app id eg com.blahblah. Then try sending again

[antonfire]

Will need to ensure new installation objects are populated with this field on push registration client side

[flovilmart]

One could as well make a fix in the adapter.

[antonfire]

I would suggest removing the appIdentifier as a requirement in the installation class as this is specified in the push adaptor settings in index.js. I can have a look at a pull request for this if that would help?

[flovilmart]

That certainly would help

[antonfire]

Ok I’ll get on it

[brianyyz]

@antonfire are you thinking that this would be an either / or approach (look for appIdentifier first and if absent use push adapter settings from index.js) or just ignore the appIdentifier in Installation and rely solely on config from index.js?

[flovilmart]

That’s not that simple, if the installation does not specify an app identifier and multiple are configured, the adapter need to retry. In any case you should also fix your clients so they set a proper appIdentifier.

Not setting an appIdentifier is not conform to the specification. And for all that I care, I don’t have to merge or provide a fix. All the official SDKs set this value.

[brianyyz]

@flovilmart that is where I was headed. This starts to feel like it would be breaking away from the way that the sdks behave.

[flovilmart]

Initially, the fact that we didn’t take into account the appIdentifier was a bug.

[antonfire]

Not everyone uses the sdk to register push, a waterfall approach would be best to prevent breaking changes for users that haven’t upgraded or installed since the breaking change

[brianyyz]

@antonfire how do we know how many users of parse are not using an sdk? I think this would be an important metric to evaluate whether a breaking change is justified. At a minimum it would inform how to approach the change.

[flovilmart]

@antonfire I am not sure where you’re getting at. The spec in the rest documentation has always been to pass an appIdentifier. That the server before 2.6 was permissive was a bug on the server.

If you feel any of this is wrong, this is an opensource project, so you know what to do.

[dongalor]

@antonfire Oh My God! Now, when I'm using topic/bundleId value in my test user Installation record, I have an error! I think we are very close to beat this problem of my. Thank you so much in advance, guys!

Here's what log says:

verbose: _PushStatus 7GzcpE2zDK: sending push to installations with 1 batches
verbose: Sending push to 1
ERR! parse-server-push-adapter APNS APNS error transmitting to device 5083c519e51624bd6e1ff3dca9c1dac8afce4433190319d71e2015876e0bb922 with status 400 and reason BadDeviceToken
verbose: _PushStatus 7GzcpE2zDK: sent push! 0 success, 1 failures
verbose: _PushStatus 7GzcpE2zDK: needs cleanup devicesToRemove=[5083c519e51624bd6e1ff3dca9c1dac8afce4433190319d71e2015876e0bb922]

[flovilmart]

That’s likely because the token is invalid

[mman]

@dongalor Error 400 happens when you try to send push from parse server using a production certificate or token to a debug build of your app, IOW the push token of your app is a one that is valid for sandbox. servers.

[antonfire]

@dongalor Set up like this so works on production and development:

push: {
ios: [
{
pfx: 'PushDevCert.p12',
topic: 'com.xxxxxx.xxxxx',
production: false
},
{
pfx: 'PushDistCert.p12',
topic: 'com.xxxxxx.xxxxx',
production: true
}]

}

[antonfire]

@flovilmart @brianyyz I was just thinking of a fallback to the bundle id where appIdentifier is undefined

[antonfire]

@dongalor Ensure your certificates are current and valid. Ensure you are sending to current valid device tokens.

[brianyyz]

@dongalor (going all the way back to the beginning) In your AppDelegate are you saving the device token that returns in

func application(application: UIApplication, didRegisterForRemoteNotificationsWithDeviceToken deviceToken: NSData) {
  // Store the deviceToken in the current Installation and save it to Parse
  let installation = PFInstallation.currentInstallation()
  installation.setDeviceTokenFromData(deviceToken)
  installation.saveInBackground()
}

before sending a push to the device? The last updated time stamp on the Installation should get updated if you are. Just want to make sure we’re covering all the bases.

[dongalor]

@flovilmart @mman @antonfire I @brianyyz My IOS developer registered a new user on 3.1.1 parse and it's working! :) So, I guess the problem now was with deviceToken.

Thank you so much for the support, now I can move all backend to 3.1.1!

P.S. We can remove wontfix label now, I guess!

[flovilmart]

Glad to hear. Also the issue is on the push adapter, so it won’t be fixed in this version of parse-server

[brianyyz]

Excellent news!

[dongalor]

@flovilmart got it, right :)
@brianyyz 👍