Feat: Add support for XAPPLEPUSHSERVICE

[titanism]

Ref: https://docker-mailserver.github.io/docker-mailserver/latest/examples/use-cases/ios-mail-push-support/

[titanism]

XAPPLEPUSHSERVICE aps-version 2 aps-account-id 0715A26B-CA09-4730-A419-793000CA982E aps-device-token 2918390218931890821908309283098109381029309829018310983092892829 aps-subtopic com.apple.mobilemail mailboxes (INBOX Notes)

• aps-version - always set to "2"
• aps-account-id - a unique id the iOS device has associated with this account
• aps-device-token - the APS device token
• aps-subtopic - always set to "com.apple.mobilemail"
• mailboxes - list of mailboxes to send notifications for

Ref: https://github.com/freswa/dovecot-xaps-plugin/blob/197d68e9d0f4f802aff06f90ffd2c1957394a380/xaps-imap-plugin.c#L40-L63

So we need to parse and validate these parameters, and then push the values to a daemon/runner/API that will then record the mapping between the account and the client. We could copy the validation from https://github.com/freswa/dovecot-xaps-plugin/blob/197d68e9d0f4f802aff06f90ffd2c1957394a380/xaps-imap-plugin.c#L64-L107. (e.g. aps-version is always "2" and all the params must have a length > 0.

[louis-lau]

I agree this would be very nice to have! May not be that easy, as there's no publically available official spec.

[titanism]

It looks like the only requirement is that you need to be registered iOS developer, so you can download macOS server.

Please note that it is not possible to use this project without legally owning a copy of OS X Server. You can purchase OS X Server on the Mac App Store or download it for free if you are a registered Mac or iOS developer.

As per https://github.com/freswa/dovecot-xaps-daemon?tab=readme-ov-file#what-is-this.

[titanism]

@louis-lau I'm digging in, there are two repos we can get inspiration from, both https://github.com/freswa/dovecot-xaps-plugin/tree/master and https://github.com/freswa/dovecot-xaps-daemon.

[louis-lau]

Nice! The fact that people have reverse engineered this before should make things easier. Looking into this was on my list somewhere, but there's about 999 things above it haha. Good luck!

[JDENredden]

I can’t contribute anything technically, but I would love to see this implemented. Major quality of life win for iOS users.

[titanism]

I can’t contribute anything technically, but I would love to see this implemented. Major quality of life win for iOS users.

We'll see what we can do, we have a lot going on at https://forwardemail.net - but this is highly requested of course for our IMAP users 😄

[titanism]

There's apparently a way to notify too for non-INBOX as per comment here: https://github.com/Rjevski/apache-james-xapsd-registration-extension/blob/f694b993c08966485e9ad185c31fe038d42c4962/README.md?plain=1#L55C1-L55C115

[titanism]

A few additional notes:

• Daemon/job should renew existing certs ~30 days before they expire (https://github.com/freswa/dovecot-xaps-daemon/blob/abce2f14cf1b5afa56329ebb4d923c9c2aebdfe3/pkg/apple_xserver_certs/request.go#L114-L140)

When creating a certificate, the request body to Apple looks like https://github.com/freswa/dovecot-xaps-daemon/blob/abce2f14cf1b5afa56329ebb4d923c9c2aebdfe3/pkg/apple_xserver_certs/request.go#L118-L139:

{
  PushCertCertificateChainPushCertCertificateChain: Buffer, // signing certificate chain
  PushCertRequestPlist: Buffer, // push certificate request plist
  PushCertSignature: Buffer, // new push certificate signature using push certificate request plist and signing key
  PushCertSignedRequest: Buffer
}

The signing key is derived from x509 parsed PKCS1PrivateKey

Cert renewals are sent as HTTP POST request to "https://identity.apple.com/pushcert/caservice/renew and cert creations (new ones) are sent to https://identity.apple.com/pushcert/caservice/new.

Headers for this request are as follows:

• Content-Type set to text/x-xml-plist
• User-Agent set to Servermgrd%20Plugin/6.0 CFNetwork/811.11 Darwin/16.7.0 (x86_64)
• Accept set to */*
• Accept-Language set to en-us

[titanism]

Looks like vendor certs are used, which is derived from this repo https://github.com/scintill/macos-server-apns-certs?tab=readme-ov-file#download-and-configure.

This repo has a great guide for creating a certificate.

[titanism]

Basically the way that this whole thing works with adding XAPPLEPUSHSERVICE is that we basically spoof being a macOS Server, and use an Apple ID (e.g. your Apple Developer Portal ID) to generate a cert, and then send push notifications.

[titanism]

The vendor certificates we need to include (and randomly pick one it seems) are here https://github.com/freswa/dovecot-xaps-daemon/blob/1e589be2e2f54fc94189b03e3db274f86bb7357c/pkg/apple_xserver_certs/request.go#L21-L116.

You can see the signing cert is randomly selected via mrand.Seed(time.Now().UnixNano()) and signingCerts := vendorCerts[mrand.Intn(10)] which gets a random integer between 1 and 10 because there are 10 total vendor certificates to choose from.

[titanism]

The value for PushCertRequestPlist is a plist created file (encoded as a Buffer it seems). They use a Go package at https://github.com/freswa/go-plist but we could use the package plist at https://www.npmjs.com/package/plist probably which is available on GitHub at https://github.com/TooTallNate/plist.js.

var plist = require('plist');

var json = [
  "metadata",
  {
    "bundle-identifier": "com.company.app",
    "bundle-version": "0.1.1",
    "kind": "software",
    "title": "AppName"
  }
];

console.log(plist.build(json));

// <?xml version="1.0" encoding="UTF-8"?>
// <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
// <plist version="1.0">
//   <key>metadata</key>
//   <dict>
//     <key>bundle-identifier</key>
//     <string>com.company.app</string>
//     <key>bundle-version</key>
//     <string>0.1.1</string>
//     <key>kind</key>
//     <string>software</string>
//     <key>title</key>
//     <string>AppName</string>
//   </dict>
// </plist>

And we need to use certs, username (Apple ID), and passwordhash` (Apple ID Password Hash) to generate it.

https://github.com/freswa/dovecot-xaps-daemon/blob/1e589be2e2f54fc94189b03e3db274f86bb7357c/pkg/apple_xserver_certs/request.go#L174-L207

Looks like they have max line length of 64 and indentation using two spaces, and no escaping.

So I'm assuming there are two dict, something like this:

var plist = require('plist');

var json = [
  "Header",
  {
    ClientApplicationCredential: "1",
    ClientApplicationName:       "XServer",
    ClientIPAddress:             "1",
    ClientOSName:                "MAC OSX",
    ClientOSVersion:             "2.1",
    LanguagePreference:          "1",
    TransactionId:               "1",
    Version:                     "1",
  },
  "Request",
  {
    ProfileType:   "Production",
    RequesterType: "XServer",
    "User",
    {
      AccountName:  username,
      PasswordHash: passwordhash
    }
    "CertRequestList",
    ... values here ....
  }
];

For values here it's derived from the code here https://github.com/freswa/dovecot-xaps-daemon/blob/1e589be2e2f54fc94189b03e3db274f86bb7357c/pkg/apple_xserver_certs/request.go#L209-L255.

[titanism]

Now we just need to figure out how to make it so we take action based off provided values from the command, e.g. (XAPPLEPUSHSERVICE aps-version 2 aps-account-id 0715A26B-CA09-4730-A419-793000CA982E aps-device-token 2918390218931890821908309283098109381029309829018310983092892829 aps-subtopic com.apple.mobilemail mailboxes (INBOX Notes)).

[titanism]

It looks like Dovecot takes the XAPPLEPUSHSERVICE and registers a struct to listen/register requests via IMAP.

https://github.com/freswa/dovecot-xaps-daemon/blob/abce2f14cf1b5afa56329ebb4d923c9c2aebdfe3/internal/socket.go#L18-L30

This is done here https://github.com/freswa/dovecot-xaps-daemon/blob/abce2f14cf1b5afa56329ebb4d923c9c2aebdfe3/internal/socket.go#L68-L104

So that's how they register it internally. Now how do they send the actual notifications via Apple Push Notifications is what we need to figure out next.

[titanism]

Searching for DeviceToken and AccountId in the repo leads us to how they send the Apple Push Notification upon a new message being appended to INBOX for example:

https://github.com/freswa/dovecot-xaps-daemon/blob/abce2f14cf1b5afa56329ebb4d923c9c2aebdfe3/internal/apns.go#L142-L185

They use the Go package apns at https://pkg.go.dev/github.com/mduvall/go-apns.

Now in Node.js land, we can do it with apn at https://github.com/node-apn/node-apn:

So basically whenever new mail is received, we'd simply call that, which will then trigger INBOX to fetch new messages.

[titanism]

@louis-lau @andris9 can you please review and merge this PR, and release with #705 merged as well under #689 for WildDuck v1.43.4 to npm so that we can test and then follow-up with how to implement here in production?

PR to add XAPPLEPUSHSERVICE support is at: #712

Many thanks 🙏

[titanism]

See forwardemail/forwardemail.net@6398425 for example integration. We are testing it out now.

[titanism]

Got it working!!!! 🎉

This PR is ready for merge, please review and merge, thank you! ✅

If you'd like to see the working implementation, see these files:

• Our handler for XAPPLEPUSHSERVICE @ https://github.com/forwardemail/forwardemail.net/blob/master/helpers/imap/on-xapplepushservice.js which simply stores to database the user's APS data provided via this command
• Sending of Apple Push Notification (APN) @ https://github.com/forwardemail/forwardemail.net/blob/master/helpers/send-apn.js

Feel free to try it out on https://forwardemail.net with your iOS device (setup an IMAP account)

PUSH now supported on iOS!!!! ⚡ (our alternative since IDLE not supported on iOS Mail)

[titanism]

We're debugging a few issues, and it's not actually working (yet). After fixing an issue with BadExpirationDate (since you need to use seconds, rounded via Math.floor, not milliseconds), we receive TopicDisallowed. Will keep you posted.

[titanism]

Additionally it appears we can't just set aps_topic to com.apple.mobilemail. Instead it's something like this com.apple.mail.XServer.xxxxxxxxxxxxxxx which is extracted from the common name of a certificate from OSX Server.

Looks like we do indeed need to spoof mac OSX Server.

[titanism]

Almost done here, figured out 90% of it just parsing the body and adding caching. That was a headache indeed.

[titanism]

Folks, it's finally WORKING! I will clean up my commits (and push once ready) and share more here once done and deployed.

This was very, very painful to get working, and we still need to test it more thoroughly, but we were able to generate all the certs, verify them, and send a successful APN payload to the device token and account id.

[titanism]

Okay so despite it working and the APN being successfully delivered – it doesn't appear that APPL is actually delivering it to the device/account pair. I've reached out to folks at APPL, Linux (maintainers of that Dovecot project), and the team at Fastmail. Hoping we can figure out if this is possible still or not, and if not, what are the alternatives. It seems that Yahoo and Fastmail can only do this approach because of a custom licensing that permits them for com.apple.mobilemail push notifications.

Ref: st3fan/dovecot-xaps-daemon#46 (comment)

[JDENredden]

Mailbox.org is another one that has managed to make push email work on iOS. They’re another one to reach out to.  Zoho Mail rely on ActiveSync to do push on iOS.

[titanism]

Thank you @JDENredden, we've pinged Mailbox.org folks too.

[titanism]

We figured out what was wrong, and are fixing it now.

[titanism]

Special thanks to @freswa per freswa/dovecot-xaps-daemon#39 (comment) (PR at #719).

We finally got to the bottom of this and will be testing in production shortly, and will follow up here once done.

[titanism]

@andris9 Here is our implementation:

• onXAPPLEPUSHSERVICE handler for IMAP @ https://github.com/forwardemail/forwardemail.net/blob/master/helpers/imap/on-xapplepushservice.js (note this returns a topic as the callback data, which is parsed from certs
• getApnCerts function @ https://github.com/forwardemail/forwardemail.net/blob/master/helpers/get-apn-certs.js (this is the core of the write and involves complicated X509 cert/key generation combined with a random selection of helpers/vendorcerts key and chain
• modification to onAppend @ https://github.com/forwardemail/forwardemail.net/blob/e3942a140b9b27dafc041e9d36284de2ced73641/helpers/imap/on-append.js#L79-L82 (so push notifications go through for the specific mailbox)
• sendApn function which leverages Redis to get the keys/topic, and accepts a mailbox/alias ID and path (e.g. INBOX is the default appended but we leverage path from APPEND command) @ https://github.com/forwardemail/forwardemail.net/blob/master/helpers/send-apn.js (this has all the core logic for sending a push notification to Apple)

There are still some future TODO's – see the TODO's in the codebase at those links, e.g. cert renewals; right now they expire after 360d in redis cache).

We are still testing stuff, so await our final confirmation before this is smooth implementation.

P.S. @andris9 awesome work with mobileconfig package, it was super useful here https://github.com/forwardemail/forwardemail.net/blob/master/app/controllers/web/mobile-config.js and here's a screenshot of it in action combined with qrcode:

[titanism]

Works great!

Had to make a few more updates to the above linked files in #711 (comment) and now it's good to go!

[titanism]

Feel free to try it out, use coupon code GITHUB for 100% off discount at https://forwardemail.net

Once you sign up and add your domain, simply follow these instructions:

https://forwardemail.net/faq#do-you-support-receiving-email-with-imap