diasend/camaps cloud sync

[bewest]

Creating a place to discuss this.
Related: https://github.com/burnedikt/diasend-nightscout-bridge

#7810 (comment)

This looks really promising. We've been considering refactoring these; the current architecture implicitly multiplies the dependency tree which has grown hard to manage. I think it would be better to have one project that wraps all cloud vendors as a subdirectory with shared dependencies, retry logic, and can inject everything needed for Nightscout integration as a sidecar or plugin once: Dexcom Share, Medtronic Carelink, Glooko/OP5, Diasend/CAMAPS, Abbott Libre View, Tandem tconnectsync, Tidepool, self (another NS api).

In particular, it's very useful to have telemetry to help track login attempts and the collect more information about the behavior and performance of third party cloud providers. The typescript reads nicely, and I plan to use axios across the board. How would you feel about participating in this effort, or even expanding diasend-nightscout-bridge to accommodate some of these ideas? It's a bigger effort, but makes things easier over time, including making it easier to introduce a new vendor like this in the future.

[sulkaharo]

One part of refactor worth mention is, the plugins need to be split to separate client and server plugins, so plugins with functionality on both sides don't pull in unnecessary dependencies into the client.

[sulkaharo]

Ping @burnedikt to you notice this. +1 to getting this merged into the mainline at some point.

[burnedikt]

First of all, thanks for considering merging the diasend bridge upstream into nightscout. I wasn't aware there already was another PR attempting this so thanks for mentioning it here.

In particular, it's very useful to have telemetry to help track login attempts and the collect more information about the behavior and performance of third party cloud providers. The typescript reads nicely, and I plan to use axios across the board. How would you feel about participating in this effort, or even expanding diasend-nightscout-bridge to accommodate some of these ideas? It's a bigger effort, but makes things easier over time, including making it easier to introduce a new vendor like this in the future.

I am very much open to help integrating the diasend-bridge within nightscout. I've also noticed that there's quite some similarities between other 3rd party cloud provider integrations that work in a similar manner (in fact the diasend bridge's plugin code was built on the mmconnect plugin code). So having some kind of shared code or at least plugin architecture / interfaces to implement would definitely help.

Can you share more about what idea's you already have in mind regarding the refactoring?

[burnedikt]

Regarding your comment on the increased memory requirements for CI:

The increased memory limit only applies to the testing stage, i.e. not for mere npm install commands as nightscout users would perform. So from my understanding, it should still run as expected on the mentioned ~512MB RAM machines.

But to be completely fair, I haven't looked into specifically why the test-ci script runs out of memory with the diasend-bridge installed, increasing the memory for testing just was the fastest way to get it working.

[sulkaharo]

We get a lot of accidental PRs from user who are trying to merge changes to their own forks. I just closed the other PR as I'm pretty much 100% sure it's one of those.

[burnedikt]

It's maybe worth noting that the dependencies of the diasend-nightscout bridge apparently break nightscout on nodejs version 10.x. See this issue.

My assumption was that the officially supported nodejs versions for nightscout are 12.x, 14.x and 16.x as indicated in the package.json but just wanted to raise awareness that popular 3rd party installers like the one from xDrip will lead to a broken state.

[bewest]

Are there any blockers for integrating into dev and mainline? Node 10 is not supported.

Regarding the future there are several implementations that would benefit from a single architecture and dependency management.

• diasend/camps cloud sync (here)
• minimed-connect-to-nightscout
• https://github.com/nightscout/share2nightscout-bridge
• several similar glooko2nightscout-bridge versions
• a python version of tconnectsync
• librelinkup for libre3 to Nightscout https://github.com/timoschlueter/nightscout-librelink-up
• sync/view another Nightscout (self/remote-mirror)

They should all reside in a single shared repo have their http library injected once, rather than each requiring their own http libraries. The plugin integration with Nightscout requires access to Nightscouts databus, and to setInterval. There are areas that deserve special consideration:

• sensitivity to retries - retrying too often can taint the credentials, make the account unreachable and potentially degrade performance of the app for other users. (Dexcom and Medtronic)
• telemetry - its useful to collect data as a community on the availability of services (Dexcom and Medtronic commonly experience outages)
• feedback - we have the outlines for a notification system that presents messages to authorized users. right now it's used for security notifications, but it would be nice to be able to tell the user that the plugin is disabled because the credentials aren't working or because the service is down. In addition perhaps there could be a way to modify the credentials by referring to a secret rather than having to restart the server.
• first sync vs nominal iteration - it's useful to get a list of gaps from Nightscout that potentially need to be filled. it's common to need this the first time as well as useful to avoid duplicating data when there are multiple sources of data.

They are all identical in the sense that the overall workflow looks something like:

  exchangeCredentialsForAuth credentials
  | authenticateSession
  | fetchData
  | storeMissingData

[bewest]

Nice work on this! Make sure to merge or rebase to Nightscout's dev branch to catch up and help make sure the tests run. Let me know if you need help. We may be closing on dev branch soon and I'm starting to think about and work on branches not necessarily aimed at this release. In particular, since there are 7 plugins that basically use this same architecture/style, I think it would be good to think about generalizing it to accommodate similar cloud providers so that the http lib can be managed/injected once. Maybe something like cloud-to-nightscout-bridges where adding a new vendor might mean adding a few files to a new subdirectory, and some tests.

[burnedikt]

Make sure to merge or rebase to Nightscout's dev branch to catch up and help make sure the tests run.

I've put up a rebased version of this PR at #7872. Since a bunch of forks are already running of my fork's master branch, I didn't want to mess up their update paths (yet), hence why there's a rebased dev branch now

In particular, since there are 7 plugins that basically use this same architecture/style, I think it would be good to think about generalizing it to accommodate similar cloud providers so that the http lib can be managed/injected once. Maybe something like cloud-to-nightscout-bridges where adding a new vendor might mean adding a few files to a new subdirectory, and some tests.

I like the idea generally and it would have helped to kickstart my project if there was some base code available upon which to build the functionality. That being said, not sure if all the "bridge" authors have the time to rework their bridges to a shared structure so in a way, it's also a strength to not force them into one and the same schema. Also, although there is a shared core, each bridge will work a bit differently, e.g. on diasend, not everything is exposed via the API and some things need to be scraped off their website which leads to additional dependencies.

One thing that would've definitely helped me personally would've been an existing (typed) client for nightscout's (internal) API (in the end I created my own), there was a bit of trial and error involved to figure out how to retrieve and store entries / treatments.

[bewest]

Feel free to close this PR in favor of the new one.
I've been prototyping a single point of entry for plugins like this.
So far it's working with self-Nightscout and Dexcom Share. https://github.com/bewest/nightscout-connect/tree/wip/bewest/axios-har-tracker

This architecture allows us to maintain and add features without requiring additions to Nightscout core code (#7983). I'd like to get Medtronic and Diasend implemented for inclusion into Nightscout.

[burnedikt]

Feel free to close this PR in favor of the new one.

Since I'm not the author of this PR, I can not close it. But this should definitely be closed. Not only because #7872 supersedes this but also because even #7872 is obsolete by now due to the fact that diasend has been replaced / migrated to glooko which is an entirely different service and therefore the sync doesn't work anymore.