Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding support for Meshtastic (Bircom) #301

Closed
ketan opened this issue Sep 1, 2024 · 47 comments · Fixed by #303
Closed

Adding support for Meshtastic (Bircom) #301

ketan opened this issue Sep 1, 2024 · 47 comments · Fixed by #303

Comments

@ketan
Copy link
Contributor

ketan commented Sep 1, 2024

Hello there -

As you may or may not be aware, sat comm devices (like Inreach and Spot) are illegal to operate in India without an operating license from the government. I'm part of volunteer team that is building a tracking network for pilots in Bir/Biling in India based on a mesh technology called meshtastic mesh network. We already have a tracker in place at https://tracker.bircom.in.

Since flyxc is a great app that's used and trusted by many pilots - I'm wondering if you'd be open to integrating with our tracker. I have some questions in terms of how you integrate with other trackers right now, so I can contribute back to this repository.

For context - our mesh network captures the following details:

  • unique device identifier
  • short name (initials) to show on the map
  • long name - for better identification
  • gps co-ordinates, along with timestamps
  • baro altitude, along with timestamps
  • (optional) some device details - hardware and firmware versions

From a design point of view, we can push these packets to you as and when we see the packets (we have a mqtt backend), or we can offer any APIs as you see fit for you to periodically pull this information.

Let me know, thanks!

@vicb
Copy link
Owner

vicb commented Sep 1, 2024 via email

@vicb
Copy link
Owner

vicb commented Sep 1, 2024

Ketan,

First of all thanks for contacting flyXC.

I'm all for integrating your platform in flyXC.

A few notes on how flyXC works:

  1. Pilots have to create an account on flyXC via the live tracking > settings menu. There they can connect to any of the platforms that are supported by flyXC. If they connect to multiple platforms, their track will be the union of those platforms. It's usual that pilots connect a satellite tracker for the reliability and an app (ie XCtrack) to get a more detailed trace when they have phone coverage.

  2. flyXC supports pulling data from those platforms (most of them) and also supports push from the platforms (OGN, Zoleo)

For 1, there needs to be some kind of id/SSO to connect your platform to flyXC. The pilot name is defined on the flyXC account so that every platform uses the same name.

For 2, pushing to flyXC sounds better to me so that flyXC does not keep pulling your server while nobody is flying. flyXC needs to receive lat/lon/alt and some kind of id. You can optional send the speed, heading and messages (landed ok, need help, ...)

I'm based on France, Paris TZ. We can discuss more tomorrow. Please send any question you have.

@ketan
Copy link
Contributor Author

ketan commented Sep 1, 2024

For the push based system - do you have some sample code that I can look at? I suppose that for the identifier bit, you'd probably expect that a tile is added to https://flyxc.app/devices that allows collecting some identifier/token corresponding to the user on my tracking system?

@cyberorg
Copy link

cyberorg commented Sep 2, 2024

Here is our current way of getting these devices to flyxc via skylines. https://bircom.in

We would like to avoid skylines step, we could use the same way to push to flyxc as we do to skylines if that works.

{"payload":"leolive=1&user=1B8D47F&lat=32.036799099999996&lon=76.70709049999999&sid=33412863&password=123&sog=0&alt=1356&tm=1725239499","_msgid":"21c1c55a9ea8c718","method":"POST","url":"https://skylines.aero/track.php","headers":{"Content-Type":"application/x-www-form-urlencoded","Content-Length":122}}

Please note that the devices don’t have internet connection, the data is sent by base stations that are connected to internet. We cannot collect passwords so the push has to be key based or dummy password like skylines. Livetrack24 doesn’t work as it needsa proper password.

This is what our raw data looks like:
https://tracker.bircom.in/api/nodes

@vicb
Copy link
Owner

vicb commented Sep 2, 2024

Yes, there would be a new card on the devices page.

I can create a POST endpoint on flyxc.app (i.e. flyxc.app/api/bircom/push) for your platform to push the data. It would be similar to what is available for Zoleo

// Hook called by zoleo.
router.post('/push', auth, async (req: Request, res: Response) => {
try {
const json = JSON.stringify(parseMessage(req.body));
if (json != null) {
const pipeline = redis.pipeline();
pushListCap(pipeline, Keys.zoleoMsgQueue, [json], ZOLEO_MAX_MSG, ZOLEO_MAX_MSG_SIZE);
await pipeline.exec();
}
res.sendStatus(200);
} catch (e) {
console.error(e);
res.sendStatus(500);
}
});
return router;

I can update the code for those 2 steps - it should not take me much longer than 1h.

What is left to be defined:

  • what token/id should be entered in flyXC (or created by flyXC and entered on your platform)
  • what mechanism flyXC uses to authenticate your push requests. It can be basic Auth (same as zooleo), a secret passed in an HTTP header or in the JSON payload, ... I guess using https is possible on your end?

@cyberorg
Copy link

cyberorg commented Sep 2, 2024

Each of the device has unique ID like "!674c3f41" which we can ask user to register their device with flyxc.

This is the raw packet we get from the device.

{"payload":[{"lat":32.0367098,"lon":76.7070858,"alt":1359,"tm":1725257499,"satsInView":5,"groundSpeed":1,"groundTrack":0,"pDOP":394},{"from":"!674c3f41","fromShortName":"Pgid","fromLongName":"Paraguide In/5652D866","fromHardware":"T_ECHO","to":"!ffffffff","toShortName":"mesh","toLongName":"Mesh","toHardware":"MESH","channelId":"LongFast","gatewayId":"!da52924c"}],"_msgid":"c126d40bd21bbad7"}

@vicb
Copy link
Owner

vicb commented Sep 2, 2024

Each of the device has unique ID like "!674c3f41" which we can ask user to register their device with flyxc.

How can we validate those id:

  • Is there a known structure, i.e. "!" followed by 8 hex digits that we can use?
  • Can I query an endpoint to make sure the id is registered?

About the payload:

  • How many messages would a single push contain?
  • tm seems to be the timestamp in sec
  • Is groundSpeed km/h
  • Is groundTrack degree
  • What is pDOP
  • Can you send text messages (Looks like https://tracker.bircom.in/api/nodes has some)

@cyberorg
Copy link

cyberorg commented Sep 2, 2024

Is there a known structure, i.e. "!" followed by 8 hex digits that we can use?

Yes, this is the known structure "!" followed by 8 hex digits, like the skylines user ID, users can add this to flyxc settings.

Can I query an endpoint to make sure the id is registered?

Converting that hex get you the decimal, which can be used to query like this:
https://tracker.bircom.in/api/node/1733050177/positions

How many messages would a single push contain?

One message like the packet example shared per push.

tm seems to be the timestamp in sec

Yes, unix timestamp

Is groundSpeed km/h

yes

Is groundTrack degree

not sure about that

What is pDOP

Position Dilution of Precision (PDOP), which can be ignored

Can you send text messages (Looks like https://tracker.bircom.in/api/nodes has some)

yes, this is what the text packet look like:

{"payload":{"packet":{"from":1733050177,"to":4294967295,"channel":0,"decoded":{"portnum":1,"payload":"Test for flyxc","wantResponse":false,"dest":0,"source":0,"requestId":0,"replyId":0,"emoji":0},"id":3631729303,"rxTime":1725260218,"rxSnr":6.75,"hopLimit":3,"wantAck":false,"priority":0,"rxRssi":-52,"delayed":0,"toHex":"!ffffffff","fromHex":"!674c3f41","fromShortName":"Pgid","fromLongName":"Paraguide In/5652D866","fromHardware":"T_ECHO","toShortName":"mesh","toLongName":"Mesh","toHardware":"MESH"},"channelId":"LongFast","gatewayId":"!da52924c"},"_msgid":"c7f75a8c20e963cd"}

@vicb
Copy link
Owner

vicb commented Sep 2, 2024

yes, this is what the text packet look like:

Is there any way to add the lat/lon/alt in the packet on your side? If not I can get the position at around rxTime.

What name should I use for your platform? "meshtastic", "bircom", ... and what stable link can I use to give users more info about the platform.

I think I can the implementation ready by tomorrow.

@vicb
Copy link
Owner

vicb commented Sep 2, 2024

What would help is a payload sample for each of the message you can push (position, text message, any other?)

Thanks!

@cyberorg
Copy link

cyberorg commented Sep 2, 2024

yes, this is what the text packet look like:

Is there any way to add the lat/lon/alt in the packet on your side? If not I can get the position at around rxTime.

Nope, you can use last position received.

What name should I use for your platform? "meshtastic", "bircom", ... and what stable link can I use to give users more info about the platform.

Meshtastic, as that is the project firmware we are using. More information on https://bircom.in. We will update the page for getting the devices registered on flyxc.

I think I can the implementation ready by tomorrow.

You're the best :)

@cyberorg
Copy link

cyberorg commented Sep 2, 2024

What would help is a payload sample for each of the message you can push (position, text message, any other?)

Position packet:

{"payload":[{"lat":32.036736499999996,"lon":76.7071666,"alt":1343,"tm":1725261844,"satsInView":8,"groundSpeed":0,"groundTrack":0,"pDOP":155},{"from":"!674c3f41","fromShortName":"Pgid","fromLongName":"Paraguide In/5652D866","fromHardware":"T_ECHO","to":"!ffffffff","toShortName":"mesh","toLongName":"Mesh","toHardware":"MESH","channelId":"LongFast","gatewayId":"!da52924c"}],"_msgid":"4244095cfa19969f"}
Text packet:

{"payload":{"packet":{"from":1733050177,"to":4294967295,"channel":0,"decoded":{"portnum":1,"payload":"One more test for flyxc","wantResponse":false,"dest":0,"source":0,"requestId":0,"replyId":0,"emoji":0},"id":3631729304,"rxTime":1725261821,"rxSnr":6.25,"hopLimit":3,"wantAck":false,"priority":0,"rxRssi":-54,"delayed":0,"toHex":"!ffffffff","fromHex":"!674c3f41","fromShortName":"Pgid","fromLongName":"Paraguide In/5652D866","fromHardware":"T_ECHO","toShortName":"mesh","toLongName":"Mesh","toHardware":"MESH"},"channelId":"LongFast","gatewayId":"!da52924c"},"_msgid":"0b104298b56d0074"}

Thanks!

🙏

@vicb
Copy link
Owner

vicb commented Sep 2, 2024

You can email me at victor at the app domain to agree on a way to authenticate your push requests

@cyberorg
Copy link

cyberorg commented Sep 2, 2024

We are revisiting device ID. Thinking of using hash of id + shortname + longname. This way the device can be used by different pilots and the tracks will show under their account.

@vicb
Copy link
Owner

vicb commented Sep 2, 2024

We are revisiting device ID. Thinking of using hash of id + shortname + longname. This way the device can be used by different pilots and the tracks will show under their account.

Wouldn't that be a problem if a pilot updates its name? (they would have to update the ID in flyxc, right?)

@cyberorg
Copy link

cyberorg commented Sep 2, 2024

Yes, new pilot will get unique key if he uses the device used by someone else already registered to flyxc and if pilot changes his name then they'll update the key on flyxc.

@vicb
Copy link
Owner

vicb commented Sep 2, 2024

Let me know what the format looks like so that I can validate with a regexp when a device is registered

@cyberorg
Copy link

cyberorg commented Sep 2, 2024

Is groundTrack degree

Yes, 0-359

@ketan
Copy link
Contributor Author

ketan commented Sep 2, 2024

Hi @vicb - I'll take over this implementation from @cyberorg from pushing the data point of view. Here's what I can send you as a reference. Can change the contract as you wish, if there's any optional data or additional that you can use, let me know:

Again, all of this is just indicative as reference, and open to change based on your needs.

curl -X POST https://flyxc.app/api/bircom/push \
      -H 'Accept: application/json' \
      -H 'Content-Type: application/json' \
      -H 'Authorization: bearer [TOKEN]' \
      -d '{
            "type": "message",
            "user_id": "xxxxx",
            "time": "2024-09-02T10:01:12.000Z",
            "message": "I have landed safely, waiting for pickup"
      }'
curl -X POST https://flyxc.app/api/bircom/push \
      -H 'Accept: application/json' \
      -H 'Content-Type: application/json' \
      -H 'Authorization: bearer [TOKEN]' \
      -d '{
            "type": "position",
            "user_id": "xxxxx",
            "time": "2024-09-02T10:01:12.000Z",
            "latitude": 32.1927,
            "longitude": 76.4506,
            "altitude": 1778.12,
            "baro_pressure": 1012.24,
            "ground_speed": 30,
            "heading": 90,
            "satsInView": 12,
      }'

The payload structure. This is the data we have:

Attribute Type Description
type string Type of payload (message, position)
user_id string Unique identifier for the device (not user)
time string ISO 8601 time 2024-09-02T10:01:12.000Z (can be seconds/millseconds from epoch)
latitude number Latitude ± 180 range
longitude number Longitude ±90 range
altitude number GPS altitude, in meters
baro_pressure number Barometric pressure, if available, in hPa.
message string For type message, the text message from the user
ground_speed number The GPS speed over ground (kmph), if available
heading number The GPS heading, if available (0-359 degrees), if available
satsInView number The number of GPS sats in view, if available

@ketan
Copy link
Contributor Author

ketan commented Sep 2, 2024

Let me know what the format looks like so that I can validate with a regexp when a device is registered

I believe this may not be needed, we'll ensure that we're sending valid device IDs right away. That said, we'll be making improvements (probably in the next few weeks/months) to allow signups so we get more stable user-ids.

@ketan
Copy link
Contributor Author

ketan commented Sep 2, 2024

If you'd like to have a quick connect, let me know - we can setup a quick zoom per your convenience.

@vicb
Copy link
Owner

vicb commented Sep 2, 2024

Nice format!

A few comments:

  • I'd prefer to have timeMSec in milliseconds rather than a date as a string
  • satsInView should probably be sats_in_view for consistent naming - but flyXC will not use it anyways
  • It would be nice to known the format of user_id so that I can do some validation when a device is registered - to be sure that there is no typo/copy paste error

@ketan
Copy link
Contributor Author

ketan commented Sep 2, 2024

Nice format!

Thanks!

A few comments:
* I'd prefer to have timeMSec in milliseconds rather than a date as a string
* satsInView should probably be sats_in_view for consistent naming - but flyXC will not use it anyways
* It would be nice to known the format of user_id so that I can do some validation when a device is registered - to be sure that there is no typo/copy paste error

Updating based on your inputs

Attribute Type Description
type string Type of payload (message, position)
user_id string Unique identifier for the device (not user) (uuid format, for e.g. aa752cea-8222-5bc8-acd9-555b090c0ccb)
time number milliseconds since epoch
latitude number Latitude ± 180 range
longitude number Longitude ±90 range
altitude number GPS altitude, in meters
baro_pressure number Barometric pressure, if available, in hPa.
message string For type message, the text message from the user
ground_speed number The GPS speed over ground (kmph), if available
heading number The GPS heading, if available (0-359 degrees), if available

Removed fields

Attribute Type Description
satsInView number The number of GPS sats in view, if available

vicb added a commit that referenced this issue Sep 2, 2024
@vicb vicb closed this as completed in #303 Sep 2, 2024
@vicb vicb changed the title Adding support for another tracker Adding support for Meshtastic (Bircom) Sep 2, 2024
@vicb
Copy link
Owner

vicb commented Sep 2, 2024

flyXC changes have been merged, let me know how the testing goes.

Thanks 🙏

@cyberorg
Copy link

cyberorg commented Sep 3, 2024 via email

@ketan
Copy link
Contributor Author

ketan commented Sep 3, 2024

Hi @vicb - thanks for implementing this quickly.

I'm unable to push to the endpoint, getting a 405 method not allowed.

[start:be]   meshmap:decode Sending job to https://flyxc.app/api/bircom/push +3s
[start:be]   meshmap:decode { headers: { Accept: 'application/json', 'Content-Type': 'application/json', Authorization: 'Bearer xxx' } } +0ms
[start:be]   meshmap:decode { body: '{"time":1725331945607,"type":"message","message":"Test","user_id":"xxx"}' } +0ms
[start:be]   meshmap:decode Response Response { status: 405, statusText: 'Method Not Allowed', headers: Headers { date: 'Tue, 03 Sep 2024 02:52:28 GMT', 'content-length': '0', connection: 'keep-alive', 'access-control-allow-origin': '*', 'referrer-policy': 'strict-origin-when-cross-origin', 'report-to': '{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v4?s=N1Zh%2FWz3GcklGS9cHcFIdWGtj2yFXXX5n678a%2BQJmktUYxKX82QzbAuNQMtirQ%2FOkyajeAgpO6qOFVzXAFdShnh1c%2B%2FYJV0bgKY%2FOfJCwoTLXG0wFT5CN3d%2F7h7htt6KXQByibWeGgc%3D"}],"group":"cf-nel","max_age":604800}', nel: '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}', vary: 'Accept-Encoding', 'cf-cache-status': 'DYNAMIC', server: 'cloudflare', 'cf-ray': '8bd271a7ed15d381-FRA', 'alt-svc': 'h3=":443"; ma=86400' }, body: ReadableStream { locked: false, state: 'readable', supportsBYOB: true }, bodyUsed: false, ok: false, redirected: false, type: 'basic', url: 'https://flyxc.app/api/bircom/push' } +0ms

I also tried with plain old curl, to no avail.

@ketan
Copy link
Contributor Author

ketan commented Sep 3, 2024

Quick update - I changed the post url to api.flyxc.app/api/bircom/push, and I'm now getting a 500 internal server error.

When you get the time, please let me know if this is because of some input errors?

Thanks

@vicb
Copy link
Owner

vicb commented Sep 3, 2024

That's probably a validation error
Could you email me the payload - I have to update a SSH key in my VM to be able to retrieve the logs

@vicb
Copy link
Owner

vicb commented Sep 3, 2024

Reopening this issue:

  • to fix positions
  • to add tes message

@vicb vicb reopened this Sep 3, 2024
@ketan
Copy link
Contributor Author

ketan commented Sep 3, 2024

Here's my curl request (I've removed the token, my user ID is specified here)

$ curl https://api.flyxc.app/api/bircom/push -H 'Accept: application/json' -H 'Content-Type application/json' -H 'Authorization: Bearer XXXXXX'  -d '{"altitude":1393,"time":1725340010504,"type":"position","user_id":"6d032188-142a-5538-8c6a-ff14a67ac166","latitude":32.0411907,"longitude":76.7091656,"ground_speed":20}' -D -
HTTP/2 500
content-type: text/plain; charset=utf-8
vary: Origin
access-control-allow-credentials: true
access-control-expose-headers: xsrf-token
etag: W/"15-/6VXivhc2MKdLfIkLcUE47K6aH0"
x-cloud-trace-context: c66fd0bc9ed6ac38f69a1bd6c6a137cb
date: Tue, 03 Sep 2024 07:01:19 GMT
server: Google Frontend
content-length: 21

Internal Server Error

@vicb
Copy link
Owner

vicb commented Sep 3, 2024

You have a typo, it should be

curl https://api.flyxc.app/api/bircom/push \
-H 'Content-Type: application/json'  \  <- missing ":"
-H 'Authorization: Bearer XXXXXX'  \
-d '...'

or

curl https://api.flyxc.app/api/bircom/push \
-H 'Authorization: Bearer XXXXXX'  \
--json '...'

@vicb
Copy link
Owner

vicb commented Sep 4, 2024

I added support for Bircom messages to flyXC.

Please test the integration and report on that issue.

Your platform is very nice. Hopefully more PG communities around the world will get inspired by your use of Meshtastic.
Thanks for reaching out in the first place and being super responsive on this issue.

Have a great day

@ketan
Copy link
Contributor Author

ketan commented Sep 4, 2024

I'm able to see myself on the map! Thanks a bunch, although unsure why @cyberorg is not able to see himself.

image

One interesting side-effect of how this is implemented is that I'm seeing 500 errors for when submitting data points for devices that are not registered on flyxc. Could you perhaps respond with a 404 or something if the device_id is not available at your end? That way I won't attempt to retry sending packets if the device is not registered?

@vicb
Copy link
Owner

vicb commented Sep 4, 2024

There is no validation of the account on the endpoint

router.post('/push', async (req: Request, res: Response) => {
const [bearer, value] = req.headers.authorization.split(' ');
if (bearer.toLowerCase() !== 'bearer' || value !== Secrets.MESHBIR_AUTH_TOKEN) {
return res.sendStatus(403);
}
try {
const message = parseMessage(req.body);
const pipeline = redis.pipeline();
pushListCap(pipeline, Keys.meshBirMsgQueue, [JSON.stringify(message)], MESHBIR_MAX_MSG, MESHBIR_MAX_MSG_SIZE);
await pipeline.exec();
res.sendStatus(200);
} catch (e) {
console.error(e);
res.sendStatus(500);
}
});

The reason why you receive a 500 error is that one of the field is malformed (user_id?)

Anyway I'll change the return status to:

  • 403: auth error (as of today)
  • 429: queue full, if you ever receive that, I can bump the queue size (new)
  • 400: validation error (currently 500)

@ketan
Copy link
Contributor Author

ketan commented Sep 4, 2024

The reason why you receive a 500 error is that one of the field is malformed (user_id?)

You are right. Then I don't believe a change is necessary at your end. I'm getting a 200 OK if I send a device id that looks valid, but does not exist in your DB, and that I think is reasonable. You accepted the packet, it looked good, and you accepted it, decided to do nothing with it :)

About 429 - how long is the queue length, and processing rate?

@ketan
Copy link
Contributor Author

ketan commented Sep 4, 2024

Could you please also regenerate our token please? @cyberorg accidentally pasted the token here. My email address is `ketanpadegaonkar [at] gmail.com]. Thanks

Apologies for the inconvenience.

@vicb
Copy link
Owner

vicb commented Sep 4, 2024

See #307 for the status changes.

MESHBIR_MAX_MSG is 4000 (messages/min) - happy to bump this when/if needed.

np, I'll email you an updated token

@vicb
Copy link
Owner

vicb commented Sep 4, 2024

The VM that processes the messages ("fetcher") has no static/public IP currently.

That's why the server (Google App Engine) collects the messages and enqueue them in Redis for the fetcher to consume on every tick (1min). The fetcher will discard any message that is not associated with a registered ID.

@vicb
Copy link
Owner

vicb commented Sep 4, 2024

And for @cyberorg, the only reasons why you might not show up on the map after a 200 are:

  • the user_id is not wrong (not associated with a flyXC account)
  • the fix is too old (currently older than 20min)

@vicb
Copy link
Owner

vicb commented Sep 4, 2024

Looks like everything is working now 🎉

@vicb vicb closed this as completed Sep 4, 2024
@cyberorg
Copy link

cyberorg commented Sep 4, 2024

It worked, then it stopped again! What would you want me to try?

position is not being updated and my device messages haven’t showed up yet.

@vicb
Copy link
Owner

vicb commented Sep 4, 2024

@cyberorg Are you using the updated token?
@ketan emailed me that everything is working for him now.

BTW did you ever receive the stickers?

@cyberorg
Copy link

cyberorg commented Sep 4, 2024

Yes, updated token, kept my device on, it is sending positions every 2 min.

Nope something messed up with post if you sent by that.

@vicb
Copy link
Owner

vicb commented Sep 4, 2024

Can you capture a request and email it to me?

@vicb
Copy link
Owner

vicb commented Sep 4, 2024

I have some incoming requests with "user_id":"eda3... which doesn't match any registered pilots, could that explain the pb?

@cyberorg
Copy link

cyberorg commented Sep 4, 2024

Mine is 9d5bb902-...

Sent you mail about txt message.

For position, curl works, so @ketan will have to figure out at his end why push from there is not working.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants