Javascript 120Hz devicemotion events for high end inertial applications

[lukes3315]

Browsers only sample sensor data at a varying 67Hz (drops down to 1Hz sometimes), which makes the readings unusable for high end use cases...
We need a robust 120Hz steady sampling rate for our human inertial navigation technology. iOS and Android do have this natively, but apparently current browser technology does not offer the same quality as the native sampling... Sadly :(

[tobie]

Thanks for your input. This is addressed by the spec for sensors which support periodic reporting mode.

Which particular sensors would you need to poll at 120Hz?

Why 120Hz and not 1KHz or 67Hz?

What are you aiming to achieve with this? E.g. is this a latency issue, are you filtering the data and need a number of data points to do so? etc.

What are the precise use cases you're targeting?

Thanks for your input.

[lukes3315]

We would need:

• Accelerometer
• Gyroscope
• Magnetometer

And if possible:

• Barometer

Well 120Hz is sufficient granularity for our application. 67Hz is not granular enough, 1Khz, iOS and Android will not let you poll at such a high frequency. But if you want to have a sample rate of 1Khz that isn't an issue as long as it is adjustable.

We are working on an indoor location technology based on inertial navigation. The current issue is the inconsistency in the data points, for example:
Timing between samples seems to be inconsistent and inaccurate.

Check us out:
www.navisens.com

[tobie]

Thanks for this added information.

[tobie]

@lukes3315

I have a couple of options for this, please LMK which are acceptable.

1. polling at 120 Hz, getting roughly two data points together each time requestAnimationFrame is called?
2. polling at 120 Hz, getting only the latest data point when requestAnimationFrame is called (this is good for latency, not so much if you want all data point, e.g. for filtering)?
3. polling at 120Hz, receiving updates at 120Hz via event observers.

[tobie]

Expecting feedback from Navisens shortly, see: https://twitter.com/navisens/status/824627997797801985.

[lukes3315]

Hi Tobie,

We think there should be two acquisition methods supported:
1/ Polling sensor samples where you request 1 to N samples
which are then received at each polling loop
2/ Passing a callback function to receive 1 to N samples (preventing polling
for each individual sample).

Having the option to batch N samples would allow for a tradeoff between low latency
and greater efficiency depending on the application.

[lukes3315]

Please feel free to reach out if you need more information.

[tobie]

Thanks, @lukes3315.

What do you mean by "polling loop"? I'm also not sure what you mean by option 2 either. :-/

Maybe code examples or f2f would be helpful?

[tobie]

Had a meeting with Luke (@lukes3315) and Ash (@adonikian) of Navisens. Navisens does indoors positioning without touching the GPS, relying instead on motion sensors in users' mobile phones. It collects readings on the device an sends them to remote servers for analysis where it notably corrects for drift using machine learning.

Currently, Navisens is available as native apps. The current max sampling rate of the DeviceOrientation API (~60 Hz) isn't enough for their requirements.

They's ideally need 120 Hz, but could work with 100 Hz.

The sampling intervals need to be consistant. And it's critical that no samples are lost. [Ed: I'd love to understand more about about why that's important.]

Latency, on the other hand, is less an issue, as the readings as sent to a server for analysis anyway.

The kind of API they have in mind would allow them to set the size of the sample batch they want to work with and get an event every time the batch is full so they can send the data over to their servers for analysis.

[Ed: it would be useful to get a better understanding of the kind of analysis that's being done on the readings to justify the high sampling rates requirements and the necessity of not dropping samples and have an idea of the batch sizes.]

[tobie]

Hearing similar use cases from @padenot:

for example, gyroscope at 120Hz sending control information via websocket to a web audio api program, controling various parameter of a synthetizer

@padenot: would be amazingly helpful if you could provide requirements around:

• latency (from time the data point is collected to time you get to use it),
• frequency (how many samples per seconds you need), and
• whether it's OK to batch the samples together (and if so, how often you need these batches sent).

[padenot]

latency (from time the data point is collected to time you get to use it),

This is very dependent on your application. It is generally considered that a normal human (i.e. not a professional drummer, for example) considers that sound happens "immediately" if the delay between doing the thing that makes your sensor react (be it a gyroscope, a fader, a knob, a keyboard press, etc.) is less than 20ms.

Now, you should experiment. Depending on the interaction, the type of sound, the parameter that is being modulated, etc., it is possible that you'll find that 100ms is acceptable. Keep in mind that the output latency of a normal computer using the Web Audio API is between 10ms and 40ms. It's possible to bring this down with special hardware and browser builds.

frequency (how many samples per seconds you need), and

It depends on the use case, for controlling parameters. Any rate is manageable with the Web Audio API, since you're simply doing js calls. There is a possibility to schedule a bunch of things in advance, or to react just in time, etc.

whether it's OK to batch the samples together (and if so, how often you need these batches sent).

This is ok, iff you can take the latency hit caused by the packetizing of the data.

[tobie]

@padenot thanks for your comments.

(For the fun story, I was a professional drummer in a previous life, dabbled with controlling early versions of Live and NI software with a drumKAT and experienced sub 20ms issues first hand when studios moved from analog to digital recording.)

To give you a bit more context, there are security concerns about providing > 60 Hz frequency and even if we managed to work around this, there are perf concerns around providing the data samples faster than the animation frame rate.

If this is a concern for some use cases, I'd like to hear about it, to see if those are use cases we want to fight for or knowingly exclude.

[padenot]

Not for music in the normal sense, but critical for VR and related technologies. You can't really do anything competitive in VR with 60Hz data.

[tobie]

@padenot OK, cool. (I'm aware of the VR requirements, they generally want higher frequency to reduce latency, but don't really care about more than one data point per animation frame. When they do, they're fine getting them batched).

[tobie]

From the feedback gathered so far here, it seems a reporting frequency whose max is tied to the animation frame rate is sufficient.

On the other hand, a faster polling frequency seems necessary to either:

1. gather more data points (indoor navigation use case), or
2. decrease latency (VR use case).

There are privacy concerns with sampling > 60 Hz, though my understanding is research by @maryammjd seems to indicate this is an issue at much lower frequency rates already. (@maryammjd can you please confirm, plus point to relevant papers?)

When polling frequency is higher than reporting frequency and we don't want to loose samples (so not the VR case), we need to buffer samples. This needs discussion as to how best to achieve this. BYOB? Dedicated API? Etc.

[adonikian]

@tobie my response to:

The sampling intervals need to be consistent. And it's critical that no samples are lost. [Ed: I'd love to understand more about about why that's important.]

This is very important, because most applications using inertial sensors use incremental estimation where the current state (e.g. position, velocity, attitude) is composed of a a set of successive relative state estimates. For inertial sensors, this is not just dependent on the sensor readings but also having correct timing information, since the sensor readings are often integrated. Thus, if a sample is lost, it is generally unrecoverable (other than trying to compensate with global information such as the magnetometer).

[maryammjd]

Hi all, may I please ask you to clarify what is the exact problem? In terms of sampling rate, we have never experience 1 Hz with motion and orientation (we have been testing different browsers for years). Is this something browser-related or network-related (e.g. connection problems)?

From the security point of view, as long as the right permission is given to the eligible code, we don't mind the sampling rate. In fact, we believe increasing the sampling rate both on native apps and js in inevitable.

[tobie]

@maryammjd, thanks for your reply. A couple of comments:

Hi all, may I please ask you to clarify what is the exact problem?\

Security and privacy concerns of using sensors at frequencies above 60 Hz.

In terms of sampling rate, we have never experience 1 Hz with motion and orientation (we have been testing different browsers for years). Is this something browser-related or network-related (e.g. connection problems)?

Not sure what you mean. I don't see any reference to 1 Hz in the whole thread. Did you misread 1 KHz way above?

From the security point of view, as long as the right permission is given to the eligible code, we don't mind the sampling rate. In fact, we believe increasing the sampling rate both on native apps and js in inevitable.

Vendor security teams do care however. I'm interested in determining whether they are right to care or not. That is, does increasing frequency create new attack scenarios (e.g. key logging of a nearby keyboard, pin logging on touch phones, crude voice recording, etc). And if so, at what frequency do they appear? What mitigation strategy exist beyond limiting the frequency (e.g. making sure the sensor doesn't report reading when the user is entering their pin code).

[maryammjd]

The very first comment of this thread says: "Browsers only sample sensor data at a varying 67Hz (drops down to 1Hz sometimes)".

Research-wise, sensors at any frequency (even 20 Hz) are able to reveal something about the user's activity (touch actions, PINs, etc.) or his surrounding environment. We can't say 5 hz or 60 hz or 200 hz is safe or not. However, the higher frequency, the better attack results; the lower frequency, the more expensive machine learning techniques, the attackers will find their way through anyway. That is why we advise to have design-level security models rather than implementation limitation solutions i.e. playing with the sampling rates.

[tobie]

The very first comment of this thread says: "Browsers only sample sensor data at a varying 67Hz (drops down to 1Hz sometimes)".

Indeed. Sorry. :)

Research-wise, sensors at any frequency (even 20 Hz) are able to reveal something about the user's activity (touch actions, PINs, etc.) or his surrounding environment. We can't say 5 hz or 60 hz or 200 hz is safe or not. However, the higher frequency, the better attack results; the lower frequency, the more expensive machine learning techniques, the attackers will find their way through anyway. That is why we advise to have design-level security models rather than implementation limitation solutions i.e. playing with the sampling rates.

Do you have pointers to papers that discuss this? That would be super helpful!

[maryammjd]

TouchSignatures: Identification of User Touch Actions and PINs Based on Mobile Sensor Data via JavaScript

Results are presented for Touch Actions on iPhone (20 Hz), and PIN digits on iPhone (20 Hz) and Android (<60 Hz)

[tobie]

Thanks are you aware of other papers that would show similar patterns for other privacy/security concerns?

[tobie]

@maryammjd Amazing!

I'm in the process of doing a writeup on permissions which I'm planning to share shortly.

Maybe @owencm has suggestions here too.

[timvolodine]

@tobie by "user gesture" I meant an input from the user (e.g. permission or dialog notifying that sensors are being enabled) which may help to avoid activation "by default" and inform the user something is happening (which the user could refuse in some way). Not sure how helpful this is from the UI perspective though..

[alexshalamov]

@maryammjd Based on your research work and also papers from other security researchers (gyromic, ACCessory, Pin skimming, ALS Pin skimming).

It looks like the security & privacy risks come from the data exposed by low-level sensors (gyro,accel, als and magn).

What would be the risk estimation for a high-level orientation sensor?

We are collecting data for the security & privacy section of the Chromium implementation design document to find out optimal permission model for sensors. Your feedback would be very much appreciated.

[tobie]

@alexshalamov the research @maryammjd carried out in "TouchSignatures: Identification of User Touch Actions and PINs Based on Mobile Sensor Data via JavaScript" relies on the DeviceOrientationEvent, itself pretty high-level. :)

[alexshalamov]

@tobie If I remember correctly, it uses motion + orientation. Would be nice to know security risks just for orientation sensor, without motion data.

[tobie]

I think @maryammjd addresses this in the above-mentioned paper (p. 21) by comparing her team's work with TouchLogger which only relies on the orientation sensor:

Taplogger performs its experiments on Android devices and identifies 36.4%
of the digit positions in the first attempt by using accelerometer and orientation
sensors. On the other hand, TouchLogger is able to identify the digits with
71.5% accuracy on an Android device by using device orientation.

The paper goes on to note that once some of the discrepancies in methodology between both techniques are addressed, they both provide similar results.

[maryammjd]

I believe all of these high-level sensor data come from acc+gyr+mag. Any combination of this high and low sensor data would reveal sth about the user.

[alexshalamov]

@maryammjd Thanks.
@tobie Thanks for the hint, I will add reference to Taplogger paper and update risk values in the table.

[tobie]

@alexshalamov it would be kind of cool to find an online reference to it. If by chance you came across something, do please share here. :)

[alexshalamov]

@tobie here you go :D
Xu, Z., Bai, K. and Zhu, S., 2012. TapLogger: Inferring User Inputs On Smartphone Touchscreens Using On-board Motion Sensors.

[tobie]

Oh, I had read this!

Need to read it again. :)

[alexshalamov]

@tobie Did you read paper about PIN skimming using ALS? Is there a sensor that do not leak user info? :D

Spreitzer, Raphael. "Pin skimming: Exploiting the ambient-light sensor in mobile devices." Proceedings of the 4th ACM Workshop on Security and Privacy in Smartphones & Mobile Devices. ACM, 2014.

[tobie]

I did not. o_O

This is one of the reasons this whole permission thing is hard. :(

[saabi]

Hi everybody,

I've been reading through the thread and it strikes me that basic usage (approximately what we already get through the old sensor API) should not need special permissions.

Reading sensor data on the background should definitely require permissions explaining about eavesdropping and that one should understand why permission should be granted.

High frequency sequencing is the gray area. So it should ask for permissions with a may-be-dangerous warning, or perhaps just the warning.

High frequency and background sensing should be different permissions.

EDIT: I should add that I'm actually referring to motion and gyro sensors only. I haven't payed attention to the other sensors.

EDIT2: Also, so far, at least the Android store permission system isn't warning users about these issues. That's however something that should be fixed elsewhere. Browser implementors should secure their side indistinctly.

[adonikian]

Hi everyone, we actually just release an in-browser demo on our website which uses the existing sensor API for tracking, check it out at: https://www.navisens.com

There are still many limitations compared to native sampling but we've tried to work around these as much as possible.