support Sec-CH-UA Client Hints in addition to UserAgent

[Findus23]

(I thought I wrote about this before, but couldn't find an issue)

The Chrome team is working on a large change that plans to deprecate the user-agent string.
The current(?) Chrome 83 already freezes the string and Chrome 85 will unify it to one common string for all desktop and one for all mobile users.

As a replacement the client hints standard is proposed: https://github.com/WICG/ua-client-hints

This allows to get similar data about a device already separated by things like brand, browser, os, etc. (meaning device-detector would not be needed anymore).

There is both an HTTP header (that only returns non-vague results if the server requests them) and a JS API for accessing the data.

Things that might complicate this:

the main UA value is not a string, but a list/set

Chrome"; v="73", "ChromiumBasedBrowser"; v="60", "Chromium"; v="73" (and the corresponding dictionary from JS) is a valid response and browser vendors are encouraged to "randomly [include] additional, intentionally incorrect, comma-separated entries with arbitrary ordering".

---

It seems like at the moment only Chromium-based browsers are working on implementing this standard, but as it seems like these are by far the majority now, that doesn't really matter.

I added this to 4.0 RC for now as it impacts every Matomo user as the most popular user agent will display incorrect results.

Links:
https://groups.google.com/a/chromium.org/forum/#!msg/blink-dev/-2JIRNMWJ7s/yHe4tQNLCgAJ
https://www.chromestatus.com/feature/5995832180473856
https://github.com/WICG/ua-client-hints
https://wicg.github.io/client-hints-infrastructure/
https://github.com/mozilla/standards-positions/issues/202#issuecomment-558294095 (Position of the firefox team on this standard)

[tsteur]

BTW @Findus23 just tested using latest Chrome 83 and Chromium 86. Both of them did not send any of the new headers, neither is the new navigator property set to read this through JS, nor did it look like the user agent was frozen. Wonder if these things were delayed eg due to covid 19

[tsteur]

Seems this is still in development: https://www.chromestatus.com/feature/5704553745874944 https://www.chromestatus.com/feature/5704553745874944

[tsteur]

checked again and seems there's still no update

[tsteur]

This feature is still in development in Chrome

[sgiehl]

It's at least meanwhile possible to read the values after enabling them in Chrome > 85:
See https://user-agent-client-hints.glitch.me/

[tsteur]

After enabling the experimental flag this worked for me. It was setting by default eg these headers along the tracking request

sec-ch-ua: "\\Not;A\"Brand";v="99", "Google Chrome";v="85", "Chromium";v="85"
sec-ch-ua-mobile: ?0

Platform headers would not be sent. Be great to focus on the other issues first in the RC milestone and towards the end on this one maybe.

Is the plan that we support these new headers in the device detector @sgiehl ? Or maybe we instead make eventually use of the JS api navigator.userAgentData and send this information along the tracking request once we are able to retrieve it so over time we in a few years we might be able to no longer use device detector which be great (as it is taking too much work). Of course depends if other browsers will implement this as well.

And it will be up to Matomo/website owner to send the correct headers so we receive the needed information?

[sgiehl]

I think I wouldn't add that to device detector actually. We should handle that within Matomo...

[tsteur]

@sgiehl @mattab I'll move this one to Matomo 4.1

[tsteur]

@sgiehl @Findus23 it's not clear to me when they will start freezing user agent. Is it though already worth looking into this what we need to do to make this work?

[sgiehl]

I already had a look on that a few days ago. It's possible to access those client hints using javascript like this:

navigator.userAgentData.getHighEntropyValues(["model", "platform", "platformVersion", "uaFullVersion"])
   .then(function(ua) {
         var model = ua.model;
         var os = ua.platform;
         var osVersion = ua.platformVersion;
         var browserVersion = ua.uaFullVersion;
});

As you can see it's accessible using a promise only, which will make it a bit more difficult to integrate into piwik.js I think. Additionally I did not yet find any useful information which values the variables can have. I only see the values my Chrome has...

What we need to do on Matomo tracking side is to identify the short codes for browser, os and brand based on the given values. Not sure how easy that will be. But guess we need to see that as soon as some more details on the values are known.

[Findus23]

I honestly can't find any news on this that is newer than my initial post. It seems like Google has posponed the planned changes. The only news are that Chrome now has an experimental flag for enabling the client hints.

And I think if the proposal is actually implemented as intended by Google¹ then I am not sure if it is useful to record the output in Matomo as it will intentionally give randomized incorrect results. https://wicg.github.io/ua-client-hints/#grease

https://github.com/w3ctag/design-reviews/issues/467 shows a bit more discussion about the topic of freezing ua strings (and the least helpful "graph" I have ever seen).

---

¹ This proposal is still only created and discussed by Google (and Microsoft Edge) developers, no non-chromium-browser engines have any plans on implementing it.

[Findus23]

> Will the delay also include freezing the UA string (i.e. will the current UA string be up-to-date until some change in 2021).
That is correct

from https://groups.google.com/a/chromium.org/g/blink-dev/c/-2JIRNMWJ7s/m/yHe4tQNLCgAJ

[tsteur]

👍 let's maybe wait until they announce it again (which I think they will).

[ampaze]

Waiting for the Browsers to start freezing the User Agent before implementing Client Hints is really asking for trouble...

Client Hints are enabled by default in Edge 90 and Chrome 90, it won't be that long before User Agent freezing is here.

[Findus23]

It would be great if Client Hints could become a browser standard (both https://wicg.github.io/ua-client-hints/ and https://datatracker.ietf.org/doc/html/rfc8942 seem to be a draft and experimental) and be implemented by more than one browser (engine). Also I don't see any specific information that User Agent freezing is no longer delayed indefinitly (https://bugs.chromium.org/p/chromium/issues/detail?id=955620).

But then again it seems like that's not how the Google Monopoly on browser standards works at the moment, so maybe it might be useful to implement this one day before Google suddenly adds it to all browsers like it did with FLoC.

[ampaze]

But then again it seems like that's not how the Google Monopoly on browser standards works at the moment, so maybe it might be useful to implement this one day before Google suddenly adds it to all browsers like it did with FLoC.

That is most likely how it will work out, even if other browsers don't follow. Chrome alone has enough users, that it might be wise to be prepared for a potential UA-Freeze, even if it will be postponed by Google for a while.

[tsteur]

Moved it back to 4.3 milestone 👍

[tsteur]

fyi https://blog.chromium.org/2021/05/update-on-user-agent-string-reduction.html says it won't happen this year (2021) for chrome and likely won't come any time soon

[DannyRo22]

@tsteur This week Chrome published an update for UA Reduction plans with projected timelines and Chrome milestones: https://blog.chromium.org/2021/09/user-agent-reduction-origin-trial-and-dates.html

[tsteur]

Thanks for this @DannyRo22 I'll move the issue in an earlier milestone so we keep monitoring things.

[tsteur]

https://youtu.be/Qq4rRt8BNjM?t=198 talks about this new feature.

There's also now https://developer.chrome.com/blog/user-agent-reduction-origin-trial/

This will enable sites to discover and fix problems before the reduced UA becomes the default behavior in Chrome (the reduction is planned to start in the second quarter of 2022)

[tsteur]

Someone needs to look at these changes, write up a brief of what needs to be done technically to still detect browsers, devices, browser plugins and operating systems. Then we should split this big issue into lots of smaller issues. For example we could have one issue detected to device, another issue detected to operating systems. It's unclear at this stage how we can best split this up.

We might not fully cover this issue yet in 4.7 and we might need to do this over several releases.

This issue might be a lot of work. For example mapping all the browser names to our browser names etc.

[sgiehl]

We are planning to extend the device detector, so it's also able to handle client hints.
So in the future it would take the useragent and an array with the client hints as parameters and return the best result out of the combined results. If there are usable results in the client hints those will be used first, falling back to the useragent results.

So the remaining work on Matomo side would be to somehow gather the client hints so they are available serverside to be passed to the device detector.
Also we might maybe need to consider to include the client hints in the finger print maybe.

I'll report back here once a version of device detector has been release supporting client hints.
In the meantime someone could already start implementing the other required stuff...

[Findus23]

@sgiehl So does Matomo want to use the JS client hint API and append the data to the matomo.php request? Or ask the browser to send the HTTP headers with the matomo.php request and then use this data server-side?

[sgiehl]

@Findus23 that actually still needs to be defined, but I guess it will be easier to do that in javascript. Guess we couldn't do that with the first tracking request, as we first would need to send the according headers and would need another request to receive the values. Or maybe I misunderstood how that works 🤔

[tsteur]

@sgiehl it be great to define what the other "required stuff" is? What exactly does device detector do and what not?

Also does device detector handle the mapping of the different browser, operating system, ... name to the names we are using?

[sgiehl]

I have just started implementing, but it should take the client hints (either HTTP headers, or the javascript values) and parse them in a way that the results are the same when parsing user agents. So yes, the mapping will be done by device detector.

From what I understood, I would say there are actually only the tasks mentioned above:

• Change the tracking that it receives the client hints
  Without having a closer look I would say the simplest way would be to adjust the detectBrowserFeatures javascript method, to also fetch the client hints. That way also the fingerprint would include them.
• Adjust the php code, so it passes the received client hints to device detector (along with the useragent)

Everything else should imho then work out of the box. Or am I missing something?

[MatomoForumNotifications]

This issue has been mentioned on Matomo forums. There might be relevant details there:

https://forum.matomo.org/t/when-will-support-sec-ch-ua-be-implemented/46570/1

[MatomoForumNotifications]

This issue has been mentioned on Matomo forums. There might be relevant details there:

https://forum.matomo.org/t/getting-rendering-engine-version-not-browser-version/53683/2