Classify web pages under cache-control for Brave Ads

[tmancey]

Resolves brave/brave-browser#9021

Submitter Checklist:

• Submitted a ticket for my issue if one did not already exist.
• Used Github auto-closing keywords in the commit message.
• Added/updated tests for this change (for new code or code which already has tests).
• Verified that these changes build without errors on
  • Android
  • iOS
  • Linux
  • macOS
  • Windows
• Verified that these changes pass automated tests (unit, browser, security tests) on
  • iOS
  • Linux
  • macOS
  • Windows
• Verified that all lint errors/warnings are resolved (npm run lint)
• Ran git rebase master (if needed).
• Ran git rebase -i to squash commits (if needed).
• Tagged reviewers and labelled the pull request as needed.
• Request a security/privacy review as needed.
• Add appropriate QA labels (QA/Yes or QA/No) to include the closed issue in milestone
• Public documentation has been updated as necessary. For instance:
  • https://github.com/brave/brave-browser/wiki/Deviations-from-Chromium-(features-we-disable-or-remove)
  • https://github.com/brave/brave-browser/wiki/Proxy-redirected-URLs
  • https://github.com/brave/brave-browser/wiki/Fingerprinting-Protections
  • https://github.com/brave/brave-browser/wiki/Brave%E2%80%99s-Use-of-Referral-Codes
  • https://github.com/brave/brave-browser/wiki/Custom-Headers
  • https://github.com/brave/brave-browser/wiki/Web-compatibility-issues-with-tracking-protection
  • https://github.com/brave/brave-browser/wiki/QA-Guide

Test Plan:

Confirm pages with cache-control are classified (that have enough content)

Reviewer Checklist:

• New files have MPL-2.0 license header.
• Request a security/privacy review as needed.
• Adequate test coverage exists to prevent regressions
• Verify test plan is specified in PR before merging to source

After-merge Checklist:

• The associated issue milestone is set to the smallest version that the
  changes has landed on.
• All relevant documentation has been updated.

[tmancey]

@kylehickinson Raised brave/brave-ios#2444 for iOS

[tmancey]

@kylehickinson I raised an issue for brave-ios so remove the cache control business logic from the iOS app (comment above updated to reflect the needed change)

[kjozwiak]

@tmancey do you have any specific examples of websites that are using cache-control that didn't work before this fix landed? Is there a specific directive that we should be checking? cache-control: private or cache-control: public?

[tmancey]

@kjozwiak microsoft.com did not work before the change, thanks

[kjozwiak]

@kjozwiak microsoft.com did not work before the change, thanks

Awesome, thanks @tmancey 👍

Reproduced the original issue on macOS 10.15.3 x64 Catalina using the following build:

Brave | 1.7.92 Chromium: 80.0.3987.163 (Official Build) (64-bit)
--- | ---
Revision | e7fbe071abe9328cdce4ffedac9822435fbd3656-refs/branch-heads/3987@{#1037}
OS | macOS Version 10.15.3 (Build 19D76)

Ensured that https://www.microsoft.com/en-ca/ was using no-store directive for cache-control. Example:

[29456:775:0415/120101.483927:INFO:ads_impl.cc(476)] OnTabUpdated.IsFocused for tab id: 2 and url: https://www.microsoft.com/en-ca/
[29456:775:0415/120101.502170:INFO:client.cc(668)] Successfully saved client state
[29456:775:0415/120101.838409:INFO:ads_impl.cc(492)] OnTabUpdated.IsBlurred for tab id: 2 and url: https://www.microsoft.com/en-ca/
[29456:775:0415/120101.863137:INFO:client.cc(668)] Successfully saved client state

Verification PASSED on macOS 10.15.3 x64 Catalina using the following build:

Brave | 1.9.23 Chromium: 81.0.4044.92 (Official Build) nightly (64-bit)
--- | ---
Revision | 32921c79b6f01a0fb2deef0e1d45b42f96581051-refs/branch-heads/4044@{#883}
OS | macOS Version 10.15.3 (Build 19D76)

Ensured that visiting https://www.microsoft.com/en-ca/ classified the website correctly:

[29567:775:0415/120701.349426:INFO:ads_impl.cc(459)] OnTabUpdated.IsFocused for tab id: 7
[29567:775:0415/120707.704980:INFO:ad_conversions.cc(67)] Checking ad conversions
[29567:775:0415/120707.713464:INFO:ads_impl.cc(762)] Successfully classified page as technology & computing-software. Winning category over time is technology & computing-software
[29567:775:0415/120707.723147:INFO:client.cc(593)] Successfully saved client state