Implement Purchase Intent Classifier for Brave Ads #8047

moritzhaller · 2020-02-04T18:02:54Z

For requirements search "Purchase Intent MiniSpec" on shared drive.

btlechowski · 2020-04-01T01:36:24Z

Test plan

Test case 1: Check for conversions item in ads catalog

Connect to US
Open https://ads-serve.bravesoftware.com/v2/catalog
Search for purchase intent.

Expected behavior:
Able to find a segment with purchase intent

Test case 2: record purchase intent from live catalog on staging

Connect to US
Run Brave with command line: /usr/bin/brave-browser-dev --enable-logging=stderr --vmodule=brave_ads=3 --brave-ads-staging --rewards=staging=true
Enable Rewards
Search for Audi a5 in the URL bar
Check Default/ads_service/client.json for automotive purchase intent by make-audi

Expected behavior:
automotive purchase intent by make-audi is not empty and contains one element

{
	"timestamp_in_seconds": ,
	"weight": 
}

Test case 3: show purchase intent ad

Overwrite the catalog with Charles
Catalog: purchase_intent_1set.txt
Run Brave with command line: /usr/bin/brave-browser-dev --enable-logging=stderr --vmodule=brave_ads=3 --brave-ads-staging --rewards=staging=true
Enable Rewards
Search for Audi a5 in the URL bar
Reload the page 9 times
Trigger an ad

Expected behavior:
Ad associated with automotive purchase intent by make-audi is shown (Audi ad)
automotive purchase intent by make-audi in Default/ads_service/client.json is not empty and contains 10 elements
Search logs for Serving ad from categories and check if automotive purchase intent by make-audi is listed

Test Case 4: keywords to funnel stage

Connect to US
Run Brave with command line: /usr/bin/brave-browser-dev --enable-logging=stderr --vmodule=brave_ads=3 --brave-ads-staging --rewards=staging=true
Enable Rewards
Search for audi a5 dealer opening times in the URL bar

Note: you can use any keywords to funnel stage from the Purchase Intent Taxonomy spreadsheet

Expected behavior:
automotive purchase intent by make-audi in Default/ads_service/client.json is not empty and contains 1 element.
The element has weight of 3
automotive purchase intent by category-entry luxury car in Default/ads_service/client.json is not empty and contains 1 element.
The element has weight of 3

Test Case 5: site

Connect to US
Run Brave with command line: /usr/bin/brave-browser-dev --enable-logging=stderr --vmodule=brave_ads=3 --brave-ads-staging --rewards=staging=true
Enable Rewards
Open http://buyatoyota.com in the URL bar

Note: you can use any automotive - low funnel sites from the Purchase Intent Taxonomy spreadsheet

Expected behavior:
purchaseIntentSignalHistory in Default/ads_service/client.json contains many elements (over 20).

Test case 6: Round robin

Overwrite the catalog with Charles
Catalog: purchase_intent_2automotive.txt
Run Brave with command line: /usr/bin/brave-browser-dev --enable-logging=stderr --vmodule=brave_ads=3 --brave-ads-staging --rewards=staging=true
Enable Rewards
Search 4 times for audi a4 in stock (should give 12 weight in total)
Open cars.com (should trigger automotive category)
Trigger ad -> Remember which ad was shown. This is Ad1.
Trigger ad -> another ad is shown than 1st ad. This is Ad2.
Trigger ad -> Ad1 is shown
Trigger ad -> Ad2 is shown
Trigger ad -> Ad1 is shown
Trigger ad -> Ad2 is shown

Note: single ad can not be shown more frequently than once per hour.

Expected behavior:
Same ad is not shown twice in a raw.

btlechowski · 2020-04-01T01:57:40Z

Verification passed on

Brave	1.7.78 Chromium: 80.0.3987.149 (Official Build) dev (64-bit)
Revision	5f4eb224680e5d7dca88504586e9fd951840cac6-refs/branch-heads/3987_137@{#16}
OS	Ubuntu 18.04 LTS

Verified test plan from #8047 (comment) and brave/brave-core#4607

Test case 1
Verified able to find a segment with purchase intent

"segments": [{
		"code": "RuHxHavV2",
		"name": "automotive purchase intent by category-mid luxury car",
		"parentCode": "5W5gsHA15"
	}, {
		"code": "mqiPIYccl",
		"name": "automotive purchase intent by make-audi",
		"parentCode": "rRWCQL4-i"
	}
]

Test case 2
Verified automotive purchase intent by make-audi is not empty and contains one element

"automotive purchase intent by make-audi": [{
		"timestamp_in_seconds": 1585701815,
		"weight": 1
	}
]

Test case 3
Verified Ad associated with automotive purchase intent by make-audi is shown (Audi ad)

Verified automotive purchase intent by make-audi in Default/ads_service/client.json is not empty and contains 10 elements
Needed to reload 10 times (11 in weight). Logged #8941

"automotive purchase intent by make-audi": [{
		"timestamp_in_seconds": 1585613989,
		"weight": 1
	}, {
		"timestamp_in_seconds": 1585614444,
		"weight": 1
	}, {
		"timestamp_in_seconds": 1585614708,
		"weight": 1
	}, {
		"timestamp_in_seconds": 1585615063,
		"weight": 1
	}, {
		"timestamp_in_seconds": 1585615477,
		"weight": 1
	}, {
		"timestamp_in_seconds": 1585615480,
		"weight": 1
	}, {
		"timestamp_in_seconds": 1585615486,
		"weight": 1
	}, {
		"timestamp_in_seconds": 1585615492,
		"weight": 1
	}, {
		"timestamp_in_seconds": 1585615498,
		"weight": 1
	}, {
		"timestamp_in_seconds": 1585615542,
		"weight": 1
	}, {
		"timestamp_in_seconds": 1585615805,
		"weight": 1
	}
]

Search logs for Serving ad from categories and check if automotive purchase intent by make-audi is listed

[16633:1:0331/030309.784666:INFO:ads_impl.cc(1177)] Serving ad from categories:
[16633:1:0331/030309.784772:INFO:ads_impl.cc(1179)]   automotive-car brands
[16633:1:0331/030309.784861:INFO:ads_impl.cc(1179)]   automotive-automotive
[16633:1:0331/030309.784929:INFO:ads_impl.cc(1179)]   automotive-motorcycles
[16633:1:0331/030309.784993:INFO:ads_impl.cc(1179)]   automotive purchase intent by make-audi
[16633:1:0331/030309.786854:INFO:client.cc(668)] Successfully saved client state
[16633:1:0331/030309.787981:INFO:ads_impl.cc(1285)] Found 1 eligible ads
[16633:1:0331/030309.788540:INFO:ads_impl.cc(1534)] Ad notification shown:
  uuid: f2980c48-a97d-4956-b3aa-ffafc4121077
  parentUuid: 7483c3d7-9247-496c-a0bb-17652acffce3
  creativeInstanceId: d9dc0028-8f86-4c92-b25d-981bc1bdb903
  creativeSetId: dea8df38-27b6-4b53-a699-77629934db80
  category: automotive purchase intent by make-audi
  title: Audi
  body: Check out the best models of the year 2020
  targetUrl: https://brave.com/purchase-intent-test

Test case 4
Verified automotive purchase intent by make-audi in Default/ads_service/client.json is not empty and contains 1 element.
Verified The element has weight of 3
Verified automotive purchase intent by category-entry luxury car in Default/ads_service/client.json is not empty and contains 1 element.
Verified The element has weight of 3

"automotive purchase intent by category-entry luxury car": [{
		"timestamp_in_seconds": 1585686239,
		"weight": 3
	}
],
"automotive purchase intent by make-audi": [{
		"timestamp_in_seconds": 1585686239,
		"weight": 3
	}
]

Test case 5
Verified purchaseIntentSignalHistory in Default/ads_service/client.json contains many elements (over 20).
purchase_intent_list_for_site.txt

Test case 6 Round robin
Verified same ad is not shown twice in a row.

Verification is in progress

Brave	1.7.86 Chromium: 80.0.3987.163 (Official Build) (64-bit)
Revision	e7fbe071abe9328cdce4ffedac9822435fbd3656-refs/branch-heads/3987@{#1037}
OS	Windows 10 OS Version 1803 (Build 17134.1006)

Verified the test plan from Implement Purchase Intent Classifier for Brave Ads #8047 (comment)
Verified there is no Purchase intent segment display for locale India
Clean profile enabled rewards and verified that the purchaseIntentSignalHistory array is empty in client.json file
Clean profile enabled rewards and searched audi 5 in URL bar and confirmed purchaseIntentSignalHistory is NOT empty

Test case 1
Verified able to find a segment with purchase intent in catalog.json file

"segments": [
            {
              "code": "RuHxHavV2",
              "name": "automotive purchase intent by category-mid luxury car",
              "parentCode": "5W5gsHA15"
            },
            {
              "code": "mqiPIYccl",
              "name": "automotive purchase intent by make-audi",
              "parentCode": "rRWCQL4-i"
            }
          ],

Test case 2
Verified automotive purchase intent by make-audi is not empty and contains one element

"automotive purchase intent by make-audi": [
      {
        "timestamp_in_seconds": 1586166501,
        "weight": 1
      }
    ]

Test case 3
Verified Ad associated with automotive purchase intent by make-audi is shown (Audi ad)

Verified automotive purchase intent by make-audi in Default/ads_service/client.json is not empty and contains 10 elements
Encountered #8941

"automotive purchase intent by make-audi": [
      {
        "timestamp_in_seconds": 1586169720,
        "weight": 1
      },
      {
        "timestamp_in_seconds": 1586169724,
        "weight": 1
      },
      {
        "timestamp_in_seconds": 1586169728,
        "weight": 1
      },
      {
        "timestamp_in_seconds": 1586169733,
        "weight": 1
      },
      {
        "timestamp_in_seconds": 1586169738,
        "weight": 1
      },
      {
        "timestamp_in_seconds": 1586169742,
        "weight": 1
      },
      {
        "timestamp_in_seconds": 1586169746,
        "weight": 1
      },
      {
        "timestamp_in_seconds": 1586169750,
        "weight": 1
      },
      {
        "timestamp_in_seconds": 1586169754,
        "weight": 1
      },
      {
        "timestamp_in_seconds": 1586169758,
        "weight": 1
      },
      {
        "timestamp_in_seconds": 1586169800,
        "weight": 1
      }
    ]

Search logs for Serving ad from categories and check if automotive purchase intent by make-audi is listed

[19496:19440:0406/161352.937:INFO:ads_impl.cc(1177)] Serving ad from categories:
[19496:19440:0406/161352.937:INFO:ads_impl.cc(1179)]   automotive purchase intent by make-audi
[19496:19440:0406/161352.937:INFO:ads_impl.cc(1179)]   automotive purchase intent by category-entry luxury car
[19496:19440:0406/161352.937:INFO:client.cc(668)] Successfully saved client state
[19496:19440:0406/161352.939:INFO:ads_impl.cc(1285)] Found 1 eligible ads
[19496:19440:0406/161352.939:INFO:ads_impl.cc(1534)] Ad notification shown:
  uuid: 04421911-7723-476d-b0a0-366951b30c08
  parentUuid: 0aad6973-6c36-42a9-8f71-12e75b364086
  creativeInstanceId: d9dc0028-8f86-4c92-b25d-981bc1bdb903
  creativeSetId: dea8df38-27b6-4b53-a699-77629934db80
  category: automotive purchase intent by make-audi
  title: Audi
  body: Check out the best models of the year 2020
  targetUrl: https://brave.com/purchase-intent-test
[19496:19440:0406/161352.940:INFO:ad_notifications.cc(296)] Saving notifications state
[3168:11464:0406/161352.940:INFO:confirmations_impl.cc(1141)] Confirm ad:
  creativeInstanceId: d9dc0028-8f86-4c92-b25d-981bc1bdb903
  creativeSetId: dea8df38-27b6-4b53-a699-77629934db80
  category: automotive purchase intent by make-audi
  targetUrl: https://brave.com/purchase-intent-test
  confirmationType: view

Test case 4
Verified automotive purchase intent by make-audi in Default/ads_service/client.json is not empty and contains 1 element.
Verified The element has weight of 3
Verified automotive purchase intent by category-entry luxury car in Default/ads_service/client.json is not empty and contains 1 element.
Verified The element has weight of 3

"automotive purchase intent by category-entry luxury car": [
      {
        "timestamp_in_seconds": 1586179268,
        "weight": 3
      }
    ],
    "automotive purchase intent by make-audi": [
      {
        "timestamp_in_seconds": 1586179268,
        "weight": 3
      }
    ]

Test case 5
Verified purchaseIntentSignalHistory in Default/ads_service/client.json contains many elements (over 20) for the site car.com.
Purchase intent list for site.txt

Test case 6 Round robin
Verified same ad is not shown twice in a row.

Verification in progress with

Brave	1.7.86 Chromium: 80.0.3987.163 (Official Build) (64-bit)
Revision	e7fbe071abe9328cdce4ffedac9822435fbd3656-refs/branch-heads/3987@{#1037}
OS	macOS Version 10.14.6 (Build 18G3020)

Using a clean profile, I enabled rewards and confirmed that the purchaseIntentSignalHistory array is empty in client.json file
Using a clean profile, I enabled rewards and searched "audi a5" in URL bar (using google.com as default SE) and confirmed that purchaseIntentSignalHistory is not empty
Verified the test plan from Implement Purchase Intent Classifier for Brave Ads #8047 (comment):

Case 1 - PASS

Confirmed able to find a segment with purchase intent in ad catalog.

Case 2 - PASS

Confirmed automotive purchase intent by make-audi is not empty and contains one element.

Case 3 - PASS

Confirmed automotive purchase intent by make-audi in Default/ads_service/client.json is not empty and contains 10 elements
However, encountered 11 purchase intent weight is needed to trigger the ad #8941, so needed to refresh the page an 11th time to trigger the expected ad
Once I refreshed the 11th time as indicated above, I was able to confirm an ad associated with automotive purchase intent by make-audi was shown

Searched logs for Serving ad from categories and confirmed automotive purchase intent by make-audi was listed

[975:775:0407/092717.367588:INFO:ads_impl.cc(1177)] Serving ad from categories:
[975:775:0407/092717.367812:INFO:ads_impl.cc(1179)]   automotive purchase intent by make-audi
[975:775:0407/092717.368077:INFO:ads_impl.cc(1179)]   automotive purchase intent by category-entry luxury car
[975:775:0407/092717.377688:INFO:client.cc(668)] Successfully saved client state
[975:775:0407/092717.378751:INFO:ads_impl.cc(1285)] Found 1 eligible ads
[975:775:0407/092717.379142:INFO:ads_impl.cc(1534)] Ad notification shown:
  uuid: a8fb8059-2e70-4050-8918-a4d5b8b85c14
  parentUuid: 054fc6bf-c452-49b6-9b01-3315a5e5392c
  creativeInstanceId: d9dc0028-8f86-4c92-b25d-981bc1bdb903
  creativeSetId: dea8df38-27b6-4b53-a699-77629934db80
  category: automotive purchase intent by make-audi
  title: Audi
  body: Check out the best models of the year 2020
  targetUrl: https://brave.com/purchase-intent-test
[975:775:0407/092717.379408:INFO:ad_notifications.cc(296)] Saving notifications state
[974:775:0407/092717.381189:INFO:confirmations_impl.cc(1141)] Confirm ad:
  creativeInstanceId: d9dc0028-8f86-4c92-b25d-981bc1bdb903
  creativeSetId: dea8df38-27b6-4b53-a699-77629934db80
  category: automotive purchase intent by make-audi
  targetUrl: https://brave.com/purchase-intent-test
  confirmationType: view

Case 4 - PASS

Confirmed automotive purchase intent by make-audi in Default/ads_service/client.json is not empty and contains 1 element. Confirmed the element has weight of 3.
Confirmed automotive purchase intent by category-entry luxury car in Default/ads_service/client.json is not empty and contains 1 element. Confirmed the element has weight of 3.

"purchaseIntentSignalHistory": {
    "automotive purchase intent by category-entry luxury car": [
      {
        "timestamp_in_seconds": 1586266801,
        "weight": 3
      }
    ],
    "automotive purchase intent by make-audi": [
      {
        "timestamp_in_seconds": 1586266801,
        "weight": 3
      }
    ]
  },

Case 5 - PASS

Confirmed purchaseIntentSignalHistory in Default/ads_service/client.json contains many elements (over 20) for the site buyatoyota.com.

Case 6 - PASS

Confirmed same ad is not shown twice in a row.
Encountered [Desktop] Round robin - ad with only parent category is shown once #8971 while testing

moritzhaller added the feature/ads label Feb 4, 2020

moritzhaller mentioned this issue Feb 12, 2020

Implement purchase intent classifier for brave ads 8047 brave/brave-core#4607

Merged

32 tasks

moritzhaller self-assigned this Feb 13, 2020

tmancey added enhancement priority/P3 The next thing for us to work on. It'll ride the trains. QA/Yes labels Feb 15, 2020

moritzhaller closed this as completed in brave/brave-core#4607 Mar 6, 2020

moritzhaller modified the milestones: 1.5.x - Release Hotfix 2, 1.7.x - Beta Mar 23, 2020

This was referenced Mar 31, 2020

Sometimes purchase intent weight in not increased on google #8939

Closed

11 purchase intent weight is needed to trigger the ad #8941

Closed

Single word keywords does not increase purchase intent when searching #8942

Closed

btlechowski added the QA Pass-Linux label Apr 1, 2020

This was referenced Apr 1, 2020

Ads Purchase Intent funnel keyword with higher weight is ignored #8965

Closed

Parent segment is not defined in ads catalog for purchase intent ads #8968

Closed

[Desktop] Round robin - ad with only parent category is shown once #8971

Closed

LaurenWags added the release-notes/include label Apr 2, 2020

This was referenced Apr 2, 2020

Manual test run on OS X for 1.7.x - Release #9002

Closed

Manual test run on Windows x64 for 1.7.x - Release #9003

Closed

GeetaSarvadnya added the QA Pass-Win64 label Apr 6, 2020

LaurenWags added the QA Pass-macOS label Apr 7, 2020

kjozwiak mentioned this issue Apr 9, 2020

Release Notes for 1.7.x (Release Channel) #9109

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Purchase Intent Classifier for Brave Ads #8047

Implement Purchase Intent Classifier for Brave Ads #8047

moritzhaller commented Feb 4, 2020 •

edited

Loading

btlechowski commented Apr 1, 2020 •

edited

Loading

btlechowski commented Apr 1, 2020 •

edited by LaurenWags

Loading

Implement Purchase Intent Classifier for Brave Ads #8047

Implement Purchase Intent Classifier for Brave Ads #8047

Comments

moritzhaller commented Feb 4, 2020 • edited Loading

btlechowski commented Apr 1, 2020 • edited Loading

btlechowski commented Apr 1, 2020 • edited by LaurenWags Loading

moritzhaller commented Feb 4, 2020 •

edited

Loading

btlechowski commented Apr 1, 2020 •

edited

Loading

btlechowski commented Apr 1, 2020 •

edited by LaurenWags

Loading