Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setup completion tracking no longer working as expected #9039

Closed
4 tasks
aaemnnosttv opened this issue Jul 18, 2024 · 8 comments
Closed
4 tasks

Setup completion tracking no longer working as expected #9039

aaemnnosttv opened this issue Jul 18, 2024 · 8 comments
Labels
P0 High priority Team S Issues for Squad 1 Type: Bug Something isn't working

Comments

@aaemnnosttv
Copy link
Collaborator

aaemnnosttv commented Jul 18, 2024

Bug Description

There are a few internal events that are tracked to measure the success of SK onboarding flows which complement start_site_setup and start_user_setup events respectively.

  • complete_site_setup when the initial SK setup is completed
  • complete_user_setup when a user successfully signs in with Google (initial connection, not each time when granting new scopes)

These are currently implemented in our success banner component which isn't really the right place (for a few reasons) anyways.

Steps to reproduce

  1. Setup SK with tracking opted-in
  2. See these events aren't tracked

Do not alter or remove anything below. The following sections will be managed by moderators only.

Acceptance criteria

  • The following internal GA events should be reimplemented and their current non-functional implementations removed
  • Reasonable effort should be taken to avoid duplicate tracking so as to capture the event once per start
Event Name Category Label Trigger Condition
complete_site_setup {viewContext}_setup proxy or custom-oauth SK Page load Initial SK setup is completed
complete_user_setup {viewContext}_setup proxy or custom-oauth SK Page load Initial user connection is completed (becomes authenticated)

Note: these events are not mutually exclusive

Implementation Brief

  • Update the points where starting site and user setup are tracked (both proxy + GCP components) to set a flag in client side cache to mark the beginning of the event/journey (introducing a utility may be a good idea)
    • The cache related behavior doesn't need to be conditional based on tracking opt-in as it's local state only
  • Introduce a new custom hook for global tracking effects (e.g. useGlobalTrackingEffect)
    • For now this can simply check for the added cache flags on page load, and if present, conditionitionally track these events as defined in the AC
      • Before calling trackEvent, delete the related cache item for the event about to be tracked. This way it shouldn't be necessary to also track state for the purpose of avoiding duplicate tracking
    • In the past, we've used a useMount for this kind of thing but we could probably also use a normal useEffect with viewContext as the dependency as a more SPA-friendly approach (if/when we get there)
  • Update the Header component to use the new effect
    • Putting this in the Header ensures that these events are tracked reliably in situations where a user might not land back on the dashboard right away (such as the site setup including GA)
  • Update SetupSuccessBannerNotification to remove everything related to the tracking of these events

Test Coverage

  • Add test coverage for the added hook

QA Brief

  • Tick the tracking checkbox Help us improve Site Kit by sharing anonymous usage data. and start the Site Kit setup
  • Verify that events listed in AC are being tracked after successful setup, regardless of setting it up with GA4 or without
    • Verify that if you cancel the flow and return back to initial screen with an error message, events are not tracked
  • Confirm this for GCP flow as well. You can check this thread on info how to trigger the GCP flow
  • Connecting/signing in from view only dashboard menu Sign in with Google, upon returning complete_user_setup event should be tracked

Changelog entry

  • Fix measurement of successful setup completion events.
@aaemnnosttv aaemnnosttv added Type: Bug Something isn't working P0 High priority labels Jul 18, 2024
@tofumatt tofumatt self-assigned this Jul 19, 2024
@binnieshah binnieshah added the Team S Issues for Squad 1 label Jul 19, 2024
@tofumatt
Copy link
Collaborator

IB ✅

@tofumatt tofumatt assigned zutigrm and unassigned tofumatt Jul 19, 2024
@zutigrm zutigrm mentioned this issue Jul 22, 2024
18 tasks
@zutigrm zutigrm removed their assignment Jul 22, 2024
@zutigrm zutigrm assigned aaemnnosttv and unassigned zutigrm Jul 23, 2024
@aaemnnosttv aaemnnosttv assigned zutigrm and unassigned aaemnnosttv Jul 23, 2024
@zutigrm zutigrm assigned aaemnnosttv and unassigned zutigrm Jul 23, 2024
@aaemnnosttv aaemnnosttv removed their assignment Jul 23, 2024
@wpdarren wpdarren self-assigned this Jul 24, 2024
@wpdarren
Copy link
Collaborator

wpdarren commented Jul 24, 2024

QA Update: ❌

@zutigrm I have three observations:

  1. The complete_site_setup and complete_user_setup GA events are not getting triggered when setting up Site Kit with the GCP flow. I tried this on two different sites.
  2. I am unable to see the complete_user_setup GA event being triggered when connecting/signing in from view only dashboard menu Sign in with Google See screencast.
  3. On the splash screen where it says, Site Kit has already been configured by another admin of this site. and I set up Site Kit with another Google account, and 2nd admin user, the GA events are not getting triggered here. This might be expected but would like to check.
ga-events.mp4

@wpdarren wpdarren assigned zutigrm and unassigned wpdarren Jul 24, 2024
@zutigrm
Copy link
Collaborator

zutigrm commented Jul 24, 2024

Thanks @wpdarren , the points 2 & 3 were just an added bonus not something that was tracked before, or part of the requirements, so I will remove tracking from view only, as it will consume to much effort, will focus only on first point around GCP setup

@zutigrm zutigrm mentioned this issue Jul 24, 2024
18 tasks
@zutigrm zutigrm assigned aaemnnosttv and unassigned zutigrm Jul 24, 2024
@aaemnnosttv
Copy link
Collaborator Author

@wpdarren I was just looking at this with @zutigrm and confirmed the only issue to fix here is around the tracking when not using the proxy. We verified that the second admin is tracked correctly as well as the sign in from the view-only user menu. It seems the reason you didn't see it is due to not opting in to the tracking, as can be seen in your screencast so that would be WAI ;)

@zutigrm is working on finishing up his follow-up PR for the GCP side and this should be back in QA soon.

@aaemnnosttv
Copy link
Collaborator Author

Back to you @wpdarren 👍

@wpdarren
Copy link
Collaborator

@aaemnnosttv oops, that was stupid of me, noted.

QA Update: ⚠️

@zutigrm I noticed on the GCP side that the complete_user_setup is getting triggered but I can't see complete_site_setup GA Event. I have tried this a few times. I made sure oAuth was cleared, and also local and session storage. According to the AC this should be tracked but not seeing this. Could you take a look?

ga.mp4

@zutigrm
Copy link
Collaborator

zutigrm commented Jul 25, 2024

Thanks @wpdarren , I left my comment in slack thread, with some info for @aaemnnosttv - as I will need your confirmation on the direction, TLDR - since connected status is always true we will need to always track site setup event, regardless if Site Kit is setup first time or by secondary admins, since we can't distinguish the connections status in GCP, like we can with Proxy component

@wpdarren
Copy link
Collaborator

QA Update: ✅

Verified:

  • Events listed in AC are being tracked after successful setup, regardless of setting it up with or without GA4.
  • When I cancel the flow and return back with an error message, events are not tracked.

Note: we decided not to concern ourselves with GCP GA events, and I can confirm for the events are getting tracked as per the AC after successful setup, regardless of setting it up with GA4 or without.

Screenshots

image
image
image
image
image
image
image

@wpdarren wpdarren removed their assignment Jul 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P0 High priority Team S Issues for Squad 1 Type: Bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants