Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

201021 Fix weekly email report #1954

Merged
merged 6 commits into from
Oct 22, 2020
Merged

201021 Fix weekly email report #1954

merged 6 commits into from
Oct 22, 2020

Conversation

paolodamico
Copy link
Contributor

Changes

Closes #1937.

  • Optimizes processing of reports by processing each team in a separate task.
  • Actual SMTP emails are also sent in a separate celery task.

Checklist

  • All querysets/queries filter by Team (if this PR affects any querysets/queries)
  • Backend tests (if this PR affects the backend)
  • N/A. Cypress E2E tests (if this PR affects the front and/or backend)

@timgl timgl had a problem deploying to posthog-201021-fix-week-3bzddq October 21, 2020 12:42 Failure
@timgl timgl temporarily deployed to posthog-201021-fix-week-3bzddq October 21, 2020 12:47 Inactive
@timgl timgl temporarily deployed to posthog-201021-fix-week-snwjuu October 22, 2020 11:51 Inactive
@paolodamico paolodamico marked this pull request as ready for review October 22, 2020 11:58
@paolodamico
Copy link
Contributor Author

tagging involved in the incident response @fuziontech, @mariusandra & @fuziontech in case you want to review

@timgl timgl merged commit 44bfcff into master Oct 22, 2020
@timgl timgl deleted the 201021-fix-weekly-email branch October 22, 2020 16:46
posthog/migrations/0091_messagingrecord.py Show resolved Hide resolved
raw_email = kwargs.pop("raw_email", None)

if raw_email:
kwargs["email_hash"] = hashlib.sha256(f"{settings.SECRET_KEY}_{raw_email}".encode()).hexdigest()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't quite get why the actual email address is destroyed by hashing here. IMO it could actually come in handy to know who in particular wasn't reached by emailing due to a bug or something.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reasoning behind this is for privacy-preservation reasons. Having the email here would require extra effort to maintain it so that PII is cleared when it has to. This way, even if user data must be permanently deleted from the system, the model will not contain any. I guess if you were trying to debug some delivery issue and you know the email address you could compute the hash and look up using the hash. POV?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, that was also my reasoning for keeping messaging records tied to users, they could get cascade deleted then.

Comment on lines +20 to +29

objects = MessagingRecordManager()

email_hash: models.CharField = models.CharField(max_length=1024)
campaign_key: models.CharField = models.CharField(max_length=128)
sent_at: models.DateTimeField = models.DateTimeField(null=True)
created_at: models.DateTimeField = models.DateTimeField(auto_now_add=True)

class Meta:
unique_together = ("email_hash", "campaign_key") # can only send campaign once to each email
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks a lot UserMessagingRecord in posthog-production, but email address- instead of user-centric. Could we maybe do user-centric here as well to use the keep track of messaging using a single scheme across both repos?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've already created an issue (#1983) to remind us to remove UserMessagingRecord in favor of this model. As for why no user-centric, I do believe we consider this improvement but it has trade-offs, the reason for doing this email-centric at first was due to the way SMTP connections are processed as a completely separate async task.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fix weekly email report
3 participants