Create email job queue and Reaction.Email namespace #1282

jshimko · 2016-08-11T19:04:24Z

creates a job queue for sending emails with error handling, retries, and logging
create Reaction.Email namespace
deprecate ReactionEmailTemplate() in favor of Reaction.Email.getTemplate() (same args)
deprecate Meteor's Email.send() in favor of Reaction.Email.send() (same args, plus job Q)
refactor/cleanup of any email-related methods
add DDP rate limiting for notifications and Meteor accounts methods
add placeholder content to empty fallback email template
add "order complete" email template to proper path (email template paths are in serious need of attention)
update Meteor and NPM deps
Fixes Guest Email Bug #959 and fixes Make sending email async #1200

This is also the first step toward replacing Meteor's accounts email methods with our own so we can use the email job queue for everything the app sends.

API changes

Sending Email

// Original Meteor Email  
Email.send({
  from: '[email protected]',
  to: '[email protected]',
  subject: 'RE: old email API',
  html: '<body>Heyo!</body>'
});

// is now
Reaction.Email.send({
  from: '[email protected]',
  to: '[email protected]',
  subject: 'RE: new email API',
  html: '<body>Heyo!</body>'
});

// identical except for "Reaction." in front of it

Retrieving Email Templates

// old
ReactionEmailTemplate('path/of/template');

// new
Reaction.Email.getTemplate('path/of/template');

// template paths are currently relative to the /private directory

Example usage

// prepare the template
SSR.compileTemplate('some-name', Reaction.Email.getTemplate('path/of/template.html'));

// render the HTML and send
Reaction.Email.send({
  from: '[email protected]',
  to: '[email protected]',
  subject: 'RE: new email API',
  html: SSR.render('some-name', { shopUrl: Meteor.absoluteUrl() })
});

That will add an email job to the job queue. The worker will then process it immediately (in batches of up to 10) and will retry failures up to 5 times (waiting 3 mins between each try) before failing completely. All email sending attempts are logged in the job collection, so these changes are also a first step toward being able to build a UI for admins to keep track of or control email notifications.

brent-hoover · 2016-08-12T02:27:23Z

package.json

@@ -23,15 +23,15 @@
  },
  "dependencies": {
    "accounting-js": "^1.1.1",
-    "authorize-net": "ongoworks/node-authorize-net",
+    "authorize-net": "github:ongoworks/node-authorize-net",


Why this change? From NPM docs "As of version 1.1.65, you can refer to GitHub urls as just "foo": "user/foo-project". Just as with git URLs, a commit-ish suffix can be included. For example:"

I didn't manually change that. npm update --save did it.

brent-hoover · 2016-08-12T03:02:20Z

If I create an anonymous order and then add an email I get this error when I mark it shipped:

I20160812-11:02:20.478(8)? 03:02:20.477Z ERROR Reaction: orders/shipmentShipped: Failed to send notification
I20160812-11:02:20.478(8)?   ReferenceError: Logger is not defined
I20160812-11:02:20.479(8)?       at Object.getTemplate (server/api/core/email.js:56:5)
I20160812-11:02:20.479(8)?       at [object Object].ordersSendNotification (server/methods/core/orders.js:352:45)
I20160812-11:02:20.479(8)?       at packages/check/match.js:107:1
I20160812-11:02:20.479(8)?       at [object Object]._.extend.withValue (packages/meteor/dynamics_nodejs.js:56:1)
I20160812-11:02:20.479(8)?       at Object.exports.Match._failIfArgumentsAreNotAllChecked (packages/check/match.js:106:1)
I20160812-11:02:20.479(8)?       at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1708:18)
I20160812-11:02:20.480(8)?       at packages/ddp-server/livedata_server.js:1624:18

brent-hoover · 2016-08-12T03:11:13Z

server/api/core/email.js

+  try {
+    return Assets.getText(file);
+  } catch (e) {
+    Logger.warn(`Template not found: ${file}. Falling back to coreDefault.html`);


missing Import here, causing the error

nm. comment redacted

brent-hoover · 2016-08-12T03:34:40Z

Except for that minor error this all looks good to me. I got all the emails, etc.

jshimko · 2016-08-12T10:41:32Z

server/api/core/email.js

+import { Meteor } from "meteor/meteor";
+import { check } from "meteor/check";
+import { Job } from "meteor/vsivsi:job-collection";
+import { Jobs, Packages, Templates } from "/lib/collections";


Yes, the Assets import is missing, but that's not where it comes from. It's a Meteor global and...

Currently, it is not possible to import Assets as an ES6 module. Any of the Assets methods below can simply be called directly in any Meteor server code.

https://docs.meteor.com/api/assets.html

And in that case, that import is correct. There is an Assets collection, but Assets.getText isn't associated. An unfortunate namespace collision. You wouldn't be able to use both in the same file without importing the Assets collection as something else.
And that's why globals are bad, Meteor.

Yeah, my mistake. I had already removed my comment.

jshimko · 2016-08-12T10:47:35Z

Missing Logger import fixed.

brent-hoover · 2016-08-12T11:06:03Z

Approved

aaronjudd · 2016-08-14T23:27:54Z

server/methods/accounts/accounts.js

+      Reaction.Email.send({
+        to: email,
+        from: `${shop.name} <${shop.emails[0].address}>`,
+        subject: `You have been invited to join ${shop.name}`,


The email header needs to be i18n strings, and will also need to be editable in email template editor.
This work was in progress in:
#1203
#1136

aaronjudd · 2016-08-14T23:36:02Z

server/methods/core/orders.js

+    const tpl = `orders/${order.workflow.status}`;
+    SSR.compileTemplate(tpl, Reaction.Email.getTemplate(tpl));
+
+    Reaction.Email.send({


Another place we will need translations, perhaps another method like "Reaction.email.submit" that had all the translation logic before Reaction.email.send could make this a bit easier to replace through the codebase, and give a convenient location for future email transforms.

aaronjudd · 2016-08-14T23:39:24Z

👍 @jshimko I'm good with this as is, but made some comments that we should address. I think we've discussed these requirements as well. This pr will make the merging of #1136 and #1203 not very likely so I will close those, but translation logic is a requirement for v0.16.0.

aaronjudd · 2016-08-14T23:54:18Z

server/jobs/email.js

+
+      try {
+        Email.send({ from, to, subject, html });
+        Logger.info(`Successfully sent email to ${to}`);


Not sure it's a good practice to log emails out to the server, maybe "debug"? @Capt-Slow what say you?

I thought of that too. Maybe just redact the full email and leave an identifiable stub?

should be enough info

* create launchdock-connect plugin (#1241) * rebuild launchdock-connect as a plugin module * build more flexible admin email verification method * add Stripe dependency for launchdock-connect * fix linting issues * Adjust inventory on shipment (#1240) * Trap order-completed change * Add missing import * Add cartItemId field to order so that we can modify inventory * Add new "inventory/sold" method * Explicitly set from and to statuses * Handle order insert hook to move ordered inventory to "sold" status * Handle moving inventory from "reserved" to "sold" status * Fix email import * Fix tests * Not so much logging * Call status change method with explicit values * Add test for moving inventory from "reserved" to "sold" * Add test for moving inventory from "sold" to "shipped" * Force eslint to recognize that switch/case should be indented (#1250) * Force eslint to recognize that swtich/case should be indented eslint/eslint#1797 * remove eslint-plugin-meteor - we need to tune this, removing until we have time * enable eslint-2 codeclimate - remove meteor-eslint plugin, failing to load with eslint-2 * updated switch indentation - test eslint switch, case indent * Add route hooks API (#1253) * build initial implementation of route hooks * extend Router with Hooks and register all hooks * more route hooks tweaks * router namespace cleanup * removed unused values from registry entry (linter) * Refactor inventory (#1251) * As noted in comments from previous PR, just some reorg kept separate from the functionality PR * Correct matching * Fix for decrement cart function (#1273) * Add failing test * Fix for update Mongo command * Add failing test for "decrease below zero" * If removeQuantity is more than quantity remove the entire line * updated linting error * linting error fix * listing issues * more cleanup and linting tweaks * fix eslint config for object key quotes * fix tests after error message update * revert quote-props linter change * revert lint changes * Simplify logic per CR comment * import moment-timezone (#1261) - fixes error .names undefined - for loading tz in i18n settings * Fix inventory tests (#1254) * Reduce size of the cart so that orders don't take so long to process. * Add a wait before pulling record * Add a wait before pulling record * Wait after calling Meteor method * Longer timeout * Longer timeout * Temporarily bypass failiing inventory test (#1280) * fix for #1072 (#1247) fix for #1072 * Use forked version of authorize.net that doesn't have vulnerability (#1252) * Update circle node (#1281) * update circle node to v4.4.7 * remove unused packages - remove eslint-plugin-meteor - remove bunyan-loggly (insecure dependency) - currently not in use * updated package.json (#1286) - updates jquery-i18next - updates i18next-browser-languagedetector * Create email job queue and Reaction.Email namespace (#1282) * build job queue for email and refactor related methods * put placeholder content in coreDefault email template * add rate limiting for accounts and notification methods * put order completed template at correct path * update old jquery-i18next * remove log * remove unused var * remove unnecessary filler in placeholder template * log warning when email template isn’t found * return job object from Reaction.Email.send() * fix missing import * fix comment for email method * remove debugging logger * updated to remove name from email should be enough info * logout and hasPermission updates (#1290) * updated hasPermissions, router behaviour - resolves #1122 - refactors client hasPermissions to wait for a userId if one isn’t immediately found - intentionally not redirecting to home page (not sure if that’s the best behavior, better to have login?) - adds subscription manager to a few more collections * fix typo * import lodash might work better if _.find exists * check for existing route table * Fix PayPal PayFlow discounts and refunds (#1275) * allow discounts and refunds with PayPal PayFlow - Fix discounts so they are working - Allow 100% discounts - Fix refunds so they are working * wrap PayPal PayFlow in wrapper for easier testing * PayPal PayFlow refund test * removed temporary file * updated cc error message * update cardNumber schemas cardNumber schemas were not allowing valid credit cards, such as Amex (15 digits), some Visa (13 digits), and foreign Maestro (12 - 19 digits). This updates to allow these lengths to be input. * Remove bash script from postinstall to fix Windows installs (#1299) * don’t run bash scripts in postinstall because Windows * add fallback fonts back to public dir * release 0.15.0 - updated package in preparation for new release * Allow any user who has the createProduct permission to also delete products (#1263) * - ignoring my custom entrypoint (for running prod image with src) * - allow users with createProduct permission to delete products (for marketplace withmultiple sellers) * - removing .gitignore files that is not welcome in repo * Fix Braintree discounts and refunds (#1265) * add Braintree Payment error to en.json * enable discounts for Braintree payments Braintree was not capturing discounts, instead using the amount in the original authorization for the capturing. This update changes the amount to the paymentMethod.amount, which includes discounts. * removed else statement after an if containing a return Since the return inside the IF statement will effectively kill the process if it’s hit, there is no need for the else. * added testing for Braintree refunds This code still needs some love, just wanted to get it up for others to take a look at. * remove no longer needed commented code * remove no longer needed commented code * added field to expected response for testing * moved braintree/refund/list methods into BraintreeApi wrapper Creating a wrapper for Braintree in order to more easily perform testing * move braintree methods into new file * reconfigure all braintree payment code to no longer use ValidateMethod * fixed lint issues * removed code used to skip over 24 hour braintree delay (for testing) * Rename braintreeapi.js to braintreeApi.js * display absolute number of adjustedTotal due to various instances of rounding numbers for display purposes, the adjustedTotal would sometimes display -0.00, as the adjusted total was technically -.00000000000000000000001, even though we display 0.00 when we round it. This update just shows the absolute number, as this is simple a display number and does not have any affect on what is being sent to and from the payment provider. * updated schema to match supported payment methods The current Schema had a 16 number minimum for credit cards, however braintree supports cards which have number lengths ranging from 12-19 * min -> max * removed comments * test testing * update braintree test * update exports of braintreeApi functions * fixed callback error when action has no callback * Updated error message to make more sense to a human user * linter fixes * updated 'Logger.info' to Logger.debug * removed unused test * removed commented callback * don't log full order details on transaction error * fix refunds and discounts for authorize.net (#1279) - Fix discounts so that they are correctly sent to authorize.net - Allow 100% discounts by adding the “voidTransaction” function and voiding transactions - Update error messaging for Refunds, as we do not (yet) allow them from Authorize.net * Fix Stripe refunds and Double Discounts (#1304) * added comment to explain voiding vs applying discount * add ___ for correctly calculating Stripe adjusted totals Stripe doesn’t “do” discounts, instead they take the discount sent to them, and apply it as a refund. This was causing any discounts to show up twice - once as a discount, once as a refund - in our adjusted total. This update fixes that by re-adding the discount price into the adjusted total when the payment provider is Stripe. * also 100% discounts on Stripe * humanizing an error code * removed unintentional text * linting fixes * removed unused commented code * removed unused commented code * Taxes (#1289) * initial commit tax-rebase plugin Initial commit for issue #972 * split providers to plugins/included * Griddle updates - fork MeteorGriddle into core/ui-grid - move fetchTI - add i18n taxSettings * initial grid editing * add taxSettings labels * initial custom add / edit tax rates - initial working forms, edit toggling.. bit rough.. * rename tax-base to taxes * updated custom grid - adds row selection * add custom rate form reset * updated custom settings * add cart hooks to trigger taxes/calculate * update hooks, taxes/deleteRate - also adds taxes/deleteRate * migrate schemas to plugins - also set taxCloud jobs to not run by default * add custom tax rates calculation - add taxes, tax to cart schema - taxes not published to client - updates global cartTotal helpers * check for shipping * migrate settings to plugins * add tax hooks to plugins -abort idea on “provides” as a package property * Avalara Tax Lookup - adds taxes/setRate method * comment unused fields - comment out first implementation unused fields These are fields that will be used future enhancements to taxes. They are commented out for now. * logger cleanup - move info to debug * updated product data with default taxable items - add taxCode placeholder to schema * taxCloud tax rate calculations * disable taxjar, check for packages - disable while module is not functional - WIP fetch rates and api configuration * custom rate ui cleanup * taxes method test - a weak attempt at a test but gets the ball rolling and actually fixes a todo * updated test, versions - add done to test - updated meteor package versions * remove unused template registry entries - final cleanup, ready for PR to development - pending tests - pending docs * avoid running country-data unless address exists * add tax rate delete - adds delete and confirm to edit form * linted griddle.js * remove unused import * fix display of taxes in order workflow perhaps this should be updated to “tax”, instead of taxes in the invoice object. * line item calculations - added for custom rates - added for avalara rates are calculated by individual line items but on a general rate for the cart. * update case linted * updated variant methods - update products/updateProductField to handle boolean - kludge update of childVariants when parentVariant is updated - remove default value from form - small cleanups

* development: (27 commits) Release 0.15.1 (#1314) Fix wrong method hooks context (#1307) - function isSoldOut() should return true if quantity is <= 0, not only if it is === 0. (#1311) Remove jquery-ui (#1310) Taxes (#1289) Fix Stripe refunds and Double Discounts (#1304) fix refunds and discounts for authorize.net (#1279) Fix Braintree discounts and refunds (#1265) Allow any user who has the createProduct permission to also delete products (#1263) Remove bash script from postinstall to fix Windows installs (#1299) update cardNumber schemas Fix PayPal PayFlow discounts and refunds (#1275) logout and hasPermission updates (#1290) Create email job queue and Reaction.Email namespace (#1282) updated package.json (#1286) Update circle node (#1281) Use forked version of authorize.net that doesn't have vulnerability (#1252) fix for #1072 (#1247) Temporarily bypass failiing inventory test (#1280) Fix inventory tests (#1254) ... # Conflicts: # .codeclimate.yml # .eslintignore # .meteor/versions # client/modules/core/main.js # imports/plugins/core/orders/client/templates/workflow/shippingTracking.js # imports/plugins/core/taxes/client/settings/custom.js # imports/plugins/core/taxes/client/settings/settings.js # imports/plugins/core/taxes/server/methods/methods.js # imports/plugins/included/authnet/server/methods/authnet.js # imports/plugins/included/braintree/server/methods/braintreeApi.js # imports/plugins/included/braintree/server/methods/braintreeMethods.js # imports/plugins/included/braintree/server/methods/braintreeapi-methods-refund.app-test.js # imports/plugins/included/example-paymentmethod/server/methods/example-payment-methods.app-test.js # imports/plugins/included/inventory/server/hooks/hooks.js # imports/plugins/included/inventory/server/hooks/inventory-hooks.app-test.js # imports/plugins/included/inventory/server/methods/inventory.js # imports/plugins/included/inventory/server/methods/statusChanges.js # imports/plugins/included/launchdock-connect/client/templates/dashboard.js # imports/plugins/included/paypal/server/methods/payflowpro-methods-refund.app-test.js # imports/plugins/included/paypal/server/methods/payflowproApi.js # imports/plugins/included/paypal/server/methods/payflowproMethods.js # imports/plugins/included/product-variant/client/templates/products/productDetail/variants/variantForm/variantForm.js # imports/plugins/included/taxes-avalara/server/hooks/hooks.js # imports/plugins/included/taxes-taxcloud/server/hooks/hooks.js # imports/plugins/included/taxes-taxjar/server/hooks/hooks.js # lib/collections/helpers.js # package.json # server/api/core/import.js # server/methods/catalog.js # server/methods/core/cart-remove.app-test.js # server/methods/core/orders.js # server/methods/core/shop.js

jshimko added 7 commits August 11, 2016 13:53

build job queue for email and refactor related methods

16ddda5

put placeholder content in coreDefault email template

f06eb8b

add rate limiting for accounts and notification methods

2a19b54

put order completed template at correct path

7005092

update old jquery-i18next

e59f485

remove log

619ad45

remove unused var

d8cc07a

jshimko added the in progress label Aug 11, 2016

jshimko assigned jshimko, brent-hoover and aaronjudd Aug 11, 2016

jshimko added 3 commits August 11, 2016 15:08

remove unnecessary filler in placeholder template

3d86525

log warning when email template isn’t found

15afd3a

return job object from Reaction.Email.send()

2842ee1

brent-hoover reviewed Aug 12, 2016
View reviewed changes

jshimko reviewed Aug 12, 2016
View reviewed changes

fix missing import

8e822d1

jshimko mentioned this pull request Aug 12, 2016

Add DDPRateLimit to all exposed client methods #1267

Closed

jshimko and others added 4 commits August 12, 2016 07:45

fix comment for email method

d08405f

remove debugging logger

202e764

merge updates from development

6c2a5d7

Merge branch 'development' into 959-email-defer

34113a2

aaronjudd mentioned this pull request Aug 14, 2016

[Tutorial]: Docs for email process / customization reactioncommerce/reaction-docs#78

Closed

3 tasks

aaronjudd reviewed Aug 14, 2016
View reviewed changes

updated to remove name from email

c47abbd

should be enough info

aaronjudd merged commit 5135cbe into development Aug 15, 2016

aaronjudd deleted the 959-email-defer branch August 15, 2016 00:05

aaronjudd removed the in progress label Aug 15, 2016

This was referenced Aug 15, 2016

Guest Email Bug #959

Closed

Make sending email async #1200

Closed

brent-hoover mentioned this pull request Aug 18, 2016

Release v0.15.0 #1308

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create email job queue and Reaction.Email namespace #1282

Create email job queue and Reaction.Email namespace #1282

jshimko commented Aug 11, 2016 •

edited by aaronjudd

Loading

brent-hoover Aug 12, 2016

jshimko Aug 12, 2016

brent-hoover commented Aug 12, 2016 •

edited

Loading

brent-hoover Aug 12, 2016

jshimko Aug 12, 2016

brent-hoover commented Aug 12, 2016

jshimko Aug 12, 2016

jshimko Aug 12, 2016

brent-hoover Aug 12, 2016

jshimko commented Aug 12, 2016

brent-hoover commented Aug 12, 2016 •

edited

Loading

aaronjudd Aug 14, 2016 •

edited

Loading

aaronjudd Aug 14, 2016

aaronjudd commented Aug 14, 2016 •

edited

Loading

aaronjudd Aug 14, 2016

brent-hoover Aug 14, 2016

Create email job queue and Reaction.Email namespace #1282

Create email job queue and Reaction.Email namespace #1282

Conversation

jshimko commented Aug 11, 2016 • edited by aaronjudd Loading

API changes

Sending Email

Retrieving Email Templates

Example usage

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brent-hoover commented Aug 12, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brent-hoover commented Aug 12, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jshimko commented Aug 12, 2016

brent-hoover commented Aug 12, 2016 • edited Loading

aaronjudd Aug 14, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aaronjudd commented Aug 14, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jshimko commented Aug 11, 2016 •

edited by aaronjudd

Loading

brent-hoover commented Aug 12, 2016 •

edited

Loading

brent-hoover commented Aug 12, 2016 •

edited

Loading

aaronjudd Aug 14, 2016 •

edited

Loading

aaronjudd commented Aug 14, 2016 •

edited

Loading