From e7371bbdb094102115df1a8a957e3d38dc41f40b Mon Sep 17 00:00:00 2001 From: Jay Vercellone Date: Fri, 28 Jun 2024 13:48:50 -0700 Subject: [PATCH] Document process for supporting DB components (#12618) * Fix linting issues in component dev guide * Add guideline section for DB components --- docs-v2/pages/components/guidelines.mdx | 480 +++++++++++++++++------- 1 file changed, 347 insertions(+), 133 deletions(-) diff --git a/docs-v2/pages/components/guidelines.mdx b/docs-v2/pages/components/guidelines.mdx index 202be64eabe16..b517a9bbe1875 100644 --- a/docs-v2/pages/components/guidelines.mdx +++ b/docs-v2/pages/components/guidelines.mdx @@ -2,54 +2,65 @@ import Callout from '@/components/Callout' # Components Guidelines & Patterns -For a component to be accepted into the Pipedream registry, it should follow these guidelines below. These guidelines help ensure components are high quality, are intutive for both Pipedream users and component developers to use and extend. +For a component to be accepted into the Pipedream registry, it should follow +these guidelines below. These guidelines help ensure components are high +quality, are intuitive for both Pipedream users and component developers to use +and extend. Questions about best practices? -Join the discussion with fellow Pipedream component developers at the [#contribute channel](https://pipedream-users.slack.com/archives/C01E5KCTR16) in Slack or [on Discourse](https://pipedream.com/community/c/dev/11). +Join the discussion with fellow Pipedream component developers at the +[#contribute channel](https://pipedream-users.slack.com/archives/C01E5KCTR16) in +Slack or [on Discourse](https://pipedream.com/community/c/dev/11). + ## Local Checks When submitting pull requests, the new code will run through a series of automated checks like linting the code. If you want to run those checks locally -for quicker feedback you must have [pnpm](https://pnpm.io/) installed and -run the following commands at the root of the project: +for quicker feedback you must have [pnpm](https://pnpm.io/) installed and run +the following commands at the root of the project: 1. To install all the project's dependencies (only needed once): -```shell -pnpm install -``` + ```shell + pnpm install + ``` 2. To install all required dependencies: -```shell -npx pnpm install -r -``` + ```shell + npx pnpm install -r + ``` -3. To run the linter checks against your code (assuming that your changes are located at `components/foo` for example): - -```shell -npx eslint components/foo -``` +3. To run the linter checks against your code (assuming that your changes are + located at `components/foo` for example): -4. Optionally, you can automatically fix any linter issues by running the following command: + ```shell + npx eslint components/foo + ``` -```shell -npx eslint --fix components/foo -``` +4. Optionally, you can automatically fix any linter issues by running the + following command: -Keep in mind that not all issues can be automatically fixed by the linter -since they could alter the behaviour of the code. + ```shell + npx eslint --fix components/foo + ``` +Keep in mind that not all issues can be automatically fixed by the linter since +they could alter the behaviour of the code. ## General -### Components should be ES modules +### Components Should Be ES Modules -The Node.js community has started publishing [ESM-only](https://flaviocopes.com/es-modules/) packages that do not work with [CommonJS modules](https://nodejs.org/docs/latest/api/modules.html#modules_modules_commonjs_modules). This means you must `import` the package. You can't use `require`. +The Node.js community has started publishing +[ESM-only](https://flaviocopes.com/es-modules/) packages that do not work with +[CommonJS +modules](https://nodejs.org/docs/latest/api/modules.html#modules_modules_commonjs_modules). +This means you must `import` the package. You can't use `require`. You also cannot mix ESM with CJS. This will **not** work: @@ -59,7 +70,7 @@ import axios from "axios"; // CommonJS - this should be `export default` module.exports = { - ... + // ... } ``` @@ -69,15 +80,21 @@ Therefore, all components should be written as ES modules: import axios from "axios"; export default { - ... + //... } ``` -**You'll need to use [the `.mjs` file extension](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules#aside_%E2%80%94_.mjs_versus_.js) for any components written as ES modules**. +**You'll need to use [the `.mjs` file +extension](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules#aside_%E2%80%94_.mjs_versus_.js) +for any components written as ES modules**. -You'll notice that many of the existing components are written as CommonJS modules. Please fix these and submit a pull request as you refactor related code. For example, if you're developing new Spotify actions, and you notice the existing event sources use CommonJS, change them to ESM: +You'll notice that many of the existing components are written as CommonJS +modules. Please fix these and submit a pull request as you refactor related +code. For example, if you're developing new Spotify actions, and you notice the +existing event sources use CommonJS, change them to ESM: -1. Rename the file extension from `.js` to `.mjs` using `git mv` (e.g. `git mv source.js source.mjs`). +1. Rename the file extension from `.js` to `.mjs` using `git mv` (e.g. `git mv + source.js source.mjs`). 2. Change all `require` statements to `import`s. 3. Change instances of `module.exports` to `export default`. @@ -96,10 +113,10 @@ scoped components are easier for users to understand and use. ### Required Metadata -Registry [components](/components/api/#component-structure) require a unique `key` and -`version`, and a friendly `name` and `description`. Action components require a -`type` field to be set to `action` (sources will require a type to be set in the -future). +Registry [components](/components/api/#component-structure) require a unique +`key` and `version`, and a friendly `name` and `description`. Action components +require a `type` field to be set to `action` (sources will require a type to be +set in the future). ```javascript export default { @@ -117,21 +134,31 @@ across registry components and should follow the pattern: `app_name_slug`-`slugified-component-name` -**Source** keys should use past tense verbs that describe the event that occurred (e.g., `linear_app-issue-created-instant`). For **action** keys, use active verbs to describe the action that will occur, (e.g., `linear_app-create-issue`). +**Source** keys should use past tense verbs that describe the event that +occurred (e.g., `linear_app-issue-created-instant`). For **action** keys, use +active verbs to describe the action that will occur, (e.g., +`linear_app-create-issue`). ### Versioning When you first publish a component to the registry, set its version to `0.0.1`. -Pipedream registry components try to follow [semantic versioning](https://semver.org/). From their site: +Pipedream registry components try to follow [semantic +versioning](https://semver.org/). From their site: Given a version number `MAJOR.MINOR.PATCH`, increment the: 1. `MAJOR` version when you make incompatible API changes, -2. `MINOR` version when you add functionality in a backwards compatible manner, and +2. `MINOR` version when you add functionality in a backwards compatible manner, + and 3. `PATCH` version when you make backwards compatible bug fixes. -When you're developing actions locally, and you've incremented the version in your account multiple times, make sure to set it to the version it should be at in the registry prior to submitting your PR. For example, when you add an action to the registry, the version should be `0.0.1`. If the action was at version `0.1.0` and you've fixed a bug, change it to `0.1.1` when committing your final code. +When you're developing actions locally, and you've incremented the version in +your account multiple times, make sure to set it to the version it should be at +in the registry prior to submitting your PR. For example, when you add an action +to the registry, the version should be `0.0.1`. If the action was at version +`0.1.0` and you've fixed a bug, change it to `0.1.1` when committing your final +code. ### Folder Structure @@ -168,14 +195,18 @@ directory](https://github.com/pipedreamhq/pipedream/tree/master/components). #### Using APIs vs Client Libraries If the app has a well-supported [Node.js client -library](/components/api/#using-npm-packages), feel free to use that instead of manually -constructing API requests. +library](/components/api/#using-npm-packages), feel free to use that instead of +manually constructing API requests. ### `package.json` -Each app should have a `package.json` in its root folder. If one doesn't exist, run `npm init` in the app's root folder and customize the file using [this `package.json`](https://github.com/PipedreamHQ/pipedream/blob/55236b3aa993cbcb545e245803d8654c6358b0a2/components/stripe/package.json) as a template. +Each app should have a `package.json` in its root folder. If one doesn't exist, +run `npm init` in the app's root folder and customize the file using [this +`package.json`](https://github.com/PipedreamHQ/pipedream/blob/55236b3aa993cbcb545e245803d8654c6358b0a2/components/stripe/package.json) +as a template. -Each time you change the code for an app file, or change the dependencies for any app component, modify the package `version`. +Each time you change the code for an app file, or change the dependencies for +any app component, modify the package `version`. Save any dependencies in the component app directory: @@ -184,17 +215,25 @@ npm i --save package npm i --save-dev package ``` -#### Error-handling and input validation +#### Error-Handling and Input Validation -When you use the SDK of a popular API, the SDK might raise clear errors to the user. For example, if the user is asked to pass an email address, and that email address doesn't validate, the library might raise that in the error message. +When you use the SDK of a popular API, the SDK might raise clear errors to the +user. For example, if the user is asked to pass an email address, and that email +address doesn't validate, the library might raise that in the error message. -But other libraries will _not_ raise clear errors. In these cases, you may need to `throw` your own custom error that wraps the error from the API / lib. [See the Airtable components](https://github.com/PipedreamHQ/pipedream/blob/9e4e400cda62335dfabfae384d9224e04a585beb/components/airtable/airtable.app.js#L70) for an example of custom error-handling and input validation. +But other libraries will _not_ raise clear errors. In these cases, you may need +to `throw` your own custom error that wraps the error from the API / lib. [See +the Airtable +components](https://github.com/PipedreamHQ/pipedream/blob/9e4e400cda62335dfabfae384d9224e04a585beb/components/airtable/airtable.app.js#L70) +for an example of custom error-handling and input validation. -In general, **imagine you are a user troubleshooting an issue. Is the error easy-to-understand? If not, `throw` a better error**. +In general, **imagine you are a user troubleshooting an issue. Is the error +easy-to-understand? If not, `throw` a better error**. ### `README` files -New actions and sources should include `README.md` files within the same directory to describe how to use the action or source to users. +New actions and sources should include `README.md` files within the same +directory to describe how to use the action or source to users. Here's an example `README.md` structure: @@ -210,11 +249,17 @@ Here's an example `README.md` structure: ``` -These sections will appear within the correponding app, source and action page, along with any subheadings and content. +These sections will appear within the correponding app, source and action page, +along with any subheadings and content. -Here's an example of an [app `README.md` within the `discord` component on the Pipedream registry](https://github.com/PipedreamHQ/pipedream/blob/master/components/discord/README.md). That same content is rendered within the [Pipedream integration page for the Discord app](https://pipedream.com/apps/discord). +Here's an example of an [app `README.md` within the `discord` component on the +Pipedream +registry](https://github.com/PipedreamHQ/pipedream/blob/master/components/discord/README.md). +That same content is rendered within the [Pipedream integration page for the +Discord app](https://pipedream.com/apps/discord). -You can add additional subheadings to each of the top level `Overview`, `Example Use Cases`, `Getting Started` and `Troubleshooting` headings: +You can add additional subheadings to each of the top level `Overview`, `Example +Use Cases`, `Getting Started` and `Troubleshooting` headings: ```markdown # Overview @@ -231,20 +276,22 @@ Perhaps there are some limitations about the API that users should know about. # Getting Started -## Generating an API key +## Generating an API Key Instructions on how to generate an API key from within the service's dashboard. # Troubleshooting -## Required OAuth scopes - -Please take note, you'll need to have sufficient privileges in order to complete authentication. +## Required OAuth Scopes +Please take note, you'll need to have sufficient privileges in order to complete +authentication. ``` -Only these three top level headings `Overview`, `Getting Starting` and `Troubleshooting` will appear within the corresponding App Marketplace page. All other headings will be ignored. +Only these three top level headings `Overview`, `Getting Starting` and +`Troubleshooting` will appear within the corresponding App Marketplace page. All +other headings will be ignored. #### Pagination @@ -271,7 +318,6 @@ logic should: component](https://github.com/PipedreamHQ/pipedream/tree/master/components/microsoft_onedrive/sources/new-file/new-file.mjs) for an example. - #### Capturing Sensitive Data If users are required to enter sensitive data, always use @@ -289,7 +335,8 @@ out](https://pipedream.com/community/c/dev/11). ##### Prop Definitions -Whenever possible, reuse existing [prop definitions](/components/api/#prop-definitions-example). +Whenever possible, reuse existing [prop +definitions](/components/api/#prop-definitions-example). If a prop definition does not exist and you are adding an app-specific prop that may be reused in future components, add it as a prop definition to the app file. @@ -297,11 +344,9 @@ Prop definitions will also be surfaced for apps the Pipedream marketplace. ##### Methods -Whenever possible, reuse -[methods](/components/api/#methods) -defined in the app file. If you need to use an API for which a method is not -defined and it may be used in future components, define a new method in the app -file. +Whenever possible, reuse [methods](/components/api/#methods) defined in the app +file. If you need to use an API for which a method is not defined and it may be +used in future components, define a new method in the app file. Use the [JS Docs](https://jsdoc.app/about-getting-started.html) pattern for lightweight documentation of each method in the app file. Provide a description @@ -349,7 +394,7 @@ changes to an app file that may impact other sources, you must currently test potentially impacted components to confirm their functionality is not negatively affected. We expect to support a testing framework in the future. -### Common Files (optional) +### Common Files (Optional) An optional pattern to improve reusability is to use a `common` module to abstract elements that are used across to multiple components. The trade-off @@ -371,9 +416,9 @@ approach, the general pattern is: etc) and potentially redefining any inherited methods. See [Google -Drive](https://github.com/PipedreamHQ/pipedream/tree/master/components/google_drive) for an -example of this pattern. When using this approach, prop definitions should still -be maintained in the app file. +Drive](https://github.com/PipedreamHQ/pipedream/tree/master/components/google_drive) +for an example of this pattern. When using this approach, prop definitions +should still be maintained in the app file. Please note that the name `common` is just a convention and depending on each case it might make sense to name any common module differently. For example, the @@ -384,27 +429,35 @@ contains several modules that are shared between different event sources. ## Props -As a general rule of thumb, we should strive to only incorporate the 3-4 most relevant options from a given API as props. This is not a hard limit, but the goal is to optimize for usability. We should aim to solve specific use cases as simply as possible. +As a general rule of thumb, we should strive to only incorporate the 3-4 most +relevant options from a given API as props. This is not a hard limit, but the +goal is to optimize for usability. We should aim to solve specific use cases as +simply as possible. ### Labels -Use [prop](/components/api/#user-input-props) labels to customize the name of a prop or -propDefinition (independent of the variable name in the code). The label should -mirror the name users of an app are familiar with; i.e., it should mirror the -equivalent label in the app's UI. This applies to usage in labels, descriptions, -etc. E.g., the Twitter API property for search keywords is “q”, but but label is -set to “Search Term”. +Use [prop](/components/api/#user-input-props) labels to customize the name of a +prop or propDefinition (independent of the variable name in the code). The label +should mirror the name users of an app are familiar with; i.e., it should mirror +the equivalent label in the app’s UI. This applies to usage in labels, +descriptions, etc. E.g., the Twitter API property for search keywords is “q”, +but its label is set to “Search Term”. ### Descriptions -Include a description for [props](/components/api/#user-input-props) if it helps the -user understand what they need to do. Use Markdown as appropriate -to improve the clarity of the description or instructions. When using Markdown: +Include a description for [props](/components/api/#user-input-props) if it helps +the user understand what they need to do. Use Markdown as appropriate to improve +the clarity of the description or instructions. When using Markdown: - Enclose sample input values in backticks (`` ` ``) - Refer to other props using **bold** by surrounding with double asterisks (\*) - Use Markdown links with descriptive text rather than displaying a full URL. -- If the description isn't self-explanatory, link to the API docs of the relevant method to further clarify how the prop works. When the value of the prop is complex (for example, an object with many properties), link to the section of the API docs that include details on this format. Users may pass values from previous steps using expressions, so they'll need to know how to structure the input data. +- If the description isn't self-explanatory, link to the API docs of the + relevant method to further clarify how the prop works. When the value of the + prop is complex (for example, an object with many properties), link to the + section of the API docs that include details on this format. Users may pass + values from previous steps using expressions, so they'll need to know how to + structure the input data. Examples: @@ -421,8 +474,8 @@ Examples: ### Optional vs Required Props -Use optional [props](/components/api/#user-input-props) whenever possible to minimize the -input fields required to use a component. +Use optional [props](/components/api/#user-input-props) whenever possible to +minimize the input fields required to use a component. For example, the Twitter search mentions source only requires that a user connect their account and enter a search term. The remaining fields are optional @@ -433,24 +486,25 @@ activate the source: ### Default Values -Provide [default values](/components/api/#user-input-props) whenever possible. NOTE: the -best default for a source doesn’t always map to the default recommended by the -app. For example, Twitter defaults search results to an algorithm that balances -recency and popularity. However, the best default for the use case on Pipedream -is recency. +Provide [default values](/components/api/#user-input-props) whenever possible. +NOTE: the best default for a source doesn’t always map to the default +recommended by the app. For example, Twitter defaults search results to an +algorithm that balances recency and popularity. However, the best default for +the use case on Pipedream is recency. ### Async Options Avoid asking users to enter ID values. Use [async -options](/components/api/#async-options-example) (with label/value definitions) so users -can make selections from a drop down menu. For example, Todoist identifies -projects by numeric IDs (e.g., 12345). The async option to select a project -displays the name of the project as the label, so that’s the value the user sees -when interacting with the source (e.g., “My Project”). The code referencing the -selection receives the numeric ID (12345). - -Async options should also support [pagination](/components/api/#async-options-example) -(so users can navigate across multiple pages of options for long lists). See +options](/components/api/#async-options-example) (with label/value definitions) +so users can make selections from a drop down menu. For example, Todoist +identifies projects by numeric IDs (e.g., 12345). The async option to select a +project displays the name of the project as the label, so that’s the value the +user sees when interacting with the source (e.g., “My Project”). The code +referencing the selection receives the numeric ID (12345). + +Async options should also support +[pagination](/components/api/#async-options-example) (so users can navigate +across multiple pages of options for long lists). See [Hubspot](https://github.com/PipedreamHQ/pipedream/blob/a9b45d8be3b84504dc22bb2748d925f0d5c1541f/components/hubspot/hubspot.app.mjs#L136) for an example of offset-based pagination. See [Twitter](https://github.com/PipedreamHQ/pipedream/blob/d240752028e2a17f7cca1a512b40725566ea97bd/components/twitter/twitter.app.mjs#L200) @@ -458,7 +512,11 @@ for an example of cursor-based pagination. ### Dynamic Props -[Dynamic props](/components/api/#dynamic-props) can improve the user experience for components. They let you render props in the Pipedream dynamically, based on the value of other props, and can be used to collect more specific information that can make it easier to use the component. See the Google Sheets example in the linked component API docs. +[Dynamic props](/components/api/#dynamic-props) can improve the user experience +for components. They let you render props in Pipedream dynamically, based on the +value of other props, and can be used to collect more specific information that +can make it easier to use the component. See the Google Sheets example in the +linked component API docs. ### Interface & Service Props @@ -509,40 +567,63 @@ search criteria”. ### Emit a Summary -Always [emit a summary](/components/api/#emit) for each event. For example, the summary -for each new Tweet emitted by the Search Mentions source is the content of the -Tweet itself. +Always [emit a summary](/components/api/#emit) for each event. For example, the +summary for each new Tweet emitted by the Search Mentions source is the content +of the Tweet itself. If no sensible summary can be identified, submit the event payload in string format as the summary. ### Deduping -Use built-in [deduping strategies](/components/api/#dedupe-strategies) whenever possible -(`unique`, `greatest`, `last`) vs developing custom deduping code. Develop -custom deduping code if the existing strategies do not support the requirements -for a source. +Use built-in [deduping strategies](/components/api/#dedupe-strategies) whenever +possible (`unique`, `greatest`, `last`) vs developing custom deduping code. +Develop custom deduping code if the existing strategies do not support the +requirements for a source. ### Surfacing Test Events -In order to provide users with source events that they can immediately reference when building their workflow, we should implement 2 strategies whenever possible: +In order to provide users with source events that they can immediately reference +when building their workflow, we should implement 2 strategies whenever +possible: + +#### Emit Events on First Run + +- Polling sources should always emit events on the first run (see the [Spotify: + New + Playlist](https://github.com/PipedreamHQ/pipedream/blob/master/components/spotify/sources/new-playlist/new-playlist.mjs) + source as an example) +- Webhook-based sources should attempt to fetch existing events in the + `deploy()` hook during source creation (see the [Jotform: New + Submission](https://github.com/PipedreamHQ/pipedream/blob/master/components/jotform/sources/new-submission/new-submission.mjs) + source) -#### Emit Events on First Run: -- Polling sources should always emit events on the first run (see the [Spotify: New Playlist](https://github.com/PipedreamHQ/pipedream/blob/master/components/spotify/sources/new-playlist/new-playlist.mjs) source as an example) -- Webhook-based sources should attempt to fetch existing events in the `deploy()` hook during source creation (see the [Jotform: New Submission](https://github.com/PipedreamHQ/pipedream/blob/master/components/jotform/sources/new-submission/new-submission.mjs) source) +_Note – make sure to emit the most recent events (considering pagination), and +limit the count to no more than 50 events._ -_Note – make sure to emit the most recent events (considering pagination), and limit the count to no more than 50 events._ +#### Include a Static Sample Event -#### Include a static sample event: -There are times where there may not be any historical events available (think about sources that emit less frequently, like "New Customer" or "New Order", etc). In these cases, we should include a static sample event so users can see the event shape and reference it while building their workflow, even if it's using fake data. +There are times where there may not be any historical events available (think +about sources that emit less frequently, like "New Customer" or "New Order", +etc). In these cases, we should include a static sample event so users can see +the event shape and reference it while building their workflow, even if it's +using fake data. -To do this, -1. Copy the JSON output from the source's emit (what you get from `steps.trigger.event`) and **make sure to remove or scrub any sensitive or personal data** (you can also copy this from the app's API docs) -2. Add a new file called `test-event.mjs` in the same directory as the component source and export the JSON event via `export default` ([example](https://github.com/PipedreamHQ/pipedream/blob/master/components/jotform/sources/new-submission/test-event.mjs)) -3. In the source component code, make sure to import that file as `sampleEmit` ([example](https://github.com/PipedreamHQ/pipedream/blob/master/components/jotform/sources/new-submission/new-submission.mjs#L2)) -4. And finally, export the `sampleEmit` object ([example](https://github.com/PipedreamHQ/pipedream/blob/master/components/jotform/sources/new-submission/new-submission.mjs#L96)) +To achieve this, follow these steps: -This will render a "Generate Test Event" button in the UI for users to emit that sample event: +1. Copy the JSON output from the source's emit (what you get from + `steps.trigger.event`) and **make sure to remove or scrub any sensitive or + personal data** (you can also copy this from the app's API docs) +2. Add a new file called `test-event.mjs` in the same directory as the component + source and export the JSON event via `export default` + ([example](https://github.com/PipedreamHQ/pipedream/blob/master/components/jotform/sources/new-submission/test-event.mjs)) +3. In the source component code, make sure to import that file as `sampleEmit` + ([example](https://github.com/PipedreamHQ/pipedream/blob/master/components/jotform/sources/new-submission/new-submission.mjs#L2)) +4. And finally, export the `sampleEmit` object + ([example](https://github.com/PipedreamHQ/pipedream/blob/master/components/jotform/sources/new-submission/new-submission.mjs#L96)) + +This will render a "Generate Test Event" button in the UI for users to emit that +sample event: ![generate-sample-event](https://res.cloudinary.com/pipedreamin/image/upload/v1690488844/generate-test-event_drjykm.gif) @@ -554,7 +635,9 @@ As a general heuristic, set the default timer interval to 15 minutes. However, you may set a custom interval (greater or less than 15 minutes) if appropriate for the specific source. Users may also override the default value at any time. -For polling sources in the Pipedream registry, the default polling interval is set as a global config. Individual sources can access that default within the props definition: +For polling sources in the Pipedream registry, the default polling interval is +set as a global config. Individual sources can access that default within the +props definition: ``` javascript import { DEFAULT_POLLING_SOURCE_TIMER_INTERVAL } from "@pipedream/platform"; @@ -568,7 +651,7 @@ export default { }, }, }, - // rest of component + // rest of component... } ``` @@ -590,11 +673,12 @@ of just letting the error bubble up). #### Hooks -[Hooks](/components/api/#hooks) are methods that are automatically invoked by Pipedream -at different stages of the [component lifecycle](/components/api/#source-lifecycle). -Webhook subscriptions are typically created when components are instantiated or -activated via the `activate()` hook, and deleted when components are deactivated -or deleted via the `deactivate()` hook. +[Hooks](/components/api/#hooks) are methods that are automatically invoked by +Pipedream at different stages of the [component +lifecycle](/components/api/#source-lifecycle). Webhook subscriptions are +typically created when components are instantiated or activated via the +`activate()` hook, and deleted when components are deactivated or deleted via +the `deactivate()` hook. #### Helper Methods @@ -634,28 +718,35 @@ name, should not be slugified, and should not include the app name. As a general pattern, articles are not included in the action name. For example, instead of "Create a Post", use "Create Post". -#### Use `@pipedream/platform` axios for all HTTP requests +#### Use `@pipedream/platform` axios for all HTTP Requests -By default, the standard `axios` package doesn't return useful debugging data to the user when it `throw`s errors on HTTP 4XX and 5XX status codes. This makes it hard for the user to troubleshoot the issue. +By default, the standard `axios` package doesn't return useful debugging data to +the user when it `throw`s errors on HTTP 4XX and 5XX status codes. This makes it +hard for the user to troubleshoot the issue. Instead, [use `@pipedream/platform` axios](/pipedream-axios/). -#### Return JavaScript objects +#### Return JavaScript Objects -When you `return` data from an action, it's exposed as a [step export](/workflows/steps/#step-exports) for users to reference in future steps of their workflow. Return JavaScript objects in all cases, unless there's a specific reason not to. +When you `return` data from an action, it's exposed as a [step +export](/workflows/steps/#step-exports) for users to reference in future steps +of their workflow. Return JavaScript objects in all cases, unless there's a +specific reason not to. -For example, some APIs return XML responses. If you return XML from the step, it's harder for users to parse and reference in future steps. Convert the XML to a JavaScript object, and return that, instead. +For example, some APIs return XML responses. If you return XML from the step, +it's harder for users to parse and reference in future steps. Convert the XML to +a JavaScript object, and return that, instead. -### "List" actions +### "List" Actions -#### Return an array of objects +#### Return an Array of Objects To simplify using results from "list"/"search" actions in future steps of a workflow, return an array of the items being listed rather than an object with a nested array. [See this example for Airtable](https://github.com/PipedreamHQ/pipedream/blob/cb4b830d93e1495d8622b0c7dbd80cd3664e4eb3/components/airtable/actions/common-list.js#L48-L63). -#### Handle pagination +#### Handle Pagination For actions that return a list of items, the common use case is to retrieve all items. Handle pagination within the action to remove the complexity of needing @@ -672,14 +763,137 @@ components](https://github.com/PipedreamHQ/pipedream/blob/e2bb7b7bea2fdf5869f18e for an example of using a `maxRecords` prop to optionally limit the maximum number of records to return. -### Use `$.summary` to summarize what happened +### Use `$.summary` to Summarize What Happened -[Describe what happened](/components/api/#returning-data-from-steps) when an action succeeds by following these guidelines: +[Describe what happened](/components/api/#returning-data-from-steps) when an +action succeeds by following these guidelines: -- Use plain language and provide helpful and contextually relevant information (especially the count of items) +- Use plain language and provide helpful and contextually relevant information + (especially the count of items) - Whenever possible, use names and titles instead of IDs -- Basic structure: _Successfully [action performed (like added, removed, updated)] “[relevant destination]”_ +- Basic structure: _Successfully [action performed (like added, removed, + updated)] “[relevant destination]”_ + +### Don't Export Data You Know Will Be Large + +Browsers can crash when users load large exports (many MBs of data). When you +know the content being returned is likely to be large – e.g. files — don't +export the full content. Consider writing the data to the `/tmp` directory and +exporting a reference to the file. + +## Database Components + +Pipedream supports a special category of apps called ["databases"](/databases), +such as +[MySQL](https://github.com/PipedreamHQ/pipedream/tree/master/components/mysql), +[PostgreSQL](https://github.com/PipedreamHQ/pipedream/tree/master/components/postgresql), +[Snowflake](https://github.com/PipedreamHQ/pipedream/tree/master/components/snowflake), +etc. Components tied to these apps offer unique features _as long as_ they +comply with some requirements. The most important features are: + +1. A built-in SQL editor that allows users to input a SQL query to be run + against their DB +2. Proxied execution of commands against a DB, which guarantees that such + requests are always being made from the same range of static IPs (see the + [shared static IPs docs](databases#send-requests-from-a-shared-static-ip)) + +When dealing with database components, the Pipedream runtime performs certain +actions internally to make these features work. For this reason, these +components must implement specific interfaces that allows the runtime to +properly interact with their code. These interfaces are usually defined in the +[`@pipedream/platform`](https://github.com/PipedreamHQ/pipedream/tree/master/platform) +package. + +### SQL Editor + +This code editor is rendered specifically for props of type `sql`, and it uses +(whenever possible) the underlying's database schema information to provide +auto-complete suggestions. Each database engine not only has its own SQL +dialect, but also its own way of inspecting the schemas and table information it +stores. For this reason, each app file must implement the logic that's +applicable to the target engine. + +To support the schema retrieval, the app file must implement a method called +`getSchema` that takes no parameters, and returns a data structure with a format +like this: -### Don't export data you know will be large +```javascript +{ + users: { // The entries at the root correspond to table names + metadata: { + rowCount: 100, + }, + schema: { + id: { // The entries under `schema` correspond to column names + columnDefault: null, + dataType: "number", + isNullable: false, + tableSchema: "public", + }, + email: { + columnDefault: null, + dataType: "varchar", + isNullable: false, + tableSchema: "public", + }, + dateOfBirth: { + columnDefault: null, + dataType: "varchar", + isNullable: true, + tableSchema: "public", + }, + }, + }, +} +``` -Browsers can crash when users load large exports (many MBs of data). When you know the content being returned is likely to be large – e.g. files — don't export the full content. Consider writing the data to the `/tmp` directory and exporting a reference to the file. +The +[`lib/sql-prop.ts`](https://github.com/PipedreamHQ/pipedream/blob/master/platform/lib/sql-prop.ts) +file in the `@pipedream/platform` package define the schema format and the +signature of the `getSchema` method. You can also check out existing examples in +the +[MySQL](https://github.com/PipedreamHQ/pipedream/blob/master/components/mysql/mysql.app.mjs), +[PostgreSQL](https://github.com/PipedreamHQ/pipedream/blob/master/components/postgresql/postgresql.app.mjs) +and +[Snowflake](https://github.com/PipedreamHQ/pipedream/blob/master/components/snowflake/snowflake.app.mjs) +components. + +### Shared Static IPs + +When a user runs a SQL query against a database, the request is proxied through +a separate internal service that's guaranteed to always use the same range of +static IPs when making outbound requests. This is important for users that have +their databases protected behind a firewall, as they can whitelist these IPs to +allow Pipedream components to access their databases. + +To make this work, the app file must implement the interface defined in the +[`lib/sql-proxy.ts`](https://github.com/PipedreamHQ/pipedream/blob/master/platform/lib/sql-proxy.ts) +file in the `@pipedream/platform` package. This interface defines the following +methods: + +1. **`getClientConfiguration`**: This method takes no parameters and returns an + object that can be fed directly to the database's client library to + initialize/establish a connection to the database. This guarantees that both + the component and the proxy service use the same connection settings, **so + make sure the component uses this method when initializing the client**. +2. **`executeQuery`**: This method takes a query object and returns the result + of executing the query against the database. The Pipedream runtime will + replace this method with a call to the proxy service, so **every component + must make use of this method in order to support this feature**. +3. **`proxyAdapter`**: This method allows the proxy service to take the + arguments passed to the `executeQuery` method and transform them into a + "generic" query object that the service can then use. The expected format + looks something like this: + ```javascript + { + query: "SELECT * FROM users WHERE id = ?", + params: [42], + } + ``` + +You can check out these example pull requests that allowed components to support +this proxy feature: + +- [#11201 (MySQL)](https://github.com/PipedreamHQ/pipedream/pull/11201) +- [#11202 (PostgreSQL)](https://github.com/PipedreamHQ/pipedream/pull/11202) +- [#12511 (Snowflake)](https://github.com/PipedreamHQ/pipedream/pull/12511)