=>
async(page, ...injects) => {
- let pipe: Pipe = createEmptyPipe()
+ let pipeObject: Pipe = createEmptyPipe()
if (injectsHavePipe(injects)) {
- pipe = getInjectsPipeOrEmptyPipe(injects)
+ pipeObject = getInjectsPipeOrEmptyPipe(injects)
injects = injects.slice(0, injects.length - 1)
}
for(const action of actions) {
- const nextPipeValueOrUndefined: PipeValue|void = await action(page, ...injects, pipe)
- pipe = wrapValueInPipe(nextPipeValueOrUndefined as PipeValue|undefined)
+ const nextPipeValueOrUndefined: AbortLineSignal|PipeValue|void = await action(page, ...injects, pipeObject)
+
+ if (isAbortLineSignal(nextPipeValueOrUndefined)) {
+ return processAbortLineSignal(nextPipeValueOrUndefined)
+ }
+
+ pipeObject = wrapValueInPipe(nextPipeValueOrUndefined as PipeValue|undefined)
}
- return pipe.value as any as PipeValue
- }
-```
\ No newline at end of file
+ return pipeObject.value
+ }
+```
+
+This BotAction handles AbortLineSignal's the same way as Pipe's do. It takes one `assembledLines` to abort out of a `pipeRunner()()`.
\ No newline at end of file
diff --git a/src/docs/api/errors.mdx b/src/docs/api/errors.mdx
index 69e5e00..d27c0af 100644
--- a/src/docs/api/errors.mdx
+++ b/src/docs/api/errors.mdx
@@ -13,7 +13,7 @@ Errors assembles BotAction's of the second call `errors()()` in a try/catch to s
If the returned BotAction catches an error, it does not disrupt it's line of assembly.
```typescript
-export const errors =
+const errors =
(errorsBlockName: string = 'Unnamed Errors Block') =>
(...actions: BotAction[]): BotAction =>
async(page, ...injects) => {
@@ -21,15 +21,14 @@ export const errors =
if (injectsHavePipe(injects)) {
return await pipe()(...actions)(page, ...injects)
}
-
- await chain(...(actions as BotAction[]))(page, ...injects)
+ return await chain(...(actions as BotAction[]))(page, ...injects)
} catch(error) {
logError('caught in ' + errorsBlockName)
console.error(error)
- console.log('\n') // append artificial "margin-bottom"
+ console.log('\n')
}
}
```
-`errors()()()` detects for a Pipe object in the injects before choosing the assembly method, to maintain consistency with its own assembly.
+`errors()()()` detects for a Pipe object in the injects before choosing the assembly method, to maintain consistency with its own assembly. In that, it maintains the assembly aborting behavior of either Chain or Pipe, depending on which assembly style is used.
For an usage example, see [Errors BotAction](/advanced/error-handling#errors-botaction)
\ No newline at end of file
diff --git a/src/docs/api/files.mdx b/src/docs/api/files.mdx
index 0dce020..4113839 100644
--- a/src/docs/api/files.mdx
+++ b/src/docs/api/files.mdx
@@ -42,16 +42,12 @@ Takes one screenshot of each URL provided. Filenames are created from urls by re
```typescript
const screenshotAll = (urls: string[], botFileOptions?: Partial): BotFilesAction =>
- async(page, options) => {
- const hydratedOptions = enrichBotFileOptionsWithDefaults({...options, ...botFileOptions})
-
- await forAll(urls)(
- url => ([
- goTo(url),
- screenshot(url.replace(/[^a-zA-Z]/g, '_'))
- ])
- )(page, hydratedOptions)
- }
+ forAll(urls)(
+ url => ([
+ goTo(url),
+ screenshot(url.replace(/[^a-zA-Z]/g, '_'), botFileOptions) // filenames are created from urls by replacing nonsafe characters with underscores
+ ])
+ )
```
By default, saves the file in the local directory. You can change the directory where it's saved by overloading the `BotFileOptions`.
diff --git a/src/docs/api/inject.mdx b/src/docs/api/inject.mdx
index 8e768d2..174779f 100644
--- a/src/docs/api/inject.mdx
+++ b/src/docs/api/inject.mdx
@@ -14,4 +14,7 @@ const inject =
async(page, ...injects) =>
await assemblyLine()(...actions)(page, ...newInjects, ...injects)
```
+
+Since `inject()()` directly uses [assemblyLine()()](/api/assembly-lines#assembly-line), it follows its aborting behavior.
+
Here is an usage [example](/advanced/injects#inject-botaction).
\ No newline at end of file
diff --git a/src/docs/api/navigation.mdx b/src/docs/api/navigation.mdx
index 703ac18..c8c0668 100644
--- a/src/docs/api/navigation.mdx
+++ b/src/docs/api/navigation.mdx
@@ -90,6 +90,38 @@ const waitForNavigation: BotAction = async(page) => {
For an usage example, see the [Instagram login() BotAction](/sites/instagram#login-botaction).
+## Wait
+This BotAction pauses for the time provided before the next BotAction assembled runs.
+
+```typescript
+const wait = (milliseconds: number): BotAction => async() => {
+ await sleep(milliseconds)
+}
+```
+This Bot Action is not like the others in this group, but a utility never the less.
+
+Example:
+```typescript
+await chain(
+ goTo('https://google.com'),
+ wait(5000), // wait 5 seconds before going to duckduckgo.com
+ goTo('https://duckduckgo.com')
+)(page)
+```
+
+## Scroll To
+This BotAction evaluates a `scrollToElement()` helper function in the context of the Puppeteer page based on the HTML selector provided. It calls the HTML node element's `scrollIntoView()` method. Once that method is called, it artificially waits for it to complete scrolling with a timespan set in milliseconds, default of 2500.
+
+```typescript
+const scrollTo = (htmlSelector: string, waitTimeForScroll: number = 2500): BotAction =>
+ chain(
+ evaluate(scrollToElement, htmlSelector),
+ wait(waitTimeForScroll)
+ )
+```
+
+For an example, see LinkedIn's Feed [ifPostNotLoadedTriggerLoadingThenScrape()](https://github.com/mrWh1te/Botmation/blob/e968078fc5e7767db6831c0461afce8b574ac1fe/src/botmation/sites/linkedin/actions/feed.ts#L45) BotAction, which scrolls to an off screen feed post, to cause lazy loading of its content.
+
## Helpers
### enrichGoToPageOptions()
@@ -100,4 +132,20 @@ const enrichGoToPageOptions = (overloadDefaultOptions: Partial =>
+ new Promise(resolve => setTimeout(resolve, milliseconds))
+```
+
+### scrollToElement()
+
+This helper function is ran inside the Page's context to scroll to a particular HTML node element.
+```typescript
+const scrollToElement = (htmlSelector: string) =>
+ document.querySelector(htmlSelector)?.scrollIntoView({behavior: 'smooth'})
```
\ No newline at end of file
diff --git a/src/docs/api/pipe.mdx b/src/docs/api/pipe.mdx
index b53e9b2..03277e4 100644
--- a/src/docs/api/pipe.mdx
+++ b/src/docs/api/pipe.mdx
@@ -45,6 +45,125 @@ await pipe('Hello World')(
)(page)
```
+## Pipe Case
+This BotAction is similar to [givenThat()()](/api/utilities#given-that) in that it evaluates a condition, or in this case conditions, and if at least one condition passes (is true), then the assembled BotAction's are ran. Instead of providing ConditionalBotAction's, the first call `pipeCase()` accepts a spread array of PipeValue's, including Functions. The values provided are tested against the pipe object's value. If the value is a function, it's called as a callback with the pipe object's value (as the first param) to test the return value as true.
+
+> Pipe Case is similar to an if statement where the conditions of the expression are combined with `||` or. At least one case must evaluate as `true` in order for the if statement to be considered true.
+
+```typescript
+const pipeCase =
+ (...valuesToTest: CaseValue[]) =>
+ (...actions: BotAction[]): BotAction =>
+ async(page, ...injects) => {
+ if (injectsHavePipe(injects)) {
+ const pipeObjectValue = getInjectsPipeValue(injects)
+
+ const matches: Dictionary = valuesToTest.reduce((foundMatches, value, index) => {
+ if (typeof value === 'function') {
+ if (value(pipeObjectValue)) {
+ (foundMatches as Dictionary)[index] = value
+ }
+ } else {
+ if (value === pipeObjectValue) {
+ (foundMatches as Dictionary)[index] = value
+ }
+ }
+
+ return foundMatches
+ }, {}) as Dictionary
+
+ if (Object.keys(matches).length > 0) {
+ const returnValue:PipeValue|AbortLineSignal = await pipe()(...actions)(page, ...injects)
+
+ if (isAbortLineSignal(returnValue)) {
+ return processAbortLineSignal(returnValue)
+ } else {
+ return createCasesSignal(matches, true, returnValue)
+ }
+ } else {
+ return createCasesSignal(matches, false, pipeObjectValue)
+ }
+ }
+
+ return createCasesSignal()
+ }
+```
+
+`pipeCase()()` similar to `pipeCases()()` returns an unique JSON object with the [CasesSignal type](https://github.com/mrWh1te/Botmation/blob/master/src/botmation/types/cases.ts). This standardizes the response to interpret what case(s) matched the pipe object value, whether or not the if statement, as a whole condition, passed. Furthermore, there's an optional `pipeValue` to carry back from the result of the assembled BotAction's, which are ran in a Pipe.
+
+It takes `assembledLines` of at least 2 to [abort](/advanced/aborting) a `pipeCase()()` from an assembled BotAction. Here's a table that explains it's aborting behavior given the number of `assembledLines` of an AbortLineSignal returned by an assembled BotAction.
+
+| assembledLines | effect |
+| - | - |
+| 0 | breaks pipeCase()() line, returns the same AbortLineSignal |
+| 1 | breaks pipeCase()() line, returns a CasesSignal with the AbortLineSignal.pipeValue |
+| 2 | breaks pipeCase()() line, returns AbortLineSignal's pipeValue |
+| 3+ | breaks pipeCase()() line, returns same AbortLineSignal with its assembledLines count reduced by 2 |
+
+For an example, see LinkedIn's [likeUserPostsFrom()](https://github.com/mrWh1te/Botmation/blob/e968078fc5e7767db6831c0461afce8b574ac1fe/src/botmation/sites/linkedin/actions/feed.ts#L79) BotAction.
+
+## Pipe Cases
+This BotAction is similar to [givenThat()()](/api/utilities#given-that) in that it evaluates a condition, or in this case conditions, and if all conditions pass (are true), then the assembled BotAction's are ran. Instead of providing ConditionalBotAction's, the first call `pipeCases()` accepts a spread array of PipeValue's, including functions. The values provided are tested against the pipe object's value. If the value is a function, it's called as a callback with the pipe object's value (as the first param) to test the return value as true.
+
+> Pipe Cases is similar to an if statement where the conditions of the expression are combined with `&&` and. All of the cases must evaluate to `true`, in order for the if statement to be considered true.
+
+```typescript
+const pipeCases =
+ (...valuesToTest: CaseValue[]) =>
+ (...actions: BotAction[]): BotAction =>
+ async(page, ...injects) => {
+ const matches: Dictionary = {}
+
+ if (injectsHavePipe(injects)) {
+ const pipeObjectValue = getInjectsPipeValue(injects)
+
+ for (const [i, value] of valuesToTest.entries()) {
+ if (typeof value === 'function') {
+ if (value(pipeObjectValue)) {
+ matches[i] = value
+ } else {
+ break
+ }
+ } else {
+ if (value === pipeObjectValue) {
+ matches[i] = value
+ } else {
+ break
+ }
+ }
+ }
+
+ if (Object.keys(matches).length === valuesToTest.length) {
+ const returnValue:PipeValue|AbortLineSignal = await pipe()(...actions)(page, ...injects)
+
+ if (isAbortLineSignal(returnValue)) {
+ return processAbortLineSignal(returnValue)
+ } else {
+ return createCasesSignal(matches, true, returnValue)
+ }
+ } else {
+ return createCasesSignal(matches, false, pipeObjectValue)
+ }
+ }
+
+ return createCasesSignal(matches)
+ }
+
+```
+
+`pipeCases()()` similar to `pipeCase()()` returns an unique JSON object with the [CasesSignal type](https://github.com/mrWh1te/Botmation/blob/master/src/botmation/types/cases.ts). This standardizes the response to interpret what case(s) matched the pipe object value, whether or not the if statement, as a whole condition, passed. Furthermore, there's an optional `pipeValue` to carry back from the result of the assembled BotAction's, which are ran in a Pipe.
+
+It takes `assembledLines` of at least 2 to [abort](/advanced/aborting) a `pipeCase()()` from an assembled BotAction. Here's a table that explains it's aborting behavior given the number of `assembledLines` of an AbortLineSignal returned by an assembled BotAction.
+
+| assembledLines | effect |
+| - | - |
+| 0 | breaks pipeCase()() line, returns the same AbortLineSignal |
+| 1 | breaks pipeCase()() line, returns a CasesSignal with the AbortLineSignal.pipeValue |
+| 2 | breaks pipeCase()() line, returns AbortLineSignal's pipeValue |
+| 3+ | breaks pipeCase()() line, returns same AbortLineSignal with its assembledLines count reduced by 2 |
+
+For an example, see LinkedIn's [likeUserPostsFrom()](https://github.com/mrWh1te/Botmation/blob/e968078fc5e7767db6831c0461afce8b574ac1fe/src/botmation/sites/linkedin/actions/feed.ts#L79) BotAction.
+
## Helpers
These Helpers are for all BotAction's, including the ones focused just on Chain. They are functions designed to be Assembly Line safe, for creating BotAction's that can safely use a Pipe, but safely run in Chain with safe fallbacks.
@@ -148,4 +267,53 @@ const pipeInjects = (injects: any[]): any[] => {
return [...injects, createEmptyPipe()]
}
```
-This can be used to simulate a BotAction being assembled in a Pipe.
\ No newline at end of file
+This can be used to simulate a BotAction being assembled in a Pipe.
+
+### createCasesSignal()
+This function is used by `pipeCase()()` and `pipeCases()()` to create a `CasesSignal` object to return. If no params are provided, the safe default is a `CasesSignal` that has no matches, the overall condition did not pass, and the pipe value is undefined.
+
+This function is found in a separate `helpers/cases.ts` file
+
+```typescript
+const createCasesSignal = (matches: Dictionary = {}, conditionPass: boolean = false, pipeValue?: PipeValue): CasesSignal => ({
+ brand: 'Cases_Signal',
+ conditionPass,
+ matches,
+ pipeValue
+})
+```
+
+### casesSignalToPipeValue()
+This function is helpful with the [map()](/api/pipe#map) BotAction. It can be provided directly as a means to map the CasesSignal object to its pipe value. Therefore, when combined after a `pipeCase()()` call, it will change the pipe object value to the `CasesSignal.pipeValue`.
+
+This function is found in a separate `helpers/cases.ts` file
+
+```typescript
+const casesSignalToPipeValue = (casesSignal: CasesSignal|any): PipeValue =>
+ isCasesSignal(casesSignal) ? casesSignal.pipeValue : undefined
+```
+
+Example:
+```typescript
+await pipe(5)(
+ pipeCase(5)(
+ // some actions that will run
+ // last BotAction returns a value
+ ),
+ // CasesSignal has pipeValue with last returned value
+ map(casesSignalToPipeValue),
+ log('Cases Signal pipe value')
+)(page)
+```
+
+### isCasesSignal()
+This function is a type guard for the `CasesSignal` type. It's found in the `types/cases.ts` file.
+
+```typescript
+const isCasesSignal = (value: any): value is CasesSignal =>
+ typeof value === 'object' &&
+ value !== null &&
+ value.brand === 'Cases_Signal' &&
+ typeof value.conditionPass === 'boolean' &&
+ isDictionary(value.matches)
+```
\ No newline at end of file
diff --git a/src/docs/api/scrapers.mdx b/src/docs/api/scrapers.mdx
new file mode 100644
index 0000000..51e1403
--- /dev/null
+++ b/src/docs/api/scrapers.mdx
@@ -0,0 +1,143 @@
+---
+title: 'Scrapers'
+---
+
+These BotAction's scrape the Page's `document` by doing something within the Page's context.
+
+Scrapers use a HTML parser to convert the `outerHTML` property of an HTML Element(s) into an interactive object.
+
+The default HTML parser is [cheerio's](https://github.com/cheeriojs/cheerio) [load() function](https://github.com/cheeriojs/cheerio#loading).
+
+You can override the HTML parser by providing a function to operate on the `outerHTML` via the optional higher-order param `higherOrderHTMLParser`, or the optional first [inject](/advanced/injects). If both are provided, the higher-order param will override the injected one, so if you want to use multiple, feel free too.
+
+## HTML Parser
+This higher-order BotAction will [inject](/advanced/injects) the provided HTML parser function of the first `htmlParser()` call, into the assembled BotAction's of the second call `htmlParser()()`.
+
+A method for BotAction's implementing the `ScraperBotAction` interface to override the default HTML parser.
+
+Assembled BotAction's can override the injected HTML parser by providing their own via their own optional higher-order param.
+
+```typescript
+const htmlParser = (htmlParser: Function) =>
+ (...actions: BotAction[]): BotAction =>
+ pipe()(
+ inject(htmlParser)(
+ errors('htmlParser()()')(...actions)
+ )
+ )
+```
+
+For an usage example, see the [LinkedIn example](https://github.com/mrWh1te/Botmation/blob/master/src/examples/linkedin.ts).
+
+## $
+This ScraperBotAction will parse the HTML of the first Element in the document that matches the provided selector.
+
+```typescript
+const $ = (htmlSelector: string, higherOrderHTMLParser?: Function): ScraperBotAction =>
+ async(page, injectedHTMLParser) => {
+ let parser: Function
+
+ if (!higherOrderHTMLParser) {
+ if (injectedHTMLParser) {
+ parser = injectedHTMLParser
+ } else {
+ parser = cheerio.load
+ }
+ } else {
+ parser = higherOrderHTMLParser
+ }
+
+ const scrapedHTML = await page.evaluate(getElementOuterHTML, htmlSelector)
+ return parser(scrapedHTML)
+ }
+```
+
+Example:
+```typescript
+await pipe()(
+ goToDashboard, // fake -> load site with "dashboard"
+ // look at the header for the badge UI representing # of notifications
+ $('header .user .notifications .badge'),
+ // by default, it returns a CheerioStatic instance
+ map(notificationsCount => notificationsCount.text()), // grab the text from the "element"
+ log('# of Notifications') // Pipe: 2
+)(page)
+```
+
+## $$
+This ScraperBotAction will parse the HTML of all Elements in the document that match the provided selector.
+
+```typescript
+const $$ = (htmlSelector: string, higherOrderHTMLParser?: Function): ScraperBotAction =>
+ async(page, ...injects) => {
+ let parser: Function
+
+ // Future support piping the HTML selector with higher-order overriding
+ const [,injectedHTMLParser] = unpipeInjects(injects, 1)
+
+ if (higherOrderHTMLParser) {
+ parser = higherOrderHTMLParser
+ } else {
+ if (injectedHTMLParser) {
+ parser = injectedHTMLParser
+ } else {
+ parser = cheerio.load
+ }
+ }
+
+ const scrapedHTMLs = await page.evaluate(getElementsOuterHTML, htmlSelector)
+ const cheerioEls: CheerioStatic[] = scrapedHTMLs.map(scrapedHTML => parser(scrapedHTML))
+ return cheerioEls as any as R
+ }
+```
+
+Example:
+```typescript
+await pipe()(
+ goToNewsFeed, // fake -> load site with "news feed"
+ // selector to grab each "element" from the Feed container
+ // Each element represents a Post in the Feed with children
+ // elements representing data like Author, Date, Actual Text, etc
+ $$('section.app .feed [data-id]'),
+ // by default, it returns a CheerioStatic[]
+ forAll()( // loop the CheerioStatic[]
+ feedPostEl => ([
+ // scroll and "like" the post IF the post belongs to a friend
+ // in the `friends` array (ie by name matching post authors)
+ givenThat(postBelongsTo(feedPostEl, ...friends))(
+ scrollTo(feedPostEl.attr('id')),
+ like(feedPostEl)
+ )
+ ])
+ )
+)(page)
+```
+
+## Evaluate
+
+This BotAction isn't a Scraper but an important building block for scraping. It's a simple BotAction that wraps Puppeteer Page's [evaluate()](https://devdocs.io/puppeteer/index#pageevaluatepagefunction-args) method to execute JavaScript in the Page. It takes a function, and parameters and returns what the function returns.
+
+```typescript
+const evaluate = (functionToEvaluate: EvaluateFn, ...functionParams: any[]): BotAction =>
+ async(page) => await page.evaluate(functionToEvaluate, ...functionParams)
+```
+
+For an example, see the Navigation BotAction [scrollTo()](/api/navigation#scroll-to)
+
+## Helpers
+
+In order to "scrape" the page's document's HTML, these Helper functions are evaluated in the page's context. To avoid race conditions with the document's nodes, the `outerHTML` property (string representation of the HTML element and children) is returned to the NodeJS context than parsed with a HTML parser.
+
+### getElementOuterHTML()
+
+```typescript
+const getElementOuterHTML = (htmlSelector: string): string|undefined =>
+ document.querySelector(htmlSelector)?.outerHTML
+```
+
+### getElementsOuterHTML()
+
+```typescript
+const getElementsOuterHTML = (htmlSelector: string): string[] =>
+ Array.from(document.querySelectorAll(htmlSelector)).map(el => el.outerHTML)
+```
\ No newline at end of file
diff --git a/src/docs/api/utilities.mdx b/src/docs/api/utilities.mdx
index d08e8ba..c352201 100644
--- a/src/docs/api/utilities.mdx
+++ b/src/docs/api/utilities.mdx
@@ -13,36 +13,88 @@ A functional "if statement".
```typescript
const givenThat =
(condition: ConditionalBotAction) =>
- (...actions: BotAction[]): BotAction =>
+ (...actions: BotAction[]): BotAction =>
async(page, ...injects) => {
- if (await condition(page, ...pipeInjects(injects))) {
- await pipe()(...actions)(page, ...injects)
+ const resolvedConditionValue: AbortLineSignal|boolean = await condition(page, ...pipeInjects(injects))
+
+ if (isAbortLineSignal(resolvedConditionValue)) {
+ return processAbortLineSignal(resolvedConditionValue)
+ }
+
+ if (resolvedConditionValue) {
+ const returnValue: PipeValue|AbortLineSignal = await pipe()(...actions)(page, ...injects)
+
+ if (isAbortLineSignal(returnValue)) {
+ return returnValue
+ }
}
}
```
Takes an async function, similar to a `BotAction` (same parameters but called a [ConditionalBotAction](/advanced/conditionals#conditional-botaction)) that is expected to return a promise that resolves to a Boolean value. If that promise resolves to `true`, only then, will it run the assembled actions.
+This Utility BotAction has unique aborting behavior. First, the ConditionalBotAction can return an AbortLineSignal which is processed normally and returned, aborting the BotAction. Secondly, if an assembled BotAction returns an AbortLineSignal, it aborts the line, except it won't return an AbortLineSignal `pipeValue`. In order to do that, it takes at least `assembledLines` of 2. That's in order to keep this function type-safe with Chain's.
+
The [example Instagram bot](https://github.com/mrWh1te/Botmation/blob/master/src/examples/instagram.ts), uses `givenThat()()`, to attempt login, only if the bot is a Guest on the page then only operates as a logged in User, after checking if the bot is logged in on the page.
## For All
It's a functional "forEach loop".
```typescript
const forAll =
- (collection: any[] | Dictionary) =>
- (botActionOrActionsFactory: (...args: any[]) => BotAction[] | BotAction): BotAction =>
+ (collection?: Collection) =>
+ (botActionOrActionsFactory: (...args: [any, string, Collection]) => BotAction[] | BotAction): BotAction =>
async(page, ...injects) => {
+ if (!collection) {
+ collection = getInjectsPipeValue(injects)
+ }
+
+ if (!collection) {
+ logWarning('Utilities forAll() missing collection')
+ return
+ }
+
+ let returnValue: AbortLineSignal|PipeValue
+
if (Array.isArray(collection)) {
- for(let i = 0; i < collection.length; i++) {
- await pipeActionOrActions(botActionOrActionsFactory(collection[i]))(page, ...injects)
+ for(let index = 0; index < collection.length; index++) {
+ injects = removePipe(injects)
+ injects.push(wrapValueInPipe([collection[index], index+'', collection]))
+
+ returnValue = await pipeActionOrActions(botActionOrActionsFactory(collection[index], index+'', collection))(page, ...injects)
+
+ if (isAbortLineSignal(returnValue)) {
+ return processAbortLineSignal(returnValue)
+ }
}
} else {
- for (const [key, value] of Object.entries(collection)) {
- await pipeActionOrActions(botActionOrActionsFactory(key, value))(page, ...injects)
+ if (isDictionary(collection)) {
+ for (const [key, value] of Object.entries(collection)) {
+ injects = removePipe(injects)
+ injects.push(wrapValueInPipe([value, collection, key]))
+
+ returnValue = await pipeActionOrActions(botActionOrActionsFactory(value, key, collection))(page, ...injects)
+
+ if (isAbortLineSignal(returnValue)) {
+ return processAbortLineSignal(returnValue)
+ }
+ }
}
}
}
```
-This `BotAction` takes a collection, either an array of any type or a simple json object with key/value pairs to iterate with a callback function that returns a Bot Action or an array of Bot Action's to iterate against.
+This `BotAction` takes a collection, either an array of any type or a simple json object with key/value pairs, to iterate with a callback function that returns a Bot Action or an array of Bot Action's to run in.
+
+It's possible to pipe in the `collection` as the first `forAll()` call's parameter is optional.
+
+The callback function is provided three params that are the collection's iterated value, the index casted as a string, and the collection itself. The index is normalized as a string to support the case of iterating an object's key->value pairs.
+
+With each loop iteration, the pipe value injected into the initial assembled BotAction changes. It's created before each loop iteration line, with the same params as the callback, except wrapped in an array. The three values will be the collection's iterated value, the index the loop is iterating on, and the collection itself.
+
+This function assembles a line of BotAction's with each loop iteration, it runs many BotAction lines. Therefore, it has advanced [aborting](/advanced/aborting) behavior to give granular control. Here's a table to understand the effects AbortLineSignal's have with varying `assembledLines` counts:
+
+| assembledLines | effect |
+| - | - |
+| 1 | break the loop iteration line, but do not abort the loop |
+| 2+ | break the loop iteration line, break the loop and return the AbortLineSignal with 2 assembledLines processed |
The [screenshotAll()](https://github.com/mrWh1te/Botmation/blob/master/src/botmation/actions/files.ts) Bot Action wraps the `forAll()()` Bot Action, to run `goTo()` and `screenshot()` actions, on a collection of urls provided.
@@ -52,69 +104,65 @@ It's a functional "while loop".
```typescript
const forAsLong =
(condition: ConditionalBotAction) =>
- (...actions: BotAction[]): BotAction =>
+ (...actions: BotAction[]): BotAction =>
async(page, ...injects) => {
+ let returnValue: PipeValue|AbortLineSignal
let resolvedCondition = await condition(page, ...pipeInjects(injects))
+ if (isAbortLineSignal(resolvedCondition)) {
+ return processAbortLineSignal(resolvedCondition)
+ }
+
while (resolvedCondition) {
- await pipe()(...actions)(page, ...pipeInjects(injects))
+ returnValue = await pipe()(...actions)(page, ...pipeInjects(injects))
+
+ if (isAbortLineSignal(returnValue)) {
+ return processAbortLineSignal(returnValue)
+ }
resolvedCondition = false
resolvedCondition = await condition(page, ...pipeInjects(injects))
+
+ if (isAbortLineSignal(resolvedCondition)) {
+ return processAbortLineSignal(resolvedCondition)
+ }
}
}
```
It resolves the [ConditionalBotAction](/advanced/conditionals#conditional-botaction) before running the actions each time. It stops looping if the condition resolves False or rejects.
+To support granular aborting, it takes two `assembledLines` in an AbortLineSignal to fully abort out of `forAsLong()()`. An AbortLineSignal with 1 `assembledLines` will abort a loop iteration line of BotAction's, but not the loop itself. It's similar to [forAll()()](/api/utilities#for-all), except the ConditionalBotAction of the first `forAsLong()` call can return an AbortLineSignal, which is handled normally. Therefore, if a ConditionalBotAction returns an AbortLineSignal with one `assembledLines`, it aborts fully out of this BotAction.
+
For a concept example, see [Loops: For As Long](/advanced/loops#for-as-long).
## Do While
It's a functional "doWhile loop".
```typescript
-const doWhile =
+const const doWhile =
(condition: ConditionalBotAction) =>
- (...actions: BotAction[]): BotAction =>
+ (...actions: BotAction[]): BotAction =>
async(page, ...injects) => {
- let resolvedCondition = true
+ let returnValue: PipeValue|AbortLineSignal
+ let resolvedCondition: boolean|AbortLineSignal = true
while (resolvedCondition) {
- await pipe()(...actions)(page, ...injects)
+ returnValue = await pipe()(...actions)(page, ...injects)
+
+ if (isAbortLineSignal(returnValue)) {
+ return processAbortLineSignal(returnValue)
+ }
resolvedCondition = false
resolvedCondition = await condition(page, ...pipeInjects(injects))
+
+ if (isAbortLineSignal(resolvedCondition)) {
+ return processAbortLineSignal(resolvedCondition)
+ }
}
}
```
It runs the BotAction's first, then resolves the [ConditionalBotAction](/advanced/conditionals#conditional-botaction) as to whether or not it should run the actions again. It will keep running the pipe of Botaction's in a loop until the condition resolves `false` or rejects.
-For a concept example, see [Loops: Do While](/advanced/loops#do-while).
+This BotAction follows the same aborting behavior as [forAsLong()()](/api/utilities#for-as-long). The ConditionalBotAction can abort the function normally, while assembled BotAction's can abort a loop iteration line only, or that and abort the `doWhile()()` BotAction.
-## Wait
-This BotAction pauses for the time provided before the next BotAction assembled runs.
-
-```typescript
-const wait = (milliseconds: number): BotAction => async() => {
- await sleep(milliseconds)
-}
-```
-This Bot Action is not like the others in this group, but a utility never the less.
-
-Example:
-```typescript
-await chain(
- goTo('https://google.com'),
- wait(5000), // wait 5 seconds before going to duckduckgo.com
- goTo('https://duckduckgo.com')
-)(page)
-```
-
-## Helpers
-
-The one helper here is for the [wait() BotAction](/api/utilities#wait).
-
-### sleep()
-This async function returns a Promise that does not resolve until the time provided lapses.
-```typescript
-const sleep = async(milliseconds: number): Promise =>
- new Promise(resolve => setTimeout(resolve, milliseconds))
-```
+For a concept example, see [Loops: Do While](/advanced/loops#do-while).
diff --git a/src/docs/contributors.mdx b/src/docs/contributors.mdx
index 7232d77..c74e24b 100644
--- a/src/docs/contributors.mdx
+++ b/src/docs/contributors.mdx
@@ -14,12 +14,13 @@ title: Contributors
π§ββπ’ Shout out to all contributors of the code Botmation depends on! Without you, none of this is possible. You all help make *Open Source Software* amazing in exponential ways.
## Infrastructure
-Shout out to all the companies providing free services to open-source projects such as Botmation. Their services help keep Botmation secure, distributable, up-to-date, informational and running smoothly, in no particular order:
+π§ββπ’ Shout out to all the companies providing free services to open-source projects such as Botmation. Their services help keep Botmation secure, distributable, up-to-date, informational and running smoothly, in no particular order:
- [Github](https://gituhb.com)
- [npm](https://npmjs.com)
- [TravisCI](https://travis-ci.com)
- [David](https://david-dm.org)
- [Codecov](https://codecov.io)
+ - [SonarCloud](https://sonarcloud.io)
- [Mergify](https://mergify.io/)
- [Snyk](https://snyk.io/)
- [LGTM](https://lgtm.com)
diff --git a/src/docs/install.mdx b/src/docs/install.mdx
index 12c4cdd..af99b6d 100644
--- a/src/docs/install.mdx
+++ b/src/docs/install.mdx
@@ -33,6 +33,6 @@ Then import any function, BotAction, Helper, and Instagram specific from the `bo
import { chain, goTo, screenshot } from 'botmation';
```
-As a reference, there are 12 groups of BotAction's, as of v2.0.0. Each one is linked in the sidebar under API.
+Each one is documented with code and examples in the sidebar under API.
After that, you are ready π
\ No newline at end of file
diff --git a/src/docs/overview.mdx b/src/docs/overview.mdx
index 4691a0c..bdc1aa7 100644
--- a/src/docs/overview.mdx
+++ b/src/docs/overview.mdx
@@ -2,34 +2,32 @@
title: 'Overview'
---
-Botmation is simple declarative framework for building web bots with reusable functions called BotAction's.
+Botmation is a simple declarative framework for building web bots with composable functions called BotAction's.
> βEverything should be made as simple as possible, but no simpler.β - Albert Einstein
-BotAction's are async functions that handle various tasks in the web, from specific to broad. They are *composable*, in that they can be created from higher order functions, and are easily assembled into web bots. They are like bricks, you can lay them down to build walls in any direction.
+BotAction's are async functions that handle various tasks in the web, from specific things like "click this button" to broad flows like "scrape this feed and like my friends' posts". They are *composable*. They can be assembled and created from higher order functions. They are like bricks, you can lay them down to build walls, then use the walls to build buildings, then use the buildings to build cities and so forth.
-Imagine a line of people at a coffee shop β. As the line forms, the last person who enters the shop joins the end of the line. The first person served is the first person in line.
+Imagine a line of people at a coffee shop β. As π§ π§ π§ customers enter, a π§π§π§ line forms with the last person who entered the shop at the end of the line. The first person served is the first person in.
-> Classic, first come, first served
+> First come -> First served
-Botmation bots work like that. They run the BotAction's in the declared order, from first to last.
+BotAction lines work like that. BotAction's are ran in the order declared, from first to last.
-Let's take the metaphor further, into Alice in Wonderland... come follow the white rabbit π for a second.
+But wait, there's more. Any π§ person in π§π§π§ line, can actually be a separate π§π§π§ line of people. Then any one of those people in this sub-line, can be another π§π§π§ line of people, and βΎοΈ infinitely deep π. It's all composable. A BotAction can be a specific async operation or an assembled line of other BotAction's.
-In Botmation, any one of those people in line π§, can actually be a whole other line of people π§π§π§. Then any one of those people in this embedded line, can also be a whole other line of people π§π§π§π§π§, and infinitely deep βΎοΈ. It doesn't matter, because a BotAction can be a small specific async operation or a line of other BotAction's.
+The possibilities are endless.-
-The possibilities for assembling bots are endless.
-
-> If you are new to Functional programming, but are familiar with Object-Oriented, consider this. Functional programming replaces Classes with a composable system for assembling functions together, like building blocks, to *compose* bigger functions as instances of Classes, using only the functionality required. The pattern removes Classes as a code sharing barrier and minimizes runtime overhead.
+> If you are new to Functional programming, but are familiar with Object-Oriented, consider this. Functional programming replaces Classes with a composable system for assembling functions together, like building blocks, to *compose* bigger functions as instances of classes, using only the functionality required. The pattern removes classes as a code sharing barrier and minimizes runtime overhead.
## What is a BotAction?
-A BotAction is in essence, a web bot part. They are assembled together, like building blocks, to build web bots and compose bigger more powerful BotAction's.
+BotAction's are building blocks for assembling web bots.
-A BotAction is an async function that does a task, small or big. For example, change the page URL, take a screenshot of the window, type something with a keyboard, click something with a mouse, manage your Instagram account (login, like friends' posts), run all the tasks of your bot in managing multiple online identities, etc.
+They are async functions that do something, big or small. For example, they can change the page URL, take a screenshot of the window, type a comment with a keyboard, click a button with a mouse, manage your Instagram account (login, like friends' posts), run all the tasks of your bot in managing multiple online identities, etc.
-Botmation's BotAction's are organized by type. Some are simple, while others are complex. Some are customized with higher-order functions or composed with other BotAction's. But, more on that later.
+To manage all the possibilities, Botmation's BotAction's are organized by type, listed under API in this site's navigation. It is not necessary to learn them all, but only what you need.
## Running BotAction's
@@ -146,7 +144,7 @@ const login = (username: string, password: string): BotAction =>
)
```
-It looks magical, but the strong typing keeps it all in check for us.
+It looks magical, but the strong typing keeps it all in check.
`login()` is a higher-order sync function that composes a chain of BotAction's to run in a line. This works because the first call of `chain()` returns a BotAction. The second call of chain is the async BotAction that runs the declared BotAction's with the customizing parameters from the the higher-order `login()` function.
diff --git a/src/docs/playground.mdx b/src/docs/playground.mdx
index 58c396a..913dd69 100644
--- a/src/docs/playground.mdx
+++ b/src/docs/playground.mdx
@@ -47,9 +47,10 @@ npm run playground
```bash
npm run examples/simple_objectoriented
npm run examples/simple_functional
-npm run examples/instagram
npm run examples/screenshots
npm run examples/pdf
+npm run examples/instagram
+npm run examples/linkedin
```
Have fun!
\ No newline at end of file
diff --git a/src/docs/roadmap.mdx b/src/docs/roadmap.mdx
index 5286395..0245d96 100644
--- a/src/docs/roadmap.mdx
+++ b/src/docs/roadmap.mdx
@@ -2,22 +2,28 @@
title: Roadmap
---
-## Botmation NPM Module
+## Botmation NPM package
### v2.x
- - Site specific BotAction's (everything in the bots directory) is deprecated, to be published in separate npm modules.
+ - Site specific BotAction's are deprecated from `botmation` package and to be published in separate npm packages `botmation-*`.
- New minor releases for proposed BotAction's [here](https://github.com/mrWh1te/Botmation/issues).
- New patch releases for bug & security fixes
### v3.x
- - Site specific BotAction's are deleted. See site specific npm modules for their replacements.
+ - Site specific BotAction's removed from botmation npm package
- Fully working (including tests) with latest Puppeteer major versions 4 & 5
+ - Site specific BotAction's are published in site specific npm packages
+
+### v4.x (subject to change)
+ - De-compose `page` from `BotAction` [proposal](https://github.com/mrWh1te/Botmation/issues/54#issuecomment-689843796)
+ - Create more than web bots with this composable framework like an API
## Working on now
- - Separate NPM modules, rename bots/ -> sites/, v3 prep
+ - Separate site specific npm packages
- New BotAction's proposed (see issues labeled "proposal" [here](https://github.com/mrWh1te/Botmation/issues))
-## Future
- - Stabilizing e2e testing with latest Puppeteer version
- - Major v3 release
+### Future
+ - More site specific npm packages
+
+
-*Last updated Aug 23, 2020*
+*Last updated Sep 14, 2020*
diff --git a/src/docs/sites/instagram.mdx b/src/docs/sites/instagram.mdx
index 3cb3310..a621510 100644
--- a/src/docs/sites/instagram.mdx
+++ b/src/docs/sites/instagram.mdx
@@ -12,7 +12,11 @@ The current set of BotAction's are for the initial login flow into Instagram's w
They can all be imported from the npm `botmation` module, like every other function in the API.
-## login() BotAction
+## Auth
+
+These BotAction's facilitate the login process of Instagram's web app.
+
+### login()
```typescript
const login = ({username, password}: {username: string, password: string}): BotAction =>
chain(
@@ -33,7 +37,7 @@ This BotAction is a composition that uses [errors()()](/api/errors#errors) to wr
The composition is declarative, therefore needs little explaining. `login()` is a higher-order function that takes a typed object for the `username` and `password`.
-## isGuest ConditionalBotAction
+### isGuest
```typescript
const isGuest: ConditionalBotAction =
indexedDBStore('redux', 'paths')(
@@ -43,7 +47,7 @@ const isGuest: ConditionalBotAction =
```
This BotAction is a composition that uses [indexedDBStore()()](/api/indexed-db#indexeddbstore) to assemble BotAction's with the IndexedDB Database name and Store name [injected](/advanced/injects). This simply grabs the value for the `users.viewerId` key then [maps](/api/pipe#map) it to the corresponding boolean value.
-## isLoggedIn ConditionalBotAction
+### isLoggedIn
```typescript
const isLoggedIn: ConditionalBotAction =
indexedDBStore('redux', 'paths')(
@@ -53,7 +57,7 @@ const isLoggedIn: ConditionalBotAction =
```
This BotAction is a composition that uses [indexedDBStore()()](/api/indexed-db#indexeddbstore) to assemble BotAction's with the IndexedDB Database name and Store name [injected](/advanced/injects). This simply grabs the value for the `users.viewerId` key then [maps](/api/pipe#map) it to the corresponding boolean value.
-## isTurnOnNotificationsModalActive ConditionalBotAction
+### isTurnOnNotificationsModalActive
```typescript
const isTurnOnNotificationsModalActive: ConditionalBotAction = async(page) => {
const modalHeader = await page.$(MAIN_MODAL_HEADER_SELECTOR)
@@ -64,7 +68,7 @@ const isTurnOnNotificationsModalActive: ConditionalBotAction = async(page) => {
```
Sometimes after login, Instagram prompts the User with a modal about turning on notifications. This ConditionalBotAction returns `true` if that modal is open.
-## closeTurnOnNotificationsModal BotAction
+### closeTurnOnNotificationsModal
```typescript
const closeTurnOnNotificationsModal: BotAction = async (page) => {
// click button with text "Not Now" inside the dialog
@@ -76,6 +80,22 @@ const closeTurnOnNotificationsModal: BotAction = async (page) => {
```
This BotAction will close the "Turn on Notifications" modal. Please use [givenThat()()](/api/utilities#given-that) with [isTurnOnNotificationsModalActive](/sites/instagram#isturnonnotificationsmodalactive-conditionalbotaction) to run this BotAction only if that modal is open.
+## Selectors
+
+Helpful HTML element selectors in the Instagram web app, can be imported directly from the main `botmation` package.
+
+These are the following:
+```typescript
+// Auth - Login Page
+const FORM_AUTH_USERNAME_INPUT_SELECTOR = 'html body section main form input[name="username"]'
+const FORM_AUTH_PASSWORD_INPUT_SELECTOR = 'html body section main form input[name="password"]'
+const FORM_AUTH_SUBMIT_BUTTON_SELECTOR = 'html body section main form button[type="submit"]'
+
+// Modals - Home Page
+const MAIN_MODAL_SELECTOR = 'html body div[role="dialog"]'
+const MAIN_MODAL_HEADER_SELECTOR = 'html body div[role="dialog"] h2'
+```
+
## Helpers
### getInstagramBaseUrl()
@@ -90,4 +110,4 @@ Returns Instagram's web app's base URL without a trailing slash, ie: `https://ww
const getInstagramLoginUrl = () =>
getInstagramBaseUrl() + INSTAGRAM_URL_EXT_LOGIN + '/'
```
-Returns Instagram's web app's URL for its login page
+Returns Instagram's web app's URL for its login page
\ No newline at end of file
diff --git a/src/docs/sites/linkedin.mdx b/src/docs/sites/linkedin.mdx
new file mode 100644
index 0000000..392109b
--- /dev/null
+++ b/src/docs/sites/linkedin.mdx
@@ -0,0 +1,264 @@
+---
+title: LinkedIn
+---
+
+These BotAction's focus on logging into LinkedIn with an account then processing the account's Feed. From logging in, to liking feed posts, it's a small collection to help get started.
+
+> These functions are yet to be published in their own npm package, but for now, you're welcome to copy and paste them into your own code from this page.
+
+## Overview
+
+The current set of BotAction's are for logging in and liking posts in the feed. Here is a working [example](https://github.com/mrWh1te/Botmation/blob/master/src/examples/linkedin.ts) of them in use.
+
+## Navigation
+Simple functions for navigating to various parts of the LinkedIn web app.
+
+### goHome
+```typescript
+const goHome: BotAction =
+ goTo('https://www.linkedin.com/', {waitUntil: 'domcontentloaded'})
+```
+
+### goToFeed
+```typescript
+const goToFeed: BotAction =
+ goTo('https://www.linkedin.com/feed/', {waitUntil: 'domcontentloaded'})
+```
+
+### goToMessaging
+```typescript
+const goToMessaging: BotAction =
+ goTo('https://www.linkedin.com/messaging/', {waitUntil: 'domcontentloaded'})
+```
+
+### goToNotifications
+```typescript
+const goToNotifications: BotAction =
+ goTo('https://www.linkedin.com/notifications/', {waitUntil: 'domcontentloaded'})
+```
+
+## Auth
+
+These BotAction's focus on logging into the LinkedIn web app.
+
+### login()
+```typescript
+const login = (emailOrPhone: string, password: string): BotAction =>
+ chain(
+ errors('LinkedIn login()')(
+ goTo('https://www.linkedin.com/login'),
+ click('form input[id="username"]'),
+ type(emailOrPhone),
+ click('form input[id="password"]'),
+ type(password),
+ click('form button[type="submit"]'),
+ waitForNavigation,
+ log('LinkedIn Login Complete')
+ )
+ )
+```
+This BotAction is a composition that uses [errors()()](/api/errors#errors) to wrap the main assembled BotAction's, in case something goes wrong here (ie a selector is updated so `click()` fails), dev's will have an easier time debugging when errors are thrown here.
+
+The composition is declarative, therefore needs little explaining. `login()` is a higher-order function that takes a `emailOrPhone` and `password` strings to automatically perform the login flow in the web app.
+
+### isGuest
+```typescript
+const isGuest: ConditionalBotAction = pipe()(
+ // data feature for user notifications
+ getLocalStorageItem('voyager-web:badges'),
+ map(value => value === null) // Local Storage returns null if not found
+)
+```
+After some investigating, it appears that Local Storage is used to maintain the state of the application's features, where each key is a major app feature. The particular key here references, what appears to be, a "Notifications" feature which is global to the app (in the Layout) and belongs only to Users.
+
+### isLoggedIn
+```typescript
+const isLoggedIn: ConditionalBotAction = pipe()(
+ // data feature for user notifications
+ getLocalStorageItem('voyager-web:badges'),
+ map(value => typeof value === 'string') // Local Storage returns string value if found
+)
+```
+After some investigating, it appears that Local Storage is used to maintain the state of the application's features, where each key is a major app feature. The particular key here references, what appears to be, a "Notifications" feature which is global to the app (in the Layout) and belongs only to Users.
+
+## Feed
+These BotAction's focus on operating in the main Feed of LinkedIn's web app.
+
+### Scrape Feed Post
+This BotAction takes a specific post html attribute `data-id` value to scrape it with the provided HTML parser.
+
+```typescript
+const scrapeFeedPost = (postDataId: string): BotAction =>
+ $('.application-outlet .feed-outlet [role="main"] [data-id="'+ postDataId + '"]')
+```
+
+### If Post Not Loaded Cause Loading Then Scrape
+Linkedin's feed lazily loads the content of its offscreen posts. It uses div containers, each with their own `data-id` attribute, as placeholders for the content to be loaded in. This BotAction tests to see if a particular scraped post was fully loaded and if not, it causes the app to load it by scrolling to it. Then it scrapes the fully loaded container.
+
+```typescript
+const ifPostNotLoadedCauseLoadingThenScrape = (post: CheerioStatic): BotAction =>
+ pipe(post)(
+ errors('LinkedIn causeLazyLoadingThenScrapePost()')(
+ pipeCase(postHasntFullyLoadedYet)(
+ scrollTo('.application-outlet .feed-outlet [role="main"] [data-id="'+ post('[data-id]').attr('data-id') + '"]'),
+ scrapeFeedPost(post('[data-id]').attr('data-id') + '')
+ ),
+ map(casesSignalToPipeValue)
+ )
+ )
+```
+
+### Like User Post
+It takes a scraped feed post, to grab identifying information for the "Like" button. It will throw and catch an error if the "Like" button was already liked, given how Puppeteer's `page.click()` handles elements not found. A "liked" button has a slightly different selector.
+
+```typescript
+const likeUserPost = (post: CheerioStatic): BotAction =>
+ errors('LinkedIn like() - Could not Like Post: Either already Liked or button not found')(
+ click( 'div[data-id="' + post('div[data-id]').attr('data-id') + '"] button[aria-label="Like ' + post(feedPostAuthorSelector).text() + 'βs post"]')
+ )
+```
+
+### Like User Posts From
+It takes a spread array of strings of the exact names of people, to like their posts in the feed. The function itself goes beyond the required scope, but to serve as a decent starting point for more complex feed interactions.
+
+```typescript
+const likeUserPostsFrom = (...peopleNames: string[]): BotAction =>
+ pipe()(
+ scrapeFeedPosts,
+ forAll()(
+ post => pipe(post)(
+ ifPostNotLoadedCauseLoadingThenScrape(post),
+ switchPipe()(
+ pipeCase(postIsPromotion)(
+ map((promotionPost: CheerioStatic) => promotionPost('[data-id]').attr('data-id')),
+ log('Promoted Content')
+ ),
+ abort(),
+ pipeCase(postIsJobPostings)(
+ map((jobPostingsPost: CheerioStatic) => jobPostingsPost('[data-id]').attr('data-id')),
+ log('Job Postings')
+ ),
+ abort(),
+ pipeCase(postIsUserInteraction)(
+ map((userInteractionPost: CheerioStatic) => userInteractionPost('[data-id]').attr('data-id')),
+ log(`Followed User's Interaction (ie like, comment, etc)`)
+ ),
+ abort(),
+ pipeCase(postIsUserPost)(
+ pipeCase(postIsAuthoredByAPerson(...peopleNames))(
+ likeUserPost(post),
+ log('User Article "liked"')
+ ),
+ emptyPipe,
+ log('User Article')
+ ),
+ abort(),
+ // default case
+ pipe()(
+ map((unhandledPost: CheerioStatic) => unhandledPost('[data-id]').text()),
+ log('Unhandled Post Case')
+ )
+ )
+ )
+ )
+ )
+```
+
+## Messaging
+These BotAction's focus on operating in the main Messaging area of LinkedIn's web app.
+
+### toggleMessagingOverlay
+```typescript
+const toggleMessagingOverlay: BotAction =
+ click(messagingOverlayHeaderSelector)
+```
+
+By default, when someone logs into the LinkedIn web app, the "Messaging" center in the bottom-right is open, covering part of the web app. This BotAction will toggle that "Messaging" overlay open and close.
+
+## Selectors
+
+Helpful HTML element selectors in the LinkedIn web app:
+```typescript
+// Selectors for Messaging Overlay
+export const messagingOverlayHeaderSelector = 'header.msg-overlay-bubble-header'
+
+// Selectors for the main News Feeds
+export const feedPostsSelector = '.application-outlet .feed-outlet [role="main"] div[data-id]'
+export const feedPostAuthorSelector = '.feed-shared-actor__title'
+```
+
+## Helpers
+
+These functions are not BotAction's, but useful in creating a web bot for Linkedin.
+
+### postIsUserPost()
+This function is a `ConditionalCallback` that fits with `pipeCase()` and `pipeCases()`. It tests the provided `CheerioStatic` function, a "post" from the feed, on whether or not it fits the criteria for a User post, the common published posts.
+
+```typescript
+const postIsUserPost: ConditionalCallback = (post: CheerioStatic) => {
+ const sharedActorFeedSupplementaryInfo = post('.feed-shared-actor__supplementary-actor-info').text().trim().toLowerCase()
+
+ return sharedActorFeedSupplementaryInfo.includes('1st') || sharedActorFeedSupplementaryInfo.includes('following')
+}
+```
+It checks for a particular part of the HTML where the connection meta information is displayed, ie to what degree of connection are you, or if not connected, are you following this User.
+
+### postIsAuthoredByAPerson()
+This higher-order function returns a `ConditionalCallback` that fits with `pipeCase()` and `pipeCases()`. It tests the provided `CheerioStatic` function, a "post" from the feed, on whether or not it was authored by a particular person.
+
+```typescript
+const postIsAuthoredByAPerson = (...peopleNames: string[]): ConditionalCallback => (post: CheerioStatic) =>
+ peopleNames.some(name => name.toLowerCase() === post(feedPostAuthorSelector).text().toLowerCase())
+```
+
+### postIsPromotion()
+This function is a `ConditionalCallback` that fits with `pipeCase()` and `pipeCases()`. It tests the provided `CheerioStatic` function, a "post" from the feed, on whether or not it fits the criteria for a Promoted post, aka an advertisement.
+
+```typescript
+const postIsPromotion: ConditionalCallback = (post: CheerioStatic) =>
+ post('.feed-shared-actor__sub-description').text().trim().toLowerCase().includes('promoted')
+```
+
+### postIsJobPostings()
+This function is a `ConditionalCallback` that fits with `pipeCase()` and `pipeCases()`. It tests the provided `CheerioStatic` function, a "post" from the feed, on whether or not it fits the criteria for a Job Postings post.
+
+```typescript
+const postIsJobPostings: ConditionalCallback = (post: CheerioStatic) => {
+ const dataId = post('[data-id]').attr('data-id')
+
+ if(!dataId) return false
+
+ const dataIdParts = dataId.split(':')
+
+ return dataIdParts.length >= 5 &&
+ dataIdParts[2] === 'aggregate' &&
+ dataIdParts.slice(3).some(part => part === 'jobPosting')
+}
+```
+
+### postIsUserInteraction()
+This function is a `ConditionalCallback` that fits with `pipeCase()` and `pipeCases()`. It tests the provided `CheerioStatic` function, a "post" from the feed, on whether or not it fits the criteria for an User Interaction post.
+
+The LinkedIn web app sometimes presents posts to highlight an User you're connected with, or following, has recently "reacted" to a post in the feed. This includes "liking", "loving", "celebrating", "commenting on" posts.
+
+```typescript
+const postIsUserInteraction: ConditionalCallback = (post: CheerioStatic) => {
+ const feedPostSiblingText = post('h2.visually-hidden:contains("Feed post") + div span').text().trim().toLowerCase()
+
+ return feedPostSiblingText.includes('likes this') ||
+ feedPostSiblingText.includes('loves this') ||
+ feedPostSiblingText.includes('celebrates this') ||
+ feedPostSiblingText.includes('commented on this')
+}
+```
+
+### postHasntFullyLoadedYet()
+This function is a `ConditionalCallback` that fits with `pipeCase()` and `pipeCases()`. It tests the provided `CheerioStatic` function, a "post" from the feed, on whether or not it fits the criteria for being a fully loaded post.
+
+The LinkedIn web app lazily loads offscreen content to expediate rendering performance of the feed. In doing such, it leaves container div's that represent posts, that may get scraped, but lack all the important information to make the informed decision. Therefore, this ConditionalCallback can be used to run code for loading a post fully, in case it has not.
+
+```typescript
+const postHasntFullyLoadedYet: ConditionalCallback = (post: CheerioStatic) => {
+ return post('[data-id]').text().trim() === ''
+}
+```
\ No newline at end of file
](https://david-dm.org/mrWh1te/Botmation)
-[Botmation](https://github.com/mrWh1te/Botmation) is a simple TypeScript framework for building web bots in a functional, composable and declarative way.
+[Botmation](https://github.com/mrWh1te/Botmation) is a simple, functional, TypeScript framework for building web bots in a composable and declarative way.
-It's built on top of [Puppeteer](https://github.com/puppeteer/puppeteer), a NodeJS library which provides a high-level API to control Chrome or Chromium over the [DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/).
+It's built with [Puppeteer](https://github.com/puppeteer/puppeteer), a NodeJS library that provides a high-level API to control Chrome or Chromium over the [DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/).
-With [Puppeteer](https://github.com/puppeteer/puppeteer), you build web bots that automate your social media, take screenshots or generate PDF's of websites, surf & scrape the web and much more. ππ
+With [Puppeteer](https://github.com/puppeteer/puppeteer), it's few steps away from building web bots that automate your social media, take screenshots & generate PDF's of websites, surf while scraping the web and much more. ππ
-With [Botmation](https://github.com/mrWh1te/Botmation), you get a simple, composable, scalable, declarative pattern to build web bots with Puppeteer. It reduces code reusing barriers and maximizes web bots' code *readability*, *maintainability* and *testability*.
+With [Botmation](https://github.com/mrWh1te/Botmation), you get a simple, scalable, declarative pattern to build Puppeteer web bots. It reduces code reusing barriers with *composition* and maximizes your web bots' code *readability*, *maintainability* and *testability*.
## Features
- π€ Learn only what you need
- π£οΈ Simple pattern to follow
-- β‘οΈ Write readable, testable & composable code
- π₯ 100% Library Test Coverage
- π Get started with Functional Programming
+- β‘οΈ Write readable, testable & composable code
- π’ Use higher order functions
- 𧱠Assemble complex async functionality
- and much more π₯
diff --git a/src/config/sidebar.yml b/src/config/sidebar.yml
index 01cae72..4c9bcaa 100644
--- a/src/config/sidebar.yml
+++ b/src/config/sidebar.yml
@@ -13,8 +13,17 @@
- label: 'Install'
link: '/install'
+- label: 'Sites'
+ items:
+ - label: 'Instagram'
+ link: '/sites/instagram'
+ - label: 'LinkedIn'
+ link: '/sites/linkedin'
+
- label: 'Advanced'
items:
+ - label: 'Aborting'
+ link: '/advanced/aborting'
- label: 'Piping'
link: '/advanced/piping'
- label: 'Injects'
@@ -30,13 +39,10 @@
- label: 'Tutorial'
link: '/advanced/tutorial'
-- label: 'Sites'
- items:
- - label: 'Instagram'
- link: '/sites/instagram'
-
- label: 'API'
items:
+ - label: 'Abort'
+ link: '/api/abort'
- label: 'Assembly Lines'
link: '/api/assembly-lines'
- label: 'Console'
@@ -59,6 +65,8 @@
link: '/api/navigation'
- label: 'Pipe'
link: '/api/pipe'
+ - label: 'Scrapers'
+ link: '/api/scrapers'
- label: 'Utilities'
link: '/api/utilities'
diff --git a/src/docs/advanced/aborting.mdx b/src/docs/advanced/aborting.mdx
new file mode 100644
index 0000000..6a1220d
--- /dev/null
+++ b/src/docs/advanced/aborting.mdx
@@ -0,0 +1,56 @@
+---
+title: Aborting
+---
+
+## AbortLine Signal
+
+Sometimes it's necessary to abort an assembly line(s) of BotAction's. Perhaps something is removed from a web page or a condition wasn't met that subsequent BotAction's rely on. When this happens, a BotAction returns an `AbortLineSignal`.
+
+```typescript
+type AbortLineSignal = {
+ brand: 'Abort_Signal',
+ assembledLines: number,
+ pipeValue?: PipeValue
+}
+```
+
+AbortLine signals are safely created with the [createAbortLineSignal()](/api/abort#createabortlinesignal) helper. It's a simple JSON object with potentially two to three properties.
+
+### Assembled Lines Count
+
+An AbortLine signal's important property is called `assembledLines`. It informs the assembler how many lines of assembly to break. One is one line, two is two, three is three and so forth.
+
+The default count of `assembledLines` is one for both the [abort()](/api/abort#abort) BotAction and the [createAbortLineSignal()](/api/abort#createabortlinesignal) helper. It can be any positive number, including zero.
+
+## Infinity AbortLine Signal
+It's possible to abort all lines of assembly without knowing how many there are by returning an Infinity AbortLine signal.
+
+Infinity AbortLine signals have `assembledLines` set to `0`. When an assembler processes an Infinity AbortLine signal, it aborts what it's doing and returns the same signal without change.
+
+### Optional Pipe Value
+`AbortLineSignal` has an optional param called `pipeValue` that carries a `PipeValue` back to the final aborted assembler for return.
+
+> The optional `pipeValue` param does not get returned by a [Chain](/api/assembly-lines#chain), when it's the final assembler to process an AbortLine signal.
+
+## Process AbortLine Signal
+
+Almost every assembler processes an AbortLine signal with the [processAbortLineSignal()](/api/abort#processabortlinesignal) helper. It supports all main use-cases of `assembledLines`: 0, 1, and 2+.
+
+Here is a table for the default assemblers' aborting behavior:
+
+| assembledLines | effect |
+| - | - |
+| 0 | Infinity AbortLine Signal aborts every BotAction and returns itself without change on each abort |
+| 1 | Default AbortLine Signal aborts only one line of assembled BotActions and returns the abortLineSignal.pipeValue |
+| 2+ | Level specific AbortLine Signal aborts one line of assembled BotActions at a time while reducing its assembledLines count to 1 for final abort |
+
+Each assembler will detect the value returned by the resolved BotAction for an AbortLine signal. If it is one, it processes it to the effect listed above.
+
+This is the default aborting behavior. There are exceptions for some BotAction's such as the Utilities, Switch Pipe, and Pipe Case(s), which implement their own unique aborting behavior.
+
+## Unique Aborting behavior
+[Switch Pipe](/api/assembly-lines#switch-pipe) implements its own AbortLine signal behavior in order to support its own unique functionality. When it hasn't received a [CasesSignal](/advanced/conditionals/#cases-signal) with its condition passed, it ignores default AbortLine signals. That's because [abort()](/api/abort#abort) is treated like `break;` from the traditional `switch(){}` code block, inside a [Switch Pipe](/api/assembly-lines#switch-pipe).
+
+Some BotAction's, like Utilities, have unique AbortLine signal behavior for aborting parts of the unique assemblers. For example, a [For All](/api/utilities#for-all) BotAction's loop iteration (runs an assembled line of BotActions) can abort out of its loop iteration with a default AbortLine signal or abort the entire loop with two or more `assembledLines`.
+
+BotAction's with unique aborting behavior have their aborting functionality explained in their documentation, listed under API in the sidebar.
\ No newline at end of file
diff --git a/src/docs/advanced/conditionals.mdx b/src/docs/advanced/conditionals.mdx
index 506ed72..02b21a7 100644
--- a/src/docs/advanced/conditionals.mdx
+++ b/src/docs/advanced/conditionals.mdx
@@ -2,7 +2,7 @@
title: Conditionals
---
-Botmation has a special type of BotAction called Utilities. These allow you to build lines of BotAction's but with unique behavior.
+Botmation has a special BotAction's that assemble BotAction's that run *if* a condition is met.
## Functional if statement
@@ -30,10 +30,51 @@ await chain(
> All [Utility BotAction's](/api/utilities) run assembled BotAction's in a Pipe
-## Conditional BotAction
+### Conditional BotAction
The first call `givenThat()` accepts a special kind of BotAction called ConditionalBotAction. ConditionalBotAction are like BotAction's, except they *always* return a boolean value.
-`isGuest` and `isLoggedIn` are ConditionalBotAction's for Instagram's web app. They are not intended to change the state of the `page` but check the `page` for a condition, then report on it. This way, the Instagram `login()` BotAction is ran only if needed. Check the [Instagram example](https://github.com/mrWh1te/Botmation/tree/master/src/examples/instagram.ts) for more like saving & loading cookies, to skip subsequent logins.
+`isGuest` and `isLoggedIn` are ConditionalBotAction's for Instagram's web app. They are not intended to change the state of the `page` but check the `page` for a condition, then report on it. This way, the Instagram `login()` BotAction is ran only if needed. Check the [Instagram example](https://github.com/mrWh1te/Botmation/tree/master/src/examples/instagram.ts) for more including saving & loading cookies, to skip subsequent logins.
-For a more technical explanation, `givenThat()()` returns a customized BotAction for the whole flow. When you resolve the returned BotAction, it runs through a series of steps. First, it resolves the `ConditionalBotAction` in a Pipe for the value it returns. Then runs the assembled BotAction's if the value was `true`.
\ No newline at end of file
+For a more technical explanation, `givenThat()()` returns a customized BotAction for the whole flow. When you resolve the returned BotAction, it runs through a series of steps. First, it resolves the `ConditionalBotAction` in a Pipe for the value it returns. Then runs the assembled BotAction's if the value was `true`. However, it does not return the final pipe value to be compatible with Chain's.
+
+## Conditionals evaluating Pipe Value
+
+There are other BotAction's like `givenThat()()` for running a line of BotActions if a condition passes. They evaluate the equality of values supplied against the Pipe object value. They are called [pipeCase()()](/api/pipe#pipe-case) && [pipeCases()()](/api/pipe#pipe-cases) which run the assembled BotAction's of their second call, if their condition in the first call passes.
+
+### Conditional Callback
+
+Instead of handling a ConditionalBotAction in their first call, they expect a Pipe value, which can be almost anything, including a synchronous function. When a synchronous function is provided, it's called as a callback with the Pipe value as its only parameter. These functions are called ConditionalCallback's. Here is the interface:
+
+```typescript
+interface ConditionalCallback