Skip to content

Commit

Permalink
pipeCases()() and LinkedIn likeArticlesFrom() (#75)
Browse files Browse the repository at this point in the history
* pipeCases()() and likeAllFrom() #61 #55 
- pipeCases()() like pipeCase()() except ALL cases must match for it to
  return a MatchesSignal with matches and pipeValue
- LinkedIn likeAllFrom(...peopleNames) uses switchPipe() and
  pipeCase()() to test post for type
* Renamed MatchesSignal -> CasesSignal + new prop `conditionPass`
- added `conditionPass` property to CasesSignal so it can return
  matching cases while still presenting itself as not ran since
  the overall condition failed (did not pass)
* pipeCases()() unit-tests
* pipeCases()() & pipeCase()() abortlinesignal tweak
- similar to switchPipe handling abort line signals, it takes multiple
  abort assembledLines to fully abort out of pipeCase(s) cz its 1 to
  abort the line assembled and a second to abort the CasesSignal return
  versus an AbortLineSignal return
- if pipeValue returned, it was abort(2)'s pipeValue
* pipeAbort(), house cleaning, conditionalCallbacks
* pipeCase(s) specifies type CaseValues for callback of testable value
* pipeAbort() supports conditional callback or a pipevalue to test
  against pipevalue and if true, then return AbortLineSignal(1,
  pipeValue?)
* linkedin example using likeArticlesFrom()
- going beyond function scope but to show a more detailed example on how
  a feed could be processed with switchPipe()()
* introduced new botaction scrollTo() which includes a static wait of
  2.5 seconds for the scroll to finish (add more if you need more)
- this update includes the ability to let the LinkedIn app lazily load
  each post if it hasnt been loaded yet, with a special check
- complete but not recently tested likeArticle() functionality since using
  fictional character names
* helpers/navigation unit-testing
* helpers/cases unit-test
* navigation/actions unit-test scrollTo()
* evaluate() botaction with integration test
  • Loading branch information
mrWh1te authored Sep 9, 2020
1 parent 472efae commit e968078
Show file tree
Hide file tree
Showing 24 changed files with 757 additions and 321 deletions.
16 changes: 15 additions & 1 deletion src/botmation/actions/abort.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ import { BotAction } from "../interfaces"
import { AbortLineSignal } from '../types'
import { PipeValue } from "../types/pipe-value"
import { createAbortLineSignal } from "../helpers/abort"
import { pipeCase } from "./pipe"
import { CasesSignal, CaseValue } from "../types/cases"

/**
* BotAction to return an AbortLineSignal to be processed by an assembler for an effect of aborting assembled lines (including parent(s) if specified a number greater than 1 or 0 for all)
Expand All @@ -11,4 +13,16 @@ import { createAbortLineSignal } from "../helpers/abort"
or pass it along to the next assembler if the assembledLines is greater than 1 as the pipeValue is returned by the final aborted assembler
*/
export const abort = (assembledLines = 1, pipeValue?: PipeValue): BotAction<AbortLineSignal> =>
async() => createAbortLineSignal(assembledLines, pipeValue)
async() => createAbortLineSignal(assembledLines, pipeValue)

/**
* Return an AbortLineSignal of 1 assembledLine if the value provided equals the pipe value or the value provided is a callback function when given the pipe value returns true
* cb gets the pipeValue. If cb returns true, then abort the pipe line
* @param value the value to test against the pipeValue for equality unless function then call function with value and if function returns truthy then Abort
* @param abortPipeValue the pipeValue of the AbortLineSignal returned
* @param assembledLines the assembledLines of the AbortLineSignal returned
*/
export const abortPipe = (value: CaseValue, abortPipeValue: PipeValue = undefined, assembledLines: number = 1): BotAction<AbortLineSignal|PipeValue|CasesSignal> =>
pipeCase(value)(
abort(assembledLines + 2, abortPipeValue)
) // returns AbortLineSignal(1, abortPipeValue?) if value(pipeValue) is truthy || value === pipeValue
24 changes: 4 additions & 20 deletions src/botmation/actions/assembly-lines.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,7 @@ import {
import { PipeValue } from "../types/pipe-value"
import { AbortLineSignal, isAbortLineSignal } from "../types/abort-line-signal"
import { processAbortLineSignal } from "../helpers/abort"
import { isMatchesSignal, MatchesSignal } from "botmation/types/matches-signal"
import { hasAtLeastOneMatch } from "botmation/helpers/matches"
import { isCasesSignal, CasesSignal } from "../types/cases"

/**
* @description chain() BotAction for running a chain of BotAction's safely and optimized
Expand Down Expand Up @@ -121,29 +120,14 @@ export const pipe =
* @param toPipe BotAction to resolve and inject as a wrapped Pipe object in EACH assembled BotAction
*/
export const switchPipe =
(toPipe?: BotAction | Exclude<PipeValue, Function>) =>
(...actions: BotAction<PipeValue|AbortLineSignal|MatchesSignal|void>[]): BotAction<any[]|AbortLineSignal|PipeValue> =>
(toPipe?: PipeValue) =>
(...actions: BotAction<PipeValue|AbortLineSignal|CasesSignal|void>[]): BotAction<any[]|AbortLineSignal|PipeValue> =>
async(page, ...injects) => {
// fallback is injects pipe value
if (!toPipe) {
toPipe = getInjectsPipeValue(injects)
}

// if function, it's an async BotAction function
// resolve it to Pipe the resolved value
if (typeof toPipe === 'function') {
if(injectsHavePipe(injects)) {
toPipe = await toPipe(page, ...injects)
} else {
// simulate pipe
toPipe = await toPipe(page, ...injects, createEmptyPipe())
}

if (isAbortLineSignal(toPipe)) {
return processAbortLineSignal(toPipe)
}
}

// remove pipe from injects if there is one to set the one for all actions
if (injectsHavePipe(injects)) {
injects = injects.slice(0, injects.length - 1)
Expand All @@ -162,7 +146,7 @@ export const switchPipe =
// resolvedActionResult can be of 3 things
// 1. MatchesSignal 2. AbortLineSignal 3. PipeValue
// switchPipe will return (if not aborted) an array of all the resolved results of each BotAction assembled in the switchPipe()() 2nd call
if (isMatchesSignal(resolvedActionResult) && hasAtLeastOneMatch(resolvedActionResult)) {
if (isCasesSignal(resolvedActionResult) && resolvedActionResult.conditionPass) {
hasAtLeastOneCaseMatch = true
actionsResults.push(resolvedActionResult)
} else if (isAbortLineSignal(resolvedActionResult)) {
Expand Down
14 changes: 13 additions & 1 deletion src/botmation/actions/navigation.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
import { DirectNavigationOptions, NavigationOptions } from 'puppeteer'

import { BotAction } from '../interfaces/bot-actions'
import { enrichGoToPageOptions, sleep } from '../helpers/navigation'
import { enrichGoToPageOptions, sleep, scrollToElement } from '../helpers/navigation'
import { chain } from './assembly-lines'
import { evaluate } from './scrapers'

/**
* @description Go to url provided in the current page
Expand Down Expand Up @@ -63,3 +65,13 @@ export const waitForNavigation: BotAction = async(page) => {
export const wait = (milliseconds: number): BotAction => async() => {
await sleep(milliseconds)
}

/**
*
* @param htmlSelector
*/
export const scrollTo = (htmlSelector: string): BotAction =>
chain(
evaluate(scrollToElement, htmlSelector),
wait(2500) // wait for scroll to complete
)
81 changes: 69 additions & 12 deletions src/botmation/actions/pipe.ts
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
import { BotAction } from "../interfaces"
import { PipeValue } from "../types/pipe-value"
import { getInjectsPipeValue, injectsHavePipe } from "../helpers/pipe"
import { MatchesSignal } from "../types/matches-signal"
import { CasesSignal, CaseValue } from "../types/cases"
import { AbortLineSignal, Dictionary, isAbortLineSignal } from "../types"
import { pipe } from "./assembly-lines"
import { createMatchesSignal } from "../helpers/matches"
import { createCasesSignal } from "../helpers/cases"
import { processAbortLineSignal } from "../helpers/abort"

//
// BotAction's Focused on Piping
Expand All @@ -15,11 +16,11 @@ import { createMatchesSignal } from "../helpers/matches"
* @description Mapper function for Mapping Piped Values to whatever you want through a function
* If the Pipe is missing from the `injects`, undefined will be past into the mapFunction, like an empty Pipe
* @param mapFunction pure function to change the piped value to something else
* Rename pipeMap ?
*/
export const map = <R extends PipeValue = PipeValue>(mapFunction: (pipedValue: any) => R): BotAction<R> =>
export const map = <R = any>(mapFunction: (pipedValue: any) => R): BotAction<R> =>
async (page, ...injects) =>
mapFunction(getInjectsPipeValue(injects))


/**
* @description Sets the Pipe's value for the next BotAction
Expand All @@ -39,12 +40,13 @@ export const emptyPipe: BotAction = async () => undefined
* @return AbortLineSignal|MatchesSignal
* If no matches are found or matches are found, a MatchesSignal is returned
* It is determined if signal has matches by using hasAtLeastOneMatch() helper
* If assembled BotAction aborts(1), it still returns a MatchesSignal with the matches
* If assembled BotAction aborts(2+), it returns a processed AbortLineSignal
* If assembled BotAction aborts(1), it breaks line & returns a MatchesSignal with the matches
* If assembled BotAction aborts(2), it breaks line & returns AbortLineSignal.pipeValue
* If assembled BotAction aborts(3+), it returns AbortLineSignal(2-)
*/
export const pipeCase =
(...valuesToTest: PipeValue[]) =>
(...actions: BotAction<PipeValue|AbortLineSignal|void>[]): BotAction<AbortLineSignal|MatchesSignal> =>
(...valuesToTest: CaseValue[]) =>
(...actions: BotAction<PipeValue|AbortLineSignal|void>[]): BotAction<PipeValue|AbortLineSignal|CasesSignal> =>
async(page, ...injects) => {
// if any of the values matches the injected pipe object value
// then run the assembled actions
Expand All @@ -69,14 +71,69 @@ export const pipeCase =
const returnValue:PipeValue|AbortLineSignal = await pipe()(...actions)(page, ...injects)

if (isAbortLineSignal(returnValue)) {
return returnValue // processed by pipe()
return processAbortLineSignal(returnValue)
} else {
// signal that a case matched
return createMatchesSignal(matches, returnValue)
return createCasesSignal(matches, true, returnValue)
}
} else {
return createCasesSignal(matches, false, pipeObjectValue) // pass through original pipe object as CasesSignal.pipeValue
}
}

// no pipe (nothing to test) or the test resulted in no matches
return createMatchesSignal() // empty matches signal (no matches)
}
return createCasesSignal() // empty matches signal (no matches)
}

/**
* runs assembled actions ONLY if ALL cases pass otherwise it breaks the case checking and immediately returns an empty MatchesSignal
* it's like if (case && case && case ...)
* Same AbortLineSignal behavior as pipeCase()()
* @param valuesToTest
*/
export const pipeCases =
(...valuesToTest: CaseValue[]) =>
(...actions: BotAction<PipeValue|AbortLineSignal|void>[]): BotAction<PipeValue|AbortLineSignal|CasesSignal> =>
async(page, ...injects) => {
// if any of the values matches the injected pipe object value
// then run the assembled actions
const matches: Dictionary = {}

if (injectsHavePipe(injects)) {
const pipeObjectValue = getInjectsPipeValue(injects)

for (const [i, value] of valuesToTest.entries()) {
if (typeof value === 'function') {
if (value(pipeObjectValue)) {
matches[i] = value
} else {
break
}
} else {
if (value === pipeObjectValue) {
matches[i] = value
} else {
break
}
}
}

// only run assembled BotAction's if ALL cases pass
if (Object.keys(matches).length === valuesToTest.length) {
const returnValue:PipeValue|AbortLineSignal = await pipe()(...actions)(page, ...injects)

if (isAbortLineSignal(returnValue)) {
return processAbortLineSignal(returnValue) // processed by pipe() to abort the line
// processed a 2nd time to abort the returning CasesSignal feature
} else {
// signal that All cases matched
return createCasesSignal(matches, true, returnValue)
}
} else {
return createCasesSignal(matches, false, pipeObjectValue) // pass through original pipe object as CasesSignal.pipeValue
}
}

return createCasesSignal(matches) // partial matches signal with conditionPass false
}

8 changes: 8 additions & 0 deletions src/botmation/actions/scrapers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
* BotAction's dedicated to scraping web pages
*/

import { EvaluateFn } from 'puppeteer'
import * as cheerio from 'cheerio'

import { BotAction, ScraperBotAction } from '../interfaces/bot-actions'
Expand Down Expand Up @@ -78,3 +79,10 @@ export const $$ = <R = CheerioStatic[]>(htmlSelector: string, higherOrderHTMLPar
return cheerioEls as any as R
}

/**
* Evaluate functions inside the `page` context
* @param functionToEvaluate
* @param functionParams
*/
export const evaluate = (functionToEvaluate: EvaluateFn<any>, ...functionParams: any[]): BotAction<any> =>
async(page) => await page.evaluate(functionToEvaluate, ...functionParams)
21 changes: 21 additions & 0 deletions src/botmation/helpers/cases.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
import { Dictionary, PipeValue } from "../types"
import { CasesSignal, isCasesSignal } from "../types/cases"

/**
* Create a CasesSignal object with safe defaults for no params provided (default is no matches, pipe value undefined, and conditionPass false)
* @param matches
* @param pipeValue
*/
export const createCasesSignal = <V = any>(matches: Dictionary<V> = {}, conditionPass: boolean = false, pipeValue?: PipeValue): CasesSignal<V> => ({
brand: 'Cases_Signal',
conditionPass,
matches,
pipeValue
})

/**
* If a CasesSignal is provided, its pipeValue is returned, otherwise `undefined` is returned
* @param casesSignal
*/
export const casesSignalToPipeValue = (casesSignal: CasesSignal|any): PipeValue =>
isCasesSignal(casesSignal) ? casesSignal.pipeValue : undefined
20 changes: 0 additions & 20 deletions src/botmation/helpers/matches.ts

This file was deleted.

10 changes: 9 additions & 1 deletion src/botmation/helpers/navigation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,12 @@ export const enrichGoToPageOptions = (overloadDefaultOptions: Partial<DirectNavi
* @param milliseconds
*/
export const sleep = async(milliseconds: number): Promise<NodeJS.Timeout> =>
new Promise(resolve => setTimeout(resolve, milliseconds))
new Promise(resolve => setTimeout(resolve, milliseconds))

/**
*
* @param htmlSelector
*/
export const scrollToElement = (htmlSelector: string) =>
document.querySelector(htmlSelector)?.scrollIntoView({behavior: 'smooth'})

Loading

0 comments on commit e968078

Please sign in to comment.