Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v2.1.0 #20

Merged
merged 36 commits into from
Sep 14, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
116078a
Scrapers
mrWh1te Aug 27, 2020
acb2144
v2.0.x -> v2.1
mrWh1te Aug 28, 2020
73e57b0
Scrapers Docs
mrWh1te Aug 28, 2020
0d7ba8e
Instagram selectors
mrWh1te Aug 28, 2020
c9cd4c4
Sites LinkedIn
mrWh1te Aug 28, 2020
b86a863
Sidebar - add link to sites/linkedin
mrWh1te Aug 28, 2020
eec8e9a
LinkedIn TOC improved
mrWh1te Aug 28, 2020
e118a8e
Instagram site toc improved
mrWh1te Aug 28, 2020
e3ed152
Move wait() into Navigation
mrWh1te Aug 28, 2020
b171d6f
screenshotAll() code updated
mrWh1te Aug 28, 2020
0c8b300
Add SonarCloud to Infrastructure
mrWh1te Aug 29, 2020
d0e8b1e
Replace LGTM badge with sonarcloud
mrWh1te Aug 29, 2020
bebc916
scrollTo() botaction
mrWh1te Sep 10, 2020
5e8a8a9
navigation/helpers scrollToElement()
mrWh1te Sep 10, 2020
f583031
api/scrapers Evaluate()
mrWh1te Sep 10, 2020
6f28176
Advanced Aborting, abort(), abortPipe(), abort helpers
mrWh1te Sep 10, 2020
06e5275
pipeCase & pipeCases
mrWh1te Sep 10, 2020
2ae1347
isCasesSignal() type guard
mrWh1te Sep 10, 2020
904af9c
Updated sites/ LinkedIn
mrWh1te Sep 10, 2020
6fec48f
sites/linkedin helpers
mrWh1te Sep 10, 2020
762b33f
api/assembly-lines Switch Pipe
mrWh1te Sep 11, 2020
effdce1
Switch Pipe copy improvement
mrWh1te Sep 11, 2020
51e5941
Update Assembly-Lines
mrWh1te Sep 11, 2020
68f2638
errors & inject aborting behavior
mrWh1te Sep 11, 2020
8b1202c
Utilties BotAction's updated code
mrWh1te Sep 11, 2020
8ae23a0
forAll() extended cb/pipe params
mrWh1te Sep 11, 2020
6540588
Extended Advanced Conditionals with pipeCase(s)
mrWh1te Sep 11, 2020
05e7c75
Advanced/Conditionals expanded
mrWh1te Sep 11, 2020
bf23123
Advanced/aborting wip
mrWh1te Sep 11, 2020
b8d1bb7
wip home page copy & advanced/aborting
mrWh1te Sep 12, 2020
544d896
roadmap update
mrWh1te Sep 14, 2020
90f179c
advanced aborting doc wip
mrWh1te Sep 14, 2020
ee49a67
advanced aborting
mrWh1te Sep 14, 2020
f792311
sidebar navigation change
mrWh1te Sep 14, 2020
83f86cf
helpers at bottom
mrWh1te Sep 14, 2020
1b6c703
review/polish
mrWh1te Sep 14, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion gatsby-config.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ module.exports = {
siteMetadata: {
siteTitle: `Botmation Documentation`,
defaultTitle: `Botmation Documentation`,
siteTitleShort: `Botmation Docs v2.0.x`,
siteTitleShort: `Botmation Docs v2.1`,
siteDescription: `Compose web bots declaratively with the simple TypeScript framework called Botmation`,
siteUrl: `https://botmation.dev`,
siteAuthor: `Michael Lage`,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,9 @@ export const Wrapper = (props) => {
);
}

// original

//
// Original
export const Main = styled.main`
padding: 0 40px;
height: 100%;
Expand Down
2 changes: 1 addition & 1 deletion src/@rocketseat/gatsby-theme-docs/components/Logo.js
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ export default () => {
return (
<h1>
Botmation
<VersionWrapper>v2.0.x</VersionWrapper>
<VersionWrapper>v2.1</VersionWrapper>
</h1>
);
}
12 changes: 6 additions & 6 deletions src/@rocketseat/gatsby-theme-docs/text/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,25 +5,25 @@
[![npm](https://img.shields.io/npm/v/botmation)](https://www.npmjs.com/package/botmation)
[![Build Status](https://travis-ci.com/mrWh1te/Botmation.svg?branch=master)](https://travis-ci.com/mrWh1te/Botmation)
[![codecov](https://img.shields.io/codecov/c/github/mrWh1te/Botmation/master?label=codecov)](https://codecov.io/gh/mrWh1te/Botmation)
[![LGTM Grade](https://img.shields.io/lgtm/grade/javascript/github/mrWh1te/Botmation)](https://lgtm.com/projects/g/mrWh1te/Botmation)
[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=mrWh1te_Botmation&metric=alert_status)](https://sonarcloud.io/dashboard?id=mrWh1te_Botmation)
[<img src="https://david-dm.org/mrWh1te/Botmation/status.svg" alt="" />](https://david-dm.org/mrWh1te/Botmation)
![GitHub](https://img.shields.io/github/license/mrWh1te/Botmation)

[Botmation](https://github.com/mrWh1te/Botmation) is a simple TypeScript framework for building web bots in a functional, composable and declarative way.
[Botmation](https://github.com/mrWh1te/Botmation) is a simple, functional, TypeScript framework for building web bots in a composable and declarative way.

It's built on top of [Puppeteer](https://github.com/puppeteer/puppeteer), a NodeJS library which provides a high-level API to control Chrome or Chromium over the [DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/).
It's built with [Puppeteer](https://github.com/puppeteer/puppeteer), a NodeJS library that provides a high-level API to control Chrome or Chromium over the [DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/).

With [Puppeteer](https://github.com/puppeteer/puppeteer), you build web bots that automate your social media, take screenshots or generate PDF's of websites, surf & scrape the web and much more. 🌊🏄
With [Puppeteer](https://github.com/puppeteer/puppeteer), it's few steps away from building web bots that automate your social media, take screenshots & generate PDF's of websites, surf while scraping the web and much more. 🌊🏄

With [Botmation](https://github.com/mrWh1te/Botmation), you get a simple, composable, scalable, declarative pattern to build web bots with Puppeteer. It reduces code reusing barriers and maximizes web bots' code *readability*, *maintainability* and *testability*.
With [Botmation](https://github.com/mrWh1te/Botmation), you get a simple, scalable, declarative pattern to build Puppeteer web bots. It reduces code reusing barriers with *composition* and maximizes your web bots' code *readability*, *maintainability* and *testability*.

## Features

- 🤓 Learn only what you need
- 🛣️ Simple pattern to follow
- ⚡️ Write readable, testable & composable code
- 🥇 100% Library Test Coverage
- 📚 Get started with Functional Programming
- ⚡️ Write readable, testable & composable code
- 🏢 Use higher order functions
- 🧱 Assemble complex async functionality
- and much more 🔥
Expand Down
18 changes: 13 additions & 5 deletions src/config/sidebar.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,17 @@
- label: 'Install'
link: '/install'

- label: 'Sites'
items:
- label: 'Instagram'
link: '/sites/instagram'
- label: 'LinkedIn'
link: '/sites/linkedin'

- label: 'Advanced'
items:
- label: 'Aborting'
link: '/advanced/aborting'
- label: 'Piping'
link: '/advanced/piping'
- label: 'Injects'
Expand All @@ -30,13 +39,10 @@
- label: 'Tutorial'
link: '/advanced/tutorial'

- label: 'Sites'
items:
- label: 'Instagram'
link: '/sites/instagram'

- label: 'API'
items:
- label: 'Abort'
link: '/api/abort'
- label: 'Assembly Lines'
link: '/api/assembly-lines'
- label: 'Console'
Expand All @@ -59,6 +65,8 @@
link: '/api/navigation'
- label: 'Pipe'
link: '/api/pipe'
- label: 'Scrapers'
link: '/api/scrapers'
- label: 'Utilities'
link: '/api/utilities'

Expand Down
56 changes: 56 additions & 0 deletions src/docs/advanced/aborting.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
---
title: Aborting
---

## AbortLine Signal

Sometimes it's necessary to abort an assembly line(s) of BotAction's. Perhaps something is removed from a web page or a condition wasn't met that subsequent BotAction's rely on. When this happens, a BotAction returns an `AbortLineSignal`.

```typescript
type AbortLineSignal = {
brand: 'Abort_Signal',
assembledLines: number,
pipeValue?: PipeValue
}
```

AbortLine signals are safely created with the [createAbortLineSignal()](/api/abort#createabortlinesignal) helper. It's a simple JSON object with potentially two to three properties.

### Assembled Lines Count

An AbortLine signal's important property is called `assembledLines`. It informs the assembler how many lines of assembly to break. One is one line, two is two, three is three and so forth.

The default count of `assembledLines` is one for both the [abort()](/api/abort#abort) BotAction and the [createAbortLineSignal()](/api/abort#createabortlinesignal) helper. It can be any positive number, including zero.

## Infinity AbortLine Signal
It's possible to abort all lines of assembly without knowing how many there are by returning an Infinity AbortLine signal.

Infinity AbortLine signals have `assembledLines` set to `0`. When an assembler processes an Infinity AbortLine signal, it aborts what it's doing and returns the same signal without change.

### Optional Pipe Value
`AbortLineSignal` has an optional param called `pipeValue` that carries a `PipeValue` back to the final aborted assembler for return.

> The optional `pipeValue` param does not get returned by a [Chain](/api/assembly-lines#chain), when it's the final assembler to process an AbortLine signal.

## Process AbortLine Signal

Almost every assembler processes an AbortLine signal with the [processAbortLineSignal()](/api/abort#processabortlinesignal) helper. It supports all main use-cases of `assembledLines`: 0, 1, and 2+.

Here is a table for the default assemblers' aborting behavior:

| assembledLines | effect |
| - | - |
| 0 | Infinity AbortLine Signal aborts every BotAction and returns itself without change on each abort |
| 1 | Default AbortLine Signal aborts only one line of assembled BotActions and returns the abortLineSignal.pipeValue |
| 2+ | Level specific AbortLine Signal aborts one line of assembled BotActions at a time while reducing its assembledLines count to 1 for final abort |

Each assembler will detect the value returned by the resolved BotAction for an AbortLine signal. If it is one, it processes it to the effect listed above.

This is the default aborting behavior. There are exceptions for some BotAction's such as the Utilities, Switch Pipe, and Pipe Case(s), which implement their own unique aborting behavior.

## Unique Aborting behavior
[Switch Pipe](/api/assembly-lines#switch-pipe) implements its own AbortLine signal behavior in order to support its own unique functionality. When it hasn't received a [CasesSignal](/advanced/conditionals/#cases-signal) with its condition passed, it ignores default AbortLine signals. That's because [abort()](/api/abort#abort) is treated like `break;` from the traditional `switch(){}` code block, inside a [Switch Pipe](/api/assembly-lines#switch-pipe).

Some BotAction's, like Utilities, have unique AbortLine signal behavior for aborting parts of the unique assemblers. For example, a [For All](/api/utilities#for-all) BotAction's loop iteration (runs an assembled line of BotActions) can abort out of its loop iteration with a default AbortLine signal or abort the entire loop with two or more `assembledLines`.

BotAction's with unique aborting behavior have their aborting functionality explained in their documentation, listed under API in the sidebar.
49 changes: 45 additions & 4 deletions src/docs/advanced/conditionals.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Conditionals
---

Botmation has a special type of BotAction called Utilities. These allow you to build lines of BotAction's but with unique behavior.
Botmation has a special BotAction's that assemble BotAction's that run *if* a condition is met.

## Functional if statement

Expand Down Expand Up @@ -30,10 +30,51 @@ await chain(

> All [Utility BotAction's](/api/utilities) run assembled BotAction's in a Pipe

## Conditional BotAction
### Conditional BotAction

The first call `givenThat()` accepts a special kind of BotAction called ConditionalBotAction. ConditionalBotAction are like BotAction's, except they *always* return a boolean value.

`isGuest` and `isLoggedIn` are ConditionalBotAction's for Instagram's web app. They are not intended to change the state of the `page` but check the `page` for a condition, then report on it. This way, the Instagram `login()` BotAction is ran only if needed. Check the [Instagram example](https://github.com/mrWh1te/Botmation/tree/master/src/examples/instagram.ts) for more like saving & loading cookies, to skip subsequent logins.
`isGuest` and `isLoggedIn` are ConditionalBotAction's for Instagram's web app. They are not intended to change the state of the `page` but check the `page` for a condition, then report on it. This way, the Instagram `login()` BotAction is ran only if needed. Check the [Instagram example](https://github.com/mrWh1te/Botmation/tree/master/src/examples/instagram.ts) for more including saving & loading cookies, to skip subsequent logins.

For a more technical explanation, `givenThat()()` returns a customized BotAction for the whole flow. When you resolve the returned BotAction, it runs through a series of steps. First, it resolves the `ConditionalBotAction` in a Pipe for the value it returns. Then runs the assembled BotAction's if the value was `true`.
For a more technical explanation, `givenThat()()` returns a customized BotAction for the whole flow. When you resolve the returned BotAction, it runs through a series of steps. First, it resolves the `ConditionalBotAction` in a Pipe for the value it returns. Then runs the assembled BotAction's if the value was `true`. However, it does not return the final pipe value to be compatible with Chain's.

## Conditionals evaluating Pipe Value

There are other BotAction's like `givenThat()()` for running a line of BotActions if a condition passes. They evaluate the equality of values supplied against the Pipe object value. They are called [pipeCase()()](/api/pipe#pipe-case) && [pipeCases()()](/api/pipe#pipe-cases) which run the assembled BotAction's of their second call, if their condition in the first call passes.

### Conditional Callback

Instead of handling a ConditionalBotAction in their first call, they expect a Pipe value, which can be almost anything, including a synchronous function. When a synchronous function is provided, it's called as a callback with the Pipe value as its only parameter. These functions are called ConditionalCallback's. Here is the interface:

```typescript
interface ConditionalCallback<V = PipeValue> extends Function {
(value: V) : boolean
}
```

They take a PipeValue and operate on it to return a boolean value, representing if the condition past or not.

These BotAction's support multiple values, and handle their overall "if condition" differently. [pipeCase()()](/api/pipe#pipe-case) will test all of its values as separate expressions to match against the Pipe value. If *one or more* of those expressions evaluates as `true` then it runs its assembled BotAction's. In essence, it combines each value in the conditional expression with `||`.

[pipeCases()()](/api/pipe#pipe-cases), on the otherhand, will test all of its values until one of them evaluates as `false`, breaking the condition evaluation. All of its values *must* match the Pipe value, in order for the assembled BotAction's to run. In essence, it combines each value in the conditional expression with `&&`.

With this approach, you can supply simple sync functions to create conditional expressions to determine whether or not some functionality needs to run. Both BotAction's will pipe in the same Pipe object that they are testing into their first assembled BotAction's. Therefore, it's possible to nest these to create more elaborate "if conditions" that combine or/and `||`/`&&`.

Finally, you can compose these BotAction's with a [Switch Pipe](/api/assembly-lines#switch-pipe), to create a switch, case(s), and break(s) (with the [Abort](/api/abort#abort) BotAction) flow to handle more use-cases. For an example of all three combined, see [LinkedIn's likeUserPostsFrom()](/sites/linkedin#like-user-posts-from) BotAction.

While [Switch Pipe](/api/assembly-lines#switch-pipe), [Pipe Case](/api/pipe#pipe-case), [Pipe Cases](/api/pipe#pipe-cases), and [Abort](/api/abort#abort) were all created together to work together in this particular way, they are all BotAction's that can be assembled with or without each other.

### Cases Signal

[Pipe Case](/api/pipe#pipe-case) and [Pipe Cases](/api/pipe#pipe-cases) return an unique object called `CasesSignal`, unless they are aborted completely. The CasesSignal has an important key called `conditionPass` which is a boolean to explain whether or not their condition evaluated as `true`. Also, CasesSignal includes the values that matched the pipe object value, including the functions, on a `matches` key. It's a simple JSON object with key->value pairs. The keys are the index'es of the values matched, set to their corresponding values. If you need to know which cases of [Pipe Case](/api/pipe#pipe-case) matched, you can read the returned CasesSignal `matches` object.

```typescript
type CasesSignal<V = any> = {
brand: 'Cases_Signal',
matches: Dictionary<V>,
conditionPass: boolean,
pipeValue?: PipeValue
}
```

CasesSignal is recognized by [Switch Pipe](/api/assembly-lines#switch-pipe), and once a CasesSignal `conditionPass` is `true`, [Switch Pipe](/api/assembly-lines#switch-pipe) lowers its `assembledLines` for aborting by one. That's how `abort()` does not break a [Switch Pipe](/api/assembly-lines#switch-pipe) line, until a Pipe Case(s) has returned a CasesSignal with `conditionPass` as `true`. Therefore, it mirrors the behavior of a traditional switch, case, break code block. For an example, see [LinkedIn's likeUserPostsFrom()](/sites/linkedin#like-user-posts-from) BotAction.
4 changes: 2 additions & 2 deletions src/docs/advanced/injects.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Botmation's pattern of composing BotAction's has support for injecting other dep

Let's take a look at the actual BotAction Function interface:
```typescript
interface BotAction<R = void, I extends Array<any> = any[]> extends Function {
interface BotAction<R = void|AbortLineSignal, I extends Array<any> = any[]> extends Function {
(page: Page, ...injects: I) : Promise<R>
}
```
Expand All @@ -30,7 +30,7 @@ await chain(

The assembled BotAction's in the `chain()` will be called with `service` and `todaysDate` as the second and third parameters.

## Inject BotAction
## Inject()() BotAction

Now, what if you want to compose a line of actions, but with new injects? You can use the `inject()()` BotAction to pass in new injects. The higher level injects will be appended to the new ones.

Expand Down
4 changes: 3 additions & 1 deletion src/docs/advanced/loops.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@ await chain(
```
Here we pass in a collection of site domain's in the first call of `forAll()` then in the second call pass in a callback function, to run BotAction(s) on each iteration of the collection. Therefore, the BotAction's returned by the callback function, in the second call, are ran in a loop for each item of the iteration, in this case, each domain name.

When the [forAll()()()](/api/utilities#for-all) BotAction completes, it will have visited and taken screenshots of each site in the array.
When the [forAll()()()](/api/utilities#for-all) BotAction completes, it will have visited and taken screenshots of each site in the array.

Learn more about `forAll()()` by reading its [documentation](/api/utilities#for-all).

## For As Long

Expand Down
6 changes: 6 additions & 0 deletions src/docs/advanced/piping.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,12 @@ const isGuest: ConditionalBotAction =

Here we are getting a value from the `redux` IndexedDB database, in a store called `paths`, by the key `users.viewerId`. The returned value (Pipe value) is mapped to the correct value for `isGuest`. Basically, if the `viewerId` is defined, then the Bot is considered logged in, and `isGuest` is `false` otherwise `true`.

## Switch Pipe

For a different kind of Piping assembler, check out [Switch Pipe](/api/assembly-lines#switch-pipe). It works similar to `pipe()()` except it injects the same Pipe object, from the its injects or the higher-order call, into each assembled BotAction.

To learn more about `switchPipe()()` read its [documentation](/api/assembly-lines#switch-pipe).

## Helpers for Piping

Also, there are separate functions in this library, that are found in the module, that are not BotAction's, but regular functions, to help reduce boilerplate called Helpers. For example, there are many Helper functions for piping.
Expand Down
3 changes: 3 additions & 0 deletions src/docs/advanced/tutorial.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -140,3 +140,6 @@ Simple, declarative, and composable. We can reuse our `linkedin_bot` multiple ti
How would you go about making `linkedin_bot` dynamic as in a function that accepts the auth information so the bot could be used in multiple pages with varying credentials?

Hint: Use a higher-order function

## Going Further
How about scraping the news feed to like posts published by your a select group of people? Check out this working [example](https://github.com/mrWh1te/Botmation/blob/master/src/examples/linkedin.ts) for how.
Loading