Add types for jobs

[mariusandra]

This is a WIP that adds types to an experimental syntax for jobs as described here: PostHog/plugin-server#325

The following will work after this is merged:

import { CreatePluginMeta, MetaJobsInput } from '@posthog/plugin-scaffold'

type Meta = CreatePluginMeta<{
    config: {
        localhostIP: string
    }
    attachments: {
        maxmindMmdb?: PluginAttachment
    }
    global: {
        ipLookup?: Reader<Response>
    }
    jobs: {
        flushBatch: (ops: { batch: string[]; retryCount?: number }) => Promise<void>
    }
}>

export const jobs: MetaJobsInput<Meta> = {
    flushBatch: async ({ batch, retryCount = 0 }, { jobs }) => {
        const resp = await fetch('https://httpbin.org/post', {
            method: 'post',
            body: JSON.stringify(batch),
            headers: { 'Content-Type': 'application/json' },
        })
        if (resp.status !== 200) {
            if (retryCount > 5) {
                console.error('Could not post batch', batch)
                return
            }
            jobs.runIn(30, 'seconds').flushBatch({ batch, retryCount: retryCount + 1 })
        }
    },
}

export async function setupPlugin({ jobs }: Meta) {
    await jobs.runNow().flushBatch({ batch: [] })
}

The meta.jobs is fully typed. The jobs export unfortunately has to go through MetaJobsInput<Meta> when declaring (or it could be Meta['__jobsInput'] perhaps?), as the output of meta.jobs has a different structure than the input. meta.jobs runs all the jobs via a wrapper like jobs.runIn(30, 'minutes').flushBatch()....

Obviously this should be merged only once we agree on the structure...

[yakkomajuri]

I'm not particularly opinionated, but my one comment would be that I would intuitively expect to pass a function as a parameter to another to run a task/job. That's the case with setTimeout, setInterval, node-schedule, etc.

Where instead of:

jobs.runNow().flushBatch({ batch: [] })

We'd have:

jobs.runNow(() => flushBatch({ batch: [] }))

// or

jobs.runNow(someJobFunc)

This discussion is completely external to the implementation, so just speaking about what would feel "natural" to me. I do understand we have a tradeoff between feel and the implementation behind the scenes.

And like I said, not very opinionated.

[mariusandra]

It might feel natural, but like you said yourself, the implementation gets in the way. We have no idea when this function will be run... nor on which server. The only thing we know is that we need to store jobName and payload in the database somehow.

Within those constraints I'm open to any kind of suggestion or simplification of the API!

[Twixes]

What file is the implementation of this in (in PostHog/plugin-server#325)?

[Twixes]

Ah, so it's PostHog/plugin-server#351

[Twixes]

Looks like this needs reworking because of jobs API changes PostHog/plugin-server#351 (comment)

[Twixes]

Sorry, wrong review, I wanted this to be 🔴 🟥 🙅 RED

[mariusandra]

Let's try again!

[mariusandra]

Here's a plugin using this code: https://github.com/PostHog/session-tracker-plugin/pull/2/files

[Twixes]

Made two suggestions, implemented them in #17

[Twixes]

I think this could be called PluginJobs, which would be more similar to other such types and more obvious?

[mariusandra]

Good for me

[Twixes]

This is named like a function (starting with a verb) on top of plain PluginMeta, but if this is the way we'd like devs to use for typing, we should only expose this really. Could we rename this to PluginMeta and rename the "dumb" base interface to BasePluginMeta?

[mariusandra]

Great!