Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V2 init #2

Open
wants to merge 73 commits into
base: development
Choose a base branch
from

Conversation

Keyrxng
Copy link
Contributor

@Keyrxng Keyrxng commented Aug 30, 2024

Resolves ubiquity-os/plugins-wishlist#25
Required by #1

Dual action/worker plugin serving as Telegram Bot kernel and an UbiquityOS plugin.

Requirements

Config:

  - uses:
    - plugin: ubiquity-os-marketplace/telegram-bridge@development
      with:
        botId: 7487982261
  - uses:  
    - plugin: http://localhost:3000
      with:
        botId: 7487982261
  • Database: Run the migration inside the Supabase SQL editor. Obtain url and service key.
  • Repository: Make yourself an admin of the repo. Mint a classic PAT, repo scoped.
  • TG Bot: Contact the BotFather and create a new bot, obtain the token for it.
  • TG Account: Head to my.telegram and obtain the api_hash and app_id.
  • Smee: Create a new url at https://smee.io/new.
  • yarn setup-env: It'll run through each set of env vars, store and upload them. Enter 0 for BOT_ADMINS for now, don't wrap input in quotes and if you need to run it more than once and you are copy pasting from .dev.vars you should sanitize the additional \ applied to line breaks in the private key.
  • yarn sms-auth: It'll ask for details via the terminal, store the session and exit.
  • TG Webhook: smee -u <url> -P "/webhook" in terminal B
  • yarn worker: should be working at this point. Speak to your bot on TG and use /setcommands.

If you have any issues at this stage let me know please.

To use the workflow functions simply create a new task in a repo and assign a price label. Check the actions tab of the telegram-bot repo and wait. Close, repeat. Reopen, repeat.

Everything should be running smoothly. I E2E tested with new DBs etc a couple of times so hopefully there is no issues for anyone testing.


Features

Commands

  • /setcommands - private, admin
  • /chatid - private, groups
  • /botid - private
  • /myid - private

Workflow functions

  • create-chat: Creates a workroom per task once price labeled. @org/repo#1 url schema and posts an invite link on the task
  • close-chat: Kicks everyone once the task is closed, archives the chat and snapshots userIds.
  • reopen-chat: Unarchives the chat, re-invites everyone previously involved.

Environment helpers

  • setup-env: Uses the CLI to walk through saving env vars locally and pushing them to repo secrets.
  • sms-auth: Requires the personal account holder to create a session via 2FA confirmed via a Telegram client, i.e you'll get a desktop notification from TG with the code.

Copy link

github-actions bot commented Aug 30, 2024

Unused files (1)

src/types/typeguards.ts

Unused dependencies (1)

Filename dependencies
package.json octokit

@Keyrxng
Copy link
Contributor Author

Keyrxng commented Aug 30, 2024

rfc @0x4007 @gentlementlegen @rndquu @whilefoo

I'll clean up and streamline things a lot if I should continue ahead with this approach

Copy link

@whilefoo whilefoo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know what's going in the code, I'm guessing it's a lot of boilerplate. But the full picture I got is that when issue is created the kernel sends event to this plugin which creates a workroom and deletes it when the issue is deleted.
I also saw there is a telegram webhook, what is the purpose of that?

@0x4007
Copy link

0x4007 commented Sep 2, 2024

Let's do individual chat rooms instead of a group with topics. This is to maximize signal to noise.

Also the data should not be deleted, instead, the users should be kicked when the project is closed as complete. This is to reduce clutter on one's telegram inbox and search history. If necessary they can always rejoin from the original link.

@Keyrxng
Copy link
Contributor Author

Keyrxng commented Sep 2, 2024

@whilefoo sorry mate I had some commits I hadn't pushed but to sum things up so far:

This worker is split into two distinct parts.

A: Hono app which supports the telegram bot
B: Standard Ubiquity-OS plugin architecture

A - We are hooked into our BotFather bot and this is the "kernel" for it. So we'll create all commands, functions etc through this worker which is separate from but able to interact with the plugin side.

B - We are also hooked into the kernel as a plugin and in this PR you see the callbacks relating to the "workrooms" feature only.


I'm guessing it's a lot of boilerplate

Yeah still a little bit of boilerplate and a few helper commands I've added for setup idk if they'll be kept. I expected that if this PR progressed I'd need to build in basic commands and features so I haven't cleaned up completely.

But the full picture I got is that when issue is created the kernel sends event to this plugin which creates a workroom and deletes it when the issue is deleted

  1. issues.labeled fires and kernel sends payload to this worker
  2. Github path uses the tg bot api to create/close workrooms, post github comment etc
  3. Bot path can hear the event for new chat created and we can act on forwarding that message etc.
  4. We do not delete chats but we do update the DB with a "closed" status and close/kick everyone from it.
  5. Still to add some safety checks like is it a price label that's been added etc.

This feature aside, it'll house all telegram bot chat commands and features etc.


The task discussion lays out a spec to enable the opening of new group chats from a personal account vs this implementation which is using a main supergroup and opening forums (topics). I spent hours trying various libs and approaches and decided to progress everything else and either it can be tasked out or if it's a blocker more time spent on it.

Here is the troubles with that so far:

  1. The endpoint is not REST and requires special auth handling. There are libraries and frameworks built for the MTProto API however, because we are building Cloudflare workers and not hosting a node server for example packages do not work out of the box and/or they are just incompatible with that env.
  2. Frameworks and libs are built targeting browser or node, while cloudlfare is a weird custom mix of both for using modules such as crypto which is essential in these libs for handling auth and sessions which would need patched for deployment.
  3. We could attempt to handle all that these frameworks do re: MTProto auth, sessions etc and build with Cloudflare V8 in mind but I'm not sure that's worth the time investment.
  4. Implementation details aside, it's not ergonomic for a number of reasons to use separate group chats for each new workroom. The idea behind them is empowering collaboration and oversight of off-site task related activity.
  • If they are separate chats we have no global overview.
  • 3rd parties like investors eg cannot see activity in these without joining them directly.
  • They'll only have the team member(s) that join the workroom when it's created unless we automate that

@Keyrxng
Copy link
Contributor Author

Keyrxng commented Sep 2, 2024

Let's do individual chat rooms instead of a group with topics. This is to maximize signal to noise.

Also the data should not be deleted, instead, the users should be kicked when the project is closed as complete. This is to reduce clutter on one's telegram inbox and search history. If necessary they can always rejoin from the original link.

Only saw this after posting my comment above.

Agreed, no chats are deleted. I mentioned my troubles with the MTProto API so far and tried all that I could think over the weekend but had very little luck and it involved a lot of "hacks" like patching node_modules because cloudflare wants any Node module prefixed with "node:" for a start, then the methods differ etc.

Anyway, I'll spend more time on it if need be but in regards to everything else, the way it communicates with the kernel and the way I have built callbacks and what that means for adding additional "plugin"-like features that respond to events, happy for me to proceed with this approach?

@0x4007
Copy link

0x4007 commented Sep 2, 2024

Why not make some sort of GitHub actions based micro service that just opens chat rooms?

It sounds to me that cloudflare is limiting this due to incompatible node modules.

@Keyrxng
Copy link
Contributor Author

Keyrxng commented Sep 2, 2024

Why not make some sort of GitHub actions based micro service that just opens chat rooms?

It sounds to me that cloudflare is limiting this due to incompatible node modules.

I had considered that we'd likely use the workflows and the worker in unison depending on what sort of features we add but hadn't considered it for this feature but I will attempt to do that although it probably isn't pretty.

  1. issues.labeled fires, kernel forwards to this worker.
  2. worker needs to repository_dispatch to kick off the workflow (I think that's the best way)
  3. this worker cannot wait until the response so it would require a dedicated workflow and ts script
  4. The script may need packages separate from src although hopefully not.
  5. I struggle to think rn how to facilitate this worker communicating with itself elegantly through workflows and passing data between.

I'll see how I get on with it and report back

@whilefoo
Copy link

whilefoo commented Sep 3, 2024

2. worker needs to repository_dispatch to kick off the workflow (I think that's the best way)

Why does the worker need to start the workflow, the kernel can directly start the workflow? Actually is there any reason why even use a worker if we move everything to the workflow?

@0x4007
Copy link

0x4007 commented Sep 3, 2024

  1. worker needs to repository_dispatch to kick off the workflow (I think that's the best way)

Why does the worker need to start the workflow, the kernel can directly start the workflow? Actually is there any reason why even use a worker if we move everything to the workflow?

Eventually the bot should reply to messages on Telegram, so building for speed is important.

@Keyrxng
Copy link
Contributor Author

Keyrxng commented Sep 3, 2024

Actually is there any reason why even use a worker if we move everything to the workflow?

The workflow is a tool that the worker will use, it is not replacing the worker instance. So it would be a workflow which handles one action: create a group chat using MTProto API. The cf worker can dispatch to it's own repo to execute functions using a different env than what cf provides, which if it works could be awesome.

Why does the worker need to start the workflow, the kernel can directly start the workflow?

So this worker would repo_dispatch to it's own repo to execute the createGroupchat.yml workflow. This way it is not either-or for worker or workflow we can have both for this bot. I don't think it's possible to tell the kernel to fire off an arbitrary workflow is it as we have one entry which is compute.yml?

Eventually the bot should reply to messages on Telegram, so building for speed is important.

This is how I thought of things. If this plugin is a worker we have instant response on the telegram and github side instead of having a dedicated worker just handling telegram requests and us building plugins/features as plugins which interact with the telegram bot. While also having the benefit of dispatching workflows to it's own repo although I thought that would be much further down the line lol

@0x4007
Copy link

0x4007 commented Sep 25, 2024

defined in the config twice

Using YAML anchors is our solution.

When will this be merged @Keyrxng ?

@Keyrxng
Copy link
Contributor Author

Keyrxng commented Sep 26, 2024

Merged hopefully before the beginning of next week, I'll be working on this for the rest of today.

ubiquity-os-marketplace deleted a comment from Tonyalbert01 last week

I'm pretty sure that it was me that deleted those two comments, what's going on there any ideas?

@0x4007
Copy link

0x4007 commented Sep 26, 2024

Just focus on this no other tasks. You have too many pulls open for weeks on end (I've seen some for months) and none get merged so nothing is getting done.

@Keyrxng
Copy link
Contributor Author

Keyrxng commented Sep 26, 2024

Who is handling deployment, myself?

Tried anchors and I don't think the kernel is built for them, I tried a bunch of different configurations. Altering the URL only would cause the with: section to disappear and if we need to redefine both then it's not practical. Asked GPT for a hand too but no joy with.

@Keyrxng
Copy link
Contributor Author

Keyrxng commented Sep 26, 2024

This is QA-able/mergeable by the way, everything working fine for me locally and deployed to CF.

@whilefoo
Copy link

Tried anchors and I don't think the kernel is built for them, I tried a bunch of different configurations.

it's not a feature of the kernel, but of YAML 1.2, which our parser supports

@0x4007
Copy link

0x4007 commented Sep 26, 2024

Fix knip and delete the tests so you can pass CI

Then merge and lets try in production

Eventually we can use my https://t.me/UbiquityOS account but that will require me to set up the secrets and such which we can handle later

@Keyrxng
Copy link
Contributor Author

Keyrxng commented Sep 27, 2024

it's not a feature of the kernel, but of YAML 1.2, which our parser supports

This is a new thing for me but I realize that the kernel does support it and perhaps my wording was incorrect.

image

I did attempt it's last suggestion but as I said the YAML processor pulled in the entire anchor and not just the with: section which meant that I'd need to build an anchor for just that section too and overall it felt like "a long way for a shortcut".

Fix knip and delete the tests so you can pass CI

Then merge and lets try in production

Eventually we can use my https://t.me/UbiquityOS account but that will require me to set up the secrets and such which we can handle later

Cool so we'll use the -beta account that I made until such times then.

README.md Outdated Show resolved Hide resolved
@0x4007
Copy link

0x4007 commented Sep 27, 2024

Can you get this running in production today?

@Keyrxng
Copy link
Contributor Author

Keyrxng commented Sep 27, 2024

Can you get this running in production today?

Should be able to as I said everything is working for me.

@gentlementlegen did you manage to QA and get it deployed etc?

https://t.me/UbiquityOS_Beta_Bot - is currently deployed to my cloudflare, message it with /myid and it'll respond. Actually everyone that is going to be considered an admin of this bot should use the /myid command so they can be saved as .env vars

@whilefoo
Copy link

I did attempt it's last suggestion but as I said the YAML processor pulled in the entire anchor and not just the with: section which meant that I'd need to build an anchor for just that section too and overall it felt like "a long way for a shortcut".

You can do like this:

plugins:
    - uses:
      - plugin: https://test.workers.dev
        with: &with
          botId: 123456
    - uses:
      - plugin: ubiquity-os/test
        with: *with

@Keyrxng
Copy link
Contributor Author

Keyrxng commented Sep 27, 2024

I did attempt it's last suggestion but as I said the YAML processor pulled in the entire anchor and not just the with: section which meant that I'd need to build an anchor for just that section too and overall it felt like "a long way for a shortcut".

You can do like this:

plugins:
    - uses:
      - plugin: https://test.workers.dev
        with: &with
          botId: 123456
    - uses:
      - plugin: ubiquity-os/test
        with: *with

Ik that is what I realized but 1. It's not what we originally intended which was to just change the URL and 2. because it only has one "with" prop it felt like it was exactly the same as just pasting the plugin twice and changing only the url. If we added more settings then I guess it would make more sense to use anchors.

https://t.me/UbiquityOS_Beta_Bot - is currently deployed to my cloudflare, message it with /myid and it'll respond. Actually everyone that is going to be considered an admin of this bot should use the /myid command so they can be saved as .env vars

@Keyrxng
Copy link
Contributor Author

Keyrxng commented Sep 27, 2024

Then merge and lets try in production

I'm not able to deploy or interact with the official workers so I won't be able to deploy to production, at best I might be able to use my worker and my workflow url installed in the official config but it's improper and I'd prefer if it was deployed officially as it caused some confusion when I did it previously for embeddings.

This plugin' setup requirements are a little (I'm not sure of the word to use) sketchy so it makes me more anxious as I can't debug CF or set env vars etc the best I can do is help whoever is going to deploy it to prod (I've tried to make it as well-documented and as easy as I could) so if any questions or help is needed msg me here or on TG, thank you.

It'll be smoother I think if it's deployed manually during reviewer QA that way all of the variables are already set, session is active etc and when this is merged and the worker-deploy workflow runs it has the correct env to begin with. Otherwise it'll require running the workflow multiple times and the set/resetting of variables.

@0x4007
Copy link

0x4007 commented Sep 27, 2024

Every repo here automatically deploys to our Cloudflare Workers on push.

Your ID is 350240305

@Keyrxng
Copy link
Contributor Author

Keyrxng commented Sep 27, 2024

Every repo here automatically deploys to our Cloudflare Workers on push.

So long as the repo secrets are set correctly the workflow pushes them when deploying so things should be okay once past the initial setup.

Anyone that wants the UbiquityOS Beta mobile number to login with just dm me on TG.

Your ID is 5112548588

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Telegram GitHub Issue "Workroom"
5 participants