Support go plugins for forges and agent backends #2751

qwerty287 · 2023-11-05T12:01:08Z

Support to load new forges and agent backends at runtime using go's plugin system. (https://pkg.go.dev/plugin)

I also added a simple example addon (a new forge which just prints log statements), it should be removed later of course, but you can see an example.

for more information, see https://pre-commit.ci

anbraten · 2023-11-05T12:26:22Z

Looks pretty nice already.
Two questions / ideas that came up to me:

How can we add a plugin to the docker image? Would we create a docker image using the "normal" image as base and simply add the plugin binary?
Can we combine this with the requested compiler, secrets, registry addons somehow? Problem would be as those should be selectable by users they would need to be executed in a rather separated environment like a step by the agent and therefore we might need to consider sth like https://github.com/hashicorp/go-plugin

qwerty287 · 2023-11-05T12:29:36Z

How can we add a plugin to the docker image? Would we create a docker image using the "normal" image as base and simply add the plugin binary?

Plugins are just files at runtime. You can mount them using docker volumes.

Can we combine this with the requested compiler, secrets, registry addons somehow? Problem would be as those should be selectable by users they would need to be executed in a rather separated environment like a step by the agent and therefore we might need to consider sth like https://github.com/hashicorp/go-plugin

That's why I wrote that go plugins might not be the best solution in the PR description :) I'm open for new ideas and can implement them if we decide it.

docs/docs/30-administration/75-addons/20-creating-addons.md

mzampetakis · 2023-11-06T06:12:17Z

This looks great!

It seems that any loaded forge through this add-on system will override the server's configuration for any forge. This could be documented at someplace (e.g. docs/docs/30-administration/75-addons/00-addons.md) so that the administrators will have a more clear view about server's bevahiour.

woodpecker-bot · 2023-11-07T15:56:41Z

Deployment of preview was successful: https://woodpecker-ci-woodpecker-pr-2751.surge.sh

for more information, see https://pre-commit.ci

qwerty287 · 2023-11-22T15:01:38Z

So, I've been playing with hashicorp/go-plugin the last few hours but I'm not really convinced yet. It gets really complex if you use complex arguments and return types ("complex" mainly means bidirectional execution of funcs). Maybe I'm also doing something wrong, I pushed a commit with the changes (it does not work though).

mzampetakis · 2023-11-22T15:05:26Z

Is there any particular reason for using hashicorp/go-plugin instead of the built in one?
I believe that it might be more convenient to stick with the go's built in one in terms of upgradability.
What do you think?

qwerty287 · 2023-11-22T15:08:38Z

Yes, go's plugins are executed in the same process which has some issues, especially when coming to security because it's not separated from the main part. This means that e.g. a panic in the plugin could panic woodpecker itself too.
I'm open about which mechanism we use though. @anbraten suggested it

qwerty287 · 2023-11-22T15:10:02Z

You can see the "warnings" section here: https://pkg.go.dev/plugin

qwerty287 · 2023-11-22T15:27:31Z

@anbraten I just read your comment again.

Can we combine this with the requested compiler, secrets, registry addons somehow? Problem would be as those should be selectable by users they would need to be executed in a rather separated environment like a step by the agent and therefore we might need to consider sth like https://github.com/hashicorp/go-plugin

What do you mean by "users"? Do you mean these should be usable also by non-admins? Because I'd say that's very hard. It would require e.g. communication via internet (hashicorp doesn't support this as well).

anbraten · 2023-11-22T16:13:26Z

"users" would be all users (including non-instance-admins) in my idea. The idea I had some time ago was that the server asks an agent to start a service (exp container) (similar to running a pipeline somewhere). This service would then provide an http endpoint. This endpoint could for example provide the config-service endpoint, so it could be used as an compiler. So as soon as the server needs the config it would check that the service is running and forwards the http request through the grpc connection via the agent to the service. But all of that was just a first idea.

mzampetakis · 2023-12-02T17:14:53Z

Is this completed or blocked by something? I could run some tests with a dummy addon forge to check if everything is working as expected. Please let me know if I could provide any assistance.

Co-authored-by: Michalis Zampetakis <[email protected]>

cmd/server/setup.go

cmd/agent/agent.go

docs/docs/30-administration/75-addons/00-overview.md

docs/docs/30-administration/75-addons/20-creating-addons.md

shared/addon/addon.go

Co-authored-by: Anbraten <[email protected]>

for more information, see https://pre-commit.ci

cmd/agent/agent.go

cmd/agent/flags.go

docs/docs/30-administration/75-addons/20-creating-addons.md

cmd/agent/flags.go

Co-authored-by: Michalis Zampetakis <[email protected]>

mzampetakis

Looks great! ♥️

docs/docs/30-administration/75-addons/20-creating-addons.md

## [2.1.0](https://github.com/woodpecker-ci/woodpecker/releases/tag/2.1.0) - 2023-12-26 ### ✨ Features - Add pull request closed event [[#2684](#2684)] - Add depends_on support for steps [[#2771](#2771)] - gitlab: support nested repos [[#2981](#2981)] - Support go plugins for forges and agent backends [[#2751](#2751)] ### 📈 Enhancement - Show default branch on top [[#3019](#3019)] - Support more addon types [[#2984](#2984)] - Hide PR tab if PRs are disabled [[#3004](#3004)] - Switch to ULID [[#2986](#2986)] - Ignore pipelines without config [[#2949](#2949)] - Link labels to input and select [[#2974](#2974)] - Register Agent with hostname [[#2936](#2936)] - Update slogan & logo [[#2962](#2962)] - Improve error handling when activating a repository [[#2965](#2965)] - Add check for storage where repo/org name is empty [[#2968](#2968)] - Update pipeline icons [[#2783](#2783)] - Kubernetes refactor [[#2794](#2794)] - Export changed files via builtin environment variables [[#2935](#2935)] - Show secrets from org and global level [[#2873](#2873)] - Only update pipelineStatus in one place [[#2952](#2952)] - Rename `engine` to `backend` [[#2950](#2950)] - Add linting for `log.Fatal()` [[#2946](#2946)] - Remove separate root path config [[#2943](#2943)] - init CI_COMMIT_TAG if commit ref is a tag [[#2934](#2934)] - Update go module path for major version 2 [[#2905](#2905)] - Unify date/time dependencies [[#2891](#2891)] - Add linting for `any` [[#2893](#2893)] - Fix vite deprecations [[#2885](#2885)] - Migrate to Xormigrate [[#2711](#2711)] - Simple security context options (Kubernetes) [[#2550](#2550)] - Changes PullRequest Index to ForgeRemoteID type [[#2823](#2823)] ### 🐛 Bug Fixes - Hide queue visualization if nothing to show [[#3003](#3003)] - fix and lint swagger file [[#3007](#3007)] - Fix IPv6 host aliases for kubernetes [[#2992](#2992)] - Fix cli lint throwing error on warnings [[#2995](#2995)] - Fix static file caching [[#2975](#2975)] - Gitea driver: ignore GetOrg error if we get a valid user. [[#2967](#2967)] - feat(k8s): Add a port name to service definition [[#2933](#2933)] - Fix error container overflow [[#2957](#2957)] - ignore some errors on repairAllRepos [[#2792](#2792)] - Allow to restart pipelines that has warnings [[#2939](#2939)] - Fix skipped pipelines model [[#2923](#2923)] - fix: Add `backend_options` to service linter entry [[#2930](#2930)] - Fix flags added multiple times [[#2914](#2914)] - Fix schema validation with array syntax for clone and services [[#2920](#2920)] - Fix prometheus docs [[#2919](#2919)] - Fix podman agent container in v2 [[#2897](#2897)] - Fix bitbucket org fetching [[#2874](#2874)] - Only deploy docs on `main` [[#2892](#2892)] - Fix pipeline-related environment [[#2876](#2876)] - Fix version check partially [[#2871](#2871)] - Fix unregistering agents when using agent tokens [[#2870](#2870)] ### 📚 Documentation - [Awesome Woodpecker] added yet another autoscaler [[#3011](#3011)] - Add cookbook blog and improve docs [[#3002](#3002)] - Replace multi-pipelines with workflows on docs frontpage [[#2990](#2990)] - Update README badges [[#2956](#2956)] - Update 20-kubernetes.md [[#2927](#2927)] - Add release documentation to CONTRIBUTING [[#2917](#2917)] - Add nix-attic plugin to the index [[#2889](#2889)] - Add usage with Tunnelmole to docs [[#2881](#2881)] - Improve code blocks in docs [[#2879](#2879)] - Add a blog post [[#2877](#2877)] - Add documentation on Kubernetes securityContext [[#2822](#2822)] - Add default page to categories [[#2869](#2869)] - Use same format for Github docs as used for the other forges [[#2866](#2866)] ### Misc - chore(deps): update dependency isomorphic-dompurify to v2 [[#3001](#3001)] - fix(deps): update dependency @intlify/unplugin-vue-i18n to v2 [[#2998](#2998)] - Fix go in gitpod [[#2973](#2973)] - fix(deps): update module google.golang.org/grpc to v1.60.1 [[#2969](#2969)] - chore(deps): update docker.io/alpine docker tag to v3.19 [[#2970](#2970)] - Fix broken gated repos [[#2959](#2959)] - fix(deps): update golang (packages) [[#2958](#2958)] - Update docker.io/techknowlogick/xgo Docker tag to go-1.21.5 [[#2926](#2926)] - Update docker.io/golang Docker tag to v1.21.5 [[#2925](#2925)] - Lock file maintenance [[#2910](#2910)] - Update web npm deps non-major [[#2909](#2909)] - Update docs npm deps non-major [[#2908](#2908)] - Update golang (packages) [[#2904](#2904)] - Update module github.com/google/go-github/v56 to v57 [[#2899](#2899)] - Update dependency marked to v11 [[#2898](#2898)] - Update dependency vite-svg-loader to v5 [[#2837](#2837)] - Update golang (packages) [[#2894](#2894)] - Update web npm deps non-major [[#2895](#2895)] - Update web npm deps non-major [[#2884](#2884)] - Update docker.io/woodpeckerci/plugin-docker-buildx Docker tag to v2.2.1 [[#2883](#2883)]

qwerty287 added 5 commits November 1, 2023 20:26

Some basic work on addons

851fdf8

Support agent backends

205852f

improve/fix docs

9e99f60

Merge branch 'main' into go-addon

0f37be8

remove old gomod

7b82fab

qwerty287 added the feature add new functionality label Nov 5, 2023

qwerty287 and others added 2 commits November 5, 2023 13:01

Merge branch 'main' into go-addon

b2af545

[pre-commit.ci] auto fixes from pre-commit.com hooks [CI SKIP]

cbcd301

for more information, see https://pre-commit.ci

qwerty287 added this to the 2.1.0 milestone Nov 5, 2023

mzampetakis reviewed Nov 6, 2023

View reviewed changes

docs/docs/30-administration/75-addons/20-creating-addons.md Outdated Show resolved Hide resolved

Merge branch 'main' into go-addon

7c8a0fa

qwerty287 and others added 7 commits November 7, 2023 16:57

fix imports

b86f8d2

fix imports

c8527fe

fix style

9728118

add missing newline

6111df8

Merge branch 'main' into go-addon

0676ab4

[pre-commit.ci] auto fixes from pre-commit.com hooks [CI SKIP]

f5bbbbe

for more information, see https://pre-commit.ci

some move to hashicorp/go-plugin

91afda4

some more tests

11ac924

qwerty287 and others added 3 commits December 14, 2023 07:23

Update docs/docs/30-administration/75-addons/00-overview.md

2a7a675

Co-authored-by: Michalis Zampetakis <[email protected]>

Merge branch 'main' into go-addon

92e9108

as per reviews

548a85b

anbraten reviewed Dec 14, 2023

View reviewed changes

qwerty287 and others added 7 commits December 14, 2023 10:58

Apply suggestions from code review

09b4034

Co-authored-by: Anbraten <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks [CI SKIP]

10fea7d

for more information, see https://pre-commit.ci

Rename new occurences to backend

94a89eb

add trust warning

81084e4

different wording

4d10d0e

Merge branch 'main' into go-addon

893b857

Merge branch 'main' into go-addon

f458289

mzampetakis reviewed Dec 17, 2023

View reviewed changes

cmd/agent/agent.go Outdated Show resolved Hide resolved

cmd/agent/flags.go Show resolved Hide resolved

docs/docs/30-administration/75-addons/20-creating-addons.md Outdated Show resolved Hide resolved

docs/docs/30-administration/75-addons/20-creating-addons.md Outdated Show resolved Hide resolved

6543 reviewed Dec 19, 2023

View reviewed changes

cmd/agent/flags.go Show resolved Hide resolved

6543 and others added 3 commits December 19, 2023 14:26

Apply suggestions from code review

47c0fee

Co-authored-by: Michalis Zampetakis <[email protected]>

Merge branch 'main' into go-addon

63299c0

fix

ea53cb2

mzampetakis approved these changes Dec 20, 2023

View reviewed changes

Merge branch 'main' into go-addon

0ed6471

qwerty287 requested a review from a team December 20, 2023 11:03

rename to getBackendEngine

861d9e8

6543 reviewed Dec 20, 2023

View reviewed changes

docs/docs/30-administration/75-addons/20-creating-addons.md Outdated Show resolved Hide resolved

6543 added 2 commits December 20, 2023 14:14

Update docs/docs/30-administration/75-addons/20-creating-addons.md

8b9c744

fmt

2c4546d

6543 approved these changes Dec 20, 2023

View reviewed changes

6543 enabled auto-merge (squash) December 20, 2023 13:16

6543 merged commit dfc2c26 into woodpecker-ci:main Dec 20, 2023
6 checks passed

woodpecker-bot mentioned this pull request Dec 20, 2023

🎉 Release 2.1.0 #2868

Merged

1 task

qwerty287 deleted the go-addon branch December 21, 2023 16:04

BrewTestBot mentioned this pull request Dec 26, 2023

woodpecker-cli 2.1.0 Homebrew/homebrew-core#158332

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support go plugins for forges and agent backends #2751

Support go plugins for forges and agent backends #2751

qwerty287 commented Nov 5, 2023 •

edited by 6543

Loading

anbraten commented Nov 5, 2023

qwerty287 commented Nov 5, 2023

mzampetakis commented Nov 6, 2023

woodpecker-bot commented Nov 7, 2023 •

edited

Loading

qwerty287 commented Nov 22, 2023 •

edited

Loading

mzampetakis commented Nov 22, 2023

qwerty287 commented Nov 22, 2023 •

edited

Loading

qwerty287 commented Nov 22, 2023

qwerty287 commented Nov 22, 2023

anbraten commented Nov 22, 2023

mzampetakis commented Dec 2, 2023

mzampetakis left a comment

Support go plugins for forges and agent backends #2751

Support go plugins for forges and agent backends #2751

Conversation

qwerty287 commented Nov 5, 2023 • edited by 6543 Loading

anbraten commented Nov 5, 2023

qwerty287 commented Nov 5, 2023

mzampetakis commented Nov 6, 2023

woodpecker-bot commented Nov 7, 2023 • edited Loading

qwerty287 commented Nov 22, 2023 • edited Loading

mzampetakis commented Nov 22, 2023

qwerty287 commented Nov 22, 2023 • edited Loading

qwerty287 commented Nov 22, 2023

qwerty287 commented Nov 22, 2023

anbraten commented Nov 22, 2023

mzampetakis commented Dec 2, 2023

mzampetakis left a comment

Choose a reason for hiding this comment

qwerty287 commented Nov 5, 2023 •

edited by 6543

Loading

woodpecker-bot commented Nov 7, 2023 •

edited

Loading

qwerty287 commented Nov 22, 2023 •

edited

Loading

qwerty287 commented Nov 22, 2023 •

edited

Loading