Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

self-hosted organization runner does not pick up jobs #831

Closed
igagis opened this issue Nov 27, 2020 · 8 comments
Closed

self-hosted organization runner does not pick up jobs #831

igagis opened this issue Nov 27, 2020 · 8 comments
Labels
bug Something isn't working

Comments

@igagis
Copy link

igagis commented Nov 27, 2020

Describe the bug
The organization-wide self-hosted runner does not pick up jobs. Note, that repository self-hosted runners work ok.

To Reproduce
Steps to reproduce the behavior:

  1. Register a new self-hosted runner to an organization
  2. Push a new commit to a repository which belongs to the organization from step 1
  3. Check actions jobs status. Jobs which have runs-on: [self-hosted] are not picked up, constantly showing "Starting your workflow run...". After some timeout those jobs fail with "This check failed".

Expected behavior
The jobs which have runs-on: [self-hosted] are picked up by the self-hosted runner.

Note, if on step 1 the runner is registered to a repository instead of organization, then it works.

Runner Version and Platform

Runner version: 2.274.2
Ubuntu-18.04

What's not working?

After workflow fails there are messages "No runner matching the specified labels was found: self-hosted." on the "This run" page.

Job Log Output

N/A

Runner and Worker's Diagnostic Logs

ivan@geiger:~/actions-runner/_diag$ cat Runner_20201127-224042-utc.log 
[2020-11-27 22:40:42Z INFO HostContext] No proxy settings were found based on environmental variables (http_proxy/https_proxy/HTTP_PROXY/HTTPS_PROXY)
[2020-11-27 22:40:42Z INFO HostContext] Well known directory 'Bin': '/home/ivan/actions-runner/bin'
[2020-11-27 22:40:42Z INFO HostContext] Well known directory 'Root': '/home/ivan/actions-runner'
[2020-11-27 22:40:42Z INFO HostContext] Well known config file 'Credentials': '/home/ivan/actions-runner/.credentials'
[2020-11-27 22:40:42Z INFO Listener] Runner is built for Linux (X64) - linux-x64.
[2020-11-27 22:40:42Z INFO Listener] RuntimeInformation: Linux 5.3.0-1016-oracle #18~18.04.1-Ubuntu SMP Mon Apr 27 10:16:19 UTC 2020.
[2020-11-27 22:40:42Z INFO Listener] Version: 2.274.2
[2020-11-27 22:40:42Z INFO Listener] Commit: 9987a8d024a6075cd2357d07c5aec9785126d5a9
[2020-11-27 22:40:42Z INFO Listener] Culture: 
[2020-11-27 22:40:42Z INFO Listener] UI Culture: 
[2020-11-27 22:40:42Z INFO HostContext] Well known directory 'Bin': '/home/ivan/actions-runner/bin'
[2020-11-27 22:40:42Z INFO HostContext] Well known directory 'Root': '/home/ivan/actions-runner'
[2020-11-27 22:40:42Z INFO Listener] Validating directory permissions for: '/home/ivan/actions-runner'
[2020-11-27 22:40:42Z INFO CommandLineParser] Parse
[2020-11-27 22:40:42Z INFO CommandLineParser] Parsing 1 args
[2020-11-27 22:40:42Z INFO CommandLineParser] parsing argument
[2020-11-27 22:40:42Z INFO CommandLineParser] HasArgs: False
[2020-11-27 22:40:42Z INFO CommandLineParser] Adding Command: run
[2020-11-27 22:40:42Z INFO Listener] Arguments parsed
[2020-11-27 22:40:42Z INFO Runner] ExecuteCommand
[2020-11-27 22:40:42Z INFO ConfigurationStore] currentAssemblyLocation: /home/ivan/actions-runner/bin/Runner.Listener.dll
[2020-11-27 22:40:42Z INFO HostContext] Well known directory 'Bin': '/home/ivan/actions-runner/bin'
[2020-11-27 22:40:42Z INFO ConfigurationStore] binPath: /home/ivan/actions-runner/bin
[2020-11-27 22:40:42Z INFO HostContext] Well known directory 'Bin': '/home/ivan/actions-runner/bin'
[2020-11-27 22:40:42Z INFO HostContext] Well known directory 'Root': '/home/ivan/actions-runner'
[2020-11-27 22:40:42Z INFO ConfigurationStore] RootFolder: /home/ivan/actions-runner
[2020-11-27 22:40:42Z INFO HostContext] Well known directory 'Bin': '/home/ivan/actions-runner/bin'
[2020-11-27 22:40:42Z INFO HostContext] Well known directory 'Root': '/home/ivan/actions-runner'
[2020-11-27 22:40:42Z INFO HostContext] Well known config file 'Runner': '/home/ivan/actions-runner/.runner'
[2020-11-27 22:40:42Z INFO ConfigurationStore] ConfigFilePath: /home/ivan/actions-runner/.runner
[2020-11-27 22:40:42Z INFO HostContext] Well known directory 'Bin': '/home/ivan/actions-runner/bin'
[2020-11-27 22:40:42Z INFO HostContext] Well known directory 'Root': '/home/ivan/actions-runner'
[2020-11-27 22:40:42Z INFO HostContext] Well known config file 'Credentials': '/home/ivan/actions-runner/.credentials'
[2020-11-27 22:40:42Z INFO ConfigurationStore] CredFilePath: /home/ivan/actions-runner/.credentials
[2020-11-27 22:40:42Z INFO HostContext] Well known directory 'Bin': '/home/ivan/actions-runner/bin'
[2020-11-27 22:40:42Z INFO HostContext] Well known directory 'Root': '/home/ivan/actions-runner'
[2020-11-27 22:40:42Z INFO HostContext] Well known config file 'MigratedCredentials': '/home/ivan/actions-runner/.credentials_migrated'
[2020-11-27 22:40:42Z INFO ConfigurationStore] MigratedCredFilePath: /home/ivan/actions-runner/.credentials_migrated
[2020-11-27 22:40:42Z INFO HostContext] Well known directory 'Bin': '/home/ivan/actions-runner/bin'
[2020-11-27 22:40:42Z INFO HostContext] Well known directory 'Root': '/home/ivan/actions-runner'
[2020-11-27 22:40:42Z INFO HostContext] Well known config file 'Service': '/home/ivan/actions-runner/.service'
[2020-11-27 22:40:42Z INFO ConfigurationStore] ServiceConfigFilePath: /home/ivan/actions-runner/.service
[2020-11-27 22:40:42Z INFO CommandSettings] Flag 'help': 'False'
[2020-11-27 22:40:42Z INFO CommandSettings] Flag 'version': 'False'
[2020-11-27 22:40:42Z INFO CommandSettings] Flag 'commit': 'False'
[2020-11-27 22:40:42Z INFO CommandSettings] Command 'configure': 'False'
[2020-11-27 22:40:42Z INFO CommandSettings] Command 'remove': 'False'
[2020-11-27 22:40:42Z INFO CommandSettings] Command 'warmup': 'False'
[2020-11-27 22:40:42Z INFO ConfigurationManager] LoadSettings
[2020-11-27 22:40:42Z INFO ConfigurationStore] IsConfigured()
[2020-11-27 22:40:42Z INFO ConfigurationStore] IsConfigured: True
[2020-11-27 22:40:42Z INFO ConfigurationManager] Is configured: True
[2020-11-27 22:40:42Z INFO ConfigurationStore] Read setting file: 269 chars
[2020-11-27 22:40:42Z INFO ConfigurationManager] Settings Loaded
[2020-11-27 22:40:42Z INFO ConfigurationStore] IsServiceConfigured()
[2020-11-27 22:40:42Z INFO ConfigurationStore] IsServiceConfigured: False
[2020-11-27 22:40:42Z INFO CommandSettings] Command 'run': 'True'
[2020-11-27 22:40:42Z INFO ConfigurationStore] IsConfigured()
[2020-11-27 22:40:42Z INFO ConfigurationStore] IsConfigured: True
[2020-11-27 22:40:42Z INFO ConfigurationManager] Is configured: True
[2020-11-27 22:40:42Z INFO Runner] Could not parse the argument value '' for StartupType. Defaulting to Manual
[2020-11-27 22:40:42Z INFO Runner] Set runner startup type - Manual
[2020-11-27 22:40:42Z INFO CommandSettings] Flag 'once': 'False'
[2020-11-27 22:40:42Z INFO Runner] RunAsync
[2020-11-27 22:40:42Z INFO ConfigurationManager] LoadSettings
[2020-11-27 22:40:42Z INFO ConfigurationStore] IsConfigured()
[2020-11-27 22:40:42Z INFO ConfigurationStore] IsConfigured: True
[2020-11-27 22:40:42Z INFO ConfigurationManager] Is configured: True
[2020-11-27 22:40:42Z INFO ConfigurationManager] Settings Loaded
[2020-11-27 22:40:42Z INFO MessageListener] {
  "AgentId": 62,
  "AgentName": "geiger",
  "PoolId": 1,
  "PoolName": "Default",
  "ServerUrl": "https://pipelines.actions.githubusercontent.com/eMc1GyGdYp1Pn3AIiLBt1AQy39VBhA0ak6WnGv2vWqR163Rx43",
  "GitHubUrl": "https://github.com/cppfw",
  "WorkFolder": "_work"
}
[2020-11-27 22:40:42Z INFO MessageListener] Loading Credentials
[2020-11-27 22:40:42Z INFO ConfigurationStore] HasCredentials()
[2020-11-27 22:40:42Z INFO ConfigurationStore] stored True
[2020-11-27 22:40:42Z INFO CredentialManager] GetCredentialProvider
[2020-11-27 22:40:42Z INFO CredentialManager] Creating type OAuth
[2020-11-27 22:40:42Z INFO CredentialManager] Creating credential type: OAuth
[2020-11-27 22:40:42Z INFO HostContext] Well known directory 'Bin': '/home/ivan/actions-runner/bin'
[2020-11-27 22:40:42Z INFO HostContext] Well known directory 'Root': '/home/ivan/actions-runner'
[2020-11-27 22:40:42Z INFO HostContext] Well known config file 'RSACredentials': '/home/ivan/actions-runner/.credentials_rsaparams'
[2020-11-27 22:40:42Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/ivan/actions-runner/.credentials_rsaparams
[2020-11-27 22:40:42Z INFO MessageListener] Attempt to create session.
[2020-11-27 22:40:42Z INFO MessageListener] Connecting to the Runner Server...
[2020-11-27 22:40:42Z INFO RunnerServer] Establish connection with 100 seconds timeout.
[2020-11-27 22:40:43Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-11-27 22:40:43Z INFO RunnerServer] Establish connection with 60 seconds timeout.
[2020-11-27 22:40:43Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-11-27 22:40:43Z INFO RunnerServer] Establish connection with 60 seconds timeout.
[2020-11-27 22:40:43Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-11-27 22:40:44Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-11-27 22:40:44Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-11-27 22:40:44Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-11-27 22:40:44Z INFO MessageListener] VssConnection created
[2020-11-27 22:40:44Z INFO Terminal] WRITE LINE: 
[2020-11-27 22:40:44Z INFO Terminal] WRITE LINE: 
[2020-11-27 22:40:44Z WARN GitHubActionsService] Authentication failed with status code 401.
WWW-Authenticate: Bearer
Strict-Transport-Security: max-age=2592000
X-TFS-ProcessId: 70fa918e-f5a5-4e0d-82b6-4722607071c4
ActivityId: 621134b9-ae5c-46e5-b2e7-e2d1049b425f
X-TFS-Session: 13947296-095b-42eb-aa57-eba4d3575a7b
X-VSS-E2EID: 8939e4f7-287c-4695-8b53-fae18f3e9259
X-VSS-SenderDeploymentId: a07ab14e-025a-39c3-8d53-788cd7ce207f
X-TFS-ServiceError: The+user+%27System%3aPublicAccess%3baaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa%27+is+not+authorized+to+access+this+resource.
X-VSS-S2STargetService: 0000005A-0000-8888-8000-000000000000/visualstudio.com
X-MSEdge-Ref: Ref A: CC099E07D0EB4B028AE159734D8C21E7 Ref B: FRAEDGE0913 Ref C: 2020-11-27T22:40:44Z
Date: Fri, 27 Nov 2020 22:40:43 GMT

[2020-11-27 22:40:44Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/ivan/actions-runner/.credentials_rsaparams
[2020-11-27 22:40:44Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/ivan/actions-runner/.credentials_rsaparams
[2020-11-27 22:40:44Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
[2020-11-27 22:40:45Z INFO MessageListener] Session created.
[2020-11-27 22:40:45Z INFO Terminal] WRITE LINE: 2020-11-27 22:40:45Z: Listening for Jobs
[2020-11-27 22:40:45Z INFO JobDispatcher] Set runner/worker IPC timeout to 30 seconds.
[2020-11-27 22:50:26Z INFO Terminal] WRITE LINE: Exiting...
[2020-11-27 22:50:26Z INFO ConfigurationStore] IsServiceConfigured()
[2020-11-27 22:50:26Z INFO ConfigurationStore] IsServiceConfigured: False
[2020-11-27 22:50:26Z INFO HostContext] Runner will be shutdown for UserCancelled
[2020-11-27 22:50:26Z WARN GitHubActionsService] GET request to https://pipelines.actions.githubusercontent.com/eMc1GyGdYp1Pn3AIiLBt1AQy39VBhA0ak6WnGv2vWqR163Rx43/_apis/distributedtask/pools/1/messages?sessionId=3b349ea5-34d0-4329-8077-81c7c7c6fb9a has been cancelled.
[2020-11-27 22:50:26Z INFO MessageListener] Get next message has been cancelled.
[2020-11-27 22:50:26Z INFO JobDispatcher] Shutting down JobDispatcher. Make sure all WorkerDispatcher has finished.
[2020-11-27 22:50:26Z INFO Listener] Runner execution been cancelled.
@igagis igagis added the bug Something isn't working label Nov 27, 2020
@igagis igagis mentioned this issue Nov 28, 2020
Closed
7 tasks
@tfenster
Copy link

tfenster commented Dec 9, 2020

Just to add: I am running into the same issue with runners on Windows

@dhoppe
Copy link

dhoppe commented Dec 19, 2020

I changed my account to an organisation so that I do not have to configure the self-hosted runners for each repository individually.

At the beginning I also had the error message mentioned above. Are you sure it is a private repository? The self-hosted runners are only allowed for private repositories for security reasons.
github_org_runners

The self-hosted runners are only displayed for private repositories.
Private repository:
github_private_repository
https://github.com/dhoppeIT/ansible-role-demo/actions/runs/432991963

Public repository:
github_public_repository
https://github.com/dhoppeIT/ansible-role-demo/actions/runs/432997701

I think the error message is a bit misleading. Instead of stating that no self-hosted runner was found with the label in question, it should be pointed out that self-hosted runners are only allowed for private repositories.

@igagis
Copy link
Author

igagis commented Dec 19, 2020

@dhoppe why do you think self-hosted renners are only allowed from private repos? Where is it stated?
As I wrote in the original bug description, in case I register my self-hoster runner to a single repo (not to the organization) then it does pick up the jobs. Aren't there security vulnerability in this case?
Yes, my repos are public, but I don't see reasons why single repo self-hosted runner works, but organization-wide one doesn't.

@dhoppe
Copy link

dhoppe commented Dec 19, 2020

https://docs.github.com/en/free-pro-team@latest/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories

Just take a look at my first screenshot.

All repositories, excluding public repositories

You could create a new group which also allows public repositories, but this requires a GitHub Enterprise account.

@igagis
Copy link
Author

igagis commented Dec 20, 2020

@dhoppe Thank you, that worked! I even was able to modify the Default group to allow self-hosted runners for public repos and I don't have Gihub Enterprise account, so it is not needed for that.

I think this issue can be closed then.

@igagis igagis closed this as completed Dec 20, 2020
@rajbos
Copy link
Contributor

rajbos commented Dec 21, 2021

Just got here after scratching my head on what was happening. Would be good to improve the error message in the logs to state this more clearly! So a job that would be eligible on that runner seems to trigger and then fails with this message hidden deep in the logs as a hint?

@dexterzep
Copy link

  1. go to your organization page.
  2. go to settings > actions > runner-groups
  3. click on self-hosted runners
  4. change "selected repos" to "all repos private or public"

image

@hasufell hasufell mentioned this issue Jan 14, 2023
Closed
@cboettig
Copy link

I have this same problem. I can get my self-hosted runners will work just fine on public repos using the fully permissive setup (Selecting "all repositories" and "all workflows"), my self-hosted runners do not deploy if I try restricting access to some repositories or workflows.

I've created a "runner group" with access to "selected public repos", and I can see the runner listed in the settings->actions->runner page of the public repo (screenshot below). But when I set runs-on: self-hosted it just stalls out "waiting for a runner to pick up this job". What did I miss?

image (2)

I also list Public-Infrastructure as the group under the runs-on section of the yaml (this seems redundant with the org-level settings configuration, but maybe both are necessary? Anyway no dice).

Only when using private repos, or otherwise, if I allowing runs on all public repos will my self-hosted runner pick up the jobs. What am I missing here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@cboettig @igagis @dhoppe @rajbos @tfenster @dexterzep