Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running together with kubernetes-plugin prevents ANSI Color from working #201

Closed
frezbo opened this issue Aug 10, 2020 · 15 comments · Fixed by #203
Closed

Running together with kubernetes-plugin prevents ANSI Color from working #201

frezbo opened this issue Aug 10, 2020 · 15 comments · Fixed by #203
Assignees
Labels
bug ❓ Possibly an ansicolor plugin bug
Milestone

Comments

@frezbo
Copy link

frezbo commented Aug 10, 2020

Here's my simple pipeline DSL:

pipeline {
  agent any

  options {
    ansiColor('xterm')
  }

  stages {
    stage('Run') {
      steps {
          echo '\033[32mcolored output\033[0m'
      }
    }
  }
}

When run on the leader (I changed the no of executors of leader to 1 for this), I get this output:

image

Reverting back the executors to zero so that the agent run on Kubernetes pods, this is the build output:

image

@dblock dblock added the bug ❓ Possibly an ansicolor plugin bug label Aug 10, 2020
@tszmytka
Copy link
Collaborator

Thank you for the screenshots, it would be however helpful if you could provide the details as requested in the issue template:

- Jenkins Version:
- AnsiColor Version:
- Operating System running Jenkins:

Since you are running this in k8s could you also specify:

- kubernetes-plugin version
- name and tag of the jenkins agent image you used to run this

Also could you try specifying your agent inside your pipeline like

agent {
    label 'my-agent';
}

instead of

agent any

and running the same again?

@frezbo
Copy link
Author

frezbo commented Aug 11, 2020

@tszmytka I think the agent section doesn't matter since all jobs runs in a kubernetes pod. To get a nomral output I just allowed the jenkins leaders to run jobs. Let me know if I want to explicitly do that. Extra info:

Jenkins version - 2.235.3
AnsiColor version - 0.7.2

The jenkins is running in a standard Amazon EKS cluster.

Kubernetes plugin version - 1.26.4

Jenkins leader image: jenkins/jenkins:lts
Jenkins agent image: jenkins/inbound-agent:4.3-4

@tszmytka
Copy link
Collaborator

the agent section doesn't matter

Maybe it won't but it's worth taking a shot. Please do that if you have the chance.

The thing is ansicolor v0.7.2 works with agents correctly. I have built countless jobs this way.
There might be some additional setting/another plugin/network configuration which interferes and causes this issue. We've just had a pretty long-running issue where timestamper-plugin turned out to be the offending party - luckily you don't seem to be using it. Can you think of any exotic or lesser-popular plugins you are utilizing that may influence log output?

Just to make sure I've set up a clean environment with:

Jenkins: jenkins/jenkins:2.235.3-lts-alpine
AnsiColor: 0.7.2
Agent: jenkins/inbound-agent:4.3-4-alpine 

and the pipeline you provided renders green output. I connected the agent directly (no k8s cluster) but other elements are the same.

I noticed that your screenshot shows log output before Running on jenkins-agent-wb19c.
Could you post the entire thing (from Started by to Finished)?

@frezbo
Copy link
Author

frezbo commented Aug 11, 2020

Here's all the plugin's that are installed:

- ansicolor:latest
- artifactory:latest
- authorize-project:latest
- configuration-as-code:latest
- envinject:latest
- git:latest
- job-dsl:latest
- kubernetes-credentials-provider:latest
- kubernetes:latest
- pipeline-cps-http:latest
- rebuild:latest
- ssh-agent:latest
- uno-choice:latest
- workflow-aggregator:latest
- workflow-cps-global-lib-http:latest
- workflow-cps-global-lib:latest
- htmlpublisher:latest

Here's the complete job output:

Started by user admin
Running as admin
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node

Still waiting to schedule task
Waiting for next available executor

Agent jenkins-agent-bz40t is provisioned from template jenkins-agent
---
apiVersion: "v1"
kind: "Pod"
metadata:
  annotations: {}
  labels:
    jenkins/jenkins-frezbo-jenkins-slave: "true"
    jenkins/label: "jenkins-frezbo-jenkins-slavex"
  name: "jenkins-agent-bz40t"
spec:
  containers:
  - command:
    - "cat"
    env:
    - name: "DOCKER_HOST"
      value: "tcp://dind:2375"
    image: "<redacted>:2.8.18"
    imagePullPolicy: "Always"
    name: "dev"
    securityContext:
      runAsUser: 1000
    tty: true
    volumeMounts:
    - mountPath: "/home/ec2-user/.gitconfig"
      name: "git-config"
      subPath: ".gitconfig"
    - mountPath: "/home/ec2-user/.ssh/config"
      name: "ssh-config"
      subPath: "config"
    - mountPath: "/home/ec2-user/.gemrc"
      name: "gem-sources"
      subPath: ".gemrc"
    - mountPath: "/home/ec2-user/.bundle/config"
      name: "bundle-config"
      subPath: "config"
    - mountPath: "/home/jenkins"
      name: "workspace-volume"
      readOnly: false
    workingDir: "/home/jenkins"
  - args:
    - "********"
    - "jenkins-agent-bz40t"
    env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "jenkins-frezbo-agent.frezbo:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "jenkins-agent-bz40t"
    - name: "AGENT_NAME"
      value: "dev-env"
    - name: "JENKINS_NAME"
      value: "jenkins-agent-bz40t"
    - name: "JENKINS_AGENT_WORKDIR"
      value: "/home/jenkins"
    - name: "JENKINS_URL"
      value: "http://jenkins-frezbo.frezbo.svc.cluster.local:8080"
    image: "<redacted>.<redacted>/jenkins/inbound-agent:4.3-4"
    imagePullPolicy: "Always"
    name: "jnlp"
    resources:
      limits:
        memory: "512Mi"
        cpu: "512m"
      requests:
        memory: "512Mi"
        cpu: "512m"
    securityContext:
      privileged: false
      runAsUser: 1000
    tty: false
    volumeMounts:
    - mountPath: "/home/jenkins"
      name: "workspace-volume"
      readOnly: false
    workingDir: "/home/jenkins"
  dnsPolicy: "ClusterFirstWithHostNet"
  hostNetwork: true
  imagePullSecrets:
  - name: "jenkins-frezbo-agent-docker-secret"
  nodeSelector:
    node-role.kubernetes.io/jenkins-agent: "true"
  restartPolicy: "Never"
  serviceAccount: "jenkins-frezbo-agent"
  volumes:
  - configMap:
      defaultMode: 420
      name: "jenkins-frezbo-git-config"
    name: "git-config"
  - name: "bundle-config"
    secret:
      defaultMode: 420
      secretName: "jenkins-frezbo-bundle-config"
  - name: "gem-sources"
    secret:
      defaultMode: 420
      secretName: "jenkins-frezbo-gem-sources"
  - configMap:
      defaultMode: 420
      name: "jenkins-frezbo-ssh-config"
    name: "ssh-config"
  - emptyDir:
      medium: ""
    name: "workspace-volume"

Running on jenkins-agent-bz40t in /home/jenkins/workspace/spock
[Pipeline] {
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (Run)
[Pipeline] echo
�[32mcolored output�[0m
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

@ryancurrah
Copy link

We are seeing the same issue

@teomanS
Copy link

teomanS commented Aug 14, 2020

I have the same issue.

My environment is;
Jenkins version : 2.235.3
AnsiColor plugin version: 0.7.2
Openshift version: 4.5.5

@tszmytka
Copy link
Collaborator

@frezbo
I've installed the plugins from your list and still can't reproduce.

I would appreciate 2 additional pieces of information:

  1. List of plugins with the actual version and whether they are enabled visible under Plugins at your.jenkins.url/systemInfo
  2. Screenshot of the Configure Global Security page available at your.jenkins.url/configureSecurity. You can redact any information you don't want to share but the page doesn't show passwords or tokens etc.

@frezbo
Copy link
Author

frezbo commented Aug 14, 2020

@tszmytka the jenkins is completely configured with JcaSC, should I sent that config file over or do you want the screenshots? I need to spin up a new one, so please be patient. I might not be able to get to it until monday. Thanks for the help.

@tszmytka
Copy link
Collaborator

@tszmytka the jenkins is completely configured with JcaSC, should I sent that config file over or do you want the screenshots? I need to spin up a new one, so please be patient. I might not be able to get to it until monday. Thanks for the help.

Whatever is easier for you. JcaSC config file would also be fine - provided you are not changing anything manually after initial setup ;-)

@frezbo
Copy link
Author

frezbo commented Aug 14, 2020

The plugins list are the same. This is the jcasc url, the values in ${} gets replaced. There is no manul config after jenkins starts back up, except creating the test job mentioned here. I will try to get the screenshots after the weekend.

jenkins:
  systemMessage: "Jenkins"
  securityRealm:
    local:
      allowsSignup: false
      users:
        - id: "${admin_username}"
          password: "${admin_password}"
  authorizationStrategy:
    loggedInUsersCanDoAnything:
      allowAnonymousRead: false
security:
  envInject:
    enableLoadingFromMaster: false
    enablePermissions: true
    hideInjectedVars: true
  globalJobDslSecurityConfiguration:
    useScriptSecurity: true
  # eventually we need to use project specific authorization and have the seed jobs run as admin user
  queueItemAuthenticator:
    authenticators:
    - global:
        strategy: "triggeringUsersAuthorizationStrategy"
unclassified:
  artifactorybuilder:
    useCredentialsPlugin: true
    artifactoryServers:
      - serverId: artifactory
        artifactoryUrl: ${artifactory_endpoint}
        deployerCredentialsConfig:
          credentialsId: "${artifactory_credentials_id}"
          overridingCredentials: false
  globalLibraries:
    libraries:
      - defaultVersion: "${jenkins_library_version}"
        name: "${jenkins_library_name}"
        retriever:
          http:
            credentialsId: "${artifactory_credentials_id}"
            httpURL: "${artifactory_endpoint}/${jenkins_library_artifactory_repo}/${jenkins_library_remote_repo_path}/^$${library.${jenkins_library_name}.version}/${jenkins_library_remote_repo_path}-groovy-^$${library.${jenkins_library_name}.version}.zip"
            preemptiveAuth: true
  resourceRoot:
    url: "${jenkins_assets_url}"
jobs:
  - file: /var/jenkins_home/casc_configs/seeds.groovy

@Kryan90
Copy link

Kryan90 commented Aug 21, 2020

We are seeing the same issue with ansible output on k8s agents

sample output

[0;35m[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to �[0m
�[0;35mallow bad characters in group names by default, this will change, but still be �[0m
�[0;35muser configurable on deprecation. This feature will be removed in version 2.10.�[0m
�[0;35m Deprecation warnings can be disabled by setting deprecation_warnings=False in �[0m
�[0;35mansible.cfg.�[0m
�[1;35m [WARNING]: Invalid characters were found in group names but not replaced, use�[0m
�[1;35m-vvvv to see details�[0m
�[1;35m�[0m

@atikhono
Copy link

Hi, same issue in my Jenkins installation.
Jenkins 2.222.4
ansiColor 0.7.2 (timestamper plugin is not installed)
kubernetes-plugin 1.26.4

In Jenkins master (running in Kubernetes as well), ansiColor step works as expected. In a Kubernetes agent, there're control characters like reported above instead of colorized output.

@tszmytka
Copy link
Collaborator

I finally got to setting up a totally fresh Jenkins instance together with a k8s cluster using the same setup @frezbo provided. This indeed makes logs show raw escape sequences.

After digging a bit deeper and comparing different run results it turned out that the reason lies in kubernetes-plugin and is very similar to #193. The plugin swallows up some ConsoleNotes and prevents ansicolor from working correctly.

A fix should be available in the coming days.

There is a way to get back colored output immediately though (at least for the time being) - install timestamper-plugin (you don't have to use it in any way). It triggers the workaround implemented for #193 and it will also help in this case until a new version is out.

@tszmytka tszmytka changed the title ANSI Color not working with Kubernetes agents Running together with kubernetes-plugin prevents ANSI Color from working Aug 24, 2020
@tszmytka tszmytka added this to the v0.7.3 milestone Aug 24, 2020
@tszmytka tszmytka self-assigned this Aug 24, 2020
@mrdima
Copy link

mrdima commented Aug 25, 2020

Thanks @tszmytka We have the same issue and I can confirm that the workaround of installing the timestamper-plugin works fine for us. Thanks for that...good to have that til the fix is available!

@Kryan90
Copy link

Kryan90 commented Sep 3, 2020

I can also confirm the workaround fixes ansible color output. Thanks!

tszmytka added a commit to tszmytka/ansicolor-plugin that referenced this issue Sep 10, 2020
… for kubernetes plugin. Test integration with timestamper
tszmytka added a commit to tszmytka/ansicolor-plugin that referenced this issue Sep 10, 2020
…tificial NL to allow ansicolor to work despite. Add tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug ❓ Possibly an ansicolor plugin bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants