Running together with kubernetes-plugin prevents ANSI Color from working

[frezbo]

Here's my simple pipeline DSL:

pipeline {
  agent any

  options {
    ansiColor('xterm')
  }

  stages {
    stage('Run') {
      steps {
          echo '\033[32mcolored output\033[0m'
      }
    }
  }
}

When run on the leader (I changed the no of executors of leader to 1 for this), I get this output:

Reverting back the executors to zero so that the agent run on Kubernetes pods, this is the build output:

[tszmytka]

Thank you for the screenshots, it would be however helpful if you could provide the details as requested in the issue template:

- Jenkins Version:
- AnsiColor Version:
- Operating System running Jenkins:

Since you are running this in k8s could you also specify:

- kubernetes-plugin version
- name and tag of the jenkins agent image you used to run this

Also could you try specifying your agent inside your pipeline like

agent {
    label 'my-agent';
}

instead of

agent any

and running the same again?

[frezbo]

@tszmytka I think the agent section doesn't matter since all jobs runs in a kubernetes pod. To get a nomral output I just allowed the jenkins leaders to run jobs. Let me know if I want to explicitly do that. Extra info:

Jenkins version - 2.235.3
AnsiColor version - 0.7.2

The jenkins is running in a standard Amazon EKS cluster.

Kubernetes plugin version - 1.26.4

Jenkins leader image: jenkins/jenkins:lts
Jenkins agent image: jenkins/inbound-agent:4.3-4

[tszmytka]

the agent section doesn't matter

Maybe it won't but it's worth taking a shot. Please do that if you have the chance.

The thing is ansicolor v0.7.2 works with agents correctly. I have built countless jobs this way.
There might be some additional setting/another plugin/network configuration which interferes and causes this issue. We've just had a pretty long-running issue where timestamper-plugin turned out to be the offending party - luckily you don't seem to be using it. Can you think of any exotic or lesser-popular plugins you are utilizing that may influence log output?

Just to make sure I've set up a clean environment with:

Jenkins: jenkins/jenkins:2.235.3-lts-alpine
AnsiColor: 0.7.2
Agent: jenkins/inbound-agent:4.3-4-alpine 

and the pipeline you provided renders green output. I connected the agent directly (no k8s cluster) but other elements are the same.

I noticed that your screenshot shows log output before Running on jenkins-agent-wb19c.
Could you post the entire thing (from Started by to Finished)?

[frezbo]

Here's all the plugin's that are installed:

- ansicolor:latest
- artifactory:latest
- authorize-project:latest
- configuration-as-code:latest
- envinject:latest
- git:latest
- job-dsl:latest
- kubernetes-credentials-provider:latest
- kubernetes:latest
- pipeline-cps-http:latest
- rebuild:latest
- ssh-agent:latest
- uno-choice:latest
- workflow-aggregator:latest
- workflow-cps-global-lib-http:latest
- workflow-cps-global-lib:latest
- htmlpublisher:latest

Here's the complete job output:

Started by user admin
Running as admin
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node

Still waiting to schedule task
Waiting for next available executor

Agent jenkins-agent-bz40t is provisioned from template jenkins-agent
---
apiVersion: "v1"
kind: "Pod"
metadata:
  annotations: {}
  labels:
    jenkins/jenkins-frezbo-jenkins-slave: "true"
    jenkins/label: "jenkins-frezbo-jenkins-slavex"
  name: "jenkins-agent-bz40t"
spec:
  containers:
  - command:
    - "cat"
    env:
    - name: "DOCKER_HOST"
      value: "tcp://dind:2375"
    image: "<redacted>:2.8.18"
    imagePullPolicy: "Always"
    name: "dev"
    securityContext:
      runAsUser: 1000
    tty: true
    volumeMounts:
    - mountPath: "/home/ec2-user/.gitconfig"
      name: "git-config"
      subPath: ".gitconfig"
    - mountPath: "/home/ec2-user/.ssh/config"
      name: "ssh-config"
      subPath: "config"
    - mountPath: "/home/ec2-user/.gemrc"
      name: "gem-sources"
      subPath: ".gemrc"
    - mountPath: "/home/ec2-user/.bundle/config"
      name: "bundle-config"
      subPath: "config"
    - mountPath: "/home/jenkins"
      name: "workspace-volume"
      readOnly: false
    workingDir: "/home/jenkins"
  - args:
    - "********"
    - "jenkins-agent-bz40t"
    env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "jenkins-frezbo-agent.frezbo:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "jenkins-agent-bz40t"
    - name: "AGENT_NAME"
      value: "dev-env"
    - name: "JENKINS_NAME"
      value: "jenkins-agent-bz40t"
    - name: "JENKINS_AGENT_WORKDIR"
      value: "/home/jenkins"
    - name: "JENKINS_URL"
      value: "http://jenkins-frezbo.frezbo.svc.cluster.local:8080"
    image: "<redacted>.<redacted>/jenkins/inbound-agent:4.3-4"
    imagePullPolicy: "Always"
    name: "jnlp"
    resources:
      limits:
        memory: "512Mi"
        cpu: "512m"
      requests:
        memory: "512Mi"
        cpu: "512m"
    securityContext:
      privileged: false
      runAsUser: 1000
    tty: false
    volumeMounts:
    - mountPath: "/home/jenkins"
      name: "workspace-volume"
      readOnly: false
    workingDir: "/home/jenkins"
  dnsPolicy: "ClusterFirstWithHostNet"
  hostNetwork: true
  imagePullSecrets:
  - name: "jenkins-frezbo-agent-docker-secret"
  nodeSelector:
    node-role.kubernetes.io/jenkins-agent: "true"
  restartPolicy: "Never"
  serviceAccount: "jenkins-frezbo-agent"
  volumes:
  - configMap:
      defaultMode: 420
      name: "jenkins-frezbo-git-config"
    name: "git-config"
  - name: "bundle-config"
    secret:
      defaultMode: 420
      secretName: "jenkins-frezbo-bundle-config"
  - name: "gem-sources"
    secret:
      defaultMode: 420
      secretName: "jenkins-frezbo-gem-sources"
  - configMap:
      defaultMode: 420
      name: "jenkins-frezbo-ssh-config"
    name: "ssh-config"
  - emptyDir:
      medium: ""
    name: "workspace-volume"

Running on jenkins-agent-bz40t in /home/jenkins/workspace/spock
[Pipeline] {
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (Run)
[Pipeline] echo
�[32mcolored output�[0m
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

[ryancurrah]

We are seeing the same issue

[teomanS]

I have the same issue.

My environment is;
Jenkins version : 2.235.3
AnsiColor plugin version: 0.7.2
Openshift version: 4.5.5

[tszmytka]

@frezbo
I've installed the plugins from your list and still can't reproduce.

I would appreciate 2 additional pieces of information:

1. List of plugins with the actual version and whether they are enabled visible under Plugins at your.jenkins.url/systemInfo
2. Screenshot of the Configure Global Security page available at your.jenkins.url/configureSecurity. You can redact any information you don't want to share but the page doesn't show passwords or tokens etc.

[frezbo]

@tszmytka the jenkins is completely configured with JcaSC, should I sent that config file over or do you want the screenshots? I need to spin up a new one, so please be patient. I might not be able to get to it until monday. Thanks for the help.

[tszmytka]

@tszmytka the jenkins is completely configured with JcaSC, should I sent that config file over or do you want the screenshots? I need to spin up a new one, so please be patient. I might not be able to get to it until monday. Thanks for the help.

Whatever is easier for you. JcaSC config file would also be fine - provided you are not changing anything manually after initial setup ;-)

[frezbo]

The plugins list are the same. This is the jcasc url, the values in ${} gets replaced. There is no manul config after jenkins starts back up, except creating the test job mentioned here. I will try to get the screenshots after the weekend.

jenkins:
  systemMessage: "Jenkins"
  securityRealm:
    local:
      allowsSignup: false
      users:
        - id: "${admin_username}"
          password: "${admin_password}"
  authorizationStrategy:
    loggedInUsersCanDoAnything:
      allowAnonymousRead: false
security:
  envInject:
    enableLoadingFromMaster: false
    enablePermissions: true
    hideInjectedVars: true
  globalJobDslSecurityConfiguration:
    useScriptSecurity: true
  # eventually we need to use project specific authorization and have the seed jobs run as admin user
  queueItemAuthenticator:
    authenticators:
    - global:
        strategy: "triggeringUsersAuthorizationStrategy"
unclassified:
  artifactorybuilder:
    useCredentialsPlugin: true
    artifactoryServers:
      - serverId: artifactory
        artifactoryUrl: ${artifactory_endpoint}
        deployerCredentialsConfig:
          credentialsId: "${artifactory_credentials_id}"
          overridingCredentials: false
  globalLibraries:
    libraries:
      - defaultVersion: "${jenkins_library_version}"
        name: "${jenkins_library_name}"
        retriever:
          http:
            credentialsId: "${artifactory_credentials_id}"
            httpURL: "${artifactory_endpoint}/${jenkins_library_artifactory_repo}/${jenkins_library_remote_repo_path}/^$${library.${jenkins_library_name}.version}/${jenkins_library_remote_repo_path}-groovy-^$${library.${jenkins_library_name}.version}.zip"
            preemptiveAuth: true
  resourceRoot:
    url: "${jenkins_assets_url}"
jobs:
  - file: /var/jenkins_home/casc_configs/seeds.groovy

[Kryan90]

We are seeing the same issue with ansible output on k8s agents

sample output

[0;35m[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to �[0m
�[0;35mallow bad characters in group names by default, this will change, but still be �[0m
�[0;35muser configurable on deprecation. This feature will be removed in version 2.10.�[0m
�[0;35m Deprecation warnings can be disabled by setting deprecation_warnings=False in �[0m
�[0;35mansible.cfg.�[0m
�[1;35m [WARNING]: Invalid characters were found in group names but not replaced, use�[0m
�[1;35m-vvvv to see details�[0m
�[1;35m�[0m

[atikhono]

Hi, same issue in my Jenkins installation.
Jenkins 2.222.4
ansiColor 0.7.2 (timestamper plugin is not installed)
kubernetes-plugin 1.26.4

In Jenkins master (running in Kubernetes as well), ansiColor step works as expected. In a Kubernetes agent, there're control characters like reported above instead of colorized output.

[tszmytka]

I finally got to setting up a totally fresh Jenkins instance together with a k8s cluster using the same setup @frezbo provided. This indeed makes logs show raw escape sequences.

After digging a bit deeper and comparing different run results it turned out that the reason lies in kubernetes-plugin and is very similar to #193. The plugin swallows up some ConsoleNotes and prevents ansicolor from working correctly.

A fix should be available in the coming days.

There is a way to get back colored output immediately though (at least for the time being) - install timestamper-plugin (you don't have to use it in any way). It triggers the workaround implemented for #193 and it will also help in this case until a new version is out.

[mrdima]

Thanks @tszmytka We have the same issue and I can confirm that the workaround of installing the timestamper-plugin works fine for us. Thanks for that...good to have that til the fix is available!

[Kryan90]

I can also confirm the workaround fixes ansible color output. Thanks!