Executing a Pipeline with a single Task results in multiple TaskRuns #3126
Comments
|
This might be related to #2813 🤔 |
yep!
hmm in that one, a container within a pod was restarted, im not sure that would be caused by the same thing, since this sounds like the controller is deciding to create more than one TaskRun @Fabian-K do you happen to have the PipelineRun status information available? it would be interesting to see if it included more information, e.g. did it include just the one TaskRun or both Also does this happen consistently or just sometimes? |
|
Thanks for the quick reply. This is not the first time I noticed multiple taskrun creations but it´s difficult to say how often this happens - also because this particular pipeline can be executed multiple times without an issue so it might also happen unnoticed. Background Info: We also run https://github.com/fluxcd/flux in the cluster. Flux is creating the Pipeline and the Task objects - but should not do anything related to PipelineRuns. Those are created via the Kubernetes API. I´m currently thinking if some of the flux-related annotations/labels were added by the flux controller (?) after the initial creation and that this modification causes an unexpected reconciliation which with bad timing results in multiple taskruns being created? 🤔 Although this is just a guess, not sure if the annotations are added after creation or if they are propagated by tekton from the pipeline?! Here is the PipelineRun: |
|
interesting, it looks like uiveri5-lunchmenu-1598256551145-uiveri5-d7fkh ended up in the status and not uiveri5-lunchmenu-1598256551145-uiveri5-d7fkh i wonder if something could happen that interrupts the reconciler from updating the status and so it thinks it needs to create another taskrun?? very curious!! |
|
The log shows that the pipelinerun reconciler issued a "started" event for the pipelinerun again as well before creating the duplicate TaskRun. This confirms that it is reading "old" PR status. The log does not show any errors updating the status (and genreconciler code would log that). The pipelinerun reconciler has a function called As far as why only one TaskRun ends up in the PR's status, this is because the pipelineState can hold only one TaskRun per pipeline task so one of them wins. The pipelineState is used to completely rewrite the PR's TaskRun statuses at the end of the reconcile. cc @afrittoli |
ghost
added
the
priority/critical-urgent
|
I've been trying to reproduce this but am struggling to do so. I've rolled my cluster back to Pipelines 0.15.2 and then written up some resources:
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: test-task
spec:
params:
- name: user-secret
type: string
resources:
inputs:
- name: workspace
type: git
steps:
- name: uiveri5
image: alpine:3.12.0
script: |
echo "$(params.user-secret)"
cd /workspace/workspace
ls
sleep 20
exit 1
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
name: test-pipeline
spec:
params:
- description: Secret containing the user for test execution
name: user-secret
type: string
resources:
- name: source-repo
type: git
tasks:
- name: uiveri5
taskRef:
name: test-task
params:
- name: user-secret
value: $(params.user-secret)
resources:
inputs:
- name: workspace
resource: source-repo
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
generateName: test-pr
spec:
pipelineRef:
name: test-pipeline
params:
- name: user-secret
value: foo
resources:
- name: source-repo
resourceSpec:
type: git
params:
- name: url
value: https://github.com/tektoncd/pipeline.git
for i in {1..100} ; do kubectl create -f ./test.yaml ; done
kubectl get pipelineruns | wc -l
kubectl get taskruns | wc -lI've repeated this test maybe 20 times and haven't been able to repro. I'm running on GKE with a 1.16 server so I'm going through the steps now to upgrade to a 1.18 server to match the original post and see if that makes a difference. @Fabian-K which platform are you running on here? Is it a bare metal cluster or something else? |
|
I've updated to k8s 1.18 and have written a script to try and replicate the annotation/label behaviour of fluxcd described in #3126 (comment) #!/usr/bin/env bash
set -x
for i in {1..40} ; do
kubectl create -f ./test.yaml
done
for i in {1..5} ; do
runs=$(kubectl get taskruns --no-headers | awk '{ print $1 }')
for run in $runs ; do
value="$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 8 | head -n 1)"
kubectl annotate --overwrite taskrun $run foo="$value"
value="$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 8 | head -n 1)"
kubectl label --overwrite taskrun $run bar="$value"
done
sleep 2
doneI've run this maybe 10 times and haven't been able to trigger a mismatch in taskrun count vs pipelinerun count. So I'm clearly missing something here in the reproduction of the bad behaviour. 😕 |
|
thanks a bunch @sbwsg 🙏 I think the best would be hear back from @Fabian-K with the platform details. Also, if @Fabian-K can provide exact pipeline with task and resources used, might help troubleshoot this further. The reason being the pipeline in the description has some @Fabian-K it would be great if you can reproduce this on 0.18 since 0.18 has timestamps in the controller logs which would be again helpful. |
|
I am seeing this behaviour as well as commented in #3510. I'm running OpenShift Pipelines Technology Preview 1.1, which means:
|
|
Thanks for your effort in reproducing the issue! The cluster is a managed cluster by https://gardener.cloud/ with OpenStack as the infrastructure provider. I also tried to reproduce it locally using kind but so far did not manage to do so :( We are updating our two instances to 0.18 soon. Will definitely look for new occurrences and try to provide controller logs. Not sure if this is helpful but it seems to me that this tends to happen in namespaces with a lot of PipelineRuns / TaskRuns (in our case around 2000 instances). |
|
thanks @Fabian-K @psschwei did you notice any such odd behavior (multiple taskruns getting created for a pipeline with a single task) while running the stress test? |
Not that I recall, although it wasn't something I was explicitly watching for either. |
|
In my case it is not related to stress at all. Happens from the very first PipelineRun in a namespace. |
|
@michaelsauter Can you provide your pipeline controller log for the period where the duplicate taskruns are created? |
|
I've tried to run some tests and I can't recreate a duplicated TaskRun. This included several thousand runs on OpenShift 4.5 (== Kubernetes 0.18 and Tekton 0.14.3). Out of curiosity I ran an instrumented PipelineRun controller to see how quickly the informer distributes added/updated *Run objects. I was surprised to see it happens very quickly. The informer consistently notified it of a new TaskRun created by the Since I still suspect it's a problem with the informer cache, I started looking around for more details and guidance about dealing with stale objects and found this: The first piece of advice, "use deterministic names for objects you create", got me to wondering about how the PipelineRun reconciler generates random suffixes for the TaskRuns. It has to generate a suffix because the Pipeline task name might get truncated and no longer be unique. However I don't think the suffix needs to be random -- just unique within the PipelineRun. It could use the index of the pipeline task which would be unique and deterministic. I believe this should work whether the PipelineRun's name is generated or not (i.e. uses If we did this then as the article suggests the PipelineRun reconciler would get an error when it tried to create a duplicate TaskRun. Theoretically it would just skip creating that TaskRun and go on normally. It would trust that the TaskRun would eventually show up in the cache. (I guess we could debate how much trust it should have in that.) Food for thought anyway. I am curious if anyone knows why the TaskRuns should have their own random suffix, not just a unique-within-the-pipeline suffix. |
|
Wow, thanks for digging further into this @GregDritschler! Your theory regarding the cache makes sense to me and I think your proposed solution would afford us an opportunity to introduce better logging when the creation happens but fails. This would in turn validate whether the theory is on the money.
We'd need to account for retries of Tasks as well I think? Otherwise Task N might fail, retry, but be blocked from creating the new TaskRun due to name collision? So maybe <PR-name>-<Task-Index>-<Task-Retry-Count>?
Other than the retries thing I'm not aware of another reason that we'd require a random suffix in PipelineRun TaskRuns. |
|
Retries reuse the same TaskRun. They just recreate a new pod from the same TaskRun. The new pod will still have its own random suffix. This does make me wonder if the TaskRun reconciler has a duplication exposure as well, i.e. it creates a pod, gets re-driven again with stale status that doesn't have the pod name, and creates a duplicate pod. I don't recall seeing any reports of that happening. If it did happen, I guess the TaskRun reconciler could be changed to generate the pod suffix deterministically. It can compute the retry number by looking at the retriesStatus. That comment in pod.go indicates the choice to use a random suffix was "to avoid confusion when a build is deleted and re-created with the same name." I suspect that's what drove the use of a random suffix for the TaskRuns too. I'm not sure I buy the argument. When you delete a run, it deletes the owned objects as well. So the only objects that exist at a given time for a given run name are the ones from the latest run. I'm not sure if that's "confusing". Regardless I hope these days everyone uses generateName in their runs anyway. |
|
I thought about this some more and found a case where the random suffix for the TaskRun comes into play. This happens when the name of the PipelineRun itself is the max length of 64 characters. I created this PipelineRun: This resulted in a PipelineRun name of So this puts a monkey wrench in my idea about using predictable TaskRun names. I suppose the PipelineRun name could be truncated in some way to preserve its unique suffix but that makes me worry that it introduces a new exposure that that name might not be unique anymore. (That is, the name created by generateName is unique across PipelineRuns, but a truncated version of that may not be.) |
|
Finally, here are my Tekton controller logs: https://gist.github.com/michaelsauter/9af328156693e9f615cb1fa699fdd69f. There are some errors like |
|
The gist https://gist.github.com/Fabian-K/10f46e5b0cd23318985e5d8f3c7b5bc2 contains one occurrence using Tekton 0.18.0
|
|
I finally was able to recreate the problem on OpenShift with Tekton 0.18 (with some debugging statements added). I ran a script to repeatedly create a PipelineRun and then update its labels. It took several hundred runs but I finally got a PR to create a duplicate TR. The log clearly shows the reconciler being driven with state status, the TR not being found in the lister cache, and the reconciler creating a duplicate TR. I'm going to bring the problem up in the next API WG. |
@Fabian-K This is in fact the problem. The reconciler is using all of the PipelineRun's labels (which includes labels propagated from the Pipeline) to look for TaskRuns associated with the PipelineRun. If the labels change in any way, it won't find the TaskRuns that it created prior to that change and it will create duplicate TaskRuns. I have linked a pull request that changes the code to use only the @itewk @michaelsauter Please check if anything is modifying Pipeline or PipelineRun labels while the PipelineRun is running. If not, then we will have to keep digging into your scenarios. |
|
@GregDritschler At least for me, I just followed https://github.com/openshift/pipelines-tutorial. Nothing special, and there is nothing else on that cluster that would cause modifications to labels. I have another cluster available now though, with the same versions, and I can try there again ... |
|
Just as a FYI: I'm now on a OpenShift 4.6 cluster which provides OpenShift Pipelines 1.2 instead of 1.1. On 1.2, I cannot reproduce the problem above. |
|
i am on OCP 4.6 with Pipelines Operator 1.2.2 and deffinitly still seeing this. I thought this fix was gonna be in 1.3 of the operator based on how versions are matched up. |
I Have reproduce this bug on Pipeline version: v0.25.0 |
|
Similar issue is reported in #3557 |
|
@Rand01ph I have added this item in the API WG to discuss, is there anyway you can provide more details which we can discuss in the WG? It will be great if you can attend the discussion. If not, I will appreciate any more details such as, screen shots, screen recording or anything else 🙏 |
Yes, I'm very happly to join your discuss. first create taskrun{"level":"info","ts":"2021-10-18T09:03:51.721Z","logger":"tekton-pipelines-controller.event-broadcaster","caller":"record/event.go:282","msg":"Event(v1.ObjectReference{Kind:"PipelineRun", Namespace:"tyw-xxxx-api", Name:"1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41", UID:"3c30044d-1b1b-41dc-b9dd-632c8bcb5269", APIVersion:"tekton.dev/v1beta1", ResourceVersion:"114893338", FieldPath:""}): type: 'Normal' reason: 'Started' ","commit":"584b527"} {"level":"info","ts":"2021-10-18T09:03:51.729Z","logger":"tekton-pipelines-controller","caller":"pipelinerun/pipelinerun.go:353","msg":"No.1 pipelineMeta is: &ObjectMeta{Name:3b8d0ae68a6fc5c95a91570e9528807c9eb2f164,GenerateName:,Namespace:tyw-xxxx-api,SelfLink:/apis/tekton.dev/v1beta1/namespaces/tyw-xxxx-api/pipelines/3b8d0ae68a6fc5c95a91570e9528807c9eb2f164,UID:29481e0e-19e9-43fb-b036-f2e87a71bc2f,ResourceVersion:114893337,Generation:1,CreationTimestamp:2021-10-18 09:03:51 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[]OwnerReference{},Finalizers:[],ClusterName:,ManagedFields:[]ManagedFieldsEntry{ManagedFieldsEntry{Manager:OpenAPI-Generator,Operation:Update,APIVersion:tekton.dev/v1beta1,Time:2021-10-18 09:03:51 +0000 UTC,FieldsType:FieldsV1,FieldsV1:{"f:spec":{".":{},"f:description":{},"f:tasks":{},"f:workspaces":{}}},},},}, pipelineSpec is: &{Description:测试clone的问题123 Resources:[] Tasks:[{Name:3d8e52746c39bdaf383fc1026d2ebd56816b9ca8 TaskRef:0xc000dd6000 TaskSpec: Conditions:[] WhenExpressions:[] Retries:0 RunAfter:[] Resources: Params:[] Workspaces:[{Name:xxxx Workspace:shared-data SubPath:}] Timeout:nil}] Params:[] Workspaces:[{Name:shared-data Description: Optional:false}] Results:[] Finally:[]}","commit":"584b527","knative.dev/controller":"github.com.tektoncd.pipeline.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","knative.dev/traceid":"57fe491d-3862-41bf-9b53-116cff25ab80","knative.dev/key":"tyw-xxxx-api/1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41"} {"level":"info","ts":"2021-10-18T09:03:51.738Z","logger":"tekton-pipelines-controller","caller":"pipelinerun/pipelinerun.go:493","msg":"No.2 resolvePipelineState is: &[0xc0009b3ce0]","commit":"584b527","knative.dev/controller":"github.com.tektoncd.pipeline.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","knative.dev/traceid":"57fe491d-3862-41bf-9b53-116cff25ab80","knative.dev/key":"tyw-xxxx-api/1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41"} {"level":"info","ts":"2021-10-18T09:03:51.757Z","logger":"tekton-pipelines-controller","caller":"volumeclaim/pvchandler.go:67","msg":"Created PersistentVolumeClaim pvc-9b4da94f2e in namespace tyw-xxxx-api","commit":"584b527"} {"level":"info","ts":"2021-10-18T09:03:51.777Z","logger":"tekton-pipelines-controller","caller":"pipelinerun/affinity_assistant.go:66","msg":"Created StatefulSet affinity-assistant-819ff3551e in namespace tyw-xxxx-api","commit":"584b527","knative.dev/controller":"github.com.tektoncd.pipeline.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","knative.dev/traceid":"57fe491d-3862-41bf-9b53-116cff25ab80","knative.dev/key":"tyw-xxxx-api/1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41"} {"level":"info","ts":"2021-10-18T09:03:51.777Z","logger":"tekton-pipelines-controller","caller":"pipelinerun/pipelinerun.go:771","msg":"Creating a new TaskRun object 1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41-3d8e52746c-7w4sq for pipeline task 3d8e52746c39bdaf383fc1026d2ebd56816b9ca8","commit":"584b527","knative.dev/controller":"github.com.tektoncd.pipeline.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","knative.dev/traceid":"57fe491d-3862-41bf-9b53-116cff25ab80","knative.dev/key":"tyw-xxxx-api/1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41"} {"level":"info","ts":"2021-10-18T09:03:51.850Z","logger":"tekton-pipelines-controller","caller":"pipelinerun/pipelinerun.go:601","msg":"PipelineRun 1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41 status is being set to &{Succeeded Unknown {2021-10-18 09:03:51.85019029 +0000 UTC m=+440512.276097640} Running Tasks Completed: 0 (Failed: 0, Cancelled 0), Incomplete: 1, Skipped: 0}","commit":"584b527","knative.dev/controller":"github.com.tektoncd.pipeline.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","knative.dev/traceid":"57fe491d-3862-41bf-9b53-116cff25ab80","knative.dev/key":"tyw-xxxx-api/1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41"} {"level":"info","ts":"2021-10-18T09:03:51.851Z","logger":"tekton-pipelines-controller.event-broadcaster","caller":"record/event.go:282","msg":"Event(v1.ObjectReference{Kind:"PipelineRun", Namespace:"tyw-xxxx-api", Name:"1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41", UID:"3c30044d-1b1b-41dc-b9dd-632c8bcb5269", APIVersion:"tekton.dev/v1beta1", ResourceVersion:"114893338", FieldPath:""}): type: 'Normal' reason: 'Running' Tasks Completed: 0 (Failed: 0, Cancelled 0), Incomplete: 1, Skipped: 0","commit":"584b527"} {"level":"info","ts":"2021-10-18T09:03:51.921Z","logger":"tekton-pipelines-controller","caller":"controller/controller.go:538","msg":"Reconcile succeeded","commit":"584b527","knative.dev/controller":"github.com.tektoncd.pipeline.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","knative.dev/traceid":"57fe491d-3862-41bf-9b53-116cff25ab80","knative.dev/key":"tyw-xxxx-api/1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41","duration":0.202026368} {"level":"info","ts":"2021-10-18T09:03:51.921Z","logger":"tekton-pipelines-controller.event-broadcaster","caller":"record/event.go:282","msg":"Event(v1.ObjectReference{Kind:"PipelineRun", Namespace:"tyw-xxxx-api", Name:"1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41", UID:"3c30044d-1b1b-41dc-b9dd-632c8bcb5269", APIVersion:"tekton.dev/v1beta1", ResourceVersion:"114893338", FieldPath:""}): type: 'Normal' reason: 'Started' ","commit":"584b527"} second create taskrun{"level":"info","ts":"2021-10-18T09:03:51.926Z","logger":"tekton-pipelines-controller","caller":"pipelinerun/pipelinerun.go:353","msg":"No.1 pipelineMeta is: &ObjectMeta{Name:3b8d0ae68a6fc5c95a91570e9528807c9eb2f164,GenerateName:,Namespace:tyw-xxxx-api,SelfLink:/apis/tekton.dev/v1beta1/namespaces/tyw-xxxx-api/pipelines/3b8d0ae68a6fc5c95a91570e9528807c9eb2f164,UID:29481e0e-19e9-43fb-b036-f2e87a71bc2f,ResourceVersion:114893337,Generation:1,CreationTimestamp:2021-10-18 09:03:51 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[]OwnerReference{},Finalizers:[],ClusterName:,ManagedFields:[]ManagedFieldsEntry{ManagedFieldsEntry{Manager:OpenAPI-Generator,Operation:Update,APIVersion:tekton.dev/v1beta1,Time:2021-10-18 09:03:51 +0000 UTC,FieldsType:FieldsV1,FieldsV1:{"f:spec":{".":{},"f:description":{},"f:tasks":{},"f:workspaces":{}}},},},}, pipelineSpec is: &{Description:测试clone的问题123 Resources:[] Tasks:[{Name:3d8e52746c39bdaf383fc1026d2ebd56816b9ca8 TaskRef:0xc001e2d880 TaskSpec: Conditions:[] WhenExpressions:[] Retries:0 RunAfter:[] Resources: Params:[] Workspaces:[{Name:xxxx Workspace:shared-data SubPath:}] Timeout:nil}] Params:[] Workspaces:[{Name:shared-data Description: Optional:false}] Results:[] Finally:[]}","commit":"584b527","knative.dev/controller":"github.com.tektoncd.pipeline.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","knative.dev/traceid":"57c1f38d-7b73-4987-bd00-448b305f16d2","knative.dev/key":"tyw-xxxx-api/1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41"} {"level":"info","ts":"2021-10-18T09:03:51.934Z","logger":"tekton-pipelines-controller","caller":"pipelinerun/pipelinerun.go:493","msg":"No.2 resolvePipelineState is: &[0xc001a29200]","commit":"584b527","knative.dev/controller":"github.com.tektoncd.pipeline.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","knative.dev/traceid":"57c1f38d-7b73-4987-bd00-448b305f16d2","knative.dev/key":"tyw-xxxx-api/1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41"} {"level":"info","ts":"2021-10-18T09:03:51.944Z","logger":"tekton-pipelines-controller","caller":"pipelinerun/pipelinerun.go:771","msg":"Creating a new TaskRun object 1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41-3d8e52746c-wh66c for pipeline task 3d8e52746c39bdaf383fc1026d2ebd56816b9ca8","commit":"584b527","knative.dev/controller":"github.com.tektoncd.pipeline.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","knative.dev/traceid":"57c1f38d-7b73-4987-bd00-448b305f16d2","knative.dev/key":"tyw-xxxx-api/1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41"} {"level":"info","ts":"2021-10-18T09:03:51.962Z","logger":"tekton-pipelines-controller","caller":"pipelinerun/pipelinerun.go:601","msg":"PipelineRun 1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41 status is being set to &{Succeeded Unknown {2021-10-18 09:03:51.962330786 +0000 UTC m=+440512.388238183} Running Tasks Completed: 0 (Failed: 0, Cancelled 0), Incomplete: 1, Skipped: 0}","commit":"584b527","knative.dev/controller":"github.com.tektoncd.pipeline.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","knative.dev/traceid":"57c1f38d-7b73-4987-bd00-448b305f16d2","knative.dev/key":"tyw-xxxx-api/1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41"} {"level":"info","ts":"2021-10-18T09:03:51.963Z","logger":"tekton-pipelines-controller.event-broadcaster","caller":"record/event.go:282","msg":"Event(v1.ObjectReference{Kind:"PipelineRun", Namespace:"tyw-xxxx-api", Name:"1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41", UID:"3c30044d-1b1b-41dc-b9dd-632c8bcb5269", APIVersion:"tekton.dev/v1beta1", ResourceVersion:"114893338", FieldPath:""}): type: 'Normal' reason: 'Running' Tasks Completed: 0 (Failed: 0, Cancelled 0), Incomplete: 1, Skipped: 0","commit":"584b527"} {"level":"warn","ts":"2021-10-18T09:03:51.984Z","logger":"tekton-pipelines-controller","caller":"pipelinerun/pipelinerun.go:261","msg":"Failed to update PipelineRun labels/annotations{error 26 0 Operation cannot be fulfilled on pipelineruns.tekton.dev "1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41": the object has been modified; please apply your changes to the latest version and try again}","commit":"584b527","knative.dev/controller":"github.com.tektoncd.pipeline.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","knative.dev/traceid":"57c1f38d-7b73-4987-bd00-448b305f16d2","knative.dev/key":"tyw-xxxx-api/1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41"} {"level":"info","ts":"2021-10-18T09:03:51.985Z","logger":"tekton-pipelines-controller.event-broadcaster","caller":"record/event.go:282","msg":"Event(v1.ObjectReference{Kind:"PipelineRun", Namespace:"tyw-xxxx-api", Name:"1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41", UID:"3c30044d-1b1b-41dc-b9dd-632c8bcb5269", APIVersion:"tekton.dev/v1beta1", ResourceVersion:"114893338", FieldPath:""}): type: 'Warning' reason: 'Error' Operation cannot be fulfilled on pipelineruns.tekton.dev "1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41": the object has been modified; please apply your changes to the latest version and try again","commit":"584b527"} {"level":"error","ts":"2021-10-18T09:03:52.047Z","logger":"tekton-pipelines-controller","caller":"pipelinerun/reconciler.go:294","msg":"Returned an error","commit":"584b527","knative.dev/controller":"github.com.tektoncd.pipeline.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","knative.dev/traceid":"57c1f38d-7b73-4987-bd00-448b305f16d2","knative.dev/key":"tyw-xxxx-api/1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41","targetMethod":"ReconcileKind","error":"1 error occurred:\n\t* Operation cannot be fulfilled on pipelineruns.tekton.dev "1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41": the object has been modified; please apply your changes to the latest version and try again\n\n","stacktrace":"github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1beta1/pipelinerun.(*reconcilerImpl).Reconcile\n\tgithub.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1beta1/pipelinerun/reconciler.go:294\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/[email protected]/controller/controller.go:530\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/[email protected]/controller/controller.go:468"} {"level":"error","ts":"2021-10-18T09:03:52.047Z","logger":"tekton-pipelines-controller","caller":"controller/controller.go:548","msg":"Reconcile error","commit":"584b527","knative.dev/controller":"github.com.tektoncd.pipeline.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","duration":0.126541491,"error":"1 error occurred:\n\t* Operation cannot be fulfilled on pipelineruns.tekton.dev "1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41": the object has been modified; please apply your changes to the latest version and try again\n\n","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/[email protected]/controller/controller.go:548\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/[email protected]/controller/controller.go:531\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/[email protected]/controller/controller.go:468"} {"level":"info","ts":"2021-10-18T09:03:52.048Z","logger":"tekton-pipelines-controller.event-broadcaster","caller":"record/event.go:282","msg":"Event(v1.ObjectReference{Kind:"PipelineRun", Namespace:"tyw-xxxx-api", Name:"1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41", UID:"3c30044d-1b1b-41dc-b9dd-632c8bcb5269", APIVersion:"tekton.dev/v1beta1", ResourceVersion:"114893338", FieldPath:""}): type: 'Warning' reason: 'InternalError' 1 error occurred:\n\t* Operation cannot be fulfilled on pipelineruns.tekton.dev "1-3-57-b4fa7e32-70fd-475f-b7a4-00c45914ad80-41": the object has been modified; please apply your changes to the latest version and try again\n\n","commit":"584b527"} |
|
Reopening as this is seen in recent releases 😞 PR #4361 must fix this issue 🤞 |
|
This should be fixed by #4361 which will make any attempt to create a new taskrun fail. |
Hi,
I have a simple pipeline that consists only of a single task. When creating a PipelineRun for it, I expect that one TaskRun is created. Sometimes however, I observe two TaskRuns being created for the same PipelineRun. Here is an example: PipelineRun
uiveri5-lunchmenu-1598256551145resulted in two TaskRunsuiveri5-lunchmenu-1598256551145-uiveri5-d7fkhanduiveri5-lunchmenu-1598256551145-uiveri5-vllbc.Is my expectation right that only one TaskRun should be created? Any ideas what could lead to this situation?
Thank you!
tekton-pipelines-controller logs
pipeline
Additional Info
The text was updated successfully, but these errors were encountered: