-
Notifications
You must be signed in to change notification settings - Fork 9.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
A way to hide certain expected changes from the "refresh" report ("Objects have changed outside of Terraform") #28803
Comments
ignore_changes
does not suppress drift information
Hi @petkaantonov! Thanks for opening this feature request. The current behavior is reflecting the long-standing behavior that Terraform does still detect and incorporate remote objects into the state, but then while producing a plan it ignores differences between the configuration and the state. The difference reported in your example is a difference between the prior state and the current remote object, and While it might seem immaterial whether Terraform updates the state or not here, it can result in a change in behavior of your configuration if any other expressions in the module refer to that value. To be specific, if you had any reference to I assume that in your case this doesn't really matter much, because you don't have any references to However, such a rule is easier to say than to implement because what we've done here is just expose in the UI some long-standing Terraform behavior that was previously invisible, and so changing that behavior at this late state will likely require a lot of research to make sure that the changes don't break use-cases we're not currently aware of. The current output is an honest and correct account of Terraform's behavior, and so I think we need to consider here whether the right thing to do is change Terraform's behavior (which, for something this fundamental, would be challenging to do at this late stage in Terraform's life) or to change the UI to fudge the details a little so that it leaves hidden some details that surface inconvenient truths that don't actually affect configuration behavior. We won't be able to make any significant changes in this regard in the near future, because the scope of this project was just to be more explicit about what Terraform was already doing rather than to change how Terraform behaves, but we'll use this issue to represent the need and consider what we might change in future releases. Thanks again! |
I see. In that case, would it be possible to add a separate way to ignore these? e.g. |
The refresh operation doesn't currently really consider the resource configuration at all -- it's job is to synchronize the existing state with remote objects -- so making it react to anything new in the resource configuration is not a trivial design decision. We also know from our experiences with |
Could the refresh operation be made configurable in some other way? like an .ignorefile There is probably going to be 3rd party solution if it's not added anyway. Just some way to ignore these is important, otherwise the output is not as useful because it becomes just manually ignored noise and people become conditioned to ignore it regardless of what it says. |
I don't currently have any ready-to-implement designs to address this feedback. We'll need to do some more research and design work in order to address it. Until then, indeed if your system routinely makes changes to objects outside of Terraform as part of its regular function then the feature in its current state will not be so useful for you, Hopefully it can still answer the sorts of questions it is aiming to solve, though: if Terraform proposes making a change that doesn't seem justified by a corresponding change in configuration, you can refer to the note about detected changes to see if any of the changes it detected are the explanation. If the actions Terraform is proposing match what you were expecting anyway, then the changes detected are just some extra information that you can safely ignore. |
In situations where the user is forced to add an ignore - to stop Terraform from chasing it's own tail - ie with the use of an aws_autoscaling_attachment (where it is documented that you must use an ignore in corresponding autoscaling groups https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/autoscaling_attachment), I think this behaviour constitutes a bug. The drift output states that "Objects have changed outside of Terraform" when in reality the changes are made within terraform and have had to be ignored for reasons that might be justified as a bug in their own right. Choosing to output drift before taking into account ignores thus creates potentially significant noise when using Terrafom within it's documented bounds. |
I can relate to this issue a lot, our daily work now has a new pain: filtering out the noise of ignored changes. anyway, thank you guys for the good work, keep up with it! 👍 |
ignore_changes
does not suppress drift information
This feels like a pretty invasive change. I second what @koalalorenzo said. The main problem with this change is that we lean on our large team of engineers to inspect their own changes carefully on a per-team basis and apply those changes without constant infra approval. Even when diffs are small, though, we can sometimes misunderstand the plan which can lead to some unfortunate consequences. Not everyone understands the inner workings of terraform, or even the best practices, so there is always going to be mistakes. This isn't terraform's fault, we all need to learn the tool a little bit better. Now, though, we have to sift through an entire block of output and tell our team to just ignore it up to a point. Its just going to lead to less clear plan output that people put less effort into reviewing. I think its going to erode trust, too, even though I suspect the opposite is the goal. For instance, take a TBH, I have ignored many attributes in terraform because we may change them outside of terraform and have come to rely on the fact that it really doesn't and shouldn't care about the changes to attributes you don't specify. IDK. Ranting a bit. I have to be totally honest I see no point in opting everyone into this for every resource automatically. I don't understand the problem that its solving and unless there is a way for it to be completely filtered out I would love to see this change reverted in the next release. (I also don't think the solution is |
I am very surprised about this change and it adds a ton of noise to the already not exactly clear and concise plans. The extra verbosity adds a lot of cognitive load which risks at best to make people glance over the plans and at worst make them ignore them completely. I really, really don't understand the reasoning here. Again, other than that, thank you for a great product over the years! |
I can get a little worked up and forget that a lot of people put in a lot of work to get terraform where it is today. I don't want to belittle that. People like @apparentlymart (who helped us in the forums when we were starting to write our own internal terraform provider plugins) are awesome. Terraform is awesome. Obviously my job is infinitely easier because of it. ❤️ As for this feature, it was in the CHANGELOG and I did glance over it, so its on me that I upgraded to |
Just adding another resource type that now always appears in my terraform output - aws_efs_file_system. (from #28845 which I will mark as a duplicate). EFS file systems are supposed to grow and change, as many other types of resource they are explicitly designed that way. With no way to ignore, the plan/apply output is now so noisy I've gone back to terraform 0.15.3 as wading through tens of modified resources on every single plan/apply just to catch the one that might be important is not a workable pattern for me. |
Since terraform 0.15.4, I get confused messages from my team every single day. I can't imagine the impact this change had globally. Please add an option to disable displaying those by default and go back to the previous behavior, with maybe an explicit way to get more details about the changes only when needed (e.g a debug mode). This is really confusing right now, and someone needs to understand the internals of terraform to appreciate the difference between a drift and remote changes. |
One thing that surprised me a lot about this change is that the As near as I can tell, the only way to currently silence the warning is to run |
Agreed, this bit us earlier. Another example is that AWS autoscaling groups will always have "external" changes to attributes such as As the below is an ignored attribute, the resource is not affected by Terraform and the message amounts to noise: # aws_autoscaling_group.example-scaling-group has been changed
~ resource "aws_autoscaling_group" "example-scaling-group" {
~ desired_capacity = 50 -> 36
id = "example-scaling-group"
name = "example-scaling-group"
# (20 unchanged attributes hidden) A person reviewing this plan has to already know the attribute is ignored on that resource or read "below the line" in order to confirm the resource is not affected. Another way of looking at this is as a formatting issue, since the change notification looks very similar to the actual plan, and it took me and the rest of my team a while to realise it's a separate section. |
Another example - |
the size attribute of azuredevops_git_repository is another of these dynamic fields that i really don't need to see every time i plan/apply. was disappointed ignore didn't just silent it, makes sense cause someone somewhere has some TF that does something based on size changing i'm sure...thinking something like a new lifecycle prop like
that tells TF to always update state from current value without reporting it as a difference. |
same issue here:
|
It is such a big frustration that the other day I rejected a valid PR with atlantis badly because I got confused what were the changes. Please make this output OPTIONAL! Who do we contact to make this disappear in our logs? Is this ever going to be fixed? What can we do to make this a priority? Is there any contributor or Hashicorp employee taking care of UX changes in Terraform that we can ask to check this issue? Here is a cute doggo asking for help: [kmoe: edited to remove large image] |
same here! It is taking a lot of time to explain to the engineers that this output does not reflect any changes in our infrastructure. And I'm afraid if the people start to ignore the plan because they are thinking that all changes are only in the state file |
Yet another example:
The following plan will 100% not ever include actions to undo or respond to those changes, because we've explicitly and deliberately in our configuration told Terraform not to manage the instances in the ELB. (Sometimes other things need to take instances in and out, e.g. Ansible while deploying things.) It says right there in our configs that Terraform should 100% not ever care what instances are in the ELB. Our configuration is being ignored, and Terraform is now caring (albeit only at the level of a warning) about things that we explicitly and deliberately told it not to care about. This is not detecting drift. This is a bug. |
I have a suggestion. How about Terraform introduces a new sub-command: terraform state diff ...and the output of that command will be the same as we are seeing in "Terraform detected the following changes made outside of Terraform" section since Terraform 1.0. Then we can completely remove that state diff from |
That's not a great solution, as it only solves few issues. |
Terraform will still check for drift between the state and the reality. It has always been doing a refresh before plan. And I believe it's always been the case that any changes outside of Terraform could trigger plan changes (i.e. resource modification). BUT when I am looking at the plan, I am mostly interested in what changes Terraform is going to make. It may choose to undo some changes made externally, or pick up some external changes (via computed resource attributes or data sources). What I am not really interested in is the state drift that might cause plan changes, but in reality it does not. The state diff output as part of the plan would be more useful/helpful if it only included the differences that resulted in plan changes and ignored all the others. If someone wanted to check the difference between the last applied state and the reality (i.e. any external changes since last |
At this point my jobs are failing and I can't even see why, because across tens of templated JSON files totaling a few hundred thousand LOC, the (ridiculously generous) maximum log file size in GitLab of 4MB is being exceeded with (and I will be flippant here) COMPLETELY worthless Terraform information. It's been all year and there is still not even a single flag to suppress expected changes? I know HashiCorp went through an employee-quitting phase but this is like an intern task and is directly impacting production. Should've stayed on 0.15 but there is no going back at this point. |
I think the current behavior is actually a bit dangerous, because when faced with walls of drift, which is now the new normal for any somewhat large install, for each proposed change, the brain trains to ignore this noise and hit yes. It will become ever easier to miss unintended changes as you use this more. |
@kvz makes a real point. Never-mind try to explain to developers who interact with terraform infrequently "ohh ignore this bit but don't ignore this bit" it's asking for accidents to be happen. |
I feel very much the same way I have to scroll through hundreds of useless |
Same problem, updating from 0.13.5 to 1.0.9.
|
@apparentlymart @jbardin Is there an update on this issue, please?! |
I just spent a good hour debugging why my diff was so strange. Turns out it wasn't, it was just Terraform playing mind games with me. Not funny. |
Worst part...it's not even consistent..yesterday I had completely different "change detections" everytime i ran terraform plan...it was as if, the terraform missed results fom the API, and didn't report it..then the next time it detected it..and the next time it didn't..and so on...this was on v0.15.5..i rolled back to v0.15.3..terrible implementation I must say...it's the classic feature vs bug...as rightly pointed out by the others..this is something one would want to see in DEBUG logging, not in default output...Terraform team has opted a completely wrong track, in my opinion...I mean as developer, what do we care about? Just our config and the infra/reality...1:1 mapping there...who cares what Terraform wants to store in its state file...if it sees a difference between what it stores and what sees in reality, it's Terraform's problem, not mine...I just need my code to be in 1:1 sync with reality...the intermediate statefile is Terraform's headache... |
FWIW, I think functionality is handy, but as others I'd rather have it as a separate command rather than standard plan part. |
Or under a prompt:
with a flag to always pass yes or no via |
This issue got so many comments since May as other open Terraform issues since 2015. Based on this it must be obvious how pressing it is and how many people are experiencing problems in their day-to-day work due to this. There are tons of great suggestions and proposals in this thread, could someone please take a look at it already? 🙂 |
90% of comments are like "yeah me too" or "that sucks". Where the PRs guys ? Your company depends on Terraform, your a professional ? Send some PRs then. It's open source, not everybody can leech. |
|
@JordanP I think we are really here to convince the Terraform Core team that the community needs a better solution. There is not much point if we submit a PR if the team is not keen to merge it.
|
I think hashicorp indicated they don't have the bandwidth to review community PRs right now. And if they had, we should ask how they want to address this, if at all. The code itself to allow to skip showing this, will be a oneliner so that is not what is blocking this |
Hi all! Thanks for all the feedback here. It's clear that there are lots of opportunities to improve the signal to noise ratio of the current change detection. As I mentioned right back at the top of this discussion, this is Terraform making explicit some behaviors that were previously implicit and thus a common cause for confusion. However, it's also clear that various particular provider features, along with the historical confusing design of Two specific things that the Terraform team is researching to improve this feature are:
Both of these features are cross-cutting and therefore not something we can just casually implement without careful design first. However, we are indeed actively investigating both and hope to have improvements to share in a forthcoming release. Since this discussion has got quite heated and it seems like we've already gathered sufficient feedback, I'm going to lock this discussion for now in order to reduce the noise for the many folks who are following this issue. We'll unlock it again when either we have more news to share or if we need some specific feedback in order to shape solutions like what I described above. |
Hello All! Here to deliver the latest update! With the addition of #30486, we hope to address the concerns raised here, while also taking into account the users who do desire the refresh report in order to help understand the changes within a plan and detect unexpected changes. Starting with v1.2, the goal for the refresh report is that only external changes which may have contributed to changes in the plan will be shown. This means in most cases, unused attributes changing outside of terraform will not show up in the normal plan output. If there are no changes in the plan, no external changes will be shown in the CLI at all. All refresh information is still stored within the plan, and if a user wants to see all external changes of resources in the CLI, a refresh-only plan can be used. For more details on the change, see #30486. For any questions or discussion, feel free to use the community forum. Thanks! |
After upgrading to 0.15.4 terraform reports changes that are ignored. It is exactly like commented here: #28776 (comment)
Terraform Version
Terraform Configuration Files
Output
Expected Behavior
No changes should be reported because they are listed in ignored changes.
Actual Behavior
Changes are reported.
Steps to Reproduce
Change any resource outside of terraform and see that
terraform apply
reports changed even when they should be ignored.Additional Context
References
The text was updated successfully, but these errors were encountered: