Allow include/exclude for file provisioner #9671

steviecoaster · 2020-07-29T22:11:59Z

It seems #1811 was closed as won't fix for....no other reason than it would be confusing.

This is a misstep as having that functionality would severely improve the usability of the file provisioner, without having to have multiple steps or convoluted shell/powershell code to clean up after it.

Please consider adding this feature.

vexx32 · 2020-07-29T22:15:35Z

Aside-meta-note: closing issues as "wontfix" with a version target is fundamentally confusing and sends mixed messages. By all means, target for whatever future version, but closing it is a pretty clear "not gonna happen".

But yeah, please provide for this. I'd like to be able to upload the contents of a git repo without having to wait half an hour for it to slowly upload the .git hidden folder and then have to delete it after the fact. This is a basic useability feature, I don't know how y'all failed to include it in the initial implementation.

azr · 2020-07-30T09:03:36Z

Hello there, thanks for opening; I think this should be solved with the hcl2 fileset function !

SwampDragons · 2020-07-30T22:21:25Z

I think you're right that the rationale for having closed the original issue was weak. This would be a nice UI improvement.

Probably the right way to address this in the provisioner itself rather than the hcl2 fileset function would be to change the UploadDir call here https://github.com/hashicorp/packer/blob/master/provisioner/file/provisioner.go#L180 to a directory-walking function that checks against an "exlcude" list that the user can pass in as an option to the provisioner.

Since it does look like HCL provides a built-in solution for this, I probably won't direct the Packer team to work on it, but I'd be happy to review a community PR that does this.

NoahAndrews · 2020-10-05T20:31:16Z

Maybe I'm missing something, but I don't see any way to exclude specific files or directories using HCL2's fileset. I'd be happy to switch to HCL2 if it can provide an actual solution.

azr · 2020-10-06T08:23:36Z

Hello there, yes sorry, my previous message was rather short.

The HCL fileset function allows to specifically and recursively select files with a glob pattern. So it's only 'include' but it can be exclusive depending on how you call it, so to take the docs page example, if you have:

> tree pkr-consul
pkr-consul
├── build-linux.pkr.hcl
└── linux
    ├── files
    │   ├── hello.txt
    │   └── world.txt
    └── scripts
        ├── README.txt
        ├── script.bash
        ├── script-1-install.sh
        └── script-2-setup.sh

And you only want the .txt suffixed files, you can do:

$ echo 'fileset("**", "*.txt")' | packer console pkr-consul
[
  "../files/hello.txt",
  "../files/world.txt",
  "../linux/scripts/README.txt",
]

or only .sh suffixed files in linux/scripts:

$ echo 'fileset("linux/scripts", "*.sh")' | packer console pkr-consul
[
  "script-1-install.sh",
  "script-2-setup.sh",
]

I'd advise on sorting the fileset after this because fileset will not sort anything ( so depending on how your filesystem works this might differ).

packer console will help you find the best solution.

If that does not help you to solve your problem; please describe your issue a bit more so that we can help you better 🙂

micchickenburger · 2021-02-23T00:35:48Z

I'm wondering if anyone has tried to tackle this yet. I'm administering a Rails app that has a bunch of stuff I need to exclude and in my case I can't delete the excess after the fact. I'm not sure if I will have time to tackle this issue anytime soon, but I am motivated to see it solved.

SwampDragons · 2021-02-23T23:46:03Z

I don't think anyone has, but I'd be interested to know if the fileset function solves this for you.

NoahAndrews · 2021-02-24T01:42:11Z

Just for the record, I didn't respond because I ended up not using packer at all.

NoahAndrews · 2021-02-24T01:49:35Z

A docker-based solution

enykeev · 2021-08-26T10:28:52Z

I don't think anyone has, but I'd be interested to know if the fileset function solves this for you.

It sadly does not. I've removed my previous message where I thought I solved the problem with this snippet

  provisioner "file" {
    sources = [for s in fileset("source", "**") : s if !can(regex("/node_modules/|/dist/|/.git/|/data/", "./${s}"))]
    destination = "/home/ec2-user/target/"
  }

But I quickly discovered that doing it this way results in just a pile of files in target directory with no directory structure being kept. (I guess) You can potentially create a number of provisioners for every subdirectory, but even if it works I would hardly call it a solution.

msilveirabr · 2022-08-03T01:55:55Z

From the looks of my tree below, you may imagine what I've done:
dynamic source block with dynamic file and powershell provisioners blocks....

Some way to control how to ignore files would be great.
I'll have to create a powershell to recurse into the target builds to remove files with one-line comment...

windows-server-2019/
├── [ 1.2K]  buildall.sh
├── [ 1.2K]  build.sh
├── [  166]  config
│   ├── [ 1.0K]  build.pkrvars.hcl
│   ├── [  946]  common.pkrvars.hcl
│   ├── [  663]  esxi.pkrvars.hcl
│   ├── [ 1.1K]  strings_en-US.pkr.hcl
│   ├── [ 1.2K]  strings_pt-BR.pkr.hcl
│   └── [  825]  vcenter.pkrvars.hcl
├── [   59]  data
│   ├── [   25]  certificates
│   │   └── [ 7.3K]  root-ca.cer
│   ├── [   20]  drivers
│   │   └── [   19]  pvscsi
│   │       └── [   80]  amd64
│   │           ├── [  10K]  pvscsi.cat
│   │           ├── [ 4.1K]  pvscsi.inf
│   │           ├── [  63K]  pvscsi.sys
│   │           └── [  595]  txtsetup.oem
│   └── [   33]  unattended
│       ├── [   66]  core
│       │   ├── [  15K]  autounattend.pkrtpl.hcl
│       │   └── [ 9.4K]  unattended.pkrtpl.hcl
│       └── [   66]  desktop
│           ├── [  15K]  autounattend.pkrtpl.hcl
│           └── [ 9.4K]  unattended.pkrtpl.hcl
├── [ 1.3K]  debug_build.sh
├── [ 1.7K]  locals.pkr.hcl
├── [    6]  manifests
├── [  230]  NOTES.txt
├── [   44]  scripts
│   ├── [   38]  all
│   │   ├── [  102]  others
│   │   │   ├── [  759]  Extend-Partition.ps1
│   │   │   ├── [  134]  lang
│   │   │   │   ├── [   95]  AdminUsersCustomConfig.ps1
│   │   │   │   ├── [   93]  AllUsersCustomConfig.ps1
│   │   │   │   ├── [ 1.2K]  LocalUserCustomConfig.ps1
│   │   │   │   └── [   91]  UsersCustomConfig.ps1
│   │   │   ├── [   28]  post-sysprep.ps1
│   │   │   └── [   36]  Remove-Store-Taskbar.ps1
│   │   └── [  179]  unattended
│   │       ├── [   34]  custom-activesetup.ps1
│   │       ├── [   87]  lang
│   │       │   ├── [ 1.1K]  custom-activesetup.ps1
│   │       │   ├── [ 1.5K]  windows-init.ps1
│   │       │   └── [ 1.3K]  windows-prepare.ps1
│   │       ├── [  970]  reboot-bw-updates.ps1
│   │       ├── [ 1.8K]  windows-init.ps1
│   │       ├── [ 8.5K]  windows-prepare.ps1
│   │       ├── [ 3.9K]  windows-vmtools.ps1
│   │       └── [ 1.7K]  wmf51-update.ps1
│   ├── [   38]  core
│   │   ├── [  102]  others
│   │   │   ├── [   33]  Extend-Partition.ps1
│   │   │   ├── [  134]  lang
│   │   │   │   ├── [   84]  AdminUsersCustomConfig.ps1
│   │   │   │   ├── [   82]  AllUsersCustomConfig.ps1
│   │   │   │   ├── [   83]  LocalUserCustomConfig.ps1
│   │   │   │   └── [   80]  UsersCustomConfig.ps1
│   │   │   ├── [   28]  post-sysprep.ps1
│   │   │   └── [   37]  Remove-Store-Taskbar.ps1
│   │   └── [   75]  unattended
│   │       ├── [   35]  custom-activesetup.ps1
│   │       ├── [   33]  lang
│   │       │   └── [   33]  windows-prepare.ps1
│   │       └── [   32]  windows-prepare.ps1
│   └── [   38]  desktop
│       ├── [  102]  others
│       │   ├── [   33]  Extend-Partition.ps1
│       │   ├── [  171]  lang
│       │   │   ├── [  679]  AdminUsersCustomConfig.ps1
│       │   │   ├── [   85]  AllUsersCustomConfig copy.ps1
│       │   │   ├── [  980]  AllUsersCustomConfig.ps1
│       │   │   ├── [   86]  LocalUserCustomConfig.ps1
│       │   │   └── [   83]  UsersCustomConfig.ps1
│       │   ├── [   28]  post-sysprep.ps1
│       │   └── [ 2.9K]  Remove-Store-Taskbar.ps1
│       └── [   75]  unattended
│           ├── [   38]  custom-activesetup.ps1
│           ├── [   33]  lang
│           │   └── [   36]  windows-prepare.ps1
│           └── [   35]  windows-prepare.ps1
├── [   28]  strings.pkr.hcl -> config/strings_pt-BR.pkr.hcl
├── [ 1.2K]  validate.sh
├── [  11K]  variables.pkr.hcl
├── [ 4.0K]  windows-server.auto.pkrvars.hcl
└── [  16K]  windows-server.pkr.hcl

Resolves: hashicorp#9671

steviecoaster changed the title ~~Allow include/exclude for file provisioned~~ Allow include/exclude for file provisioner Jul 29, 2020

azr added the enhancement label Jul 30, 2020

SwampDragons added help wanted provisioner/file labels Jul 30, 2020

Glyphack added a commit to Glyphack/packer that referenced this issue Feb 18, 2023

file: use excludes list to excl files when uploading a directory

961928b

Resolves: hashicorp#9671

Glyphack linked a pull request Feb 18, 2023 that will close this issue

[WIP] file: use excludes list to excl files when uploading a directory #12280

Open

Glyphack added a commit to Glyphack/packer that referenced this issue Mar 21, 2023

file: use excludes list to excl files when uploading a directory

33fc558

Resolves: hashicorp#9671

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow include/exclude for file provisioner #9671

Allow include/exclude for file provisioner #9671

steviecoaster commented Jul 29, 2020 •

edited

Loading

vexx32 commented Jul 29, 2020

azr commented Jul 30, 2020

SwampDragons commented Jul 30, 2020

NoahAndrews commented Oct 5, 2020 •

edited

Loading

azr commented Oct 6, 2020 •

edited

Loading

micchickenburger commented Feb 23, 2021

SwampDragons commented Feb 23, 2021

NoahAndrews commented Feb 24, 2021

NoahAndrews commented Feb 24, 2021

enykeev commented Aug 26, 2021

msilveirabr commented Aug 3, 2022

Allow include/exclude for file provisioner #9671

Allow include/exclude for file provisioner #9671

Comments

steviecoaster commented Jul 29, 2020 • edited Loading

vexx32 commented Jul 29, 2020

azr commented Jul 30, 2020

SwampDragons commented Jul 30, 2020

NoahAndrews commented Oct 5, 2020 • edited Loading

azr commented Oct 6, 2020 • edited Loading

micchickenburger commented Feb 23, 2021

SwampDragons commented Feb 23, 2021

NoahAndrews commented Feb 24, 2021

NoahAndrews commented Feb 24, 2021

enykeev commented Aug 26, 2021

msilveirabr commented Aug 3, 2022

steviecoaster commented Jul 29, 2020 •

edited

Loading

NoahAndrews commented Oct 5, 2020 •

edited

Loading

azr commented Oct 6, 2020 •

edited

Loading