fix: Check all directories with changes and pass all args in terrascan hook

[carlosbustillordguez]

Put an x into the box if that apply:

• This PR introduces breaking change.
• This PR fixes a bug.
• This PR adds new functionality.
• This PR enhances existing functionality.

Description of your changes

This PR introduces the following changes:

• Configure terrascan hook to run only with Terraform config files.
• Pass to the terrascan command all supplied arguments.
• Allow inspecting each directory with TF config files for the current commit and fetch the final terrascan exit code.
• Add hook usage notes and an example for terrascan.

How has this code been tested

The current implementation of terrascan.sh only adds to the terrascan command the first argument of the ARGS array. Try to use the following configuration to reproduce the problem (the --policy-type azure is never passed as an argument to terrascan):

repos:
- repo: https://github.com/antonbabenko/pre-commit-terraform
  rev: v1.62.1 # Get the latest from: https://github.com/antonbabenko/pre-commit-terraform/releases
  hooks:
    - id: terraform_validate
    - id: terraform_fmt
      args:
        - --args=-no-color
        - --args=-diff
        - --args=-write=false
    # Detect compliance and security violations across IaC to mitigate risk before provisioning cloud native infrastructure.
    - id: terrascan
      args:
        - --args=--non-recursive # avoids scan errors on subdirectories without Terraform config files
        - --args=--policy-type azure
    # Security scanner for your Terraform code.
    - id: terraform_tfsec
    # A Pluggable Terraform Linter
    - id: terraform_tflint
      args:
        - --args=--config=__GIT_WORKING_DIR__/.tflint.hcl
        - --args=--module

terrascan is executed over all committed files, which produce errors on subdirectories without Terraform config files:

$ git commit -m "test1"
Terraform validate.......................................................Passed
Terraform fmt............................................................Passed
terrascan................................................................Failed
- hook id: terrascan
- exit code: 1

2021-12-19T23:09:08.531+0100    error   cli/run.go:132  scan run failed{error 26 0  1 error occurred:
        * directory '/home/carlos/devel/azure/terraform/azure-kubernetes-services-iac/modules/cert-manager/templates' has no terraform config files

}

Terraform validate with tfsec............................................Passed
Terraform validate with tflint...........................................Passed

Also, only the first directory of the FILES array is only analyzed with terrascan.

[MaxymVlasov]

docs review: Please add a link to terrascan section ([Hook notes](#terrascan)) in Available Hooks table

[yermulnik]

@MaxymVlasov Code looks good to me.

[MaxymVlasov]

Also, I tested performance, and the hook in this branch work 3 times slower than in v1.62.1 and 6 times slower than in v1.62.2

took 20s

repos:
- repo: https://github.com/antonbabenko/pre-commit-terraform
  rev: v1.62.1
  hooks:
    - id: terrascan

tag v1.62.2 (with require_serial: true) took 9s.

took 1m 3s

repos:
- repo: https://github.com/carlosbustillordguez/pre-commit-terraform
  rev: 5f9e0cce9974741a7dc1723c7625082a4c1cc904
  hooks:
    - id: terrascan

Try to add debug outputs to script and find what goes wrong

[MaxymVlasov]

Still facing problems with performance. I will try to investigate why

UPD. It is because v1.62.2 check only 1 directory, when this branch checks all.
Try to decrease cont of run for terrascan

[carlosbustillordguez]

Also, I tested performance, and the hook in this branch work 3 times slower than in v1.62.1 and 6 times slower than in v1.62.2

took 20s

repos:
- repo: https://github.com/antonbabenko/pre-commit-terraform
  rev: v1.62.1
  hooks:
    - id: terrascan

tag v1.62.2 (with require_serial: true) took 9s.

took 1m 3s

repos:
- repo: https://github.com/carlosbustillordguez/pre-commit-terraform
  rev: 5f9e0cce9974741a7dc1723c7625082a4c1cc904
  hooks:
    - id: terrascan

Try to add debug outputs to script and find what goes wrong

Also, I tested performance, and the hook in this branch work 3 times slower than in v1.62.1 and 6 times slower than in v1.62.2

took 20s

repos:
- repo: https://github.com/antonbabenko/pre-commit-terraform
  rev: v1.62.1
  hooks:
    - id: terrascan

tag v1.62.2 (with require_serial: true) took 9s.

took 1m 3s

repos:
- repo: https://github.com/carlosbustillordguez/pre-commit-terraform
  rev: 5f9e0cce9974741a7dc1723c7625082a4c1cc904
  hooks:
    - id: terrascan

Try to add debug outputs to script and find what goes wrong

I tested the hooks following the CONTRIBUTING.md guide, against the following repos:

repos:
-   repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.62.2
    hooks:
    -   id: terrascan

repos:
-   repo: https://github.com/carlosbustillordguez/pre-commit-terraform
    rev: 0d1c7164a4cde2d8c8e3d71846a2a6a2bb4472dd
    hooks:
    -   id: terrascan

The tests result are:

20 runs 'terrascan v1.62.2:'

time command	max	min	mean	median
users seconds	2.49	2.02	2.155	2.155
system seconds	0.23	0.11	0.17	0.165
CPU %	116	96	103.5	104
Total time	2.47	2.02	2.243	2.235

20 runs 'terrascan PR #305:'

time command	max	min	mean	median
users seconds	2.3	1.99	2.1825	2.175
system seconds	0.25	0.11	0.1755	0.175
CPU %	113	97	105.3	105.5
Total time	2.46	2.1	2.2355	2.22

The total time is practically the same for both repositories. Please, note that the current implementation for terrascan.sh only analyze the directory of the first file passed to the terrascan_ function:

pre-commit-terraform/terrascan.sh

Line 9 in 04ecd10

terrascan_ "$ARGS" "$FILES"

Because the argument is not passed as an array, in my PR I pass all arguments and files to the terrascan_ function:

pre-commit-terraform/terrascan.sh

Line 7 in 0d1c716

terrascan_ "${ARGS[*]}" "${FILES[@]}"

This can make the difference in bigger Terraform repositories because the root module and internal modules will be inspected.

Can you share which tests are you using?

[carlosbustillordguez]

Still facing problems with performance. I will try to investigate why

UPD. It is because v1.62.2 check only 1 directory, when this branch checks all. Try to decrease cont of run for terrascan

I noted that if we inspect only the first directory, works fine with a module without internal modules. For a module with internal modules, if the changed files are in different internal modules only the first one in the FILES array will be inspected. With the approach of my branch that is covered but I am aware can introduce some delay when all files are scanned.

For my tests I am using a Terraform module with the following structure:

├── LICENSE
├── README.md
├── aks-cluster.tf
├── main.tf
├── modules
│   ├── agic-internal
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   ├── cert-manager
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   ├── templates
│   │   │   ├── clusterissuer-le-production.yaml.tpl
│   │   │   ├── clusterissuer-le-staging.yaml.tpl
│   │   └── variables.tf
│   ├── kv-certificates
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   └── managed-identities-apps
│       ├── main.tf
│       ├── outputs.tf
│       ├── templates
│       │   ├── azure-identity-binding.yaml.tpl
│       │   └── azure-identity.yaml.tpl
│       └── variables.tf
├── outputs.tf
├── scripts
│   ├── README.md
│   ├── add-ssl-cert-appgw.sh
│   └── custom-role.md
├── terraform.tfvars
└── variables.tf

[MaxymVlasov]

pre-commit automatically parallel checks to exiting cores, and I suppose you have >=4 CPU cores, when terrascan checks only 4 dirs in your repo.

Try to run on bigger repo, like this one.

.
├── README.md
├── aws-nuke.yaml
├── environment
│   ├── prd
│   │   ├── backends.tf
│   │   ├── data.tf
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   ├── providers.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   └── qa
│       ├── backends.tf
│       ├── data.tf
│       ├── main.tf
│       ├── outputs.tf
│       ├── providers.tf
│       ├── variables.tf
│       └── versions.tf
├── modules
│   ├── aws-acm-certificate
│   │   ├── acm.tf
│   │   ├── output.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── aws-api-gateway
│   │   ├── output.tf
│   │   ├── rest.tf
│   │   ├── route53.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── aws-chatbot
│   │   ├── README.md
│   │   ├── data.tf
│   │   ├── helm.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── aws-dynamodb-table
│   │   ├── dynamodb.tf
│   │   ├── output.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── aws-ecr-repo
│   │   ├── outputs.tf
│   │   ├── repo.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── aws-eks-cluster
│   │   ├── cluster.tf
│   │   ├── data.tf
│   │   ├── iam.tf
│   │   ├── outputs.tf
│   │   ├── permissions.tf
│   │   ├── policies.tf
│   │   ├── providers.tf
│   │   ├── security.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── aws-eks-node-group
│   │   ├── data.tf
│   │   ├── iam.tf
│   │   ├── instances.tf
│   │   ├── output.tf
│   │   ├── policies.tf
│   │   ├── security.tf
│   │   ├── user_data
│   │   │   └── userdata.sh.tpl
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── aws-elasticache-cluster
│   │   ├── alarm.tf
│   │   ├── clusters.tf
│   │   ├── outputs.tf
│   │   ├── security.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── aws-environment
│   │   ├── data.tf
│   │   ├── endpoints.tf
│   │   ├── flow_log.tf
│   │   ├── iam.tf
│   │   ├── instances.tf
│   │   ├── lambdas.tf
│   │   ├── network.tf
│   │   ├── outputs.tf
│   │   ├── policies.tf
│   │   ├── provisioner.tf
│   │   ├── route53.tf
│   │   ├── security.tf
│   │   ├── sns.tf
│   │   ├── templates
│   │   │   └── github-users-sync.tpl
│   │   ├── user_data
│   │   │   └── bastion.sh
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── aws-gitlab-ec2-manager
│   │   ├── README.md
│   │   ├── ec2.tf
│   │   ├── iam.tf
│   │   ├── outputs.tf
│   │   ├── provisioner.tf
│   │   ├── s3.tf
│   │   ├── sg.tf
│   │   ├── templates
│   │   │   ├── config.sh
│   │   │   └── init.sh
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── aws-grafana
│   │   ├── alarm.tf
│   │   ├── data.tf
│   │   ├── db.tf
│   │   ├── helm.tf
│   │   ├── iam.tf
│   │   ├── providers.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── aws-iam-group
│   │   ├── iam.tf
│   │   ├── outputs.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── aws-iam-role
│   │   ├── iam.tf
│   │   ├── outputs.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── aws-iam-user
│   │   ├── iam.tf
│   │   ├── outputs.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── aws-lambda-function
│   │   ├── cloudwatch.tf
│   │   ├── iam.tf
│   │   ├── lambda.tf
│   │   ├── output.tf
│   │   ├── permissions.tf
│   │   ├── policy.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── aws-msk-cluster
│   │   ├── cloudwatch.tf
│   │   ├── kms.tf
│   │   ├── msk.tf
│   │   ├── outputs.tf
│   │   ├── security.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── aws-rds-cluster
│   │   ├── alarm.tf
│   │   ├── iam.tf
│   │   ├── instances.tf
│   │   ├── outputs.tf
│   │   ├── policies.tf
│   │   ├── security.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── aws-route53-zone
│   │   ├── outputs.tf
│   │   ├── variables.tf
│   │   ├── versions.tf
│   │   └── zone.tf
│   ├── aws-s3-bucket
│   │   ├── iam.tf
│   │   ├── outputs.tf
│   │   ├── policy.tf
│   │   ├── s3.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── aws-sns-topic
│   │   ├── output.tf
│   │   ├── sns.tf
│   │   ├── variable.tf
│   │   └── versions.tf
│   ├── helm-release
│   │   ├── helm.tf
│   │   ├── namespace.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── kube-manifest
│   │   ├── manifest.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── postgresql-env
│   │   ├── db.tf
│   │   ├── outputs.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   └── rapp-services
│       ├── aws-core-api
│       │   ├── README.md
│       │   ├── data.tf
│       │   ├── db.tf
│       │   ├── helm.tf
│       │   ├── iam.tf
│       │   ├── vars.tf
│       │   └── versions.tf
│       ├── aws-iot-segway-max-plus
│       │   ├── README.md
│       │   ├── data.tf
│       │   ├── helm.tf
│       │   ├── vars.tf
│       │   └── versions.tf
│       ├── aws-iot-segway-mock
│       │   ├── README.md
│       │   ├── data.tf
│       │   ├── db.tf
│       │   ├── helm.tf
│       │   ├── variables.tf
│       │   └── versions.tf
│       └── aws-ops-web
│           ├── README.md
│           ├── alarm.tf
│           ├── helm.tf
│           ├── providers.tf
│           ├── variables.tf
│           └── versions.tf
├── path
├── results.json
├── run-305.sh
├── run1.62.2.sh
├── run305-dir.sh
├── services
│   ├── alerting
│   │   └── us-east-1
│   │       ├── init.tf
│   │       ├── main.tf
│   │       └── vars.tf
│   ├── chatbot
│   │   └── global
│   │       ├── 0-init.tf
│   │       ├── 1-ssm-params.tf
│   │       ├── 2-eks.tf
│   │       ├── 3-api-gw.tf
│   │       ├── 3-iam.tf
│   │       ├── 3-lambda.tf
│   │       └── vars.tf
│   ├── dns
│   │   └── global
│   │       ├── backends.tf
│   │       ├── main.tf
│   │       ├── outputs.tf
│   │       ├── providers.tf
│   │       ├── variables.tf
│   │       └── versions.tf
│   ├── docker-registry
│   │   └── global
│   │       ├── backends.tf
│   │       ├── main.tf
│   │       ├── outputs.tf
│   │       ├── providers.tf
│   │       ├── variables.tf
│   │       └── versions.tf
│   ├── grafana
│   │   └── qa
│   │       ├── backends.tf
│   │       ├── data.tf
│   │       ├── main.tf
│   │       ├── providers.tf
│   │       ├── variables.tf
│   │       └── versions.tf
│   ├── iam
│   │   └── global
│   │       ├── backends.tf
│   │       ├── data.tf
│   │       ├── main.tf
│   │       ├── outputs.tf
│   │       ├── providers.tf
│   │       ├── variables.tf
│   │       └── versions.tf
│   ├── kafka
│   │   ├── prd
│   │   │   ├── init.tf
│   │   │   ├── main.tf
│   │   │   ├── outputs.tf
│   │   │   └── vars.tf
│   │   └── qa
│   │       ├── backends.tf
│   │       ├── data.tf
│   │       ├── main.tf
│   │       ├── outputs.tf
│   │       ├── providers.tf
│   │       ├── variables.tf
│   │       └── versions.tf
│   ├── kubernetes
│   │   ├── README.md
│   │   ├── prd
│   │   │   ├── backends.tf
│   │   │   ├── data.tf
│   │   │   ├── kube-prometheus-stack.yaml
│   │   │   ├── main.tf
│   │   │   ├── outputs.tf
│   │   │   ├── providers.tf
│   │   │   ├── variables.tf
│   │   │   └── versions.tf
│   │   └── qa
│   │       ├── backends.tf
│   │       ├── data.tf
│   │       ├── kube-prometheus-stack.yaml
│   │       ├── main.tf
│   │       ├── outputs.tf
│   │       ├── providers.tf
│   │       ├── variables.tf
│   │       └── versions.tf
│   ├── postgresql
│   │   └── qa
│   │       ├── backends.tf
│   │       ├── data.tf
│   │       ├── main.tf
│   │       ├── outputs.tf
│   │       ├── providers.tf
│   │       ├── variables.tf
│   │       └── versions.tf
│   ├── rapp
│   │   ├── prd
│   │   │   └── p-1
│   │   │       ├── core-api
│   │   │       │   ├── init.tf
│   │   │       │   ├── main.tf
│   │   │       │   └── vars.tf
│   │   │       ├── iot-segway-max-plus
│   │   │       │   ├── init.tf
│   │   │       │   ├── main.tf
│   │   │       │   └── vars.tf
│   │   │       └── ops-web
│   │   │           ├── init.tf
│   │   │           ├── main.tf
│   │   │           └── vars.tf
│   │   └── qa
│   │       ├── README.md
│   │       ├── commons
│   │       │   ├── README.md
│   │       │   ├── backends.tf
│   │       │   ├── data.tf
│   │       │   ├── main.tf
│   │       │   ├── outputs.tf
│   │       │   ├── providers.tf
│   │       │   ├── variables.tf
│   │       │   └── versions.tf
│   │       ├── qa-1
│   │       │   ├── core-api.tf
│   │       │   ├── init.tf
│   │       │   ├── iot-segway-max-plus-mock.tf
│   │       │   ├── iot-segway-max-plus.tf
│   │       │   ├── main.tf
│   │       │   └── vars.tf
│   │       ├── qa-2
│   │       │   ├── core-api.tf
│   │       │   ├── init.tf
│   │       │   ├── iot-segway-max-plus-mock.tf
│   │       │   ├── iot-segway-max-plus.tf
│   │       │   ├── main.tf
│   │       │   └── vars.tf
│   │       ├── qa-3
│   │       │   ├── core-api.tf
│   │       │   ├── init.tf
│   │       │   ├── iot-segway-max-plus-mock.tf
│   │       │   ├── iot-segway-max-plus.tf
│   │       │   ├── main.tf
│   │       │   └── vars.tf
│   │       └── stg-1
│   │           ├── core-api.tf
│   │           ├── init.tf
│   │           ├── iot-segway-max-plus-mock.tf
│   │           ├── iot-segway-max-plus.tf
│   │           ├── main.tf
│   │           └── vars.tf
│   ├── redis
│   │   ├── prd
│   │   │   ├── init.tf
│   │   │   ├── main.tf
│   │   │   ├── outputs.tf
│   │   │   └── vars.tf
│   │   └── qa
│   │       ├── backends.tf
│   │       ├── data.tf
│   │       ├── main.tf
│   │       ├── outputs.tf
│   │       ├── providers.tf
│   │       ├── variables.tf
│   │       └── versions.tf
│   └── s3
│       ├── global
│       │   ├── backend
│       │   │   ├── backend.tf
│       │   │   ├── initial.tfstate
│       │   │   ├── initial.tfstate.backup
│       │   │   ├── main.tf
│       │   │   ├── providers.tf
│       │   │   ├── variables.tf
│       │   │   └── versions.tf
│       │   ├── backends.tf
│       │   ├── main.tf
│       │   ├── providers.tf
│       │   ├── variables.tf
│       │   └── versions.tf
│       └── qa
│           ├── backends.tf
│           ├── main.tf
│           ├── providers.tf
│           ├── variables.tf
│           └── versions.tf
├── terraform.tfstate
└── test

78 directories, 328 files

And you will get result like this:

5 runs 'terrascan v1.62.2:'

time command	max	min	mean	median
users seconds	2.52	2.35	2.47	2.5
system seconds	0.28	0.17	0.226	0.24
CPU %	64	59	61.2	60
Total time	4.53	4.16	4.374	4.38

Run details

• Test Start: Wed Dec 22 16:08:33 UTC 2021
• Test End: Wed Dec 22 16:08:55 UTC 2021

Variable name	Value
TEST_NUM	5
TEST_COMMAND	pre-commit try-repo -a /mnt/c/Users/vm/code/open-source/pre-commit-terraform terrascan
TEST_DIR	/home/vm/code/Oslo
TEST_DESCRIPTION	5 runs 'terrascan v1.62.2:'
RAW_TEST_RESULTS_FILE_NAME	pr305

Memory info (head -n 6 /proc/meminfo):

MemTotal:       12765352 kB
MemFree:         6649596 kB
MemAvailable:    9093208 kB
Buffers:          324424 kB
Cached:          2127924 kB
SwapCached:            0 kB

CPU info:

Real procs: 6
Virtual (hyper-threading) procs: 12

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 165
model name	: Intel(R) Core(TM) i7-10850H CPU @ 2.70GHz
stepping	: 2
microcode	: 0xffffffff
cpu MHz		: 2712.007
cache size	: 12288 KB
physical id	: 0
siblings	: 12
core id		: 5
cpu cores	: 6
apicid		: 11
initial apicid	: 11
fpu		: yes
fpu_exception	: yes
cpuid level	: 21
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves flush_l1d arch_capabilities
bugs		: spectre_v1 spectre_v2 spec_store_bypass swapgs itlb_multihit
bogomips	: 5424.01
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

5 runs 'terrascanPR #305:'

time command	max	min	mean	median
users seconds	95.68	2.35	46.986	45.57
system seconds	4.23	0.17	1.997	1.8
CPU %	168	59	111.5	110
Total time	60.56	4.16	31.537	30.35

Run details

• Test Start: Wed Dec 22 16:09:13 UTC 2021
• Test End: Wed Dec 22 16:14:07 UTC 2021

Variable name	Value
TEST_NUM	5
TEST_COMMAND	pre-commit try-repo -a /tmp/305 terrascan
TEST_DIR	/home/vm/code/Oslo
TEST_DESCRIPTION	5 runs 'terrascanPR #305:'
RAW_TEST_RESULTS_FILE_NAME	pr305

Memory info (head -n 6 /proc/meminfo):

MemTotal:       12765352 kB
MemFree:         6618580 kB
MemAvailable:    9065164 kB
Buffers:          326176 kB
Cached:          2128860 kB
SwapCached:            0 kB

CPU info:

Real procs: 6
Virtual (hyper-threading) procs: 12

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 165
model name	: Intel(R) Core(TM) i7-10850H CPU @ 2.70GHz
stepping	: 2
microcode	: 0xffffffff
cpu MHz		: 2712.007
cache size	: 12288 KB
physical id	: 0
siblings	: 12
core id		: 5
cpu cores	: 6
apicid		: 11
initial apicid	: 11
fpu		: yes
fpu_exception	: yes
cpuid level	: 21
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves flush_l1d arch_capabilities
bugs		: spectre_v1 spectre_v2 spec_store_bypass swapgs itlb_multihit
bogomips	: 5424.01
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

[carlosbustillordguez]

Still facing problems with performance. I will try to investigate why

UPD. It is because v1.62.2 check only 1 directory, when this branch checks all. Try to decrease cont of run for terrascan

I just changed the code to scan only one directory:

pre-commit-terraform/terrascan.sh

Lines 15 to 22 in 677912b

	# consume modified files passed from pre-commit so that
	# terrascan runs against only those relevant directories
	# shellcheck disable=SC2128 # scan the first directory of the FILES array
	for file_with_path in $files; do
	file_with_path="${file_with_path// /__REPLACED__SPACE__}"
	paths[index]=$(dirname "$file_with_path")
	index=$((index + 1))
	done

It will be good to catch if the terrascan try-repo was issued with the -a argument, to only run terrascan in the first directory otherwise, run in directories with changed .tf files. But I don't know how to do that... any clue?

Please, let me know if the new change has improved the performance for your repo.

[MaxymVlasov]

I little bit check how terrascan works and what I found:
terrascan recursively check all files in provided dir.

So performance degradation exists only in the pre-commit run --all situation, because it will provide all existing files to hook:

pre-commit-terraform/terrascan.sh

Lines 15 to 19 in e6ffbcd

	# consume modified files passed from pre-commit so that
	# terrascan runs against only those relevant directories
	for file_with_path in "${files[@]}"; do
	file_with_path="${file_with_path// /__REPLACED__SPACE__}"
	paths[index]=$(dirname "$file_with_path")

Then, unique paths are found and run terrascan for each folder:

pre-commit-terraform/terrascan.sh

Lines 29 to 30 in e6ffbcd

	# for each path run terrascan
	for path_uniq in $(echo "${paths[*]}" | tr ' ' '\n' | sort -u); do

It works literally how it should work: checks only diffs.

At the same time, need to know when the --all (-a) argument passed to pre-commit and just run terrascan -d GIT_REPO_ROOT.
Because it is not very related to this PR, I propose to merge that PR as is, and open issue for improving performance during pre-commit -a run.

Because for now

only the first directory of the FILES array is only analyzed with terrascan.

And that is not a performance issue in day-to-day usage cases, it's a bug that should be fixed.

@carlosbustillordguez if you'd like to fix that too, you're welcome to fix #309, right after this PR merge.

@antonbabenko ready to merge

[antonbabenko]

@MaxymVlasov Could you please write a bit more descriptive title for the PRs? What exactly does this PR fix in the hook? It will be used in CHANGELOG and release notes.

[MaxymVlasov]

@antonbabenko done

[antonbabenko]

Keep in mind the length. :)

[antonbabenko]

This PR is included in version 1.62.3 🎉

[MaxymVlasov]

@carlosbustillordguez thank you for help during debugging.