Skip to content

Commit

Permalink
Merge pull request #1692 from snyk/IAC-2657/remove-deep-mode
Browse files Browse the repository at this point in the history
Remove deep mode
  • Loading branch information
chdorner-snyk committed Oct 25, 2023
2 parents 82ddf50 + 26ffd12 commit 90874b6
Show file tree
Hide file tree
Showing 259 changed files with 3,071 additions and 5,274 deletions.
10 changes: 10 additions & 0 deletions .gitleaksignore
Original file line number Diff line number Diff line change
Expand Up @@ -281,3 +281,13 @@ ff72de8e77f908fba61df50bc0938744270d1b51:pkg/remote/aws/test/iam_role_multiple/r
ff72de8e77f908fba61df50bc0938744270d1b51:pkg/remote/aws/test/iam_user_multiple/results.golden.json:aws-access-token:12
ff72de8e77f908fba61df50bc0938744270d1b51:pkg/remote/aws/test/iam_user_multiple/results.golden.json:aws-access-token:24
ff72de8e77f908fba61df50bc0938744270d1b51:pkg/remote/aws/test/iam_user_multiple/results.golden.json:aws-access-token:36
20e6356a49e6dd18f00cd6c36b735ef1a850ac55:enumeration/remote/aws_iam_scanner_test.go:aws-access-token:823
20e6356a49e6dd18f00cd6c36b735ef1a850ac55:enumeration/remote/aws_iam_scanner_test.go:aws-access-token:826
651ab697db3ff60ba195c22dc0570d1204a97f7e:enumeration/remote/aws_iam_scanner_test.go:aws-access-token:823
651ab697db3ff60ba195c22dc0570d1204a97f7e:enumeration/remote/aws_iam_scanner_test.go:aws-access-token:826
6cf09f996d8637c30ad06884a450ff66920d9798:enumeration/remote/aws_iam_scanner_test.go:aws-access-token:823
6cf09f996d8637c30ad06884a450ff66920d9798:enumeration/remote/aws_iam_scanner_test.go:aws-access-token:826
6d204a7f446251a3c1519bfc4b80599529eec279:enumeration/remote/aws_iam_scanner_test.go:aws-access-token:823
6d204a7f446251a3c1519bfc4b80599529eec279:enumeration/remote/aws_iam_scanner_test.go:aws-access-token:826
40f68d61a91d8c10c09e43263cbd36b380cca90a:enumeration/remote/aws_iam_scanner_test.go:aws-access-token:823
40f68d61a91d8c10c09e43263cbd36b380cca90a:enumeration/remote/aws_iam_scanner_test.go:aws-access-token:826
5 changes: 5 additions & 0 deletions .snyk
Original file line number Diff line number Diff line change
Expand Up @@ -85,4 +85,9 @@ ignore:
- '*':
reason: This license is addressed by including acknowledgments in each release
created: 2022-09-09T14:25:05.042Z
SNYK-GOLANG-GOLANGORGXNETHTTP2-5953327:
- '*':
reason: Not affected because CLI
expires: 2024-04-15T15:15:28.330Z
created: 2023-10-16T15:15:28.356Z
patch: {}
1 change: 0 additions & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,3 @@ Resource listing is done using cloud providers SDK. Resource details retrieval i
- `Remote` is a representation of a cloud provider
- `Resource` is an abstract representation of a cloud provider resource (e.g. S3 bucket, EC2 instance, etc ...)
- `Enumerator` is used to list resources of a given type from a given remote and return a resource list, it should exist only one Enumerator per resource
- `DetailsFetcher` is used to retrieve resource's details of a given type, this is an optional layer and is used only in deep mode.
Binary file modified docs/media/generalflow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/media/resource.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
13 changes: 1 addition & 12 deletions docs/media/resource.puml
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,5 @@ RemoteSDK --> Enumerator: Attrs
end
Enumerator --> Scanner: []Resource with limited attributes
end
alt if deep mode enabled
hnote across: Details fetching phase
loop for each enumerated resource
Scanner -> DetailsFetcher: ReadDetails(res)
DetailsFetcher -> TerraformProvider: ReadResource()
TerraformProvider --> DetailsFetcher: CTYValue
DetailsFetcher -> Deserializer: Deserialize()
Deserializer -> DetailsFetcher: Resource
DetailsFetcher -> Scanner: Resource with\nfull attributes
end
end
Scanner --> driftctl: []Resource
@enduml
@enduml
43 changes: 2 additions & 41 deletions docs/new-resource.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,22 +13,12 @@ Then, you'll find below a more detailed flow of how we handle the enumeration an
First step would be to add a file called `pkg/resource/<providername>/<resourcetype>.go`.
This file will define a string constant that will be the resource type identifier in driftctl.

Optionally, if your resource is to be supported by driftctl experimental deep mode, you can add a function that will be applied to this resource at creation.
This allows to prevent useless diffs to be displayed.
You can also add metadata to fields so that they are compared or displayed differently.

For example this defines the `aws_iam_role` resource:

```go
const AwsIamRoleResourceType = "aws_iam_role"

func initAwsIAMRoleMetaData(resourceSchemaRepository resource.SchemaRepositoryInterface) {
// assume_role_policy drifts will be displayed as json
resourceSchemaRepository.UpdateSchema(AwsIamRoleResourceType, map[string]func(attributeSchema *resource.AttributeSchema){
"assume_role_policy": func(attributeSchema *resource.AttributeSchema) {
attributeSchema.JsonString = true
},
})
// force_detach_policies should not be compared so it will be removed before the comparison
resourceSchemaRepository.SetNormalizeFunc(AwsIamRoleResourceType, func(res *resource.Resource) {
val := res.Attrs
Expand Down Expand Up @@ -57,17 +47,12 @@ var supportedTypes = map[string]struct{}{
All resources inside driftctl are `resource.Resource` structs.
All the other attributes are represented inside a `map[string]interface`

## Repository, Enumerator and DetailsFetcher
## Repository, Enumerator

Then you will have to implement two interfaces:
Then you will have to implement one interface:

- Repositories are the way we decided to hide direct calls to SDK and pagination logic. It's a common abstraction pattern for data retrieval.
- `remote.common.Enumerator` is used to enumerate resources. It will call the cloud provider SDK to get the list of resources.
For some resource it could make other call to enrich the resource with additional attributes when driftctl is used in deep mode
- `remote.common.DetailsFetcher` is used to retrieve resource's details. It makes a call to Terraform provider `ReadResource`.
This implementation is optional and is only needed if your resource type is to be supported by experimental deep mode.
Please also note that it exists a generic implementation called `remote.common.GenericDetailsFetcher` that can be used with most resource types.


### Repository

Expand Down Expand Up @@ -126,7 +111,6 @@ Most of the resource returned by enumerator have empty attributes: they only rep

**There are exceptions to this**:
- Sometimes, you will need more information about resources for them to be fetched in the `DetailsFetcher`. For those cases, you will add specific attributes to the map of data.
- For complex cases (e.g. middlewares) where you would need driftctl to run as expected in deep and non-deep mode, you would need to enumerate resources as well as to fetch manually specific attributes, using the remote SDK, before adding them to the map of data.

You can use an already implemented Enumerator as example.

Expand Down Expand Up @@ -192,26 +176,3 @@ Once the enumerator is written you have to add it to the remote initialization l
```go
remoteLibrary.AddEnumerator(NewEC2InstanceEnumerator(s3Repository, factory))
```

### DetailsFetcher

DetailsFetchers are only used by driftctl experimental deep mode.

This is the component that call Terraform provider to retrieve all attributes for each resource.
We do not want to reimplement what has already been done in each Terraform provider. Thus, you should not call the remote SDK there.

If `common.GenericDetailsFetcher` satisfies your needs you should always prefer using it instead of implementing a custom `DetailsFetcher` in a new struct.

The `DetailsFetcher` should also be added to `pkg/remote/<providername>/init.go` even if you use the generic version:

```go
remoteLibrary.AddDetailsFetcher(aws.AwsEbsVolumeResourceType, common.NewGenericDetailsFetcher(aws.AwsEbsVolumeResourceType, provider, deserializer))
```

***Don't forget to add unit tests after adding a new resource.***

You can find example of **functional tests** in `pkg/remote/<type>_scanner_test.go`.

You should also add **acceptance tests** if you think it makes sense. They are located next to the resource definition described in the first step.

More information about adding tests can be found in [testing documentation](testing.md)
Loading

0 comments on commit 90874b6

Please sign in to comment.