Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Change target for container vuln scanning tests #343

Merged
merged 6 commits into from
Mar 20, 2021
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,10 @@ executors:
resource_class: arm.medium

jobs:
prepare-test-resources:
executor: go-executor
steps:
- run: make test-resources
unit-test:
executor: go-executor
steps:
Expand Down Expand Up @@ -224,6 +228,7 @@ workflows:
branches:
only: main
jobs:
- prepare-test-resources
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we only want to run this on nightly?

- unit-test
- build-cli
- integration-test-linux:
Expand Down
3 changes: 3 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,9 @@ generate-databox:
generate-docs:
go generate cli/cmd/docs.go

test-resources:
scripts/prepare_test_resources.sh all

install-cli: build-cli-cross-platform
ifeq (x86_64, $(shell uname -m))
mv bin/$(PACKAGENAME)-$(shell uname -s | tr '[:upper:]' '[:lower:]')-amd64 /usr/local/bin/$(CLINAME)
Expand Down
3 changes: 1 addition & 2 deletions api/_examples/pagerduty-alert-channel/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,7 @@ func main() {

alert := api.NewPagerDutyAlertChannel("pagerduty-alert-from-golang",
api.PagerDutyData{
IntegrationKey: "1234abc8901abc567abc123abc78e012",
MinAlertSeverity: 5,
IntegrationKey: "1234abc8901abc567abc123abc78e012",
},
)

Expand Down
26 changes: 12 additions & 14 deletions integration/container_vulnerability_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,8 @@ import (
)

const (
registry = "index.docker.io"
repository = "lacework/lacework-cli"
tag1 = "ubuntu-1804"
tag2 = "debian-10"
registry = "index.docker.io"
cleanRepository = "techallylw/test-cli-clean"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How so you are using the clean docker image? I see it does have one vulnerability:

https://4451-242235794-gh.circle-artifacts.com/0/circleci-artifacts/techallylw-test-cli-clean-sha256%3A7bf7cef98cceeb038437e7af713153ea28e93a00875d11f284b6f7c6dcb26ba9.html

Two questions:

  • Shouldn't we need to assert that the clean image has NO vulnerabilities? (but it does 🤔)
  • Shouldn't we be using the dirty image here instead?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only if the tests expect vulns we should use dirty container. It was vuln free initially. Having something guaranteed to be vuln free will be difficult. Best we can do is hope that patches happen quickly for new CVE's. Perhaps there's a base image with a focus on quickly patching new cve's?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But you're right for 1 of the tests that rely on the assessment output from this scan expect vulns. So we should switch to to the dirty container.

)

func TestContainerVulnerabilityCommandAliases(t *testing.T) {
Expand Down Expand Up @@ -75,7 +73,7 @@ func TestContainerVulnerabilityCommandListAssessments(t *testing.T) {
// verify that at least our lacework-cli container is listed
expectedFields := []string{
registry,
repository,
cleanRepository,
"Success", // status
"sha256:", // image digest
}
Expand All @@ -92,11 +90,11 @@ func TestContainerVulnerabilityCommandScanHumanReadablePollGenerateHtml(t *testi
home := createTOMLConfigFromCIvars()
defer os.RemoveAll(home)
out, err, exitcode := LaceworkCLIWithHome(home,
"vulnerability", "container", "scan", registry, repository, tag2, "--poll", "--html")
"vulnerability", "container", "scan", registry, cleanRepository, "latest", "--poll", "--html")

assert.Contains(t, out.String(), "A new vulnerability scan has been requested. (request_id:",
"STDOUT changed, please check")
assert.Contains(t, out.String(), "The container vulnerability assessment was stored at 'lacework-lacework-cli-sha256",
assert.Contains(t, out.String(), "The container vulnerability assessment was stored at 'techallylw-test-cli-clean-sha256",
"STDOUT changed, please check")
assert.Empty(t,
err.String(),
Expand All @@ -111,7 +109,7 @@ func TestContainerVulnerabilityCommandScanHumanReadablePollGenerateHtml(t *testi
imageDigest = shas[len(shas)-1]
)
assert.NotEmpty(t, imageDigest, "unable to extract image digest")
htmlFile := path.Join(home, fmt.Sprintf("lacework-lacework-cli-%s.html", imageDigest))
htmlFile := path.Join(home, fmt.Sprintf("techallylw-test-cli-clean-%s.html", imageDigest))
assert.FileExists(t, htmlFile, "the HTML file was not generated")
storeFileInCircleCI(htmlFile)
})
Expand All @@ -134,9 +132,9 @@ func TestContainerVulnerabilityCommandsEndToEnd(t *testing.T) {
// "requestId": "e94f2774-5662-4510-8ebf-2d5e3cd317f6",
// "status": "Scanning"
// }
t.Run(fmt.Sprintf("run scan for %s/%s:%s", registry, repository, tag1), func(t *testing.T) {
t.Run(fmt.Sprintf("run scan for %s/%s", registry, cleanRepository), func(t *testing.T) {
out, err, exitcode = LaceworkCLIWithTOMLConfig(
"vulnerability", "container", "scan", registry, repository, tag1, "--json")
"vulnerability", "container", "scan", registry, cleanRepository, "latest", "--json")
assert.Empty(t,
err.String(),
"STDERR should be empty")
Expand Down Expand Up @@ -189,8 +187,8 @@ func TestContainerVulnerabilityCommandsEndToEnd(t *testing.T) {

// fields
"Registry " + registry,
"Repository " + repository,
"Tags " + tag1,
"Repository " + cleanRepository,
"Tags " + "latest",
"Size",
"ID",
"Digest",
Expand Down Expand Up @@ -271,7 +269,7 @@ func TestContainerVulnerabilityCommandsEndToEnd(t *testing.T) {
"STDERR should be empty")
assert.Equal(t, 0, exitcode,
"EXITCODE is not the expected one")
assert.Contains(t, out.String(), "The container vulnerability assessment was stored at 'lacework-lacework-cli-sha256",
assert.Contains(t, out.String(), "The container vulnerability assessment was stored at 'techallylw-test-cli-clean-sha256",
"STDOUT changed, please check")

assert.NotContains(t, out.String(), "Try adding '--details' to increase details shown about the vulnerability assessment.",
Expand All @@ -284,7 +282,7 @@ func TestContainerVulnerabilityCommandsEndToEnd(t *testing.T) {
imageDigest = shas[len(shas)-1]
)
assert.NotEmpty(t, imageDigest, "unable to extract image digest")
htmlFile := path.Join(home, fmt.Sprintf("lacework-lacework-cli-%s.html", imageDigest))
htmlFile := path.Join(home, fmt.Sprintf("techallylw-test-cli-clean-%s.html", imageDigest))
assert.FileExists(t, htmlFile, "the HTML file was not generated")
storeFileInCircleCI(htmlFile)
})
Expand Down
2 changes: 2 additions & 0 deletions integration/test_resources/clean.Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
FROM alpine:latest as alpine

1 change: 1 addition & 0 deletions integration/test_resources/vuln_scan/dirty.Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
FROM node:latest
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pin this version instead of using latest

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, the latest Node container is dirty?!!! 😮

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, quick and easy way to get a vulnerable container. Just add NPM.

42 changes: 42 additions & 0 deletions scripts/prepare_test_resources.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
#!/bin/bash
#
# Name:: prepare_test_resources.sh
# Description:: Make ready resources required by integration tests
# Author:: Darren Murray (<[email protected]>)
#

main() {
if [[ -z $DOCKERHUB_PASS ]]; then
echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin
fi

case "${1:-}" in
clean)
build_clean
;;
dirty)
build_dirty
;;
all)
build_clean
build_dirty
;;
*)
echo "invalid argument"
;;
esac
}

build_clean() {
echo "building clean container"
docker build --no-cache -f "integration/test_resources/clean.Dockerfile" -t techallylw/test-cli-clean .
docker push techallylw/test-cli-clean
}

build_dirty() {
echo "building dirty container"
docker build --no-cache -f "integration/test_resources/vuln_scan/dirty.Dockerfile" -t techallylw/test-cli-dirty .
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for --no-cache on this one

docker push techallylw/test-cli-dirty
}

main "$@" || exit 99