Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[infra] Improve srcmap support for Go projects (#3355, #2714). #3664

Merged
merged 2 commits into from
Apr 17, 2020
Merged

[infra] Improve srcmap support for Go projects (#3355, #2714). #3664

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d69555c
[infra] Improve srcmap support for Go projects (#3355, #2714).
Dor1s Apr 16, 2020
c3b93e7
address review feedback
Dor1s Apr 17, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions infra/base-images/base-builder/srcmap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,14 @@ function jq_inplace() {
F=$(tempfile) && cat $1 | jq "$2" > $F && mv $F $1
}

PATHS_TO_SCAN="$SRC"

if [[ $FUZZING_LANGUAGE == "go" ]]; then
PATHS_TO_SCAN="$PATHS_TO_SCAN $GOPATH"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's in $GOPATH if this isn't a Go project? Can we use that to determining whether or not to scan it, rather than introducing another env var?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's always set to /root/go in the base-builder and always has a few packages we install, so scanning unconditionally will result in irrelevant Go packages revisions in srcmap of any project we build

fi

# Git
for DOT_GIT_DIR in $(find $SRC -name ".git" -type d); do
for DOT_GIT_DIR in $(find $PATHS_TO_SCAN -name ".git" -type d); do
GIT_DIR=$(dirname $DOT_GIT_DIR)
cd $GIT_DIR
GIT_URL=$(git config --get remote.origin.url)
Expand All @@ -35,7 +41,7 @@ for DOT_GIT_DIR in $(find $SRC -name ".git" -type d); do
done

# Subversion
for DOT_SVN_DIR in $(find $SRC -name ".svn" -type d); do
for DOT_SVN_DIR in $(find $PATHS_TO_SCAN -name ".svn" -type d); do
SVN_DIR=$(dirname $DOT_SVN_DIR)
cd $SVN_DIR
SVN_URL=$(svn info | grep "^URL:" | sed 's/URL: //g')
Expand All @@ -44,7 +50,7 @@ for DOT_SVN_DIR in $(find $SRC -name ".svn" -type d); do
done

# Mercurial
for DOT_HG_DIR in $(find $SRC -name ".hg" -type d); do
for DOT_HG_DIR in $(find $PATHS_TO_SCAN -name ".hg" -type d); do
HG_DIR=$(dirname $DOT_HG_DIR)
cd $HG_DIR
HG_URL=$(hg paths default)
Expand Down
29 changes: 2 additions & 27 deletions infra/gcb/build_and_run_coverage.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,35 +71,10 @@ def get_build_steps(project_dir):
dockerfile_path = os.path.join(project_dir, 'Dockerfile')
name = project_yaml['name']
image = project_yaml['image']
language = project_yaml['language']
report_date = datetime.datetime.now().strftime('%Y%m%d')

build_steps = [
{
'args': [
'clone',
'https://github.com/google/oss-fuzz.git',
],
'name': 'gcr.io/cloud-builders/git',
},
{
'name': 'gcr.io/cloud-builders/docker',
'args': [
'build',
'-t',
image,
'.',
],
'dir': 'oss-fuzz/projects/' + name,
},
{
'name': image,
'args': [
'bash', '-c',
'srcmap > /workspace/srcmap.json && cat /workspace/srcmap.json'
],
'env': ['OSSFUZZ_REVISION=$REVISION_ID'],
},
]
build_steps = build_lib.project_image_steps(name, image, language)

env = CONFIGURATION[:]
out = '/workspace/out/' + SANITIZER
Expand Down
33 changes: 33 additions & 0 deletions infra/gcb/build_lib.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,3 +165,36 @@ def gsutil_rm_rf_step(url):
],
}
return step


def project_image_steps(name, image, language):
"""Returns GCB steps to build OSS-Fuzz project image."""
steps = [{
'args': [
'clone',
'https://github.com/google/oss-fuzz.git',
],
'name': 'gcr.io/cloud-builders/git',
}, {
'name': 'gcr.io/cloud-builders/docker',
'args': [
'build',
'-t',
image,
'.',
],
'dir': 'oss-fuzz/projects/' + name,
}, {
'name':
image,
'args': [
'bash', '-c',
'srcmap > /workspace/srcmap.json && cat /workspace/srcmap.json'
],
'env': [
'OSSFUZZ_REVISION=$REVISION_ID',
'FUZZING_LANGUAGE=%s' % language,
],
}]

return steps
47 changes: 12 additions & 35 deletions infra/gcb/build_project.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,45 +109,22 @@ def get_build_steps(project_dir):
dockerfile_path = os.path.join(project_dir, 'Dockerfile')
name = project_yaml['name']
image = project_yaml['image']
language = project_yaml['language']
run_tests = project_yaml['run_tests']

ts = datetime.datetime.now().strftime('%Y%m%d%H%M')

build_steps = [
{
'args': [
'clone',
'https://github.com/google/oss-fuzz.git',
],
'name': 'gcr.io/cloud-builders/git',
},
{
'name': 'gcr.io/cloud-builders/docker',
'args': [
'build',
'-t',
image,
'.',
],
'dir': 'oss-fuzz/projects/' + name,
},
{
'name': image,
'args': [
'bash', '-c',
'srcmap > /workspace/srcmap.json && cat /workspace/srcmap.json'
],
'env': ['OSSFUZZ_REVISION=$REVISION_ID'],
},
{
'name': 'gcr.io/oss-fuzz-base/msan-builder',
'args': [
'bash',
'-c',
'cp -r /msan /workspace',
],
},
]
build_steps = build_lib.project_image_steps(name, image, language)

# Copy over MSan instrumented libraries.
build_steps.append({
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a comment on why msan-builder is only used here ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done, it's not used in the coverage job because we don't use MSan there

'name': 'gcr.io/oss-fuzz-base/msan-builder',
'args': [
'bash',
'-c',
'cp -r /msan /workspace',
],
})

for fuzzing_engine in project_yaml['fuzzing_engines']:
for sanitizer in get_sanitizers(project_yaml):
Expand Down