Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an option to ensure product references are correct #55

Merged
merged 8 commits into from
Sep 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions .github/workflows/CI.yml
Original file line number Diff line number Diff line change
Expand Up @@ -63,14 +63,24 @@ jobs:
run: |
shopt -s globstar

license-scanner scan \
! license-scanner scan \
--ensure-licenses Apache-2.0 GPL-3.0-only \
--exclude ./**/target ./**/weights \
-- ./**/src/**/*.rs \
2>out.txt \
&& exit 1 || exit 0
2>out.txt
# We expected it to fail because there are some unlicensed files left.

grep -q 'No license detected in reconstruct.rs. Exact file path:' ./out.txt
grep -q 'No license detected in mod.rs. Exact file path:' ./out.txt
working-directory: polkadot
- name: Enforce product references in headers
run: |
! license-scanner scan \
--ensure-product 'Polkadot' \
-- ./xcm/src/lib.rs \
2>out.txt
# We expected it to fail because there are some copy-paste errors.

grep -q 'Product mismatch' ./out.txt
grep -q 'Expected "Polkadot", detected "Substrate" in line:' ./out.txt
working-directory: polkadot
22 changes: 22 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -276,6 +276,28 @@ yarn start -- scan --ensure-any-license /directory/or/file
Those options are conflicting with each other so only one should be specified.
By default, no licensing is enforced.

## `--ensure-product` <a name="ensure-product"></a>

If configured, the scan will make sure that if a license header references a product,
it will be the correct product and not a result of a copy-paste error.

For example, this fragment references the `Substrate` product.

```text
// Substrate is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
```

Examples:

```bash
yarn start -- scan --ensure-product Polkadot -- /directory/or/file
```

It treats a different product reference as an error, but it allows a generic "this program".

## `--exclude` <a name="exclude"></a>

Can be used to exclude files or directories from the scan.
Expand Down
2 changes: 2 additions & 0 deletions license-scanner/cli/scan.ts
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ export const executeScan = async function ({
detectionOverrides,
logLevel,
ensureLicenses,
ensureProduct,
exclude,
}: ScanCliArgs) {
const licenses = await loadLicensesNormalized(joinPath(buildRoot, "licenses"), {
Expand Down Expand Up @@ -119,6 +120,7 @@ export const executeScan = async function ({
detectionOverrides: detectionOverrides ?? null,
logger,
ensureLicenses,
ensureProduct,
});
allLicensingErrors.push(...licensingErrors);
}
Expand Down
28 changes: 28 additions & 0 deletions license-scanner/license.ts
Original file line number Diff line number Diff line change
Expand Up @@ -299,6 +299,34 @@ export const ensureLicensesInResult = function ({
}
};

/**
* If a product is mentioned in this file,
* ensure that it is the correct product,
* and not a copy-paste error from a different product.
*/
export const ensureProductInFile = function (filePath: string, product: string | undefined): Error | undefined {
if (!product) return;
const lines = fs.readFileSync(filePath, "utf8").split(/\r?\n/);
for (const regexp of [
new RegExp("This file is part of (.*)\\."),
new RegExp("// (.*) is free software"),
new RegExp("// (.*) is distributed in the hope"),
new RegExp("// along with (.+?)\\.(.*)gnu.org"),
]) {
for (const line of lines) {
if (regexp.test(line)) {
const matches = regexp.exec(line);
assert(matches);
if (matches[1] !== product && matches[1].toLowerCase() !== "this program") {
return new Error(
`Product mismatch in ${filePath}. Expected "${product}", detected "${matches[1]}" in line: "${line}".`,
);
}
}
}
}
};

export const throwLicensingErrors = function (licensingErrors: Error[]) {
if (licensingErrors.length === 0) return;
throw new Error(
Expand Down
7 changes: 7 additions & 0 deletions license-scanner/main.ts
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,12 @@ program
"If configured, the scan will make sure that all scanned files are licensed with any license.",
).conflicts("ensureLicenses"),
)
.addOption(
new Option(
"--ensure-product <product>",
"If configured, the scan will make sure the product mentioned in the license headers is correct.",
),
)
.option("--exclude <exclude...>", "Can be used to exclude files or directories from the scan.")
// It's actually correct usage but @commander-js/extra-typings is wrong on this one.
// eslint-disable-next-line @typescript-eslint/no-misused-promises
Expand All @@ -60,6 +66,7 @@ program
exclude: options.exclude ?? [],
logLevel: options.logLevel as LogLevel,
ensureLicenses: readEnsureLicenses(options),
ensureProduct: options.ensureProduct,
});
} catch (e: any) {
logger.debug(e.stack);
Expand Down
5 changes: 4 additions & 1 deletion license-scanner/scanner.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import assert from "assert";
import { dirname, join as joinPath, relative as relativePath } from "path";

import { getOrDownloadCrate, getVersionedCrateName } from "./crate";
import { ensureLicensesInResult } from "./license";
import { ensureLicensesInResult, ensureProductInFile } from "./license";
import { getOrDownloadRepository } from "./repository";
import { scanQueue, scanQueueSize } from "./synchronization";
import {
Expand Down Expand Up @@ -133,6 +133,7 @@ export const scan = async function (options: ScanOptions): Promise<ScanResult> {
tracker,
logger,
ensureLicenses = false,
ensureProduct,
} = options;

const licensingErrors: Error[] = [];
Expand Down Expand Up @@ -179,6 +180,8 @@ export const scan = async function (options: ScanOptions): Promise<ScanResult> {
const result = await matchLicense(file.path);
const licensingError = ensureLicensesInResult({ file, result, ensureLicenses });
if (licensingError) licensingErrors.push(licensingError);
const productError = ensureProductInFile(file.path, ensureProduct);
if (productError) licensingErrors.push(productError);
if (result === undefined) {
return;
}
Expand Down
6 changes: 6 additions & 0 deletions license-scanner/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,11 @@ export type ScanOptions = {
* all source files have one of those licenses detected.
*/
ensureLicenses?: boolean | string[];
/**
* If true, the scan will make sure that
* the license headers contain the correct product name.
*/
ensureProduct?: string | undefined;
};

export type LicenseInput = {
Expand Down Expand Up @@ -150,6 +155,7 @@ export interface ScanCliArgs {
detectionOverrides: DetectionOverride[];
logLevel: LogLevel;
ensureLicenses: boolean | string[];
ensureProduct: string | undefined;
}

export interface DumpCliArgs {
Expand Down
Loading