Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dependencies check script #3941

Merged
merged 6 commits into from
May 22, 2019
Merged

Add dependencies check script #3941

merged 6 commits into from
May 22, 2019

Conversation

htynkn
Copy link
Member

@htynkn htynkn commented Apr 27, 2019

What is the purpose of the change

Run script to get dependencies license list

For example:
licenseCheck.sh - will check whole project
licenseCheck.sh module-name - will only check specific module

Result will save in license-list.txt file

Brief changelog

  1. Add script
  2. Add license plugin config
  3. Add missing file

Verifying this change

test script

Follow this checklist to help us incorporate your contribution quickly and easily:

  • Make sure there is a GITHUB_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GITHUB issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
  • Format the pull request title like [Dubbo-XXX] Fix UnknownException when host config not exist #XXX. Each commit in the pull request should have a meaningful subject line and body.
  • Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
  • Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
  • Run mvn clean install -DskipTests=false & mvn clean test-compile failsafe:integration-test to make sure unit-test and integration-test pass.
  • If this contribution is large, please follow the Software Donation Guide.

@htynkn
Copy link
Member Author

htynkn commented Apr 27, 2019

Close #3840
Close #3790

@htynkn
Copy link
Member Author

htynkn commented Apr 27, 2019

Some example for how license-list.txt file looks like

==============================================
./dubbo-cluster/target/generated-sources/license/THIRD-PARTY.txt

Lists of 41 third-party dependencies.
     (Apache License, Version 2.0) Code Generation Library (cglib:cglib-nodep:2.2 - http://cglib.sourceforge.net/)
     (Apache License, Version 2.0) fastjson (com.alibaba:fastjson:1.2.46 - https://github.com/alibaba/fastjson)
     (Apache License, Version 2.0) Hessian Lite(Alibaba embed version) (com.alibaba:hessian-lite:3.2.5 - https://github.com/dubbo/hessian-lite)
     (Apache License, Version 2.0) java-util (com.cedarsoftware:java-util:1.9.0 - https://github.com/jdereg/java-util)
     (Apache License, Version 2.0) json-io (com.cedarsoftware:json-io:2.5.1 - https://github.com/jdereg/json-io)
     (3-Clause BSD License) Kryo (com.esotericsoftware:kryo:4.0.1 - https://github.com/EsotericSoftware/kryo/kryo)
     (The 3-Clause BSD License) MinLog (com.esotericsoftware:minlog:1.3.0 - https://github.com/EsotericSoftware/minlog)
     (The 3-Clause BSD License) ReflectASM (com.esotericsoftware:reflectasm:1.11.3 - https://github.com/EsotericSoftware/reflectasm)
     (Apache License, Version 2.0) Jackson-core (com.fasterxml.jackson.core:jackson-core:2.8.6 - https://github.com/FasterXML/jackson-core)
     (Apache License, Version 2.0) Guava: Google Core Libraries for Java (com.google.guava:guava:20.0 - https://github.com/google/guava/guava)
     (Apache License, Version 2.0) Apache Commons Logging (commons-logging:commons-logging:1.2 - http://commons.apache.org/proper/commons-logging/)
     (Apache License, Version 2.0) kryo serializers (de.javakaffee:kryo-serializers:0.42 - https://github.com/magro/kryo-serializers)
     (Apache License, Version 2.0) fst (de.ruedigermoeller:fst:2.48-jdk-6 - http://ruedigermoeller.github.io/fast-serialization/)
     (The 3-Clause BSD License) JLine (jline:jline:0.9.94 - http://jline.sourceforge.net)
     (Apache License, Version 2.0) Apache Log4j (log4j:log4j:1.2.16 - http://logging.apache.org/log4j/1.2/)
     (Apache License, Version 2.0) Byte Buddy (without dependencies) (net.bytebuddy:byte-buddy:1.9.3 - http://bytebuddy.net/byte-buddy)
     (Apache License, Version 2.0) Byte Buddy Java agent (net.bytebuddy:byte-buddy-agent:1.9.3 - http://bytebuddy.net/byte-buddy-agent)
     (Apache License, Version 2.0) Curator Client (org.apache.curator:curator-client:4.0.1 - http://curator.apache.org/curator-client)
     (Apache License, Version 2.0) Curator Framework (org.apache.curator:curator-framework:4.0.1 - http://curator.apache.org/curator-framework)
     (Apache License, Version 2.0) dubbo-common (org.apache.dubbo:dubbo-common:2.7.2-SNAPSHOT - https://github.com/apache/incubator-dubbo/dubbo-common)
     (Apache License, Version 2.0) dubbo-configcenter-api (org.apache.dubbo:dubbo-configcenter-api:2.7.2-SNAPSHOT - https://github.com/apache/incubator-dubbo/dubbo-configcenter/dubbo-configcenter-api)
     (Apache License, Version 2.0) dubbo-remoting-api (org.apache.dubbo:dubbo-remoting-api:2.7.2-SNAPSHOT - https://github.com/apache/incubator-dubbo/dubbo-remoting/dubbo-remoting-api)
     (Apache License, Version 2.0) dubbo-rpc-api (org.apache.dubbo:dubbo-rpc-api:2.7.2-SNAPSHOT - https://github.com/apache/incubator-dubbo/dubbo-rpc/dubbo-rpc-api)
     (Apache License, Version 2.0) dubbo-serialization-api (org.apache.dubbo:dubbo-serialization-api:2.7.2-SNAPSHOT - https://github.com/apache/incubator-dubbo/dubbo-serialization/dubbo-serialization-api)
     (Apache License, Version 2.0) Apache Yetus - Audience Annotations (org.apache.yetus:audience-annotations:0.5.0 - https://yetus.apache.org/audience-annotations)
     (Apache License, Version 2.0) zookeeper (org.apache.zookeeper:zookeeper:3.4.13 - no url defined)
     (Apache License, Version 2.0) org.apiguardian:apiguardian-api (org.apiguardian:apiguardian-api:1.0.0 - https://github.com/apiguardian-team/apiguardian)
     (The 3-Clause BSD License) Hamcrest All (org.hamcrest:hamcrest-all:1.3 - https://github.com/hamcrest/JavaHamcrest/hamcrest-all)
     (Apache License, Version 2.0) (LGPL 2.1) (MPL 1.1) Javassist (org.javassist:javassist:3.20.0-GA - http://www.javassist.org/)
     (Eclipse Public License v2.0) JUnit Jupiter API (org.junit.jupiter:junit-jupiter-api:5.4.0 - https://junit.org/junit5/)
     (Eclipse Public License v2.0) JUnit Jupiter Engine (org.junit.jupiter:junit-jupiter-engine:5.4.0 - https://junit.org/junit5/)
     (Eclipse Public License v2.0) JUnit Jupiter Params (org.junit.jupiter:junit-jupiter-params:5.4.0 - https://junit.org/junit5/)
     (Eclipse Public License v2.0) JUnit Platform Commons (org.junit.platform:junit-platform-commons:1.4.0 - https://junit.org/junit5/)
     (Eclipse Public License v2.0) JUnit Platform Engine API (org.junit.platform:junit-platform-engine:1.4.0 - https://junit.org/junit5/)
     (The MIT License) mockito-core (org.mockito:mockito-core:2.23.4 - https://github.com/mockito/mockito)
     (Apache License, Version 2.0) Objenesis (org.objenesis:objenesis:2.6 - http://objenesis.org)
     (Apache License, Version 2.0) org.opentest4j:opentest4j (org.opentest4j:opentest4j:1.1.1 - https://github.com/ota4j-team/opentest4j)
     (The 3-Clause BSD License) ASM Core (org.ow2.asm:asm:5.0.4 - http://asm.objectweb.org/asm/)
     (The MIT License) SLF4J API Module (org.slf4j:slf4j-api:1.7.25 - http://www.slf4j.org)
     (The MIT License) SLF4J LOG4J-12 Binding (org.slf4j:slf4j-log4j12:1.7.25 - http://www.slf4j.org)
     (Apache License, Version 2.0) SnakeYAML (org.yaml:snakeyaml:1.20 - http://www.snakeyaml.org)
==============================================
./dubbo-container/dubbo-container-spring/target/generated-sources/license/THIRD-PARTY.txt

Lists of 34 third-party dependencies.
     (Apache License, Version 2.0) Code Generation Library (cglib:cglib-nodep:2.2 - http://cglib.sourceforge.net/)
     (Apache License, Version 2.0) fastjson (com.alibaba:fastjson:1.2.46 - https://github.com/alibaba/fastjson)
     (Apache License, Version 2.0) Hessian Lite(Alibaba embed version) (com.alibaba:hessian-lite:3.2.5 - https://github.com/dubbo/hessian-lite)
     (Apache License, Version 2.0) java-util (com.cedarsoftware:java-util:1.9.0 - https://github.com/jdereg/java-util)
     (Apache License, Version 2.0) json-io (com.cedarsoftware:json-io:2.5.1 - https://github.com/jdereg/json-io)
     (3-Clause BSD License) Kryo (com.esotericsoftware:kryo:4.0.1 - https://github.com/EsotericSoftware/kryo/kryo)
     (The 3-Clause BSD License) MinLog (com.esotericsoftware:minlog:1.3.0 - https://github.com/EsotericSoftware/minlog)
     (The 3-Clause BSD License) ReflectASM (com.esotericsoftware:reflectasm:1.11.3 - https://github.com/EsotericSoftware/reflectasm)
     (Apache License, Version 2.0) Jackson-core (com.fasterxml.jackson.core:jackson-core:2.8.6 - https://github.com/FasterXML/jackson-core)
     (Apache License, Version 2.0) Apache Commons Logging (commons-logging:commons-logging:1.2 - http://commons.apache.org/proper/commons-logging/)
     (Apache License, Version 2.0) kryo serializers (de.javakaffee:kryo-serializers:0.42 - https://github.com/magro/kryo-serializers)
     (Apache License, Version 2.0) fst (de.ruedigermoeller:fst:2.48-jdk-6 - http://ruedigermoeller.github.io/fast-serialization/)
     (Apache License, Version 2.0) Apache Log4j (log4j:log4j:1.2.16 - http://logging.apache.org/log4j/1.2/)
     (Apache License, Version 2.0) Byte Buddy (without dependencies) (net.bytebuddy:byte-buddy:1.9.3 - http://bytebuddy.net/byte-buddy)
     (Apache License, Version 2.0) Byte Buddy Java agent (net.bytebuddy:byte-buddy-agent:1.9.3 - http://bytebuddy.net/byte-buddy-agent)
     (Apache License, Version 2.0) dubbo-common (org.apache.dubbo:dubbo-common:2.7.2-SNAPSHOT - https://github.com/apache/incubator-dubbo/dubbo-common)
     (Apache License, Version 2.0) dubbo-container-api (org.apache.dubbo:dubbo-container-api:2.7.2-SNAPSHOT - https://github.com/apache/incubator-dubbo/dubbo-container/dubbo-container-api)
     (Apache License, Version 2.0) org.apiguardian:apiguardian-api (org.apiguardian:apiguardian-api:1.0.0 - https://github.com/apiguardian-team/apiguardian)
     (The 3-Clause BSD License) Hamcrest All (org.hamcrest:hamcrest-all:1.3 - https://github.com/hamcrest/JavaHamcrest/hamcrest-all)
     (Apache License, Version 2.0) (LGPL 2.1) (MPL 1.1) Javassist (org.javassist:javassist:3.20.0-GA - http://www.javassist.org/)
     (Eclipse Public License v2.0) JUnit Jupiter API (org.junit.jupiter:junit-jupiter-api:5.4.0 - https://junit.org/junit5/)
     (Eclipse Public License v2.0) JUnit Jupiter Engine (org.junit.jupiter:junit-jupiter-engine:5.4.0 - https://junit.org/junit5/)
     (Eclipse Public License v2.0) JUnit Jupiter Params (org.junit.jupiter:junit-jupiter-params:5.4.0 - https://junit.org/junit5/)
     (Eclipse Public License v2.0) JUnit Platform Commons (org.junit.platform:junit-platform-commons:1.4.0 - https://junit.org/junit5/)
     (Eclipse Public License v2.0) JUnit Platform Engine API (org.junit.platform:junit-platform-engine:1.4.0 - https://junit.org/junit5/)
     (The MIT License) mockito-core (org.mockito:mockito-core:2.23.4 - https://github.com/mockito/mockito)

@codecov-io
Copy link

codecov-io commented Apr 30, 2019

Codecov Report

Merging #3941 into master will decrease coverage by 0.07%.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #3941      +/-   ##
============================================
- Coverage     63.94%   63.87%   -0.08%     
  Complexity       98       98              
============================================
  Files           715      715              
  Lines         31493    31493              
  Branches       5074     5074              
============================================
- Hits          20139    20115      -24     
- Misses         9052     9074      +22     
- Partials       2302     2304       +2
Impacted Files Coverage Δ Complexity Δ
.../apache/dubbo/qos/protocol/QosProtocolWrapper.java 64.1% <0%> (-17.95%) 0% <0%> (ø)
...ache/dubbo/remoting/p2p/support/AbstractGroup.java 45.45% <0%> (-11.37%) 0% <0%> (ø)
.../apache/dubbo/remoting/transport/AbstractPeer.java 63.04% <0%> (-8.7%) 0% <0%> (ø)
...ng/exchange/support/header/HeartbeatTimerTask.java 73.68% <0%> (-5.27%) 0% <0%> (ø)
...pache/dubbo/registry/support/AbstractRegistry.java 78.16% <0%> (-4.22%) 0% <0%> (ø)
.../exchange/support/header/HeaderExchangeServer.java 66.98% <0%> (-1.89%) 0% <0%> (ø)
...dubbo/rpc/protocol/dubbo/CallbackServiceCodec.java 80.88% <0%> (+0.73%) 0% <0%> (ø) ⬇️
...dubbo/remoting/exchange/support/DefaultFuture.java 75.51% <0%> (+2.04%) 0% <0%> (ø) ⬇️
...apache/dubbo/common/config/ConfigurationUtils.java 72% <0%> (+8%) 0% <0%> (ø) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 78c7509...1947248. Read the comment docs.

@ralf0131
Copy link
Contributor

I suggest to add a check to ensure there is no incompatible license, below is what I used:

cat license-list.txt| grep -E 'GPL|General Public License' | grep -v CDDL | grep -v Apache | grep -v "Eclipse Public License" | grep -v "MIT" | grep -v "Common Development and Distribution License"

If everything is ok, there should be no output.

@htynkn
Copy link
Member Author

htynkn commented Apr 30, 2019

I suggest to add a check to ensure there is no incompatible license, below is what I used:

cat license-list.txt| grep -E 'GPL|General Public License' | grep -v CDDL | grep -v Apache | grep -v "Eclipse Public License" | grep -v "MIT" | grep -v "Common Development and Distribution License"

If everything is ok, there should be no output.

Yeah, we can do this. but I'm wondering are we able to list all license which is not compatible with dubbo? does GPL is the only one we need check?

@ralf0131
Copy link
Contributor

I suggest to add a check to ensure there is no incompatible license, below is what I used:

cat license-list.txt| grep -E 'GPL|General Public License' | grep -v CDDL | grep -v Apache | grep -v "Eclipse Public License" | grep -v "MIT" | grep -v "Common Development and Distribution License"

If everything is ok, there should be no output.

Yeah, we can do this. but I'm wondering are we able to list all license which is not compatible with dubbo? does GPL is the only one we need check?

No. All the incompatible license are listed here: https://www.apache.org/legal/resolved.html#category-x

@htynkn
Copy link
Member Author

htynkn commented May 1, 2019

@ralf0131 I update script and put all stuff into one script, please review again.

Below are dependencies which I'm not sure if need fix:

@ralf0131 ralf0131 added this to the 2.7.2 milestone May 13, 2019
@ralf0131 ralf0131 self-assigned this May 13, 2019
@ralf0131
Copy link
Contributor

Below are dependencies which I'm not sure if need fix:

Apache 2.0 License: OK

  • (GNU LESSER GENERAL PUBLIC LICENSE, Version 2.1) SpotBugs Annotations (com.github.spotbugs:spotbugs-annotations:3.1.10 - https://spotbugs.github.io/)

LGPL: which is imcomplatible with Apache, I am investigating it.

It is actually a 2-Clause BSD License , so I think it is ok.

I found some more reported by the script:

  • (BEA licensed) "Java Concurrency in Practice" book annotations (net.jcip:jcip-annotations:1.0 - http://jcip.net/)

BEA license is said to be incompatible with Apache, need to investigate it.

  • (Common Public License Version 1.0) JUnit (junit:junit:3.8.1 - http://junit.org)

CPL 1.0 is OK, but junit 3.8.1 should not be included, need to exclude it if possible.

Standard BSD license is compatible with Apache.

CPL is compatible with Apache.

My suggestion to improve the script is to add the following to allowLicense:

  • Common Public License Version 1.0
  • The 2-Clause BSD License

Copy link
Contributor

@ralf0131 ralf0131 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let merge it and improve it later.

@ralf0131 ralf0131 merged commit 3fc6909 into apache:master May 22, 2019
@htynkn
Copy link
Member Author

htynkn commented May 22, 2019

Let merge it and improve it later.

Sorry, I'm busy recently. will try to do some improvement when I get time. and also happy to see if anyone else can help on it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants