go-cve-dictionary

This is tool to build a local copy of the NVD (National Vulnerabilities Database) [1] and the Japanese JVN [2], which contain security vulnerabilities according to their CVE identifiers [3] including exhaustive information and a risk score. The local copy is generated in sqlite format, and the tool has a server mode for easy querying.

[1] https://en.wikipedia.org/wiki/National_Vulnerability_Database
[2] https://en.wikipedia.org/wiki/Common_Vulnerabilities_and_Exposures
[3] http://jvndb.jvn.jp/apis/termsofuse.html

Installation

Install requirements

go-cve-dictionary requires the following packages.

SQLite3, MySQL, PostgreSQL or Redis
git
gcc
go v1.7.1 or later
- https://golang.org/doc/install

Here's an example for Amazon EC2 server.

$ ssh [email protected]  -i ~/.ssh/private.pem
$ sudo yum -y install sqlite git gcc
$ wget https://storage.googleapis.com/golang/go1.7.1.linux-amd64.tar.gz
$ sudo tar -C /usr/local -xzf go1.7.1.linux-amd64.tar.gz
$ mkdir $HOME/go

Put these lines into /etc/profile.d/goenv.sh

export GOPATH=$HOME/go
export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin

Set the OS environment variable to current shell

$ source /etc/profile.d/goenv.sh

Deploy go-cve-dictionary

To install:

$ mkdir -p $GOPATH/src/github.com/vulsio
$ cd $GOPATH/src/github.com/vulsio
$ git clone https://github.com/vulsio/go-cve-dictionary.git
$ cd go-cve-dictionary
$ make install

Create a log output directory. You can use another directory on the command line option (--log-dir).

$ sudo mkdir /var/log/go-cve-dictionary
$ sudo chown ec2-user /var/log/go-cve-dictionary
$ sudo chmod 700 /var/log/go-cve-dictionary

Fetch vulnerability data from NVD.

$ go-cve-dictionary fetch nvd
... snip ...
$ ls -alh cve.sqlite3
-rw-r--r-- 1 ec2-user ec2-user 7.0M Mar 24 13:20 cve.sqlite3

Now we have vulnerability data. Start go-cve-dictionary as server mode.

$ go-cve-dictionary server
[Mar 24 15:21:55]  INFO Opening DB. datafile: /home/ec2-user/cve.sqlite3
[Mar 24 15:21:55]  INFO Migrating DB
[Mar 24 15:21:56]  INFO Starting HTTP Sever...
[Mar 24 15:21:56]  INFO Listening on 127.0.0.1:1323

Update go-cve-dictionary

If the DB schema was changed, please specify new SQLite3, MySQL, PostgreSQL or Redis DB file.

$ cd $GOPATH/src/github.com/vulsio/go-cve-dictionary
$ git pull
$ rm -r vendor
$ make install

Binary files are created under $GOPATH/bin

Sample data sources

Hello HeartBleed

$ curl http://127.0.0.1:1323/cves/CVE-2014-0160 | jq "."
{
  "CveID": "CVE-2014-0160",
  "Nvd": {
    "Summary": "The (1) TLS and (2) DTLS implementations in OpenSSL 1.0.1 before 1.0.1g do not properly handle Heartbeat Extension packets, which allows remote attackers to obtain sensitive information from process memory via crafted packets that trigger a buffer over-read, as demonstrated by reading private keys, related to d1_both.c and t1_lib.c, aka the Heartbleed bug.",
    "Score": 5,
    "AccessVector": "NETWORK",
    "AccessComplexity": "LOW",
    "Authentication": "NONE",
    "ConfidentialityImpact": "PARTIAL",
    "IntegrityImpact": "NONE",
    "AvailabilityImpact": "NONE",
    "Cpes": null,
    "References": [
      {
        "Source": "CERT",
        "Link": "http://www.us-cert.gov/ncas/alerts/TA14-098A"
      },
      ...snip...
    ],
    "PublishedDate": "2014-04-07T18:55:03.893-04:00",
    "LastModifiedDate": "2015-10-22T10:19:38.453-04:00"
  },
  "Jvn": {
    "Title": "OpenSSL の heartbeat 拡張に情報漏えいの脆弱性",
    "Summary": "OpenSSL の heartbeat 拡張の実装には、情報漏えいの脆弱性が存在します。TLS や DTLS 通信において OpenSSL のコードを実行しているプロセスのメモリ内容が通信相手に漏えいする可能性があります。",
    "JvnLink": "http://jvndb.jvn.jp/ja/contents/2014/JVNDB-2014-001920.html",
    "JvnID": "JVNDB-2014-001920",
    "Score": 5,
    "Severity": "Medium",
    "Vector": "(AV:N/AC:L/Au:N/C:P/I:N/A:N)",
    "References": [
      {
        "Source": "AT-POLICE",
        "Link": "http://www.npa.go.jp/cyberpolice/detect/pdf/20140410.pdf"
      },
      ...snip...
    ],
    "Cpes": null,
    "PublishedDate": "2014-04-08T16:13:59+09:00",
    "LastModifiedDate": "2014-04-08T16:13:59+09:00"
  }
}

Hello Ruby on Rails 4.0.2

$ curl -v -H "Accept: application/json" -H "Content-type: application/json" -X POST -d '{"name": "cpe:/a:rubyonrails:ruby_on_rails:4.0.2:-"}' http://localhost:1323/cpes | jq "."
[
  {
    "CveID": "CVE-2016-0751",
    "Nvd": {
      "CveDetailID": 345,
      "Summary": "actionpack/lib/action_dispatch/http/mime_type.rb in Action Pack in Ruby on Rails before 3.2.22.1, 4.0.x and 4.1.x before 4.1.14.1, 4.2.x before 4.2.5.1, and 5.x before 5.0.0.beta1.1 does not properly restrict use of the MIME type cache, which allows remote attackers to cause a denial of service (memory consumption) via a crafted HTTP Accept header.",
      "Score": 5,
      "AccessVector": "NETWORK",
      "AccessComplexity": "LOW",
      "Authentication": "NONE",
      "ConfidentialityImpact": "NONE",
      "IntegrityImpact": "NONE",
      "AvailabilityImpact": "PARTIAL",
      "Cpes": null,
      "References": [
        {
          "Source": "MLIST",
          "Link": "https://groups.google.com/forum/message/raw?msg=ruby-security-ann/9oLY_FCzvoc/5CDXbvpYEgAJ"
        },
        {
          "Source": "MLIST",
          "Link": "http://www.openwall.com/lists/oss-security/2016/01/25/9"
        }
      ],
      "PublishedDate": "2016-02-15T21:59:05.877-05:00",
      "LastModifiedDate": "2016-03-18T21:02:43.817-04:00"
    },
    "Jvn": {
      "Title": "",
      "Summary": "",
      "JvnLink": "",
      "JvnID": "",
      "Score": 0,
      "Severity": "",
      "Vector": "",
      "References": null,
      "Cpes": null,
      "PublishedDate": "0001-01-01T00:00:00Z",
      "LastModifiedDate": "0001-01-01T00:00:00Z"
    }
  },
  ... snip ...
]

Usage

$ go-cve-dictionary --help
GO CVE Dictionary

Usage:
  go-cve-dictionary [command]

Available Commands:
  completion  generate the autocompletion script for the specified shell
  fetch       Fetch Vulnerability dictionary
  help        Help about any command
  server      Start CVE dictionary HTTP Server
  version     Show version

Flags:
      --config string       config file (default is $HOME/.go-cve-dictionary.yaml)
      --dbpath string       /path/to/sqlite3 or SQL connection string (default "$PWD/cve.sqlite3")
      --dbtype string       Database type to store data in (sqlite3, mysql, postgres or redis supported) (default "sqlite3")
      --debug               debug mode (default: false)
      --debug-sql           SQL debug mode
  -h, --help                help for go-cve-dictionary
      --http-proxy string   http://proxy-url:port (default: empty)
      --log-dir string      /path/to/log (default "/var/log/go-cve-dictionary")
      --log-json            output log as JSON
      --log-to-file         output log to file

Use "go-cve-dictionary [command] --help" for more information about a command.

Usage: Fetch Command

$ go-cve-dictionary fetch --help
Fetch Vulnerability dictionary

Usage:
  go-cve-dictionary fetch [command]

Available Commands:
  jvn         Fetch Vulnerability dictionary from JVN
  nvd         Fetch Vulnerability dictionary from NVD

Flags:
      --batch-size int   The number of batch size to insert. NOTE: This Option does not work for dbtype: redis. (default 5)
  -h, --help             help for fetch

Global Flags:
      --config string       config file (default is $HOME/.go-cve-dictionary.yaml)
      --dbpath string       /path/to/sqlite3 or SQL connection string (default "$PWD/cve.sqlite3")
      --dbtype string       Database type to store data in (sqlite3, mysql, postgres or redis supported) (default "sqlite3")
      --debug               debug mode (default: false)
      --debug-sql           SQL debug mode
      --http-proxy string   http://proxy-url:port (default: empty)
      --log-dir string      /path/to/log (default "/var/log/go-cve-dictionary")
      --log-json            output log as JSON
      --log-to-file         output log to file

Use "go-cve-dictionary fetch [command] --help" for more information about a command.

Fetch NVD data

to fetch all years

$ go-cve-dictionary fetch nvd

to fetch specific years

$ go-cve-dictionary fetch nvd 2021

Fetch JVN data

to fetch all years

$ go-cve-dictionary fetch jvn

to fetch specific years

$ go-cve-dictionary fetch jvn 2021

Usage: Run HTTP Server

$ go-cve-dictionary server --help
Start CVE dictionary HTTP Server

Usage:
  go-cve-dictionary server [flags]

Flags:
      --bind string   HTTP server bind to IP address (default "127.0.0.1")
  -h, --help          help for server
      --port string   HTTP server port number (default "1323")

Global Flags:
      --config string       config file (default is $HOME/.go-cve-dictionary.yaml)
      --dbpath string       /path/to/sqlite3 or SQL connection string (default "$PWD/cve.sqlite3")
      --dbtype string       Database type to store data in (sqlite3, mysql, postgres or redis supported) (default "sqlite3")
      --debug               debug mode (default: false)
      --debug-sql           SQL debug mode
      --http-proxy string   http://proxy-url:port (default: empty)
      --log-dir string      /path/to/log (default "/var/log/go-cve-dictionary")
      --log-json            output log as JSON
      --log-to-file         output log to file

Usage: Use MySQL as a DB storage back-end

fetch nvd

$ go-cve-dictionary fetch nvd \
      --dbtype mysql \
      --dbpath "user:pass@tcp(localhost:3306)/dbname?parseTime=true"

fetch jvn

$ go-cve-dictionary fetch jvn \
      --dbtype mysql \
      --dbpath "user:pass@tcp(localhost:3306)/dbname?parseTime=true"

server

$ go-cve-dictionary server \
      --dbtype mysql \
      --dbpath "user:pass@tcp(localhost:3306)/dbname?parseTime=true"

Usage: Use Postgres as a DB storage back-end

fetch nvd

$ go-cve-dictionary fetch nvd \
      --dbtype postgres \
      --dbpath "host=myhost user=user dbname=dbname sslmode=disable password=password"

fetch jvn

$ go-cve-dictionary fetch jvn \
      --dbtype postgres \
      --dbpath "host=myhost user=user dbname=dbname sslmode=disable password=password"

server

$ go-cve-dictionary server \
      --dbtype postgres \
      --dbpath "host=myhost user=user dbname=dbname sslmode=disable password=password"

Usage: Use Redis as a DB storage back-end

fetch nvd

$ go-cve-dictionary fetch nvd \
      --dbtype redis \
      --dbpath "redis://localhost/0"

fetch jvn

$ go-cve-dictionary fetch jvn \
      --dbtype redis \
      --dbpath "redis://localhost/0"

server

$ go-cve-dictionary server \
      --dbtype redis \
      --dbpath "redis://localhost/0"

Misc

HTTP Proxy Support
If your system at behind HTTP proxy, you have to specify -http-proxy option.
How to daemonize go-cve-dictionary
Use Systemd, Upstart or supervisord, daemontools...

How to cross compile

$ cd /path/to/your/local-git-repository/go-cve-dictionary
$ GOOS=linux GOARCH=amd64 go build -o cvedict.amd64

Logging
go-cve-dictionary writes a log under -log-path specified directory (default is /var/log/go-cve-dictionary/).
Debug
Run with --debug, --debug-sql option.

Completion Support

bash

$ go-cve-dictionary completion bash > /usr/share/bash-completion/completions/go-cve-dictionary

zsh

$ go-cve-dictionary completion zsh > ~/.zsh-completions/go-cve-dictionary

fish

$ go-cve-dictionary completion fish > ~/.config/fish/completions/go-cve-dictionary.fish

Data Source

Authors

kotakanbe (@kotakanbe) created go-cve-dictionary and these fine people have contributed.

How to Contribute

fork a repository: github.com/vulsio/go-cve-dictionary to github.com/you/repository
get original code: github.com/vulsio/go-cve-dictionary
work on original code
add remote to your repository: git remote add myfork https://github.com/you/repo.git
push your changes: git push myfork
create a new Pull Request

see GitHub and Go: forking, pull requests, and go-getting

Licence

Please see LICENSE.

Additional License

NVD

How can my organization use the NVD data within our own products and services? All NVD data is freely available from our XML Data Feeds. There are no fees, licensing restrictions, or even a requirement to register. All NIST publications are available in the public domain according to Title 17 of the United States Code. Acknowledgment of the NVD when using our information is appreciated. In addition, please email [email protected] to let us know how the information is being used.

JVN

Name		Name	Last commit message	Last commit date
Latest commit History 229 Commits
.github		.github
commands		commands
config		config
db		db
fetcher		fetcher
integration		integration
log		log
models		models
server		server
util		util
.dockerignore		.dockerignore
.gitignore		.gitignore
.golangci.yml		.golangci.yml
.goreleaser.yml		.goreleaser.yml
.revive.toml		.revive.toml
Dockerfile		Dockerfile
GNUmakefile		GNUmakefile
LICENSE		LICENSE
README.md		README.md
go.mod		go.mod
go.sum		go.sum
main.go		main.go

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

go-cve-dictionary

Installation

Install requirements

Deploy go-cve-dictionary

Update go-cve-dictionary

Sample data sources

Hello HeartBleed

Hello Ruby on Rails 4.0.2

Usage

Usage: Fetch Command

Fetch NVD data

Fetch JVN data

Usage: Run HTTP Server

Usage: Use MySQL as a DB storage back-end

Usage: Use Postgres as a DB storage back-end

Usage: Use Redis as a DB storage back-end

Misc

Data Source

Authors

How to Contribute

Licence

Additional License

About

Releases

Packages

Languages

License

aplunk/go-cve-dictionary

Folders and files

Latest commit

History

Repository files navigation

go-cve-dictionary

Installation

Install requirements

Deploy go-cve-dictionary

Update go-cve-dictionary

Sample data sources

Hello HeartBleed

Hello Ruby on Rails 4.0.2

Usage

Usage: Fetch Command

Fetch NVD data

Fetch JVN data

Usage: Run HTTP Server

Usage: Use MySQL as a DB storage back-end

Usage: Use Postgres as a DB storage back-end

Usage: Use Redis as a DB storage back-end

Misc

Data Source

Authors

How to Contribute

Licence

Additional License

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages