diff --git a/CERTS.md b/CERTS.md deleted file mode 100644 index d7402e4c..00000000 --- a/CERTS.md +++ /dev/null @@ -1,7 +0,0 @@ -# ICGC DCC - Storage - Certificates - -First run [gencert.sh](../score-server/bin/gencert.sh): - -```shell -cd score-server -bin/gencert.sh \ No newline at end of file diff --git a/CHANGES.md b/CHANGES.md deleted file mode 100644 index 63bd52b1..00000000 --- a/CHANGES.md +++ /dev/null @@ -1,47 +0,0 @@ -# ICGC DCC - Storage - Change Log - -Change log for the ICGC storage system - -1.0.23 --- - - Updated (default) **``filename``** output layout. Files are no longer written as: -``` - . - └── output-dir - ├── filename-1 - │   └── object-id - └── filename-2 - └── object-id -``` - but instead: -``` - . - └── output-dir - ├── filename-1 - └── filename-2 -``` - - if duplicate filenames (but different object id's) are encountered, warning messages will be displayed/logged - -1.0.22 --- - - Added support for Azure - -1.0.14 --- - - Batch Slicing support in View command - - Validate repository values in download manifest files against client profile - - Remove duplicates in result pagination - - Add client check for correct Java version - -1.0.13 --- - - Fix end-of-stream bug in data channel - -1.0.12 --- - - Fix HTTP timeout settings not being applied - -0.0.20 --- - - Fix `--force` overwrite option diff --git a/INSTALL.md b/INSTALL.md deleted file mode 100644 index 2272c632..00000000 --- a/INSTALL.md +++ /dev/null @@ -1,116 +0,0 @@ -# Installation Guide -Installing the ICGC Storage System - - -# Storage System Overview -Backend Components: -- storage-server: allows authenticated users to interact with entities in the storage system -- metadata-server: allows authenticated users to register entities with the storage system -- auth-server: authenticates users by granting access tokens via a REST api - -Client Components: -- metadata-client: registers entities with the storage system -- storage-client: primary client for interaction with storage system - -Each component is a Spring Boot java application packaged in a JAR. Look in src/main/resources/application.yml for default configuration properties, which can be overridden by specifying java system properties when running the jar or by adding an application.properties file via -Dspring.config.location. - - -# Installation -This guide describes setting up the ICGC Storage System on a single Ubuntu EC2 instance. - -Before getting started: -- Ensure you have access to the dcc-auth, Song, and Score source -- Make an S3 bucket to hold the storage system data -- Make a KMS Master Key to encrypt data stored in S3 using the web console (http://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html). -- Make an IAM role with permission to write to s3 (AmazonS3FullAccess). -- Launch an Ubuntu EC2 with the newly created IAM role and a static IP. -- Open ports 8444 and 5431 of the EC2 for anybody who will use the storage system as a client and open port 8443 to anybody who will generate access tokens to be given to end users. - - Also, open ports 8443, 8444, 5431, and 27017 of the EC2 to the ip of the EC2 to ensure the servers can communicate with each other. -- Get a domain and point it towards the new EC2's IP address. The command shown in this guide use the domain storage.ucsc-cgl.org; this should be replaced with the desired domain. -- Install Java 11 on the EC2 instance. - -Add $DCC_HOME environment variable that points to the directory to hold all storage system files. -``` -# set up $DCC_HOME -mkdir ~/dcc -printf "# ICGC Storage System\nexport DCC_HOME=~/dcc\n" >> ~/.bashrc -source ~/.bashrc -# add conf directories -mkdir $DCC_HOME/conf -mkdir $DCC_HOME/conf/ssl -mkdir $DCC_HOME/conf/maven -``` - -Maven version must be between 3.0.3 and 3.2.5 (inclusive). -``` -# install mvnvm (http://mvnvm.org/) -curl -s https://bitbucket.org/mjensen/mvnvm/raw/master/mvn > $DCC_HOME/conf/maven/mvn -chmod 0755 $DCC_HOME/conf/maven/mvn -sudo ln -s $DCC_HOME/conf/maven/mvn /usr/local/bin/mvn -echo "mvn_version=3.2.5" >$DCC_HOME/conf/maven/mvnvm.properties -``` - -Also, install unzip if it's not already installed. -``` -sudo apt-get install -y unzip -``` - -Create an SSL certificate to be used across the storage system. This can be done using the LetsEncrypt certbot client (Note: this will require temporarily opening access to port 443 on the EC2). The root account may need to be used for some of this. -``` -git clone https://github.com/certbot/certbot -cd certbot -./certbot-auto certonly --standalone --email -d storage.ucsc-cgl.org -cd /etc/letsencrypt/archive/storage.ucsc-cgl.org/ # or wherever output from the previous command points you -# convert pem files to pkcs12 -openssl pkcs12 -export -in cert1.pem -inkey privkey1.pem -out ucsc-storage.p12 -name tomcat -CAfile chain1.pem -caname root -chain -# convert pkcs12 to jks -keytool -importkeystore -destkeystore ucsc-storage.jks -deststorepass password -srckeystore ucsc-storage.p12 -srcstoretype PKCS12 -srcstorepass password -chown ubuntu:ubuntu ucsc-storage.p12 -chown ubuntu:ubuntu ucsc-storage.jks -mv ucsc-storage.p12 ucsc-storage.jks $DCC_HOME/conf/ssl -``` - -The LetsEncrypt root CA certificate has to be added to the JVM truststore to tell the JVM to trust our newly generated certificate. To avoid altering the original, a copy is made that can be specified upon invocation of java clients. -``` -# create copy of jvm truststore with LetsEncrypt cert added -cp /usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts $DCC_HOME/conf/ssl/ -keytool -import -file chain1.pem -alias LetsEncryptCA -keystore cacerts -storepass changeit -``` - -Install and configure MongoDB metadata-server dependency (https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/). The commands shown below leave access to mongodb unrestricted. The port that mongod listens on shouldn't be open to external IPs, and in production systems access restriction should be enabled. -``` -sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927 -echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list -sudo apt-get update -sudo apt-get install -y mongodb-org -# printf 'use admin\ndb.createUser({user:"%s",pwd:"%s", roles:[{role:"userAdminAnyDatabase",db:"admin"}]})' metadata pass | mongo -# printf '\n# Enable auth\nauth=true\n' | sudo tee -a /etc/mongod.conf >/dev/null 2>&1 -# sudo service mongod restart -``` - -Install git. -``` -sudo apt-get install -y git -``` - -Pull in and build the storage system source, linking to the ssl certificate while you're at it. -``` -# clone storage system source -cd $DCC_HOME -git clone git@github.com:overture-stack/score.git -git clone git@github.com:BD2KGenomics/dcc-auth.git -git clone git@github.com:BD2KGenomics/dcc-metadata.git -# link mvnvm.properties and ssl certificate then build -for f in $DCC_HOME/dcc-*; do ln -s $DCC_HOME/conf/maven/mvnvm.properties $f/mvnvm.properties && ln -s $DCC_HOME/conf/ssl/ucsc-storage.jks $f/ucsc-storage.jks && cd $f && mvn; done; -``` - -Run the system. -``` -# run the auth-server (TODO: no description of config properties file) -cd $DCC_HOME/dcc-auth/dcc-auth-server/ && java -Dspring.profiles.active=dev,no_scope_validation -Dserver.ssl.key-store=ucsc-storage.jks -Dserver.ssl.key-store-password=password -Dserver.ssl.key-store-type=JKS -Dlogging.file=/var/log/dcc/auth-server/auth-server.log -Dserver.port=8443 -Dmanagement.port=8543 -jar $DCC_HOME/dcc-auth/dcc-auth-server/target/dcc-auth-server-1.0.13-SNAPSHOT.jar -# run the metadata-server -cd $DCC_HOME/dcc-metadata/dcc-metadata-server && java -Djavax.net.ssl.trustStore=$DCC_HOME/conf/ssl/cacerts -Djavax.net.ssl.trustStorePassword=changeit -Dspring.profiles.active=development,secure -Dserver.port=8444 -Dmanagement.port=8544 -Dlogging.file=/var/log/dcc/metadata-server/metadata-server.log -Dauth.server.url=https://storage.ucsc-cgl.org:8443/oauth/check_token -Dauth.server.clientId=metadata -Dauth.server.clientsecret=pass -Dspring.data.mongodb.uri=mongodb://localhost:27017/dcc-metadata -Dserver.ssl.key-store=ucsc-storage.jks -Dserver.ssl.key-store-password=password -Dserver.ssl.key-store-type=JKS -jar target/dcc-metadata-server-0.0.16-SNAPSHOT.jar -# run the storage-server -cd $DCC_HOME/score/score-server && java -Djavax.net.ssl.trustStore=$DCC_HOME/conf/ssl/cacerts -Djavax.net.ssl.trustStorePassword=changeit -Dspring.profiles.active=secure,default -Dlogging.file=/var/log/dcc/storage-server/storage-server.log -Dserver.port=5431 -Dbucket.name.object= -Dbucket.name.state= -Dauth.server.url=https://storage.ucsc-cgl.org:8443/oauth/check_token -Dauth.server.clientId=storage -Dauth.server.clientsecret=pass -Dmetadata.url=https://storage.ucsc-cgl.org:8444 -Dendpoints.jmx.domain=storage -Ds3.endpoint=https://s3.amazonaws.com -Ds3.accessKey=foo -Ds3.secretKey=bar -Ds3.masterEncryptionKeyId=baz -Ds3.secured=true -Dupload.clean.enabled=false -Dserver.ssl.key-store=ucsc-storage.jks -Dserver.ssl.key-store-password=password -Dserver.ssl.key-store-type=JKS -jar target/score-server-1.0.14-SNAPSHOT.jar -``` -Note: passwords (and ideally all configuration) should be specified in configuration files in production systems. diff --git a/score-client/README.md b/score-client/README.md index fedec157..fd9795fe 100644 --- a/score-client/README.md +++ b/score-client/README.md @@ -35,13 +35,13 @@ An example usage of the container which will download a remote file (with associ pull overture/score # Publish token -export ACCESSTOKEN= +export ACCESSTOKEN= # Make life easy for usage alias score-client="docker run -it --rm -u $(id -u):$(id -g) -e ACCESSTOKEN -v /tmp:/data score-client bin/score-client" -# Usage with an example object-id from https://dcc.icgc.org +# Usage with an example object-id score-client download --object-id 5b845b9a-3dcd-59ef-9f56-9a99396e988f --output-dir /data --output-layout bundle ``` diff --git a/score-core/README.md b/score-core/README.md index b75111df..d96cfe4f 100644 --- a/score-core/README.md +++ b/score-core/README.md @@ -1,6 +1,6 @@ -# ICGC DCC - Storage Core +# SCORe - Storage Core -Core library / shared classes for ICGC storage system. +Core library / shared classes for SCORe. ## Build diff --git a/score-fs/README.md b/score-fs/README.md index e1db3aae..bcb88845 100644 --- a/score-fs/README.md +++ b/score-fs/README.md @@ -1,7 +1,7 @@ -ICGC DCC - Storage File System +SCORe - Storage File System === -ICGC storage file system for the ICGC storage system. +SCORe storage file system for the SCORe. ## Build diff --git a/score-server/README.md b/score-server/README.md index eb773e04..873f4e87 100644 --- a/score-server/README.md +++ b/score-server/README.md @@ -1,6 +1,4 @@ -# ICGC DCC - Storage Server - -Storage server for ICGC storage system. +# SCORe - Storage Server ## Libraries diff --git a/score-test/README.md b/score-test/README.md index 8ff1395b..0ba48e90 100644 --- a/score-test/README.md +++ b/score-test/README.md @@ -1,4 +1,4 @@ -# ICGC DCC - Storage Test +# SCORe - Storage Test Module used for integration testing.