Skip to content

Latest commit

 

History

History
271 lines (226 loc) · 18.4 KB

README.md

File metadata and controls

271 lines (226 loc) · 18.4 KB
Raw

SciDB

Tags

Five different tags are available for SciDB. Three are intended for the end user, while two are used for building.

16.9

The tags intended for the end user are:

Tag From Download Size Image Size Comments
scidb:16.9 scidb:16.9-pre 1.866GB SciDB (w/ Shim)
scidb:16.9-deb debian:8 410.6MB SciDB (w/ Shim) from Debian packages (Bintray)
scidb:16.9-ext scidb:16.9 1.899GB SciDB (w/ Shim, dev_tools, and accelerated_io_tools)

The tags used for building are:

Tag From Download Size Image Size Comments
scidb:16.9-pre debian:8 1.452GB Install and build SciDB dependencies and libs, respectively
scidb:16.9-pkg scidb:16.9 2.213GB Build SciDB Debian packages (.deb)

15.12

The tags intended for the end user are:

Tag From Download Size Image Size Comments
scidb:15.12 scidb:15.12-pre 1.892GB SciDB (w/ Shim)
scidb:15.12-deb debian:8 529.3MB SciDB (w/ Shim) from Debian packages (Bintray)
scidb:15.12-ext scidb:15.12 1.926GB SciDB (w/ Shim, dev_tools, and accelerated_io_tools)

The tags used for building are:

Tag From Download Size Image Size Comments
scidb:15.12-pre debian:8 1.546GB Install and build SciDB dependencies and libs, respectively
scidb:15.12-pkg scidb:15.12 2.246GB Build SciDB Debian packages (.deb)

15.7

The tags intended for the end user are:

Tag From Download Size Image Size Comments
scidb:15.7 scidb:15.7-pre 1.835GB SciDB (w/ Shim)
scidb:15.7-deb debian:8 527MB SciDB (w/ Shim) from Debian packages (Bintray)
scidb:15.7-ext scidb:15.7 1.867GB SciDB (w/ Shim, dev_tools, and accelerated_io_tools)

The tags used for building are:

Tag From Download Size Image Size Comments
scidb:15.7-pre debian:8 1.437GB Install and build SciDB dependencies and libs, respectively
scidb:15.7-pkg scidb:15.7 2.138GB Build SciDB Debian packages (.deb)

Usage

When started, the images intended for the end user, use an ENTRYPOINT script to start SSH, PostgreSQL, SciDB, and Shim. As a final command the entrypoint script tails the SciDB log. For example to start a base container and redirect its output to the console, use:

> docker run --tty rvernica/scidb:16.9
[ ok ] Starting OpenBSD Secure Shell server: sshd.
[ ok ] Starting PostgreSQL 9.4 database server: main.
Starting shim
shim: SciDB HTTP service started on port(s) 8080,8083s with web root [/var/lib/shim/wwwroot], talking to SciDB on port 1239
scidb.py: INFO: Found 0 scidb processes
scidb.py: INFO: start((server 0 (127.0.0.1) local instance 0))
scidb.py: INFO: Starting SciDB server.
scidb.py: INFO: start((server 0 (127.0.0.1) local instance 1))
scidb.py: INFO: Starting SciDB server.
load = fn(output_array,input_file,instance_id,format,max_errors,shadow_array,isStrict){store(input(output_array,input_file,instance_id,format,max_errors,shadow_array,isStrict),output_array)};
sys_create_array_aux = fn(_A_,_E_,_C_){join(aggregate(apply(_A_,_t_,_E_),approxdc(_t_)),build(<values_per_chunk:uint64 null>[i=0:0,1,0],_C_))};
sys_create_array_att = fn(_L_,_S_,_D_){redimension(join(build(<n:int64 null,lo:int64 null,hi:int64 null,ci:int64 null,co:int64 null>[No=0:0,1,0],_S_,true),cast(aggregate(_L_,min(_D_),max(_D_),approxdc(_D_)),<min:int64 null,max:int64 null,count:int64 null>[No=0:0,1,0])),<lo:int64 null,hi:int64 null,ci:int64 null,co:int64 null,min:int64 null,max:int64 null,count:int64 null>[n=0:*,?,0])};
sys_create_array_dim = fn(_L_,_S_,_D_){redimension(join(build(<n:int64 null,lo:int64 null,hi:int64 null,ci:int64 null,co:int64 null>[No=0:0,1,0],_S_,true),cast(aggregate(apply(aggregate(_L_,count(*),_D_),_t_,_D_),min(_t_),max(_t_),count(*)),<min:int64 null,max:int64 null,count:int64 null>[No=0:0,1,0])),<lo:int64 null,hi:int64 null,ci:int64 null,co:int64 null,min:int64 null,max:int64 null,count:int64 null>[n=0:*,?,0])}
2016-12-23 01:55:54,209 [0x7f0e097df880] [DEBUG]: Network manager is intialized
2016-12-23 01:55:54,209 [0x7f0e097df880] [DEBUG]: NetworkManager::run()
2016-12-23 01:55:54,209 [0x7f0e097df880] [DEBUG]: server-id = 0
2016-12-23 01:55:54,209 [0x7f0e097df880] [DEBUG]: server-instance-id = 0
2016-12-23 01:55:54,212 [0x7f0e097df880] [DEBUG]: Registered instance # 0
2016-12-23 01:55:54,212 [0x7f0e097df880] [INFO ]: SciDB instance. SciDB Version: 16.9.0. Build Type: Release. Commit: db1a98f. Copyright (C) 2008-2015 SciDB, Inc. is exiting.

Any additional arguments provided when a container is started are executed at the end of this script. So, to get access to the container in interactive mode, append bash at the end of the docker run command. For example, to start a base container in interactive mode, use:

> docker run --tty --interactive rvernica/scidb:16.9 bash
[ ok ] Starting OpenBSD Secure Shell server: sshd.
[ ok ] Starting PostgreSQL 9.4 database server: main.
Starting shim
shim: SciDB HTTP service started on port(s) 8080,8083s with web root [/var/lib/shim/wwwroot], talking to SciDB on port 1239
scidb.py: INFO: Found 0 scidb processes
scidb.py: INFO: start((server 0 (127.0.0.1) local instance 0))
scidb.py: INFO: Starting SciDB server.
scidb.py: INFO: start((server 0 (127.0.0.1) local instance 1))
scidb.py: INFO: Starting SciDB server.
root@a86f0f5867dd:/# iquery --afl --query "list('libraries')"
{inst,n} name,major,minor,patch,build,build_type
{0,0} 'SciDB',16,9,0,229747087,'Release'
{1,0} 'SciDB',16,9,0,229747087,'Release'
root@a86f0f5867dd:/# exit

Here is an example for starting and interacting with a detached extended container:

> docker run --detach rvernica/scidb:16.9-ext
16b699486f2d802d9bd34cf36486d47762b6863a9cd0cb294a9e520dc7f4ef1f
> docker logs 16b699
Starting OpenBSD Secure Shell server: sshd.
Starting PostgreSQL 9.4 database server: main.
Starting shim
scidb.py: INFO: Found 0 scidb processes
scidb.py: INFO: start((server 0 (127.0.0.1) local instance 0))
scidb.py: INFO: Starting SciDB server.
scidb.py: INFO: start((server 0 (127.0.0.1) local instance 1))
scidb.py: INFO: Starting SciDB server.
load = fn(output_array,input_file,instance_id,format,max_errors,shadow_array,isStrict){store(input(output_array,input_file,instance_id,format,max_errors,shadow_array,isStrict),output_array)};
sys_create_array_aux = fn(_A_,_E_,_C_){join(aggregate(apply(_A_,_t_,_E_),approxdc(_t_)),build(<values_per_chunk:uint64 null>[i=0:0,1,0],_C_))};
sys_create_array_att = fn(_L_,_S_,_D_){redimension(join(build(<n:int64 null,lo:int64 null,hi:int64 null,ci:int64 null,co:int64 null>[No=0:0,1,0],_S_,true),cast(aggregate(_L_,min(_D_),max(_D_),approxdc(_D_)),<min:int64 null,max:int64 null,count:int64 null>[No=0:0,1,0])),<lo:int64 null,hi:int64 null,ci:int64 null,co:int64 null,min:int64 null,max:int64 null,count:int64 null>[n=0:*,?,0])};
sys_create_array_dim = fn(_L_,_S_,_D_){redimension(join(build(<n:int64 null,lo:int64 null,hi:int64 null,ci:int64 null,co:int64 null>[No=0:0,1,0],_S_,true),cast(aggregate(apply(aggregate(_L_,count(*),_D_),_t_,_D_),min(_t_),max(_t_),count(*)),<min:int64 null,max:int64 null,count:int64 null>[No=0:0,1,0])),<lo:int64 null,hi:int64 null,ci:int64 null,co:int64 null,min:int64 null,max:int64 null,count:int64 null>[n=0:*,?,0])}
2016-12-23 01:55:54,209 [0x7f0e097df880] [DEBUG]: Network manager is intialized
2016-12-23 01:55:54,209 [0x7f0e097df880] [DEBUG]: NetworkManager::run()
2016-12-23 01:55:54,209 [0x7f0e097df880] [DEBUG]: server-id = 0
2016-12-23 01:55:54,209 [0x7f0e097df880] [DEBUG]: server-instance-id = 0
2016-12-23 01:55:54,212 [0x7f0e097df880] [DEBUG]: Registered instance # 0
2016-12-23 01:55:54,212 [0x7f0e097df880] [INFO ]: SciDB instance. SciDB Version: 16.9.0. Build Type: Release. Commit: db1a98f. Copyright (C) 2008-2015 SciDB, Inc. is exiting.
> docker exec --tty 16b699 iquery --afl --query "load_library('accelerated_io_tools'); list('libraries')"
Query was executed successfully
{inst,n} name,major,minor,patch,build,build_type
{0,0} 'SciDB',16,9,0,229747087,'Release'
{0,1} 'libaccelerated_io_tools.so',16,9,0,229747087,null
{0,2} 'libdev_tools.so',16,9,0,229747087,null
{1,0} 'SciDB',16,9,0,229747087,'Release'
{1,1} 'libaccelerated_io_tools.so',16,9,0,229747087,null
{1,2} 'libdev_tools.so',16,9,0,229747087,null

Here is an example for starting a debian-package-based container in interactive mode:

> docker run --tty --interactive rvernica/scidb:16.9-deb bash
[ ok ] Starting OpenBSD Secure Shell server: sshd.
[ ok ] Starting PostgreSQL 9.4 database server: main.
Starting shim
shim: SciDB HTTP service started on port(s) 8080,8083s with web root [/var/lib/shim/wwwroot], talking to SciDB on port 1239
scidb.py: INFO: Found 0 scidb processes
scidb.py: INFO: start((server 0 (127.0.0.1) local instance 0))
scidb.py: INFO: Starting SciDB server.
scidb.py: INFO: start((server 0 (127.0.0.1) local instance 1))
scidb.py: INFO: Starting SciDB server.
root@2222a680cde1:/# iquery --afl --query "list('libraries')"
{inst,n} name,major,minor,patch,build,build_type
{0,0} 'SciDB',16,9,0,229747087,'Release'
{1,0} 'SciDB',16,9,0,229747087,'Release'
root@2222a680cde1:/# exit

Here is an example for starting a debian-package-based container and Docker volumes for PostgreSQL and SciDB data:

> docker run --tty --interactive --volume postgres:/var/lib/postgresql/9.4/main --volume scidb:/opt/scidb/16.9/DB-scidb rvernica/scidb:16.9-deb iquery --afl --query "store(build(<val:double>[i=0:1; j=0:1], i*2+j), matrix)"
[ ok ] Starting OpenBSD Secure Shell server: sshd.
[ ok ] Starting PostgreSQL 9.4 database server: main.
Starting shim
shim: SciDB HTTP service started on port(s) 8080,8083s with web root [/var/lib/shim/wwwroot], talking to SciDB on port 1239
scidb.py: INFO: Found 0 scidb processes
scidb.py: INFO: start((server 0 (127.0.0.1) local instance 0))
scidb.py: INFO: Starting SciDB server.
scidb.py: INFO: start((server 0 (127.0.0.1) local instance 1))
scidb.py: INFO: Starting SciDB server.
{i,j} val
{0,0} 0
{0,1} 1
{1,0} 2
{1,1} 3
> docker volume ls
DRIVER              VOLUME NAME
local               postgres
local               scidb
> docker run --tty --interactive --volume postgres:/var/lib/postgresql/9.4/main --volume scidb:/opt/scidb/16.9/DB-scidb rvernica/scidb:16.9-deb iquery --afl --query "scan(matrix)"
[ ok ] Starting OpenBSD Secure Shell server: sshd.
[ ok ] Starting PostgreSQL 9.4 database server: main.
Starting shim
shim: SciDB HTTP service started on port(s) 8080,8083s with web root [/var/lib/shim/wwwroot], talking to SciDB on port 1239
scidb.py: INFO: Found 0 scidb processes
scidb.py: INFO: start((server 0 (127.0.0.1) local instance 0))
scidb.py: INFO: Starting SciDB server.
scidb.py: INFO: start((server 0 (127.0.0.1) local instance 1))
scidb.py: INFO: Starting SciDB server.
{i,j} val
{0,0} 0
{0,1} 1
{1,0} 2
{1,1} 3

Ports Exposed

The image exposes the following ports:

Port Application Usage
1239 SciDB iquery
8080 SciDB Shim (HTTP) http://localhost:8080
8083 SciDB Shim (HTTPS) https://localhost:8080

Build Details

The scidb:16.9-pre, scidb:16.9, and scidb:16.9-ext images are build as follows:

  1. Base image is set to the official debian:8 image;
  2. Set environment variables for SciDB build (version, paths, etc.);
  3. Install dependencies from Debian 8 repository;
  4. Install openjdk-8-jdk from backports repository;
  5. Install dependencies from Paradigm 4 (Ubuntu 14.04) repository;
  6. Download SciDB source code from SciDB Forum;
  7. Apply patches required to build SciDB on Debian 8;
  8. Build SciDB libraries;
  9. scidb:16.9-pre is created (necessary due to 2h time limit for builds on Docker Hub);
  10. Build SciDB;
  11. Set environment variables for SciDB instance (number of instances, database name, etc.);
  12. Setup password-less SSH;
  13. Setup PostgreSQL credentials;
  14. Install SciDB;
  15. Install Shim using .deb package provided on Paradigm4 GitHub;
  16. Add and setup ENTRYPOINT script;
  17. Expose SciDB (1239) and Shim (8080 and 8083) ports;
  18. scidb:16.9 is created;
  19. Install dependencies from Debian 8 repository;
  20. Set SHA-1 commit hash for dev_tools and accelerated_io_tools;
  21. Download and build dev_tools;
  22. Start SciDB;
  23. Load dev_tools in SciDB and install accelerated_io_tools from SciDB;
  24. scidb:16.9-ext is created.

The scidb:16.9-pkg image is build as follows:

  1. Base image is set to scidb:16.9;
  2. Install dependencies from Debian 8 repository;
  3. Build SciDB .deb packages;
  4. scidb:16.9-pkg is created.

Once the SciDB .deb packages generated in the scidb:16.9-pkg image are uploaded to the rvernica/deb Bintray repository, the scidb:16.9-deb image is build as follows:

  1. Base image is set to the official debian:8 image;
  2. Set environment variables for SciDB (version, database name, etc.);
  3. Install dependencies from Debian 8 repository;
  4. Add Paradigm 4 (Ubuntu 14.04) and rvernica/deb Bintray repositories;
  5. Install SciDB from .deb packages;
  6. Setup password-less SSH;
  7. Setup config.ini for SciDB;
  8. Setup PostgreSQL credentials;
  9. Initialize SciDB (init-syscat and init-all);
  10. Install Shim using .deb package provided on Paradigm4 GitHub;
  11. Add and setup ENTRYPOINT script;
  12. Expose SciDB (1239) and Shim (8080 and 8083) ports;
  13. scidb:16.9-deb is created.