Skip to content

ToolboxInstallation

Jump to bottom Edit New page
Hugo Oliveira edited this page Jun 2, 2021 · 33 revisions

Toolbox Installation

This page provides an overview of the steps required to get the IMOS Toolbox up and running.

Updating your Installation via Git

If you are using git, need to update to the latest stable release, and has no local changes in the toolbox code that need to be kept, you can do the following within the toolbox root folder:

git fetch origin
git checkout 2.6
git pull

If you have local changes, git may block the checkout since some files would be overwritten. Git will warn you about stashing or committing the changes if this is the case. A simple way to circumvent this is to create a new branch to save your changes:

git checkout -b my_local_changes
git add <file1> <file2> ... 
git commit -m 'my local changes before update to version 2.6.1'
git checkout 2.6

After that, you can resolve the conflicts from your 'my_local_changes' branch the way you find more appropriate. For example, a simple diff can be viewed with:

git diff my_local_changes

You can then change the files manually, as in here, or you may want to merge the branch 2.6 into the my_local_branch, and resolve conflicts (if any):

git checkout my_local_changes
git merge 2.6
#Resolve conflicts
git merge --continue

This way, you will be located at your local branch ('my_local_changes') with merged updates from the latest stable release. Please note that conflict resolving may be problematic if changes are too complex - you need to be familiar with diffs, the way git mark the conflicts, and git conflict resolution.

Standalone installation

Requirements

The stable version requires:

  • Windows or Linux (64bit operating system)
  • The Matlab Component Runtime R2018b for your architecture - Win64 or GLNXA64 (Linux). The installer is called MCR_R2018b_{ARCH}_installer, with version tag 9.5 or v95). click here to download
  • An disk partition win long filename support (filename > 160 characters long) (windows only).

For reading metadata from a database:

  • A database with a proper schema and data entries - also known as a deployment database). We provide a basic template here.
  • Java JRE (1.6 or newer) to be able to query the deployment database with the toolbox.

Optional:

  • Microsoft Access 64 bit (the UCanAccess library is used otherwise - e.g. Linux).

The Matlab Component Runtime

The runtime library provides all the hooks and dependencies to run Matlab Apps without the need to install the Matlab software. The toolbox stand-alone binary is a Matlab App, and its GUI and functions are able to run as a sole app as long as the runtime is available/installed.

A hard-requirement to allow using the toolbox as an stand-alone app is to install the correct runtime version. This is usually associated with the toolbox releases:

  • For installations between v2.3 >= imos-toolbox < v2.6.1 , use the MCRInstaller_8.0.
  • For versions >=2.6.1 : MCRInstaller_R2018b_v9.5

You only need to run the installer once.

For Windows, it comes as a straightforward executable.

For Linux, you will need to execute the install application within the zip file. You may want to install the library as root or as a user. During installation, you might receive a warning about requiring the .NET Framework - you may safely ignore it.

Installing with root credentials in the default installation path may automatically include the MCR applications into your PATH (but this is not guaranteed since it can be distribution dependent).

I recommend installing it as a user within your local home folder. However, you will need to update your PATH environment. If using the bash shell, you may want to include this into your .bashrc file (or equivalent) after the installation finishes:

MCR_ROOT="$HOME/apps/MATLAB/MATLAB_Runtime/v95"
MCR_APPS="$MCR_ROOT/bin:$MCR_ROOT/glnxa64"
export PATH="$PATH:$MCR_APPS"

Stand-alone application Startup

Windows:

Start the toolbox by double-clicking on imosToolbox_Win64.bat. The script will execute imosToolbox_Win64.exe within a prompt shell, so any messages thrown by the toolbox software are visible. This is the recommended way, since executing the .exe file will not provide any information, warning or error message that may be useful.

Linux:

A sh script imosToolbox_Linux64.sh is provided to setup the correct library paths to execute the stand-alone binary file. You will need to provide the root folder of the Matlab Component Runtime as a first argument to the script:

./imosToolbox_Linux64.sh $MCR_ROOT

where $MCR_ROOT is the deployed MCR root path (e.g. /home/ggalibert/apps/MATLAB/MATLAB_Compiler_Runtime/v95)

In newer Linux distributions using intel graphics with the iris, you may face a libGL error when starting the stand-alone app. You may export a variable to override the OpenGL configuration just for Matlab:

MESA_LOADER_DRIVER_OVERRIDE=i965 ./imosToolbox_Linux64.sh $MCR_ROOT

See also the this matlab issue and why you need to do it.

The IMOS Toolbox within Matlab (as a library/GUI)

Another usage option is to use the Toolbox within Matlab, as a library and/or GUI. You will need to call the imosToolbox function within the command-line interface to start the graphical interface.

Requirements

For stable releases after v.2.6.1, it's required to have a valid Matlab R2018b or newer installation. If you plan on using the Spike tests, you will also need to have both the statistics and the signal processing toolboxes.

Installation

There are two options:

  • Obtain the latest stable release as a zip file, as required for the Stand-alone application.

  • Checkout the stable, tagged version repository from gitHub, using git:

cd FOLDER_TO_WHERE_I_WANT_THE_IMOS_TOOLBOX_REPO/
git clone https://github.com/aodn/imos-toolbox.git
cd imos-toolbox
git checkout 2.6

Most of the users should use the first method. However, this requires you to regularly re-download the package at every major (minor) stable (hotfix) release. The latter option, using git, allows you to keep a local copy of the repository and regularly updated it to any major/minor release. This requires some familiarity with Git.

Git can be easily installed on Linux, Mac and Windows. Git for Windows comes with Git BASH which provides a BASH environment like in any Linux/Unix/Mac terminal. Hence, the command above can be used to install the repository in both windows and Linux.

Startup within Matlab

You may want to add the root directory to your path:

TOOLBOX_ROOT='/the/path/to/the/imos_toolbox/'
addpath(TOOLBOX_ROOT)

before execute the imosToolbox function. Otherwise, you may go to the root directory and execute the function there.

Since version 2.5 there is no need to add the toolbox directory and its sub-directories to the Matlab path. This is automatically done every time you run the toolbox for the current Matlab session. Nonetheless, if you are using older versions, doing testing, or using just some particular components, there is a utility function to add all children folders to your path. This function, setToolboxPaths, is located in <toolbox_root>/Util/Path. For example, to add all children folders to your path:

cd <toolbox_root>
addpath('<toolbox_root>Util/Path');
setToolboxPaths(pwd)

This function is likely a requirement if you are performing development or running unit Tests.

The folder structure and organisation

Assuming that the toolbox is installed in C:\imos-toolbox, the repository is organized with the following paths:

  • C:\imos-toolbox\
    • C:\imos-toolbox\AutomaticQC
    • C:\imos-toolbox\DDB
    • C:\imos-toolbox\FlowManager
    • C:\imos-toolbox\GUI
    • C:\imos-toolbox\Graph
      • C:\imos-toolbox\Graph\DepthProfile
      • C:\imos-toolbox\Graph\TimeSeries
      • C:\imos-toolbox\Graph\Transect
    • C:\imos-toolbox\IMOS
    • C:\imos-toolbox\NetCDF
    • C:\imos-toolbox\Parser
    • C:\imos-toolbox\Preprocessing
      • C:\imos-toolbox\Preprocessing\Transform
    • C:\imos-toolbox\Seawater
    • C:\imos-toolbox\Util

Within the above folders, some toolboxes may be defined. Currently, we got the following toolboxes:

C:\Preprocessing\+TeledyneADCP C:\Parser\+Workhorse C:\Util\+IMOS

In particular, the +IMOS toolbox contains several sub-toolboxes that include new functionality to deal with toolbox datasets.

Regular updates with git

Every quarter and/or minor release, we test the toolbox for stability and merge changes into the current major version branch. If you wish to update to the latest stable release, you have to position yourself in the respective branch and pull the new changes:

git fetch origin
git checkout 2.6
git pull

The above will ensure you are at the "2.6" branch before pulling the changes locally into your directory. Now you have a clone of the 2.6 branch. Any critical fixes will require futures "pulls" or changes to a new branch (for example when a new major version branch happens).

Finally, we only encourage users to use the master branch if they are comfortable with git or plan to contribute/develop the toolbox further.

If you have created a GitHub account, you can watch the toolbox GitHub repo and be notified about anything that is happening related to the toolbox. You can watch for new releases by selecting "Releases Only" in the Watch button in the top-right corner of the main repository page. Websites like coderelease can also provide you with this service.

Conflicts

The toolbox has some configuration files that are open to change by any user. In this scenario, merge conflicts can occur: By pulling new changes, your config files will drift from the default options presented in the repo.

If a conflict appears, you will need to resolve it - see example of conflict.

Configure/Set the Deployment Database

If you seek to pull metadata information from a deployment database, you need to make sure the database is correctly configured. configure your database.

Since imos-toolbox version 2.0, in addition to the MS-Access deployment database (template provided in the downloads page), it is also possible to connect to any user-specific database, as long as its schema complies to minimum requirements.

Since version 2.5.8, a deployment database in CSV file format is supported. A template with example entries can be found in the downloads page).

MS Access database

The toolbox can read the MS Access deployment database using:

  • Microsoft ODBC driver,
  • UCanAccess JDBC driver.

Using the Microsoft Access ODBC drivers

This solution is only recommended when the architecture version of Microsoft Access matches the one of Matlab or standalone binary architecture. For example, if your Microsoft Access software is 32bit, but your Matlab is 64bit, then this won't work.

Operating system Microsof Access Matlab or standalone binary Compatibility
32bit 32bit 32bit Ok
64bit 32bit 32bit Ok
64bit 32bit 64bit KO
64bit 64bit 32bit KO
64bit 64bit 64bit Ok

The Microsoft Access software comes with a Microsoft Access ODBC driver which can be used to turn your MS Access file into an ODBC source following the described steps below:

  • Start
    • Control Panel
      • Administrative Tools
        • Data Sources (ODBC)
          • User DSN
            • Add
              • Driver: "Microsoft Access Driver (*.mdb)"
              • Data Source Name: "imos-ddb" (or whatever you wish to call it)
              • Select -> navigate to the ddb MS Access file

Once you have configured the ddb as an ODBC source, set the toolbox.ddb property (in toolboxProperties.txt) to the ODBC Data Source Name.

Some limitations have been observed where it is not possible to open the current 32bit version of the deployment database with a Microsoft Access software 64bit. A workaround may be to convert the .mdb to .accdb. Be aware that once you convert to .accdb, you cannot go back to .mdb without a fair amount of manual copying to rebuild the .mdb.

With UCanAccess JDBC drivers

This is the only solution for Linux operating systems.

The toolbox will automatically use the JDBC drivers if you simply set the toolbox.ddb property (in toolboxProperties.txt) to your MS Access file full path.

The UCANAccess driver is a java application library. One drawback of using this library is that you may need to increase the amount of Java Heap memory needed by the Java Virtual Machine. This is so in cases with large databases. The usual error message is something akin to a java.lang.OutOfMemoryError: Java heap space error message.

The easiest fix is to tell Java to allocate sufficient heap space memory when the Java Virtual Machine starts up.

If you are using the stand-alone binary version, you need to add a java.opts file in the root directory of the toolbox, with the heap memory directive option (-Xmx). For example, a file with -Xmx2g will ask Java to allocate 2Gb of Heap memory. Now you will be able to start the standalone binary executable from the root directory of the toolbox. This is the preferred method and the only available one for the standalone version.

If you are using the toolbox from matlab, you need to open the Preferences menu -> General -> Java Heap Memory option, and set it to a given value. For example, a 2Gb is a minimum requirement for the AIMS database). See also [here] for matlab support(https://www.mathworks.com/help/matlab/matlab_external/java-heap-memory-preferences.html)

CSV files as Deployment databases

A deployment database template (existing entries are provided as examples) is available here and follows the structure of the MS-Access deployment database schema.

Similarly to the case above, You need to inform the toolbox where the Deployment database is by setting the toolbox.ddb entry in the toolboxProperties.txt configuration file:

toolbox.ddb = path_to_the_directory_where_you_keep_the_csv_files

Other databases

Your deployment database architecture has to follow a minimum interface requirements to prevent the toolbox from crashing. Then, to be able to query it through the toolbox, you must put the driver (.jar file) used to connect to it in the Java directory and set the relevant toolbox.ddb.* properties (in toolboxProperties.txt). Since version 2.5, postgresql-9.4-1201.jdbc4.jar is included by default since CSIRO uses Posgresql for their deployment database.

Contributing

Please feel free to send any enhancements or proposals on the github issues page.

Pull Requests are also welcome but please be aware that we need to maintain consistency, check compatibility, and test the features before we merge them into the main repository. This may take some time, depending on the proposed changes, code consistency and testing.

If you would like to do further development for Push requests, we recommend you create a fork first and work from there, preferably from the master branch or the stable release (so the number of conflicts/rebases are reduced).

Using the Toolbox

See the overview on using the toolbox to get started.

Add a custom footer
Add a custom sidebar
Clone this wiki locally