Merge pull request #42 from brianmajor/main

integrating read-the-docs content

docs/general/ALMA_Desktop/archive_download.md

@@ -0,0 +1,63 @@
+# Download data from the ALMA archive (web browser) {#desktop_download}
+
+Downloading data directly from an ALMA archive web query is quite
+straightforward. Double-click on the Firefox icon on the desktop, and
+navigate to the ALMA archive.
+
+> ![image](images/archive/1_firefox_browse_archive.png)
+
+You can use the filters to help identify the data you are interested in.
+For more information on using the archive query effectively, see this
+instructional video:
+<https://almascience.nrao.edu/alma-data/archive/archive-video-tutorials>
+
+Select the dataset(s) of interest by ticking the box(es) to the left of
+each project.
+
+> ![image](images/archive/2_select_data.png)
+
+On the upper right panel, click the downward-pointing arrow, then click
+the green \'Explore and download\' button that pops up.
+
+> ![image](images/archive/3_download_data1.png)
+
+In the next window that pops up, select the files that you are
+interested in and click \'Download Selected\'.
+
+> ![image](images/archive/4_download_data2.png)
+
+In the pop up window that next appears, click the \'Download Script\'
+button.
+
+> ![image](images/archive/5_download_data3.png)
+
+Save this file to your desktop session.
+
+> ![image](images/archive/6_download_data4.png)
+
+To locate the file, click the downward-pointing arrow in the top left of
+the firefox browser, and then double-click the folder icon in the right
+of the window that pops up.
+
+> ![image](images/archive/7_download_data5.png)
+
+This will open the File Manager in the directory where your ALMA data
+download script has been saved. The default is usually under Downloads/
+within your home directory.
+
+> ![image](images/archive/8_download_data6.png)
+
+::: {#desktop_run_download_script}
+Open a terminal (double-click the terminal icon), go to the Downloads
+directory, and run the data download script.
+:::
+
+> ![image](images/archive/9_download_data7.png)
+
+When all of the files have been downloaded, you should see a query about
+un-tarring (uncompressing) them. Click \'y\' for yes.
+
+> ![image](images/archive/10_download_data8.png)
+
+You can now open a CASA terminal to begin your data reduction or
+analysis.

docs/general/ALMA_Desktop/archive_script_download.md

@@ -0,0 +1,17 @@
+# Download data from the ALMA archive (transfer script) {#desktop_script_download}
+
+Downloading data from the ALMA archive directly using a web query is
+shown in `this tutorial<desktop_download>`{.interpreted-text
+role="ref"}. Some users, however, may prefer to interact with the ALMA
+archive on their local machine and instead transfer the resulting data
+download script to their Desktop session. For more information on using
+the archive query effectively, see this instructional video:
+<https://almascience.nrao.edu/alma-data/archive/archive-video-tutorials>
+
+Once you have a data download script on your local computer, there are a
+variety of methods to `transfer files <filetransfer>`{.interpreted-text
+role="ref"} into your Science Portal diskspace. Once the script is
+uploaded and thus accessible to your Desktop session, you can run the
+script within a terminal container and download the data, as shown in
+`this tutorial<desktop_run_download_script>`{.interpreted-text
+role="ref"}.

docs/general/ALMA_Desktop/start_casa.md

@@ -0,0 +1,38 @@
+# Starting CASA {#desktop_start_casa}
+
+Once you have
+`launched a Desktop session<launch_desktop>`{.interpreted-text
+role="ref"}, it is straightforward to run CASA in a terminal.
+
+> ![image](images/start_casa/1_new_desktop.png)
+
+To start a CASA-enabled terminal, first click on the \'Applications\'
+menu at the top left corner of your screen
+
+> ![image](images/start_casa/2_applications_menu.png)
+
+Click through the \'AstroSoftware\' and main CASA version to select the
+specific version that you want to use. Every version of CASA back to
+CASA 3.4.0 is available (this is the version of CASA which is tied to
+the scripts distributed with ALMA Cycle 0 data on the archive; see link
+[here](https://casaguides.nrao.edu/index.php?title=Updating_a_script_to_work_with_CASA_4.2))
+
+> ![image](images/start_casa/3_choose_casa.png)
+
+Clicking on your prefered version of CASA will open a terminal in which
+you can run CASA in the standard manner (typing either \'casa\' or
+\'casa \--pipeline\' depending on the mode you wish to use). \[NB: there
+are two dashes before \'pipeline\' in the previous command\]
+
+> ![image](images/start_casa/4_casa_launched.png)
+>
+> ![image](images/start_casa/5_run_casa.png)
+
+You can open a regular (non-CASA) terminal by double-clicking the
+\'terminal\' icon at the lower left side of the screen. Note that each
+terminal has a label on top which denotes what type of terminal it is
+(i.e., a plain terminal or the CASA version that is enabled). Only basic
+linux commands are recognized in the CASA terminals, so it is preferable
+to use a non-CASA terminal for all regular linux uses.
+
+> ![image](images/start_casa/6_casa_and_terminal.png)

docs/general/ALMA_Desktop/typical_reduction.md

@@ -0,0 +1,75 @@
+# Example: reduce and image data {#typical_reduction}
+
+This tutorial walks through an example of reducing and imaging and ALMA
+dataset, and copying the final images to a personal computer. First, you
+need to download your ALMA data onto your Desktop Session. See
+`here<desktop_download>`{.interpreted-text role="ref"} or
+`here<desktop_script_download>`{.interpreted-text role="ref"} for
+tutorials on how to download data directly from the ALMA Science
+Archive, or `here<filetransfer>`{.interpreted-text role="ref"} for
+instructions on how to transfer files from your personal computer or
+VOSpace to the Desktop Session, if you already have your ALMA data
+downloaded there.
+
+Next, open a CASA container (see
+`this tutorial<desktop_start_casa>`{.interpreted-text role="ref"}) with
+the version that you need to run. Then, start CASA in either interactive
+or pipeline mode, depending on what is required for your reduction
+script, by typing `casa` or `casa --pipeline`
+
+> *Note: The version of CASA originally used to reduce your dataset can
+> usually be found listed in the log files and/or near the top of the
+> reduction scripts. At present, it is generally recommended that you
+> use the same version of CASA to reduce the data as was originally
+> used. The presence of scripts with \'pipe\' in their names in the
+> script directory indicate that CASA must be initiated in pipeline
+> mode.*
+>
+> ![image](images/typical_reduc/1_start_casa.png)
+
+Once you have started CASA, run the data reduction script (usually
+called scriptForPI.py) by typing `execfile('scriptForPI.py')`
+
+> ![image](images/typical_reduc/2_run_scriptForPI.png)
+
+After the reduction script runs, you will find the calibrated
+measurement set(s) in the calibrated/ directory. The scriptForImaging.py
+script that is usually distributed with the dataset will help you to
+image your dataset. Usually, this script needs to be copied into the
+calibrated/ directory before running it. In some instances, you may need
+to start up a different version of casa to run the script. In the
+example shown here, the calibration script uses the pipeline version of
+CASA (\'casa-4.3.1-pipe\'), while the imaging script uses the
+interactive version of CASA (\'casa-4.3.1\'), and these two CASA
+versions were distributed independently with the names as noted in
+parenthesis. Thus, to image the data, you would need to open a new
+\'casa-4.3.1\' container and type `casa` in the prompt.
+
+During imaging, you might find it helpful to open up a regular terminal
+to view and/or edit the imaging script, for example, to copy and paste
+the commands one line at a time into casa (see the
+`Using the Clipboard<clipboard>`{.interpreted-text role="ref"} tutorial
+for more information).
+
+> ![image](images/typical_reduc/3_ready_for_imaging.png)
+
+Interacting with the CASA viewer while running the tast *tclean*, for
+example, works in the same manner as on a personal desktop machine.
+
+> ![image](images/typical_reduc/4_interactive_clean.png)
+
+Once imaging and any desired analysis is complete, you can transfer your
+final files off of the system using one of the many options available to
+`transfer files<filetransfer>`{.interpreted-text role="ref"}.
+
+> ![image](images/typical_reduc/5_imaging_done_copy_out.png)
+
+In the example above, a small subset of files are being transfered to a
+personal VOSpace page using the `vcp<vostools>`{.interpreted-text
+role="ref"} command:
+
+    vcp calibrated_final_cont_image_162622-24225.* vos:helenkirk/
+
+Note that the Science Portal is not intended to be used for long-term
+data storage needs - VOSpace is instead recommended as it has a robust
+backup system in place.

docs/general/General_tools/File_transfers.md

@@ -0,0 +1,22 @@
+# File Transfer Overview {#filetransfer}
+
+There are a variety of methods available to move files into and out of
+the Science Portal. This page provides a short summary of the options,
+with links to further resources that are available.
+
+-   upload individual files \[Notebook\]: Small files can be easily
+    uploaded within a Notebook session using the upload icon. A tutorial
+    is available `here<notebook_transfer_file>`{.interpreted-text
+    role="ref"}
+-   upload/download files or directories using the [web browser
+    interface](https://www.canfar.net/storage/arc/list/home) for the
+    storage associated with the Science Portal. A tutorial is available
+    `here<webstorage>`{.interpreted-text role="ref"}.
+-   upload/download files or directories from the command line using the
+    VOS Tools software. See a basic tutorial
+    `here<vostools>`{.interpreted-text role="ref"}.
+-   mount the file system to your local computer using sshfs. See a
+    basic tutorial `here<sshfs>`{.interpreted-text role="ref"}.
+-   command line interaction to upload/download files through a direct
+    url. See a brief description `here<directurl>`{.interpreted-text
+    role="ref"}.

docs/general/General_tools/Group_management.md

@@ -0,0 +1,80 @@
+# Group Management Tools {#groupmanagement}
+
+The group management tools allow you to define a set of people who have
+permission to read, write, or execute your files.
+
+Start by going to `canfar.net` and clicking on the Group Management
+icon.
+
+> ![image](images/groupmanagement/1_canfar_landing.png)
+
+You can then create a new \'group\' (list of CADC usernames) which you
+will be granting access to your files. Click on the \'New Group\' button
+to get started.
+
+> ![image](images/groupmanagement/2_group_management_landing.png)
+
+A pop up window will appear that allows you to provide a group name and
+brief description.
+
+> ![image](images/groupmanagement/3_create_group.png)
+
+This group will then appear on the master list on the main page. You can
+add your collaborators by clicking the \'Edit\' button in the Membership
+column.
+
+> ![image](images/groupmanagement/4_group_landing_add.png)
+
+Start typing the name of your collaborator into the \'Enter a name\'
+box, and a window will pop up with matches that you can use to find the
+person. (NB: the search is based on the actual name and not CADC
+username). Once you have selected the person, click the \'Add member\'
+button. They will be added to the group even though the listing on the
+pop-up window is not auto-refreshed. You can close the pop-up window and
+click the \'Edit\' button a second time to confirm that they are now a
+member of your group.
+
+> ![image](images/groupmanagement/5_add_members.png)
+>
+> ![image](images/groupmanagement/6_updated_members.png)
+
+If you wish to grant others in the group permission to add further
+members, etc, you can do so by clicking the \'Edit\' button in the
+\'Administrators\' column and following the same procedure.
+
+> ![image](images/groupmanagement/7_add_admin.png)
+
+With your group created, you navigate to CANFAR\'s webpage for all
+projects, <https://www.canfar.net/storage/arc/list/projects> and edit
+file access there. In this example, the project directory is
+`ALMAcores`, and the group `HKirk_plus_grads` is already granted access
+to read and write files.
+
+> ![image](images/groupmanagement/8_browse_projects.png)
+
+Click the pencil button to edit the group permissions. In this example,
+the current group with write permissions appears in the associated box.
+
+> ![image](images/groupmanagement/9_edit_permissions1.png)
+
+Start typing in your desired group name, then select the group from the
+pop-up list.
+
+> ![image](images/groupmanagement/10_edit_permissions2.png)
+
+Click the \'Save\' button
+
+> ![image](images/groupmanagement/11_edit_permissions3.png)
+
+then click \'Ok\' on the pop-up window.
+
+> ![image](images/groupmanagement/12_edit_permissions4.png)
+
+Your updated group permissions are now shown on the main page.
+
+> ![image](images/groupmanagement/13_permissions_updated.png)
+
+Management of groups can also be done via the command line, and is the
+prefered option for more complex settings. See [this
+page](https://github.com/opencadc/science-platform/tree/master/doc#groups-and-permissions)
+for instructions.

docs/general/General_tools/Using_sshfs.md

@@ -0,0 +1,94 @@
+# Using SSHFS {#sshfs}
+
+SSHFS, or the secure shell file system, allows users to interact with
+the directories and files from their Science Portal account directly on
+their local machine.
+
+## Installation
+
+Software installation is required to use this tool.
+
+*Linux*: SSHFS is Linux-based software that needs to be installed on
+your local computer. On Ubuntu and Debian based systems, it can be
+installed through apt-get:
+
+    sudo apt-get install sshfs
+
+*Mac OSX*: Often SSHFS is already installed; if not, you will need to
+download FUSE and SSHFS from the [osxfuse
+site](https://osxfuse.github.io)
+
+## Mount the Remote File System
+
+For Ubuntu/Debian Linux or Mac OSX, the instructions are below.
+Instructions for Windows users can be found at the bottom of
+[this](https://www.digitalocean.com/community/tutorials/how-to-use-sshfs-to-mount-remote-file-systems-over-ssh)
+page.
+
+To start, we will need to create a local directory in which to mount the
+file system, \"arc\":
+
+    mkdir $HOME/arc
+
+Now we can mount the file system locally using the following command,
+based on which OS you are running. You will be asked for your CADC
+password during this step.
+
+*On Ubuntu/Debian*:
+
+    sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=10 -p 64022 [your_cadc_username]@ws-uv.canfar.net:/ $HOME/arc
+
+*On Mac OSX*:
+
+    sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=10,defer_permissions -p 64022 [your_cadc_username]@ws-uv.canfar.net:/ $HOME/arc
+
+The extra `defer_permissions` switch works around issues with OSX
+permission handling. See [this
+page](https://github.com/osxfuse/osxfuse/wiki/Mount-options#default_permissions-and-defer_permissions)
+for more details.
+
+## Synch Local and Remote Directories with rsync
+
+With the steps above in place, the rsync (\"remote synch\") command can
+be used. rsync uses an algorithm that minimizes the amount of data
+copied by only moving the portions of files that have changed. Further
+rsync examples and docs are
+[here](https://www.digitalocean.com/community/tutorials/how-to-use-rsync-to-sync-local-and-remote-directories).
+
+The synch is performed using the following:
+
+    rsync -vrltP source_dir $HOME/arc/destination_dir/
+
+where
+
+-   `-v` increases verbosity
+-   `-r` recurses into directories
+-   `-l` copies symlinks as symlinks
+-   `-t` preserves modification times (use the command \"man rsync\" for
+    more details on why this option prevents resending already
+    transferred data when not using `-a`)
+-   `-P` keeps partially transferred files and shows progress during
+    transfer
+
+Pro tip: including a `/` after source_dir in the command above will
+transfer the directory contents without the main directory itself. i.e.,
+if your source_dir contains a file called test, then :
+
+    rsync -vrltP source_dir $HOME/arc/destination_dir/
+
+will add in the file as `$HOME/arc/destination_dir/source_dir/test`
+whereas:
+
+    rsync -vrltP source_dir/ $HOME/arc/destination_dir/
+
+will add in the file as `$HOME/arc/destination_dir/test`
+
+## Unmounting the File System
+
+If you have finished working with your files and want to disconnect from
+the remote file system, you can do this by:
+
+    umount $HOME/arc
+
+NB: If you run into problems with the original sshfs command and need to
+run it again, you will likely need to unmount first.