Skip to content

Programming

Jump to bottom
William Stein edited this page Mar 27, 2018 · 36 revisions

Programming Software from CoCalc

This FAQ is all about programming your own software (C, Fortran, Java, Julia, Octave, R, etc.) from CoCalc. A highly related page is the FAQ for Utilizing External Tools from CoCalc. There is also a separate page for Sage and Jupyter.

Remember: if you don't find what you need, or if you'd like to ask a question, then please email [email protected] at any time. We'd love to hear from you! Please include a link to any relevant project or document (copy and paste the URL address in your browser) as part of your email.

List of Questions:

Question: I would like to create, compile and run a C program.

Click here to view a video about this question.

  1. Click +New, type a filename ending in ".c", e.g., foo.c, and click "Create File" (or just press return).

  2. Paste this code into the file:

#include<stdio.h>
int main(void) {
    printf("Hello World\n");
    printf("this is CoCalc!\n");
}
  1. Open a terminal by clicking +New, clicking "Command Line Terminal" (or typing a filename ending in .term), and type gcc foo.c. Finally, run the program by typing ./a.out.

Question: I would like to create, compile and run a Fortran F90 program.

Click here to view a video about this question.

  1. Create a file sum.f90 by clicking +New, which I copy and paste from http://en.wikibooks.org/wiki/Fortran/Fortran_examples#Area_Of_a_Triangle_program

  2. Create a terminal and type gfortran sum.f90

  3. Run the program by typing ./a.out

Question: I would like to create, compile and run a Java program.

Click here to view a video about this question.

  1. Create a file HelloWorld.java containing
public class HelloWorld {
    public static void main (String[] args) {
	System.out.println ("Hello World!");
    }
}

  1. Create a terminal and run javac HelloWorld.java to compile your program.

  2. Run java HelloWorld to see the output.

Question: I would like to use Octave

I've put an example Octave Jupyter notebook and an Octave CoCalc worksheet here:

https://cocalc.com/projects/4a5f0542-5873-4eed-a85c-a18c706e8bcd/files/cloud-examples/octave/

Unfortunately, neither Jupyter nor CoCalc worksheets in Octave mode are "rock solid". For something that is rock solid, type "+New", click terminal, and type "octave" on the command line, and this should work well. You can type "+New", enter a filename that ends with .m, and edit it, then load it into the command line (by typing the filename without the extension).

You might also find this useful: https://github.com/sagemath/cloud/issues/97

Question: I would like to use Julia in a CoCalc worksheet.

  1. Click +New, type a filename, then click the "Sage Worksheet" button.

  2. To evaluate code using Julia, begin the cell with %julia, type the code, then press shift+enter.

  3. Type %default_mode julia in a cell and press shift+enter; now all cells will be evaluated using Julia by default. If you need to switch back, use %default_mode sage (or %sage to just switch back for one cell).

Question: How do I adjust the output cut-offs, for example, to protect against infinite loops?

It is an extremely common programming mistake to write an infinite loop, particularly when first learning about loops. Because CoCalc assumes an experienced programmer, the "cutoff limits" are set rather high. Users new to programming might want to set that limit lower, so that their screen isn't overflowing with repeated lines in the event that they inadvertently code up an infinite loop. (By the way, this works in all languages, not just Sage, e.g. R, C, FORTRAN, whatever you'd like.)

You can type

print sage_server.MAX_STDOUT_SIZE

at any time to find out the current limit. By default, it is 40,000.

Then, you can change it by typing something like this:

sage_server.MAX_STDOUT_SIZE = 500

Note, this is 500 characters. Take care to ensure that the setting of this variable will be executed before your code starts. If you type

sage_server.MAX_STDOUT_SIZE = 500

for i in range(0,1000):
    print i

then it will be cut off somewhere in the middle of printing 152, because you need to count each digit, as well as the invisible "end of line" symbol. At the 501st character, the computation is stopped, and there is no more output.

By the way, it isn't just the case that the output is truncated at this point. The computation is halted as well. (The technical term for this is that "the process is killed.")

Question: I want to install Python3 and use it as my default.

With full network access enabled, you can download and compile Python 3 this way. Last line sets a symlink to make it your default!

apt-get source python3.4-dev
cd python3.4-3.4.0/
./configure --prefix=$HOME
make
make install
cd ..
pip3 install numpy
pip3 install scipy
pip3 install matplotlib
pip3 install ipython
pip3 install pyzmq
pip3 install jinja2
pip3 install tornado
ln -s ~/bin/python3 ~/bin/python

Question: How to fix an exception related to Sage's Integer(...) vs. Python ints?

By default, Sage parses the input commands and replaces some elements with its own parts and also adds some syntactic sugar. For example, an integer like 234 is translated to Integer(234) in order to be more powerful and live as a part of Sage. To avoid this behaviour, either append an r to the number, like 234r (the extra r tells Sage to consider this as "raw" input) or change the mode of the cell to Python by adding %python at the top. You can also switch to pure Python mode by default via %default_mode python. Alternatively, you can type Integer=int and possibly also RealNumber=float.

Question: How can I install Python packages from PyPI using pip?

WARNING: Due to people launching attacks from CoCalc, access to the internet from within a free project is disabled and hence using pip install --user will not work. A small subscription enables internet access for your project.

First, you need to know if it is Python 2 or 3. Let's suppose the package is called ggplot. Create a new terminal in a project (+New-->Terminal) and type

pip2 install --user ggplot

or for Python 3

pip3 install --user ...

Now ggplot should be available in your project's worksheets. In case you run a CoCalc worksheet, you need to restart the worksheet server (in the project's settings) and then the worksheet itself via the "restart" button.

The $HOME/.local/ path is the "canonical" root for some overlay directories of Linux's standard directory layout. In the case of Python 2, $HOME/.local/lib/python2.7/site-packages/ will contain the package you've installed.

To use binaries installed by pip add export PATH=~/.local/bin:$PATH to ~/.bashrc and run source ~/.bashrc

In case your Python environment can't find the package, you might have to add your ~/.local/... directory dynamically during runtime like that:

import sys, os
sys.path.insert(0, os.path.expanduser('~/.local/lib/python2.7/site-packages'))

Make sure, the path is correct. I.e. for Python 3 this could be one of python3.4, python3.5, python3.6...

Question: I would like to develop a webserver in Python

Here's a simple example of an HTTP server written using the Python SimpleHTTPServer class. Open a project and click "+New" then paste in this link, then click the "From Web" button (or use wget or copy/paste into a file): https://gist.githubusercontent.com/williamstein/c69001187923c2a62f1b42126a4a77e8/raw/b50f9410a6d1a55144931ad419affa9466094386/server.py

Then in a terminal type

chmod +x server.py
./server.py

You'll see a URL in a box that looks something like this

 https://cocalc.com/4a5f0542-5873-4eed-a85c-a18c706e8bcd/port/10000/

Just paste it into your browser.

IMPORTANT: This web server is visible exactly to collaborators on your project and nobody else, and all communication is SSL encrypted. Thus you can't use this method to create a general purpose webserver available to the world. On the other hand, this is a good way to collaborate with a controlled group of people on development, without having to worry about security.

If you just want an http view of your files, use the raw servers, which is already available by default at https://cocalc.com/project_id/raw/. The point of the above script is that you could modify it to provide all sorts of interesting functionality.

Question: I would like to install new R packages.

Open a terminal windows and type

R

Then you can install packages as usual

install.packages('packagename')

The above will install R packages for use with CoCalc worksheets (%r mode) and Jupyter notebooks using default R. The Sage binary may be built with a different release of R. Use R-sage instead of R to install packages for it.

Note that you must also upgrade your project to have network access (requires a subscription).

Question: I would like to process an R Markdown (.Rmd) document.

Suppose you have folder "RMD" under your home directory in CoCalc containing file "myproject.Rmd", and you would like to create a .pdf or .html file from the R markdown.

Then open a CoCalc worksheet, i.e. a .sagews file and do these two steps:

1. Use knitr in an R cell to create a plain markdown file.

%r
# create .md from .Rmd
library(markdown)
library(knitr)
knit("myproject.Rmd")

2. Use pandoc in an sh cell to generate the .pdf.

%sh
pandoc myproject.md -V geometry:margin=0.75in -o myproject.pdf

For .html output, change the file extension at the end of the last line from .pdf to .html. View or download the generated .md and .pdf or .html files from the CoCalc file browser.

Question: I would like to use R in a CoCalc worksheet.

Video

  1. Click +New, type a filename, then click the "Sage Worksheet" button.

  2. To evaluate code using R, begin the cell with %r, type the code, then press shift+enter.

  3. Type %default_mode r in a cell and press shift+enter; now all cells will be evaluated using R by default. If you need to switch back, use %default_mode sage.

Question: I would like to override the default width and height for R svg figures in a CoCalc worksheet.

To set width to 10 inches and height to 4 inches, use the sage command:

r.set_plot_options(width=10, height=4)

If you have set default_mode to r, then enter the command in a sage mode cell:

%sage r.set_plot_options(width=10, height=4)

You can change it by typing it again.

Question: I need to spell check my LaTeX documents

Whenever you save the LaTeX document, it will run a spell checker and underline the words that are not spelled correctly. By default, it uses the language you've set in your web browser.

You can change the autosave interval to be very short in account settings (under editor) if you need the spell checking to update frequently.

Seeing a list of alternative words (correct spellings) isn't supported directly in the editor yet https://github.com/sagemath/cloud/issues/50. For now, a workaround is to run LaTeX-aware aspell, e.g.

1. open a terminal
2. aspell -t -c <filename.tex>

Question: How can I properly work with multi-document LaTeX projects?

Suppose your LaTeX project is composed of one master.tex file and several chapter-1.tex, chapter-2.tex, etc. CoCalc's LaTeX editor only knows about the currently opened file, and using \import{} doesn't work, because the chapter-*.tex parts are not proper documents.

To solve this, use the subfiles package instead. It does not only collect the partial documents into one, but also extracts the preamble of the master.tex file for each chapter-*.tex in order to create valid subdocuments.

Following the documentation, do this:

  1. \usepackage{subfiles} in master.tex

  2. \subfile{⟨subfile name ⟩} for each subfile in master.tex's document environment (i.e. instead of \include or \import).

  3. For each chapter-*.tex subfile:

      \documentclass[⟨master.tex file-name⟩]{subfiles}
  \begin{document}
  ⟨text, graphics, etc.⟩
  \end{document}

After that, all *.tex files can be compiled and all other features like forward/inverse search work, too.

Question: I want to browse and view (or edit) files outside of my project's home directory.

You can make a symbolic link to the root of the filesystem by typing

 ln -s / root

Now you can explore the complete filesystem. Note that there many files that you can only read and not write.

Question: Is .bashrc or .bash_profile called on startup?

~/.bashrc is run on startup and ~/.bash_profile is not! Hence, use ~/.bashrc to customize your setup, and you can also use ~/.bash_aliases for your aliases (see ~/.bashrc).

Make sure you have at least 6GB disk space (look at quotas in project settings -- if you don't, don't want to upgrade, and are just playing around, cd to /tmp instead), then type

 install-latest-sage-release

Then type this, assuming the name of the install you just got is sage-6.7:

cd; mkdir -p bin; cd bin; ln -sf ~/sage-6.7/sage .

See the discussion about worksheet servers in the next FAQ question below. Also, now that you have your own copy of Sage, you can change anything in Sage! Do so, send us patches, etc. See http://www.sagemath.org/doc/developer/ for step-by-step instructions.

Question: How to turn off the character accent selector and re-enable key repetition on Mac OS X ?

launch a Terminal and run the command

 defaults write -g ApplePressAndHoldEnabled -bool false

Question: I would like that all of my worksheets know where to find a given module that I write or install.

Put an executable file with this content in $HOME/bin/sage:

#!/usr/bin/env bash
SAGE_PATH=$HOME/NEW_MODULE /usr/local/bin/sage "$@"

You could do this by making a new directory called bin, then a new file in there called "sage". Then in the terminal type the following to make "sage" executable.

  cd; cd bin; chmod +x sage

This is also the file "sage" attached to this message.

Then restart the worksheet server by going project settings and clicking "Restart --> Worksheet server".

Now any newly (re-)started worksheet will run with the above modified SAGE_PATH. Since SAGE_PATH is added to PYTHONPATH when Sage starts, this does what you want.

Obviously, I plan to add a simple way to do something equivalent to the above, by filling in some settings box in the UI. I'll update this FAQ entry once I do that.

(From Nathan Dunfield) Another approach, which also works now and doesn't require the custom "$HOME/bin/sage", is to use http://docs.python.org/2/install/#alternate-installation-the-user-scheme

That is, one installs a module with sage -python setup.py install --user and it's dumped into

$HOME/.local/lib/python2.7/site-packages

This location is searched automatically by Sage's Python without any intervention on the part of the user. (However, I did have to restart the worksheet server to access newly installed modules from a worksheet.) One can also put modules into the user's site-packages by hand and Sage will find them.

Question: How can I tell if my code is running in a CoCalc worksheet (a .sagews file)?

If your code is running in a CoCalc worksheet, then the global variable __SAGEWS__ will be defined.

Question: How do I access functionality specific to CoCalc worksheets (.sagews files)?

Type salvus.[tab key].

Question: I want to run Sage locally, on my own machine. How do I do that?

There is a lovely tutorial on the web to help you do exactly that:

http://doc.sagemath.org/html/en/installation/index.html

Question: I want to use my own custom built Sage binaries on CoCalc

See the instructions, immediately below, on using a custom built-from-scratch copy of Sage. Just substitute your own .tar.gz file for the official build of Sage.

Question: I want to use my own custom built-from-source copy of Sage (Warning: this takes hours.)

Open a terminal. Grab the source tarball (requires network access). You can browse http://sage.math.washington.edu/home/release/ to find recent releases and testing versions.

To build, do the following in your terminal (no need to worry about screen or tmux, of course, since sessions are persistent even if your browser leaves), and check back in a few hours:

tar xvf sage-6.10.tar.gz && cd sage-6.10 && make

WARNING: Building can easily take more than 2 hours. By default CoCalc projects have an idle timeout that is smaller. (See What is an "idle timeout?".) If you aren't editing files in the project, your build will get killed part of the way through. If you're doing legit Sage development, email THE LINK TO YOUR PROJECT to [email protected] and we will increase the idle timeout, disk space, RAM, etc, so you can contribute to Sage.

After doing that, do something like this in the terminal:

cd; mkdir -p bin; cd bin; ln -s ~/sage-6.10/sage .

Then restart your worksheet server (in project settings). Then for that project, you'll have your own 100% customizable copy of Sage; and moreover, when the system-wide Sage is upgraded, your project isn't impacted at all -- that sort of stability is a major win for some people. This also uses little extra disk space in backups/snapshots, because of de-duplication. You can of course also install any custom packages you want into this copy of Sage. You can also help improve Sage: http://www.sagemath.org/doc/developer/

If you want to do Sage development see http://mathandhats.blogspot.com/2014/06/how-to-develop-for-sage-using-sage-math.html.

Important: Whenever you change Python code installed in that copy of Sage, you may have to restart the worksheet server and any running worksheets. This is inconvenient, but is necessary because the worksheet server starts one copy of Sage, then forks off additional copies each time you open a new worksheet, which greatly reduces the time from when you open a worksheet until it actually starts computing things.

Also Important: If your copy of Sage is messed up in some way, then the worksheet server can't start, hence worksheets won't open. To debug this, open a terminal and do this:

~$ cd .smc
~/.smc$ sage sage_server.py
you should see an error here, e.g.,

and fix whatever error you see. Also look at log files in ~/.smc/sage_server/

Question: How do I install some software into my own Anaconda environment in CoCalc?

The task below is to create a custom Anaconda overlay environment called myconda and, just for the sake of explanation,

  1. install "Microsoft's Open R" (which is an enhanced version of R by microsoft).
  2. Install the plotly library from PyPI

To get it installed in Anaconda as a user, do this:

  1. Open a terminal.

  2. Type anaconda3

  3. Type conda create -n myconda -c mro r This creates a new local environment called "myconda" (name it as you wish) with the package "r" as its source coming from the channel "mro" (Microsoft's Open R). Instead of that, you can add any other anaconda package in that spot. The example from the documentation is biopython, see http://conda.pydata.org/docs/using/envs.html#create-an-environment.

  4. When installing, it briefly shows you that it ends up in ~/.conda/envs/myconda/.... in your local files. Now that we have it installed, we can get out of this "root" environment via source deactivate or restart the session. In any case, you are back in the the normal Linux terminal environment.

  5. Now run this: source ~/.conda/envs/myconda/bin/activate myconda Note that myconda is the name specified above, and the prompt switches to (myconda) $. Typing which R shows: /projects/xxx-xxx-xxx/.conda/envs/myconda/bin/R and of course, just running R gives:

R version 3.2.3 (2015-12-10) -- "Wooden Christmas-Tree"
Copyright (C) 2015 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
[...]
Microsoft R Open 3.2.3
Default CRAN mirror snapshot taken on 2016-01-01
The enhanced R distribution from Microsoft
  1. In the very same spirit, you can also run pip installations:
(myconda)~$ pip install plotly
Downloading/unpacking plotly
[...]
Successfully installed plotly requests six pytz
(myconda)~$ python -c 'import plotly; print(plotly)'
<module 'plotly' from '/projects/20e4a191-73ea-4921-80e9-0a5d792fc511/.local/lib/python2.7/site-packages/plotly/__init__.pyc'>

Note that since I'm still in my own "myconda" overlay environment, the --user switch in pip install wasn't necessary. (Otherwise, it would be necessary.)

Question: How do I configure a Jupyter kernel for my custom Anaconda environment?

With Anaconda's conda command, you can create custom environments with specific versions of Python, R, and their packages. This is similar to capabilities provided by Python's environment manager, virtualenv.

Suppose you want to create a custom Anaconda environment with the mdtraj package and be able to use this environment in a Jupyter notebook. Here's how.

  1. Follow these steps in a .term file in CoCalc. In the last step, the display name of the new kernel is changed so that it does not duplicate the name of kernel installed by CoCalc.

     ~$ mkdir -p ~/.local/share/jupyter/kernels
 ~$ anaconda3
 (root) ~$ conda create --name mymdtraj mdtraj
 (root) ~$ source activate mymdtraj
 (mymdtraj) ~$ conda install ipykernel
 (mymdtraj) ~$ source deactivate
 ~$ mv ~/.conda/envs/mymdtraj/share/jupyter/kernels/python3 ~/.local/share/jupyter/kernels/mymdtraj
 ~$ open ~/.local/share/jupyter/kernels/mymdtraj/kernel.json
 ## change display_name from "Python 3" to "My mdtraj" and save the file

  2. Open a new Jupyter notebook in CoCalc.

  3. Click on the Kernel button and look for your new kernel, "My mdtraj", or whatever you used for display_name in kernel.json. If you don't see your new kernel, scroll to the bottom of the Kernel menu and click Refresh Kernel List, and your new kernel should appear.

  4. Select the new kernel. You will now be running the environment you created with the conda create command.

Question: I want to start long-running numerically intensive computations on CoCalc. What are the current limitations?

Open your project and click on Settings. The default limitations are listed under "Quotas" in the lower left. These can be raised, as mentioned there. Notes:

  1. Projects without "member hosting" upgrade can get restarted regularly (these are hosted on Google preemptible instances). You can check if a VM rebooted by typing "uptime". crontab files are persistent.

  2. If a project isn't used (via the web-based UI) for the idle timeout (as listed in quotas), then all processes in that project are terminated and the user is removed (so ssh into the project also is not possible). You can pay to raise the idle timeout. See also, What is an "idle timeout?".

Question: I want some scratch space

Use /tmp. Files in /tmp may be deleted at any time, and aren't backed up.

Question: Will my code keep running if I disconnect? Even if my computer is put to sleep? Or do I need to have a machine open in order for the process to run?

You definitely do not need to have your computer awake, or a window open, for your project to keep working. However, this is controlled by something called an "idle timeout," described in the next question.

Question: What is an "idle timeout?"

Under project settings (that's the wrench icon) there is an entry under "Project Usage and Quotas" (left-hand side), which will tell you how long the process will run "in the background." There is an idle timeout for each project, and it will be completely stopped (the technical term in UNIX is "killed") if you don't actively edit a file for that amount of time.

The default for free projects is 1 hour. You can increase this to 24 hours for only $14 per month. This means that if you use your project a little bit once per day, then it will never timeout.

However, free projects have another limitation. A free project can be "killed" (stopped) at any time, whatsoever. This will happen at least once per day. You have to keep this in mind when designing your project. (For example, use checkpointing.) In contrast, all paid projects are immune to this issue. See also Subscription and Pricing Information.

The next question will discuss the output of your processes.

Question: If I have code that has been running for a while, and it times out or is otherwise "killed" (see previous question), what happens to the output?

If you are using a Jupyter notebook, then all output that is printed will be lost if no browser is viewing it. This is a major design flaw in Jupyter.

In contrast, CoCalc worksheets will capture output even if no browser is observing them.

You can also (of course) write to a file on disk, which might be preferable in some cases.

Question: I want to see all processes running in my project

Type exactly the following in a full terminal (+New--> Terminal) to see all processes running in a project:

htop

You can kill things, etc. See http://linux.die.net/man/1/htop.

Question: I want to know how much memory I am using

Type exactly the following in a full terminal (+New--> Terminal):

smem -tk

It lists all processes and the bottom line shows the total sum. The last RSS column is probably the most interesting one, for more consult man smem. The total used memory is also listed under 'Project usage and quotas" in project settings.

Question: How do I raise the limit on the number of output messages per cell in a CoCalc worksheet?

import sage_server
sage_server.MAX_OUTPUT_MESSAGES=100000

See this published worksheet for more details.

Also, type sage_server.[tab key] to see information about other limitations.

Question: I want to XXX, but I don't see XXX above.

Do not hesitate to request support, either by using the "Help" button at the top right of any CoCalc window, or by sending an email (be sure to include a LINK TO THE CoCalc PROJECT where you need help) to [email protected]. You can also ask questions on the CoCalc mailing list.

Analytics

This Wiki is for CoCalc.com.

A more structured documentation is the CoCalc User Manual.

For further questions, please contact us.

Clone this wiki locally