Before you begin to work with Azure Big Compute & HPC, we recommend that you take a few minutes to download and install some relevant Azure client side tools that will make working with Azure a lot faster and easier (and more fun)! These utilities can either be installed on your laptop, or you can provision a linux or windows VM in Azure, and install them there.
Most of these tools are universal and run on Linux, Windows or MacOS.
If you are looking for a quick platform to start working with Big Compute, then the BigComputeBench ARM template is a great way to get started fast.
If you're going to be using Linux and you are using Windows 10 or later on your desktop, it's a good idea to install the new Linux/Bash capability. Find the instructions here and here. You can run bash in the old CMD/DOS shell, but for a much more Linux-native experience, run the Ubuntu or SUSE shells directly (search in the start menu for eg: Ubuntu):
Azure CLI is THE tool to interact with Azure from the command line. You can use to create objects such as virtual machines, networks, scale-sets & clusters, storage, disks, and so on. Azure CLI download and instructions are available here.
Note that you can also invoke Azure CLI via thecloud shell from the azure portal.
StorageExplorer is a feature-rich graphical tool for managing Azure Storage resources & data securely. Grab it from storageexplorer.com
- Copy Data to and from Azure, and across Azure storage containers & subscriptions.
- Share datasets and collaborate with your colleagues inside & outside your organization
Azcopy is a utility for copying and moving files around to and from Azure or between Azure storage containers. It's very efficient at copying large datasets comprised of large files or 1000's of small files, and can easily be incorporated into scripts. It's also embedded into StorageExplorer as the copy engine there. Use AzCopy 10+ in preference to slower alternatives such as blobxfer, az storage upload, and the older AzCopy pre version 10 - you'll find it much higher performant and more efficient at saturating links to upload/download data to and from azure as quickly as possible.
These days most of what you need for the common Azure Batch tasks is included in the Azure Batch CLI (see above) under the "az batch" sub-command. However, for the labs we will need some extra tools normally reserved for black belts:
Batch Labs is THE graphical tool for interacting with Azure Batch, and replaces the old Batch Explorer tool. You can download the latest build for your operating system. Batch Labs source code is also online here.
Batch Shipyard is a set of client-side tools that makes running container based Big Compute jobs on Azure Batch a breeze. Information and overview here. Clone the github repository to your workstation and follow the install instructions linked on the main README.
The Batch CLI Exensions provide a bunch of new advanced functionality for Azure Batch from the CLI including shipyard style templates, Task Factory, automated file upload/download and new package management features. After installing Azure Batch CLI, follow the instructions on the github README to install the extensions.
In case you hadn't heard, infrastructure in the cloud is nothing but CODE. In Azure we call it ARM - Azure Resource Manager.
Get yourself a github account. You'll find lots and lots of example templates, including plenty for HPC:
- General Azure Examples Repository
- 5 Click Templates for HPC Clusters & Apps
- Production HPC Cluster Deployment
- Simple, Flexible Modular, Multi-OS HPC Deployment
If you want to getting into coding for Azure, here are two great expert tools that have direct Azure & Github integration:
- Visual Studio Code is a free and lightweight version of Visual Studio, and is great for rapid code editing and github integration.
- Visual Studio is the most comprehensive IDE on the planet, with cloud and mobile integration. This tool is a must for serious developers.