Last month the ASA team updated azure-streamanalytics-cicd, their companion npm package. It now includes the addtestcase and test commands that add support for query unit testing. These commands are well documented and quite easy to use.
Release 1.1.0 of asa.unittest adds helper tools to migrate existing tests to the new official tool. I document the process here
If you're getting started, go directly to azure-streamanalytics-cicd instead of asa.unittest.
A PowerShell Gallery module for unit testing Azure Stream Analytics (ASA) jobs, the serverless complex-event-processing service running in Azure.
In this article:
At the time of writing, the major IDEs that support ASA (VSCode and Visual Studio) do not offer native unit testing capabilities -> Not true anymore, see above.
This tool intends to fill that gap by enabling:
- fully local, repeatable executions over multiple test cases
- automated evaluation of the resulting outputs against expected ones
For that it leverages the local testing with sample data capabilities of either VSCode or Visual Studio, as unit testing should not rely on external services (no live input).
Local runs are scripted thanks to the sa.exe tool from the Microsoft.Azure.StreamAnalytics.CICD package.
The results are then evaluated against reference data sets thanks to the Compare-Object PowerShell cmdlet.
The whole thing is wired together in a PowerShell script based on a predefined test fixture (folder structure + naming convention):
figure 1 - High level overview
This repository provides an installation script (New-AutProject), in addition to the test script (Start-AutRun), to automate most of the setup. This installation script also allows automated executions in a continuous build pipeline such as Azure DevOps Pipelines.
Please note that this solution is currently available only on Windows as it depends on Microsoft.Azure.StreamAnalytics.CICD.
From Wikipedia:
Unit tests are typically automated tests written and run by software developers to ensure that a section of an application (known as the "unit") meets its design and behaves as intended.
Here the unit is an individual output of an ASA job / query. The test runner will need all the test inputs required for the job, but it will calculate test results only for outputs having a reference data file provided.
For practical reason (limiting the number of tests mean limiting the number of parallel runs to do), a single test can involve multiple outputs, as is demonstrated in the sample files.
Unit tests should not rely on external services, so all runs are done via local runs on sample data. Using live sources, or the Cloud service, to run tests would not qualify as unit testing.
This solution leverages PowerShell, and a nuget package to enable ASA unit testing:
- For PowerShell, any recent version should do (assets tab under a specific release)
- The nuget CLI will be downloaded via the provided installation script, but it requires the .NET Framework 4.7.2 or above to run.
From there, the installation script will take care of the other dependencies (including the nuget CLI).
To be noted that those requirements are installed by default on every Azure DevOps Pipelines agents.
The scripts will expect the following folder structure to run properly:
- mySolutionFolder <- Potentially new top solution folder
- ASATest1 <- Existing ASA project folder, containing the
.asaqlfile and inputs folder - ASATest1.Tests <- New folder for the test project
- 1_arrange <- New folder that will contain test cases
- 2_act <- New folder that will contain dependencies and scripts
- 3_assert <- New folder that will contain test run results
- ASATest1 <- Existing ASA project folder, containing the
The step-by-step processes below explain how to set up this environment from scratch.
The following steps show how to download and run the solution with the included Hello World ASA project:
- Check all the requirements are installed
- Import the module from the PowerShell Gallery
- Open a Powershell host (terminal, ISE, VSCode...)
- Run
Install-Module -Name asa.unittestto import the module from the PowerShell Gallery
- Clone/download the Examples folder from this repository, save it to a convenient location (
C:\temp\examplesfrom now on) - Only once - execute the installer:
New-AutProject- In the Powershell host
- Run
New-AutProject -installPath "C:\temp\examples\ASAHelloWorld.Tests" -verbosewithinstallPaththe absolute path to the test folder, not the ASA project folder 
- In case of issues see troubleshooting
- Execute the test runner:
Start-AutRun- In the Powershell host
- Run
Start-AutRun -solutionPath "C:\temp\examples" -asaProjectName "ASAHelloWorld" -verbosewithsolutionPaththe absolute path to the folder containing both the ASA and the Test projects.Start-AutRunoffers additional parameters that can be discovered via its help 
- Here it is expected that the test ends with 2 errors, in test case 003. The folder
C:\temp\examples\ASAHelloWorld.Tests\3_assertwill contain the full trace of the run - In case of issues see troubleshooting
The following steps show how to download and run the solution on an existing ASA project:
- Check all requirements are installed
- Prepare the ASA Project
- If it doesn't exist, create a solution folder (simple top folder,
C:\temp\examplesin the HelloWorld above) - Copy or move the existing ASA project to the solution folder
- In ASA, add local inputs for every source used in the query (see VSCode / Visual Studio)
- If it doesn't exist, create a solution folder (simple top folder,
- Only once - Import the module from the PowerShell Gallery
- Open a Powershell host (terminal, ISE, VSCode...)
- Run
Install-Module -Name asa.unittestto import the module from the PowerShell Gallery - In case of issues see troubleshooting
- Only once - execute the installer:
New-AutProject- In the Powershell host
- Run
New-AutProject -installPath "MySolutionPath\MyTestFolder" -verbosewithinstallPaththe absolute path to the test folder, not the ASA project folder. Usually theMyTestFoldersub-folder is of the formMyAsaProject.Tests
Once the installation is done:
- Configure a test case
- Execute the test runner:
Start-AutRun- Open a Powershell host (terminal, ISE...)
- Run
Start-AutRun -solutionPath "MySolutionPath" -asaProjectName "MyAsaProject" -verbosewithsolutionPaththe absolute path to the folder containing both the ASA and the Test projects.Start-AutRunoffers additional parameters that can be discovered via its help - In case of issues see troubleshooting
For local development, the recommended way of running jobs is via a terminal window.
A test case is made of at least 2 files : a test input data file and a reference output data file. The test runner will start a local job ingesting the test input data, and compare the output of that job to the reference output data. Mapping inputs and outputs will be done following a file name convention.
Reminder : every input source used in the job query to be tested must have a local source configured in the ASA project (VSCode, Visual Studio).
Once this is done:
- In
MyTestFolder\1_arrange, prepare input files:- Copy the test input data file to be used in the test case for the test case
- Rename it according to the file name convention :
xxx~input~sourceAlias~testLabel.yyyxxx: for the test case number (grouping multiple inputs and outputs together), for example : 001, 002...~: as the separatorinput: flags the file as an input filesourceAlias: alias of the source in the querytestLabel: a label to identify the test (nominal,missingField,nullValue...)yyy: any of the supported data format extension (csv, json, avro)
- In
MyTestFolder\1_arrange, prepare output files:- Copy the reference output data file to be used in the test case
- Rename it according to the file name convention :
xxx~output~sinkAlias~testLabel.jsonxxx: for the test case number (grouping multiple inputs and outputs together), for example : 001, 002...~: as the separatoroutput: flags the file as an input filesinkAlias: alias of the destination in the querytestLabel: a label to identify the test (nominal,missingField,nullValue...)json: the data must be in JSON format as it's the only format currently supported for local output (the ASA engine doesn't honor the output format for local runs)
Note that live extracts are a good way to generate test input data. For reference output data, the best is to leverage the output files in the LocalRunOutputs folder, located in the ASA project after a successful local run. Note that these files may be generated in a line separated format which is not supported by the test runner. They will need to be corrected as follow:
Unsupported format:
{"EventId":"3","EventMessage":"Hello"}
{"EventId":"4","EventMessage":"Friends"}Supported format (brackets and commas):
[
{"EventId":"3","EventMessage":"Hello"},
{"EventId":"4","EventMessage":"Friends"}
]The solution comes with a couple of pre-configured test cases to illustrate the format and naming convention.
Use a PowerShell task to run the installation script first, and the test runner script second.
Note that both scripts use default values for most parameters. These default values are wired to the build variables provided by Azure DevOps. As such, they can be left unassigned in the task.
The parameters are:
- For the installation script (
New-AutProject)$installPath, the folder containing the test fixture
- For the test runner (
Start-AutRun)$asaProjectName$unittestFolderis defaulted to$asaProjectName.Tests$solutionPathis defaulted to$ENV:BUILD_SOURCESDIRECTORY
The main causes of error are:
PowerShell execution policies. PowerShell comes with that mechanism which is supposed to make a user deeply aware of the fact they're running a script they haven't written. For users with administrative rights, it's an easy to solve issue via an admin session and the command Set-ExecutionPolicy -ExecutionPolicy Unrestricted (doc). For non admins, the easiest is to create new powershell scripts (text files with the .ps1 extension) and copy/paste the content of each script (install, Start-AutRun). Note that the VSCode Integrated PowerShell environment has an issue with execution policies and should be avoided (use terminal instead)
PowerShell remoting for jobs. Depending on the version of PowerShell (older), it may require remoting to be enabled to start background jobs. Background jobs are used by the test runner to start runs in parallel. This should not be necessary, but the command to enable remoting is Enable-PSRemoting
- Both the ASA project folder and the unittest folder need to be in the same solution folder
- The ASA project folder, ASA script file (
.asaql) and the XML asaproj file (.asaproj) needs to have the same name - Bad format for JSON data files: test input JSON files and all reference output file need to be array of records (
[{...},{...}]) - see Configuring a test case
- October 2020 :
- Add helpers command to migrate existing tests to the new official tool
- May 2020 :
- Update doc to reflect the removal of the jsondiffpatch dependency
- Fix open issues
- March 2020 :
- Full refactoring to allow publication in the PowerShell Gallery
- Complete unit test coverage (Pester) and linting
- February 2020 :
- Automated the generation of the XML asaproj file (required by sa.exe) from the JSON one (for VSCode project)
- Renamed scripts to follow PowerShell standards, with backward compatibility
This solution uses the following components:
- the Microsoft.Azure.StreamAnalytics.CICD nuget package, which provides
sa.exe, a Windows-only executable that allows to run an ASA job via command line- the nuget CLI, to download and install the package above
- PowerShell as the shell to run intermediary tasks and execute the required commands
Slow executionSomewhat mitigated via the parallel execution of tests- No integration to the usual IDEs
- No cross-platform options
- Kevin Marquette for his invaluable blog on PowerShell
- jsondiffpatch : Diff & patch JavaScript objects (used actively up to release 1.0.7)
- Tabler icons : A set of over 400 free MIT-licensed high-quality SVG icons for you to use in your web projects