Add automated tests for demo code snippets

.travis.yml

@@ -0,0 +1,17 @@
+dist: xenial
+language: python
+
+matrix:
+    include:
+        - python: "2.7"
+        - python: "3.5"
+        - python: "3.6"
+        - python: "3.7"
+
+install:
+    - pip install -r requirements.txt
+
+script:
+    - python run_demo_md.py
+
+

README.md

@@ -1,4 +1,4 @@
-# in-toto demo
+# in-toto demo [![Build Status](https://travis-ci.com/in-toto/demo.svg?branch=master)](https://travis-ci.com/in-toto/demo)
 
 In this demo, we will use in-toto to secure a software supply chain with a very
 simple workflow.  Bob is a developer for a project, Carl packages the software, and
@@ -13,13 +13,13 @@ This is, you will perform the commands on behalf of Alice, Bob and Carl as well
 as the client who verifies the final product.
 
 
-### Download and setup in-toto on *NIX (Linux, OS X, ..)
+### Download and setup in-toto on \*NIX (Linux, OS X, ..)
 __Virtual Environments (optional)__
 
 We highly recommend to install `in-toto` and its dependencies in a [`virtualenv`](https://virtualenv.pypa.io/en/stable/). Just copy-paste the following snippet to install
 [`virtualenvwrapper`](https://virtualenvwrapper.readthedocs.io/en/latest/) and create a virtual environment:
 
-```shell
+```bash
 # Install virtualenvwrapper
 pip install virtualenvwrapper
 
@@ -35,7 +35,7 @@ mkvirtualenv in-toto-demo
 ```
 
 __Get demo files and install in-toto__
-```shell
+```bash
 # Fetch the demo repo using git
 git clone https://github.com/in-toto/demo.git
 
@@ -51,9 +51,10 @@ all the [dependencies installed on your system](https://github.com/in-toto/in-to
 Inside the demo directory you will find four directories: `owner_alice`,
 `functionary_bob`, `functionary_carl` and `final_product`. Alice, Bob and Carl
 already have RSA keys in each of their directories. This is what you see:
-```shell
+```bash
 tree  # If you don't have tree, try 'find .' instead
 # the tree command gives you the following output
+# .
 # ├── README.md
 # ├── final_product
 # ├── functionary_bob
@@ -63,10 +64,12 @@ tree  # If you don't have tree, try 'find .' instead
 # │   ├── carl
 # │   └── carl.pub
 # ├── owner_alice
-# │   ├── alice
-# │   ├── alice.pub
-# │   └── create_layout.py
-# └── run_demo.py
+# │   ├── alice
+# │   ├── alice.pub
+# │   └── create_layout.py
+# ├── requirements.txt
+# ├── run_demo.py
+# └── run_demo_md.py
 ```
 
 ### Define software supply chain layout (Alice)
@@ -124,9 +127,10 @@ in-toto-record start --step-name update-version --key bob --materials demo-proje
 
 Then Bob uses an editor of his choice to update the version number in `demo-project/foo.py`, e.g.:
 
-```python
-# In demo-project/foo.py
+```shell
+cat <<EOF > demo-project/foo.py
 VERSION = "foo-v1"
+EOF
 ```
 
 And finally he records the state of files after the modification and produces
@@ -198,13 +202,13 @@ malicious code.
 
 ```shell
 cd ../functionary_carl
-echo "something evil" >> demo-project/foo.py
+echo something evil >> demo-project/foo.py
 ```
-Carl thought that this is the sane code he got from Bob and
+Carl thought that this is the genuine code he got from Bob and
 unwittingly packages the tampered version of foo.py
 
 ```shell
-in-toto-run --step-name package --materials demo-project/foo.py --products demo-project.tar.gz --key carl -- tar --exclude '.git' -zcvf demo-project.tar.gz demo-project
+in-toto-run --step-name package --materials demo-project/foo.py --products demo-project.tar.gz --key carl -- tar --exclude ".git" -zcvf demo-project.tar.gz demo-project
 ```
 and ships everything out as final product to the client:
 ```shell
@@ -237,15 +241,15 @@ and how to use it on [in-toto's Github page](https://in-toto.github.io/).
 ### Clean slate
 If you want to run the demo again, you can use the following script to remove all the files you created above.
 
-```shell
+```bash
 cd .. # You have to be the demo directory
 python run_demo.py -c
 ```
 
 ### Tired of copy-pasting commands?
 The same script can be used to sequentially execute all commands listed above. Just change into the `demo` directory, run `python run_demo.py` without flags and observe the output.
 
-```shell
+```bash
 # In the demo directory
 python run_demo.py
-```
+```

requirements.txt

@@ -1 +1 @@
-in-toto==0.2.3
+in-toto==0.3.0

run_demo_md.py

@@ -0,0 +1,123 @@
+"""
+<Program Name>
+  run_demo_md.py
+
+<Author>
+  Lukas Puehringer <[email protected]>
+
+<Started>
+  Jul 17, 2019
+
+<Purpose>
+  Provides a script that extracts the demo code snippets from README.md and
+  runs them in a shell, raising `SystemExit`, if the output is not as expected.
+
+  virtualenv setup and installation of in-toto, as described in the demo
+  instructions, is not performed by this script and must be done before running
+  it. Snippets are run in a temporary directory, which is removed afterwards.
+
+  NOTE: Currently, the script runs all snippets marked as `shell` snippets (see
+  `SNIPPET_PATTERN`). To exclude a snippet from execution it must be marked as
+  something else (e.g. `bash` to get the same syntax highlighting).
+
+"""
+import os
+import re
+import shutil
+import tempfile
+import six
+
+if six.PY2:
+  import subprocess32 as subprocess
+else:
+  import subprocess
+
+# The file pointed to by `INSTRUCTIONS_FN` contains `shell` code snippets that
+# may be extracted using the regex defined in `SNIPPET_PATTERN`, and executed
+# to generate a combined stdout/stderr equal to `EXPECTED_STDOUT`.
+INSTRUCTIONS_FN = "README.md"
+SNIPPET_PATTERN = r"```shell\n([\s\S]*?)\n```"
+
+EXPECTED_STDOUT = \
+"""+ cd owner_alice
++ python create_layout.py
++ cd ../functionary_bob
++ in-toto-run --step-name clone --products demo-project/foo.py --key bob -- git clone https://github.com/in-toto/demo-project.git
++ in-toto-record start --step-name update-version --key bob --materials demo-project/foo.py
++ cat
++ in-toto-record stop --step-name update-version --key bob --products demo-project/foo.py
++ cp -r demo-project ../functionary_carl/
++ cd ../functionary_carl
++ in-toto-run --step-name package --materials demo-project/foo.py --products demo-project.tar.gz --key carl -- tar --exclude .git -zcvf demo-project.tar.gz demo-project
++ cd ..
++ cp owner_alice/root.layout functionary_bob/clone.776a00e2.link functionary_bob/update-version.776a00e2.link functionary_carl/package.2f89b927.link functionary_carl/demo-project.tar.gz final_product/
++ cd final_product
++ cp ../owner_alice/alice.pub .
++ in-toto-verify --layout root.layout --layout-key alice.pub
++ echo 0
+0
++ cd ../functionary_carl
++ echo something evil
++ in-toto-run --step-name package --materials demo-project/foo.py --products demo-project.tar.gz --key carl -- tar --exclude .git -zcvf demo-project.tar.gz demo-project
++ cd ..
++ cp owner_alice/root.layout functionary_bob/clone.776a00e2.link functionary_bob/update-version.776a00e2.link functionary_carl/package.2f89b927.link functionary_carl/demo-project.tar.gz final_product/
++ cd final_product
++ in-toto-verify --layout root.layout --layout-key alice.pub
+(in-toto-verify) RuleVerificationError: 'DISALLOW *' matched the following artifacts: ['demo-project/foo.py']
+Full trace for 'expected_materials' of item 'package':
+Available materials (used for queue):
+['demo-project/foo.py']
+Available products:
+['demo-project.tar.gz']
+Queue after 'MATCH demo-project/* WITH PRODUCTS FROM update-version':
+['demo-project/foo.py']
+
++ echo 1
+1
+"""
+
+# NOTE: Very ugly hack to make this work on Python 2
+if six.PY2:
+  EXPECTED_STDOUT = EXPECTED_STDOUT.replace("['", "[u'")
+
+
+# Setup a test directory with all necessary demo files and change into it. This
+# lets us easily clean up all the files created during the demo eventually.
+demo_dir = os.path.dirname(os.path.realpath(__file__))
+tmp_dir = os.path.realpath(tempfile.mkdtemp())
+test_dir = os.path.join(tmp_dir, os.path.basename(demo_dir))
+shutil.copytree(demo_dir, test_dir)
+os.chdir(test_dir)
+
+# Wrap test code in try/finally to always tear down test directory and files
+try:
+  # Extract all shell code snippets from demo instructions
+  with open(INSTRUCTIONS_FN) as fp:
+    readme = fp.read()
+  snippets = re.findall(SNIPPET_PATTERN, readme)
+
+  # Create script from all snippets, with shell xtrace mode (set -x) for
+  # detailed output and make sure that it has the expected prefix (PS4='+ ')
+  script = "PS4='+ '\nset -x\n{}".format("\n".join(snippets))
+
+  # Execute script in one shell so we can run commands like `cd`
+  # NOTE: Would be nice to use `in_toto.process.run_duplicate_streams` to show
+  # output in real time, but the method does not support the required kwargs.
+  proc = subprocess.Popen(
+      ["/bin/sh", "-c", script],
+      stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+      universal_newlines=True)
+  stdout, _ = proc.communicate()
+
+  # Fail if the output is not what we expected
+  if stdout != EXPECTED_STDOUT:
+    raise SystemExit(
+        "#### EXPECTED:\n-\n{}\n-\n#### GOT:\n-\n{}\n-\nDemo test failed due "
+        "to unexpected output (see above). :(".format(EXPECTED_STDOUT, stdout))
+
+  print("{}\nDemo test ran as expected. :)".format(stdout))
+
+finally:
+  # Change back to where we were in the beginning and tear down test directory
+  os.chdir(demo_dir)
+  shutil.rmtree(test_dir)