Merge pull request #455 from ibpsa/issue448-service-submit

Issue448 service submit

.env

@@ -11,3 +11,4 @@ MONGO_URL=mongodb://mongo:27017/
 MONGO_DB_NAME=alfalfa
 REGION=us-west-1
 BOPTEST_SUBMIT_URL=http://web
+BOPTEST_DASHBOARD_SERVER=https://dashboard.boptest.net/

.pre-commit-config.yaml

@@ -7,7 +7,7 @@
 #  )
 
 repos:
-  - repo: git://github.com/pre-commit/pre-commit-hooks
+  - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v2.2.3
     hooks:
       - id: trailing-whitespace

.travis.yml

@@ -25,4 +25,46 @@ jobs:
   include:
   - python: 3.9
     install: pip install --upgrade pip && pip install pandas==1.2.5 numpy==1.20.2 matplotlib==3.3.4 requests==2.25.1
-    script: cd testing && make test_all
+    script: cd testing && make build_jm_image && make build_boptest_image && make test_parser
+  - python: 3.9
+    install: pip install --upgrade pip && pip install pandas==1.2.5 numpy==1.20.2 matplotlib==3.3.4 requests==2.25.1
+    script: cd testing && make build_jm_image && make build_boptest_image && make test_data
+  - python: 3.9
+    install: pip install --upgrade pip && pip install pandas==1.2.5 numpy==1.20.2 matplotlib==3.3.4 requests==2.25.1
+    script: cd testing && make build_jm_image && make build_boptest_image && make test_forecast
+  - python: 3.9
+    install: pip install --upgrade pip && pip install pandas==1.2.5 numpy==1.20.2 matplotlib==3.3.4 requests==2.25.1
+    script: cd testing && make build_jm_image && make build_boptest_image && make test_kpis
+#  - python: 3.9
+#    install: pip install --upgrade pip && pip install pandas==1.2.5 numpy==1.20.2 matplotlib==3.3.4 requests==2.25.1
+#    script: cd testing && make build_jm_image && make build_boptest_image && make test_readme_commands
+  - python: 3.9
+    install: pip install --upgrade pip && pip install pandas==1.2.5 numpy==1.20.2 matplotlib==3.3.4 requests==2.25.1
+    script: cd testing && make build_jm_image && make build_boptest_image && make test_testcase1
+  - python: 3.9
+    install: pip install --upgrade pip && pip install pandas==1.2.5 numpy==1.20.2 matplotlib==3.3.4 requests==2.25.1
+    script: cd testing && make build_jm_image && make build_boptest_image && make test_testcase2
+  - python: 3.9
+    install: pip install --upgrade pip && pip install pandas==1.2.5 numpy==1.20.2 matplotlib==3.3.4 requests==2.25.1
+    script: cd testing && make build_jm_image && make build_boptest_image && make test_testcase3
+  - python: 3.9
+    install: pip install --upgrade pip && pip install pandas==1.2.5 numpy==1.20.2 matplotlib==3.3.4 requests==2.25.1
+    script: cd testing && make build_jm_image && make build_boptest_image && make test_bestest_air
+  - python: 3.9
+    install: pip install --upgrade pip && pip install pandas==1.2.5 numpy==1.20.2 matplotlib==3.3.4 requests==2.25.1
+    script: cd testing && make build_jm_image && make build_boptest_image && make test_bestest_hydronic
+  - python: 3.9
+    install: pip install --upgrade pip && pip install pandas==1.2.5 numpy==1.20.2 matplotlib==3.3.4 requests==2.25.1
+    script: cd testing && make build_jm_image && make build_boptest_image && make test_bestest_hydronic_heat_pump
+  - python: 3.9
+    install: pip install --upgrade pip && pip install pandas==1.2.5 numpy==1.20.2 matplotlib==3.3.4 requests==2.25.1
+    script: cd testing && make build_jm_image && make build_boptest_image && make test_multizone_residential_hydronic
+  - python: 3.9
+    install: pip install --upgrade pip && pip install pandas==1.2.5 numpy==1.20.2 matplotlib==3.3.4 requests==2.25.1
+    script: cd testing && make build_jm_image && make build_boptest_image && make test_singlezone_commercial_hydronic
+  - python: 3.9
+    install: pip install --upgrade pip && pip install pandas==1.2.5 numpy==1.20.2 matplotlib==3.3.4 requests==2.25.1
+    script: cd testing && make build_jm_image && make build_boptest_image && make test_multizone_office_simple_air
+#  - python: 2.7
+#    install: pip install --upgrade pip && pip install pandas==0.24.2 numpy==1.16.6 matplotlib==2.1.1 requests==2.18.4
+#    script: cd testing && make test_python2

Dockerfile

@@ -12,16 +12,23 @@ USER root
 # Edit pyfmi to event update at start of simulation for ME2
 RUN sed -i "350 i \\\n        if isinstance(self.model, fmi.FMUModelME2):\n            self.model.event_update()" $JMODELICA_HOME/Python/pyfmi/fmi_algorithm_drivers.py
 
+# Install commands for Spawn
+ENV SPAWN_VERSION=0.3.0-8d93151657
+RUN wget https://spawn.s3.amazonaws.com/custom/Spawn-$SPAWN_VERSION-Linux.tar.gz \
+    && tar -xzf Spawn-$SPAWN_VERSION-Linux.tar.gz \
+    && ln -s /Spawn-$SPAWN_VERSION-Linux/bin/spawn-$SPAWN_VERSION /usr/local/bin/
+
 USER developer
 
 WORKDIR $HOME
 
-RUN pip install --user flask-restful==0.3.9 pandas==0.24.2 flask_cors==3.0.10
+RUN pip install --user flask-restful==0.3.9 pandas==0.24.2 flask_cors==3.0.10 requests==2.27.1
 
 RUN mkdir models && \
     mkdir doc
 
 ENV PYTHONPATH $PYTHONPATH:$HOME
+ENV BOPTEST_DASHBOARD_SERVER https://api.boptest.net:8081/
 
 CMD python restapi.py && bash
 

README.md

@@ -5,7 +5,9 @@
 Building Optimization Performance Tests
 
 This repository contains code for the Building Optimization Performance Test framework (BOPTEST)
-that is being developed as part of the IBPSA Project 1 (https://ibpsa.github.io/project1/).
+that is being developed as part of the [IBPSA Project 1](https://ibpsa.github.io/project1/).
+
+Visit the [BOPTEST Home Page](https://ibpsa.github.io/project1-boptest/) for more information about the project, software, and documentation.
 
 ## Structure
 - ``/testcases`` contains test cases, including docs, models, and configuration settings.
@@ -17,13 +19,21 @@ that is being developed as part of the IBPSA Project 1 (https://ibpsa.github.io/
 - ``/kpis`` contains code for calculating key performance indicators.
 - ``/docs`` contains design requirements and guide documentation.
 
-##2) Build the test case by ``$ make build`` where <testcase_dir_name> is the name of the test case subdirectory located in ``/testcases``.
 ## Quick-Start to Run Test Cases
-1) Install [Docker](https://docs.docker.com/get-docker/).
-2) Build the BOPTEST Service by ``$ make build``.
-3) Start the BOPTEST Service by ``$ make run``.
-4) In a separate process, use the test case API defined below to interact with a test case using your test controller.  Alternatively, view and run an example test controller as described in the next step.
-5) Run an example test controller:
+1) Download this repository.
+2) Install [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/).
+3) Build and deploy a test case using the following commands executed in the root directory of this repository and where <testcase_dir_name> is the name of the test case subdirectory located in [/testcases](https://github.com/ibpsa/project1-boptest/tree/master/testcases):
+
+  * ``docker-compose up``
+  * A couple notes:
+    * The first time this command is run, the image ``boptest_base`` will be built.  This takes about a minute.  Subsequent usage will use the already-built image and deploy much faster.
+    * If you update your BOPTEST repository, use the command ``docker rmi boptest_base`` to remove the image so it can be re-built with the updated repository upon next deployment.
+    * ``TESTCASE`` is simply an environment variable.  Consistent with use of docker-compose, you may also edit the value of this variable in the ``.env`` file and then use ``docker-compose up``.
+
+4) In a separate process, use the test case API defined below to interact with the test case using your test controller.  Alternatively, view and run an example test controller as described below.
+5) Shutdown the test case by the command ``docker-compose down`` executed in the root directory of this repository
+
+## Run an example test controller:
 
 * For Python-based example controllers:
   * Build and run the BOPTEST Service.  Then, in a separate terminal, use ``$ cd examples/python/ && python testcase1.py`` to test a simple proportional feedback controller on this test case over a two-day period.
@@ -38,6 +48,7 @@ that is being developed as part of the IBPSA Project 1 (https://ibpsa.github.io/
 ## Test Case RESTful API
 - To run a test, use the API defined in the table below by sending RESTful requests to: ``http://127.0.0.1/<request>/<testid>``
 - In BOPTEST Service there can be many tests running concurently. The ``select`` API is used to begin a test and retrieve a unique ``testid`` which is used for further API requests related to the test: ``curl -X POST  http://127.0.0.1/testcases/testcase1/select``. The value ``testcase1`` can be any one of the available BOPTEST test cases reported by the API: GET ``http://127.0.0.1/testcases/``.
+- The API will return a JSON in the form ``{"status":<status_code_int>, "message":<message_str>, "payload":<relevant_return_data>}``. Status codes in ``"status"`` are integers: ``200`` for successful with or without warning, ``400`` for bad input error, or ``500`` for internal error.  Data returned in ``"payload"`` is the data of interest relvant to the specific API request, while the string in ``"message"`` will report any warnings or error messages to help debug encountered problems.
 
 Example RESTful interaction:
 
@@ -50,23 +61,25 @@ Example RESTful interaction:
 | Interaction                                                           | Request                                                   |
 |-----------------------------------------------------------------------|-----------------------------------------------------------|
 | List available test cases.                                             |  GET ``testcases`` |
-| Select a test case and begin a new test.                               |  POST ``testcases/{testcase_name}/select`` |
+| Select a test case and begin a new test. A unique ``testid`` will be returned.                                |  POST ``testcases/{testcase_name}/select`` |
 | Stop a running test.                                                   |  PUT ``stop/{testid}`` |
-| Advance simulation with control input and receive measurements.        |  POST ``advance/{testid}`` with json data "{<input_name>:<value>}" |
-| Initialize simulation to a start time using a warmup period in seconds.     |  PUT ``initialize/{testid}`` with arguments ``start_time=<value>``, ``warmup_time=<value>``|
+| Advance simulation with control input and receive measurements.        |  POST ``advance/{testid}`` with optional json data "{<input_name>:<value>}" |
+| Initialize simulation to a start time using a warmup period in seconds.  Also resets point data history and KPI calculations.     |  PUT ``initialize/{testid}`` with required arguments ``start_time=<value>``, ``warmup_period=<value>``|
 | Receive communication step in seconds.                                 |  GET ``step/{testid}``                                             |
-| Set communication step in seconds.                                     |  PUT ``step/{testid}`` with argument ``step=<value>``              |
+| Set communication step in seconds.                                     |  PUT ``step/{testid}`` with required argument ``step=<value>``              |
 | Receive sensor signal point names (y) and metadata.                          |  GET ``measurements/{testid}``                                     |
 | Receive control signal point names (u) and metadata.                        |  GET ``inputs/{testid}``                                           |
-| Receive test result data for the given point name between the start and final time in seconds. |  PUT ``results/{testid}`` with arguments ``point_name=<string>``, ``start_time=<value>``, ``final_time=<value>``|
+| Receive test result data for the given point name between the start and final time in seconds. |  PUT ``results/{testid}`` with required arguments ``point_name=<string>``, ``start_time=<value>``, ``final_time=<value>``|
 | Receive test KPIs.                                                     |  GET ``kpi/{testid}``                                              |
 | Receive test case name.                                                |  GET ``name/{testid}``                                             |
 | Receive boundary condition forecast from current communication step.   |  GET ``forecast/{testid}``                                         |
 | Receive boundary condition forecast parameters in seconds.             |  GET ``forecast_parameters/{testid}``                              |
-| Set boundary condition forecast parameters in seconds.                 |  PUT ``forecast_parameters/{testid}`` with arguments ``horizon=<value>``, ``interval=<value>``|
+| Set boundary condition forecast parameters in seconds.                 |  PUT ``forecast_parameters/{testid}`` with required arguments ``horizon=<value>``, ``interval=<value>``|
 | Receive current test scenario.                                         |  GET ``scenario/{testid}``                                   |
 | Set test scenario. Setting the argument ``time_period`` performs an initialization with predefined start time and warmup period and will only simulate for predefined duration. |  PUT ``scenario/{testid}`` with optional arguments ``electricity_price=<string>``, ``time_period=<string>``.  See README in [/testcases](https://github.com/ibpsa/project1-boptest/tree/master/testcases) for options and test case documentation for details.|
 | Receive BOPTEST version.                                               |  GET ``version``                                             |
+| Submit KPIs, other test information, and optional string tags (up to 10) to online dashboard.  Requires a formal test scenario to be completed, initialized using the PUT ``scenario`` API. |  POST ``submit`` with required argument ``api_key=<string>`` and optional arguments ``tag#=<string>`` where # is an integer between 1 and 10.  The API key can be obtained from the user account registered wth the online dashboard at (url coming soon).|
+
 ## Development
 
 This repository uses pre-commit to ensure that the files meet standard formatting conventions (such as line spacing, layout, etc).
@@ -82,4 +95,26 @@ See the [wiki](https://github.com/ibpsa/project1-boptest/wiki) for use cases and
 A dashboard for aggregating and sharing test results is under development here: https://github.com/NREL/boptest-dashboard.
 
 ## Publications
-D. Blum, F. Jorissen, S. Huang, Y. Chen, J. Arroyo, K. Benne, Y. Li, V. Gavan, L. Rivalin, L. Helsen, D. Vrabie, M. Wetter, and M. Sofos. (2019). “Prototyping the BOPTEST framework for simulation-based testing of advanced control strategies in buildings.” In *Proc. of the 16th International Conference of IBPSA*, Sep 2 – 4. Rome, Italy.
+### To cite, please use:
+D. Blum, J. Arroyo, S. Huang, J. Drgona, F. Jorissen, H.T. Walnum, Y. Chen, K. Benne, D. Vrabie, M. Wetter, and L. Helsen. (2021). ["Building optimization testing framework (BOPTEST) for simulation-based benchmarking of control strategies in buildings."](https://doi.org/10.1080/19401493.2021.1986574) *Journal of Building Performance Simulation*, 14(5), 586-610.
+
+### Additional publications:
+J. Arroyo, F. Spiessens, and L. Helsen. (2022). ["Comparison of Optimal Control Techniques for Building Energy Management."](https://doi.org/10.3389/fbuil.2022.849754) *Frontiers in Built Environment* 8.
+
+T. Marzullo, S. Dey, N. Long, J. L. Vilaplana, and G. Henze. (2022). ["A high-fidelity building performance simulation test bed for the development and evaluation of advanced controls"](https://doi.org/10.1080/19401493.2022.2058091) *Journal of Building Performance Simulation*, 15(3), 379-397.
+
+J. Arroyo, C. Manna, F. Spiessens, and L. Helsen. (2022). ["Reinforced model predictive control (RL-MPC) for building energy management."](https://doi.org/10.1016/j.apenergy.2021.118346) *Applied Energy* 309: 118346.
+
+J. Arroyo, C. Manna, F. Spiessens, and L. Helsen. (2021). [“An OpenAI-Gym Environment for the Building Optimization Testing (BOPTEST) Framework.”](https://www.researchgate.net/profile/Javier-Arroyo/publication/354386346_An_OpenAI-Gym_environment_for_the_Building_Optimization_Testing_BOPTEST_framework/links/613616690360302a0082ffc1/An-OpenAI-Gym-environment-for-the-Building-Optimization-Testing-BOPTEST-framework.pdf) In *Proceedings of the 17th IBPSA Conference*, Sep 1 - 3. Bruges, Belgium.
+
+F. Bünning, C. Pfister, A. Aboudonia, P. Heer, and J. Lygeros. (2021). [“Comparing Machine Learning Based Methods to Standard Regression Methods for MPC on a Virtual Testbed.”](https://www.research-collection.ethz.ch/bitstream/handle/20.500.11850/524933/BS2021_finalVersion.pdf?sequence=1&isAllowed=y) In *Proceedings of the 17th IBPSA Conference*, Sep 1 - 3. Bruges, Belgium.
+
+T. Yang, K. Filonenko, K. Arendt, and C. Veje. (2020). [“Implementation and Performance Analysis of a Multi-Energy Building Emulator.”](https://ieeexplore.ieee.org/document/9236623) In *2020 6th IEEE International Energy Conference (ENERGYCon)*, Sep 28 - Oct 1. Gammarth, Tunisia, 451–456.
+
+H. T. Walnum, I. Sartori, and M. Bagle. (2020). [“Model Predictive Control of District Heating Substations for Flexible Heating of Buildings.”](https://sintef.brage.unit.no/sintef-xmlui/handle/11250/2683181) In *SINTEF Proceedings no 5, ser. BuildSim-Nordic 2020*, Oct 13–14. Oslo, Norway: International Conference Organised by IBPSA-Nordic, 123–130.
+
+J. Arroyo, F. Spiessens, and L. Helsen. (2020). [“Identification of Multi-zone Grey-box Building Models for Use in Model Predictive Control.”](https://doi.org/10.1080/19401493.2020.1770861) *Journal of Building Performance Simulation* 13 (4): 472–486.
+
+D. Blum, F. Jorissen, S. Huang, Y. Chen, J. Arroyo, K. Benne, Y. Li, V. Gavan, L. Rivalin, L. Helsen, D. Vrabie, M. Wetter, and M. Sofos. (2019). [“Prototyping the BOPTEST framework for simulation-based testing of advanced control strategies in buildings.”](http://www.ibpsa.org/proceedings/BS2019/BS2019_211276.pdf) In *Proceedings of the 16th International Conference of IBPSA*, Sep 2 – 4. Rome, Italy.
+
+S. Huang, Y. Chen, P. W. Ehrlich, and D. L. Vrabie. (2018). [“A Control-Oriented Building Envelope and HVAC System Simulation Model for a Typical Large Office Building.”](https://www.ashrae.org/File%20Library/Conferences/Specialty%20Conferences/2018%20Building%20Performance%20Analysis%20Conference%20and%20SimBuild/Papers/C101.pdf) In *Proceedings of 2018 Building Performance Modeling Conference and SimBuild co-organized by ASHRAE and IBPSA-USA*, Sep 26 - 28. Chicago, IL.

data/data_manager.py

@@ -389,7 +389,13 @@ def load_data_and_jsons(self):
         files = []
         for f in z_fmu.namelist():
             if f.startswith('resources/') and f.endswith('.csv'):
-                files.append(f)
+                if 'resource_file_exclusion' in self.case.config_json:
+                    if f.split('/')[-1] in self.case.config_json['resource_file_exclusion']:
+                        print('{0} found on"resource_file_exclusion" list.  Not loading data into testcase.'.format(f))
+                    else:
+                        files.append(f)
+                else:
+                    files.append(f)
 
         # Find the minimum sampling resolution
         sampling = 3600.
@@ -425,7 +431,10 @@ def load_data_and_jsons(self):
                                        'you want this variable to be part of the test case data '\
                                        'make sure that the column has a key with any of the specified '\
                                        'formats in categories.json and that, if it is a zone related '\
-                                       'key, it is in the format: variable[zone_identifier] '.format(col,f))
+                                       'key, it is in the format: variable[zone_identifier].  If you '\
+                                       'do not intend for this variable to be included, consider adding '\
+                                       'the file it is contained within to the "resource_file_exclusion" '\
+                                       'list in the test case config.json.'.format(col,f))
                     else:
                         for category in self.categories:
                             # Use linear interpolation for continuous variables