Issue135 forecast uncertainty #482

laura-zabala · 2022-09-05T15:06:55Z

The function for the uncertainty emulator and an example notebook for application are included.

ErrorEmulator.py: function to generate an error with an AR model in the prediction horizon. SimulateError.ipynb: notebook to simulate the error for different uncertainty scenarios.

javiarrobas · 2022-09-13T09:34:39Z

Thanks a lot for the PR, @laura-zabala! I've gone through your additions and they are looking great. As I see it, now we have two main options to move forward:

First, further integrate existing functionality into BOPTEST architecture, then, extend functionality (e.g. for emulating uncertainty of variables other than temperature).
First, extend functionality, then integrate into BOPTEST architecture.

My opinion is that we should go for 1. In that case, the steps I suggest to follow (may be missing some) are:

Import the predict_error method in forecaster.py.
Propagate arguments needed in predict_error to get_forecast. Note that hp can be avoided as an argument since it can be parsed from forecast. We probably want to use default argument values that lead to deterministic forecast, that is: F0=0, K0=0, F=0, K=0, mu=0.
Extend the Forecast and Forecast_Parameters API calls to pass the new arguments through forecast_parameters.
Add tests to test_forecast.py. numpy seeds will need to be set to get consistent testing results. For this I suggest adding at least three tests:
1. test_bare_forecast_uncertainty. Something very similar to what is already implemented in SimulateError.ipynb. This allows getting comprehensive reference results and having a fully independent test. We may want to replicate this test for different uncertainty scenarios.
2., 3. test_get_uncertain_forecast for testcase2 (single-zone) and for testcase3 (multi-zone). These two tests come together because only need to be implemented once in PartialForecasterTest which is extended in both ForecasterSingleZoneTest and ForecasterMultiZoneTest .
Extend functionality for emulating uncertainty of variables other than temperature.
Add documentation.

laura-zabala · 2022-09-16T11:41:28Z

TY for your feedback Javier! I also think it's better to go for option 1. I'll check your suggestions

The predict_error method is imported in forecaster.py.

The arguments required by "predict_error" are propagated in get_forecast: F0, K0, F, K, mu. The hp is already defined (horizon). Default values are included for these parameters for the deterministic scenario.

laura-zabala · 2022-10-11T13:33:38Z

Additional step:

Add the error_forecast to the forecast in the proper way.

dhblum · 2022-10-17T20:36:58Z

Hey all, thanks for the PR and comments already! Just my thoughts here, sorry they're a bit late. But I also vote for 1. However, I might suggest being a little more strategic in preparation for other variables, and in particular other error models. I suggest calling predict_error() maybe predict_error_AR() to be clear there may be other error prediction models later other than just AR.

I'm also thinking the approach of propagating parameters directly to get_forecast will become unsustainable once other error methods are introduced with their own parameters, and some variables we find may warrant the same error methods but different parameters (in which case for instance there would be two sets of the current parameters, one for drybulb temperature and one for, say, humidity if both need AR). This will require more complex architecture/method of relaying what parameters go with which methods go with which variables. Maybe its ok to proceed as now proposed, and cross that refactoring bridge when we get to it as a result of new error methods needing to be introduced to new variables.

predict_error is updated to predict_error_AR to specify the error model

laura-zabala · 2022-10-18T14:39:55Z

I updated the predict_error() to predict_error_AR().
I also agree that once we have to define more parameters, we'll get to too many values to be propagated in get_forecast. Maybe we can continue working with that and have on the "to-do" an alternative way to define it.

dhblum · 2022-10-18T19:49:39Z

@laura-zabala Thanks. Sure, let's keep working this way through all what's needed from top to bottom to implement and keep track of some of these "to-dos."

laura-zabala · 2022-11-02T09:44:39Z

Additional step:

Integrate the seed (from numpy) in the error model to avoid randomness.

…into issue135_forecastUncertainty # Conflicts: # forecast/forecaster.py

…Uncertainty

…nario, and prevent the weather forecast uncertainty from being overwritten in get_forecast after setting it in the scenario.

…ironment

…n and introduced two new tests: test_forecast_temperature_are_within_range and test_forecast_solar_radiation_are_positive for improved criteria checking.

Issue135 forecast uncertainty

laura-zabala · 2024-02-27T09:17:25Z

@wfzheng I reviewed the implementation of the function error_emulator.py and how it is implemented in forecaster.py and I think the implementation is correct. I have the following doubts:

forecaster.py [Line 70] : Why is the generated error subtracted to the total temperature, instead of adding it?
forecaster.py : The irradiation is limited, not just to avoid negative values but also to limit exceeded values from a predefined max and min limits [Lines 94-100]. Should we include something analogous to the temperature?

To-do's from my side:

Review how the seed is integrated.
Review the unit test cases for single-zone and multi-zone.

laura-zabala · 2024-04-24T08:24:09Z

@wfzheng In the forecaster.py file, in the Forecaster class, I miss the description of the parameters: weather_temperature_dry_bulb, weather_solar_global_horizontal, seed, category, plot.

dhblum

Thank you @laura-zabala and @wfzheng for this implementation! I've started reviewing and left some inline comments. Can you start to address? I might have additional comments soon after another review and some thinking more on the structure of the implementation, but these are a start.

dhblum · 2024-05-21T22:08:03Z

！

Should not be committed.

dhblum · 2024-05-21T22:08:49Z

.gitignore

@@ -15,3 +15,7 @@ testcase2/doc/build/
 parser/*.fmu
 parser/*.mo
 parser/*.json
+
+xx.ipynb
+connect_boptest.py


I don't see a connect_boptest.py and don't think this should be written in the .gitignore.

dhblum · 2024-05-21T22:09:05Z

.gitignore

+
+xx.ipynb
+connect_boptest.py
+\!


This shouldn't be in the .gitignore.

dhblum · 2024-05-21T22:09:18Z

.gitignore

@@ -15,3 +15,7 @@ testcase2/doc/build/
 parser/*.fmu
 parser/*.mo
 parser/*.json
+
+xx.ipynb


I don't see why this should be in the .gitignore.

dhblum · 2024-05-21T22:11:18Z

forecast/forecaster.py

@@ -30,45 +33,72 @@ def __init__(self, testcase):
        # Point to the test case object
        self.case = testcase

-    def get_forecast(self,point_names, horizon=24*3600, interval=3600,
+    def get_forecast(self, point_names, horizon=24 * 3600, interval=3600,
+                     weather_temperature_dry_bulb=None, weather_solar_global_horizontal=None, seed=None,


I suggest shorter variable names, using three letters, such as: wea_tem_dry_bul instead of the full words as you have.

Also missing documentation of these parameters, which should be included as docstrings as in other functions, and was done previously for this one.

Thank you for your suggestions regarding the variable names, the documentation and the issue of '.gitignore' file. I have updated the variable names to shorter versions as you suggested, updated the docstrings and corrected the '.gitignore'.

I've created a pull request to merge these modifications into the issue135_forecastUncertainty branch of the forked repository at laura-zabala/project1-boptest.

dhblum · 2024-05-21T22:32:32Z

forecast/forecaster.py

+            indices = np.where(original_HGloHor > 50)[0]
+
+
+            for i in range(200):


Is it ever the case 200 is not enough? What happens if condition not met after 200?

The limit of 200 iterations is set to balance between accuracy and computational efficiency. Increasing the number of iterations could slow down the simulation.

If the condition is not met after 200 iterations, the code still ensures that the forecast values remain within the specified range by applying the np.clip function. [Line 99]

wfzheng · 2024-05-23T06:14:03Z

@laura-zabala
> * forecaster.py [Line 70] : Why is the generated error subtracted to the total temperature, instead of adding it?
Answer: In the previous code found in the notebook UNCERTAINTY_EMULATOR_AUTOREG.ipynb , the errors are calculated using the formula errors = (y.iloc[:, 1:] - y_hat.iloc[:, 1:]). This computation derives the errors by subtracting the predicted values (y_hat) from the actual values (y). Therefore, to generate the forecast (forecast = y_true - errors), we subtract these errors from the true values (y_true).

> * forecaster.py : The irradiation is limited, not just to avoid negative values but also to limit exceeded values from a predefined max and min limits [Lines 94-100]. Should we include something analogous to the temperature?
Answer: I think the current implementation, where we limit the maximum forecasted irradiation (forecast['HGloHor']) to twice the true values, is practical and should generally cover the expected variability in the data without introducing unrealistic predictions. However I'd appreciate your insights, @dhblum, on whether you think this approach might lead to any potential issues or if there's a better way to handle extremes in our forecast model. Looking forward to your feedback!

…ertainty

…o issue135_forecastUncertainty merge laura's revise

dhblum · 2024-07-30T18:25:21Z

Hey @laura-zabala, can you:

update your laura-zabala:issue135_forecastUncertainty branch with the latest ibpsa:issue135_forecastUncertainty
merge PR Issue135 forecast uncertainty laura-zabala/project1-boptest#3 which Wanfu I think uses to address my comments.

Then I think I'll be able to review things again more easily within this PR.

Issue135 forecast uncertainty

…ertainty

Issue135 forecast uncertainty

laura-zabala · 2024-08-27T14:33:29Z

Hi,
@dhblum I think it's ready now, sorry for the late response

dhblum · 2024-10-18T18:25:26Z

@laura-zabala I'm closing this PR in favor of #693 since for some reason this one seems to leave out recent changes by @wfzheng.

laura-zabala and others added 2 commits August 19, 2022 12:10

Uncertainty generator function included.

eab2f75

ErrorEmulator.py: function to generate an error with an AR model in the prediction horizon. SimulateError.ipynb: notebook to simulate the error for different uncertainty scenarios.

Uncertainty emulator function included

79cbeb2

laura.zabala added 2 commits October 11, 2022 15:13

predict_error method is imported in forcaster.

0dc3f89

The predict_error method is imported in forecaster.py.

Required arguments by predict_error are propagated to get_forecast

f4093d1

The arguments required by "predict_error" are propagated in get_forecast: F0, K0, F, K, mu. The hp is already defined (horizon). Default values are included for these parameters for the deterministic scenario.

uncertainty error added juts to dry bult temperature

407bc79

AR speficied in the function name

cac66b6

predict_error is updated to predict_error_AR to specify the error model

Formatting correction in the error_emulator function

e66ea46

wfzheng and others added 12 commits November 19, 2023 23:19

Merge remote-tracking branch 'upstream/issue135_forecastUncertainty' …

c74022e

…into issue135_forecastUncertainty # Conflicts: # forecast/forecaster.py

Merge remote-tracking branch 'upstream/master' into issue135_forecast…

9d33819

…Uncertainty

Add forecast uncertainty function

816358c

Add forecast uncertainty function

edeaf1c

Add the function of selecting weather forecast uncertainty in the sce…

0267d23

…nario, and prevent the weather forecast uncertainty from being overwritten in get_forecast after setting it in the scenario.

add forecast uncertainty test for both single zone and multi zone env…

b5db17d

…ironment

add forecast uncertainty test for both single zone and multi zone env…

2597023

…ironment

add forecast uncertainty test for both single zone and multi zone env…

b6fb4a7

…ironment

Remove file with special character from staging area

bc6f233

add forecast uncertainty test for both single zone and multi zone env…

5eebe2b

…ironment

Enhanced testcase.py by adding seed parameter to set_scenario functio…

32f727f

…n and introduced two new tests: test_forecast_temperature_are_within_range and test_forecast_solar_radiation_are_positive for improved criteria checking.

Merge pull request #2 from wfzheng/issue135_forecastUncertainty

a433f7c

Issue135 forecast uncertainty

Description of the functions to generate errors for the forecast.

c29d489

dhblum reviewed May 21, 2024

View reviewed changes

wfzheng added 8 commits May 23, 2024 14:41

Merge remote-tracking branch 'laura/master' into issue135_forecastUnc…

a7560fb

…ertainty

Merge remote-tracking branch 'laura/issue135_forecastUncertainty' int…

d81a697

…o issue135_forecastUncertainty merge laura's revise

Add missing parameter descriptions to Forecaster class

5c200c8

Add missing parameter descriptions to Forecaster class

7fb0965

Add missing parameter descriptions to Forecaster class

3ffc633

Add missing parameter descriptions to Forecaster class

b1f38f1

Shorten variable names for better readability in get_forecast

8c29dbf

revise .gitignore

f503671

laura-zabala added 3 commits August 27, 2024 16:28

Merge pull request #4 from ibpsa/issue135_forecastUncertainty

a4bc294

Issue135 forecast uncertainty

Merge branch 'issue135_forecastUncertainty' into issue135_forecastUnc…

7e001b4

…ertainty

Merge pull request #3 from wfzheng/issue135_forecastUncertainty

e229f4d

Issue135 forecast uncertainty

dhblum mentioned this pull request Oct 18, 2024

Issue135 forecast uncertainty 2 #693

Open

dhblum closed this Oct 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue135 forecast uncertainty #482

Issue135 forecast uncertainty #482

laura-zabala commented Sep 5, 2022

javiarrobas commented Sep 13, 2022

laura-zabala commented Sep 16, 2022

laura-zabala commented Oct 11, 2022 •

edited

Loading

dhblum commented Oct 17, 2022

laura-zabala commented Oct 18, 2022

dhblum commented Oct 18, 2022

laura-zabala commented Nov 2, 2022 •

edited

Loading

laura-zabala commented Feb 27, 2024

laura-zabala commented Apr 24, 2024

dhblum left a comment

dhblum May 21, 2024

dhblum May 21, 2024

dhblum May 21, 2024

dhblum May 21, 2024

dhblum May 21, 2024

wfzheng May 23, 2024

dhblum May 21, 2024

wfzheng May 23, 2024

wfzheng commented May 23, 2024 •

edited

Loading

dhblum commented Jul 30, 2024

laura-zabala commented Aug 27, 2024 •

edited

Loading

dhblum commented Oct 18, 2024

		indices = np.where(original_HGloHor > 50)[0]


		for i in range(200):

Issue135 forecast uncertainty #482

Issue135 forecast uncertainty #482

Conversation

laura-zabala commented Sep 5, 2022

javiarrobas commented Sep 13, 2022

laura-zabala commented Sep 16, 2022

laura-zabala commented Oct 11, 2022 • edited Loading

dhblum commented Oct 17, 2022

laura-zabala commented Oct 18, 2022

dhblum commented Oct 18, 2022

laura-zabala commented Nov 2, 2022 • edited Loading

laura-zabala commented Feb 27, 2024

laura-zabala commented Apr 24, 2024

dhblum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wfzheng commented May 23, 2024 • edited Loading

dhblum commented Jul 30, 2024

laura-zabala commented Aug 27, 2024 • edited Loading

dhblum commented Oct 18, 2024

laura-zabala commented Oct 11, 2022 •

edited

Loading

laura-zabala commented Nov 2, 2022 •

edited

Loading

wfzheng commented May 23, 2024 •

edited

Loading

laura-zabala commented Aug 27, 2024 •

edited

Loading