Fix bound calculation with multiple Markov states in first stage

[odow]

As reported by @rodripor, the bound calculation did not account for multiple Markov states in the first stage.

There is an unresolved question as to what the bound should be when there are multiple Markov states (or stagewise-independent noise terms for that matter). Should it just be the expectation of all of the possible first stage realizations? Or some risk-adjusted value? Even though we do not explicitly optimize the risk-adjusted value, it should attain the optimal solution from monotonicity.

Closes #120

[odow]

Pinging @rodripor, are you using Expectation() as your risk measure? What do you expect the bound to be if there are multiple Markov states in the first stage?

[aferragu]

Let me pitch in here Oscar (I'm @rodripor advisor). We are using Expectation() as risk measure. However, seems to me that it doesn't make sense to have multiple Markov states for t=0 since the cost-to-go is a function of the state, both its continuous part and its Markov part. Therefore, the bound should be for a given initial Markov state and continuous states.

There is some asymmetry in your definition of continuous and Markov states. Whenever you define a state variable with the @state macro, you give an initial condition (the third argument). However, for the Markov transitions you expect to have a probability fixed to 1 to one of them, and then a vector for the first jumps, and finally the transition matrices. Maybe a better way to go is to define the transition matrices, and have a Markov state constructor where you can specify the initial condition. I'm not sure if your code can be refactored to perform this, due to the way you define Markov transitions.

Regarding your proposed fix, I don't understand this line:
m.storage.probability[i] *= transition_matrix[1, markov_state]
unless m.storage.probability[i] is initialized to 1. However, I don't know if you need this fix or impose an initial Markov state instead.

[odow]

Maybe a better way to go is to define the transition matrices, and have a Markov state constructor where you can specify the initial condition. I'm not sure if your code can be refactored to perform this, due to the way you define Markov transitions.

In Chapter 1 of my thesis, I introduce the idea of a policy graph as a way of structuring stochastic optimization problems. Basically, instead of thinking about stages and Markov states, we should think of nodes in some (potentially cyclic) graph.

This leads to two possibilities:

1. In my thesis I define a root node. When viewed this way, it is fine to have multiple Markov states in the first "stage". Effectively you find a decision rule for each node a priori.
2. However, I have been thinking about changing this, and defining the initial conditions as a (node, noise realization, initial state) triplet. With this definition, the first "stage" problem is deterministic, and all the uncertainty happens in future stages.

Option (1) more of less corresponds to what we have at the moment. Option (2) more of less corresponds to what you are proposing.

Refactoring into policy graphs is a major re-write, so I have been putting it off. Introducing a parameter for the initial Markov state should be relatively straight forward.

Regarding your proposed fix, I don't understand this line:
m.storage.probability[i] *= transition_matrix[1, markov_state]

This is a little sleight of hand. The code isn't clear and needs to be re-written. After solving all of the problems in a stage, m.storage.probability ends up containing the probabilities of the noise outcomes ignoring the Markov transition. So for two Markov states, each with two noise realizations, m.storage.probability = [0.5, 0.5, 0.5, 0.5]. Therefore, we need to weight by the Markov transitions.

[odow]

I'm merging this as-is. I'm about to embark on a re-write that will address a lot of these questions.