fix(envoy): upgrade models in experiment

[sakoush]

What this PR does / why we need it:

This change addresses 2 issues related to Experiments:

• When upgrading to a new version of a model, an experiment route in envoy containing this model would still point to the old (and the new version). This is problematic as we unload the old versions after a new version in made available.
• When upgrading to a new version of model and in an experiment that has a Default route pointing to this model, the old versions of the model are not unloaded. This is incorrect behaviour.

The changes are respectively:

• When we process events to update an experiment due to a model change, we first remove an existing state (which had the old route) before creating the routes for the experiment which updated models etc.
• We always clean old versions of models regardless of how many routes existed for it. This was introduced initially to guard against 404s which is not true anymore as we have a first remove the routes from envoy (UnloadEnvoyRequested) before unloading a model (UnloadRequested).

Which issue(s) this PR fixes:

Fixes # INFRA-1001 (internal)

Special notes for your reviewer:

[lc525]

lgtm;
I haven't done a smoke test in kind but everything looks ok code-wise. For some reason I thought this would require a larger change.

[sakoush]

lgtm; I haven't done a smoke test in kind but everything looks ok code-wise. For some reason I thought this would require a larger change.

I have done a smoke test in kind and added the steps for release testing.