---
jupytext:
  text_representation:
    extension: .md
    format_name: myst
    format_version: 0.13
    jupytext_version: 1.13.8
kernelspec:
  display_name: Python 3
  language: python
  name: python3
---

+++ {"Collapsed": "false"}

(cut-description)=
# Cut descriptions

The model setup for the two cut description examples are available in the following formats.

- pyshop
    - [](cuts.py)
    - [](rhs_file.txt)
    - [](wv_file.txt)
- YAML
    - [](cut-model.yaml)
    - [](standard_cuts.yaml)
    - [](price_dep_cuts.yaml)
- ASCII
    - [](cut-model.ascii)
    - [](name_coupling.txt)
    - [](cut_order.txt)
    - [](standard_cuts.txt)    
    - [](price_dep_cuts.txt)        

+++ {"Collapsed": "false"}

## Standard cut description
Adding a standard [cut description](water-value-descriptions) to a simple two-reservoir system is show in this example. The cuts used in the example were generated by the prototype cut creation functionality in SHOP, and is therefore not saved to the standard text file format used by ProdRisk and EOPS.

```{code-cell} ipython3
:Collapsed: 'false'

#Necessary imports used in all examples
import pandas as pd
from pyshop import ShopSession
import plotly.graph_objs as go
pd.options.plotting.backend = "plotly"

#Functions used in this example for building and solving a simple model with cuts
from cuts import build_model, run_model, read_cuts, get_minimal_cut_surface
```

```{code-cell} ipython3
:Collapsed: 'false'

#Create a standard ShopSession
shop=ShopSession()
#Build a simple model with two reservoirs and two plants.
build_model(shop)
#Display topology to the screen
display(shop.model.build_connection_tree())
```

+++ {"Collapsed": "false"}

After the topology has been created, we read in the cuts from the text files. A new [](cut_group) object must be created to hold the right-hand side constants of the cut constraints in the [rhs](cut_group:rhs) attribute. The [water_value_input](reservoir:water_value_input) attribute on the [](reservoir) objects are used to store the cut coefficients with corresponding reference volumes. All reservoirs that are part of the cut constraints must be connected to the cut_group.

```{code-cell} ipython3
:Collapsed: 'false'

#Read the cut info
rhs_in,cut_coeffs = read_cuts()
n_cuts = len(rhs_in)

#Create a cut group object and add the RHS
my_cuts = shop.model.cut_group.add_object("my_cuts")
my_cuts.rhs.set([pd.Series(rhs_in, index=[0]*n_cuts, name=0)])

#Add the cut coefficients to the reservoirs and connect them to the cut_group
for rsv in shop.model.reservoir:
    name = rsv.get_name()
    
    v_ref = cut_coeffs[name][0]
    wv = cut_coeffs[name][1]
    
    rsv.water_value_input.set([pd.Series(wv, index=v_ref, name=0)])
    rsv.connect_to(my_cuts)    
```

+++ {"Collapsed": "false"}

The inequality constraints defined by each cut $k$ is

$c \leq \sum_i WV_{ik}\cdot(v_i - V_i^{ref}) + rhs_k \qquad \forall k,$

when formulated as profit maximization. The single cut variable $c$ is bounded from above by all the cuts, and the optimization problem tries to maximize its value. The cut coefficients $WV_{ik}$ for reservoir $i$ is multiplied with the end volume variable $v_i$ subtracted the cut reference volume. When all $v_i = V_i^{ref}$, the cut is bounded by the right hand side value $rhs_k$.

The cut constraints can be visualized as a 3D plot since there are only two reservoirs in this example. The x and y axis correspond to the end volume of each reservoir, while the z axis shows the cut value in terms of future expected income. Each cut is a plane in 3D which is spanned around a reference point with the reference volumes as x and y coordinates and the rhs as the z value. These points are plotted in 3D scatter plot below. The minimal surface spanned by all these cuts are also plotted, which is the effective bound that the optimization problem will see. The surface is found by taking the lowest (most restrictive) cut value over the set of cuts for different values of the reservoir volumes. 

Note that the reference points in red are marked as "never binding" and hover above the calculated minimal surface. This should not happen, as a cut should at least be binding exactly at the reference point. Non-convexities in the optimization program used to create these sample cuts are to blame in this case, as the SHOP cut creation functionality is still experimental. Some of the bidning cuts are also not binding at the reference point but somewhere else in the domain, which is also inaccurate.

```{code-cell} ipython3
:Collapsed: 'false'

import numpy as np
rsv1 = shop.model.reservoir.Reservoir1
rsv2 = shop.model.reservoir.Reservoir2

vmax1 = rsv1.max_vol.get()
vmax2 = rsv2.max_vol.get()

v_ref1 = np.array(cut_coeffs["Reservoir1"][0])
v_ref2 = np.array(cut_coeffs["Reservoir2"][0])

v1 = np.arange(0,vmax1,0.25)
v2 = np.arange(0,vmax2,0.25)

#Add the reference volumes to the volume ranges to have a good representation for the surface around the reference points
v1 = np.concatenate((v1,v_ref1))
v1.sort()
v2 = np.concatenate((v2,v_ref2))
v2.sort() 

#Calculate the minimal cut surface and the binding cuts
binding_cuts, minimal_surface = get_minimal_cut_surface(rhs_in,cut_coeffs,v1,v2)

#Plot the cut surface
fig = go.Figure()
fig.add_trace(go.Surface(z=minimal_surface, x=v1*100/vmax1, y=v2*100/vmax2,hoverinfo="skip"))

colors = ["red"]*n_cuts
text = [f"Cut {i+1}, never binding" for i in range(n_cuts)]
for k in binding_cuts:
    colors[k] = "black"
    text[k] = f"Cut {k+1}, binding"
    
#Plot the reference points
fig.add_trace(go.Scatter3d(z=rhs_in, x=v_ref1*100/vmax1, y=v_ref2*100/vmax2,mode='markers',marker_color=colors,hovertext=text))

fig.update_scenes(
    aspectratio=dict(x=1, y=1, z=1),
    xaxis_title='Rsv1 vol [%]',
    yaxis_title='Rsv2 vol [%]',
    zaxis_title='Cut value'
)
fig.update_layout(
    width=1000,
    height=900,
    autosize=True)
    
fig.show()
```

+++ {"Collapsed": "false"}

Now we can run the optimization and look at the results. The final value in the TXY attribute [end_value](cut_group:end_value) on the cut_group object holds the optimized cut value (future expected (negative) income). The [binding_cut_up](cut_group:binding_cut_up) attribute holds the cut constraint that was the most constraining cut in this optimization.

```{code-cell} ipython3
:Collapsed: 'false'

#Optimize model by calling "run_model"
run_model(shop)

#Get the final end value of the cut group, which is the future expected (negative) profit
cut_val = -my_cuts.end_value.get().iloc[-1]

#Get the binding cut constraint and print the cut information
cut_index = my_cuts.binding_cut_up.get().iloc[-1]
cut_index = int(cut_index)

rhs_out = my_cuts.rhs.get()[0]
active_rhs = rhs_out.values[cut_index]

print(f"The future expeced income is {cut_val:.2f} € and the binding cut is number {cut_index+1} out of {n_cuts} with the following values:")
print(f"RHS is {active_rhs:.2f} €")
for rsv in shop.model.reservoir:
    name = rsv.get_name()
    
    cut_coeffs = rsv.water_value_input.get()[0]
    v_ref = cut_coeffs.index[cut_index]
    wv = cut_coeffs.values[cut_index]

    end_vol = rsv.storage.get().iloc[-1]

    print(f"Cut coefficient {wv:.2f} €/Mm3 and reference volume {v_ref:.2f} Mm3 for {name} with end volume {end_vol:.2f} Mm3")
```

+++ {"Collapsed": "false"}

Note that the attribute [binding_cut_down](cut_group:binding_cut_down) is not used when the cuts are not price dependent, and have the default value of:

```{code-cell} ipython3
:Collapsed: 'false'

i = my_cuts.binding_cut_down.get().iloc[-1]
print(f"'binding_cut_down' attribute for standard cuts: {i}")
```

+++ {"Collapsed": "false"}

The evolution of the reservoir storage and the global water value result time series are shown in the figures below. The final value of the global water value time series is equal to the binding cut coefficient for each reservoir.

```{code-cell} ipython3
:Collapsed: 'false'

pd.DataFrame([rsv.storage.get().rename(rsv.get_name()) for rsv in shop.model.reservoir]).transpose().plot(title="Reservoir storage")
pd.DataFrame([-rsv.water_value_global_result.get().rename(rsv.get_name()) for rsv in shop.model.reservoir]).transpose().plot(title="Reservoir global water value")
```

+++ {"Collapsed": "false"}

# Price dependent cut description
To illustrate how price dependent cuts can be added to SHOP, we will use the same cuts as in the example above and synthesize some price variations. 

```{code-cell} ipython3
:Collapsed: 'false'

import numpy as np

#Create a standard ShopSession
shop=ShopSession()
#Build a simple model with two reservoirs and two plants.
build_model(shop)

#Read the cut info
rhs_in,cut_coeffs = read_cuts()
n_cuts = len(rhs_in)

#Create a cut group object
my_cuts = shop.model.cut_group.add_object("my_cuts")
for rsv in shop.model.reservoir:
    rsv.connect_to(my_cuts)
```

+++ {"Collapsed": "false"}

We use the average price in the SHOP run and create four arbitrary price levels around it. Only the two closest price levels to the average price will actually be used in SHOP, so the highest and lowest price levels are not necessary for this particular model run. The rhs and cut coefficients are then scaled linearly for each price level.

Since SHOP also uses the average price to interpolate the cuts in the two closest price levels, the solution we get form using the scaled price dependent cuts will be identical to the solution in the previous example with standard cuts. This is assuming that the models are set up in the exact same way and that a MIP gap of zero is reached in each iteration in both models. To get some different results in this example, we add some "random" noise to the scaling in the code below. Try to comment out the code adding the noise and see for yourself if the results are the same as in the original case!

```{code-cell} ipython3
:Collapsed: 'false'

#Find the average price and make some new price levels around it
market = shop.model.market.Day_ahead
average_price = np.mean(market.sale_price.get())

prices = [average_price*0.5,average_price*0.8,average_price*1.12,average_price*1.3]

scaling = [p/average_price for p in prices]

#Comment out the two lines below to remove the noise applied to the cut scaling
noise = [0.95,0.96,1.04,1.05]
scaling = [s*n for s,n in zip(scaling,noise)]

#Set the scaled RHS, the price is set as a reference value
my_cuts.rhs.set([pd.Series(np.array(rhs_in)*scale, index=[0]*n_cuts, name=price) for scale,price in zip(scaling,prices)])

#Add the scaled cut coefficients, the price is set as a reference value
for rsv in shop.model.reservoir:
    name = rsv.get_name()
    
    v_ref = cut_coeffs[name][0]
    wv = cut_coeffs[name][1]
    
    rsv.water_value_input.set([pd.Series(np.array(wv)*scale, index=v_ref, name=price) for scale,price in zip(scaling,prices)])
```

+++ {"Collapsed": "false"}

The scatter plots below shows the scaled and original cuts in terms of the rhs and cut coefficients. The values are plotted in 2D for simplicity:

```{code-cell} ipython3
:Collapsed: 'false'

#Plot the scaled and original RHS values

rhs = my_cuts.rhs.get()

fig = go.Figure(layout={'title':f"RHS",'xaxis_title':"Cut number",'yaxis_title':"RHS [€]"})
for r in rhs:
    fig.add_trace(go.Scatter(name=f"Price {r.name:.2f}",x=list(range(1,n_cuts+1)),y=r.values,mode='markers'))    

fig.add_trace(go.Scatter(name=f"Original",x=list(range(1,n_cuts+1)),y=rhs_in,mode='markers'))    
fig.show()
```

```{code-cell} ipython3
:Collapsed: 'false'

#Plot the scaled and original (ref volume, cut coefficient) pairs
for rsv in shop.model.reservoir:    
    wv_in = rsv.water_value_input.get()
    name = rsv.get_name()
    
    fig = go.Figure(layout={'title':f"{name}",'xaxis_title':"Ref volume [Mm3]",'yaxis_title':r"Cut coefficient [€/Mm3]"})
    for wv in wv_in:
        fig.add_trace(go.Scatter(name=f"Price {wv.name:.2f}",x=wv.index,y=wv.values,text=[f"Cut number {i+1}" for i in range(n_cuts)],mode='markers'))    
    
    fig.add_trace(go.Scatter(name=f"Original",x=cut_coeffs[name][0],y=cut_coeffs[name][1],text=[f"Cut number {i+1}" for i in range(n_cuts)],mode='markers'))    
    
    fig.show()
```

+++ {"Collapsed": "false"}

Now we run the model and take a look at the final cut value and the binding cuts:

```{code-cell} ipython3
:Collapsed: 'false'

#Optimize model by calling "run_model"
run_model(shop)

#Get the final end value of the cut group, which is the future expected (negative) profit
cut_val = -my_cuts.end_value.get().iloc[-1]
print(f"The future expeced income is {cut_val:.2f} €\n")

#Get the binding cut constraints in the two price levels and print the cut information
cut_index_up = my_cuts.binding_cut_up.get().iloc[-1]
cut_index_up = int(cut_index_up)

cut_index_down = my_cuts.binding_cut_down.get().iloc[-1]
cut_index_down = int(cut_index_down)

rhs_out = my_cuts.rhs.get()
active_rhs_up = rhs_out[2].values[cut_index_up]
active_rhs_down = rhs_out[1].values[cut_index_down]

print(f"The binding cut in the set of cuts over the SHOP price is number {cut_index_up+1} out of {n_cuts} with the following values:")
print(f"\tRHS is {active_rhs_up:.2f} €")
for rsv in shop.model.reservoir:
    name = rsv.get_name()
    
    cut_coeffs = rsv.water_value_input.get()[2]
    v_ref = cut_coeffs.index[cut_index_up]
    wv = cut_coeffs.values[cut_index_up]

    end_vol = rsv.storage.get().iloc[-1]

    print(f"\tCut coefficient {wv:.2f} €/Mm3 and reference volume {v_ref:.2f} Mm3 for {name} with end volume {end_vol:.2f} Mm3")
print("")

print(f"The binding cut in the set of cuts below the SHOP price is number {cut_index_down+1} out of {n_cuts} with the following values:")
print(f"\tRHS is {active_rhs_down:.2f} €")
for rsv in shop.model.reservoir:
    name = rsv.get_name()
    
    cut_coeffs = rsv.water_value_input.get()[1]
    v_ref = cut_coeffs.index[cut_index_down]
    wv = cut_coeffs.values[cut_index_down]

    end_vol = rsv.storage.get().iloc[-1]

    print(f"\tCut coefficient {wv:.2f} €/Mm3 and reference volume {v_ref:.2f} Mm3 for {name} with end volume {end_vol:.2f} Mm3")
```

+++ {"Collapsed": "false"}

The final value of the global water value result time series is equal to the weighted sum of the binding cut coefficients in the price levels above and below the average SHOP price. The interpolation weights used in SHOP are:

```{code-cell} ipython3
:Collapsed: 'false'

w_up = (average_price - prices[1])/(prices[2]-prices[1])
w_down = 1 - w_up
print(f"Weight for cuts with price {(prices[1]):.2f}: {w_down:.3f}")
print(f"Weight for cuts with price {(prices[2]):.2f}: {w_up:.3f}")
```

```{code-cell} ipython3
:Collapsed: 'false'

pd.DataFrame([rsv.storage.get().rename(rsv.get_name()) for rsv in shop.model.reservoir]).transpose().plot(title="Reservoir storage")
pd.DataFrame([-rsv.water_value_global_result.get().rename(rsv.get_name()) for rsv in shop.model.reservoir]).transpose().plot(title="Reservoir global water value")
```

+++ {"Collapsed": "false"}

(cuts.py)=
## cuts.py

```{code-cell} ipython3
:Collapsed: 'false'
:tags: ['remove-input']

with open('cuts.py', 'r') as f:
    print(f.read())
```

+++ {"Collapsed": "false"}

(rhs_file.txt)=
## rhs_file.txt

```{code-cell} ipython3
:Collapsed: 'false'
:tags: ['remove-input']

with open('rhs_file.txt', 'r') as f:
    print(f.read())
```

+++ {"Collapsed": "false"}

(wv_file.txt)=
## wv_file.txt

```{code-cell} ipython3
:Collapsed: 'false'
:tags: ['remove-input']

with open('wv_file.txt', 'r') as f:
    print(f.read())
```

+++ {"Collapsed": "false"}

(cut-model.yaml)=
## model.yaml

```{code-cell} ipython3
:Collapsed: 'false'
:tags: ['remove-input']

with open('model.yaml', 'r') as f:
    print(f.read())
```

+++ {"Collapsed": "false"}

(standard_cuts.yaml)=
## standard_cuts.yaml

```{code-cell} ipython3
:Collapsed: 'false'
:tags: ['remove-input']

with open('standard_cuts.yaml', 'r') as f:
    print(f.read())
```

+++ {"Collapsed": "false"}

(price_dep_cuts.yaml)=
## price_dep_cuts.yaml

```{code-cell} ipython3
:Collapsed: 'false'
:tags: ['remove-input']

with open('price_dep_cuts.yaml', 'r') as f:
    print(f.read())
```

+++ {"Collapsed": "false"}

(cut-model.ascii)=
## model.ascii

```{code-cell} ipython3
:Collapsed: 'false'
:tags: ['remove-input']

with open('model.ascii', 'r') as f:
    print(f.read())
```

+++ {"Collapsed": "false"}

(name_coupling.txt)=
## name_coupling.txt

```{code-cell} ipython3
:Collapsed: 'false'
:tags: ['remove-input']

with open('name_coupling.txt', 'r') as f:
    print(f.read())
```

+++ {"Collapsed": "false"}

(cut_order.txt)=
## cut_order.txt

```{code-cell} ipython3
:Collapsed: 'false'
:tags: ['remove-input']

with open('cut_order.txt', 'r') as f:
    print(f.read())
```

+++ {"Collapsed": "false"}

(standard_cuts.txt)=
## standard_cuts.txt

```{code-cell} ipython3
:Collapsed: 'false'
:tags: ['remove-input']

with open('standard_cuts.txt', 'r') as f:
    print(f.read())
```

+++ {"Collapsed": "false"}

(price_dep_cuts.txt)=
## price_dep_cuts.txt

```{code-cell} ipython3
:Collapsed: 'false'
:tags: ['remove-input']

with open('price_dep_cuts.txt', 'r') as f:
    print(f.read())
```