Serializing a SHOP model#
SHOP accepts different input formats, such as YAML and JSON. However, these formats do not save the entire internal state of the SHOP program, which makes it impossible to recreate the exact state of a SHOP run. The online optimization functionality allows for a deeper serialization of the whole SHOP run, and enables the user to dump the SHOP state to a binary file or to retrieve the state and keep it in memory as an array of bytes. This example will demonstrate how serialization and deserialization can be used to save and resume previous SHOP runs.
The SHOP_ONLINE_OPTIMIZATION license in the “Online Optimization Capabilities” license group is required to (de)serialize SHOP models. The following YAML model will be used as a base for the rest of the example:
Full model serialization#
When a full serialization is performed, the entire SHOP state is dumped to the binary format (either a file or in memory). This includes the input and output data on all objects that can normally be retrieved through the API, connections between objects, and internal attributes automatically calculated by SHOP that is not exposed in the API. After a full serialization has been performed, it can be loaded into a new SHOP run to start at the same state as the previous run. No objects should be created by the user in the new SHOP run before loading in a fully serialized model, otherwise undefined behaviour will occur.
To perform a full serialization, a model with all input data is first loaded in from a YAML file and three full and three incremental iterations are executed:
from pyshop import ShopSession
shop = ShopSession()
shop.load_yaml("model.yaml")
shop.start_sim([],3)
shop.set_code("incremental", [])
shop.start_sim([], 3)
For future reference, the resulting reservori volume of Reservoir1 after 3 + 3 iterations is shown here:
rsv = shop.model.reservoir["Reservoir1"]
print(rsv.storage.get())
2018-01-23 00:00:00 33.913043
2018-01-23 01:00:00 33.524987
2018-01-23 02:00:00 33.154691
2018-01-23 03:00:00 32.784395
2018-01-23 04:00:00 32.414099
...
2018-01-25 20:00:00 17.785555
2018-01-25 21:00:00 17.600407
2018-01-25 22:00:00 17.415259
2018-01-25 23:00:00 17.252050
2018-01-26 00:00:00 17.252050
Name: storage, Length: 73, dtype: float64
After the model has been executed, the state can be saved to a binary file with the dump_state function in pyshop (version 1.7.3 or later):
shop.dump_state("full_state.bin")
If no file name is specified, the state is returned as an array of bytes:
state = shop.dump_state()
print(type(state))
<class 'bytes'>
The SHOP run can now be resumed by loading in the state into a new SHOP run with the load_state function in pyshop. The function accepts a file name or a byte array as input:
shop_full = ShopSession()
shop_full.load_state("full_state.bin")
After loading the state, all objects and input and output attributes are loaded in. The reservoir storage of Reservoir1 is identical to the storage reported in the original SHOP run:
rsv = shop_full.model.reservoir["Reservoir1"]
print(rsv.storage.get())
2018-01-23 00:00:00 33.913043
2018-01-23 01:00:00 33.524987
2018-01-23 02:00:00 33.154691
2018-01-23 03:00:00 32.784395
2018-01-23 04:00:00 32.414099
...
2018-01-25 20:00:00 17.785555
2018-01-25 21:00:00 17.600407
2018-01-25 22:00:00 17.415259
2018-01-25 23:00:00 17.252050
2018-01-26 00:00:00 17.252050
Name: storage, Length: 73, dtype: float64
Now we can update any data we like and continue running iterations. No data is changed here, so running another iteration will give identical results to starting from scratch and running 3 full and 4 incremental iterations:
shop_full.start_sim([], 1)
Partial model serialization#
A partial serialization is intended to save the smallest possible amount of data to be able to recreate the complete SHOP state. This is achieved by ignoring all object attributes and connections that is already exposed in the API, and only saving the hidden internal states of the SHOP objects. However, this means that the user is responsible for building the topology (objects and connections) and setting all input and output data available in the API before the partial state is loaded into the new SHOP model. The new SHOP model must be created in the exact same way as the original SHOP model where the partial state originated from, meaning that objects have to be added and connected in the same order with the same names to avoid undefined behaviour.
We use the same original SHOP run as before, and set the optional partial argument to True when dumping the model state:
partial_state = shop.dump_state(partial = True)
In addition to the partial state, we need to set all objects with input and output attributes. We can do this by dumping a YAML or JSON file with all data from the original run:
shop.dump_yaml("model_w_results.yaml", input_only = False)
A new SHOP run can now be constructed from the full YAML file and the partial state. Note that setting output attributes, such as the reservoir volume, will normally result in an error or be silently ignored unless the online optimization functionality has been activated by the command activate online_optimization /on. This has to be done explicitly by the user to avoid unintended setting of output attributes in regular SHOP runs, and must be set before loading the YAML file to actually read in all attributes. Also note that dumping the YAML file from the original model includes all the executed commands. To avoid running all the iterations when loading the YAML file, we set skip_commands = True:
shop_partial = ShopSession()
shop_partial.activate_online_optimization("on", [])
shop_partial.load_yaml("model_w_results.yaml", skip_commands = True)
shop_partial.load_state(partial_state, partial = True)
Again we se that the reservoir volume of Reservoir1 is identical to the original run:
rsv = shop_partial.model.reservoir["Reservoir1"]
print(rsv.storage.get())
2018-01-23 00:00:00 33.913043
2018-01-23 01:00:00 33.524987
2018-01-23 02:00:00 33.154691
2018-01-23 03:00:00 32.784395
2018-01-23 04:00:00 32.414099
...
2018-01-25 20:00:00 17.785555
2018-01-25 21:00:00 17.600407
2018-01-25 22:00:00 17.415259
2018-01-25 23:00:00 17.252050
2018-01-26 00:00:00 17.252050
Name: storage, Length: 73, dtype: float64
Note that it is not possible to load in the fully serialized state with partial = True, or vice versa. This will likely result in a crash. Loading in a state saved from an old SHOP version will also likely crash, so a SHOP run from scratch is needed after updating SHOP.
This time, we increase the sale price in all markets with 10% to simulate receiving a new price prognosis. If this is judged to be a significant change, we can go back to full iterations to let the unit commitment of the generators adjust to the new price. Only a single full iteration is performed in this case, since we are likely pretty close to the correct unit commitment:
for m in shop_partial.model.market:
old_price = m.sale_price.get()
m.sale_price.set(1.1 * old_price)
shop_partial.set_code("full", [])
shop_partial.start_sim([], 1)
shop_partial.set_code("incremental", [])
shop_partial.start_sim([], 3)