{
"cells": [
{
"cell_type": "markdown",
"id": "04ac5031",
"metadata": {
"Collapsed": "false"
},
"source": [
"(multiple-price-bid-matrix)=\n",
"# Bidding under uncertainty\n",
"\n",
"The model setup is available in the following formats:\n",
"\n",
"- pyshop\n",
" - [](multi_price_bid_matrix_model.py)"
]
},
{
"cell_type": "markdown",
"id": "5d079f8f",
"metadata": {
"Collapsed": "false"
},
"source": [
"In this example we show how to import a spreadsheet containing multiple prices (up to 52 time series) into a dataframe. This price input will be used to create the same amount of scenarios which we in turn feed to SHOP. The output of this multiple price input run from SHOP will result into a joint bid matrix which consider all the stochastic price inputs to SHOP with all its price data intact, ready to make use of in a marked bidding situation with uncertainty regarding the price forecasts."
]
},
{
"cell_type": "markdown",
"id": "91da31ec",
"metadata": {
"Collapsed": "false"
},
"source": [
"## Imports and settings"
]
},
{
"cell_type": "markdown",
"id": "8d44656a",
"metadata": {
"Collapsed": "false"
},
"source": [
"The first thing we do is to import the needed packages. You can import whichever packages you like, however we use the following ones for this example:\n",
"\n",
"* Pandas for structuring our data into dataframes\n",
"* Pyshop in order to create a SHOP session\n",
"* Plotly as backend for dynamic graph plotting"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "82051bc5",
"metadata": {
"Collapsed": "false"
},
"outputs": [],
"source": [
"import pandas as pd\n",
"from pyshop import ShopSession\n",
"import plotly.express as px\n",
"pd.options.plotting.backend = \"plotly\""
]
},
{
"cell_type": "markdown",
"id": "95619ca1",
"metadata": {
"Collapsed": "false"
},
"source": [
"Additionally, we import basic SHOP functions and data from a predefined demo dataset, see this section of the documentation:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "21b4daa8",
"metadata": {
"Collapsed": "false"
},
"outputs": [],
"source": [
"from multi_price_bid_matrix_model import build_model, run_model"
]
},
{
"cell_type": "markdown",
"id": "2bb819b8",
"metadata": {
"Collapsed": "false"
},
"source": [
"## Instancing SHOP and building the model"
]
},
{
"cell_type": "markdown",
"id": "595d7f77",
"metadata": {
"Collapsed": "false"
},
"source": [
"In order to have SHOP receive our inputs, run the model we create and give us results, we need to instance a running SHOP session. You may create multiple SHOP sessions simultaneously if needed."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "c34733bf",
"metadata": {
"Collapsed": "false"
},
"outputs": [],
"source": [
"#Create a standard ShopSession\n",
"shop = ShopSession()"
]
},
{
"cell_type": "markdown",
"id": "a4cd18d2",
"metadata": {
"Collapsed": "false"
},
"source": [
"We then build our model using the existing function imported from bp.py"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "507da28d",
"metadata": {
"Collapsed": "false"
},
"outputs": [],
"source": [
"#Build a simple model with one plant with two equal generators and a second plant with one large and three small generators \n",
"#by calling function \"build_model\" in bp.py\n",
"build_model(shop)"
]
},
{
"cell_type": "markdown",
"id": "b0b7ae71",
"metadata": {
"Collapsed": "false"
},
"source": [
"The imported model can now be visualized:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "128e548d",
"metadata": {
"Collapsed": "false"
},
"outputs": [
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n",
"\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"#Display topology to the screen\n",
"display(shop.model.build_connection_tree())"
]
},
{
"cell_type": "markdown",
"id": "0ccee33f",
"metadata": {
"Collapsed": "false"
},
"source": [
"## Data preperation"
]
},
{
"cell_type": "markdown",
"id": "2b124e4d",
"metadata": {
"Collapsed": "false"
},
"source": [
"Since we we need to work a bit with time horizons given that we will consider part of a period with stochastic information, we retrive the start time from the imported data"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "ef455236",
"metadata": {
"Collapsed": "false"
},
"outputs": [
{
"data": {
"text/plain": [
"{'starttime': Timestamp('2018-01-23 00:00:00'),\n",
" 'endtime': Timestamp('2018-01-26 00:00:00'),\n",
" 'timeunit': 'hour',\n",
" 'timeresolution': 2018-01-23 00:00:00 1.0\n",
" 2018-01-23 01:00:00 1.0\n",
" 2018-01-23 02:00:00 1.0\n",
" 2018-01-23 03:00:00 1.0\n",
" 2018-01-23 04:00:00 1.0\n",
" ... \n",
" 2018-01-25 20:00:00 1.0\n",
" 2018-01-25 21:00:00 1.0\n",
" 2018-01-25 22:00:00 1.0\n",
" 2018-01-25 23:00:00 1.0\n",
" 2018-01-26 00:00:00 1.0\n",
" Name: data, Length: 73, dtype: float64}"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Retrieve the start time for adding stochastic time series later\n",
"timeres=shop.get_time_resolution()\n",
"shop.get_time_resolution()"
]
},
{
"cell_type": "markdown",
"id": "ff4e9869",
"metadata": {
"Collapsed": "false"
},
"source": [
"### Importing prices from spreadsheet"
]
},
{
"cell_type": "markdown",
"id": "79650b73",
"metadata": {
"Collapsed": "false"
},
"source": [
"After the model data has been imported and read, we move on to the prices we want to consider. \n",
"We first define the number of prices we want to import"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "fe37e93d",
"metadata": {
"Collapsed": "false"
},
"outputs": [],
"source": [
"# Import number of prices\n",
"n_prices = 10"
]
},
{
"cell_type": "markdown",
"id": "761b8206",
"metadata": {
"Collapsed": "false"
},
"source": [
"which will be equal to the scenarios we will create later on. \n",
"We then define for how long of a period we want to consider the stochastic price input. This can not be a larger period than what you have data for in your price forecasts."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "09ef31f8",
"metadata": {
"Collapsed": "false"
},
"outputs": [],
"source": [
"# Set the hours to import stochastic price\n",
"stochastic_prices_start = timeres['starttime']\n",
"# In this example, we choose 24 hours\n",
"stochastic_prices_end = stochastic_prices_start + pd.Timedelta(hours=23)"
]
},
{
"cell_type": "markdown",
"id": "8aa80440",
"metadata": {
"Collapsed": "false"
},
"source": [
"Next we import the stochastic prices accordingly into a dataframe"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "686d4f37",
"metadata": {
"Collapsed": "false"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/tmp/ipykernel_2555/2553596855.py:2: Pandas4Warning: The xlrd engine is deprecated and will be removed in a future version. Use the calamine engine instead, by setting engine='calamine'.\n",
" stochastic_prices_from_file = pd.read_excel ('52_hourly_prices_random.xls', header=None,usecols=list(range(0,n_prices)))\n"
]
},
{
"data": {
"text/html": [
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Plot all stochastic prices imported from Excel\n",
"stochastic_prices_from_file.plot(title=\"Stochastic prices imported from spreadsheet\", labels=dict(index=\"Time\", value=\"EUR/MWh\", variable=\"\"))"
]
},
{
"cell_type": "markdown",
"id": "fd2d7025",
"metadata": {
"Collapsed": "false"
},
"source": [
"### Creating scenarios"
]
},
{
"cell_type": "markdown",
"id": "b89cd75c",
"metadata": {
"Collapsed": "false"
},
"source": [
"Now it is time to create scenarios that we can populate with the imported prices. We make sure to create just as many scenarios as we have prices imported."
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "9a6e36eb",
"metadata": {
"Collapsed": "false"
},
"outputs": [],
"source": [
"# Generate as many scenarios as prices\n",
"n_scenarios = n_prices\n",
"for i in range (1, n_scenarios+1):\n",
" scenario_name='S'+str(i)\n",
" # The first scenario always exists in SHOP and should not be added again\n",
" if i>1:\n",
" scenario = shop.model.scenario.add_object(scenario_name)\n",
" else:\n",
" scenario=shop.model.scenario[scenario_name]\n",
" scenario.scenario_id.set(i)\n",
" \n",
" # Set each scenario equally probable\n",
" scenario.probability.set(1.0/n_scenarios)\n",
" \n",
" # Branch immediately, i.e. at 'starttime'\n",
" scenario.common_scenario.set(pd.Series([i], index=[timeres['starttime']]))\n",
" \n",
" # Optionally set branching to start after given number of hours (all scenarios are set equal to scenario 2 before this) \n",
" #scen.common_scenario.set(pd.Series([2, i], index=[timeres['starttime'], branching_start_time]))"
]
},
{
"cell_type": "markdown",
"id": "4f42d896",
"metadata": {
"Collapsed": "false"
},
"source": [
"### Creating the new price array from stochastic and deterministic prices"
]
},
{
"cell_type": "markdown",
"id": "4dbc7bbe",
"metadata": {
"Collapsed": "false"
},
"source": [
"Since we have chosen to only consider the first 24 hours as stochastic when it comes to the price, but have longer total time horizon, we need to combine the stochastic and deterministic prices into a joint price dataframe. We have already defined the start and end time for the stochastic price, but need to make sure that we also define when the deterministic price should be valid and thus overlap each other."
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "8bd1d6e8",
"metadata": {
"Collapsed": "false"
},
"outputs": [],
"source": [
"# Use first (and only) market as index for setting stochastic data and getting results\n",
"name_list = shop.model.market.get_object_names()\n",
"\n",
"# The deterministic model only has one market, so we give stochastic price to that (da is just a name we use to indicate that we regard it to be a day ahead energy market)\n",
"day_ahead_market = shop.model.market[name_list[0]]\n",
"\n",
"# Get the deterministic price from ASCII import for correct indexing \n",
"deterministic_price = day_ahead_market.sale_price.get()\n",
"\n",
"# Create a new array for deterministic price with n_prices columns in order to combine it with the stochastic price\n",
"deterministic_price_multi_dimension_array = pd.DataFrame(index=deterministic_price.index, columns=list(range(0,n_prices)))\n",
"\n",
"# Define when to use deterministic price\n",
"deterministic_price_start = timeres['starttime']+pd.Timedelta(hours=24)\n",
"deterministic_price_end = timeres['endtime']-pd.Timedelta(hours=1)"
]
},
{
"cell_type": "markdown",
"id": "51ca8913",
"metadata": {
"Collapsed": "false"
},
"source": [
"We print out the start and end times to make sure they are correct."
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "a34ac041",
"metadata": {
"Collapsed": "false"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Stochastic price(s) start time 2018-01-23 00:00:00\n",
"Stochastic price(s) end time 2018-01-23 23:00:00\n",
"Deterministic prices start time 2018-01-24 00:00:00\n",
"Deterministic prices end time 2018-01-25 23:00:00\n"
]
}
],
"source": [
"# Print out start and end set point for prices\n",
"print(\"Stochastic price(s) start time\",stochastic_prices_start)\n",
"print(\"Stochastic price(s) end time\",stochastic_prices_end)\n",
"print(\"Deterministic prices start time\",deterministic_price_start)\n",
"print(\"Deterministic prices end time\",deterministic_price_end)"
]
},
{
"cell_type": "markdown",
"id": "64f3c65a",
"metadata": {
"Collapsed": "false"
},
"source": [
"Lastly, we need to combine the stochastic and deterministic price inputs. We do this by creating a new dataframe with the right dimensions and indexes, and then populate it according to the start and end times defined above."
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "58c36e0f",
"metadata": {
"Collapsed": "false"
},
"outputs": [],
"source": [
"# Creating a new price dataframe which will combine both stochastic and deterministic price\n",
"combined_price = pd.DataFrame(index=deterministic_price.index, columns=list(range(0,n_prices)))\n",
"\n",
"# Loop over the new price array and assign stochastic and deterministic prices\n",
"\n",
"for j in range(0,n_prices):\n",
" deterministic_price_multi_dimension_array[j]=deterministic_price\n",
"for i in pd.date_range(stochastic_prices_start,stochastic_prices_end,freq='h'):\n",
" combined_price.loc[i] = stochastic_prices_from_file.loc[i]\n",
"for i in pd.date_range(deterministic_price_start,(deterministic_price_end),freq='h'):\n",
" combined_price.loc[i] = deterministic_price_multi_dimension_array.loc[i]"
]
},
{
"cell_type": "markdown",
"id": "dee6618c",
"metadata": {
"Collapsed": "false"
},
"source": [
"Then we can plot out the resulting combined price."
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "32df7d06",
"metadata": {
"Collapsed": "false"
},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Plot the combined price\n",
"combined_price.plot(title=\"Combined price input\", labels=dict(index=\"Time\", value=\"EUR/MWh\", variable=\"\"))"
]
},
{
"cell_type": "markdown",
"id": "275f1af3",
"metadata": {
"Collapsed": "false"
},
"source": [
"This price is then set as the day ahead market price to consider for all scenarios. We also define a slightly higher buy-back price"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "109a87f6",
"metadata": {
"Collapsed": "false"
},
"outputs": [],
"source": [
"day_ahead_market.sale_price.set(combined_price)\n",
"\n",
"# Make sure that the buy price is slightly higher, so no arbitrage occurs\n",
"\n",
"day_ahead_market.buy_price.set(day_ahead_market.sale_price.get()+0.1)"
]
},
{
"cell_type": "markdown",
"id": "9d61409b",
"metadata": {
"Collapsed": "false"
},
"source": [
"### Create bid groups and configure limits"
]
},
{
"cell_type": "markdown",
"id": "a950696d",
"metadata": {
"Collapsed": "false"
},
"source": [
"Now we need to create a bid group, which is the connection between a set of hydropower plants and their bid matrix."
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "2bca4075",
"metadata": {
"Collapsed": "false"
},
"outputs": [],
"source": [
"# Create a bid group\n",
"bg=shop.model.bid_group.add_object('bg')\n",
"\n",
"# Add all plants in the system to the bid group\n",
"for plant in shop.model.plant:\n",
" bg.connect_to(plant)\n",
"\n",
"# Defining which periods the bid curve should consider\n",
"day_ahead_market.bid_flag.set(pd.Series([1,0],index=[stochastic_prices_start,stochastic_prices_end]))"
]
},
{
"cell_type": "markdown",
"id": "351886b0",
"metadata": {
"Collapsed": "false"
},
"source": [
"## Running multi price scenarios in SHOP"
]
},
{
"cell_type": "markdown",
"id": "2f006c7d",
"metadata": {
"Collapsed": "false"
},
"source": [
"It is time to optimize. We call the predefined function run_model, which calls for five full and three incremental iterations. Since we have defined scenarios in the SHOP instance earlier, all scenarios will be computed and optimized with a single call."
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "d1d8b5b7",
"metadata": {
"Collapsed": "false"
},
"outputs": [],
"source": [
"#Optimize model by calling \"run_model\" bp.py\n",
"run_model(shop)"
]
},
{
"cell_type": "markdown",
"id": "ab570cf8",
"metadata": {
"Collapsed": "false"
},
"source": [
"## Creating and plotting the bid matrix"
]
},
{
"cell_type": "markdown",
"id": "5ab5452c",
"metadata": {
"Collapsed": "false"
},
"source": [
"Once SHOP has finished optimizing all scenarios, we can retrive and start processing the bid matrix the way we want it."
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "527aec32",
"metadata": {
"Collapsed": "false"
},
"outputs": [],
"source": [
"# Get bid matrix as an array\n",
"bid_result=bg.bid_curves.get()\n",
"\n",
"# Convert bid matrix to a dataframe structure of choice\n",
"bid_matrix=pd.DataFrame(index=bid_result[0].index)\n",
"for t in range (0, 23):\n",
" bid_matrix[t]=bid_result[t].values\n",
"\n",
"# Transpose the bid matrix if necessary depending on the viewing needs \n",
"bid_matrix_transposed=bid_matrix.transpose()"
]
},
{
"cell_type": "markdown",
"id": "d87280a7",
"metadata": {
"Collapsed": "false"
},
"source": [
"When we are satisfied with a data structure that fulfill our needs, we can either print it or plot it to review it"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "77c07d99",
"metadata": {
"Collapsed": "false"
},
"outputs": [
{
"data": {
"text/html": [
"