process_storage_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Processing _Danish Energy Agency Technology Catalogue_ energy storage data for Mopo WP5\n",
    "\n",
    "The aim of this workbook is to process the desired heat storage data from the\n",
    "DEA technology catalogues into a format desired by WP5."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Julia setup\n",
    "\n",
    "This processing script is written in Julia, and some setup is required for it to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Activate (and set up) the required Julia environment\n",
    "\n",
    "using Pkg # Julia package manager.\n",
    "Pkg.activate(@__DIR__) # Activate the Julia environment in the folder this file is in (namely the `Project.toml`)\n",
    "Pkg.instantiate() # Download and install the necessary dependencies.\n",
    "\n",
    "# Load dependencies\n",
    "using XLSX\n",
    "using DataFrames\n",
    "using Statistics\n",
    "using CSV\n",
    "using MopoHeatSectorDataProcessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read DEA Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Read the storage technology catalogue\n",
    "\n",
    "raw_data = DataFrame(XLSX.readtable(\"input-data\\\\dea-technology-catalogues\\\\technology_datasheet_for_energy_storage.xlsx\", \"alldata_flat\"))\n",
    "first(raw_data, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Map DEA technologies to desired output technologies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Collect the set of all available technologies.\n",
    "\n",
    "techs = Set(raw_data[:, :Technology])\n",
    "\n",
    "# With only 15 technologies available, mapping is way easier than for heat generation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Large-scale hot water storage tanks\n",
    "\n",
    "The `LT-hot-water-tank`, which I assume stands for \"low temperature hot water tank\"\n",
    "or something? Regardless, its for district heating so large-scale in this catalogue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Map district-heating-scale hot water tank.\n",
    "\n",
    "DH_LT_hot_water_tank = filter(\n",
    "    x -> contains(lowercase(x), \"large-scale hot water tank\"),\n",
    "    techs\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Small-scale heat storage\n",
    "\n",
    "Apparently, the `heat-storage` in the WP5 case refers to distributed\n",
    "heat storage in buildings, essentially small-scale domestig hot water storage tanks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Map distributed small-scale thermal storage.\n",
    "\n",
    "distributed_hot_water_tanks = filter(\n",
    "    x -> contains(lowercase(x), \"small-scale hot water tank\"),\n",
    "    techs\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Map pit thermal storage\n",
    "\n",
    "Just for good measure since we have it available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Map pit thermal storage cause why not.\n",
    "\n",
    "pit_thermal_storage = filter(\n",
    "    x -> contains(lowercase(x), \"thermal\"),\n",
    "    techs\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculate the desired parameters\n",
    "\n",
    "Based on the _Conclusions_ in `examine_storage_data.ipynb`,\n",
    "the rough plan for the parameters is as follows:\n",
    "\n",
    "- `CAPEX Energy`: Likely some `Specific investment` or its `- energy component`.\n",
    "- `CAPEX`: Likely `Specific investment` or its `- capacity component` similar to above.\n",
    "- `FOM Energy`: Likely `Fixed O&M [EUR/MWhCapacity/year]` or similar.\n",
    "- `FOM`: Likely `Fixed O&M [EUR/MW/year]` or similar.\n",
    "- `VOM`: Easy `Variable O&M [EUR/MWh]`.\n",
    "- `Charge Efficiency`: Charging/discharging efficiencies seem to be given as components of `Round trip efficiency`, e.g. `- Charge Efficiency [%]`. I can only hope they are available separately everywhere.\n",
    "- `Discharge Efficiency`: Similar to above, but `- Discharge efficiency [%]`.\n",
    "- `Energy-to-power ratio`: Ratio between energy content and discharging power. E.g. `Energy storage capacity for one unit [MWh]` divided by `Generating capacity for one unit [MW]`.\n",
    "- `Currency`: EUR 2020, the `priceyear` column contains this information.\n",
    "- `Metadata`: Add citation to the DEA catalogue in question.\n",
    "\n",
    "**NOTE! In this data, `CAPEX Energy` and `CAPEX` are either or!**\n",
    "Both contain all the investment costs for the entire plant\n",
    "and are just alternative forms of the same costs!\n",
    "As such, they are *NOT separate investments for the energy storage\n",
    "and its charging infrastructure!\n",
    "`FOM Energy` and `FOM` are similarly just two slightly different forms\n",
    "of presenting the total fixed O&M costs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Calculate the desired parameters\n",
    "\n",
    "# Parameter name correction mapping table.\n",
    "spelling_fixes = Dict(\n",
    "    \"[EUR/MWhCapacity/year)\" => \"[EUR/MWhCapacity/year]\",\n",
    "    \"[MWh)\" => \"[MWh]\",\n",
    "    \"[kWh)\" => \"[kWh]\",\n",
    "    \"[MW)\" => \"[MW]\",\n",
    "    \"[EUR/MW/year] - 1-2% of investment\" => \"[EUR/MW/year]\",\n",
    "    \"[MW]*\" => \"[MW]\"\n",
    ")\n",
    "\n",
    "# Preprocessing\n",
    "cols = [:Technology, :par, :est, :year, :val] # Omit unnecessary cols\n",
    "data = deepcopy(raw_data[!, cols])\n",
    "data.est = lowercase.(data.est) # Avoid upper/lowercase issues\n",
    "data.year = string.(data.year) # Avoid year stack/unstack issues\n",
    "filter!(r -> r.est == \"ctrl\", data) # Filter our lower and upper estimates.\n",
    "filter!(r -> typeof(r.val) != String, data) # Filter out string values.\n",
    "map!(p -> replace(p, spelling_fixes...), data.par, data.par) # Fix bothersome misspellings.\n",
    "unique!(data)\n",
    "data = extrapolate_years( # Inter/extrapolate data to cover all years found within.\n",
    "    data;\n",
    "    extrapolation_years=[2015, 2018, 2019, 2020, 2030, 2040, 2050],\n",
    "    combine=mean\n",
    ")\n",
    "data = unstack(data, :par, :val)\n",
    "\n",
    "# Energy-to-power ratio (in hours), calculated first as this turns out to be useful for power-based CAPEX and FOM estimation.\n",
    "data[!, :energy_to_power_ratio_h] = ( # Hours of energy output for large units.\n",
    "    data[!, Symbol(\"Energy storage capacity for one unit [MWh]\")]\n",
    "    ./ data[!, Symbol(\"Output capacity for one unit [MW]\")]\n",
    ")\n",
    "missing_inds = ismissing.(data.energy_to_power_ratio_h)\n",
    "data[missing_inds, :energy_to_power_ratio_h] = ( # Hours of energy output for large units.\n",
    "    data[missing_inds, Symbol(\"Energy storage capacity for one unit [kWh]\")]\n",
    "    ./ data[missing_inds, Symbol(\"Output capacity for one unit [kW]\")]\n",
    ")\n",
    "\n",
    "# CAPEX_energy, Specific investment with the corresponding unit. Need to account for several different potential parameters\n",
    "data[!, :CAPEX_energy_MEUR_GWh] = data[!, Symbol(\"Specific investment [MEUR/GWhCapacity]\")]\n",
    "missing_inds = ismissing.(data.CAPEX_energy_MEUR_GWh)\n",
    "data[missing_inds, :CAPEX_energy_MEUR_GWh] = data[missing_inds, Symbol(\"Specific investment [EUR/kWh]\")] # No scaling to MEUR/GWh required\n",
    "missing_inds = ismissing.(data.CAPEX_energy_MEUR_GWh)\n",
    "data[missing_inds, :CAPEX_energy_MEUR_GWh] = (\n",
    "    data[missing_inds, Symbol(\"Specific investment [MEUR/MWh]\")]\n",
    "    .* 1000 # Requires scaling to reach MEUR/GWh\n",
    ")\n",
    "\n",
    "# CAPEX_power, `Specific investment` with the corresponding unit. Need to account for several params.\n",
    "data[!, :CAPEX_power_MEUR_MW] = data[!, Symbol(\"Specific investment [MEUR/MW]\")]\n",
    "missing_inds = ismissing.(data.CAPEX_power_MEUR_MW)\n",
    "data[missing_inds, :CAPEX_power_MEUR_MW] = (\n",
    "    data[missing_inds, Symbol(\"Specific investment [EUR/kW]\")]\n",
    "    ./ 1000 # Requires scaling to reach MEUR/MW\n",
    ")\n",
    "missing_inds = ismissing.(data.CAPEX_power_MEUR_MW)\n",
    "data[missing_inds, :CAPEX_power_MEUR_MW] = (\n",
    "    data[missing_inds, :CAPEX_energy_MEUR_GWh] # Based on energy CAPEX\n",
    "    ./ 1000 # Scaling to MEUR/MWh\n",
    "    .* data[missing_inds, :energy_to_power_ratio_h] # Scaling based on energy-to-power ratio\n",
    ")\n",
    "\n",
    "# CAPEX_energy based on CAPEX power if it is still missing\n",
    "missing_inds = ismissing.(data.CAPEX_energy_MEUR_GWh)\n",
    "data[missing_inds, :CAPEX_energy_MEUR_GWh] = (\n",
    "    data[missing_inds, :CAPEX_power_MEUR_MW] # Based on CAPEX power\n",
    "    .* 1000 # Scaling to MEUR/GW\n",
    "    ./ data[missing_inds, :energy_to_power_ratio_h] # Scaling based on energy-to-power ratio\n",
    ")\n",
    "\n",
    "# FOM_energy, `Fixed O&M`, although units vary. Small tanks also require specific processing.\n",
    "data[!, :FOM_energy_EUR_GWh_y] = (\n",
    "    data[!, Symbol(\"Fixed O&M [EUR/MWhCapacity/year]\")]\n",
    "    .* 1000 # Requires scaling to reach EUR/GWh/year\n",
    ")\n",
    "missing_inds = ismissing.(data.FOM_energy_EUR_GWh_y)\n",
    "data[missing_inds, :FOM_energy_EUR_GWh_y] = (\n",
    "    data[missing_inds, Symbol(\"Fixed O&M [EUR/tank/year]\")] # Handle small-scale tanks\n",
    "    ./ data[missing_inds, Symbol(\"Energy storage capacity for one unit [kWh]\")]\n",
    "    .* 1e6 # Scaling to EUR/GWh/y\n",
    ")\n",
    "\n",
    "# FOM_power, needs to handle multiple params, as well as estimate FOM costs for tanks.\n",
    "data[!, :FOM_power_EUR_MW_y] = data[!, Symbol(\"Fixed O&M [EUR/MW/year]\")]\n",
    "missing_inds = ismissing.(data.FOM_power_EUR_MW_y)\n",
    "data[missing_inds, :FOM_power_EUR_MW_y] = (\n",
    "    data[missing_inds, Symbol(\"Fixed O&M [kEUR/MW/year]\")]\n",
    "    ./ 1000 # Scaling to EUR/MW/year\n",
    ")\n",
    "missing_inds = ismissing.(data.FOM_power_EUR_MW_y)\n",
    "data[missing_inds, :FOM_power_EUR_MW_y] = ( # Estimate power fom costs for large tanks.\n",
    "    data[missing_inds, :FOM_energy_EUR_GWh_y] # Based on energy FOM cost.\n",
    "    ./ 1000 # Scaling to EUR/MWh/year\n",
    "    .* data[missing_inds, :energy_to_power_ratio_h] # Scaling with \"hours of energy\", which is the ratio of energy to power.\n",
    ")\n",
    "\n",
    "# FOM_energy using FOM_power if not yet exists.\n",
    "missing_inds = ismissing.(data.FOM_energy_EUR_GWh_y)\n",
    "data[missing_inds, :FOM_energy_EUR_GWh_y] = (\n",
    "    data[missing_inds, :FOM_power_EUR_MW_y] # Based on FOM_power\n",
    "    .* 1000 # Scaling to EUR/GWh/y\n",
    "    ./ data[missing_inds, :energy_to_power_ratio_h] # Scaling with energy-to-power ratio.\n",
    ")\n",
    "\n",
    "# VOM, handle small-scale tanks first to avoid electricity costs.\n",
    "data[!, :VOM_EUR_MWh] = data[!, Symbol(\"- of which is other O&M costs [EUR/MWh]\")]\n",
    "missing_inds = ismissing.(data.VOM_EUR_MWh)\n",
    "data[missing_inds, :VOM_EUR_MWh] = data[missing_inds, Symbol(\"Variable O&M [EUR/MWh]\")]\n",
    "missing_inds = ismissing.(data.VOM_EUR_MWh)\n",
    "data[missing_inds, :VOM_EUR_MWh] = data[missing_inds, Symbol(\"Variable O&M [EUR/MWhoutput]\")]\n",
    "\n",
    "# Charge efficiency\n",
    "data[!, :charge_efficiency_pu] = data[!, Symbol(\"- Charge efficiency [%]\")] ./ 100 # Scaling to p.u.\n",
    "\n",
    "# Discharging efficiency\n",
    "data[!, :discharge_efficiency_pu] = data[!, Symbol(\"- Discharge efficiency [%]\")] ./ 100 # Scaling to p.u.\n",
    "\n",
    "# Storage losses\n",
    "data[!, :storage_losses_pu_day] = (\n",
    "    data[!, Symbol(\"Energy losses during storage [%/day]\")]\n",
    "    ./ 100 # Scaling to p.u.\n",
    ")\n",
    "missing_inds = ismissing.(data.storage_losses_pu_day)\n",
    "data[missing_inds, :storage_losses_pu_day] = ( # Handle losses given hourly\n",
    "    1 .- ( # Losses per day estimated from exponential hourly state change.\n",
    "        1 .- data[missing_inds, Symbol(\"Energy losses during storage [%/hour]\")]\n",
    "        ./ 100 # Scaling to p.u.\n",
    "    ) .^ 24 # Exponentiate hourly state change.\n",
    ")\n",
    "missing_inds = ismissing.(data.storage_losses_pu_day)\n",
    "data[missing_inds, :storage_losses_pu_day] = ( # Estimate losses for pit thermal storage\n",
    "    data[missing_inds, Symbol(\"Energy losses during storage [K/day]\")]\n",
    "    ./ (\n",
    "        data[missing_inds, Symbol(\"Max. storage temperature, hot[⁰C]\")]\n",
    "        .- data[missing_inds, Symbol(\"Storage temperature, discharged [⁰C]\")]\n",
    "    )\n",
    ")\n",
    "\n",
    "# Lifetime\n",
    "data[!, :lifetime_y] = data[!, Symbol(\"Technical lifetime [years]\")]\n",
    "\n",
    "# Currency is 2020 Euros based on `examine_storage_data.ipynb`\n",
    "data[!, :currency] .= \"2020 EUR\"\n",
    "\n",
    "# Metadata is the DEA technology catalogue\n",
    "data[!, :metadata] .= \"Technology data for energy storage - October 2018 - Updated April 2024, Danish Energy Agency, 2024\"\n",
    "\n",
    "# Drop unnecessary columns\n",
    "param_cols = [:Technology, :est, :year]\n",
    "params = [\n",
    "    :CAPEX_energy_MEUR_GWh,\n",
    "    :CAPEX_power_MEUR_MW,\n",
    "    :FOM_energy_EUR_GWh_y,\n",
    "    :FOM_power_EUR_MW_y,\n",
    "    :VOM_EUR_MWh,\n",
    "    :charge_efficiency_pu,\n",
    "    :discharge_efficiency_pu,\n",
    "    :storage_losses_pu_day,\n",
    "    :energy_to_power_ratio_h,\n",
    "    :lifetime_y,\n",
    "    :currency,\n",
    "    :metadata\n",
    "]\n",
    "data = data[!, vcat(param_cols, params)]\n",
    "describe(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Technology mapping\n",
    "\n",
    "Map DEA technologies to desired Mopo WP5 technologies.\n",
    "This is extremely straightforward for heat storaged."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Technology mapping\n",
    "\n",
    "technology_mapping = Dict(\n",
    "    \"DH-LT-hot-water-tank\" => DH_LT_hot_water_tank,\n",
    "    \"distributed-hot-water-tanks\" => distributed_hot_water_tanks,\n",
    "    \"DH-pit-thermal-storage\" => pit_thermal_storage\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculate aggregated parameters based on the above mappings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Form the desired technology parameter table\n",
    "\n",
    "cols = [\n",
    "    :technology,\n",
    "    :year,\n",
    "    :CAPEX_energy_MEUR_GWh,\n",
    "    :CAPEX_power_MEUR_MW,\n",
    "    :FOM_energy_EUR_GWh_y,\n",
    "    :FOM_power_EUR_MW_y,\n",
    "    :VOM_EUR_MWh,\n",
    "    :charge_efficiency_pu,\n",
    "    :discharge_efficiency_pu,\n",
    "    :energy_to_power_ratio_h,\n",
    "    :storage_losses_pu_day,\n",
    "    :lifetime_y,\n",
    "    :currency,\n",
    "    :metadata\n",
    "]\n",
    "final_data = DataFrame()\n",
    "for (name, techs) in technology_mapping\n",
    "    # Filter relevant technologies\n",
    "    df = filter(\n",
    "        r -> r.Technology in techs,\n",
    "        data\n",
    "    )\n",
    "    isempty(df) && continue # Skip the rest of the loop if df is empty.\n",
    "    # Calculate average properties\n",
    "    df = unstack(\n",
    "        combine(\n",
    "            groupby(stack(df), [:year, :currency, :metadata, :variable]),\n",
    "            :value => mean\n",
    "        ),\n",
    "        :variable,\n",
    "        :value_mean\n",
    "    )\n",
    "    # Final formatting\n",
    "    df.technology .= name\n",
    "    append!(final_data, df[!, cols])\n",
    "end\n",
    "describe(final_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspect any potentially problematic missing data\n",
    "\n",
    "No missing data it seems, nice!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Format and export technology parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Format and export technology parameters\n",
    "\n",
    "# Config\n",
    "exportcols = [\n",
    "    :technology,\n",
    "    :year,\n",
    "    :CAPEX_energy_MEUR_GWh,\n",
    "    #:CAPEX_power_MEUR_MW,\n",
    "    :FOM_energy_EUR_GWh_y,\n",
    "    #:FOM_power_EUR_MW_y,\n",
    "    :VOM_EUR_MWh,\n",
    "    :charge_efficiency_pu,\n",
    "    :discharge_efficiency_pu,\n",
    "    :energy_to_power_ratio_h,\n",
    "    :storage_losses_pu_day,\n",
    "    :lifetime_y,\n",
    "    :currency,\n",
    "    :metadata\n",
    "]\n",
    "exportyears = string.([2030, 2040, 2050])\n",
    "roundcols = exportcols[3:9]\n",
    "dgts = 4 # Number of digits for rounding floats, storage losses need annoyingly many.\n",
    "\n",
    "# Export\n",
    "export_df = filter(r -> r.year in exportyears, final_data)\n",
    "export_df[!, roundcols] = round.(export_df[!, roundcols]; digits=dgts)\n",
    "CSV.write(\"output/heat_storage_technology_params.csv\", export_df[!, exportcols])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.11.1",
   "language": "julia",
   "name": "julia-1.11"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.11.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}