examine_generation_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Examining _Danish Energy Agency Technology Catalogue_ heat generation data for Mopo WP5\n",
    "\n",
    "This Jupyter Notebook contains code for examining the Danish Energy Agency data sheets.\n",
    "The main objective is to see if it we can get everything, or at least most, of what we need.\n",
    "\n",
    "\n",
    "## Julia setup\n",
    "\n",
    "The processing is done using Julia `XLSX` and `DataFrames` packages,\n",
    "so we need to do a bit of setup in the beginning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Activate (and set up) the required Julia environment\n",
    "\n",
    "using Pkg # Julia package manager.\n",
    "Pkg.activate(@__DIR__) # Activate the Julia environment in the folder this file is in (namely the `Project.toml`)\n",
    "Pkg.instantiate() # Download and install the necessary dependencies.\n",
    "\n",
    "# Load dependencies\n",
    "using XLSX\n",
    "using DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading and examining the excel data\n",
    "\n",
    "Next, we'll read the raw excel datasheets into `DataFrames` for easy processing and organization.\n",
    "Fortunately, the newer DEA technology catalogues contain the `alldata_flat` sheet,\n",
    "making them easy to read programmatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_dh_data = DataFrame(XLSX.readtable(\"input-data\\\\dea-technology-catalogues\\\\technology_data_for_el_and_dh.xlsx\", \"alldata_flat\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_heat_data = DataFrame(XLSX.readtable(\"input-data\\\\dea-technology-catalogues\\\\technology_data_heating_installations.xlsx\", \"alldata_flat\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_cc_data = DataFrame(XLSX.readtable(\"input-data\\\\dea-technology-catalogues\\\\technology_data_for_carbon_capture_transport_storage.xlsx\", \"alldata_flat\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the all the data in a relatively easy to digest format, we can check what exactly it is they contain.\n",
    "First, I just want to check if the `cat`, `priceyear`, `est`, and `year` columns are consistent between the two datasets.\n",
    "Otherwise, we'll need special treatment.\n",
    "\n",
    "First, let's check the parameter categories `cat`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dh_cat = Set(raw_dh_data[:,\"cat\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "heat_cat = Set(raw_heat_data[:,\"cat\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cc_cat = Set(raw_cc_data[:,\"cat\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "union(dh_cat, heat_cat, cc_cat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, I guess this is what could be expected. The `Electric regulation ability` category has trailing whitespaces in its name,\n",
    "doesn't match the `Regulation ability` from the electricity and district heating data,\n",
    "and the `Technology-specific data` category is missing the hyphen in the individual heat plants data.\n",
    "The carbon capture catalogue is especially bad, with both `Technical data` and `Energy/technical data`,\n",
    "as well as misspelled `Technology specific data` with trailing whitespaces.\n",
    "\n",
    "Let's check `priceyear` next:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dh_priceyear = Set(raw_dh_data[:,\"priceyear\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "heat_priceyear = Set(raw_heat_data[:,\"priceyear\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cc_priceyear = Set(raw_cc_data[:,\"priceyear\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "union(dh_priceyear, heat_priceyear, cc_priceyear)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, at least the price years are consistent.\n",
    "\n",
    "What about `est`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dh_est = Set(raw_dh_data[:,\"est\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "heat_est = Set(raw_heat_data[:,\"est\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cc_est = Set(raw_cc_data[:,\"est\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "union(dh_est, heat_est, cc_est)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The electricity and district heating catalogue and the individual heating catalogue agree,\n",
    "but again the carbon capture catalogue messes things up with mixed use of capitalization.\n",
    "Also, there's a new `Est` value, which I'm guessing is denoting an especially bad estimate.\n",
    "\n",
    "`year`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dh_year = Set(raw_dh_data[:,\"year\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "heat_year = Set(raw_heat_data[:,\"year\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cc_year = Set(raw_cc_data[:,\"year\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "union(dh_year, heat_year, cc_year)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, not terrible. The 2015 data is missing from the individual heating plants and carbon capture,\n",
    "but at least the other years are there and are consistent.\n",
    "Fortunately, we shouldn't need the 2015 data anyhow.\n",
    "\n",
    "Finally, we're interested in the parameter names, as they should tell us everything we can hope to find.\n",
    "Unfortunately, this is the set with by far the most unique entries,\n",
    "and the risk of not every similar parameter having an identical or consistent name is at its greatest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dh_par = Set(raw_dh_data[:,\"par\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "heat_par = Set(raw_heat_data[:,\"par\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cc_par = Set(raw_cc_data[:,\"par\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "union(dh_par, heat_par, cc_par)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, there's quite a lot of parameters, especially in the electricity and district heating and carbon capture data sheets.\n",
    "Worryingly, there doesn't seem to be that much overlap between the datasets,\n",
    "so we'll have to hope that everything we need can be found where we need it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "intersect(dh_par, heat_par)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "intersect(dh_par, cc_par)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well that's unfortunate.\n",
    "The few parameter names that are common between the datasets are not that important.\n",
    "Furthermore, the individual heat and carbon capture catalogues have no names in common,\n",
    "although fortunately this isn't that big of a deal.\n",
    "Seems like we'll have to do this the hard way, and search for relevant parameters using keywords instead,\n",
    "and map them between the sets.\n",
    "\n",
    "\n",
    "## Finding the desired parameters\n",
    "\n",
    "The goal of this exercise is to find values for the `CAPEX (EUR/MW)`,\n",
    "`FOM (EUR/MW/y)`, `VOM (EUR/MWh)`, `Lifetime`, `Conversion Rate (output/input)`,\n",
    "`CO2 Captured (ton CO2/MWh)`, and `Fuel Cost` for the WP5 input data excel.\n",
    "Since the parameter names don't seem to line up,\n",
    "we'll have to search for them \"by hand\".\n",
    "\n",
    "Let's start with `CAPEX`. Seems like `\"investment\"` yields relevant parameters from both tables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"investment (*total)\", lowercase(x)), dh_par)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"investment\", lowercase(x)), heat_par)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"investment\", lowercase(x)), cc_par)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Seems like `Nominal investment (*total)` is the closest thing, which I think corresponds to CAPEX anyhow.\n",
    "The units are different between the two technology catalogues, though,\n",
    "so we'll need to do some conversions from `kEUR/unit` to `MEUR/MW`.\n",
    "Regardless, CAPEX seems to be viable.\n",
    "Adding carbon-capture-related investment costs on top of generation capacity investments might also get tricky,\n",
    "as the two are using different units.\n",
    "\n",
    "Next up, we're interested in `FOM (EUR/MW/y)` costs, so let's see what we can find."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"fixed\", lowercase(x)), dh_par)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"fixed\", lowercase(x)), heat_par)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"fixed\", lowercase(x)), cc_par)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"fixed o&m\", lowercase(x)), cc_par)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Seems like this shouldn't be a massive problem,\n",
    "although we'll have to be careful whether electricity or heat capacity is the relevant one.\n",
    "Carbon capture catalogue unfortunately doesn't clearly state `(*total)` costs,\n",
    "but we can perhaps assume that the `\"Fixed O&M [EUR/tCO2/year]\"` is close?\n",
    "\n",
    "What about `VOM (EUR/MWh)`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"variable\", lowercase(x)), dh_par)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"variable\", lowercase(x)), heat_par)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"variable\", lowercase(x)), cc_par)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we have relevant parameters available,\n",
    "but we'll have to be a bit careful about the units.\n",
    "Especially `EUR/MWh_h` vs `EUR/MWh_e`.\n",
    "I'm not even sure what `EUR_MWH_i` means.\n",
    "Carbon capture costs are quite interesting, since they exclude energy costs.\n",
    "Fortunately, I think we can safely assume that any CC combined power/heat plant would use its own production for its CC.\n",
    "\n",
    "What about `Lifetime`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"lifetime\", lowercase(x)), dh_par)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"lifetime\", lowercase(x)), heat_par)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"lifetime\", lowercase(x)), cc_par)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Slightly different parameter names between the catalogues,\n",
    "but should contain what we need.\n",
    "The question becomes, do CC system lifetimes affect power plant lifetimes?\n",
    "I suppose ideally they would have to be treated as different systems,\n",
    "with different lifetimes, but that's a modelling choise.\n",
    "\n",
    "`Conversion Rate (output/input)` could be a problem?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"efficiency\", lowercase(x)), dh_par)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"efficiency\", lowercase(x)), heat_par)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"efficiency\", lowercase(x)), cc_par)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"heat  input\", lowercase(x)), cc_par)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well we have \"efficiencies\",\n",
    "which should be straight forward for individual heating plants at least.\n",
    "However, parameters for CHP plants could prove to be challenging.\n",
    "We don't have anything resembling an efficiency for carbon capture, though.\n",
    "However, there are `Heat input [MWh/tCO2]` and `Electricity input [MWh/tCO2]`,\n",
    "which I guess could be used to deduce how much the efficiency of the underlying plant suffers from self-consumption of generation for carbon capture?\n",
    "Not sure if this is worth it, though.\n",
    "\n",
    "What about Cb coefficients for CHP plants? These are required for calculating CHP plant fuel-to-heat ratios."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"cb\", lowercase(x)), dh_par)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`CO2 Captured (ton CO2/MWh)`?\n",
    "Seems like neither the electricity and district heating nor the individual heat plants have any carbon capture technologies associated with them in the catalogues.\n",
    "There's a separate catalogue for this,\n",
    "so I'm assuming we'll need to pull parameters from there?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"capacity\", lowercase(x)), cc_par)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"capture\", lowercase(x)), cc_par)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The easier parameter to work with would likely be the `CO2 capture rate, net [%]`,\n",
    "although to get `CO2 Captured (ton CO2/MWh)` we'd need to know the CO2 in per MWh.\n",
    "Regardless, this will likely require some work.\n",
    "\n",
    "Finally, what about `Fuel Cost`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"cost\", lowercase(x)), dh_par)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"cost\", lowercase(x)), heat_par)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"cost\", lowercase(x)), cc_par)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`fuel` doesn't yield any relevant parameters that aren't also found by `cost`.\n",
    "As could be expected, the technology catalogues don't really deal with them,\n",
    "as fuel costs can be seen entirely independent of the technologies themselves.\n",
    "Furthermore, I think these are irrelevant for endogenously modelled energy carriers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Individual heat data specific stuff\n",
    "\n",
    "For the individual heating systems, most of the parameters are annoyingly given\n",
    "per unit instead of per capacity.\n",
    "Thus, we'll need to extract the unit size from the data.\n",
    "\n",
    "Seems like `Heat production capacity for one unit [KW_h]` is our ticket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter(x -> occursin(\"unit\", lowercase(x)), heat_par)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusions\n",
    "\n",
    "Overall, the _DEA Technology Catalogues_ are pretty comprehensive,\n",
    "but don't contain all of the parameters we want in the exact same format we want.\n",
    "Thus, we'll have to make some assumptions, do some processing, etc. to get what we want.\n",
    "In any case, it's likely we'll have to revise some parameters or assumptions\n",
    "to get all the technologies uniform\n",
    "so doing things programmatically might've been the right call.\n",
    "\n",
    "- `CAPEX (EUR/MW)`: `Nominal investment (*total)` likely to be the best candidate, although requires some tweaking to arrive at the desired `EUR/MW` values for individual heating systems and carbon capture tech.\n",
    "\n",
    "- `FOM (EUR/MW/year)`: `Fixed O&M (*total) [EUR/MW_x/year]` seem easiest, although I'll need to figure out per what output we want these costs. Options for `x` seem to include `e` presumably for _electricity_, `h` presumably for _heat_, and `i` for _input_. Individual heating systems and carbon capture additions need some conversions to force into the desired format.\n",
    "\n",
    "- `VOM (EUR/MWh)`: `Variable O&M (*total) [EUR/MWh_x]`. Although carbon capture needs to be accounted for based on produced `tCO2`.\n",
    "\n",
    "- `Lifetime`: `Technical lifetime [years]` and `Technical economic lifetime [years]` should contain what we need. Straightforward?\n",
    "\n",
    "- `Conversion Rate (output/input)`: `Electrical efficiency (net, annual average)` and `Heat efficiency (net, annual average)` should get us most of what we need, although individual heating seems to need assumptions about the heat distribution system. Accounting for carbon capture is more challenging, though. We could maybe deduce some impacts based on the `Electricity input [MWh/tCO2]` and `Heat input [MWh/tCO2]` parameters, but likely gets tricky.\n",
    "\n",
    "- `CO2 Captured (tCO2/MWh)`: `A3] CO2 capture rate, net [%]` likely the easiest avenue, although needs some assumptions about the CO2 content in the input fuel.\n",
    "\n",
    "- `Fuel Cost`: The catalogues understandably don't consider fuel costs, as they don't really have anything to do with the technologies. These we need to obtain somewhere else, and likely need to be coordinated between different components."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.11.2",
   "language": "julia",
   "name": "julia-1.11"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}