demand_scaling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scaling and reorganizing heating/cooling demand data.\n",
    "\n",
    "This script reads, normalises, and reorganises the raw ArchetypeBuildingModel output\n",
    "for Mopo WP5.\n",
    "Depending on the exact needs, these scripts might need quite a bit of tinkering later on as well, but let's do something for now."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Julia environment setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Julia environment setup\n",
    "\n",
    "using Pkg\n",
    "Pkg.activate(@__DIR__)\n",
    "Pkg.instantiate()\n",
    "\n",
    "using CSV\n",
    "using DataFrames\n",
    "using Dates\n",
    "using Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read, reorganise, and normalise the raw ABM demand data.\n",
    "\n",
    "Essentially, the demand profiles here are normalised using their yearly sums.\n",
    "Whether this is the way WP5 eventually wants this data is still a bit unclear,\n",
    "but at least scaling and rescaling should be easy based on these.\n",
    "\n",
    "Furthermore, scaled normalised values are calculated based on the average yearly\n",
    "demands across all available weather years.\n",
    "This approach aims to preserve variability between weather years, while still allowing them to be scaled to match future projections.\n",
    "\n",
    "Essentially, we're sort of emulating what Hotmaps projections do, using average HDD and CDD from 2002-2012 as the basis for their heating and cooling demand calculations.\n",
    "However, actual HDD and CDD-based scaling suffers from data availability issues outside EU like so many other data sources.\n",
    "This is not perfect, but hopefully it will do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Config for normalising the raw data\n",
    "\n",
    "years = [1995, 2008, 2009, 2012, 2015]\n",
    "dem_paths = [\n",
    "    year => \"input-data/abm-raw-data/ideal_demands_$(year).csv\"\n",
    "    for year in years\n",
    "];"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Read, organise, and normalise demands data.\n",
    "\n",
    "# Read and organise the data.\n",
    "dem_cols = [:timestamp, :weather_year, :country, :category, :demand, :value]\n",
    "dem_data = DataFrame()\n",
    "for (year, path) in dem_paths\n",
    "    df = stack(DataFrame(CSV.File(path; header=[1,2])))\n",
    "    s = split.(df[!, :variable], '_')\n",
    "    df[!, :country] = String.(getindex.(s, 1))\n",
    "    df[!, :category] = String.(getindex.(s, 2))\n",
    "    df[!, :demand] = String.(getindex.(s, 6))\n",
    "    df[!, :weather_year] .= year\n",
    "    rename!(\n",
    "        df,\n",
    "        :building_archetype_building_process => :timestamp,\n",
    "    )\n",
    "    append!(dem_data, df[!, dem_cols])\n",
    "end\n",
    "\n",
    "# Add normalised data\n",
    "dem_sums = combine( # Calculate yearly sums\n",
    "    groupby(\n",
    "        dem_data,\n",
    "        dem_cols[2:end-1] # Skip timestamp and value.\n",
    "    ),\n",
    "    :value => sum\n",
    ")\n",
    "dem_sums_means = combine( # Calculate mean of yearly sums\n",
    "    groupby(\n",
    "        dem_sums,\n",
    "        dem_cols[3:end-1] # Group by :country, :category, and :demand\n",
    "    ),\n",
    "    :value_sum => mean\n",
    ")\n",
    "leftjoin!(dem_data, dem_sums, on=dem_cols[2:end-1])\n",
    "leftjoin!(dem_data, dem_sums_means, on=dem_cols[3:end-1])\n",
    "dem_data[!, :value_norm] = dem_data.value ./ dem_data.value_sum\n",
    "dem_data[!, :scaling_factor] = dem_data.value_sum ./ dem_data.value_sum_mean\n",
    "dem_data[!, :value_norm_scaled] = dem_data.value_norm .* dem_data.scaling_factor\n",
    "describe(dem_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Check that normalisation worked as intended\n",
    "\n",
    "dem_normalised_sums = combine(\n",
    "    groupby(\n",
    "        dem_data,\n",
    "        dem_cols[2:end-1]\n",
    "    ),\n",
    "    :value_norm => sum\n",
    ")\n",
    "all(dem_normalised_sums.value_norm_sum .≈ 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Fetch the ratios of peak demand to the yearly demand\n",
    "\n",
    "dem_normalised_peak_demands = combine(\n",
    "    groupby(\n",
    "        dem_data,\n",
    "        [:country, :demand]\n",
    "    ),\n",
    "    :value_norm => maximum\n",
    ")\n",
    "describe(dem_normalised_peak_demands)\n",
    "\n",
    "# These can be used for estimating heat-only boiler capacity later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read, organise, and process Hotmaps scenario data for demand scaling.\n",
    "\n",
    "The next step is to read the Hotmaps scenario data and process it to a format for easy scaling.\n",
    "\n",
    "\n",
    "### Norway and Switzerland missing...\n",
    "\n",
    "Turns out Norway and Switzerland are in fact **not** included in the Hotmaps scenario data,\n",
    "despite their indices showing up in the .csv files.\n",
    "They have `missing` data, so no actual numbers for the heating and cooling demand.\n",
    "Guess I'll have to use neighbouring countries scaled using population as the basis again..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Configure reading Hotmaps data\n",
    "\n",
    "hm_current_path = \"input-data/scen_current_building_demand/data/scen_current_building_demand.csv\"\n",
    "hm_ambitious_path = \"input-data/scen_ambitious_building_demand/data/scen_ambitious_building_demand.csv\";"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Read and organise the Hotmaps data\n",
    "\n",
    "hm_current_data_raw = dropmissing!(DataFrame(CSV.File(hm_current_path)))\n",
    "hm_ambitious_data_raw = dropmissing!(DataFrame(CSV.File(hm_ambitious_path)))\n",
    "hm_data = vcat(hm_current_data_raw, hm_ambitious_data_raw)\n",
    "rename!( # Column renaming for dataset compatibility.\n",
    "    hm_data,\n",
    "    Dict(\n",
    "        :NUTS0_code => :country,\n",
    "        :Year => :scenario_year,\n",
    "        :Value => :scenario_value,\n",
    "        :Unit => :unit, \n",
    "    )\n",
    ")\n",
    "replace!( # Country renaming to align datasets.\n",
    "    hm_data[!, :country],\n",
    "    \"GB\" => \"UK\",\n",
    "    \"GR\" => \"EL\"\n",
    ")\n",
    "supertype_to_category_map = Dict( # Map supertype to category\n",
    "    \"Residential\" => \"res\",\n",
    "    \"Nonresidential_private\" => \"nonres\",\n",
    "    \"Nonresidential_public\" => \"nonres\"\n",
    ")\n",
    "hm_data[!, :category] = map(\n",
    "    x -> get(supertype_to_category_map, x, missing),\n",
    "    hm_data[!, :Supertype]\n",
    ")\n",
    "type_to_demand_map = Dict( # Map types to demands, drop \"auxiliary energy demand\"\n",
    "    \"space heating\" => \"heating\",\n",
    "    \"cooling\" => \"cooling\",\n",
    "    \"hot water\" => \"DHW\"\n",
    ")\n",
    "hm_data[!, :demand] = map(\n",
    "    x -> get(type_to_demand_map, x, missing),\n",
    "    hm_data[!, :Type]\n",
    ")\n",
    "dropmissing!(hm_data)\n",
    "\n",
    "\n",
    "## Add missing data for Norway and Switzerland based on Sweden and Austria respectively.\n",
    "\n",
    "extrapolation_mappings = Dict( # Map Norway and Switzerland to existing data with population-based coefficients\n",
    "    \"NO\" => (\"SE\", 0.52),\n",
    "    \"CH\" => (\"AT\", 0.97)\n",
    ")\n",
    "for (c1, (c2, coeff)) in extrapolation_mappings\n",
    "    df = filter(r -> r.country == c2, hm_data)\n",
    "    df.country .= c1\n",
    "    df[!, :scenario_value] .*= coeff\n",
    "    append!(hm_data, df)\n",
    "end\n",
    "describe(hm_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Aggregate total distributed heating demands across the desired dims.\n",
    "\n",
    "agg_cols = [:Scenario, :scenario_year, :country, :category, :demand, :unit]\n",
    "hm_sums = combine(\n",
    "    groupby(\n",
    "        filter(r -> r.Fuel != \"District heating\", hm_data),\n",
    "        agg_cols\n",
    "    ),\n",
    "    :scenario_value => sum\n",
    ")\n",
    "hm_sums.Fuel .= \"distributed heating\"\n",
    "\n",
    "# Include district heating rows\n",
    "dh_sums = combine(\n",
    "    groupby(\n",
    "        filter(r -> r.Fuel == \"District heating\", hm_data),\n",
    "        agg_cols\n",
    "    ),\n",
    "    :scenario_value => sum\n",
    ")\n",
    "dh_sums.Fuel .= \"district heating\"\n",
    "hm_sums = vcat(hm_sums, dh_sums)\n",
    "describe(hm_sums)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Combine datasets, calculate scaled demand, and export desired timeseries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Export timeseries\n",
    "\n",
    "# Export config\n",
    "years = [1995, 2008, 2009, 2012, 2015]\n",
    "dgts = 4 # Number of digits when rounding exports\n",
    "categories = [\"res\", \"nonres\"]\n",
    "demands = [\"heating\", \"cooling\", \"DHW\"]\n",
    "index_cols = [:timestamp, :country]\n",
    "\n",
    "# Export demand timeseries csvs\n",
    "for cat in categories\n",
    "    for dem in demands\n",
    "        for year in years\n",
    "            df = filter(\n",
    "                r -> r.weather_year == year && r.category == cat && r.demand == dem,\n",
    "                dem_data\n",
    "            )\n",
    "            df = df[!, vcat(index_cols, :value_norm_scaled)]\n",
    "            df.value_norm_scaled = round.(df.value_norm_scaled .* 1000; digits=dgts) # Scaling to MW/GWh\n",
    "            df = sort!(unstack(df[!, vcat(index_cols, :value_norm_scaled)], :country, :value_norm_scaled))\n",
    "            CSV.write(\"output/demand-timeseries/$(dem)_$(cat)_wy$(year)_normalised_MW_GWh.csv\", df)\n",
    "        end\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Export scenario demands\n",
    "\n",
    "# Config\n",
    "dgts = 2 # Number of digits when rounding exports.\n",
    "\n",
    "# Export table\n",
    "hm_export = rename(\n",
    "    hm_sums,\n",
    "    Dict(\n",
    "        :Scenario => :scenario,\n",
    "        :category => :building_category,\n",
    "        :Fuel => :demand_category,\n",
    "    )\n",
    ")\n",
    "hm_export = sort(hm_export[!, [:scenario, :scenario_year, :country, :building_category, :demand_category, :demand, :unit, :scenario_value_sum]])\n",
    "hm_export.scenario_value_sum = round.(hm_export.scenario_value_sum; digits=dgts)\n",
    "hm_export = unstack(hm_export, :country, :scenario_value_sum)\n",
    "CSV.write(\"output/scenario_total_yearly_demands_GWh.csv\", hm_export)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Export country-wise peak-to-yearly-demand ratios\n",
    "\n",
    "# Config\n",
    "dgts = 4 # Digits for rounding peak demands\n",
    "\n",
    "# Export\n",
    "export_df = sort(dem_normalised_peak_demands)\n",
    "export_df.value_norm_maximum = round.(\n",
    "    export_df.value_norm_maximum .* 1000; digits=dgts\n",
    ") # Round and convert to MW/GWh\n",
    "export_df.unit .= \"MW/GWh\"\n",
    "export_df = unstack(export_df, :country, :value_norm_maximum)\n",
    "CSV.write(\n",
    "    \"output/peak_to_yearly_demand_ratios_MW_GWh.csv\",\n",
    "    export_df\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot diagnostics to see how the data looks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Scale demands based on select scenario year.\n",
    "\n",
    "scenario_year = 2025 # Select scenario year\n",
    "scenario = \"current\" # Select scenario\n",
    "\n",
    "scen_data = filter( # Filter the desired scenario and year\n",
    "    r -> r.scenario_year == scenario_year && r.Scenario == scenario,\n",
    "    hm_sums\n",
    ")\n",
    "scen_data = combine( # Sum \"district heating\" and \"distributed heating\"\n",
    "    groupby(scen_data, [:country, :category, :demand]),\n",
    "    :scenario_value_sum => sum\n",
    ")\n",
    "full_data = leftjoin(dem_data, scen_data, on=[:country, :category, :demand])\n",
    "full_data.final_scaled_value_MW = full_data.value_norm_scaled .* full_data.scenario_value_sum_sum .* 1000\n",
    "describe(full_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Plot diagnostics\n",
    "\n",
    "using StatsPlots\n",
    "\n",
    "plot_year = 1995\n",
    "plot_cols = [:timestamp, :category, :final_scaled_value_MW]\n",
    "countries = Set(full_data.country)\n",
    "f = nothing # How many timesteps to plot, `nothing` plots all\n",
    "\n",
    "for c in countries\n",
    "    for dem in demands\n",
    "        df = filter(\n",
    "            r -> r.weather_year == plot_year && r.demand == dem && r.country == c,\n",
    "            full_data\n",
    "        )\n",
    "        df = df[!, plot_cols]\n",
    "        df = unstack(df, :category, :final_scaled_value_MW)\n",
    "        isnothing(f) ? nothing : df = first(df, f)\n",
    "        plt = plot(\n",
    "            df[!, 1],\n",
    "            hcat(df[!, 2], df[!, 3]);\n",
    "            title = \"$(c) $(dem)\",\n",
    "            labels = reshape(names(df)[2:3], 1, 2)\n",
    "        )\n",
    "        display(plt)\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## plot When2Heat data for comparison\n",
    "\n",
    "f = 168\n",
    "w2h_data = DataFrame(CSV.File(\"input-data/when2heat/spaceHeat_2025_hour_country_1995.csv\"))\n",
    "for c in intersect(Set(w2h_data.country), Set(full_data.country))\n",
    "    # ABM data\n",
    "    df = filter(\n",
    "        r -> r.weather_year == 1995 && r.demand == \"heating\" && r.country == c,\n",
    "        full_data\n",
    "    )\n",
    "    df = df[!, plot_cols]\n",
    "    df = combine(\n",
    "        groupby(df, :timestamp),\n",
    "        :final_scaled_value_MW => sum\n",
    "    )\n",
    "    df.final_scaled_value_MW_sum ./= maximum(df.final_scaled_value_MW_sum)\n",
    "    isnothing(f) ? nothing : df = first(df, f)\n",
    "    plt = plot(\n",
    "        df[!, 1],\n",
    "        df[!, 2];\n",
    "        title = \"$(c)\",\n",
    "        label = \"abm\"\n",
    "    )\n",
    "\n",
    "    # When2Heat data\n",
    "    df2 = filter(r -> r.country == c, w2h_data)\n",
    "    isnothing(f) ? nothing : df2 = first(df2, f)\n",
    "    plot!(\n",
    "        plt,\n",
    "        df[!, 1],\n",
    "        df2[!, 3];\n",
    "        label = \"w2h\"\n",
    "    )\n",
    "    display(plt)\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Not great, not terrible.**\n",
    "\n",
    "There's a pretty significant difference between the heating demands from\n",
    "ArchetypeBuildingModel and the when2heat datasets.\n",
    "In a sense, it's not that surprising, as the they use wildly different\n",
    "approaches for calculating the heating demands.\n",
    "\n",
    "Overall, I would still expect when2heat data to be more reliable,\n",
    "since from what I understand it is a heating-degree-day-approach\n",
    "based on actual gas-use measurements from Germany.\n",
    "However, I'm a bit sceptical on how well the when2heat timeseries apply\n",
    "to countries with different heating preferences _(e.g. the Nordics)_.\n",
    "\n",
    "Not sure if ArchetypeBuildingModel really should even be used for\n",
    "something like this, but for now it seems to be what we're stuck with."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.11.2",
   "language": "julia",
   "name": "julia-1.11"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}