Welcome to codema-dev projects!

Download, wrangle & explore all Irish energy datasets used by the codema-dev team

⚠️ Some projects use closed-access datasets for which you will need permission from the codema-dev team to use! Email us at codema-dev@codema.ie

Setup

Run the projects in your browser by clicking on the following buttons:

⬅️ click me to launch workspace

⬅️ click me

> Binder can take a few minutes to setup this workspace, click Build logs > show to see view the build progress.

- Double click on the project you want to open

- Right click on the README.md file, Open With > Notebook and run all cells






> ❓ Binder runs this code in the cloud for free with the help of NumFocus, if you find this useful consider donating to them here

> This link was generated using:
> - Binder on https://jupyterhub.github.io/nbgitpuller/link.html
> - environment repository = https://github.com/codema-dev/projects-sandbox
> - content repository = https://github.com/codema-dev/projects-sandbox

⬅️ click me launch workspace

💻 Running locally

How-To Guides

⚠️ Accessing closed-access data

❓ FAQ

Tools

Data Storage

All raw data is saved on both Google Drive and Amazon s3. Amazon s3 is easier to query from within code than Google Drive as it is possible to authenticate via environment variables to avoid a username/password login step. Google Drive is still used for all data manipulated by Excel or QGIS. Amazon s3 enables the sharing of data between projects by storing intermediate datasets which in most cases here change only in frequently.

Code Storage & Version Control

All code is saved to GitHub which uses git for version control: updating, reverting, branching, merging etc.

Code Engine

Getting code up and running on your local machine can be somewhat involved. Code engines such as binder or Gitpod enable running this code on cloud machines for free. They automate the building of the required installations using configuration files: environment.yml for binder and .gitpod.yml + .gitpod.Dockerfile for Gitpod.

Package Management

All Python packages are installed (mostly) from the conda-forge channel using the conda package manager.

Code

NoCode

Website

Jekyll is used to generate the website from simple text (or Markdown) files and a pre-defined template.

It creates the necessary HTML, css & JavaScript files

GitHub Pages is used to build and deploy the website from the file generated by Jekyll

Why?

In previous years all data wrangling was performed solely using Microsoft Excel. Although this is useful for small datasets, it soon becomes a burden when working with multiple, large datasets.

For example, when generating the previous residential energy estimates it was necessary to create up to 16 separate workbooks for each local authority each containing as many as 15 sheets, as the datasets were too large to fit into a single workbook. Although each workbook performed the same logic to clean and merge datasets, changing this logic meant changing all of the separate workbooks one at a time.

Moving to open-source scripting tools enabled using logic written down in scripts (or text files) to wrangle and merge data files, thus separating data from the logic operating on it. This means that if any dataset is updated, re-generating outputs is as simple as running a few scripts. Furthermore these scripts can be shared without sharing the underlying datasets.

Tools Considered

Criteria: a tool capable of modelling retrofitting hundreds of thousands of buildings to estimate energy & carbon savings, BER rating improvement and costs.

EnergyPLAN is an energy system model that works well for comparing aggregated demand against renewable supply profiles. It doesn’t, however, model individual buildings and instead requires aggregated inputs for building energy demands.

SEAI’s Dwelling Energy Assessment Procedure (DEAP) Excel model, EnergyPlus and RC_BuildingSimulator can model individual buildings using simple physics-based simulations but are difficult to scale. As a result, it is necessary to create a limited number of representative archetypes (<100) in order to use them to model building stocks. At present, archetype creation for these models is a long, manual process. To avoid this limitation some scripting libraries were experimented with to see if this process could be sped up:

• DEAP: pycel enables replacing individual building characteristics specified in a DEAP Excel model via a Python process, however, as of January 2020 pycel library didn’t support all operations performed in the DEAP spreadsheet.

• EnergyPlus: eppy enables replacing building characteristics and geomeppy geometry-specific characteristics via Python. As of September 2020 these libraries are better suited to parameterising existing models than for creating them from scratch.

RC_BuildingSimulator is a Python library and so can be easily scaled. This library wasn’t used as it is not actively maintained, cumbersome to adapt to this use case and would require some validation as to its accuracy as it is not a widely used library.

CityEnergyAnalyst also models individual buildings using physics-based simulations but is designed for district-level simulations. However, it is tied to Openstreetmaps as a data source for building geometries and ages and to swiss building standards by building age for archetypes. As of October 2020 Openstreetmaps was not as complete as in Switzerland, and decoupling CityEnergyAnalyst from it proved difficult.

As a consequence, we developed rc-building-model which re-implements the DEAP model in Python. This model was tested and validated against the DEAP Excel model for individual buildings, and implemented to easily and rapidly scale to the Dublin building stock.

Keeping the global environment.yml up to date

This environment.yml is built by merging the environment.yml from each project. Binder & GitPod use it to create a sandbox environment in which all dependencies are installed.

To update this file run:

conda env create --file environment.meta.yml --name codema-dev-projects-meta
conda activate codema-dev-projects-meta
invoke merge-environment-ymls

conda env create creates a virtual environment by reading environment.meta.yml in which invoke is defined as a dependency. invoke then runs the function merge_environment_ymls from tasks.py which merges the environment.yml from each project and from environment.meta.yml together into a single environment.yml

To speed up Binder builds, Binder reads the codema-dev/projects dependencies from a separate repository codema-dev/projects-sandbox. You must also update the environment.yml here with your newly generated environment.yml to keep Binder up to date!

Every time any file is changed Binder rebuilds the entire repository and reinstalls the dependencies. By keeping the environment and the content separate Binder only reinstalls dependencies when the dependencies change. This means that it no longer has to download & resolve dependency conflicts which can take ~20 minutes.