GB Power Gross Demand ETL Pipeline at a HIGH granularity [Guideline] (Short Articles)

This is meant to be a very short Article showing how it is possible to create a highly granular power gross demand table for GB with open-source data (registrations needed). Github Repo: https://github.com/pcpp94/elexon_pipeline_GB_demand

Understanding power demand at high granularity is essential for efficient energy planning and decision-making. My GitHub repo provides the guidelines for the first and most important steps in building an ETL pipeline to create a GB Power Gross Demand Table at the following granularities: Half-hourly data, Sectoral Split (Domestic and Non-Domestic), and GSP (Grid Supply Point) areas. The repo includes some final tables including gross demand data until 2023.

The outputs you can get from the ETL pipeline:

• GB Power Demand at a half-hourly, sectoral (Domestic, Non-domestic), GSP-area (Grid Supply Point) granularity.
• Power Settlement Demand (measured from final-users consumption).
• Includes Big Power Demand (Transmission Line connected demand) by Category.
• Embedded Power Generation at SVA (Suppliers Volume Allocation) level.
• Basis to produce Power Base Demand Volumes for non-commodity costs: CfD (Contracts for Difference), RO (Renewable Obligations), FiT (Feed in Tariffs).

Example of an Analysis showing the change in Power Demand depending on Temperature at the 14 GSPs and by Sector (Domestic, Non-domestic)

The data inputs needed by Source:

Free registration to sources needed

ELEXON Open Dataflows:

• P315: “Suppliers’ Meter Volume and MPAN counts”. This is SVA (Supplier Volume Allocation) demand only. Available through FTP.
• P114: “Settlement Reports” SVA and CVA (Central Volume Allocation) demand. Available through FTP or the APIs below [API Key found on the ELEXON Portal Profile Settings]:
  - https://downloads.elexonportal.co.uk/p114/list?key=<API_KEY>&date=2018-01-17&filter=s0142
  - https://downloads.elexonportal.co.uk/p114/list?key=<API_KEY>&date=2018-01-17&filter=c0421

ELEXON MDD (Market Domain Data):

• TLMs: Transmission Loss Multipliers. Also available through the following API:
  - https://downloads.elexonportal.co.uk/file/download/TLM_FILE?key=<API_KEY>
• LLFs: Line Loss Factors. For bulk downloading/file transfer contact llfs@elexon.co.uk.
• GCFs: Grid Correction Factors. Link #2:
  - https://www.elexonportal.co.uk/gspgroupcorrectionfactordata
• PCs: Profile Coefficients. Link #2:
  - https://www.elexonportal.co.uk/d0018

BMU Dictionaries:

• ELEXON BMRS BMUnits
• Neta Reports ELEXON BMUs

The ETL Process:

To keep it brief, the ETL framework is available in my GitHub repo. Simply follow the sequence of functions in the main() function within main.py to understand the extraction, transformation, and loading process for generating the GB Power Gross Demand tables at the specified granularities.

# main.py

def load_market_domain_data(): # Load the latest MDD tables: LLfs, TLMs, GCFs, PCs.
    load_llfs()
    load_gcf()
    load_pc()
    load_tlm()


def load_bmu_dictionary_data(): # Load the BMU dictionaries.
    load_dictionary_elexon()
    load_dictionary_enappsys()
    load_merged_dictionary()


def load_p315_p114_data(): # Main dataflows.
    load_elexon_p315_276()
    load_elexon_p315_276_exports()
    load_elexon_p315_277()
    load_elexon_p114_c042()
    load_elexon_p114_s014()


def transform_elexon_demand(): # Transformation.
    transform_elexon_p315_total()
    transform_elexon_gross_demand()
    transform_cfd_cap_mech()
    transform_ro_fit()
    transform_embedded_generation()
    transform_nd_bkdwn_approx()


def main():
    load_market_domain_data()
    load_bmu_dictionary_data()
    load_p315_p114_data()
    transform_elexon_demand()


if __name__ == '__main__':
    main()

For any further questions, contact me at:

• pcparedesp@hotmail.com
• https://www.linkedin.com/in/pablo-paredes-0b247470/