EirGrid Downloader¶

A data pipeline and REST API for Irish electricity grid data sourced from the EirGrid Smart Grid Dashboard.

The pipeline runs daily (as an AWS Lambda function or a local Docker container), downloads historical grid data, stores it as Parquet files in S3, then writes pre-computed JSON summaries that the FastAPI service serves to the built-in dashboard UI.

Alongside the pipeline two small auxiliary Lambdas keep the public dashboard honest: a gapcheck function that detects missing data and schedules targeted backfills (see Gap Detection), and a stdlib-only healthcheck function that probes every public endpoint once a day and emails an alert on failure or staleness.

What data is collected¶

Area	Region(s)	Resolution
Wind generation (actual + forecast)	ROI, NI, ALL	15 min
Solar generation (actual + forecast)	ROI, NI, ALL	15 min
System demand	ROI, NI, ALL	15 min
Interconnector flows (EWIC, Greenlink, Moyle, Net)	ALL	15 min
CO₂ emissions	ROI	15 min
System frequency	ROI	5 sec (hourly avg in API)
SNSP	ALL	15 min
Generation mix by fuel type	ROI	Daily snapshot

See Upstream Data for the full list of EirGrid endpoints the pipeline calls, the parameters sent, and how raw rows map into Parquet partitions and the JSON summaries served by the API. If you came here from a panel on the dashboard, the Dashboard Panels page documents each gauge, donut, bar, and chart — what it shows, where it comes from, and how it's calculated.

Quick start¶

Prerequisites: Docker, Docker Compose, uv

# Install Python dependencies
uv sync

# Start MinIO object store + run the pipeline
make up

This downloads up to 31 days of grid data, stores Parquet files in the local object store, and generates JSON summaries.

# Start the API + dashboard (in a second terminal)
make api

Service	URL
Dashboard UI	http://localhost:6502/ui/
API (Swagger)	http://localhost:6502/docs
Project docs (this site)	http://localhost:6502/guide/
Object store browser	http://localhost:6501

Repository layout¶

src/
  sources/         # EirGrid fetch logic + BaseSource ABC
  pipeline/        # Async orchestrator + Docker entrypoint
  storage/         # S3 Parquet upload with merge-on-write
  export/          # JSON summary generation via DuckDB
  api/             # FastAPI service + built-in dashboard UI
    static/        # HTML / CSS / JS (no build step)
tests/             # pytest suite (no infrastructure required)
docs/              # This documentation (MkDocs Material — served at /guide/)
Dockerfile         # Pipeline Lambda image
Dockerfile.api     # API service image
docker-compose.yml
Makefile
pyproject.toml