Skip to content

Configuration

All configuration is via environment variables. In local development these are loaded from .env by Docker Compose. In Lambda they are set as function environment variables.

Pipeline

Variable Default Description
S3_BUCKET eirgrid-data Target S3 bucket (or MinIO bucket)
S3_PREFIX grid-data Key prefix inside the bucket
DAYS_BACK 1 Days of history to download per run
MAX_CONCURRENCY 10 Maximum concurrent HTTP requests
REQUEST_TIMEOUT 25 HTTP timeout per request (seconds)
MAX_RETRIES 8 Retry attempts per request (5xx and network errors only)
LOG_LEVEL INFO DEBUG / INFO / WARNING / ERROR
LOG_FILE /app/logs/pipeline.log Log file path (Docker volume-mounted from host)
SOURCES (blank = all) Comma-separated source ID filter, e.g. eirgrid

Backfill mode

Two additional variables activate backfill mode, which iterates day-by-day from DATE_TO back to DATE_FROM:

Variable Format Description
DATE_FROM YYYY-MM-DD Start of backfill range (inclusive)
DATE_TO YYYY-MM-DD End of backfill range (inclusive) 
docker compose run --rm \
  -e DATE_FROM=2026-03-01 \
  -e DATE_TO=2026-03-31 \
  pipeline

API service

Variable Default Description
S3_BUCKET eirgrid-data Bucket to read summaries from
S3_PREFIX grid-data Key prefix (must match pipeline)
CACHE_TTL_SECONDS 300 Seconds between S3 cache refreshes
CORS_ORIGINS * Comma-separated allowed origins
LOG_LEVEL INFO DEBUG / INFO / WARNING / ERROR

AWS / MinIO credentials

Both the pipeline and API use boto3, which picks up credentials from the standard chain:

  1. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
  2. IAM instance role / Lambda execution role (production)

In local development, Docker Compose sets these to MinIO credentials via .env.

Routing to MinIO

When AWS_ENDPOINT_URL is set, boto3 automatically routes all S3 requests to the specified endpoint — no code changes are needed between local and production.

AWS_ENDPOINT_URL=http://minio:9000

DuckDB (used by the export layer) is configured separately in _make_con() using the same AWS_ENDPOINT_URL environment variable.

Local .env file

# Object store
MINIO_ROOT_USER=<your-username>
MINIO_ROOT_PASSWORD=<your-password>

# boto3 / pipeline
AWS_ACCESS_KEY_ID=<your-username>
AWS_SECRET_ACCESS_KEY=<your-password>
AWS_DEFAULT_REGION=us-east-1
AWS_ENDPOINT_URL=http://minio:9000

# S3 layout
S3_BUCKET=eirgrid-data
S3_PREFIX=grid-data

# Pipeline tuning
DAYS_BACK=31
MAX_CONCURRENCY=10
REQUEST_TIMEOUT=25
MAX_RETRIES=8
LOG_LEVEL=INFO

# API
CACHE_TTL_SECONDS=300
CORS_ORIGINS=*

Do not commit .env

.env is listed in .gitignore. It contains local credentials and should never be committed.

Gap detection

The daily gapcheck Lambda (see Gap Detection) is tuned through environment variables on the function. All four have sensible defaults — only override when debugging.

Variable Default Purpose
GAP_WINDOW_DAYS 31 Lookback window scanned for gaps
GAP_TRAILING_HOURS 6 Recent period excluded from detection (masks end-of-day EirGrid latency)
GAP_COOLDOWN_HOURS 3 Minimum wait between retries on the same (area, region, day)
GAP_MAX_ATTEMPTS 5 Attempts before an entry is marked permanent_failure
PIPELINE_LAMBDA_ARN (unset) ARN of the pipeline Lambda used for async backfill invokes — set by Terraform in production, unset locally so make gap-check stays read-only

Healthcheck Lambda

The daily healthcheck is configured entirely through Terraform variables — there is no .env for it.

Terraform variables

Variable Default Purpose
healthcheck_schedule cron(0 8 * * ? *) EventBridge cron (UTC) for the probe
healthcheck_base_url https://grid.lennonsec.org Dashboard base URL — /api/{area} is appended
healthcheck_alert_email [email protected] Email address subscribed to the SNS topic (requires one-time click-through confirmation)
healthcheck_staleness_hours 24 Fail when any area's latest data point is older than this

Lambda environment

These are set by Terraform from the variables above; override them only when invoking by hand.

Variable Default Purpose
HEALTHCHECK_BASE_URL from TF Base URL probed
ALERT_TOPIC_ARN from TF SNS topic for failure publishes
STALENESS_HOURS 24 Per-area staleness threshold
REQUEST_TIMEOUT 30 Per-request timeout in seconds (wind's 30-day series can exceed 10 s on a cold edge)