EB-NeRD dataset

EB-NeRD is Ekstra Bladet’s public news-recommendation dataset and the Danish-language substrate for this lab: it supplies article metadata, user history, and impression logs that let the local platform move from a five-row fixture to real news-recommendation scale.

Source-feed scale

The public EB-NeRD downloads below have been materialised end-to-end on a laptop-class host (Linux, aarch64, ~32 GB RAM). The numbers are real measurements, not projections. The large feed carries the ADR-0014 scale headline.

Feed	Zip size	Articles	Impressions	Distinct users	Ingest time	Raw on-disk	Staged on-disk
demo	21 MB	11,777	50,080	1,935	~1.4s	18 MB	27 MB
small	81 MB	20,738	477,534	18,827	~3.6s	75 MB	141 MB
large	3.0 GB	125,541	24,630,275	974,791	~2 min	2.8 GB	6.0 GB

CI and fast-feedback tests use checked-in JSON fixtures under tutorial/serving/fixtures/ebnerd_sample/, but those fixtures are test-only. The public dataset id is always ebnerd.

Downloaded EB-NeRD archives are read from data/source-feeds/inbox/, stage their Parquet members under data/source-feeds/staged/ebnerd/ (concatenating EB-NeRD’s train/ and validation/ splits into root-level behaviors.parquet / history.parquet via streaming DuckDB COPY), and then ingest them through a DuckDB COPY fast path that writes the raw Parquet layout in two minutes for the largest feed. The dlt pipeline is reserved for JSON fixtures, where dlt’s heterogeneous-source normalisation actually earns its keep; for typed parquet sources, DuckDB’s vectorised UNNEST is ~55× faster than dlt’s parquet → JSONL → parquet round-trip.

Canonical layout

The hand-off is: dlt -> dbt -> FastAPI -> Dagster.

• The ingest entry point lives in serving.dlt_pipeline. Source-feed Parquet goes through _duckdb_ingest_ebnerd_source_feed, which writes the raw Parquet layout directly via DuckDB COPY (including a vectorised UNNEST for the user-history explode).
• The layout owner is dataset_layout in the storage module. EB-NeRD raw resources live under data/raw/ebnerd/ebnerd_articles/, data/raw/ebnerd/ebnerd_impressions/, and data/raw/ebnerd/ebnerd_history/.
• dbt reads those same canonical globs through tutorial/serving/dbt/dbt_project.yml, while tests can still override each parquet source explicitly.
• FastAPI reads the DuckDB serving warehouse materialised from dbt, so EB-NeRD reaches /articles, /articles/{id}, /preview, and /articles/{id}/embedding through the analytical contract.
• Dagster imports the same layout-derived globs in tutorial_orchestration.definitions.

Slice cross-reference:

• E1 made the extractor and dlt/Dagster path source-feed aware.
• E2 widened the EB-NeRD staging contract and FastAPI article/preview contract.
• E3 added editor browse and search at /articles.
• E4 added the Evidence scale panel at /dashboard/scale/.
• E5 stages the global contrastive and word2vec embedding zips as optional analytical inputs.
• E6 stages the EB-NeRD testset and predictions_large_random.zip baseline for evaluation.

Extracting and ingesting

Run from the repo root:

# 1. Make sure data/source-feeds/inbox/ebnerd.zip exists.
uv run --package tutorial-serving python -m serving.ebnerd_archive

# 2. Materialise the canonical EB-NeRD pipeline.
uv run --package tutorial-orchestration dagster asset materialize -m tutorial_orchestration.definitions --select '*'

Use small or large only when the matching archive exists in data/source-feeds/inbox/ and the machine has enough free disk for the staged copy, raw Parquet output, and DuckDB serving warehouse.

Troubleshooting

Zip not in inbox. Download the EB-NeRD archives from recsys.eb.dk and place the chosen file at data/source-feeds/inbox/ebnerd.zip. Keep the expected filename; the extractor intentionally does not chase renamed downloads or fragile direct URLs.

Schema mismatch. Run dbt parse against the serving project to find the offending model before materialising the full pipeline again:

uv run --package tutorial-serving dbt parse --project-dir tutorial/serving/dbt

Out-of-disk. Check the “Raw on-disk” and “Staged on-disk” columns above before selecting a larger feed. The largest combined footprint (large) is ~9 GB on top of the original 3 GB zip in the inbox; the staged copy is expanded train+validation Parquet and the raw copy is the ingest output dbt queries.

Links

• /dashboard/scale/ shows the Evidence scale panel from E4.
• /articles opens the E3 editor browse/search surface.
• ADR-0014 records the local-first scale story.
• ADR-0016 records DuckDB and Parquet as the analytical substrate.