valknar/worldcup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: rewrite README with accurate data pipeline documentation

Browse Source 
- Replace openfootball references with Wikipedia scraper workflow
- Document all three scripts: scrape (dev), seed (init), sync (scheduled)
- Explain rate-limit handling, incremental group detection, UTC kickoff ordering
- Add NEXT_PUBLIC_SITE_URL to env vars table
- Update project structure with data/, client.tsx pattern, wiki-scraper.ts
- Add architecture notes for server/client split, dynamic sitemap, standings seeding

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Sebastian Krüger
2026-06-16 07:50:12 +02:00
parent c721062560
commit 1ebe4613ce
1 changed files with 140 additions and 69 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+140 -69
View File
@@ -1,13 +1,13 @@
# World Cup # World Cup
A full-stack World Cup statistics web app covering every tournament from 1930 to 2026. Built with Next.js 16, TailwindCSS 4, GraphQL, and PostgreSQL. Data is sourced from [openfootball/worldcup.json](https://github.com/openfootball/worldcup.json) and synced on a schedule so live 2026 results appear within minutes. A full-stack World Cup statistics web app covering every tournament from 1930 to 2026. Built with Next.js 16, TailwindCSS 4, GraphQL, and PostgreSQL. Historical data is scraped from English Wikipedia and committed to the repo; live 2026 results are synced from Wikipedia on a schedule so scores appear within minutes of the final whistle.
## Features ## Features
- **Live 2026 matches** — detected automatically when today's date matches a scheduled fixture; Apollo polls every 60 seconds for score updates - **Live 2026 matches** — detected automatically when today's date matches a scheduled fixture; Apollo polls every 60 seconds for score updates
- **All-time statistics** — goals, hat-tricks, biggest wins, highest-scoring games, penalty stats, goals-by-minute heatmap, confederation performance, title counts - **All-time statistics** — goals, hat-tricks, biggest wins, highest-scoring games, penalty stats, goals-by-minute heatmap, confederation performance, title counts
- **Group standings** — computed from match results for every tournament, pre-seeded from openfootball's standings files where available - **Group standings** — computed from match results for every tournament, with 0-row entries seeded so all groups appear even before any matches are played
- **Deep-linked pages** — every tournament, team, and player has a permanent URL (`/tournaments/1966`, `/teams/brazil`, `/players/Pelé`) - **Deep-linked pages** — every tournament, team, and player has a permanent URL (`/tournaments/1966`, `/teams/brazil`, `/players/Pelé`) with server-side metadata for SEO
- **Full-text search** — across teams, tournaments, and players - **Full-text search** — across teams, tournaments, and players
- **Squad data** — 26-man rosters for 2026 with position, shirt number, and date of birth - **Squad data** — 26-man rosters for 2026 with position, shirt number, and date of birth
- **Qualification playoffs** — 2026 inter-confederation playoff results stored separately - **Qualification playoffs** — 2026 inter-confederation playoff results stored separately
@@ -19,9 +19,9 @@ A full-stack World Cup statistics web app covering every tournament from 1930 to
| Route | Content | | Route | Content |
|---|---| |---|---|
| `/` | Home: live matches, stat pills, latest result, upcoming fixtures, Golden Boot race | | `/` | Home: live matches, stat pills, latest result, upcoming fixtures, Golden Boot race |
| `/groups` | All 12 group tables for 2026 (P/W/D/L/GD/Pts) | | `/groups` | All 12 group tables for 2026 (P/W/D/L/GD/Pts) with results and upcoming fixtures |
| `/stats` | Historical stats: goals chart, top scorers, hat-tricks, biggest wins, goals by minute, ET/shootout stats, confederation stats | | `/stats` | Historical stats: goals chart, top scorers, hat-tricks, biggest wins, goals by minute, ET/shootout stats, confederation stats |
| `/history` | All 23 tournament cards newest-first, each with host, winner, top scorer | | `/history` | All 24 tournament cards newest-first, each with host, winner, top scorer |
| `/search?q=…` | Full-text search across teams, players, tournaments | | `/search?q=…` | Full-text search across teams, players, tournaments |
| `/tournaments/[year]` | Tournament detail: group stage with standings + matches, knockout rounds, scorer sidebar | | `/tournaments/[year]` | Tournament detail: group stage with standings + matches, knockout rounds, scorer sidebar |
| `/teams/[slug]` | Team profile: all-time record, top scorers, WC appearances | | `/teams/[slug]` | Team profile: all-time record, top scorers, WC appearances |
@@ -41,26 +41,69 @@ A full-stack World Cup statistics web app covering every tournament from 1930 to
| Fonts | Bebas Neue + Space Grotesk (Google Fonts) | | Fonts | Bebas Neue + Space Grotesk (Google Fonts) |
| Container | Docker multi-stage build, Traefik-compatible | | Container | Docker multi-stage build, Traefik-compatible |
## Data sources ## Data pipeline
All data is fetched from the [openfootball/worldcup.json](https://github.com/openfootball/worldcup.json) GitHub repository via raw URLs. The sync script fetches up to seven files per tournament year depending on availability: Data flows through three scripts that are run at different times and for different purposes.
| File | Content | Years available | ### 1. Scrape — one-time developer task
|---|---|---|
| `worldcup.json` | Matches, scores (FT/HT/ET/P), goal-scorer events | All (1930–2026) |
| `worldcup.teams.json` | Team details, FIFA codes, confederation | 2014–2026 |
| `worldcup.stadiums.json` | Stadium name, city, capacity, coordinates | 2014–2026 |
| `worldcup.groups.json` | Group compositions | 2014–2026 |
| `worldcup.standings.json` | Pre-computed group standings | 2014, 2018 |
| `worldcup.squads.json` | 26-man player rosters | 2026 |
| `worldcup.quali_playoffs.json` | Inter-confederation playoff results | 2026 |
**Note:** Individual goal-scorer records are only available from openfootball for 1930–1950, 1990, 2006, and 2014–2026. Match scores (used for standings, biggest wins, etc.) are complete for all years. ```bash
pnpm scrape # all years (1930–2022), matches + squads
pnpm scrape 2002 # single year
pnpm scrape 2002 --matches # matches, meta, stadiums, groups only
pnpm scrape 2002 --squads # squads only
```
Fetches structured match data from English Wikipedia using the [MediaWiki parse API](https://en.wikipedia.org/w/api.php) and writes JSON files to `data/{year}/`. These files are **committed to git** so the production build never needs to hit Wikipedia for historical data.
Each year produces up to five files:
| File | Content |
|---|---|
| `worldcup.json` | Matches with scores (FT/HT/ET/P) and goal-scorer events |
| `worldcup.meta.json` | Tournament metadata: host, winner, runner-up, team count |
| `worldcup.stadiums.json` | Stadium names and cities |
| `worldcup.groups.json` | Group compositions (teams per group) |
| `worldcup.squads.json` | Player rosters (where available on Wikipedia) |
The scraper has built-in rate-limit handling: it detects Wikipedia's plain-text `"You are making too many requests"` response, waits 30 seconds, and retries with exponential back-off (up to 6 attempts, 15 s × attempt delay between retries). Group sub-pages are fetched with a 3-second delay between requests.
### 2. Seed — initial database population
```bash
DATABASE_URL="postgres://wc:wc@localhost:5432/worldcup" pnpm seed
DATABASE_URL="..." pnpm seed --force # drop and re-seed from scratch
```
Reads the committed `data/{year}/` JSON files and loads them into the database. Also creates all tables (if they do not exist). Intended for first-time setup and for re-seeding after schema changes. Covers **1930–2022 only** — 2026 data is handled by sync.
Seed is **idempotent** and skips silently if data is already present (unless `--force` is passed).
### 3. Sync — scheduled live updates (2026 only)
```bash
DATABASE_URL="..." pnpm sync # normal run
DATABASE_URL="..." pnpm sync --force # clear and re-fetch all 2026 data
```
Fetches the current state of the 2026 Wikipedia pages and upserts everything into the database. Historical years (1930–2022) are not touched — they come from the committed JSON files via seed.
What sync does on each run:
1. Fetches `2026_FIFA_World_Cup` via the MediaWiki API
2. Determines which groups are fully complete (all matches have FT scores) and skips their sub-pages to save requests
3. Upserts matches, scores, and goal events
4. Fetches `2026_FIFA_World_Cup_squads` and upserts squad rosters
5. Recomputes group standings from match results
6. Seeds 0-row standing entries for groups with no played matches yet (so all groups appear in the UI)
7. Updates tournament aggregates (total goals, matches played, avg goals/game)
Sync is designed to run on a **10-minute cron** in production. Each run is safe to repeat — all writes use `ON CONFLICT DO UPDATE`.
## Database schema ## Database schema
``` ```
tournaments year PK, host, winner, runner_up, third, fourth, tournaments year PK, host, winner, runner_up, third_place, fourth_place,
teams_count, matches_count, total_goals, avg_goals_per_game teams_count, matches_count, total_goals, avg_goals_per_game
teams id, name UNIQUE, iso2, fifa_code, continent, confederation teams id, name UNIQUE, iso2, fifa_code, continent, confederation
@@ -100,10 +143,13 @@ pnpm install
# 2. Start the database # 2. Start the database
docker compose -f docker-compose.dev.yml up -d docker compose -f docker-compose.dev.yml up -d
# 3. Seed all 23 tournaments # 3. Seed historical data (1930–2022) from committed JSON files
DATABASE_URL="postgres://wc:wc@localhost:5432/worldcup" pnpm seed
# 4. Sync 2026 data from Wikipedia
DATABASE_URL="postgres://wc:wc@localhost:5432/worldcup" pnpm sync DATABASE_URL="postgres://wc:wc@localhost:5432/worldcup" pnpm sync
# 4. Start the dev server # 5. Start the dev server
DATABASE_URL="postgres://wc:wc@localhost:5432/worldcup" pnpm dev DATABASE_URL="postgres://wc:wc@localhost:5432/worldcup" pnpm dev
``` ```
@@ -111,15 +157,25 @@ Open [http://localhost:3000](http://localhost:3000).
To stop the database: `docker compose -f docker-compose.dev.yml down` To stop the database: `docker compose -f docker-compose.dev.yml down`
If you need to re-scrape historical data (e.g. after a Wikipedia article correction):
```bash
pnpm scrape 2002 # re-scrape a single year
git add data/2002/ && git commit -m "chore: refresh 2002 scraped data"
```
## Environment variables ## Environment variables
| Variable | Required | Description | | Variable | Required | Description |
|---|---|---| |---|---|---|
| `DATABASE_URL` | Yes | PostgreSQL connection string | | `DATABASE_URL` | Yes | PostgreSQL connection string |
| `NEXT_PUBLIC_SITE_URL` | Production | Public base URL, e.g. `https://worldcup.example.com` — used for sitemap and OG metadata |
| `DB_PASSWORD` | Production | Password for the `wc` DB user (used by docker-compose.yml) | | `DB_PASSWORD` | Production | Password for the `wc` DB user (used by docker-compose.yml) |
| `TRAEFIK_ENABLED` | Production | Set to `true` to activate Traefik router labels | | `TRAEFIK_ENABLED` | Production | Set to `true` to activate Traefik router labels |
| `TRAEFIK_HOST` | Production | Public hostname, e.g. `worldcup.example.com` | | `TRAEFIK_HOST` | Production | Public hostname, e.g. `worldcup.example.com` |
| `NETWORK_NAME` | Production | Name of the external Docker network Traefik is attached to | | `NETWORK_NAME` | Production | Name of the external Docker network Traefik is attached to |
| `UMAMI_ID` | Optional | Umami analytics site ID |
| `UMAMI_SRC` | Optional | Umami analytics script URL |
Copy `.env.example` to `.env` and fill in the values before deploying. Copy `.env.example` to `.env` and fill in the values before deploying.
@@ -134,6 +190,7 @@ In Coolify's environment variable editor set:
``` ```
DB_PASSWORD=<strong-random-password> DB_PASSWORD=<strong-random-password>
DATABASE_URL=postgres://wc:<DB_PASSWORD>@db:5432/worldcup DATABASE_URL=postgres://wc:<DB_PASSWORD>@db:5432/worldcup
NEXT_PUBLIC_SITE_URL=https://worldcup.yourdomain.com
TRAEFIK_ENABLED=true TRAEFIK_ENABLED=true
TRAEFIK_HOST=worldcup.yourdomain.com TRAEFIK_HOST=worldcup.yourdomain.com
NETWORK_NAME=<your-traefik-network-name> NETWORK_NAME=<your-traefik-network-name>
@@ -143,12 +200,14 @@ NETWORK_NAME=<your-traefik-network-name>
Coolify builds the Docker image via `docker compose up` and attaches the container to the Traefik network automatically. TLS certificates are issued by the `resolver` cert resolver configured in Traefik. Coolify builds the Docker image via `docker compose up` and attaches the container to the Traefik network automatically. TLS certificates are issued by the `resolver` cert resolver configured in Traefik.
### 3. Initial data sync ### 3. Initial data load
After the first deployment run the sync once manually in Coolify's terminal: After the first deployment, seed historical data and then sync 2026:
```bash ```bash
docker compose exec app pnpm sync # In Coolify's terminal for the app container:
pnpm seed # loads 1930–2022 from committed JSON files
pnpm sync # fetches 2026 from Wikipedia
``` ```
### 4. Scheduled sync (live updates) ### 4. Scheduled sync (live updates)
@@ -161,73 +220,85 @@ In Coolify → your service → **Scheduled Tasks**, add:
| Schedule | `*/10 * * * *` | | Schedule | `*/10 * * * *` |
| Container | `app` | | Container | `app` |
This re-syncs from openfootball every 10 minutes. During the 2026 group stage new match results appear within 10 minutes of the final whistle. This re-syncs 2026 from Wikipedia every 10 minutes. New match results appear within 10 minutes of the final whistle.
## Running the sync manually
```bash
# From host (dev)
DATABASE_URL="postgres://wc:wc@localhost:5432/worldcup" pnpm sync
# Inside the app container (production)
docker compose exec app pnpm sync
```
The sync is fully idempotent — safe to run repeatedly. It upserts every record and recomputes tournament aggregates at the end of each year.
## Project structure ## Project structure
``` ```
worldcup/ worldcup/
├── app/ ├── app/
│ ├── layout.tsx # Root layout: nav, fonts, Apollo provider │ ├── layout.tsx # Root layout: nav, fonts, Apollo provider, global metadata
│ ├── page.tsx # Home page │ ├── robots.ts # robots.txt (Next.js convention)
│ ├── groups/page.tsx # 2026 group standings │ ├── sitemap.ts # sitemap.xml — dynamic, rendered at request time
│ ├── stats/page.tsx # All-time statistics │ ├── page.tsx # Home — server wrapper (exports metadata)
│ ├── history/page.tsx # Tournament history cards │ ├── client.tsx # Home — Apollo/interactive client component
│ ├── search/page.tsx # Full-text search │ ├── groups/
│ ├── tournaments/[year]/page.tsx # Tournament detail │ │ ├── page.tsx # Groups — server wrapper
│ ├── teams/[slug]/page.tsx # Team profile │ │ └── client.tsx # Groups — client component
│ ├── players/[name]/page.tsx # Player profile │ ├── stats/page.tsx + client.tsx
│ └── api/graphql/route.ts # GraphQL Yoga endpoint │ ├── history/page.tsx + client.tsx
│ ├── search/page.tsx + client.tsx
│ ├── tournaments/[year]/
│ │ ├── page.tsx # generateMetadata fetches tournament from DB
│ │ └── client.tsx # Tournament detail, group standings, bracket
│ ├── teams/[slug]/page.tsx + client.tsx
│ ├── players/[name]/page.tsx + client.tsx
│ └── api/graphql/route.ts # GraphQL Yoga endpoint
├── components/ ├── components/
│ ├── apollo-provider.tsx # Apollo Client provider wrapper │ ├── apollo-provider.tsx # Apollo Client provider wrapper
│ ├── nav.tsx # Top navigation bar │ ├── nav.tsx # Top navigation bar
│ ├── team-flag.tsx # flag-icons wrapper component │ ├── team-flag.tsx # flag-icons wrapper component
│ ├── match-card.tsx # Match result / fixture card │ ├── match-card.tsx # Match result / fixture card
│ └── live-badge.tsx # Pulsing LIVE indicator │ └── live-badge.tsx # Pulsing LIVE indicator
├── lib/ ├── lib/
│ ├── db/ │ ├── db/
│ │ ├── schema.ts # Drizzle table definitions │ │ ├── schema.ts # Drizzle table definitions
│ │ └── index.ts # DB connection singleton │ │ └── index.ts # DB connection singleton
│ ├── graphql/ │ ├── graphql/
│ │ ├── schema.ts # GraphQL SDL │ │ ├── schema.ts # GraphQL SDL
│ │ ├── resolvers/index.ts # All resolvers │ │ ├── resolvers/index.ts # All resolvers
│ │ ├── hooks.ts # Apollo v4 useQuery wrapper │ │ ├── hooks.ts # Apollo v4 useQuery wrapper
│ │ └── client.ts # Apollo Client factory │ │ └── client.ts # Apollo Client factory
│ └── iso-codes.ts # Team name → ISO2 country code map │ ├── wiki-scraper.ts # Wikipedia HTML parser (cheerio), rate-limit retry
│ └── iso-codes.ts # Team name → ISO2 country code map
├── scripts/ ├── scripts/
│ └── sync.ts # Data sync script (all years, idempotent) │ ├── scrape-wikipedia.ts # Developer-only: scrape Wikipedia → data/{year}/
├── docker-compose.yml # Production (Traefik + external network) │ ├── seed.ts # Initial DB load from data/{year}/ JSON files
├── docker-compose.dev.yml # Local dev (DB only, port 5432 exposed) │ └── sync.ts # Scheduled: sync 2026 live data from Wikipedia
├── Dockerfile # Multi-stage pnpm build ├── data/
├── .env.example # Environment variable template │ ├── 1930/ … 2022/ # Committed Wikipedia scrape output (per-year JSON)
├── next.config.ts # standalone output, serverExternalPackages │ └── {year}/
├── drizzle.config.ts # Drizzle Kit config │ ├── worldcup.json # Matches + goals
│ ├── worldcup.meta.json # Tournament metadata
│ ├── worldcup.stadiums.json # Stadiums
│ ├── worldcup.groups.json # Group compositions
│ └── worldcup.squads.json # Squad rosters (where available)
├── docker-compose.yml # Production (Traefik + external network)
├── docker-compose.dev.yml # Local dev (DB only, port 5432 exposed)
├── Dockerfile # Multi-stage pnpm build
├── .env.example # Environment variable template
├── next.config.ts # standalone output, serverExternalPackages
├── drizzle.config.ts # Drizzle Kit config
└── tsconfig.json └── tsconfig.json
``` ```
## Architecture notes ## Architecture notes
**Live match detection** — A match is considered live when its date equals today and the current time is within 5 minutes before kick-off to 125 minutes after. Time zones are stripped; all times are treated as local tournament time. Apollo's `pollInterval: 60_000` re-queries `liveMatches` every minute. **Live match detection** — A match is considered live when its date equals today and the current time falls within 5 minutes before kick-off to 125 minutes after. Kick-off times are stored as `"HH:MM UTC±N"` strings; the resolver computes the UTC timestamp at query time using PostgreSQL interval arithmetic. Apollo's `pollInterval: 60_000` re-queries `liveMatches` and `recentMatches` every minute.
**UTC kickoff ordering** — Both `upcomingMatches` (ascending) and `recentMatches` (descending) sort by computed UTC kickoff time using a `CASE` expression that parses the `time_local` string and subtracts the UTC offset as an interval. This ensures correct ordering across time zones — a match starting later in a westward timezone is not incorrectly ranked ahead of an earlier match with a higher database ID.
**Server/client split** — All pages use a server wrapper `page.tsx` that exports `metadata` (or `generateMetadata`) and a `client.tsx` that contains the Apollo query and interactive rendering. This lets Next.js generate accurate `<title>`, OpenGraph, and Twitter card tags for each route without requiring server-side data fetching in client components.
**`NEXT_PUBLIC_SITE_URL`** — The public hostname is read from this environment variable in `sitemap.ts`, `robots.ts`, and `layout.tsx` (`metadataBase`). All per-page `openGraph.url` values use relative paths (`/groups`, `/tournaments/2026`, etc.) which Next.js resolves against `metadataBase` automatically. The sitemap is marked `export const dynamic = 'force-dynamic'` so it runs at request time when the database is reachable, not at build time.
**Apollo Client v4** — This project uses Apollo Client 4 which moved hooks to `@apollo/client/react` and core utilities to `@apollo/client/core`. A thin wrapper in `lib/graphql/hooks.ts` re-exports `useQuery` typed as `Record<string, any>` to avoid the v4 `TData = {}` default breaking all field accesses. **Apollo Client v4** — This project uses Apollo Client 4 which moved hooks to `@apollo/client/react` and core utilities to `@apollo/client/core`. A thin wrapper in `lib/graphql/hooks.ts` re-exports `useQuery` typed as `Record<string, any>` to avoid the v4 `TData = {}` default breaking all field accesses.
**Standalone Docker output** — `next.config.ts` sets `output: 'standalone'` which produces a self-contained `server.js`. The `scripts/` and `lib/` directories are copied separately into the runner stage so `pnpm sync` works inside the container without needing a full Node/TypeScript toolchain reinstall. **Standalone Docker output** — `next.config.ts` sets `output: 'standalone'` which produces a self-contained `server.js`. The `scripts/`, `lib/`, and `data/` directories are copied separately into the runner stage so `pnpm seed` and `pnpm sync` work inside the container without needing a full Node/TypeScript toolchain reinstall.
**Group standings** — Pre-computed standings from openfootball are stored directly. For all other years (and 2026 during the tournament) standings are computed live from match results via a SQL `GROUP BY` query in the `groupStandings` resolver. **Group standings** — Standings are computed live from match results via a SQL `GROUP BY` query in the `groupStandings` resolver. After each sync, 0-row standing entries are inserted for all teams in all 2026 groups, ensuring every group appears in the UI even before its first match is played.
**Total goals** — Tournament goal counts are derived from match score totals (`score_ft_home + score_ft_away`), not from the goals table. This ensures correct numbers for all years, including those where individual scorer records are not available in the openfootball dataset. **Wikipedia scraper rate limits** — The MediaWiki API occasionally returns a plain-text `"You are making too many requests to the API"` response instead of JSON. The scraper detects this by reading the response as text first, then parses JSON only if the body does not start with that phrase. On rate-limit (or HTTP 429), it waits 30 seconds before retrying. Retries use exponential back-off: 15 s × attempt number, up to 6 attempts per page.
## GraphQL API ## GraphQL API