fix: retry failed group subpages, add rate-limit detection in scraper

- Detect Wikipedia plain-text rate-limit response ("You are making too many requests") and wait 30s before retrying, rather than silently failing - Increase inter-attempt delay from 3s to 15s per attempt - Increase group subpage delay from 1.2s to 3s, year delay from 0.6s to 2s - Re-scrape 1982, 1998, 2002, 2006 which had failed groups; all groups now complete — e.g. 2002 now has 64 matches including Group E (Germany/Klose) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-15 18:44:54 +02:00
parent d37ebe201e
commit 7e4bf2d07c
18 changed files with 1193 additions and 16 deletions
@@ -81,7 +81,7 @@ async function main() {
      const { matches, stadiums, groups, meta } = await scrapeYear(year, mainHtml)
      writeMatches(year, matches, stadiums, groups, meta)
      process.stdout.write(`${matches.length} matches`)
-      await delay(600)
+      await delay(2000)
    }

    if (doSquads) {