valknar/worldcup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: retry failed group subpages, add rate-limit detection in scraper

Browse Source 
- Detect Wikipedia plain-text rate-limit response ("You are making too many
  requests") and wait 30s before retrying, rather than silently failing
- Increase inter-attempt delay from 3s to 15s per attempt
- Increase group subpage delay from 1.2s to 3s, year delay from 0.6s to 2s
- Re-scrape 1982, 1998, 2002, 2006 which had failed groups; all groups now
  complete — e.g. 2002 now has 64 matches including Group E (Germany/Klose)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Sebastian Krüger
2026-06-15 18:44:54 +02:00
parent d37ebe201e
commit 7e4bf2d07c
18 changed files with 1193 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
data/1978/worldcup.groups.json
+9
View File
@@ -18,6 +18,15 @@
"Mexico"
]
},
{
"name": "Group 3",
"teams": [
"Austria",
"Spain",
"Brazil",
"Sweden"
]
},
{
"name": "Group 4",
"teams": [
data/1978/worldcup.json
+137
View File
@@ -417,6 +417,143 @@
],
"ground": "Estadio Gigante de Arroyito, Rosario"
},
{
"round": "Group stage",
"group": "Group 3",
"date": "1978-06-03",
"time": "13:45",
"team1": "Austria",
"team2": "Spain",
"score": {
"ft": [
2,
1
]
},
"goals1": [
{
"name": "Walter Schachner",
"minute": 10
},
{
"name": "Hans Krankl",
"minute": 76
}
],
"goals2": [
{
"name": "Dani",
"minute": 21
}
],
"ground": "Estadio José Amalfitani, Buenos Aires"
},
{
"round": "Group stage",
"group": "Group 3",
"date": "1978-06-03",
"time": "13:45",
"team1": "Brazil",
"team2": "Sweden",
"score": {
"ft": [
1,
1
]
},
"goals1": [
{
"name": "Reinaldo",
"minute": 45
}
],
"goals2": [
{
"name": "Thomas Sjöberg",
"minute": 37
}
],
"ground": "Estadio José Maria Minella, Mar del Plata"
},
{
"round": "Group stage",
"group": "Group 3",
"date": "1978-06-07",
"time": "13:45",
"team1": "Austria",
"team2": "Sweden",
"score": {
"ft": [
1,
0
]
},
"goals1": [
{
"name": "Hans Krankl",
"minute": 42,
"penalty": true
}
],
"ground": "Estadio José Amalfitani, Buenos Aires"
},
{
"round": "Group stage",
"group": "Group 3",
"date": "1978-06-07",
"time": "13:45",
"team1": "Brazil",
"team2": "Spain",
"score": {
"ft": [
0,
0
]
},
"ground": "Estadio José Maria Minella, Mar del Plata"
},
{
"round": "Group stage",
"group": "Group 3",
"date": "1978-06-11",
"time": "13:45",
"team1": "Spain",
"team2": "Sweden",
"score": {
"ft": [
1,
0
]
},
"goals1": [
{
"name": "Juan Manuel Asensi",
"minute": 75
}
],
"ground": "Estadio José Amalfitani, Buenos Aires"
},
{
"round": "Group stage",
"group": "Group 3",
"date": "1978-06-11",
"time": "13:45",
"team1": "Brazil",
"team2": "Austria",
"score": {
"ft": [
1,
0
]
},
"goals1": [
{
"name": "Roberto Dinamite",
"minute": 40
}
],
"ground": "Estadio José Maria Minella, Mar del Plata"
},
{
"round": "Group stage",
"group": "Group 4",
data/1978/worldcup.stadiums.json
+8
View File
@@ -24,6 +24,14 @@
"name": "Estadio Olímpico Chateau Carreras",
"city": "Córdoba"
},
{
"name": "Estadio José Amalfitani",
"city": "Buenos Aires"
},
{
"name": "Estadio José Maria Minella",
"city": "Mar del Plata"
},
{
"name": "Chateau Carreras",
"city": "Córdoba"