valknar/worldcup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: retry failed group subpages, add rate-limit detection in scraper

Browse Source 
- Detect Wikipedia plain-text rate-limit response ("You are making too many
  requests") and wait 30s before retrying, rather than silently failing
- Increase inter-attempt delay from 3s to 15s per attempt
- Increase group subpage delay from 1.2s to 3s, year delay from 0.6s to 2s
- Re-scrape 1982, 1998, 2002, 2006 which had failed groups; all groups now
  complete — e.g. 2002 now has 64 matches including Group E (Germany/Klose)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Sebastian Krüger
2026-06-15 18:44:54 +02:00
parent d37ebe201e
commit 7e4bf2d07c
18 changed files with 1193 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
data/1970/worldcup.groups.json
+9
View File
@@ -26,6 +26,15 @@
"Brazil",
"Czechoslovakia"
]
},
{
"name": "Group 4",
"teams": [
"Peru",
"Bulgaria",
"Germany",
"Morocco"
]
}
]
}
data/1970/worldcup.json
+209
View File
@@ -751,6 +751,215 @@
}
],
"ground": "Estadio Jalisco, Guadalajara"
},
{
"round": "Group stage",
"group": "Group 4",
"date": "1970-06-02",
"time": "16:00",
"team1": "Peru",
"team2": "Bulgaria",
"score": {
"ft": [
3,
2
]
},
"goals1": [
{
"name": "Alberto Gallardo",
"minute": 50
},
{
"name": "Héctor Chumpitaz",
"minute": 55
},
{
"name": "Teófilo Cubillas",
"minute": 73
}
],
"goals2": [
{
"name": "Dinko Dermendzhiev",
"minute": 13
},
{
"name": "Hristo Bonev",
"minute": 49
}
],
"ground": "Estadio Nou Camp, León"
},
{
"round": "Group stage",
"group": "Group 4",
"date": "1970-06-03",
"time": "16:00",
"team1": "Germany",
"team2": "Morocco",
"score": {
"ft": [
2,
1
]
},
"goals1": [
{
"name": "Uwe Seeler",
"minute": 56
},
{
"name": "Gerd Müller",
"minute": 80
}
],
"goals2": [
{
"name": "Houmane Jarir",
"minute": 21
}
],
"ground": "Estadio Nou Camp, León"
},
{
"round": "Group stage",
"group": "Group 4",
"date": "1970-06-06",
"time": "16:00",
"team1": "Peru",
"team2": "Morocco",
"score": {
"ft": [
3,
0
]
},
"goals1": [
{
"name": "Teófilo Cubillas",
"minute": 65
},
{
"name": "Teófilo Cubillas",
"minute": 75
},
{
"name": "Roberto Challe",
"minute": 67
}
],
"ground": "Estadio Nou Camp, León"
},
{
"round": "Group stage",
"group": "Group 4",
"date": "1970-06-07",
"time": "12:00",
"team1": "Germany",
"team2": "Bulgaria",
"score": {
"ft": [
5,
2
]
},
"goals1": [
{
"name": "Reinhard Libuda",
"minute": 20
},
{
"name": "Gerd Müller",
"minute": 27
},
{
"name": "Gerd Müller",
"minute": 52,
"penalty": true
},
{
"name": "Gerd Müller",
"minute": 88
},
{
"name": "Uwe Seeler",
"minute": 70
}
],
"goals2": [
{
"name": "Asparuh Nikodimov",
"minute": 12
},
{
"name": "Todor Kolev",
"minute": 89
}
],
"ground": "Estadio Nou Camp, León"
},
{
"round": "Group stage",
"group": "Group 4",
"date": "1970-06-10",
"time": "16:00",
"team1": "Germany",
"team2": "Peru",
"score": {
"ft": [
3,
1
]
},
"goals1": [
{
"name": "Gerd Müller",
"minute": 19
},
{
"name": "Gerd Müller",
"minute": 26
},
{
"name": "Gerd Müller",
"minute": 39
}
],
"goals2": [
{
"name": "Teófilo Cubillas",
"minute": 44
}
],
"ground": "Estadio Nou Camp, León"
},
{
"round": "Group stage",
"group": "Group 4",
"date": "1970-06-11",
"time": "16:00",
"team1": "Bulgaria",
"team2": "Morocco",
"score": {
"ft": [
1,
1
]
},
"goals1": [
{
"name": "Dobromir Zhechev",
"minute": 40
}
],
"goals2": [
{
"name": "Maouhoub Ghazouani",
"minute": 61
}
],
"ground": "Estadio Nou Camp, León"
}
]
}