valknar/worldcup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: retry failed group subpages, add rate-limit detection in scraper

Browse Source 
- Detect Wikipedia plain-text rate-limit response ("You are making too many
  requests") and wait 30s before retrying, rather than silently failing
- Increase inter-attempt delay from 3s to 15s per attempt
- Increase group subpage delay from 1.2s to 3s, year delay from 0.6s to 2s
- Re-scrape 1982, 1998, 2002, 2006 which had failed groups; all groups now
  complete — e.g. 2002 now has 64 matches including Group E (Germany/Klose)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Sebastian Krüger
2026-06-15 18:44:54 +02:00
parent d37ebe201e
commit 7e4bf2d07c
18 changed files with 1193 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
data/2002/worldcup.json
+181
View File
@@ -1154,6 +1154,187 @@
],
"ground": "Daejeon World Cup Stadium, Daejeon"
},
{
"round": "Group stage",
"group": "Group E",
"date": "2002-06-01",
"time": "15:30",
"team1": "Republic of Ireland",
"team2": "Cameroon",
"score": {
"ft": [
1,
1
]
},
"goals1": [
{
"name": "Matt Holland",
"minute": 52
}
],
"goals2": [
{
"name": "Patrick M'Boma",
"minute": 39
}
],
"ground": "Niigata Stadium, Niigata"
},
{
"round": "Group stage",
"group": "Group E",
"date": "2002-06-01",
"time": "20:30",
"team1": "Germany",
"team2": "Saudi Arabia",
"score": {
"ft": [
8,
0
]
},
"goals1": [
{
"name": "Miroslav Klose",
"minute": 20
},
{
"name": "Miroslav Klose",
"minute": 25
},
{
"name": "Miroslav Klose",
"minute": 70
},
{
"name": "Michael Ballack",
"minute": 40
},
{
"name": "Carsten Jancker",
"minute": 45,
"offset": 1
},
{
"name": "Thomas Linke",
"minute": 73
},
{
"name": "Oliver Bierhoff",
"minute": 84
},
{
"name": "Bernd Schneider",
"minute": 90,
"offset": 1
}
],
"ground": "Sapporo Dome, Sapporo"
},
{
"round": "Group stage",
"group": "Group E",
"date": "2002-06-05",
"time": "20:30",
"team1": "Germany",
"team2": "Republic of Ireland",
"score": {
"ft": [
1,
1
]
},
"goals1": [
{
"name": "Miroslav Klose",
"minute": 19
}
],
"goals2": [
{
"name": "Robbie Keane",
"minute": 90,
"offset": 2
}
],
"ground": "Kashima Soccer Stadium, Ibaraki"
},
{
"round": "Group stage",
"group": "Group E",
"date": "2002-06-06",
"time": "18:00",
"team1": "Cameroon",
"team2": "Saudi Arabia",
"score": {
"ft": [
1,
0
]
},
"goals1": [
{
"name": "Samuel Eto'o",
"minute": 66
}
],
"ground": "Saitama Stadium, Saitama"
},
{
"round": "Group stage",
"group": "Group E",
"date": "2002-06-11",
"time": "20:30",
"team1": "Cameroon",
"team2": "Germany",
"score": {
"ft": [
0,
2
]
},
"goals2": [
{
"name": "Marco Bode",
"minute": 50
},
{
"name": "Miroslav Klose",
"minute": 79
}
],
"ground": "Ecopa Stadium, Shizuoka"
},
{
"round": "Group stage",
"group": "Group E",
"date": "2002-06-11",
"time": "20:30",
"team1": "Saudi Arabia",
"team2": "Republic of Ireland",
"score": {
"ft": [
0,
3
]
},
"goals2": [
{
"name": "Robbie Keane",
"minute": 7
},
{
"name": "Gary Breen",
"minute": 61
},
{
"name": "Damien Duff",
"minute": 87
}
],
"ground": "International Stadium Yokohama, Yokohama"
},
{
"round": "Group stage",
"group": "Group F",