valknar/worldcup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: retry failed group subpages, add rate-limit detection in scraper

Browse Source 
- Detect Wikipedia plain-text rate-limit response ("You are making too many
  requests") and wait 30s before retrying, rather than silently failing
- Increase inter-attempt delay from 3s to 15s per attempt
- Increase group subpage delay from 1.2s to 3s, year delay from 0.6s to 2s
- Re-scrape 1982, 1998, 2002, 2006 which had failed groups; all groups now
  complete — e.g. 2002 now has 64 matches including Group E (Germany/Klose)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Sebastian Krüger
2026-06-15 18:44:54 +02:00
parent d37ebe201e
commit 7e4bf2d07c
18 changed files with 1193 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
data/2022/worldcup.json
+171
View File
@@ -540,6 +540,177 @@
],
"ground": "Lusail Stadium, Lusail"
},
{
"round": "Group stage",
"group": "Group A",
"date": "2022-11-20",
"time": "19:00",
"team1": "Qatar",
"team2": "Ecuador",
"score": {
"ft": [
0,
2
]
},
"goals2": [
{
"name": "Enner Valencia",
"minute": 16,
"penalty": true
},
{
"name": "Enner Valencia",
"minute": 31
}
],
"ground": "Al Bayt Stadium, Al Khor"
},
{
"round": "Group stage",
"group": "Group A",
"date": "2022-11-21",
"time": "19:00",
"team1": "Senegal",
"team2": "Netherlands",
"score": {
"ft": [
0,
2
]
},
"goals2": [
{
"name": "Cody Gakpo",
"minute": 84
},
{
"name": "Davy Klaassen",
"minute": 90,
"offset": 9
}
],
"ground": "Al Thumama Stadium, Doha"
},
{
"round": "Group stage",
"group": "Group A",
"date": "2022-11-25",
"time": "16:00",
"team1": "Qatar",
"team2": "Senegal",
"score": {
"ft": [
1,
3
]
},
"goals1": [
{
"name": "Mohammed Muntari",
"minute": 78
}
],
"goals2": [
{
"name": "Boulaye Dia",
"minute": 41
},
{
"name": "Famara Diédhiou",
"minute": 48
},
{
"name": "Bamba Dieng",
"minute": 84
}
],
"ground": "Al Thumama Stadium, Doha"
},
{
"round": "Group stage",
"group": "Group A",
"date": "2022-11-25",
"time": "19:00",
"team1": "Netherlands",
"team2": "Ecuador",
"score": {
"ft": [
1,
1
]
},
"goals1": [
{
"name": "Cody Gakpo",
"minute": 6
}
],
"goals2": [
{
"name": "Enner Valencia",
"minute": 49
}
],
"ground": "Khalifa International Stadium, Al Rayyan"
},
{
"round": "Group stage",
"group": "Group A",
"date": "2022-11-29",
"time": "18:00",
"team1": "Ecuador",
"team2": "Senegal",
"score": {
"ft": [
1,
2
]
},
"goals1": [
{
"name": "Moisés Caicedo",
"minute": 67
}
],
"goals2": [
{
"name": "Ismaïla Sarr",
"minute": 44,
"penalty": true
},
{
"name": "Kalidou Koulibaly",
"minute": 70
}
],
"ground": "Khalifa International Stadium, Al Rayyan"
},
{
"round": "Group stage",
"group": "Group A",
"date": "2022-11-29",
"time": "18:00",
"team1": "Netherlands",
"team2": "Qatar",
"score": {
"ft": [
2,
0
]
},
"goals1": [
{
"name": "Cody Gakpo",
"minute": 26
},
{
"name": "Frenkie de Jong",
"minute": 49
}
],
"ground": "Al Bayt Stadium, Al Khor"
},
{
"round": "Group stage",
"group": "Group B",