valknar/worldcup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: retry failed group subpages, add rate-limit detection in scraper

Browse Source 
- Detect Wikipedia plain-text rate-limit response ("You are making too many
  requests") and wait 30s before retrying, rather than silently failing
- Increase inter-attempt delay from 3s to 15s per attempt
- Increase group subpage delay from 1.2s to 3s, year delay from 0.6s to 2s
- Re-scrape 1982, 1998, 2002, 2006 which had failed groups; all groups now
  complete — e.g. 2002 now has 64 matches including Group E (Germany/Klose)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Sebastian Krüger
2026-06-15 18:44:54 +02:00
parent d37ebe201e
commit 7e4bf2d07c
18 changed files with 1193 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
data/1998/worldcup.json
+185
View File
@@ -857,6 +857,191 @@
],
"ground": "Stade de la Beaujoire, Nantes"
},
{
"round": "Group stage",
"group": "Group C",
"date": "1998-06-12",
"time": "17:30",
"team1": "Saudi Arabia",
"team2": "Denmark",
"score": {
"ft": [
0,
1
]
},
"goals2": [
{
"name": "Marc Rieper",
"minute": 69
}
],
"ground": "Stade Félix-Bollaert, Lens"
},
{
"round": "Group stage",
"group": "Group C",
"date": "1998-06-12",
"time": "21:00",
"team1": "France",
"team2": "South Africa",
"score": {
"ft": [
3,
0
]
},
"goals1": [
{
"name": "Christophe Dugarry",
"minute": 36
},
{
"name": "Pierre Issa",
"minute": 77,
"owngoal": true
},
{
"name": "Thierry Henry",
"minute": 90,
"offset": 2
}
],
"ground": "Stade Vélodrome, Marseille"
},
{
"round": "Group stage",
"group": "Group C",
"date": "1998-06-18",
"time": "17:30",
"team1": "South Africa",
"team2": "Denmark",
"score": {
"ft": [
1,
1
]
},
"goals1": [
{
"name": "Benni McCarthy",
"minute": 51
}
],
"goals2": [
{
"name": "Allan Nielsen",
"minute": 12
}
],
"ground": "Stade de Toulouse, Toulouse"
},
{
"round": "Group stage",
"group": "Group C",
"date": "1998-06-18",
"time": "21:00",
"team1": "France",
"team2": "Saudi Arabia",
"score": {
"ft": [
4,
0
]
},
"goals1": [
{
"name": "Thierry Henry",
"minute": 37
},
{
"name": "Thierry Henry",
"minute": 78
},
{
"name": "David Trezeguet",
"minute": 68
},
{
"name": "Bixente Lizarazu",
"minute": 85
}
],
"ground": "Stade de France, Saint-Denis"
},
{
"round": "Group stage",
"group": "Group C",
"date": "1998-06-24",
"time": "16:00",
"team1": "France",
"team2": "Denmark",
"score": {
"ft": [
2,
1
]
},
"goals1": [
{
"name": "Youri Djorkaeff",
"minute": 12,
"penalty": true
},
{
"name": "Emmanuel Petit",
"minute": 56
}
],
"goals2": [
{
"name": "Michael Laudrup",
"minute": 42,
"penalty": true
}
],
"ground": "Stade Gerland, Lyon"
},
{
"round": "Group stage",
"group": "Group C",
"date": "1998-06-24",
"time": "16:00",
"team1": "South Africa",
"team2": "Saudi Arabia",
"score": {
"ft": [
2,
2
]
},
"goals1": [
{
"name": "Shaun Bartlett",
"minute": 18
},
{
"name": "Shaun Bartlett",
"minute": 90,
"offset": 3,
"penalty": true
}
],
"goals2": [
{
"name": "Sami Al-Jaber",
"minute": 45,
"offset": 2,
"penalty": true
},
{
"name": "Yousuf Al-Thunayan",
"minute": 74,
"penalty": true
}
],
"ground": "Parc Lescure, Bordeaux"
},
{
"round": "Group stage",
"group": "Group D",