CRITICAL FIX: raw.githubusercontent.com does NOT count against GitHub
API rate limits, but the code was treating all requests the same way.
Problem:
- README fetches (~25,000) were going through rateLimitedRequest()
- Added artificial delays, proactive checks, and unnecessary waits
- Build took ~7 hours instead of ~2-3 hours
- Only getRepoInfo() API calls actually count against rate limits
Solution:
1. Created fetchRawContent() function for direct raw content fetches
2. Updated getReadme() to use fetchRawContent()
3. Updated getAwesomeListsIndex() to use fetchRawContent()
4. Reduced workflow timeout: 330m → 180m (3 hours)
Impact:
- Build time: ~7 hours → ~2-3 hours (60% reduction)
- Only ~25K API calls (getRepoInfo) count against 5000/hour limit
- ~25K README fetches are now unrestricted via raw.githubusercontent.com
- Will complete well within GitHub Actions 6-hour free tier limit
Files changed:
- lib/github-api.js: Add fetchRawContent(), update getReadme() and
getAwesomeListsIndex() to use it
- .github/workflows/build-database.yml: Reduce timeout to 180 minutes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
279cc2fa25