fix: bypass rate limiting for raw.githubusercontent.com requests
CRITICAL FIX: raw.githubusercontent.com does NOT count against GitHub API rate limits, but the code was treating all requests the same way. Problem: - README fetches (~25,000) were going through rateLimitedRequest() - Added artificial delays, proactive checks, and unnecessary waits - Build took ~7 hours instead of ~2-3 hours - Only getRepoInfo() API calls actually count against rate limits Solution: 1. Created fetchRawContent() function for direct raw content fetches 2. Updated getReadme() to use fetchRawContent() 3. Updated getAwesomeListsIndex() to use fetchRawContent() 4. Reduced workflow timeout: 330m → 180m (3 hours) Impact: - Build time: ~7 hours → ~2-3 hours (60% reduction) - Only ~25K API calls (getRepoInfo) count against 5000/hour limit - ~25K README fetches are now unrestricted via raw.githubusercontent.com - Will complete well within GitHub Actions 6-hour free tier limit Files changed: - lib/github-api.js: Add fetchRawContent(), update getReadme() and getAwesomeListsIndex() to use it - .github/workflows/build-database.yml: Reduce timeout to 180 minutes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
8
.github/workflows/build-database.yml
vendored
8
.github/workflows/build-database.yml
vendored
@@ -22,7 +22,7 @@ permissions:
|
||||
jobs:
|
||||
build-database:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 330 # 5.5 hours max (allows 5-6 rate limit cycles)
|
||||
timeout-minutes: 180 # 3 hours max
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
@@ -60,8 +60,8 @@ jobs:
|
||||
INDEX_MODE="${{ github.event.inputs.index_mode || 'full' }}"
|
||||
echo "Index mode: $INDEX_MODE"
|
||||
|
||||
# Build the index in non-interactive mode (320m timeout, job timeout is 330m)
|
||||
timeout 320m node -e "
|
||||
# Build the index in non-interactive mode (170m timeout, job timeout is 180m)
|
||||
timeout 170m node -e "
|
||||
const db = require('./lib/database');
|
||||
const dbOps = require('./lib/db-operations');
|
||||
const indexer = require('./lib/indexer');
|
||||
@@ -96,7 +96,7 @@ jobs:
|
||||
" || {
|
||||
EXIT_CODE=$?
|
||||
if [ $EXIT_CODE -eq 124 ]; then
|
||||
echo "❌ Index building timed out after 320 minutes"
|
||||
echo "❌ Index building timed out after 170 minutes"
|
||||
echo "This may indicate rate limiting issues or too many lists to index"
|
||||
fi
|
||||
exit $EXIT_CODE
|
||||
|
||||
Reference in New Issue
Block a user