feat: initial commit - Scrapyd UI web interface

- Next.js 16.0.1 + React 19.2.0 + Tailwind CSS 4.1.16
- Complete Scrapyd API integration (all 12 endpoints)
- Dashboard with real-time job monitoring
- Projects management (upload, list, delete)
- Spiders management with scheduling
- Jobs monitoring with filtering and cancellation
- System status monitoring
- Dark/light theme toggle with next-themes
- Server-side authentication via environment variables
- Docker deployment with multi-stage builds
- GitHub Actions CI/CD workflow

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2025-11-05 03:32:14 +01:00
commit 971ef5426d
55 changed files with 8885 additions and 0 deletions

28
.dockerignore Normal file
View File

@@ -0,0 +1,28 @@
# Node
node_modules
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
# Next.js
.next
out
# Env
.env*.local
.env
# IDE
.vscode
.idea
.claude
# Git
.git
.gitignore
# Misc
README.md
*.md
.DS_Store

62
.github/workflows/docker-build.yml vendored Normal file
View File

@@ -0,0 +1,62 @@
name: Build and Push Docker Image
on:
push:
branches:
- main
tags:
- 'v*.*.*'
pull_request:
branches:
- main
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to GitHub Container Registry
if: github.event_name != 'pull_request'
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata (tags, labels)
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=ref,event=branch
type=ref,event=pr
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=semver,pattern={{major}}
type=sha
- name: Build and push Docker image
uses: docker/build-push-action@v5
with:
context: .
file: ./Dockerfile
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
platforms: linux/amd64,linux/arm64

41
.gitignore vendored Normal file
View File

@@ -0,0 +1,41 @@
# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
# dependencies
/node_modules
/.pnp
.pnp.*
.yarn/*
!.yarn/patches
!.yarn/plugins
!.yarn/releases
!.yarn/versions
# testing
/coverage
# next.js
/.next/
/out/
# production
/build
# misc
.DS_Store
*.pem
# debug
npm-debug.log*
yarn-debug.log*
yarn-error.log*
.pnpm-debug.log*
# env files (can opt-in for committing if needed)
.env*
# vercel
.vercel
# typescript
*.tsbuildinfo
next-env.d.ts

51
Dockerfile Normal file
View File

@@ -0,0 +1,51 @@
# Multi-stage Dockerfile for Next.js production build
# Stage 1: Dependencies
FROM node:22-alpine AS deps
RUN apk add --no-cache libc6-compat
RUN corepack enable && corepack prepare pnpm@latest --activate
WORKDIR /app
# Copy package files
COPY package.json pnpm-lock.yaml* ./
RUN pnpm install --frozen-lockfile
# Stage 2: Builder
FROM node:22-alpine AS builder
RUN corepack enable && corepack prepare pnpm@latest --activate
WORKDIR /app
# Copy dependencies from deps stage
COPY --from=deps /app/node_modules ./node_modules
COPY . .
# Set environment variables for build
ENV NEXT_TELEMETRY_DISABLED=1
# Build Next.js
RUN pnpm build
# Stage 3: Runner
FROM node:22-alpine AS runner
WORKDIR /app
ENV NODE_ENV=production
ENV NEXT_TELEMETRY_DISABLED=1
# Create non-root user
RUN addgroup --system --gid 1001 nodejs
RUN adduser --system --uid 1001 nextjs
# Copy standalone build
COPY --from=builder /app/public ./public
COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static
USER nextjs
EXPOSE 3000
ENV PORT=3000
ENV HOSTNAME="0.0.0.0"
CMD ["node", "server.js"]

20
Dockerfile.dev Normal file
View File

@@ -0,0 +1,20 @@
# Development Dockerfile with hot reload
FROM node:22-alpine
RUN corepack enable && corepack prepare pnpm@latest --activate
WORKDIR /app
# Copy package files
COPY package.json pnpm-lock.yaml* ./
# Install dependencies
RUN pnpm install
# Copy application code
COPY . .
EXPOSE 3000
CMD ["pnpm", "dev"]

283
README.md Normal file
View File

@@ -0,0 +1,283 @@
# Scrapy UI
A modern, stylish web interface for managing and monitoring [Scrapyd](https://scrapyd.readthedocs.io/) instances. Built with Next.js 15, shadcn/ui, and Tailwind CSS 4.
## Features
- **Real-time Monitoring** - Live dashboard with job statistics and system status
- **Project Management** - Upload, list, and delete Scrapy projects and versions
- **Spider Management** - Browse spiders and schedule jobs with custom arguments
- **Job Control** - Monitor running/pending/finished jobs with filtering and cancellation
- **System Status** - View Scrapyd daemon health and metrics
- **Modern UI** - Clean, responsive design with dark/light theme support
- **Secure** - Server-side authentication with environment variables
- **Docker Ready** - Multi-stage builds for production deployment
## Tech Stack
- **Next.js 15** (App Router, Server Components)
- **React 19** with Server Actions
- **TypeScript** for type safety
- **Tailwind CSS 4** for styling
- **shadcn/ui** for UI components
- **TanStack Query** for data fetching
- **Zod** for runtime validation
- **Docker** for containerization
## Prerequisites
- Node.js 22+ or Docker
- A running Scrapyd instance
- Basic auth credentials for Scrapyd
## Quick Start
### 1. Clone the repository
```bash
git clone <your-repo-url>
cd scrapy-ui
```
### 2. Configure environment variables
Copy `.env.example` to `.env.local` and update with your credentials:
```bash
cp .env.example .env.local
```
Edit `.env.local`:
```env
SCRAPYD_URL=https://scrapy.pivoine.art
SCRAPYD_USERNAME=your_username
SCRAPYD_PASSWORD=your_password
```
### 3. Run locally
#### Using pnpm:
```bash
pnpm install
pnpm dev
```
#### Using Docker Compose (development):
```bash
docker-compose -f docker-compose.dev.yml up
```
#### Using Docker Compose (production):
```bash
docker-compose up -d
```
Visit [http://localhost:3000/ui](http://localhost:3000/ui)
## Project Structure
```
scrapy-ui/
├── app/ # Next.js App Router
│ ├── (dashboard)/ # Dashboard route group
│ │ ├── page.tsx # Dashboard (/)
│ │ ├── projects/ # Projects management
│ │ ├── spiders/ # Spiders listing & scheduling
│ │ ├── jobs/ # Jobs monitoring & control
│ │ └── system/ # System status
│ ├── api/scrapyd/ # API routes (server-side)
│ │ ├── daemon/ # GET /api/scrapyd/daemon
│ │ ├── projects/ # GET/DELETE /api/scrapyd/projects
│ │ ├── versions/ # GET/POST/DELETE /api/scrapyd/versions
│ │ ├── spiders/ # GET /api/scrapyd/spiders
│ │ └── jobs/ # GET/POST/DELETE /api/scrapyd/jobs
│ └── layout.tsx # Root layout with theme provider
├── components/ # React components
│ ├── ui/ # shadcn/ui components
│ ├── sidebar.tsx # Navigation sidebar
│ ├── header.tsx # Page header
│ └── theme-toggle.tsx # Dark/light mode toggle
├── lib/ # Utilities & API client
│ ├── scrapyd-client.ts # Scrapyd API wrapper
│ ├── types.ts # TypeScript types & Zod schemas
│ └── utils.ts # Helper functions
├── Dockerfile # Production build
├── Dockerfile.dev # Development build
├── docker-compose.yml # Production deployment
└── docker-compose.dev.yml # Development deployment
```
## API Endpoints
All Scrapyd endpoints are proxied through Next.js API routes with server-side authentication:
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/scrapyd/daemon` | GET | Daemon status |
| `/api/scrapyd/projects` | GET | List projects |
| `/api/scrapyd/projects` | DELETE | Delete project |
| `/api/scrapyd/versions` | GET | List versions |
| `/api/scrapyd/versions` | POST | Upload version |
| `/api/scrapyd/versions` | DELETE | Delete version |
| `/api/scrapyd/spiders` | GET | List spiders |
| `/api/scrapyd/jobs` | GET | List jobs |
| `/api/scrapyd/jobs` | POST | Schedule job |
| `/api/scrapyd/jobs` | DELETE | Cancel job |
## Deployment
### Docker
#### Build production image:
```bash
docker build -t scrapy-ui:latest .
```
#### Run container:
```bash
docker run -d \
-p 3000:3000 \
-e SCRAPYD_URL=https://scrapy.pivoine.art \
-e SCRAPYD_USERNAME=your_username \
-e SCRAPYD_PASSWORD=your_password \
--name scrapy-ui \
scrapy-ui:latest
```
### GitHub Actions
The project includes a GitHub Actions workflow (`.github/workflows/docker-build.yml`) that automatically builds and pushes Docker images to GitHub Container Registry on push to `main` or on tagged releases.
To use it:
1. Ensure GitHub Actions has write permissions to packages
2. Push code to trigger the workflow
3. Images will be available at `ghcr.io/<username>/scrapy-ui`
### Reverse Proxy (Nginx)
To deploy under `/ui` path with Nginx:
```nginx
location /ui/ {
proxy_pass http://localhost:3000/ui/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
```
## Configuration
### Environment Variables
| Variable | Description | Required | Default |
|----------|-------------|----------|---------|
| `SCRAPYD_URL` | Scrapyd base URL | Yes | `https://scrapy.pivoine.art` |
| `SCRAPYD_USERNAME` | Basic auth username | Yes | - |
| `SCRAPYD_PASSWORD` | Basic auth password | Yes | - |
| `NODE_ENV` | Environment mode | No | `production` |
### Next.js Configuration
The `next.config.ts` includes:
- `basePath: "/ui"` - Serves app under `/ui` path
- `output: "standalone"` - Optimized Docker builds
- Optimized imports for `lucide-react`
## Development
### Install dependencies:
```bash
pnpm install
```
### Run development server:
```bash
pnpm dev
```
### Build for production:
```bash
pnpm build
pnpm start
```
### Lint code:
```bash
pnpm lint
```
### Add shadcn/ui components:
```bash
pnpm dlx shadcn@latest add <component-name>
```
## Features in Detail
### Dashboard
- Real-time job statistics (running, pending, finished)
- System health indicators
- Quick project overview
- Auto-refresh every 30 seconds
### Projects
- List all Scrapy projects
- Upload new versions (.egg files)
- View version history
- Delete projects/versions
- Drag & drop file upload
### Spiders
- Browse spiders by project
- Schedule jobs with custom arguments
- JSON argument validation
- Quick schedule dialog
### Jobs
- Filter by status (pending/running/finished)
- Real-time status updates (5-second refresh)
- Cancel running/pending jobs
- View job logs and items
- Detailed job information
### System
- Daemon status monitoring
- Job queue statistics
- Environment information
- Auto-refresh every 10 seconds
## Security
- **Server-side authentication**: Credentials are stored in environment variables and never exposed to the client
- **API proxy**: All Scrapyd requests go through Next.js API routes
- **Basic auth**: Automatic injection of credentials in request headers
- **No client-side secrets**: Zero credential exposure in browser
## Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
## License
MIT License - feel free to use this project for personal or commercial purposes.
## Acknowledgments
- Built with [Next.js](https://nextjs.org/)
- UI components from [shadcn/ui](https://ui.shadcn.com/)
- Icons from [Lucide](https://lucide.dev/)
- Designed for [Scrapyd](https://scrapyd.readthedocs.io/)

View File

@@ -0,0 +1,383 @@
"use client";
import { useState } from "react";
import { useQuery, useMutation, useQueryClient } from "@tanstack/react-query";
import { Header } from "@/components/header";
import { Button } from "@/components/ui/button";
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
import { Badge } from "@/components/ui/badge";
import { Skeleton } from "@/components/ui/skeleton";
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from "@/components/ui/select";
import {
Table,
TableBody,
TableCell,
TableHead,
TableHeader,
TableRow,
} from "@/components/ui/table";
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from "@/components/ui/dialog";
import {
BriefcaseBusiness,
Clock,
PlayCircle,
CheckCircle2,
XCircle,
ExternalLink,
AlertCircle,
} from "lucide-react";
import { ListProjects, ListJobs, Job } from "@/lib/types";
import { format } from "date-fns";
export default function JobsPage() {
const queryClient = useQueryClient();
const [selectedProject, setSelectedProject] = useState<string>("");
const [statusFilter, setStatusFilter] = useState<string>("all");
const [cancelDialogOpen, setCancelDialogOpen] = useState(false);
const [selectedJob, setSelectedJob] = useState<Job | null>(null);
// Fetch projects
const { data: projects, isLoading: isProjectsLoading } = useQuery({
queryKey: ["projects"],
queryFn: async (): Promise<ListProjects> => {
const res = await fetch("/ui/api/scrapyd/projects");
if (!res.ok) throw new Error("Failed to fetch projects");
return res.json();
},
});
// Fetch jobs for selected project
const { data: jobs, isLoading: isJobsLoading } = useQuery({
queryKey: ["jobs", selectedProject],
queryFn: async (): Promise<ListJobs> => {
const res = await fetch(`/ui/api/scrapyd/jobs?project=${selectedProject}`);
if (!res.ok) throw new Error("Failed to fetch jobs");
return res.json();
},
enabled: !!selectedProject,
refetchInterval: 5000, // Refresh every 5 seconds for real-time updates
});
// Cancel job mutation
const cancelJobMutation = useMutation({
mutationFn: async (data: { project: string; job: string }) => {
const res = await fetch("/ui/api/scrapyd/jobs", {
method: "DELETE",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(data),
});
if (!res.ok) throw new Error("Failed to cancel job");
return res.json();
},
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: ["jobs"] });
queryClient.invalidateQueries({ queryKey: ["daemon-status"] });
setCancelDialogOpen(false);
setSelectedJob(null);
},
});
const handleCancelJob = () => {
if (selectedJob) {
cancelJobMutation.mutate({
project: selectedProject,
job: selectedJob.id,
});
}
};
// Combine and filter jobs
const allJobs: Array<Job & { status: string }> = [];
if (jobs) {
allJobs.push(...jobs.pending.map((j) => ({ ...j, status: "pending" })));
allJobs.push(...jobs.running.map((j) => ({ ...j, status: "running" })));
allJobs.push(...jobs.finished.map((j) => ({ ...j, status: "finished" })));
}
const filteredJobs =
statusFilter === "all"
? allJobs
: allJobs.filter((j) => j.status === statusFilter);
const getStatusBadge = (status: string) => {
switch (status) {
case "pending":
return (
<Badge variant="outline" className="gap-1">
<Clock className="h-3 w-3" />
Pending
</Badge>
);
case "running":
return (
<Badge className="gap-1 bg-green-500">
<PlayCircle className="h-3 w-3" />
Running
</Badge>
);
case "finished":
return (
<Badge variant="secondary" className="gap-1">
<CheckCircle2 className="h-3 w-3" />
Finished
</Badge>
);
default:
return <Badge>{status}</Badge>;
}
};
return (
<div className="space-y-6">
<Header
title="Jobs"
description="Monitor and manage scraping jobs"
/>
{/* Filters */}
<div className="grid gap-4 md:grid-cols-2">
<Card>
<CardHeader>
<CardTitle>Project</CardTitle>
</CardHeader>
<CardContent>
{isProjectsLoading ? (
<Skeleton className="h-10 w-full" />
) : (
<Select value={selectedProject} onValueChange={setSelectedProject}>
<SelectTrigger>
<SelectValue placeholder="Select a project" />
</SelectTrigger>
<SelectContent>
{projects?.projects.map((project) => (
<SelectItem key={project} value={project}>
{project}
</SelectItem>
))}
</SelectContent>
</Select>
)}
</CardContent>
</Card>
<Card>
<CardHeader>
<CardTitle>Status Filter</CardTitle>
</CardHeader>
<CardContent>
<Select value={statusFilter} onValueChange={setStatusFilter}>
<SelectTrigger>
<SelectValue />
</SelectTrigger>
<SelectContent>
<SelectItem value="all">All Jobs</SelectItem>
<SelectItem value="pending">Pending</SelectItem>
<SelectItem value="running">Running</SelectItem>
<SelectItem value="finished">Finished</SelectItem>
</SelectContent>
</Select>
</CardContent>
</Card>
</div>
{/* Jobs Statistics */}
{selectedProject && jobs && (
<div className="grid gap-4 md:grid-cols-3">
<Card>
<CardHeader className="flex flex-row items-center justify-between pb-2">
<CardTitle className="text-sm font-medium">Pending</CardTitle>
<Clock className="h-4 w-4 text-yellow-500" />
</CardHeader>
<CardContent>
<div className="text-2xl font-bold">{jobs.pending.length}</div>
</CardContent>
</Card>
<Card>
<CardHeader className="flex flex-row items-center justify-between pb-2">
<CardTitle className="text-sm font-medium">Running</CardTitle>
<PlayCircle className="h-4 w-4 text-green-500" />
</CardHeader>
<CardContent>
<div className="text-2xl font-bold">{jobs.running.length}</div>
</CardContent>
</Card>
<Card>
<CardHeader className="flex flex-row items-center justify-between pb-2">
<CardTitle className="text-sm font-medium">Finished</CardTitle>
<CheckCircle2 className="h-4 w-4 text-blue-500" />
</CardHeader>
<CardContent>
<div className="text-2xl font-bold">{jobs.finished.length}</div>
</CardContent>
</Card>
</div>
)}
{/* Jobs Table */}
{selectedProject && (
<Card>
<CardHeader>
<CardTitle>
Jobs for "{selectedProject}"
</CardTitle>
</CardHeader>
<CardContent>
{isJobsLoading ? (
<div className="space-y-2">
{[1, 2, 3].map((i) => (
<Skeleton key={i} className="h-16 w-full" />
))}
</div>
) : filteredJobs.length > 0 ? (
<Table>
<TableHeader>
<TableRow>
<TableHead>Job ID</TableHead>
<TableHead>Spider</TableHead>
<TableHead>Status</TableHead>
<TableHead>Start Time</TableHead>
<TableHead>PID</TableHead>
<TableHead className="text-right">Actions</TableHead>
</TableRow>
</TableHeader>
<TableBody>
{filteredJobs.map((job) => (
<TableRow key={job.id}>
<TableCell>
<code className="rounded bg-muted px-2 py-1 text-xs">
{job.id.substring(0, 8)}...
</code>
</TableCell>
<TableCell>
<div className="flex items-center gap-2">
<BriefcaseBusiness className="h-4 w-4 text-muted-foreground" />
<span className="font-medium">{job.spider}</span>
</div>
</TableCell>
<TableCell>{getStatusBadge(job.status)}</TableCell>
<TableCell>
<span className="text-sm text-muted-foreground">
{format(new Date(job.start_time), "PPp")}
</span>
</TableCell>
<TableCell>
{job.pid ? (
<code className="text-xs">{job.pid}</code>
) : (
<span className="text-xs text-muted-foreground">-</span>
)}
</TableCell>
<TableCell className="text-right">
<div className="flex items-center justify-end gap-2">
{job.log_url && (
<Button
variant="ghost"
size="sm"
asChild
>
<a
href={job.log_url}
target="_blank"
rel="noopener noreferrer"
>
<ExternalLink className="h-4 w-4" />
</a>
</Button>
)}
{(job.status === "pending" || job.status === "running") && (
<Button
variant="ghost"
size="sm"
onClick={() => {
setSelectedJob(job);
setCancelDialogOpen(true);
}}
>
<XCircle className="h-4 w-4 text-destructive" />
</Button>
)}
</div>
</TableCell>
</TableRow>
))}
</TableBody>
</Table>
) : (
<div className="flex flex-col items-center justify-center py-12 text-center">
<AlertCircle className="mb-4 h-12 w-12 text-muted-foreground" />
<h3 className="mb-2 text-lg font-semibold">No jobs found</h3>
<p className="text-sm text-muted-foreground">
{statusFilter === "all"
? "No jobs have been scheduled for this project"
: `No ${statusFilter} jobs found`}
</p>
</div>
)}
</CardContent>
</Card>
)}
{!selectedProject && !isProjectsLoading && (
<Card>
<CardContent className="flex flex-col items-center justify-center py-12">
<BriefcaseBusiness className="mb-4 h-12 w-12 text-muted-foreground" />
<h3 className="mb-2 text-lg font-semibold">Select a project</h3>
<p className="text-sm text-muted-foreground">
Choose a project to view its jobs
</p>
</CardContent>
</Card>
)}
{/* Cancel Job Dialog */}
<Dialog open={cancelDialogOpen} onOpenChange={setCancelDialogOpen}>
<DialogContent>
<DialogHeader>
<DialogTitle>Cancel Job</DialogTitle>
<DialogDescription>
Are you sure you want to cancel this job?
{selectedJob && (
<div className="mt-2 rounded-lg border p-3">
<p className="text-sm">
<strong>Spider:</strong> {selectedJob.spider}
</p>
<p className="text-sm">
<strong>Job ID:</strong> <code>{selectedJob.id}</code>
</p>
</div>
)}
</DialogDescription>
</DialogHeader>
<DialogFooter>
<Button
variant="outline"
onClick={() => setCancelDialogOpen(false)}
>
No, keep it
</Button>
<Button
variant="destructive"
onClick={handleCancelJob}
disabled={cancelJobMutation.isPending}
>
{cancelJobMutation.isPending ? "Canceling..." : "Yes, cancel job"}
</Button>
</DialogFooter>
</DialogContent>
</Dialog>
</div>
);
}

View File

@@ -0,0 +1,19 @@
import { Sidebar } from "@/components/sidebar";
import { Providers } from "@/components/providers";
export default function DashboardLayout({
children,
}: {
children: React.ReactNode;
}) {
return (
<Providers>
<div className="flex h-screen overflow-hidden">
<Sidebar />
<main className="flex-1 overflow-y-auto">
<div className="container p-6">{children}</div>
</main>
</div>
</Providers>
);
}

173
app/(dashboard)/page.tsx Normal file
View File

@@ -0,0 +1,173 @@
"use client";
import { useQuery } from "@tanstack/react-query";
import { Header } from "@/components/header";
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
import { Badge } from "@/components/ui/badge";
import { Skeleton } from "@/components/ui/skeleton";
import {
Activity,
FolderKanban,
PlayCircle,
Clock,
CheckCircle2,
} from "lucide-react";
import { DaemonStatus } from "@/lib/types";
export default function DashboardPage() {
const { data: daemonStatus, isLoading: isDaemonLoading } = useQuery({
queryKey: ["daemon-status"],
queryFn: async (): Promise<DaemonStatus> => {
const res = await fetch("/ui/api/scrapyd/daemon");
if (!res.ok) throw new Error("Failed to fetch daemon status");
return res.json();
},
});
const { data: projects, isLoading: isProjectsLoading } = useQuery({
queryKey: ["projects"],
queryFn: async () => {
const res = await fetch("/ui/api/scrapyd/projects");
if (!res.ok) throw new Error("Failed to fetch projects");
return res.json();
},
});
const stats = [
{
title: "Running Jobs",
value: daemonStatus?.running ?? 0,
icon: PlayCircle,
color: "text-green-500",
bgColor: "bg-green-500/10",
},
{
title: "Pending Jobs",
value: daemonStatus?.pending ?? 0,
icon: Clock,
color: "text-yellow-500",
bgColor: "bg-yellow-500/10",
},
{
title: "Finished Jobs",
value: daemonStatus?.finished ?? 0,
icon: CheckCircle2,
color: "text-blue-500",
bgColor: "bg-blue-500/10",
},
{
title: "Total Projects",
value: projects?.projects?.length ?? 0,
icon: FolderKanban,
color: "text-purple-500",
bgColor: "bg-purple-500/10",
},
];
return (
<div className="space-y-6">
<Header
title="Dashboard"
description="Monitor your Scrapyd instance and scraping jobs"
/>
{/* Stats Grid */}
<div className="grid gap-4 md:grid-cols-2 lg:grid-cols-4">
{stats.map((stat, index) => (
<Card key={index}>
<CardHeader className="flex flex-row items-center justify-between gap-0 pb-2">
<CardTitle className="text-sm font-medium">
{stat.title}
</CardTitle>
<div className={`rounded-full p-2 ${stat.bgColor}`}>
<stat.icon className={`h-4 w-4 ${stat.color}`} />
</div>
</CardHeader>
<CardContent>
{isDaemonLoading || isProjectsLoading ? (
<Skeleton className="h-8 w-16" />
) : (
<div className="text-2xl font-bold">{stat.value}</div>
)}
</CardContent>
</Card>
))}
</div>
{/* System Status Card */}
<Card>
<CardHeader>
<div className="flex items-center justify-between">
<CardTitle>System Status</CardTitle>
<Badge variant="outline" className="flex items-center gap-1">
<Activity className="h-3 w-3" />
{isDaemonLoading ? (
<Skeleton className="h-4 w-12" />
) : (
daemonStatus?.status
)}
</Badge>
</div>
</CardHeader>
<CardContent>
{isDaemonLoading ? (
<div className="space-y-2">
<Skeleton className="h-4 w-full" />
<Skeleton className="h-4 w-3/4" />
</div>
) : (
<div className="space-y-2">
<div className="flex justify-between text-sm">
<span className="text-muted-foreground">Node Name:</span>
<span className="font-mono">{daemonStatus?.node_name}</span>
</div>
<div className="flex justify-between text-sm">
<span className="text-muted-foreground">Total Jobs:</span>
<span>
{(daemonStatus?.running ?? 0) +
(daemonStatus?.pending ?? 0) +
(daemonStatus?.finished ?? 0)}
</span>
</div>
</div>
)}
</CardContent>
</Card>
{/* Projects Overview */}
<Card>
<CardHeader>
<CardTitle>Projects Overview</CardTitle>
</CardHeader>
<CardContent>
{isProjectsLoading ? (
<div className="space-y-2">
{[1, 2, 3].map((i) => (
<Skeleton key={i} className="h-8 w-full" />
))}
</div>
) : projects?.projects?.length > 0 ? (
<div className="space-y-2">
{projects.projects.map((project: string) => (
<div
key={project}
className="flex items-center justify-between rounded-lg border p-3"
>
<div className="flex items-center gap-2">
<FolderKanban className="h-4 w-4 text-muted-foreground" />
<span className="font-medium">{project}</span>
</div>
<Badge variant="secondary">Active</Badge>
</div>
))}
</div>
) : (
<p className="text-center text-muted-foreground">
No projects found. Upload a project to get started.
</p>
)}
</CardContent>
</Card>
</div>
);
}

View File

@@ -0,0 +1,326 @@
"use client";
import { useState } from "react";
import { useQuery, useMutation, useQueryClient } from "@tanstack/react-query";
import { Header } from "@/components/header";
import { Button } from "@/components/ui/button";
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
import { Badge } from "@/components/ui/badge";
import { Skeleton } from "@/components/ui/skeleton";
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
DialogTrigger,
} from "@/components/ui/dialog";
import { Input } from "@/components/ui/input";
import { Label } from "@/components/ui/label";
import {
Table,
TableBody,
TableCell,
TableHead,
TableHeader,
TableRow,
} from "@/components/ui/table";
import {
FolderKanban,
Upload,
Trash2,
Package,
AlertCircle,
} from "lucide-react";
import { ListProjects, ListVersions } from "@/lib/types";
export default function ProjectsPage() {
const queryClient = useQueryClient();
const [selectedProject, setSelectedProject] = useState<string | null>(null);
const [uploadDialogOpen, setUploadDialogOpen] = useState(false);
const [deleteDialogOpen, setDeleteDialogOpen] = useState(false);
// Fetch projects
const { data: projects, isLoading: isProjectsLoading } = useQuery({
queryKey: ["projects"],
queryFn: async (): Promise<ListProjects> => {
const res = await fetch("/ui/api/scrapyd/projects");
if (!res.ok) throw new Error("Failed to fetch projects");
return res.json();
},
});
// Fetch versions for selected project
const { data: versions } = useQuery({
queryKey: ["versions", selectedProject],
queryFn: async (): Promise<ListVersions> => {
const res = await fetch(
`/ui/api/scrapyd/versions?project=${selectedProject}`
);
if (!res.ok) throw new Error("Failed to fetch versions");
return res.json();
},
enabled: !!selectedProject,
});
// Delete project mutation
const deleteProjectMutation = useMutation({
mutationFn: async (project: string) => {
const res = await fetch("/ui/api/scrapyd/projects", {
method: "DELETE",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ project }),
});
if (!res.ok) throw new Error("Failed to delete project");
return res.json();
},
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: ["projects"] });
setDeleteDialogOpen(false);
setSelectedProject(null);
},
});
// Upload version mutation
const uploadVersionMutation = useMutation({
mutationFn: async (formData: FormData) => {
const res = await fetch("/ui/api/scrapyd/versions", {
method: "POST",
body: formData,
});
if (!res.ok) throw new Error("Failed to upload version");
return res.json();
},
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: ["projects"] });
queryClient.invalidateQueries({ queryKey: ["versions"] });
setUploadDialogOpen(false);
},
});
const handleUpload = (e: React.FormEvent<HTMLFormElement>) => {
e.preventDefault();
const formData = new FormData(e.currentTarget);
uploadVersionMutation.mutate(formData);
};
return (
<div className="space-y-6">
<Header
title="Projects"
description="Manage your Scrapyd projects and versions"
action={
<Dialog open={uploadDialogOpen} onOpenChange={setUploadDialogOpen}>
<DialogTrigger asChild>
<Button>
<Upload className="mr-2 h-4 w-4" />
Upload Project
</Button>
</DialogTrigger>
<DialogContent>
<form onSubmit={handleUpload}>
<DialogHeader>
<DialogTitle>Upload Project Version</DialogTitle>
<DialogDescription>
Upload a Python egg file for your Scrapy project
</DialogDescription>
</DialogHeader>
<div className="gap-4 py-4">
<div className="space-y-2">
<Label htmlFor="project">Project Name</Label>
<Input
id="project"
name="project"
placeholder="myproject"
required
/>
</div>
<div className="space-y-2">
<Label htmlFor="version">Version</Label>
<Input
id="version"
name="version"
placeholder="1.0.0"
required
/>
</div>
<div className="space-y-2">
<Label htmlFor="egg">Egg File</Label>
<Input
id="egg"
name="egg"
type="file"
accept=".egg"
required
/>
</div>
</div>
<DialogFooter>
<Button
type="submit"
disabled={uploadVersionMutation.isPending}
>
{uploadVersionMutation.isPending
? "Uploading..."
: "Upload"}
</Button>
</DialogFooter>
</form>
</DialogContent>
</Dialog>
}
/>
{/* Projects List */}
<Card>
<CardHeader>
<CardTitle>All Projects</CardTitle>
</CardHeader>
<CardContent>
{isProjectsLoading ? (
<div className="space-y-2">
{[1, 2, 3].map((i) => (
<Skeleton key={i} className="h-16 w-full" />
))}
</div>
) : projects?.projects && projects.projects.length > 0 ? (
<Table>
<TableHeader>
<TableRow>
<TableHead>Project Name</TableHead>
<TableHead>Versions</TableHead>
<TableHead className="text-right">Actions</TableHead>
</TableRow>
</TableHeader>
<TableBody>
{projects.projects.map((project) => (
<TableRow
key={project}
onClick={() => setSelectedProject(project)}
className="cursor-pointer"
>
<TableCell>
<div className="flex items-center gap-2">
<FolderKanban className="h-4 w-4 text-muted-foreground" />
<span className="font-medium">{project}</span>
</div>
</TableCell>
<TableCell>
{selectedProject === project && versions ? (
<Badge variant="secondary">
{versions.versions.length} version(s)
</Badge>
) : (
<Badge variant="outline">Click to load</Badge>
)}
</TableCell>
<TableCell className="text-right">
<Dialog
open={deleteDialogOpen && selectedProject === project}
onOpenChange={(open) => {
setDeleteDialogOpen(open);
if (open) setSelectedProject(project);
}}
>
<DialogTrigger asChild>
<Button
variant="ghost"
size="icon"
onClick={(e) => {
e.stopPropagation();
setSelectedProject(project);
}}
>
<Trash2 className="h-4 w-4 text-destructive" />
</Button>
</DialogTrigger>
<DialogContent>
<DialogHeader>
<DialogTitle>Delete Project</DialogTitle>
<DialogDescription>
Are you sure you want to delete "{project}"? This
action cannot be undone.
</DialogDescription>
</DialogHeader>
<DialogFooter>
<Button
variant="outline"
onClick={() => setDeleteDialogOpen(false)}
>
Cancel
</Button>
<Button
variant="destructive"
onClick={() =>
deleteProjectMutation.mutate(project)
}
disabled={deleteProjectMutation.isPending}
>
{deleteProjectMutation.isPending
? "Deleting..."
: "Delete"}
</Button>
</DialogFooter>
</DialogContent>
</Dialog>
</TableCell>
</TableRow>
))}
</TableBody>
</Table>
) : (
<div className="flex flex-col items-center justify-center py-12 text-center">
<AlertCircle className="mb-4 h-12 w-12 text-muted-foreground" />
<h3 className="mb-2 text-lg font-semibold">No projects found</h3>
<p className="mb-4 text-sm text-muted-foreground">
Upload your first project to get started
</p>
<Button onClick={() => setUploadDialogOpen(true)}>
<Upload className="mr-2 h-4 w-4" />
Upload Project
</Button>
</div>
)}
</CardContent>
</Card>
{/* Project Versions */}
{selectedProject && versions && (
<Card>
<CardHeader>
<CardTitle>
Versions for "{selectedProject}"
</CardTitle>
</CardHeader>
<CardContent>
{versions.versions.length > 0 ? (
<div className="space-y-2">
{versions.versions.map((version) => (
<div
key={version}
className="flex items-center justify-between rounded-lg border p-3"
>
<div className="flex items-center gap-2">
<Package className="h-4 w-4 text-muted-foreground" />
<span className="font-mono text-sm">{version}</span>
</div>
<Badge variant="secondary">
{version === versions.versions[versions.versions.length - 1]
? "Latest"
: ""}
</Badge>
</div>
))}
</div>
) : (
<p className="text-center text-sm text-muted-foreground">
No versions found for this project
</p>
)}
</CardContent>
</Card>
)}
</div>
);
}

View File

@@ -0,0 +1,282 @@
"use client";
import { useState } from "react";
import { useQuery, useMutation, useQueryClient } from "@tanstack/react-query";
import { Header } from "@/components/header";
import { Button } from "@/components/ui/button";
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
import { Badge } from "@/components/ui/badge";
import { Skeleton } from "@/components/ui/skeleton";
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
DialogTrigger,
} from "@/components/ui/dialog";
import { Input } from "@/components/ui/input";
import { Label } from "@/components/ui/label";
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from "@/components/ui/select";
import {
Table,
TableBody,
TableCell,
TableHead,
TableHeader,
TableRow,
} from "@/components/ui/table";
import { Bug, PlayCircle, AlertCircle } from "lucide-react";
import { ListProjects, ListSpiders, ScheduleJob } from "@/lib/types";
import { Textarea } from "@/components/ui/textarea";
export default function SpidersPage() {
const queryClient = useQueryClient();
const [selectedProject, setSelectedProject] = useState<string>("");
const [scheduleDialogOpen, setScheduleDialogOpen] = useState(false);
const [selectedSpider, setSelectedSpider] = useState<string>("");
// Fetch projects
const { data: projects, isLoading: isProjectsLoading } = useQuery({
queryKey: ["projects"],
queryFn: async (): Promise<ListProjects> => {
const res = await fetch("/ui/api/scrapyd/projects");
if (!res.ok) throw new Error("Failed to fetch projects");
return res.json();
},
});
// Fetch spiders for selected project
const { data: spiders, isLoading: isSpidersLoading } = useQuery({
queryKey: ["spiders", selectedProject],
queryFn: async (): Promise<ListSpiders> => {
const res = await fetch(
`/ui/api/scrapyd/spiders?project=${selectedProject}`
);
if (!res.ok) throw new Error("Failed to fetch spiders");
return res.json();
},
enabled: !!selectedProject,
});
// Schedule job mutation
const scheduleJobMutation = useMutation({
mutationFn: async (data: {
project: string;
spider: string;
args?: Record<string, string>;
}) => {
const res = await fetch("/ui/api/scrapyd/jobs", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(data),
});
if (!res.ok) throw new Error("Failed to schedule job");
return res.json() as Promise<ScheduleJob>;
},
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: ["jobs"] });
setScheduleDialogOpen(false);
setSelectedSpider("");
},
});
const handleSchedule = (e: React.FormEvent<HTMLFormElement>) => {
e.preventDefault();
const formData = new FormData(e.currentTarget);
const argsStr = formData.get("args") as string;
let args: Record<string, string> | undefined;
if (argsStr.trim()) {
try {
args = JSON.parse(argsStr);
} catch (error) {
alert("Invalid JSON format for arguments");
return;
}
}
scheduleJobMutation.mutate({
project: selectedProject,
spider: selectedSpider,
args,
});
};
return (
<div className="space-y-6">
<Header
title="Spiders"
description="Browse and schedule spider jobs"
/>
{/* Project Selector */}
<Card>
<CardHeader>
<CardTitle>Select Project</CardTitle>
</CardHeader>
<CardContent>
{isProjectsLoading ? (
<Skeleton className="h-10 w-full" />
) : (
<Select value={selectedProject} onValueChange={setSelectedProject}>
<SelectTrigger>
<SelectValue placeholder="Select a project" />
</SelectTrigger>
<SelectContent>
{projects?.projects.map((project) => (
<SelectItem key={project} value={project}>
{project}
</SelectItem>
))}
</SelectContent>
</Select>
)}
</CardContent>
</Card>
{/* Spiders List */}
{selectedProject && (
<Card>
<CardHeader>
<CardTitle>
Spiders in "{selectedProject}"
</CardTitle>
</CardHeader>
<CardContent>
{isSpidersLoading ? (
<div className="space-y-2">
{[1, 2, 3].map((i) => (
<Skeleton key={i} className="h-16 w-full" />
))}
</div>
) : spiders?.spiders && spiders.spiders.length > 0 ? (
<Table>
<TableHeader>
<TableRow>
<TableHead>Spider Name</TableHead>
<TableHead>Status</TableHead>
<TableHead className="text-right">Actions</TableHead>
</TableRow>
</TableHeader>
<TableBody>
{spiders.spiders.map((spider) => (
<TableRow key={spider}>
<TableCell>
<div className="flex items-center gap-2">
<Bug className="h-4 w-4 text-muted-foreground" />
<span className="font-medium">{spider}</span>
</div>
</TableCell>
<TableCell>
<Badge variant="secondary">Available</Badge>
</TableCell>
<TableCell className="text-right">
<Dialog
open={scheduleDialogOpen && selectedSpider === spider}
onOpenChange={(open) => {
setScheduleDialogOpen(open);
if (open) setSelectedSpider(spider);
}}
>
<DialogTrigger asChild>
<Button
size="sm"
onClick={() => setSelectedSpider(spider)}
>
<PlayCircle className="mr-2 h-4 w-4" />
Schedule
</Button>
</DialogTrigger>
<DialogContent>
<form onSubmit={handleSchedule}>
<DialogHeader>
<DialogTitle>Schedule Spider Job</DialogTitle>
<DialogDescription>
Schedule "{spider}" to run on "{selectedProject}"
</DialogDescription>
</DialogHeader>
<div className="gap-4 py-4">
<div className="space-y-2">
<Label htmlFor="project-name">Project</Label>
<Input
id="project-name"
value={selectedProject}
disabled
/>
</div>
<div className="space-y-2">
<Label htmlFor="spider-name">Spider</Label>
<Input
id="spider-name"
value={spider}
disabled
/>
</div>
<div className="space-y-2">
<Label htmlFor="args">
Arguments (JSON)
</Label>
<Textarea
id="args"
name="args"
placeholder='{"url": "https://example.com", "pages": 10}'
className="font-mono text-sm"
/>
<p className="text-xs text-muted-foreground">
Optional: Provide spider arguments in JSON format
</p>
</div>
</div>
<DialogFooter>
<Button
type="submit"
disabled={scheduleJobMutation.isPending}
>
{scheduleJobMutation.isPending
? "Scheduling..."
: "Schedule Job"}
</Button>
</DialogFooter>
</form>
</DialogContent>
</Dialog>
</TableCell>
</TableRow>
))}
</TableBody>
</Table>
) : (
<div className="flex flex-col items-center justify-center py-12 text-center">
<AlertCircle className="mb-4 h-12 w-12 text-muted-foreground" />
<h3 className="mb-2 text-lg font-semibold">No spiders found</h3>
<p className="text-sm text-muted-foreground">
This project doesn't have any spiders yet
</p>
</div>
)}
</CardContent>
</Card>
)}
{!selectedProject && !isProjectsLoading && (
<Card>
<CardContent className="flex flex-col items-center justify-center py-12">
<Bug className="mb-4 h-12 w-12 text-muted-foreground" />
<h3 className="mb-2 text-lg font-semibold">Select a project</h3>
<p className="text-sm text-muted-foreground">
Choose a project to view its spiders
</p>
</CardContent>
</Card>
)}
</div>
);
}

View File

@@ -0,0 +1,267 @@
"use client";
import { useQuery } from "@tanstack/react-query";
import { Header } from "@/components/header";
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
import { Badge } from "@/components/ui/badge";
import { Skeleton } from "@/components/ui/skeleton";
import {
Activity,
Server,
Cpu,
HardDrive,
CheckCircle,
Clock,
PlayCircle,
CheckCircle2,
} from "lucide-react";
import { DaemonStatus, ListProjects } from "@/lib/types";
export default function SystemPage() {
const { data: daemonStatus, isLoading: isDaemonLoading } = useQuery({
queryKey: ["daemon-status"],
queryFn: async (): Promise<DaemonStatus> => {
const res = await fetch("/ui/api/scrapyd/daemon");
if (!res.ok) throw new Error("Failed to fetch daemon status");
return res.json();
},
refetchInterval: 10000, // Refresh every 10 seconds
});
const { data: projects, isLoading: isProjectsLoading } = useQuery({
queryKey: ["projects"],
queryFn: async (): Promise<ListProjects> => {
const res = await fetch("/ui/api/scrapyd/projects");
if (!res.ok) throw new Error("Failed to fetch projects");
return res.json();
},
});
const systemMetrics = [
{
label: "Daemon Status",
value: daemonStatus?.status || "unknown",
icon: Activity,
color: "text-green-500",
bgColor: "bg-green-500/10",
},
{
label: "Running Jobs",
value: daemonStatus?.running ?? 0,
icon: PlayCircle,
color: "text-blue-500",
bgColor: "bg-blue-500/10",
},
{
label: "Pending Jobs",
value: daemonStatus?.pending ?? 0,
icon: Clock,
color: "text-yellow-500",
bgColor: "bg-yellow-500/10",
},
{
label: "Finished Jobs",
value: daemonStatus?.finished ?? 0,
icon: CheckCircle2,
color: "text-purple-500",
bgColor: "bg-purple-500/10",
},
];
return (
<div className="space-y-6">
<Header
title="System Status"
description="Monitor Scrapyd daemon health and metrics"
/>
{/* Status Overview */}
<Card>
<CardHeader>
<div className="flex items-center justify-between">
<CardTitle>Daemon Information</CardTitle>
{isDaemonLoading ? (
<Skeleton className="h-6 w-16" />
) : (
<Badge
variant={daemonStatus?.status === "ok" ? "default" : "destructive"}
className="gap-1"
>
{daemonStatus?.status === "ok" ? (
<CheckCircle className="h-3 w-3" />
) : (
<Activity className="h-3 w-3" />
)}
{daemonStatus?.status?.toUpperCase()}
</Badge>
)}
</div>
</CardHeader>
<CardContent>
{isDaemonLoading ? (
<div className="space-y-3">
{[1, 2, 3].map((i) => (
<Skeleton key={i} className="h-6 w-full" />
))}
</div>
) : (
<div className="space-y-4">
<div className="flex items-center justify-between border-b pb-3">
<div className="flex items-center gap-2">
<Server className="h-4 w-4 text-muted-foreground" />
<span className="text-sm font-medium">Node Name</span>
</div>
<code className="rounded bg-muted px-2 py-1 text-sm">
{daemonStatus?.node_name}
</code>
</div>
<div className="flex items-center justify-between border-b pb-3">
<div className="flex items-center gap-2">
<Cpu className="h-4 w-4 text-muted-foreground" />
<span className="text-sm font-medium">Total Projects</span>
</div>
<span className="text-sm font-bold">
{isProjectsLoading ? (
<Skeleton className="h-5 w-8" />
) : (
projects?.projects?.length ?? 0
)}
</span>
</div>
<div className="flex items-center justify-between">
<div className="flex items-center gap-2">
<HardDrive className="h-4 w-4 text-muted-foreground" />
<span className="text-sm font-medium">Total Jobs (All Time)</span>
</div>
<span className="text-sm font-bold">
{(daemonStatus?.running ?? 0) +
(daemonStatus?.pending ?? 0) +
(daemonStatus?.finished ?? 0)}
</span>
</div>
</div>
)}
</CardContent>
</Card>
{/* System Metrics Grid */}
<div className="grid gap-4 md:grid-cols-2 lg:grid-cols-4">
{systemMetrics.map((metric, index) => (
<Card key={index}>
<CardHeader className="flex flex-row items-center justify-between gap-0 pb-2">
<CardTitle className="text-sm font-medium">
{metric.label}
</CardTitle>
<div className={`rounded-full p-2 ${metric.bgColor}`}>
<metric.icon className={`h-4 w-4 ${metric.color}`} />
</div>
</CardHeader>
<CardContent>
{isDaemonLoading ? (
<Skeleton className="h-8 w-20" />
) : (
<div className="text-2xl font-bold">{metric.value}</div>
)}
</CardContent>
</Card>
))}
</div>
{/* Job Queue Status */}
<Card>
<CardHeader>
<CardTitle>Job Queue Status</CardTitle>
</CardHeader>
<CardContent>
{isDaemonLoading ? (
<div className="space-y-3">
{[1, 2, 3].map((i) => (
<Skeleton key={i} className="h-12 w-full" />
))}
</div>
) : (
<div className="space-y-3">
<div className="flex items-center justify-between rounded-lg border p-4">
<div className="flex items-center gap-3">
<div className="rounded-full bg-blue-500/10 p-2">
<PlayCircle className="h-5 w-5 text-blue-500" />
</div>
<div>
<p className="text-sm font-medium">Running Jobs</p>
<p className="text-xs text-muted-foreground">
Currently executing
</p>
</div>
</div>
<span className="text-2xl font-bold">
{daemonStatus?.running}
</span>
</div>
<div className="flex items-center justify-between rounded-lg border p-4">
<div className="flex items-center gap-3">
<div className="rounded-full bg-yellow-500/10 p-2">
<Clock className="h-5 w-5 text-yellow-500" />
</div>
<div>
<p className="text-sm font-medium">Pending Jobs</p>
<p className="text-xs text-muted-foreground">
Waiting in queue
</p>
</div>
</div>
<span className="text-2xl font-bold">
{daemonStatus?.pending}
</span>
</div>
<div className="flex items-center justify-between rounded-lg border p-4">
<div className="flex items-center gap-3">
<div className="rounded-full bg-purple-500/10 p-2">
<CheckCircle2 className="h-5 w-5 text-purple-500" />
</div>
<div>
<p className="text-sm font-medium">Finished Jobs</p>
<p className="text-xs text-muted-foreground">
Completed successfully
</p>
</div>
</div>
<span className="text-2xl font-bold">
{daemonStatus?.finished}
</span>
</div>
</div>
)}
</CardContent>
</Card>
{/* Environment Info */}
<Card>
<CardHeader>
<CardTitle>Environment Information</CardTitle>
</CardHeader>
<CardContent>
<div className="gap-2 text-sm">
<div className="flex justify-between">
<span className="text-muted-foreground">Scrapyd URL:</span>
<code className="rounded bg-muted px-2 py-1 text-xs">
{process.env.NEXT_PUBLIC_SCRAPYD_URL || "https://scrapy.pivoine.art"}
</code>
</div>
<div className="flex justify-between">
<span className="text-muted-foreground">UI Version:</span>
<Badge variant="outline">1.0.0</Badge>
</div>
<div className="flex justify-between">
<span className="text-muted-foreground">Refresh Interval:</span>
<span>10 seconds</span>
</div>
</div>
</CardContent>
</Card>
</div>
);
}

View File

@@ -0,0 +1,15 @@
import { NextResponse } from "next/server";
import { ScrapydClient } from "@/lib/scrapyd-client";
export async function GET() {
try {
const data = await ScrapydClient.getDaemonStatus();
return NextResponse.json(data);
} catch (error) {
console.error("Failed to fetch daemon status:", error);
return NextResponse.json(
{ error: "Failed to fetch daemon status" },
{ status: 500 }
);
}
}

View File

@@ -0,0 +1,77 @@
import { NextRequest, NextResponse } from "next/server";
import { ScrapydClient } from "@/lib/scrapyd-client";
export async function GET(request: NextRequest) {
try {
const searchParams = request.nextUrl.searchParams;
const project = searchParams.get("project");
if (!project) {
return NextResponse.json(
{ error: "Project name is required" },
{ status: 400 }
);
}
const data = await ScrapydClient.listJobs({ project });
return NextResponse.json(data);
} catch (error) {
console.error("Failed to fetch jobs:", error);
return NextResponse.json(
{ error: "Failed to fetch jobs" },
{ status: 500 }
);
}
}
export async function POST(request: NextRequest) {
try {
const body = await request.json();
const { project, spider, jobid, settings, args } = body;
if (!project || !spider) {
return NextResponse.json(
{ error: "Project and spider are required" },
{ status: 400 }
);
}
const data = await ScrapydClient.scheduleJob({
project,
spider,
jobid,
settings,
args,
});
return NextResponse.json(data);
} catch (error) {
console.error("Failed to schedule job:", error);
return NextResponse.json(
{ error: "Failed to schedule job" },
{ status: 500 }
);
}
}
export async function DELETE(request: NextRequest) {
try {
const body = await request.json();
const { project, job } = body;
if (!project || !job) {
return NextResponse.json(
{ error: "Project and job ID are required" },
{ status: 400 }
);
}
const data = await ScrapydClient.cancelJob({ project, job });
return NextResponse.json(data);
} catch (error) {
console.error("Failed to cancel job:", error);
return NextResponse.json(
{ error: "Failed to cancel job" },
{ status: 500 }
);
}
}

View File

@@ -0,0 +1,38 @@
import { NextRequest, NextResponse } from "next/server";
import { ScrapydClient } from "@/lib/scrapyd-client";
export async function GET() {
try {
const data = await ScrapydClient.listProjects();
return NextResponse.json(data);
} catch (error) {
console.error("Failed to fetch projects:", error);
return NextResponse.json(
{ error: "Failed to fetch projects" },
{ status: 500 }
);
}
}
export async function DELETE(request: NextRequest) {
try {
const body = await request.json();
const { project } = body;
if (!project) {
return NextResponse.json(
{ error: "Project name is required" },
{ status: 400 }
);
}
const data = await ScrapydClient.deleteProject({ project });
return NextResponse.json(data);
} catch (error) {
console.error("Failed to delete project:", error);
return NextResponse.json(
{ error: "Failed to delete project" },
{ status: 500 }
);
}
}

View File

@@ -0,0 +1,26 @@
import { NextRequest, NextResponse } from "next/server";
import { ScrapydClient } from "@/lib/scrapyd-client";
export async function GET(request: NextRequest) {
try {
const searchParams = request.nextUrl.searchParams;
const project = searchParams.get("project");
const version = searchParams.get("version") || undefined;
if (!project) {
return NextResponse.json(
{ error: "Project name is required" },
{ status: 400 }
);
}
const data = await ScrapydClient.listSpiders({ project, version });
return NextResponse.json(data);
} catch (error) {
console.error("Failed to fetch spiders:", error);
return NextResponse.json(
{ error: "Failed to fetch spiders" },
{ status: 500 }
);
}
}

View File

@@ -0,0 +1,74 @@
import { NextRequest, NextResponse } from "next/server";
import { ScrapydClient } from "@/lib/scrapyd-client";
export async function GET(request: NextRequest) {
try {
const searchParams = request.nextUrl.searchParams;
const project = searchParams.get("project");
if (!project) {
return NextResponse.json(
{ error: "Project name is required" },
{ status: 400 }
);
}
const data = await ScrapydClient.listVersions({ project });
return NextResponse.json(data);
} catch (error) {
console.error("Failed to fetch versions:", error);
return NextResponse.json(
{ error: "Failed to fetch versions" },
{ status: 500 }
);
}
}
export async function DELETE(request: NextRequest) {
try {
const body = await request.json();
const { project, version } = body;
if (!project || !version) {
return NextResponse.json(
{ error: "Project and version are required" },
{ status: 400 }
);
}
const data = await ScrapydClient.deleteVersion({ project, version });
return NextResponse.json(data);
} catch (error) {
console.error("Failed to delete version:", error);
return NextResponse.json(
{ error: "Failed to delete version" },
{ status: 500 }
);
}
}
export async function POST(request: NextRequest) {
try {
const formData = await request.formData();
const project = formData.get("project") as string;
const version = formData.get("version") as string;
const eggFile = formData.get("egg") as File;
if (!project || !version || !eggFile) {
return NextResponse.json(
{ error: "Project, version, and egg file are required" },
{ status: 400 }
);
}
const buffer = Buffer.from(await eggFile.arrayBuffer());
const data = await ScrapydClient.addVersion(project, version, buffer);
return NextResponse.json(data);
} catch (error) {
console.error("Failed to add version:", error);
return NextResponse.json(
{ error: "Failed to add version" },
{ status: 500 }
);
}
}

BIN
app/favicon.ico Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 25 KiB

110
app/globals.css Normal file
View File

@@ -0,0 +1,110 @@
@import "tailwindcss";
@source "../components/*.{js,ts,jsx,tsx}";
@source "../components/ui/*.{js,ts,jsx,tsx}";
@source "./(dashboard)/*.{js,ts,jsx,tsx}";
@source "./(dashboard)/jobs/*.{js,ts,jsx,tsx}";
@source "./(dashboard)/projects/*.{js,ts,jsx,tsx}";
@source "./(dashboard)/spiders/*.{js,ts,jsx,tsx}";
@source "./(dashboard)/system/*.{js,ts,jsx,tsx}";
@source "*.{js,ts,jsx,tsx}";
@custom-variant dark (&:is(.dark *));
:root {
--radius: 0.5rem;
/* Light Mode Colors - Using OKLCH for better color precision */
--background: oklch(100% 0 0);
--foreground: oklch(9.8% 0 0);
--card: oklch(100% 0 0);
--card-foreground: oklch(9.8% 0 0);
--popover: oklch(100% 0 0);
--popover-foreground: oklch(9.8% 0 0);
--primary: oklch(18% 0 0);
--primary-foreground: oklch(98% 0 0);
--secondary: oklch(96.1% 0 0);
--secondary-foreground: oklch(18% 0 0);
--muted: oklch(96.1% 0 0);
--muted-foreground: oklch(55% 0 0);
--accent: oklch(96.1% 0 0);
--accent-foreground: oklch(18% 0 0);
--destructive: oklch(62.8% 0.257 29.234);
--destructive-foreground: oklch(98% 0 0);
--border: oklch(89.8% 0 0);
--input: oklch(89.8% 0 0);
--ring: oklch(9.8% 0 0);
/* Chart Colors */
--chart-1: oklch(68% 0.14 29);
--chart-2: oklch(55% 0.14 192);
--chart-3: oklch(42% 0.10 218);
--chart-4: oklch(72% 0.15 84);
--chart-5: oklch(70% 0.18 41);
}
@theme inline {
/* Tailwind v4 theme color definitions */
--color-background: var(--background);
--color-foreground: var(--foreground);
--color-card: var(--card);
--color-card-foreground: var(--card-foreground);
--color-popover: var(--popover);
--color-popover-foreground: var(--popover-foreground);
--color-primary: var(--primary);
--color-primary-foreground: var(--primary-foreground);
--color-secondary: var(--secondary);
--color-secondary-foreground: var(--secondary-foreground);
--color-muted: var(--muted);
--color-muted-foreground: var(--muted-foreground);
--color-accent: var(--accent);
--color-accent-foreground: var(--accent-foreground);
--color-destructive: var(--destructive);
--color-destructive-foreground: var(--destructive-foreground);
--color-border: var(--border);
--color-input: var(--input);
--color-ring: var(--ring);
--color-chart-1: var(--chart-1);
--color-chart-2: var(--chart-2);
--color-chart-3: var(--chart-3);
--color-chart-4: var(--chart-4);
--color-chart-5: var(--chart-5);
}
.dark {
--background: oklch(15% 0 0);
--foreground: oklch(98% 0 0);
--card: oklch(15% 0 0);
--card-foreground: oklch(98% 0 0);
--popover: oklch(15% 0 0);
--popover-foreground: oklch(98% 0 0);
--primary: oklch(98% 0 0);
--primary-foreground: oklch(18% 0 0);
--secondary: oklch(25% 0 0);
--secondary-foreground: oklch(98% 0 0);
--muted: oklch(25% 0 0);
--muted-foreground: oklch(65% 0 0);
--accent: oklch(25% 0 0);
--accent-foreground: oklch(98% 0 0);
--destructive: oklch(45% 0.15 29);
--destructive-foreground: oklch(98% 0 0);
--border: oklch(25% 0 0);
--input: oklch(25% 0 0);
--ring: oklch(83% 0 0);
/* Chart Colors Dark */
--chart-1: oklch(60% 0.14 250);
--chart-2: oklch(55% 0.12 180);
--chart-3: oklch(65% 0.15 70);
--chart-4: oklch(68% 0.13 310);
--chart-5: oklch(66% 0.14 10);
}
@layer base {
* {
@apply border-border outline-ring/50;
}
body {
@apply bg-background text-foreground;
}
}

42
app/layout.tsx Normal file
View File

@@ -0,0 +1,42 @@
import type { Metadata } from "next";
import { Geist, Geist_Mono } from "next/font/google";
import "./globals.css";
import { ThemeProvider } from "@/components/theme-provider";
const geistSans = Geist({
variable: "--font-geist-sans",
subsets: ["latin"],
});
const geistMono = Geist_Mono({
variable: "--font-geist-mono",
subsets: ["latin"],
});
export const metadata: Metadata = {
title: "Scrapy UI - Scrapyd Management Interface",
description: "Manage and monitor your Scrapyd scraping projects",
};
export default function RootLayout({
children,
}: Readonly<{
children: React.ReactNode;
}>) {
return (
<html lang="en" suppressHydrationWarning>
<body
className={`${geistSans.variable} ${geistMono.variable} antialiased`}
>
<ThemeProvider
attribute="class"
defaultTheme="system"
enableSystem
disableTransitionOnChange
>
{children}
</ThemeProvider>
</body>
</html>
);
}

20
components.json Normal file
View File

@@ -0,0 +1,20 @@
{
"$schema": "https://ui.shadcn.com/schema.json",
"style": "new-york",
"rsc": true,
"tsx": true,
"tailwind": {
"config": "",
"css": "app/globals.css",
"baseColor": "neutral",
"cssVariables": true,
"prefix": ""
},
"aliases": {
"components": "@/components",
"utils": "@/lib/utils",
"ui": "@/components/ui",
"lib": "@/lib",
"hooks": "@/hooks"
}
}

19
components/header.tsx Normal file
View File

@@ -0,0 +1,19 @@
interface HeaderProps {
title: string;
description?: string;
action?: React.ReactNode;
}
export function Header({ title, description, action }: HeaderProps) {
return (
<div className="flex items-center justify-between">
<div>
<h2 className="text-3xl font-bold tracking-tight">{title}</h2>
{description && (
<p className="text-muted-foreground">{description}</p>
)}
</div>
{action && <div>{action}</div>}
</div>
);
}

22
components/providers.tsx Normal file
View File

@@ -0,0 +1,22 @@
"use client";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { useState } from "react";
export function Providers({ children }: { children: React.ReactNode }) {
const [queryClient] = useState(
() =>
new QueryClient({
defaultOptions: {
queries: {
staleTime: 60 * 1000, // 1 minute
refetchInterval: 30 * 1000, // 30 seconds for real-time updates
},
},
})
);
return (
<QueryClientProvider client={queryClient}>{children}</QueryClientProvider>
);
}

83
components/sidebar.tsx Normal file
View File

@@ -0,0 +1,83 @@
"use client";
import Link from "next/link";
import { usePathname } from "next/navigation";
import { cn } from "@/lib/utils";
import {
LayoutDashboard,
FolderKanban,
Bug,
BriefcaseBusiness,
Activity,
} from "lucide-react";
import { ThemeToggle } from "./theme-toggle";
const routes = [
{
label: "Dashboard",
icon: LayoutDashboard,
href: "/",
exact: true,
},
{
label: "Projects",
icon: FolderKanban,
href: "/projects",
},
{
label: "Spiders",
icon: Bug,
href: "/spiders",
},
{
label: "Jobs",
icon: BriefcaseBusiness,
href: "/jobs",
},
{
label: "System",
icon: Activity,
href: "/system",
},
];
export function Sidebar() {
const pathname = usePathname();
return (
<div className="flex h-full w-64 flex-col border-r bg-card">
<div className="flex h-16 items-center border-b px-6">
<h1 className="text-xl font-bold">Scrapy UI</h1>
</div>
<nav className="flex-1 gap-1 p-4">
{routes.map((route) => {
const isActive = route.exact
? pathname === route.href
: pathname?.startsWith(route.href);
return (
<Link
key={route.href}
href={route.href}
className={cn(
"flex items-center gap-3 rounded-lg px-3 py-2 text-sm font-medium transition-colors",
isActive
? "bg-primary text-primary-foreground"
: "text-muted-foreground hover:bg-accent hover:text-accent-foreground"
)}
>
<route.icon className="h-5 w-5" />
{route.label}
</Link>
);
})}
</nav>
<div className="border-t p-4">
<div className="flex items-center justify-between">
<span className="text-sm text-muted-foreground">Theme</span>
<ThemeToggle />
</div>
</div>
</div>
);
}

View File

@@ -0,0 +1,11 @@
"use client";
import * as React from "react";
import { ThemeProvider as NextThemesProvider } from "next-themes";
export function ThemeProvider({
children,
...props
}: React.ComponentProps<typeof NextThemesProvider>) {
return <NextThemesProvider {...props}>{children}</NextThemesProvider>;
}

View File

@@ -0,0 +1,32 @@
"use client";
import * as React from "react";
import { Moon, Sun } from "lucide-react";
import { useTheme } from "next-themes";
import { Button } from "@/components/ui/button";
export function ThemeToggle() {
const { theme, setTheme } = useTheme();
const [mounted, setMounted] = React.useState(false);
React.useEffect(() => {
setMounted(true);
}, []);
if (!mounted) {
return <Button variant="ghost" size="icon" disabled />;
}
return (
<Button
variant="ghost"
size="icon"
onClick={() => setTheme(theme === "light" ? "dark" : "light")}
>
<Sun className="h-5 w-5 rotate-0 scale-100 transition-all dark:-rotate-90 dark:scale-0" />
<Moon className="absolute h-5 w-5 rotate-90 scale-0 transition-all dark:rotate-0 dark:scale-100" />
<span className="sr-only">Toggle theme</span>
</Button>
);
}

36
components/ui/badge.tsx Normal file
View File

@@ -0,0 +1,36 @@
import * as React from "react"
import { cva, type VariantProps } from "class-variance-authority"
import { cn } from "@/lib/utils"
const badgeVariants = cva(
"inline-flex items-center rounded-md border px-2.5 py-0.5 text-xs font-semibold transition-colors focus:outline-none focus:ring-2 focus:ring-ring",
{
variants: {
variant: {
default:
"border-transparent bg-primary text-primary-foreground shadow hover:bg-primary/80",
secondary:
"border-transparent bg-secondary text-secondary-foreground hover:bg-secondary/80",
destructive:
"border-transparent bg-destructive text-destructive-foreground shadow hover:bg-destructive/80",
outline: "text-foreground",
},
},
defaultVariants: {
variant: "default",
},
}
)
export interface BadgeProps
extends React.HTMLAttributes<HTMLDivElement>,
VariantProps<typeof badgeVariants> {}
function Badge({ className, variant, ...props }: BadgeProps) {
return (
<div className={cn(badgeVariants({ variant }), className)} {...props} />
)
}
export { Badge, badgeVariants }

57
components/ui/button.tsx Normal file
View File

@@ -0,0 +1,57 @@
import * as React from "react"
import { Slot } from "@radix-ui/react-slot"
import { cva, type VariantProps } from "class-variance-authority"
import { cn } from "@/lib/utils"
const buttonVariants = cva(
"inline-flex items-center justify-center gap-2 whitespace-nowrap rounded-md text-sm font-medium transition-colors focus-visible:outline-none focus-visible:ring-1 focus-visible:ring-ring disabled:pointer-events-none disabled:opacity-50 [&_svg]:pointer-events-none [&_svg]:size-4 [&_svg]:shrink-0",
{
variants: {
variant: {
default:
"bg-primary text-primary-foreground shadow hover:bg-primary/90",
destructive:
"bg-destructive text-destructive-foreground shadow-sm hover:bg-destructive/90",
outline:
"border border-input bg-background shadow-sm hover:bg-accent hover:text-accent-foreground",
secondary:
"bg-secondary text-secondary-foreground shadow-sm hover:bg-secondary/80",
ghost: "hover:bg-accent hover:text-accent-foreground",
link: "text-primary underline-offset-4 hover:underline",
},
size: {
default: "h-9 px-4 py-2",
sm: "h-8 rounded-md px-3 text-xs",
lg: "h-10 rounded-md px-8",
icon: "h-9 w-9",
},
},
defaultVariants: {
variant: "default",
size: "default",
},
}
)
export interface ButtonProps
extends React.ButtonHTMLAttributes<HTMLButtonElement>,
VariantProps<typeof buttonVariants> {
asChild?: boolean
}
const Button = React.forwardRef<HTMLButtonElement, ButtonProps>(
({ className, variant, size, asChild = false, ...props }, ref) => {
const Comp = asChild ? Slot : "button"
return (
<Comp
className={cn(buttonVariants({ variant, size, className }))}
ref={ref}
{...props}
/>
)
}
)
Button.displayName = "Button"
export { Button, buttonVariants }

76
components/ui/card.tsx Normal file
View File

@@ -0,0 +1,76 @@
import * as React from "react"
import { cn } from "@/lib/utils"
const Card = React.forwardRef<
HTMLDivElement,
React.HTMLAttributes<HTMLDivElement>
>(({ className, ...props }, ref) => (
<div
ref={ref}
className={cn(
"rounded-xl border bg-card text-card-foreground shadow",
className
)}
{...props}
/>
))
Card.displayName = "Card"
const CardHeader = React.forwardRef<
HTMLDivElement,
React.HTMLAttributes<HTMLDivElement>
>(({ className, ...props }, ref) => (
<div
ref={ref}
className={cn("flex flex-col [flex flex-col gap->*:not(:last-child)]:mb-1.5 p-6", className)}
{...props}
/>
))
CardHeader.displayName = "CardHeader"
const CardTitle = React.forwardRef<
HTMLDivElement,
React.HTMLAttributes<HTMLDivElement>
>(({ className, ...props }, ref) => (
<div
ref={ref}
className={cn("font-semibold leading-none tracking-tight", className)}
{...props}
/>
))
CardTitle.displayName = "CardTitle"
const CardDescription = React.forwardRef<
HTMLDivElement,
React.HTMLAttributes<HTMLDivElement>
>(({ className, ...props }, ref) => (
<div
ref={ref}
className={cn("text-sm text-muted-foreground", className)}
{...props}
/>
))
CardDescription.displayName = "CardDescription"
const CardContent = React.forwardRef<
HTMLDivElement,
React.HTMLAttributes<HTMLDivElement>
>(({ className, ...props }, ref) => (
<div ref={ref} className={cn("p-6 pt-0", className)} {...props} />
))
CardContent.displayName = "CardContent"
const CardFooter = React.forwardRef<
HTMLDivElement,
React.HTMLAttributes<HTMLDivElement>
>(({ className, ...props }, ref) => (
<div
ref={ref}
className={cn("flex items-center p-6 pt-0", className)}
{...props}
/>
))
CardFooter.displayName = "CardFooter"
export { Card, CardHeader, CardFooter, CardTitle, CardDescription, CardContent }

121
components/ui/dialog.tsx Normal file
View File

@@ -0,0 +1,121 @@
"use client"
import * as React from "react"
import * as DialogPrimitive from "@radix-ui/react-dialog"
import { cn } from "@/lib/utils"
import { Cross2Icon } from "@radix-ui/react-icons"
const Dialog = DialogPrimitive.Root
const DialogTrigger = DialogPrimitive.Trigger
const DialogPortal = DialogPrimitive.Portal
const DialogClose = DialogPrimitive.Close
const DialogOverlay = React.forwardRef<
React.ElementRef<typeof DialogPrimitive.Overlay>,
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Overlay>
>(({ className, ...props }, ref) => (
<DialogPrimitive.Overlay
ref={ref}
className={cn(
"fixed inset-0 z-50 bg-black/80 data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0",
className
)}
{...props}
/>
))
DialogOverlay.displayName = DialogPrimitive.Overlay.displayName
const DialogContent = React.forwardRef<
React.ElementRef<typeof DialogPrimitive.Content>,
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Content>
>(({ className, children, ...props }, ref) => (
<DialogPortal>
<DialogOverlay />
<DialogPrimitive.Content
ref={ref}
className={cn(
"fixed left-[50%] top-[50%] z-50 grid w-full max-w-lg translate-x-[-50%] translate-y-[-50%] gap-4 border bg-background p-6 shadow-lg duration-200 data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95 data-[state=closed]:slide-out-to-left-1/2 data-[state=closed]:slide-out-to-top-[48%] data-[state=open]:slide-in-from-left-1/2 data-[state=open]:slide-in-from-top-[48%] sm:rounded-lg",
className
)}
{...props}
>
{children}
<DialogPrimitive.Close className="absolute right-4 top-4 rounded-sm opacity-70 transition-opacity hover:opacity-100 focus:outline-none focus:ring-2 focus:ring-ring disabled:pointer-events-none data-[state=open]:bg-accent data-[state=open]:text-muted-foreground">
<Cross2Icon className="h-4 w-4" />
<span className="sr-only">Close</span>
</DialogPrimitive.Close>
</DialogPrimitive.Content>
</DialogPortal>
))
DialogContent.displayName = DialogPrimitive.Content.displayName
const DialogHeader = ({
className,
...props
}: React.HTMLAttributes<HTMLDivElement>) => (
<div
className={cn(
"flex flex-col [flex flex-col gap->*:not(:last-child)]:mb-1.5 text-center sm:text-left",
className
)}
{...props}
/>
)
DialogHeader.displayName = "DialogHeader"
const DialogFooter = ({
className,
...props
}: React.HTMLAttributes<HTMLDivElement>) => (
<div
className={cn(
"flex flex-col-reverse sm:flex-row sm:justify-end sm:gap-2",
className
)}
{...props}
/>
)
DialogFooter.displayName = "DialogFooter"
const DialogTitle = React.forwardRef<
React.ElementRef<typeof DialogPrimitive.Title>,
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Title>
>(({ className, ...props }, ref) => (
<DialogPrimitive.Title
ref={ref}
className={cn(
"text-lg font-semibold leading-none tracking-tight",
className
)}
{...props}
/>
))
DialogTitle.displayName = DialogPrimitive.Title.displayName
const DialogDescription = React.forwardRef<
React.ElementRef<typeof DialogPrimitive.Description>,
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Description>
>(({ className, ...props }, ref) => (
<DialogPrimitive.Description
ref={ref}
className={cn("text-sm text-muted-foreground", className)}
{...props}
/>
))
DialogDescription.displayName = DialogPrimitive.Description.displayName
export {
Dialog,
DialogPortal,
DialogOverlay,
DialogTrigger,
DialogClose,
DialogContent,
DialogHeader,
DialogFooter,
DialogTitle,
DialogDescription,
}

View File

@@ -0,0 +1,200 @@
"use client"
import * as React from "react"
import * as DropdownMenuPrimitive from "@radix-ui/react-dropdown-menu"
import { cn } from "@/lib/utils"
import { CheckIcon, ChevronRightIcon, DotFilledIcon } from "@radix-ui/react-icons"
const DropdownMenu = DropdownMenuPrimitive.Root
const DropdownMenuTrigger = DropdownMenuPrimitive.Trigger
const DropdownMenuGroup = DropdownMenuPrimitive.Group
const DropdownMenuPortal = DropdownMenuPrimitive.Portal
const DropdownMenuSub = DropdownMenuPrimitive.Sub
const DropdownMenuRadioGroup = DropdownMenuPrimitive.RadioGroup
const DropdownMenuSubTrigger = React.forwardRef<
React.ElementRef<typeof DropdownMenuPrimitive.SubTrigger>,
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.SubTrigger> & {
inset?: boolean
}
>(({ className, inset, children, ...props }, ref) => (
<DropdownMenuPrimitive.SubTrigger
ref={ref}
className={cn(
"flex cursor-default select-none items-center gap-2 rounded-sm px-2 py-1.5 text-sm outline-none focus:bg-accent data-[state=open]:bg-accent [&_svg]:pointer-events-none [&_svg]:size-4 [&_svg]:shrink-0",
inset && "pl-8",
className
)}
{...props}
>
{children}
<ChevronRightIcon className="ml-auto" />
</DropdownMenuPrimitive.SubTrigger>
))
DropdownMenuSubTrigger.displayName =
DropdownMenuPrimitive.SubTrigger.displayName
const DropdownMenuSubContent = React.forwardRef<
React.ElementRef<typeof DropdownMenuPrimitive.SubContent>,
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.SubContent>
>(({ className, ...props }, ref) => (
<DropdownMenuPrimitive.SubContent
ref={ref}
className={cn(
"z-50 min-w-[8rem] overflow-hidden rounded-md border bg-popover p-1 text-popover-foreground shadow-lg data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95 data-[side=bottom]:slide-in-from-top-2 data-[side=left]:slide-in-from-right-2 data-[side=right]:slide-in-from-left-2 data-[side=top]:slide-in-from-bottom-2 origin-[--radix-dropdown-menu-content-transform-origin]",
className
)}
{...props}
/>
))
DropdownMenuSubContent.displayName =
DropdownMenuPrimitive.SubContent.displayName
const DropdownMenuContent = React.forwardRef<
React.ElementRef<typeof DropdownMenuPrimitive.Content>,
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Content>
>(({ className, sideOffset = 4, ...props }, ref) => (
<DropdownMenuPrimitive.Portal>
<DropdownMenuPrimitive.Content
ref={ref}
sideOffset={sideOffset}
className={cn(
"z-50 max-h-[var(--radix-dropdown-menu-content-available-height)] min-w-[8rem] overflow-y-auto overflow-x-hidden rounded-md border bg-popover p-1 text-popover-foreground shadow-md",
"data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95 data-[side=bottom]:slide-in-from-top-2 data-[side=left]:slide-in-from-right-2 data-[side=right]:slide-in-from-left-2 data-[side=top]:slide-in-from-bottom-2 origin-[--radix-dropdown-menu-content-transform-origin]",
className
)}
{...props}
/>
</DropdownMenuPrimitive.Portal>
))
DropdownMenuContent.displayName = DropdownMenuPrimitive.Content.displayName
const DropdownMenuItem = React.forwardRef<
React.ElementRef<typeof DropdownMenuPrimitive.Item>,
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Item> & {
inset?: boolean
}
>(({ className, inset, ...props }, ref) => (
<DropdownMenuPrimitive.Item
ref={ref}
className={cn(
"relative flex cursor-default select-none items-center gap-2 rounded-sm px-2 py-1.5 text-sm outline-none transition-colors focus:bg-accent focus:text-accent-foreground data-[disabled]:pointer-events-none data-[disabled]:opacity-50 [&>svg]:size-4 [&>svg]:shrink-0",
inset && "pl-8",
className
)}
{...props}
/>
))
DropdownMenuItem.displayName = DropdownMenuPrimitive.Item.displayName
const DropdownMenuCheckboxItem = React.forwardRef<
React.ElementRef<typeof DropdownMenuPrimitive.CheckboxItem>,
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.CheckboxItem>
>(({ className, children, checked, ...props }, ref) => (
<DropdownMenuPrimitive.CheckboxItem
ref={ref}
className={cn(
"relative flex cursor-default select-none items-center rounded-sm py-1.5 pl-8 pr-2 text-sm outline-none transition-colors focus:bg-accent focus:text-accent-foreground data-[disabled]:pointer-events-none data-[disabled]:opacity-50",
className
)}
checked={checked}
{...props}
>
<span className="absolute left-2 flex h-3.5 w-3.5 items-center justify-center">
<DropdownMenuPrimitive.ItemIndicator>
<CheckIcon className="h-4 w-4" />
</DropdownMenuPrimitive.ItemIndicator>
</span>
{children}
</DropdownMenuPrimitive.CheckboxItem>
))
DropdownMenuCheckboxItem.displayName =
DropdownMenuPrimitive.CheckboxItem.displayName
const DropdownMenuRadioItem = React.forwardRef<
React.ElementRef<typeof DropdownMenuPrimitive.RadioItem>,
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.RadioItem>
>(({ className, children, ...props }, ref) => (
<DropdownMenuPrimitive.RadioItem
ref={ref}
className={cn(
"relative flex cursor-default select-none items-center rounded-sm py-1.5 pl-8 pr-2 text-sm outline-none transition-colors focus:bg-accent focus:text-accent-foreground data-[disabled]:pointer-events-none data-[disabled]:opacity-50",
className
)}
{...props}
>
<span className="absolute left-2 flex h-3.5 w-3.5 items-center justify-center">
<DropdownMenuPrimitive.ItemIndicator>
<DotFilledIcon className="h-2 w-2 fill-current" />
</DropdownMenuPrimitive.ItemIndicator>
</span>
{children}
</DropdownMenuPrimitive.RadioItem>
))
DropdownMenuRadioItem.displayName = DropdownMenuPrimitive.RadioItem.displayName
const DropdownMenuLabel = React.forwardRef<
React.ElementRef<typeof DropdownMenuPrimitive.Label>,
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Label> & {
inset?: boolean
}
>(({ className, inset, ...props }, ref) => (
<DropdownMenuPrimitive.Label
ref={ref}
className={cn(
"px-2 py-1.5 text-sm font-semibold",
inset && "pl-8",
className
)}
{...props}
/>
))
DropdownMenuLabel.displayName = DropdownMenuPrimitive.Label.displayName
const DropdownMenuSeparator = React.forwardRef<
React.ElementRef<typeof DropdownMenuPrimitive.Separator>,
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Separator>
>(({ className, ...props }, ref) => (
<DropdownMenuPrimitive.Separator
ref={ref}
className={cn("-mx-1 my-1 h-px bg-muted", className)}
{...props}
/>
))
DropdownMenuSeparator.displayName = DropdownMenuPrimitive.Separator.displayName
const DropdownMenuShortcut = ({
className,
...props
}: React.HTMLAttributes<HTMLSpanElement>) => {
return (
<span
className={cn("ml-auto text-xs tracking-widest opacity-60", className)}
{...props}
/>
)
}
DropdownMenuShortcut.displayName = "DropdownMenuShortcut"
export {
DropdownMenu,
DropdownMenuTrigger,
DropdownMenuContent,
DropdownMenuItem,
DropdownMenuCheckboxItem,
DropdownMenuRadioItem,
DropdownMenuLabel,
DropdownMenuSeparator,
DropdownMenuShortcut,
DropdownMenuGroup,
DropdownMenuPortal,
DropdownMenuSub,
DropdownMenuSubContent,
DropdownMenuSubTrigger,
DropdownMenuRadioGroup,
}

22
components/ui/input.tsx Normal file
View File

@@ -0,0 +1,22 @@
import * as React from "react"
import { cn } from "@/lib/utils"
const Input = React.forwardRef<HTMLInputElement, React.ComponentProps<"input">>(
({ className, type, ...props }, ref) => {
return (
<input
type={type}
className={cn(
"flex h-9 w-full rounded-md border border-input bg-transparent px-3 py-1 text-base shadow-sm transition-colors file:border-0 file:bg-transparent file:text-sm file:font-medium file:text-foreground placeholder:text-muted-foreground focus-visible:outline-none focus-visible:ring-1 focus-visible:ring-ring disabled:cursor-not-allowed disabled:opacity-50 md:text-sm",
className
)}
ref={ref}
{...props}
/>
)
}
)
Input.displayName = "Input"
export { Input }

26
components/ui/label.tsx Normal file
View File

@@ -0,0 +1,26 @@
"use client"
import * as React from "react"
import * as LabelPrimitive from "@radix-ui/react-label"
import { cva, type VariantProps } from "class-variance-authority"
import { cn } from "@/lib/utils"
const labelVariants = cva(
"text-sm font-medium leading-none peer-disabled:cursor-not-allowed peer-disabled:opacity-70"
)
const Label = React.forwardRef<
React.ElementRef<typeof LabelPrimitive.Root>,
React.ComponentPropsWithoutRef<typeof LabelPrimitive.Root> &
VariantProps<typeof labelVariants>
>(({ className, ...props }, ref) => (
<LabelPrimitive.Root
ref={ref}
className={cn(labelVariants(), className)}
{...props}
/>
))
Label.displayName = LabelPrimitive.Root.displayName
export { Label }

158
components/ui/select.tsx Normal file
View File

@@ -0,0 +1,158 @@
"use client"
import * as React from "react"
import * as SelectPrimitive from "@radix-ui/react-select"
import { cn } from "@/lib/utils"
import { CheckIcon, ChevronDownIcon, ChevronUpIcon } from "@radix-ui/react-icons"
const Select = SelectPrimitive.Root
const SelectGroup = SelectPrimitive.Group
const SelectValue = SelectPrimitive.Value
const SelectTrigger = React.forwardRef<
React.ElementRef<typeof SelectPrimitive.Trigger>,
React.ComponentPropsWithoutRef<typeof SelectPrimitive.Trigger>
>(({ className, children, ...props }, ref) => (
<SelectPrimitive.Trigger
ref={ref}
className={cn(
"flex h-9 w-full items-center justify-between whitespace-nowrap rounded-md border border-input bg-transparent px-3 py-2 text-sm shadow-sm data-[placeholder]:text-muted-foreground focus:outline-none focus:ring-1 focus:ring-ring disabled:cursor-not-allowed disabled:opacity-50 [&>span]:line-clamp-1",
className
)}
{...props}
>
{children}
<SelectPrimitive.Icon asChild>
<ChevronDownIcon className="h-4 w-4 opacity-50" />
</SelectPrimitive.Icon>
</SelectPrimitive.Trigger>
))
SelectTrigger.displayName = SelectPrimitive.Trigger.displayName
const SelectScrollUpButton = React.forwardRef<
React.ElementRef<typeof SelectPrimitive.ScrollUpButton>,
React.ComponentPropsWithoutRef<typeof SelectPrimitive.ScrollUpButton>
>(({ className, ...props }, ref) => (
<SelectPrimitive.ScrollUpButton
ref={ref}
className={cn(
"flex cursor-default items-center justify-center py-1",
className
)}
{...props}
>
<ChevronUpIcon className="h-4 w-4" />
</SelectPrimitive.ScrollUpButton>
))
SelectScrollUpButton.displayName = SelectPrimitive.ScrollUpButton.displayName
const SelectScrollDownButton = React.forwardRef<
React.ElementRef<typeof SelectPrimitive.ScrollDownButton>,
React.ComponentPropsWithoutRef<typeof SelectPrimitive.ScrollDownButton>
>(({ className, ...props }, ref) => (
<SelectPrimitive.ScrollDownButton
ref={ref}
className={cn(
"flex cursor-default items-center justify-center py-1",
className
)}
{...props}
>
<ChevronDownIcon className="h-4 w-4" />
</SelectPrimitive.ScrollDownButton>
))
SelectScrollDownButton.displayName =
SelectPrimitive.ScrollDownButton.displayName
const SelectContent = React.forwardRef<
React.ElementRef<typeof SelectPrimitive.Content>,
React.ComponentPropsWithoutRef<typeof SelectPrimitive.Content>
>(({ className, children, position = "popper", ...props }, ref) => (
<SelectPrimitive.Portal>
<SelectPrimitive.Content
ref={ref}
className={cn(
"relative z-50 max-h-[--radix-select-content-available-height] min-w-[8rem] overflow-y-auto overflow-x-hidden rounded-md border bg-popover text-popover-foreground shadow-md data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95 data-[side=bottom]:slide-in-from-top-2 data-[side=left]:slide-in-from-right-2 data-[side=right]:slide-in-from-left-2 data-[side=top]:slide-in-from-bottom-2 origin-[--radix-select-content-transform-origin]",
position === "popper" &&
"data-[side=bottom]:translate-y-1 data-[side=left]:-translate-x-1 data-[side=right]:translate-x-1 data-[side=top]:-translate-y-1",
className
)}
position={position}
{...props}
>
<SelectScrollUpButton />
<SelectPrimitive.Viewport
className={cn(
"p-1",
position === "popper" &&
"h-[var(--radix-select-trigger-height)] w-full min-w-[var(--radix-select-trigger-width)]"
)}
>
{children}
</SelectPrimitive.Viewport>
<SelectScrollDownButton />
</SelectPrimitive.Content>
</SelectPrimitive.Portal>
))
SelectContent.displayName = SelectPrimitive.Content.displayName
const SelectLabel = React.forwardRef<
React.ElementRef<typeof SelectPrimitive.Label>,
React.ComponentPropsWithoutRef<typeof SelectPrimitive.Label>
>(({ className, ...props }, ref) => (
<SelectPrimitive.Label
ref={ref}
className={cn("px-2 py-1.5 text-sm font-semibold", className)}
{...props}
/>
))
SelectLabel.displayName = SelectPrimitive.Label.displayName
const SelectItem = React.forwardRef<
React.ElementRef<typeof SelectPrimitive.Item>,
React.ComponentPropsWithoutRef<typeof SelectPrimitive.Item>
>(({ className, children, ...props }, ref) => (
<SelectPrimitive.Item
ref={ref}
className={cn(
"relative flex w-full cursor-default select-none items-center rounded-sm py-1.5 pl-2 pr-8 text-sm outline-none focus:bg-accent focus:text-accent-foreground data-[disabled]:pointer-events-none data-[disabled]:opacity-50",
className
)}
{...props}
>
<span className="absolute right-2 flex h-3.5 w-3.5 items-center justify-center">
<SelectPrimitive.ItemIndicator>
<CheckIcon className="h-4 w-4" />
</SelectPrimitive.ItemIndicator>
</span>
<SelectPrimitive.ItemText>{children}</SelectPrimitive.ItemText>
</SelectPrimitive.Item>
))
SelectItem.displayName = SelectPrimitive.Item.displayName
const SelectSeparator = React.forwardRef<
React.ElementRef<typeof SelectPrimitive.Separator>,
React.ComponentPropsWithoutRef<typeof SelectPrimitive.Separator>
>(({ className, ...props }, ref) => (
<SelectPrimitive.Separator
ref={ref}
className={cn("-mx-1 my-1 h-px bg-muted", className)}
{...props}
/>
))
SelectSeparator.displayName = SelectPrimitive.Separator.displayName
export {
Select,
SelectGroup,
SelectValue,
SelectTrigger,
SelectContent,
SelectLabel,
SelectItem,
SelectSeparator,
SelectScrollUpButton,
SelectScrollDownButton,
}

View File

@@ -0,0 +1,31 @@
"use client"
import * as React from "react"
import * as SeparatorPrimitive from "@radix-ui/react-separator"
import { cn } from "@/lib/utils"
const Separator = React.forwardRef<
React.ElementRef<typeof SeparatorPrimitive.Root>,
React.ComponentPropsWithoutRef<typeof SeparatorPrimitive.Root>
>(
(
{ className, orientation = "horizontal", decorative = true, ...props },
ref
) => (
<SeparatorPrimitive.Root
ref={ref}
decorative={decorative}
orientation={orientation}
className={cn(
"shrink-0 bg-border",
orientation === "horizontal" ? "h-[1px] w-full" : "h-full w-[1px]",
className
)}
{...props}
/>
)
)
Separator.displayName = SeparatorPrimitive.Root.displayName
export { Separator }

View File

@@ -0,0 +1,15 @@
import { cn } from "@/lib/utils"
function Skeleton({
className,
...props
}: React.HTMLAttributes<HTMLDivElement>) {
return (
<div
className={cn("animate-pulse rounded-md bg-primary/10", className)}
{...props}
/>
)
}
export { Skeleton }

120
components/ui/table.tsx Normal file
View File

@@ -0,0 +1,120 @@
import * as React from "react"
import { cn } from "@/lib/utils"
const Table = React.forwardRef<
HTMLTableElement,
React.HTMLAttributes<HTMLTableElement>
>(({ className, ...props }, ref) => (
<div className="relative w-full overflow-auto">
<table
ref={ref}
className={cn("w-full caption-bottom text-sm", className)}
{...props}
/>
</div>
))
Table.displayName = "Table"
const TableHeader = React.forwardRef<
HTMLTableSectionElement,
React.HTMLAttributes<HTMLTableSectionElement>
>(({ className, ...props }, ref) => (
<thead ref={ref} className={cn("[&_tr]:border-b", className)} {...props} />
))
TableHeader.displayName = "TableHeader"
const TableBody = React.forwardRef<
HTMLTableSectionElement,
React.HTMLAttributes<HTMLTableSectionElement>
>(({ className, ...props }, ref) => (
<tbody
ref={ref}
className={cn("[&_tr:last-child]:border-0", className)}
{...props}
/>
))
TableBody.displayName = "TableBody"
const TableFooter = React.forwardRef<
HTMLTableSectionElement,
React.HTMLAttributes<HTMLTableSectionElement>
>(({ className, ...props }, ref) => (
<tfoot
ref={ref}
className={cn(
"border-t bg-muted/50 font-medium [&>tr]:last:border-b-0",
className
)}
{...props}
/>
))
TableFooter.displayName = "TableFooter"
const TableRow = React.forwardRef<
HTMLTableRowElement,
React.HTMLAttributes<HTMLTableRowElement>
>(({ className, ...props }, ref) => (
<tr
ref={ref}
className={cn(
"border-b transition-colors hover:bg-muted/50 data-[state=selected]:bg-muted",
className
)}
{...props}
/>
))
TableRow.displayName = "TableRow"
const TableHead = React.forwardRef<
HTMLTableCellElement,
React.ThHTMLAttributes<HTMLTableCellElement>
>(({ className, ...props }, ref) => (
<th
ref={ref}
className={cn(
"h-10 px-2 text-left align-middle font-medium text-muted-foreground [&:has([role=checkbox])]:pr-0 [&>[role=checkbox]]:translate-y-[2px]",
className
)}
{...props}
/>
))
TableHead.displayName = "TableHead"
const TableCell = React.forwardRef<
HTMLTableCellElement,
React.TdHTMLAttributes<HTMLTableCellElement>
>(({ className, ...props }, ref) => (
<td
ref={ref}
className={cn(
"p-2 align-middle [&:has([role=checkbox])]:pr-0 [&>[role=checkbox]]:translate-y-[2px]",
className
)}
{...props}
/>
))
TableCell.displayName = "TableCell"
const TableCaption = React.forwardRef<
HTMLTableCaptionElement,
React.HTMLAttributes<HTMLTableCaptionElement>
>(({ className, ...props }, ref) => (
<caption
ref={ref}
className={cn("mt-4 text-sm text-muted-foreground", className)}
{...props}
/>
))
TableCaption.displayName = "TableCaption"
export {
Table,
TableHeader,
TableBody,
TableFooter,
TableHead,
TableRow,
TableCell,
TableCaption,
}

View File

@@ -0,0 +1,22 @@
import * as React from "react"
import { cn } from "@/lib/utils"
const Textarea = React.forwardRef<
HTMLTextAreaElement,
React.ComponentProps<"textarea">
>(({ className, ...props }, ref) => {
return (
<textarea
className={cn(
"flex min-h-[60px] w-full rounded-md border border-input bg-transparent px-3 py-2 text-base shadow-sm placeholder:text-muted-foreground focus-visible:outline-none focus-visible:ring-1 focus-visible:ring-ring disabled:cursor-not-allowed disabled:opacity-50 md:text-sm",
className
)}
ref={ref}
{...props}
/>
)
})
Textarea.displayName = "Textarea"
export { Textarea }

30
docker-compose.dev.yml Normal file
View File

@@ -0,0 +1,30 @@
version: '3.8'
services:
scrapy-ui-dev:
build:
context: .
dockerfile: Dockerfile.dev
image: scrapy-ui:dev
container_name: scrapy-ui-dev
restart: unless-stopped
ports:
- "3000:3000"
volumes:
- .:/app
- /app/node_modules
- /app/.next
environment:
- NODE_ENV=development
- SCRAPYD_URL=${SCRAPYD_URL:-https://scrapy.pivoine.art}
- SCRAPYD_USERNAME=${SCRAPYD_USERNAME}
- SCRAPYD_PASSWORD=${SCRAPYD_PASSWORD}
env_file:
- .env.local
networks:
- scrapy-network
command: pnpm dev
networks:
scrapy-network:
driver: bridge

25
docker-compose.yml Normal file
View File

@@ -0,0 +1,25 @@
version: '3.8'
services:
scrapy-ui:
build:
context: .
dockerfile: Dockerfile
image: scrapy-ui:latest
container_name: scrapy-ui
restart: unless-stopped
ports:
- "3000:3000"
environment:
- NODE_ENV=production
- SCRAPYD_URL=${SCRAPYD_URL:-https://scrapy.pivoine.art}
- SCRAPYD_USERNAME=${SCRAPYD_USERNAME}
- SCRAPYD_PASSWORD=${SCRAPYD_PASSWORD}
env_file:
- .env.local
networks:
- scrapy-network
networks:
scrapy-network:
driver: bridge

18
eslint.config.mjs Normal file
View File

@@ -0,0 +1,18 @@
import { defineConfig, globalIgnores } from "eslint/config";
import nextVitals from "eslint-config-next/core-web-vitals";
import nextTs from "eslint-config-next/typescript";
const eslintConfig = defineConfig([
...nextVitals,
...nextTs,
// Override default ignores of eslint-config-next.
globalIgnores([
// Default ignores of eslint-config-next:
".next/**",
"out/**",
"build/**",
"next-env.d.ts",
]),
]);
export default eslintConfig;

216
lib/scrapyd-client.ts Normal file
View File

@@ -0,0 +1,216 @@
import {
DaemonStatusSchema,
ListProjectsSchema,
ListVersionsSchema,
ListSpidersSchema,
ListJobsSchema,
ScheduleJobSchema,
CancelJobSchema,
DeleteVersionSchema,
DeleteProjectSchema,
AddVersionSchema,
type ScheduleJobParams,
type CancelJobParams,
type ListVersionsParams,
type ListSpidersParams,
type ListJobsParams,
type DeleteVersionParams,
type DeleteProjectParams,
} from "./types";
// Get credentials from environment variables (server-side only)
const SCRAPYD_URL = process.env.SCRAPYD_URL || "https://scrapy.pivoine.art";
const SCRAPYD_USERNAME = process.env.SCRAPYD_USERNAME || "";
const SCRAPYD_PASSWORD = process.env.SCRAPYD_PASSWORD || "";
/**
* Create Basic Auth header
*/
function getAuthHeader(): string {
const credentials = Buffer.from(`${SCRAPYD_USERNAME}:${SCRAPYD_PASSWORD}`).toString("base64");
return `Basic ${credentials}`;
}
/**
* Base fetch wrapper with auth
*/
async function fetchScrapyd(endpoint: string, options: RequestInit = {}) {
const url = `${SCRAPYD_URL}/${endpoint}`;
const response = await fetch(url, {
...options,
headers: {
Authorization: getAuthHeader(),
...options.headers,
},
});
if (!response.ok) {
throw new Error(`Scrapyd API error: ${response.status} ${response.statusText}`);
}
return response.json();
}
/**
* ScrapydClient - Server-side API client for Scrapyd
* All methods use environment variables for authentication
*/
export const ScrapydClient = {
/**
* Get daemon status
*/
async getDaemonStatus() {
const data = await fetchScrapyd("daemonstatus.json");
return DaemonStatusSchema.parse(data);
},
/**
* List all projects
*/
async listProjects() {
const data = await fetchScrapyd("listprojects.json");
return ListProjectsSchema.parse(data);
},
/**
* List versions for a project
*/
async listVersions(params: ListVersionsParams) {
const url = new URLSearchParams({ project: params.project });
const data = await fetchScrapyd(`listversions.json?${url}`);
return ListVersionsSchema.parse(data);
},
/**
* List spiders for a project
*/
async listSpiders(params: ListSpidersParams) {
const url = new URLSearchParams({
project: params.project,
...(params.version && { _version: params.version }),
});
const data = await fetchScrapyd(`listspiders.json?${url}`);
return ListSpidersSchema.parse(data);
},
/**
* List jobs (pending, running, finished) for a project
*/
async listJobs(params: ListJobsParams) {
const url = new URLSearchParams({ project: params.project });
const data = await fetchScrapyd(`listjobs.json?${url}`);
return ListJobsSchema.parse(data);
},
/**
* Schedule a spider job
*/
async scheduleJob(params: ScheduleJobParams) {
const formData = new URLSearchParams({
project: params.project,
spider: params.spider,
...(params.jobid && { jobid: params.jobid }),
});
// Add custom settings
if (params.settings) {
Object.entries(params.settings).forEach(([key, value]) => {
formData.append(`setting`, `${key}=${value}`);
});
}
// Add spider arguments
if (params.args) {
Object.entries(params.args).forEach(([key, value]) => {
formData.append(key, value);
});
}
const data = await fetchScrapyd("schedule.json", {
method: "POST",
headers: {
"Content-Type": "application/x-www-form-urlencoded",
},
body: formData.toString(),
});
return ScheduleJobSchema.parse(data);
},
/**
* Cancel a job
*/
async cancelJob(params: CancelJobParams) {
const formData = new URLSearchParams({
project: params.project,
job: params.job,
});
const data = await fetchScrapyd("cancel.json", {
method: "POST",
headers: {
"Content-Type": "application/x-www-form-urlencoded",
},
body: formData.toString(),
});
return CancelJobSchema.parse(data);
},
/**
* Delete a project version
*/
async deleteVersion(params: DeleteVersionParams) {
const formData = new URLSearchParams({
project: params.project,
version: params.version,
});
const data = await fetchScrapyd("delversion.json", {
method: "POST",
headers: {
"Content-Type": "application/x-www-form-urlencoded",
},
body: formData.toString(),
});
return DeleteVersionSchema.parse(data);
},
/**
* Delete a project
*/
async deleteProject(params: DeleteProjectParams) {
const formData = new URLSearchParams({
project: params.project,
});
const data = await fetchScrapyd("delproject.json", {
method: "POST",
headers: {
"Content-Type": "application/x-www-form-urlencoded",
},
body: formData.toString(),
});
return DeleteProjectSchema.parse(data);
},
/**
* Add/upload a project version (egg file)
*/
async addVersion(project: string, version: string, eggFile: Buffer) {
const formData = new FormData();
formData.append("project", project);
formData.append("version", version);
formData.append("egg", new Blob([new Uint8Array(eggFile)]), "project.egg");
const data = await fetchScrapyd("addversion.json", {
method: "POST",
body: formData,
});
return AddVersionSchema.parse(data);
},
};

120
lib/types.ts Normal file
View File

@@ -0,0 +1,120 @@
import { z } from "zod";
// Scrapyd API Response Schemas
export const DaemonStatusSchema = z.object({
status: z.literal("ok"),
node_name: z.string(),
pending: z.number(),
running: z.number(),
finished: z.number(),
});
export const ListProjectsSchema = z.object({
status: z.literal("ok"),
projects: z.array(z.string()),
});
export const ListVersionsSchema = z.object({
status: z.literal("ok"),
versions: z.array(z.string()),
});
export const ListSpidersSchema = z.object({
status: z.literal("ok"),
spiders: z.array(z.string()),
});
export const JobSchema = z.object({
id: z.string(),
spider: z.string(),
pid: z.number().optional(),
start_time: z.string(),
end_time: z.string().optional(),
log_url: z.string().optional(),
items_url: z.string().optional(),
});
export const ListJobsSchema = z.object({
status: z.literal("ok"),
pending: z.array(JobSchema),
running: z.array(JobSchema),
finished: z.array(JobSchema),
});
export const ScheduleJobSchema = z.object({
status: z.literal("ok"),
jobid: z.string(),
});
export const CancelJobSchema = z.object({
status: z.literal("ok"),
prevstate: z.enum(["pending", "running", "finished"]),
});
export const DeleteVersionSchema = z.object({
status: z.literal("ok"),
});
export const DeleteProjectSchema = z.object({
status: z.literal("ok"),
});
export const AddVersionSchema = z.object({
status: z.literal("ok"),
spiders: z.number(),
});
// TypeScript Types
export type DaemonStatus = z.infer<typeof DaemonStatusSchema>;
export type ListProjects = z.infer<typeof ListProjectsSchema>;
export type ListVersions = z.infer<typeof ListVersionsSchema>;
export type ListSpiders = z.infer<typeof ListSpidersSchema>;
export type Job = z.infer<typeof JobSchema>;
export type ListJobs = z.infer<typeof ListJobsSchema>;
export type ScheduleJob = z.infer<typeof ScheduleJobSchema>;
export type CancelJob = z.infer<typeof CancelJobSchema>;
export type DeleteVersion = z.infer<typeof DeleteVersionSchema>;
export type DeleteProject = z.infer<typeof DeleteProjectSchema>;
export type AddVersion = z.infer<typeof AddVersionSchema>;
// Request Parameters
export interface ScheduleJobParams {
project: string;
spider: string;
jobid?: string;
settings?: Record<string, string>;
args?: Record<string, string>;
}
export interface CancelJobParams {
project: string;
job: string;
}
export interface ListVersionsParams {
project: string;
}
export interface ListSpidersParams {
project: string;
version?: string;
}
export interface ListJobsParams {
project: string;
}
export interface DeleteVersionParams {
project: string;
version: string;
}
export interface DeleteProjectParams {
project: string;
}
export interface AddVersionParams {
project: string;
version: string;
egg: File;
}

6
lib/utils.ts Normal file
View File

@@ -0,0 +1,6 @@
import { clsx, type ClassValue } from "clsx"
import { twMerge } from "tailwind-merge"
export function cn(...inputs: ClassValue[]) {
return twMerge(clsx(inputs))
}

14
next.config.ts Normal file
View File

@@ -0,0 +1,14 @@
import type { NextConfig } from "next";
const nextConfig: NextConfig = {
basePath: "/ui",
output: "standalone",
experimental: {
optimizePackageImports: ["lucide-react"],
},
turbopack: {
root: process.cwd(),
}
};
export default nextConfig;

43
package.json Normal file
View File

@@ -0,0 +1,43 @@
{
"name": "scrapy-ui",
"version": "1.0.0",
"private": true,
"type": "module",
"scripts": {
"dev": "next dev",
"build": "next build",
"start": "next start",
"lint": "eslint"
},
"dependencies": {
"@radix-ui/react-dialog": "^1.1.15",
"@radix-ui/react-dropdown-menu": "^2.1.16",
"@radix-ui/react-icons": "^1.3.2",
"@radix-ui/react-label": "^2.1.8",
"@radix-ui/react-select": "^2.2.6",
"@radix-ui/react-separator": "^1.1.8",
"@radix-ui/react-slot": "^1.2.4",
"@tanstack/react-query": "^5.61.5",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"date-fns": "^4.1.0",
"lucide-react": "^0.460.0",
"next": "16.0.1",
"next-themes": "^0.4.4",
"react": "19.2.0",
"react-dom": "19.2.0",
"tailwind-merge": "^2.6.0",
"zod": "^3.23.8"
},
"devDependencies": {
"@tailwindcss/postcss": "^4.1.16",
"@types/node": "^20",
"@types/react": "^19",
"@types/react-dom": "^19",
"eslint": "^9",
"eslint-config-next": "16.0.1",
"postcss": "^8.5.6",
"tailwindcss": "^4.1.16",
"typescript": "^5"
}
}

4970
pnpm-lock.yaml generated Normal file

File diff suppressed because it is too large Load Diff

3
pnpm-workspace.yaml Normal file
View File

@@ -0,0 +1,3 @@
onlyBuiltDependencies:
- sharp
- unrs-resolver

5
postcss.config.mjs Normal file
View File

@@ -0,0 +1,5 @@
const config = {
plugins: ['@tailwindcss/postcss'],
};
export default config;

1
public/file.svg Normal file
View File

@@ -0,0 +1 @@
<svg fill="none" viewBox="0 0 16 16" xmlns="http://www.w3.org/2000/svg"><path d="M14.5 13.5V5.41a1 1 0 0 0-.3-.7L9.8.29A1 1 0 0 0 9.08 0H1.5v13.5A2.5 2.5 0 0 0 4 16h8a2.5 2.5 0 0 0 2.5-2.5m-1.5 0v-7H8v-5H3v12a1 1 0 0 0 1 1h8a1 1 0 0 0 1-1M9.5 5V2.12L12.38 5zM5.13 5h-.62v1.25h2.12V5zm-.62 3h7.12v1.25H4.5zm.62 3h-.62v1.25h7.12V11z" clip-rule="evenodd" fill="#666" fill-rule="evenodd"/></svg>

After

Width:  |  Height:  |  Size: 391 B

1
public/globe.svg Normal file
View File

@@ -0,0 +1 @@
<svg fill="none" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16"><g clip-path="url(#a)"><path fill-rule="evenodd" clip-rule="evenodd" d="M10.27 14.1a6.5 6.5 0 0 0 3.67-3.45q-1.24.21-2.7.34-.31 1.83-.97 3.1M8 16A8 8 0 1 0 8 0a8 8 0 0 0 0 16m.48-1.52a7 7 0 0 1-.96 0H7.5a4 4 0 0 1-.84-1.32q-.38-.89-.63-2.08a40 40 0 0 0 3.92 0q-.25 1.2-.63 2.08a4 4 0 0 1-.84 1.31zm2.94-4.76q1.66-.15 2.95-.43a7 7 0 0 0 0-2.58q-1.3-.27-2.95-.43a18 18 0 0 1 0 3.44m-1.27-3.54a17 17 0 0 1 0 3.64 39 39 0 0 1-4.3 0 17 17 0 0 1 0-3.64 39 39 0 0 1 4.3 0m1.1-1.17q1.45.13 2.69.34a6.5 6.5 0 0 0-3.67-3.44q.65 1.26.98 3.1M8.48 1.5l.01.02q.41.37.84 1.31.38.89.63 2.08a40 40 0 0 0-3.92 0q.25-1.2.63-2.08a4 4 0 0 1 .85-1.32 7 7 0 0 1 .96 0m-2.75.4a6.5 6.5 0 0 0-3.67 3.44 29 29 0 0 1 2.7-.34q.31-1.83.97-3.1M4.58 6.28q-1.66.16-2.95.43a7 7 0 0 0 0 2.58q1.3.27 2.95.43a18 18 0 0 1 0-3.44m.17 4.71q-1.45-.12-2.69-.34a6.5 6.5 0 0 0 3.67 3.44q-.65-1.27-.98-3.1" fill="#666"/></g><defs><clipPath id="a"><path fill="#fff" d="M0 0h16v16H0z"/></clipPath></defs></svg>

After

Width:  |  Height:  |  Size: 1.0 KiB

1
public/next.svg Normal file
View File

@@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 394 80"><path fill="#000" d="M262 0h68.5v12.7h-27.2v66.6h-13.6V12.7H262V0ZM149 0v12.7H94v20.4h44.3v12.6H94v21h55v12.6H80.5V0h68.7zm34.3 0h-17.8l63.8 79.4h17.9l-32-39.7 32-39.6h-17.9l-23 28.6-23-28.6zm18.3 56.7-9-11-27.1 33.7h17.8l18.3-22.7z"/><path fill="#000" d="M81 79.3 17 0H0v79.3h13.6V17l50.2 62.3H81Zm252.6-.4c-1 0-1.8-.4-2.5-1s-1.1-1.6-1.1-2.6.3-1.8 1-2.5 1.6-1 2.6-1 1.8.3 2.5 1a3.4 3.4 0 0 1 .6 4.3 3.7 3.7 0 0 1-3 1.8zm23.2-33.5h6v23.3c0 2.1-.4 4-1.3 5.5a9.1 9.1 0 0 1-3.8 3.5c-1.6.8-3.5 1.3-5.7 1.3-2 0-3.7-.4-5.3-1s-2.8-1.8-3.7-3.2c-.9-1.3-1.4-3-1.4-5h6c.1.8.3 1.6.7 2.2s1 1.2 1.6 1.5c.7.4 1.5.5 2.4.5 1 0 1.8-.2 2.4-.6a4 4 0 0 0 1.6-1.8c.3-.8.5-1.8.5-3V45.5zm30.9 9.1a4.4 4.4 0 0 0-2-3.3 7.5 7.5 0 0 0-4.3-1.1c-1.3 0-2.4.2-3.3.5-.9.4-1.6 1-2 1.6a3.5 3.5 0 0 0-.3 4c.3.5.7.9 1.3 1.2l1.8 1 2 .5 3.2.8c1.3.3 2.5.7 3.7 1.2a13 13 0 0 1 3.2 1.8 8.1 8.1 0 0 1 3 6.5c0 2-.5 3.7-1.5 5.1a10 10 0 0 1-4.4 3.5c-1.8.8-4.1 1.2-6.8 1.2-2.6 0-4.9-.4-6.8-1.2-2-.8-3.4-2-4.5-3.5a10 10 0 0 1-1.7-5.6h6a5 5 0 0 0 3.5 4.6c1 .4 2.2.6 3.4.6 1.3 0 2.5-.2 3.5-.6 1-.4 1.8-1 2.4-1.7a4 4 0 0 0 .8-2.4c0-.9-.2-1.6-.7-2.2a11 11 0 0 0-2.1-1.4l-3.2-1-3.8-1c-2.8-.7-5-1.7-6.6-3.2a7.2 7.2 0 0 1-2.4-5.7 8 8 0 0 1 1.7-5 10 10 0 0 1 4.3-3.5c2-.8 4-1.2 6.4-1.2 2.3 0 4.4.4 6.2 1.2 1.8.8 3.2 2 4.3 3.4 1 1.4 1.5 3 1.5 5h-5.8z"/></svg>

After

Width:  |  Height:  |  Size: 1.3 KiB

1
public/vercel.svg Normal file
View File

@@ -0,0 +1 @@
<svg fill="none" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1155 1000"><path d="m577.3 0 577.4 1000H0z" fill="#fff"/></svg>

After

Width:  |  Height:  |  Size: 128 B

1
public/window.svg Normal file
View File

@@ -0,0 +1 @@
<svg fill="none" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16"><path fill-rule="evenodd" clip-rule="evenodd" d="M1.5 2.5h13v10a1 1 0 0 1-1 1h-11a1 1 0 0 1-1-1zM0 1h16v11.5a2.5 2.5 0 0 1-2.5 2.5h-11A2.5 2.5 0 0 1 0 12.5zm3.75 4.5a.75.75 0 1 0 0-1.5.75.75 0 0 0 0 1.5M7 4.75a.75.75 0 1 1-1.5 0 .75.75 0 0 1 1.5 0m1.75.75a.75.75 0 1 0 0-1.5.75.75 0 0 0 0 1.5" fill="#666"/></svg>

After

Width:  |  Height:  |  Size: 385 B

42
tsconfig.json Normal file
View File

@@ -0,0 +1,42 @@
{
"compilerOptions": {
"target": "ES2017",
"lib": [
"dom",
"dom.iterable",
"esnext"
],
"allowJs": true,
"skipLibCheck": true,
"strict": true,
"noEmit": true,
"esModuleInterop": true,
"module": "esnext",
"moduleResolution": "bundler",
"resolveJsonModule": true,
"isolatedModules": true,
"jsx": "react-jsx",
"incremental": true,
"plugins": [
{
"name": "next"
}
],
"paths": {
"@/*": [
"./*"
]
}
},
"include": [
"next-env.d.ts",
"**/*.ts",
"**/*.tsx",
".next/types/**/*.ts",
".next/dev/types/**/*.ts",
"**/*.mts"
],
"exclude": [
"node_modules"
]
}