mirror of
https://github.com/yusufipk/RepoHub.git
synced 2026-09-11 10:36:07 +00:00
feat: add AUR repository support for Arch Linux packages
- Extended database schema and models to include 'aur' as a valid repository type alongside 'official' and 'third-party' - Added repository filter UI in package browser for Arch platform to distinguish between official and AUR packages - Enhanced init-db script with migration system to track and apply schema changes automatically
This commit is contained in:
@@ -1,211 +0,0 @@
|
|||||||
# RepoHub Sync Security Configuration
|
|
||||||
|
|
||||||
Bu doküman, RepoHub sync işlemlerinin güvenliği ve otomatikleştirilmesi için yapılandırmayı açıklar.
|
|
||||||
|
|
||||||
## Güvenlik Yapılandırması
|
|
||||||
|
|
||||||
### Environment Değişkenleri
|
|
||||||
|
|
||||||
`.env.local` dosyasına aşağı değişkenleri ekleyin:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Sync Configuration
|
|
||||||
# Enable sync operations only from the server itself (set to 'true' on production server)
|
|
||||||
SYNC_SERVER_ONLY=true
|
|
||||||
# Automatic sync frequency in days (set to 0 to disable automatic sync)
|
|
||||||
AUTO_SYNC_DAYS=1
|
|
||||||
# Secret key to authorize sync operations from server
|
|
||||||
SYNC_SECRET_KEY=your_very_secure_secret_key_here
|
|
||||||
```
|
|
||||||
|
|
||||||
### Değişkenlerin Açıklaması
|
|
||||||
|
|
||||||
- **SYNC_SERVER_ONLY**: `true` olarak ayarlandığında, sync işlemleri sadece sunucudan yapılabilir
|
|
||||||
- **AUTO_SYNC_DAYS**: Otomatik sync sıklığı (gün olarak). `0` = otomatik sync kapalı
|
|
||||||
- **SYNC_SECRET_KEY**: Sunucu sync işlemleri için gerekli gizli anahtar
|
|
||||||
|
|
||||||
## Sunucu Sync İşlemleri
|
|
||||||
|
|
||||||
### 1. Manuel Sync (Sunucudan)
|
|
||||||
|
|
||||||
Sunucuda sync işlemleri yapmak için:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Tüm platformları sync et
|
|
||||||
./scripts/server-sync.sh auto-sync
|
|
||||||
|
|
||||||
# Sadece Windows paketlerini sync et
|
|
||||||
./scripts/server-sync.sh sync-winget
|
|
||||||
|
|
||||||
# Sync durumunu kontrol et
|
|
||||||
./scripts/server-sync.sh status
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2. Cron Job ile Otomatik Sync
|
|
||||||
|
|
||||||
Otomatik sync kurulumu için:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Günlük otomatik sync kur
|
|
||||||
sudo ./scripts/cron-setup.sh install
|
|
||||||
|
|
||||||
# Haftalık otomatik sync kur (her 7 günde bir)
|
|
||||||
sudo AUTO_SYNC_DAYS=7 ./scripts/cron-setup.sh install
|
|
||||||
|
|
||||||
# Cron job durumunu kontrol et
|
|
||||||
sudo ./scripts/cron-setup.sh status
|
|
||||||
|
|
||||||
# Cron job'ı kaldır
|
|
||||||
sudo ./scripts/cron-setup.sh remove
|
|
||||||
```
|
|
||||||
|
|
||||||
## API Güvenliği
|
|
||||||
|
|
||||||
### Sync Endpoint'leri
|
|
||||||
|
|
||||||
Tüm sync endpoint'leri artık güvenlik kontrolü yapar:
|
|
||||||
|
|
||||||
- `/api/sync` - Genel sync endpoint'i
|
|
||||||
- `/api/sync-winget` - Windows paket sync'i
|
|
||||||
- `/api/sync-homebrew` - macOS paket sync'i
|
|
||||||
- `/api/sync-fedora` - Fedora paket sync'i
|
|
||||||
- `/api/sync-arch` - Arch paket sync'i
|
|
||||||
- `/api/auto-sync` - Otomatik sync
|
|
||||||
|
|
||||||
### Güvenlik Kontrolü
|
|
||||||
|
|
||||||
**Server-only modda (`SYNC_SERVER_ONLY=true`):**
|
|
||||||
|
|
||||||
1. **Secret Key Kontrolü**: Request header'ında `x-sync-secret` olmalı
|
|
||||||
2. **IP Kontrolü**: İstek localhost'tan gelmeli veya doğru secret key içermeli
|
|
||||||
|
|
||||||
**Normal modda (`SYNC_SERVER_ONLY=false`):**
|
|
||||||
|
|
||||||
- Herkes sync işlemi yapabilir (geliştirme için)
|
|
||||||
|
|
||||||
### Örnek API Kullanımı
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Server-only modda sync yapmak
|
|
||||||
curl -X POST \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-H "x-sync-secret: your_secret_key" \
|
|
||||||
-d '{}' \
|
|
||||||
http://localhost:3002/api/sync-winget
|
|
||||||
|
|
||||||
# Sync durumunu kontrol et
|
|
||||||
curl -H "x-sync-secret: your_secret_key" \
|
|
||||||
http://localhost:3002/api/sync
|
|
||||||
```
|
|
||||||
|
|
||||||
## Güvenlik İpuçları
|
|
||||||
|
|
||||||
### 1. Secret Key Güvenliği
|
|
||||||
|
|
||||||
- Güçlü ve rastgele bir secret key kullanın
|
|
||||||
- Secret key'i `.env.local` dosyasında saklayın, asla kod içine koymayın
|
|
||||||
- Secret key'i düzenli olarak değiştirin
|
|
||||||
|
|
||||||
### 2. Sunucu Güvenliği
|
|
||||||
|
|
||||||
- Sync script'lerini sadece sunucuda çalıştırın
|
|
||||||
- Cron job'ları root kullanıcısı olarak ayarlayın
|
|
||||||
- Log dosyalarını düzenli olarak kontrol edin
|
|
||||||
|
|
||||||
### 3. Ağ Güvenliği
|
|
||||||
|
|
||||||
- Sync endpoint'lerini firewall ile koruyun
|
|
||||||
- Sadece localhost'tan erişime izin verin
|
|
||||||
- SSL/TLS kullanın (production'da)
|
|
||||||
|
|
||||||
## Monitoring ve Logging
|
|
||||||
|
|
||||||
### Log Dosyaları
|
|
||||||
|
|
||||||
- **Sync Log**: `/var/log/repohub-sync.log`
|
|
||||||
- **Cron Log**: `/var/log/cron.log` (sistem bağımlı)
|
|
||||||
|
|
||||||
### Log İzleme
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Son 20 sync log satırını göster
|
|
||||||
tail -20 /var/log/repohub-sync.log
|
|
||||||
|
|
||||||
# Real-time log izleme
|
|
||||||
tail -f /var/log/repohub-sync.log
|
|
||||||
|
|
||||||
# Cron job loglarını kontrol et
|
|
||||||
sudo tail -20 /var/log/cron.log
|
|
||||||
```
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
### Yaygın Sorunlar
|
|
||||||
|
|
||||||
1. **403 Forbidden Error**
|
|
||||||
- `SYNC_SECRET_KEY` doğru ayarlanmamış
|
|
||||||
- Header'da `x-sync-secret` eksik
|
|
||||||
|
|
||||||
2. **Cron Job Çalışmıyor**
|
|
||||||
- Script executable değil
|
|
||||||
- Environment değişkenleri eksik
|
|
||||||
- Log dosyası izinleri yanlış
|
|
||||||
|
|
||||||
3. **Sync Başarısız**
|
|
||||||
- API URL yanlış
|
|
||||||
- Ağ bağlantısı sorunu
|
|
||||||
- Disk alanı yetersiz
|
|
||||||
|
|
||||||
### Debug Komutları
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Sync script test et
|
|
||||||
./scripts/server-sync.sh test
|
|
||||||
|
|
||||||
# Cron job durumunu kontrol et
|
|
||||||
sudo ./scripts/cron-setup.sh status
|
|
||||||
|
|
||||||
# Environment değişkenlerini kontrol et
|
|
||||||
env | grep SYNC
|
|
||||||
```
|
|
||||||
|
|
||||||
## Production Dağıtımı
|
|
||||||
|
|
||||||
### Adım 1: Environment Ayarı
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# .env.local dosyasını production'a kopyala
|
|
||||||
cp .env.local.example .env.local
|
|
||||||
# .env.local dosyasını production değerleriyle düzenle
|
|
||||||
```
|
|
||||||
|
|
||||||
### Adım 2: Secret Key Oluştur
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Güçlü secret key oluştur
|
|
||||||
openssl rand -hex 32
|
|
||||||
# Bu değeri .env.local dosyasına ekle
|
|
||||||
```
|
|
||||||
|
|
||||||
### Adım 3: Cron Job Kur
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Production cron job kur
|
|
||||||
sudo AUTO_SYNC_DAYS=1 ./scripts/cron-setup.sh install
|
|
||||||
```
|
|
||||||
|
|
||||||
### Adım 4: Test Et
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Sync işlemini test et
|
|
||||||
./scripts/server-sync.sh test
|
|
||||||
|
|
||||||
# Otomatik sync'i test et
|
|
||||||
curl -X POST \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-H "x-sync-secret: your_secret_key" \
|
|
||||||
-d '{}' \
|
|
||||||
http://localhost:3002/api/auto-sync
|
|
||||||
```
|
|
||||||
|
|
||||||
Bu yapılandırma ile sync işlemleriniz güvenli ve otomatik hale gelecektir.
|
|
||||||
+75
-3
@@ -36,7 +36,7 @@ async function main() {
|
|||||||
await admin.end().catch(() => {})
|
await admin.end().catch(() => {})
|
||||||
}
|
}
|
||||||
|
|
||||||
// Connect to the target DB and apply schema
|
// Connect to the target DB and apply schema, then run migrations
|
||||||
const db = new Pool({
|
const db = new Pool({
|
||||||
host: DB_HOST,
|
host: DB_HOST,
|
||||||
port: DB_PORT,
|
port: DB_PORT,
|
||||||
@@ -59,10 +59,82 @@ async function main() {
|
|||||||
} else {
|
} else {
|
||||||
console.error('❌ Schema apply error:', err)
|
console.error('❌ Schema apply error:', err)
|
||||||
process.exitCode = 1
|
process.exitCode = 1
|
||||||
|
// continue to attempt migrations anyway
|
||||||
}
|
}
|
||||||
} finally {
|
}
|
||||||
|
|
||||||
|
// Ensure schema_migrations table exists
|
||||||
|
try {
|
||||||
|
await db.query(`
|
||||||
|
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||||
|
id SERIAL PRIMARY KEY,
|
||||||
|
filename TEXT UNIQUE NOT NULL,
|
||||||
|
applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||||
|
);
|
||||||
|
`)
|
||||||
|
} catch (err) {
|
||||||
|
console.error('❌ Failed to ensure schema_migrations table:', err)
|
||||||
|
process.exitCode = 1
|
||||||
|
}
|
||||||
|
|
||||||
|
// Discover migration files
|
||||||
|
try {
|
||||||
|
const migrationsDir = path.join(__dirname, 'migrations')
|
||||||
|
const migrationFiles = []
|
||||||
|
|
||||||
|
if (fs.existsSync(migrationsDir)) {
|
||||||
|
for (const f of fs.readdirSync(migrationsDir)) {
|
||||||
|
if (f.endsWith('.sql')) {
|
||||||
|
migrationFiles.push({ filename: f, fullpath: path.join(migrationsDir, f) })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Back-compat: also pick up legacy root-level migrations named migrate-*.sql
|
||||||
|
for (const f of fs.readdirSync(__dirname)) {
|
||||||
|
if (f.startsWith('migrate-') && f.endsWith('.sql')) {
|
||||||
|
migrationFiles.push({ filename: f, fullpath: path.join(__dirname, f) })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sort by filename for deterministic order (e.g., 001_*, 002_* ...)
|
||||||
|
migrationFiles.sort((a, b) => a.filename.localeCompare(b.filename))
|
||||||
|
|
||||||
|
if (migrationFiles.length) {
|
||||||
|
console.log(`🚀 Running migrations (${migrationFiles.length} found)...`)
|
||||||
|
} else {
|
||||||
|
console.log('ℹ️ No migrations found')
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const m of migrationFiles) {
|
||||||
|
try {
|
||||||
|
const { rows } = await db.query('SELECT 1 FROM schema_migrations WHERE filename = $1', [m.filename])
|
||||||
|
if (rows.length) {
|
||||||
|
console.log(`↪️ Skipping already applied migration: ${m.filename}`)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
const sql = fs.readFileSync(m.fullpath, 'utf8')
|
||||||
|
console.log(`➡️ Applying migration: ${m.filename}`)
|
||||||
|
await db.query('BEGIN')
|
||||||
|
await db.query(sql)
|
||||||
|
await db.query('INSERT INTO schema_migrations (filename) VALUES ($1)', [m.filename])
|
||||||
|
await db.query('COMMIT')
|
||||||
|
console.log(`✅ Migration applied: ${m.filename}`)
|
||||||
|
} catch (err) {
|
||||||
|
console.error(`❌ Migration failed: ${m.filename}`, err)
|
||||||
|
try { await db.query('ROLLBACK') } catch {}
|
||||||
|
process.exitCode = 1
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch (err) {
|
||||||
|
console.error('❌ Failed during migration discovery/execution:', err)
|
||||||
|
process.exitCode = 1
|
||||||
|
}
|
||||||
|
|
||||||
|
// Done
|
||||||
await db.end().catch(() => {})
|
await db.end().catch(() => {})
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
main().catch((e) => {
|
main().catch((e) => {
|
||||||
|
|||||||
@@ -0,0 +1,5 @@
|
|||||||
|
-- Add 'aur' to repository check constraint
|
||||||
|
ALTER TABLE packages DROP CONSTRAINT IF EXISTS packages_repository_check;
|
||||||
|
ALTER TABLE packages
|
||||||
|
ADD CONSTRAINT packages_repository_check
|
||||||
|
CHECK (repository IN ('official','third-party','aur'));
|
||||||
@@ -11,7 +11,7 @@ export async function GET(request: NextRequest) {
|
|||||||
category_id: searchParams.get('category_id') ?
|
category_id: searchParams.get('category_id') ?
|
||||||
parseInt(searchParams.get('category_id')!) : undefined,
|
parseInt(searchParams.get('category_id')!) : undefined,
|
||||||
type: searchParams.get('type') as 'gui' | 'cli' | undefined,
|
type: searchParams.get('type') as 'gui' | 'cli' | undefined,
|
||||||
repository: searchParams.get('repository') as 'official' | 'third-party' | undefined,
|
repository: searchParams.get('repository') as 'official' | 'third-party' | 'aur' | undefined,
|
||||||
search: searchParams.get('search') || undefined,
|
search: searchParams.get('search') || undefined,
|
||||||
limit: searchParams.get('limit') ?
|
limit: searchParams.get('limit') ?
|
||||||
parseInt(searchParams.get('limit')!) : undefined,
|
parseInt(searchParams.get('limit')!) : undefined,
|
||||||
|
|||||||
@@ -0,0 +1,52 @@
|
|||||||
|
import { NextRequest, NextResponse } from 'next/server'
|
||||||
|
import { SyncAuth } from '@/lib/sync/auth'
|
||||||
|
import { AurPackageFetcher } from '@/services/aurPackageFetcher'
|
||||||
|
|
||||||
|
export const dynamic = 'force-dynamic'
|
||||||
|
export const maxDuration = 300
|
||||||
|
|
||||||
|
let syncInProgress = false
|
||||||
|
let syncStatus: {
|
||||||
|
status: 'idle' | 'running' | 'complete' | 'error'
|
||||||
|
error: string | null
|
||||||
|
} = { status: 'idle', error: null }
|
||||||
|
|
||||||
|
export async function GET() {
|
||||||
|
return NextResponse.json(syncStatus)
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function POST(request: NextRequest) {
|
||||||
|
const authResult = await SyncAuth.isSyncAllowed(request)
|
||||||
|
if (!authResult.allowed) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ error: 'Sync operation not allowed', reason: authResult.reason },
|
||||||
|
{ status: 403 }
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
if (syncInProgress) {
|
||||||
|
return NextResponse.json(
|
||||||
|
{ error: 'Sync already in progress' },
|
||||||
|
{ status: 409 }
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
syncInProgress = true
|
||||||
|
syncStatus = { status: 'running', error: null }
|
||||||
|
|
||||||
|
;(async () => {
|
||||||
|
try {
|
||||||
|
const fetcher = new AurPackageFetcher()
|
||||||
|
const pkgs = await fetcher.fetchAllPackages()
|
||||||
|
await fetcher.storePackages(pkgs)
|
||||||
|
syncStatus.status = 'complete'
|
||||||
|
} catch (err: any) {
|
||||||
|
syncStatus.status = 'error'
|
||||||
|
syncStatus.error = err?.message || 'Unknown error'
|
||||||
|
} finally {
|
||||||
|
syncInProgress = false
|
||||||
|
}
|
||||||
|
})()
|
||||||
|
|
||||||
|
return NextResponse.json({ message: 'AUR sync started', status: syncStatus })
|
||||||
|
}
|
||||||
@@ -31,10 +31,12 @@ export function PackageBrowserV2({
|
|||||||
const [totalCount, setTotalCount] = useState(0)
|
const [totalCount, setTotalCount] = useState(0)
|
||||||
const [searchQuery, setSearchQuery] = useState('')
|
const [searchQuery, setSearchQuery] = useState('')
|
||||||
const [typeFilter, setTypeFilter] = useState<string>('')
|
const [typeFilter, setTypeFilter] = useState<string>('')
|
||||||
|
const [repositoryFilter, setRepositoryFilter] = useState<string>('')
|
||||||
const isDebianUbuntu = useMemo(() => {
|
const isDebianUbuntu = useMemo(() => {
|
||||||
const id = selectedPlatform?.id || ''
|
const id = selectedPlatform?.id || ''
|
||||||
return id === 'debian' || id === 'ubuntu'
|
return id === 'debian' || id === 'ubuntu'
|
||||||
}, [selectedPlatform])
|
}, [selectedPlatform])
|
||||||
|
const isArch = useMemo(() => (selectedPlatform?.id || '') === 'arch', [selectedPlatform])
|
||||||
const [scrollPosition, setScrollPosition] = useState(0)
|
const [scrollPosition, setScrollPosition] = useState(0)
|
||||||
const searchInputRef = useRef<HTMLInputElement>(null)
|
const searchInputRef = useRef<HTMLInputElement>(null)
|
||||||
|
|
||||||
@@ -66,6 +68,9 @@ export function PackageBrowserV2({
|
|||||||
if (isDebianUbuntu && typeFilter && typeFilter !== 'all') {
|
if (isDebianUbuntu && typeFilter && typeFilter !== 'all') {
|
||||||
params.type = typeFilter as 'gui' | 'cli'
|
params.type = typeFilter as 'gui' | 'cli'
|
||||||
}
|
}
|
||||||
|
if (isArch && repositoryFilter && repositoryFilter !== 'all') {
|
||||||
|
params.repository = repositoryFilter as 'official' | 'aur'
|
||||||
|
}
|
||||||
|
|
||||||
console.log('🔍 Frontend: API params:', params)
|
console.log('🔍 Frontend: API params:', params)
|
||||||
const result = await apiClient.getPackages(params)
|
const result = await apiClient.getPackages(params)
|
||||||
@@ -86,7 +91,7 @@ export function PackageBrowserV2({
|
|||||||
}
|
}
|
||||||
|
|
||||||
loadPackages()
|
loadPackages()
|
||||||
}, [selectedPlatform, typeFilter, isDebianUbuntu])
|
}, [selectedPlatform, typeFilter, repositoryFilter, isDebianUbuntu, isArch])
|
||||||
|
|
||||||
// Debounced search to prevent focus loss
|
// Debounced search to prevent focus loss
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
@@ -111,6 +116,9 @@ export function PackageBrowserV2({
|
|||||||
if (isDebianUbuntu && typeFilter && typeFilter !== 'all') {
|
if (isDebianUbuntu && typeFilter && typeFilter !== 'all') {
|
||||||
params.type = typeFilter as 'gui' | 'cli'
|
params.type = typeFilter as 'gui' | 'cli'
|
||||||
}
|
}
|
||||||
|
if (isArch && repositoryFilter && repositoryFilter !== 'all') {
|
||||||
|
params.repository = repositoryFilter as 'official' | 'aur'
|
||||||
|
}
|
||||||
|
|
||||||
console.log('🔍 Frontend: Debounced API params:', params)
|
console.log('🔍 Frontend: Debounced API params:', params)
|
||||||
const result = await apiClient.getPackages(params)
|
const result = await apiClient.getPackages(params)
|
||||||
@@ -130,7 +138,7 @@ export function PackageBrowserV2({
|
|||||||
}, 300) // 300ms debounce
|
}, 300) // 300ms debounce
|
||||||
|
|
||||||
return () => clearTimeout(timeoutId)
|
return () => clearTimeout(timeoutId)
|
||||||
}, [searchQuery, isDebianUbuntu, typeFilter, selectedPlatform])
|
}, [searchQuery, isDebianUbuntu, typeFilter, isArch, repositoryFilter, selectedPlatform])
|
||||||
|
|
||||||
// Load more packages
|
// Load more packages
|
||||||
const loadMore = async () => {
|
const loadMore = async () => {
|
||||||
@@ -151,6 +159,9 @@ export function PackageBrowserV2({
|
|||||||
if (isDebianUbuntu && typeFilter && typeFilter !== 'all') {
|
if (isDebianUbuntu && typeFilter && typeFilter !== 'all') {
|
||||||
params.type = typeFilter as 'gui' | 'cli'
|
params.type = typeFilter as 'gui' | 'cli'
|
||||||
}
|
}
|
||||||
|
if (isArch && repositoryFilter && repositoryFilter !== 'all') {
|
||||||
|
params.repository = repositoryFilter as 'official' | 'aur'
|
||||||
|
}
|
||||||
|
|
||||||
const result = await apiClient.getPackages(params)
|
const result = await apiClient.getPackages(params)
|
||||||
setPackages(prev => [...prev, ...result.packages])
|
setPackages(prev => [...prev, ...result.packages])
|
||||||
@@ -257,6 +268,18 @@ export function PackageBrowserV2({
|
|||||||
</SelectContent>
|
</SelectContent>
|
||||||
</Select>
|
</Select>
|
||||||
)}
|
)}
|
||||||
|
{isArch && (
|
||||||
|
<Select value={repositoryFilter || "all"} onValueChange={(value) => setRepositoryFilter(value === "all" ? "" : value)}>
|
||||||
|
<SelectTrigger className="w-full sm:w-40">
|
||||||
|
<SelectValue placeholder="Repository" />
|
||||||
|
</SelectTrigger>
|
||||||
|
<SelectContent>
|
||||||
|
<SelectItem value="all">All</SelectItem>
|
||||||
|
<SelectItem value="official">Official</SelectItem>
|
||||||
|
<SelectItem value="aur">AUR</SelectItem>
|
||||||
|
</SelectContent>
|
||||||
|
</Select>
|
||||||
|
)}
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{/* Package List */}
|
{/* Package List */}
|
||||||
@@ -311,6 +334,11 @@ export function PackageBrowserV2({
|
|||||||
{pkg.type.toUpperCase()}
|
{pkg.type.toUpperCase()}
|
||||||
</span>
|
</span>
|
||||||
)}
|
)}
|
||||||
|
{pkg.repository === 'aur' && (
|
||||||
|
<span className="text-xs text-muted-foreground">
|
||||||
|
AUR
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
{pkg.repository === 'third-party' && (
|
{pkg.repository === 'third-party' && (
|
||||||
<span className="text-xs text-muted-foreground">
|
<span className="text-xs text-muted-foreground">
|
||||||
third-party
|
third-party
|
||||||
|
|||||||
@@ -37,7 +37,7 @@ CREATE TABLE packages (
|
|||||||
category_id INTEGER REFERENCES categories(id),
|
category_id INTEGER REFERENCES categories(id),
|
||||||
license_id INTEGER REFERENCES licenses(id),
|
license_id INTEGER REFERENCES licenses(id),
|
||||||
type VARCHAR(10) CHECK (type IN ('gui', 'cli')),
|
type VARCHAR(10) CHECK (type IN ('gui', 'cli')),
|
||||||
repository VARCHAR(20) CHECK (repository IN ('official', 'third-party')),
|
repository VARCHAR(20) CHECK (repository IN ('official', 'third-party', 'aur')),
|
||||||
homepage_url VARCHAR(255),
|
homepage_url VARCHAR(255),
|
||||||
download_url VARCHAR(255),
|
download_url VARCHAR(255),
|
||||||
last_updated TIMESTAMP WITH TIME ZONE,
|
last_updated TIMESTAMP WITH TIME ZONE,
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ export interface Package {
|
|||||||
category_id?: number
|
category_id?: number
|
||||||
license_id?: number
|
license_id?: number
|
||||||
type?: 'gui' | 'cli'
|
type?: 'gui' | 'cli'
|
||||||
repository?: 'official' | 'third-party'
|
repository?: 'official' | 'third-party' | 'aur'
|
||||||
homepage_url?: string
|
homepage_url?: string
|
||||||
download_url?: string
|
download_url?: string
|
||||||
last_updated?: Date
|
last_updated?: Date
|
||||||
@@ -53,7 +53,7 @@ export interface CreatePackageInput {
|
|||||||
category_id?: number
|
category_id?: number
|
||||||
license_id?: number
|
license_id?: number
|
||||||
type?: 'gui' | 'cli'
|
type?: 'gui' | 'cli'
|
||||||
repository?: 'official' | 'third-party'
|
repository?: 'official' | 'third-party' | 'aur'
|
||||||
homepage_url?: string
|
homepage_url?: string
|
||||||
download_url?: string
|
download_url?: string
|
||||||
popularity_score?: number
|
popularity_score?: number
|
||||||
@@ -66,7 +66,7 @@ export interface UpdatePackageInput {
|
|||||||
category_id?: number
|
category_id?: number
|
||||||
license_id?: number
|
license_id?: number
|
||||||
type?: 'gui' | 'cli'
|
type?: 'gui' | 'cli'
|
||||||
repository?: 'official' | 'third-party'
|
repository?: 'official' | 'third-party' | 'aur'
|
||||||
homepage_url?: string
|
homepage_url?: string
|
||||||
download_url?: string
|
download_url?: string
|
||||||
popularity_score?: number
|
popularity_score?: number
|
||||||
@@ -77,7 +77,7 @@ export interface PackageFilter {
|
|||||||
platform_id?: string
|
platform_id?: string
|
||||||
category_id?: number
|
category_id?: number
|
||||||
type?: 'gui' | 'cli'
|
type?: 'gui' | 'cli'
|
||||||
repository?: 'official' | 'third-party'
|
repository?: 'official' | 'third-party' | 'aur'
|
||||||
search?: string
|
search?: string
|
||||||
limit?: number
|
limit?: number
|
||||||
offset?: number
|
offset?: number
|
||||||
|
|||||||
@@ -177,7 +177,7 @@ export class ArchPackageFetcher {
|
|||||||
await query(
|
await query(
|
||||||
`INSERT INTO packages (id, name, description, version, platform_id, type, repository, popularity_score, is_active)
|
`INSERT INTO packages (id, name, description, version, platform_id, type, repository, popularity_score, is_active)
|
||||||
VALUES (gen_random_uuid(), $1, $2, $3, $4, $5, $6, $7, $8)`,
|
VALUES (gen_random_uuid(), $1, $2, $3, $4, $5, $6, $7, $8)`,
|
||||||
[pkg.name, pkg.description, pkg.version, 'arch', null, null, 0, true]
|
[pkg.name, pkg.description, pkg.version, 'arch', null, 'official', 0, true]
|
||||||
)
|
)
|
||||||
stored++
|
stored++
|
||||||
} else if (existingResult.rows[0].version !== pkg.version) {
|
} else if (existingResult.rows[0].version !== pkg.version) {
|
||||||
@@ -186,7 +186,7 @@ export class ArchPackageFetcher {
|
|||||||
`UPDATE packages
|
`UPDATE packages
|
||||||
SET version = $1, description = $2, repository = $3, updated_at = NOW()
|
SET version = $1, description = $2, repository = $3, updated_at = NOW()
|
||||||
WHERE id = $4`,
|
WHERE id = $4`,
|
||||||
[pkg.version, pkg.description, null, existingResult.rows[0].id]
|
[pkg.version, pkg.description, 'official', existingResult.rows[0].id]
|
||||||
)
|
)
|
||||||
updated++
|
updated++
|
||||||
} else {
|
} else {
|
||||||
|
|||||||
@@ -0,0 +1,223 @@
|
|||||||
|
interface AurPackage {
|
||||||
|
name: string
|
||||||
|
version: string
|
||||||
|
description: string
|
||||||
|
}
|
||||||
|
|
||||||
|
type AurRpcResult = {
|
||||||
|
ID: number
|
||||||
|
Name: string
|
||||||
|
Version: string
|
||||||
|
Description: string
|
||||||
|
}
|
||||||
|
|
||||||
|
type AurRpcResponse = {
|
||||||
|
version: number
|
||||||
|
type: string
|
||||||
|
resultcount: number
|
||||||
|
results: AurRpcResult[]
|
||||||
|
}
|
||||||
|
|
||||||
|
export class AurPackageFetcher {
|
||||||
|
private rpcBase = 'https://aur.archlinux.org/rpc/?v=5&type=search&by=name&arg='
|
||||||
|
private htmlBase = 'https://aur.archlinux.org/packages'
|
||||||
|
|
||||||
|
private async sleep(ms: number) {
|
||||||
|
return new Promise((r) => setTimeout(r, ms))
|
||||||
|
}
|
||||||
|
|
||||||
|
private async fetchWithUA(url: string) {
|
||||||
|
return fetch(url, {
|
||||||
|
headers: {
|
||||||
|
'user-agent': 'RepoHubBot/0.1 (+https://repohub.local) NodeFetch',
|
||||||
|
'accept': 'text/html,application/json;q=0.9,*/*;q=0.8',
|
||||||
|
'accept-language': 'en-US,en;q=0.9'
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
async fetchAllPackages(onProgress?: (current: number, total: number, sampleName: string) => void): Promise<AurPackage[]> {
|
||||||
|
console.log('🅰️ Starting AUR package fetch via RPC...')
|
||||||
|
|
||||||
|
const queries = 'abcdefghijklmnopqrstuvwxyz0123456789'.split('')
|
||||||
|
const seen = new Set<string>()
|
||||||
|
const out: AurPackage[] = []
|
||||||
|
let totalApprox = 0
|
||||||
|
|
||||||
|
for (let i = 0; i < queries.length; i++) {
|
||||||
|
const q = queries[i]
|
||||||
|
const url = `${this.rpcBase}${encodeURIComponent(q)}`
|
||||||
|
try {
|
||||||
|
const res = await this.fetchWithUA(url)
|
||||||
|
if (!res.ok) {
|
||||||
|
console.warn(`AUR RPC failed for '${q}': ${res.status}`)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
const data = (await res.json()) as AurRpcResponse
|
||||||
|
totalApprox += data.resultcount || 0
|
||||||
|
for (const r of data.results || []) {
|
||||||
|
if (!seen.has(r.Name)) {
|
||||||
|
seen.add(r.Name)
|
||||||
|
out.push({
|
||||||
|
name: r.Name,
|
||||||
|
version: r.Version || 'latest',
|
||||||
|
description: r.Description || `AUR package: ${r.Name}`
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (onProgress) onProgress(out.length, totalApprox || out.length, out[out.length - 1]?.name || q)
|
||||||
|
// polite delay
|
||||||
|
await this.sleep(120)
|
||||||
|
} catch (e) {
|
||||||
|
console.error(`AUR RPC error for '${q}':`, e)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// If RPC yielded nothing (or environment blocks RPC), fallback to HTML scraping
|
||||||
|
if (out.length === 0) {
|
||||||
|
console.log('ℹ️ AUR RPC returned 0 results. Falling back to HTML scraping...')
|
||||||
|
const htmlResults = await this.fetchAllPackagesHtml(onProgress)
|
||||||
|
console.log(`✅ AUR HTML fetch completed! Rows: ${htmlResults.length}`)
|
||||||
|
return htmlResults
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(`✅ AUR RPC fetch completed! Unique packages: ${out.length}`)
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
private async fetchAllPackagesHtml(onProgress?: (current: number, total: number, sampleName: string) => void): Promise<AurPackage[]> {
|
||||||
|
const results: AurPackage[] = []
|
||||||
|
let offset = 0
|
||||||
|
const perPage = 50
|
||||||
|
let total = 0
|
||||||
|
|
||||||
|
// Try first page to detect total
|
||||||
|
const firstUrl = `${this.htmlBase}?O=${offset}&SeB=nd&SB=p`
|
||||||
|
console.log(`📄 AUR HTML: fetching first page ${firstUrl}`)
|
||||||
|
const firstRes = await this.fetchWithUA(firstUrl)
|
||||||
|
if (!firstRes.ok) throw new Error(`AUR HTML HTTP ${firstRes.status}`)
|
||||||
|
const firstHtml = await firstRes.text()
|
||||||
|
total = this.extractTotal(firstHtml) || 0
|
||||||
|
const firstRows = this.parseAurRows(firstHtml)
|
||||||
|
results.push(...firstRows)
|
||||||
|
if (onProgress && firstRows.length) onProgress(results.length, total || results.length, results[results.length - 1].name)
|
||||||
|
|
||||||
|
// Iterate subsequent pages until no rows
|
||||||
|
while (true) {
|
||||||
|
offset += perPage
|
||||||
|
const url = `${this.htmlBase}?O=${offset}&SeB=nd&SB=p`
|
||||||
|
console.log(`📄 AUR HTML: fetching offset ${offset}`)
|
||||||
|
const res = await this.fetchWithUA(url)
|
||||||
|
if (!res.ok) break
|
||||||
|
const html = await res.text()
|
||||||
|
const rows = this.parseAurRows(html)
|
||||||
|
console.log(`🧩 AUR HTML: parsed ${rows.length} rows at offset ${offset}`)
|
||||||
|
if (rows.length === 0) break
|
||||||
|
results.push(...rows)
|
||||||
|
if (onProgress) onProgress(results.length, total || results.length, rows[rows.length - 1].name)
|
||||||
|
await this.sleep(100)
|
||||||
|
}
|
||||||
|
|
||||||
|
return results
|
||||||
|
}
|
||||||
|
|
||||||
|
private extractTotal(html: string): number | null {
|
||||||
|
// Matches: "101725 packages found.\n Page 1 of 2035."
|
||||||
|
const m = html.match(/([0-9][0-9,.]*)\s+packages\s+found\./i)
|
||||||
|
if (m) {
|
||||||
|
const n = parseInt(m[1].replace(/[,\.]/g, ''))
|
||||||
|
return Number.isFinite(n) ? n : null
|
||||||
|
}
|
||||||
|
return null
|
||||||
|
}
|
||||||
|
|
||||||
|
private parseAurRows(html: string): AurPackage[] {
|
||||||
|
const out: AurPackage[] = []
|
||||||
|
const rowBlockRegex = /<tr[^>]*>([\s\S]*?)<\/tr>/gi
|
||||||
|
let rowMatch: RegExpExecArray | null
|
||||||
|
while ((rowMatch = rowBlockRegex.exec(html)) !== null) {
|
||||||
|
const row = rowMatch[1]
|
||||||
|
const nameMatch = row.match(/<a\s+href="\/packages\/[^"]+"[^>]*>\s*([^<]+)\s*<\/a>/i)
|
||||||
|
if (!nameMatch) continue
|
||||||
|
const name = nameMatch[1].trim()
|
||||||
|
// find version as the first <td> after the anchor
|
||||||
|
const afterAnchorIndex = row.indexOf(nameMatch[0]) + nameMatch[0].length
|
||||||
|
const afterAnchor = row.slice(afterAnchorIndex)
|
||||||
|
const versionMatch = afterAnchor.match(/<td[^>]*>\s*([^<]+)\s*<\/td>/i)
|
||||||
|
const version = (versionMatch?.[1] || '').trim()
|
||||||
|
const descMatch = row.match(/<td[^>]*class="[^"]*wrap[^"]*"[^>]*>\s*([\s\S]*?)\s*<\/td>/i)
|
||||||
|
let description = (descMatch?.[1] || '').replace(/<[^>]*>/g, '').trim()
|
||||||
|
if (!description && name) description = `AUR package: ${name}`
|
||||||
|
if (name) {
|
||||||
|
out.push({ name, version: version || 'latest', description })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
async storePackages(packages: AurPackage[], onProgress?: (current: number, total: number) => void): Promise<void> {
|
||||||
|
const { query } = await import('@/lib/database/config')
|
||||||
|
console.log(`💾 Storing ${packages.length} AUR packages in database...`)
|
||||||
|
|
||||||
|
const batchSize = 100
|
||||||
|
let stored = 0
|
||||||
|
let updated = 0
|
||||||
|
let skipped = 0
|
||||||
|
|
||||||
|
// Preflight: ensure DB constraint allows 'aur'
|
||||||
|
try {
|
||||||
|
const check = await query(
|
||||||
|
"SELECT pg_get_constraintdef(oid) AS def FROM pg_constraint WHERE conname = 'packages_repository_check'"
|
||||||
|
)
|
||||||
|
const def = check.rows?.[0]?.def as string | undefined
|
||||||
|
if (!def || !def.toLowerCase().includes("'aur'")) {
|
||||||
|
throw new Error(
|
||||||
|
"Database constraint 'packages_repository_check' does not include 'aur'. Please run scripts/migrate-add-aur-to-repository.sql before syncing AUR."
|
||||||
|
)
|
||||||
|
}
|
||||||
|
} catch (prefErr) {
|
||||||
|
console.error('[AUR] Preflight constraint check failed:', prefErr)
|
||||||
|
throw prefErr
|
||||||
|
}
|
||||||
|
|
||||||
|
for (let i = 0; i < packages.length; i += batchSize) {
|
||||||
|
const batch = packages.slice(i, i + batchSize)
|
||||||
|
for (const pkg of batch) {
|
||||||
|
try {
|
||||||
|
const existing = await query(
|
||||||
|
'SELECT id, version FROM packages WHERE name = $1 AND platform_id = $2',
|
||||||
|
[pkg.name, 'arch']
|
||||||
|
)
|
||||||
|
if (existing.rows.length === 0) {
|
||||||
|
await query(
|
||||||
|
`INSERT INTO packages (id, name, description, version, platform_id, type, repository, popularity_score, is_active)
|
||||||
|
VALUES (gen_random_uuid(), $1, $2, $3, $4, $5, $6, $7, $8)`,
|
||||||
|
[pkg.name, pkg.description, pkg.version, 'arch', null, 'aur', 0, true]
|
||||||
|
)
|
||||||
|
stored++
|
||||||
|
} else if (existing.rows[0].version !== pkg.version) {
|
||||||
|
await query(
|
||||||
|
`UPDATE packages SET version = $1, description = $2, repository = $3, updated_at = NOW() WHERE id = $4`,
|
||||||
|
[pkg.version, pkg.description, 'aur', existing.rows[0].id]
|
||||||
|
)
|
||||||
|
updated++
|
||||||
|
} else {
|
||||||
|
skipped++
|
||||||
|
}
|
||||||
|
if (onProgress && (stored + updated + skipped) % 100 === 0) {
|
||||||
|
onProgress(stored + updated + skipped, packages.length)
|
||||||
|
}
|
||||||
|
} catch (e: any) {
|
||||||
|
if (e?.code === '23514') {
|
||||||
|
console.error('[AUR] Constraint violation detected. Abort to avoid wasting time.')
|
||||||
|
throw new Error(
|
||||||
|
"Insert failed due to 'packages_repository_check'. Run scripts/migrate-add-aur-to-repository.sql to allow 'aur'."
|
||||||
|
)
|
||||||
|
}
|
||||||
|
console.error(`Error storing AUR package ${pkg.name}:`, e)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
console.log(`✅ AUR stored: ${stored} new, ${updated} updated, ${skipped} skipped`)
|
||||||
|
}
|
||||||
|
}
|
||||||
+2
-2
@@ -16,7 +16,7 @@ export interface Package {
|
|||||||
type: 'gui' | 'cli'
|
type: 'gui' | 'cli'
|
||||||
platform?: string | Platform
|
platform?: string | Platform
|
||||||
platform_id?: string
|
platform_id?: string
|
||||||
repository: 'official' | 'third-party'
|
repository: 'official' | 'third-party' | 'aur'
|
||||||
download_url?: string
|
download_url?: string
|
||||||
lastUpdated?: string
|
lastUpdated?: string
|
||||||
downloads?: number
|
downloads?: number
|
||||||
@@ -29,7 +29,7 @@ export interface FilterOptions {
|
|||||||
platform_id?: string
|
platform_id?: string
|
||||||
category_id?: number
|
category_id?: number
|
||||||
type?: 'gui' | 'cli'
|
type?: 'gui' | 'cli'
|
||||||
repository?: 'official' | 'third-party'
|
repository?: 'official' | 'third-party' | 'aur'
|
||||||
search?: string
|
search?: string
|
||||||
limit?: number
|
limit?: number
|
||||||
offset?: number
|
offset?: number
|
||||||
|
|||||||
Reference in New Issue
Block a user