# Scrapyard

A self-hosted aggregator for refurbished laptops. Scrapes multiple Swedish second-hand stores, enriches CPU specs from Intel ARK, and serves a filterable web UI.

---

## Requirements

- Python 3.11+
- Firefox (used by Playwright for JavaScript-heavy pages)

---

## Installation

```bash
# 1. Clone the repo
git clone <repo-url>
cd scrapyard

# 2. Create and activate a virtual environment
python -m venv .venv

# Windows
.venv\Scripts\activate

# macOS / Linux
source .venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Install the Playwright Firefox browser
playwright install firefox

# 5. Start the app
python main.py
```

Open **http://localhost:5000** in your browser.

The database (`database.db`) is created automatically on first run. It ships with CPU spec data pre-populated — no scraping needed to browse those.

---

## Scraping data

Use the **Scraping** panel at the bottom of the sidebar in the UI, or trigger scrapers directly:

| Button | What it does |
|---|---|
| Scrape BilligTeknik | Crawls BilligTeknik listing and detail pages |
| Scrape Nuvoo | Crawls Nuvoo via the Shopify JSON API |
| Enrich CPUs | Fetches detailed CPU specs from Intel ARK |

Scrapers run in the background — the UI stays responsive while they work.

---

## Updating

```bash
# Pull latest changes
git pull

# Install any new dependencies
pip install -r requirements.txt

# Restart the app
python main.py
```

---

## Pre-push database wipe

The database is not committed to the repo (see `.gitignore`). Before pushing, run the purge script to clear all scraped listings while keeping CPU data intact:

```bash
python purge_db.py
```

It will show a summary and ask for confirmation before deleting anything.