Scrapyard
A self-hosted aggregator for refurbished laptops. Scrapes multiple Swedish second-hand stores, enriches CPU specs from Intel ARK, and serves a filterable web UI.
Requirements
- Python 3.11+
- Firefox (used by Playwright for JavaScript-heavy pages)
Installation
# 1. Clone the repo
git clone <repo-url>
cd scrapyard
# 2. Create and activate a virtual environment
python -m venv .venv
# Windows
.venv\Scripts\activate
# macOS / Linux
source .venv/bin/activate
# 3. Install dependencies
pip install -r requirements.txt
# 4. Install the Playwright Firefox browser
playwright install firefox
# 5. Start the app
python main.py
Open http://localhost:5000 in your browser.
The database (database.db) is created automatically on first run. It ships with CPU spec data pre-populated — no scraping needed to browse those.
Scraping data
Use the Scraping panel at the bottom of the sidebar in the UI, or trigger scrapers directly:
| Button | What it does |
|---|---|
| Scrape BilligTeknik | Crawls BilligTeknik listing and detail pages |
| Scrape Nuvoo | Crawls Nuvoo via the Shopify JSON API |
| Enrich CPUs | Fetches detailed CPU specs from Intel ARK |
Scrapers run in the background — the UI stays responsive while they work.
Updating
# Pull latest changes
git pull
# Install any new dependencies
pip install -r requirements.txt
# Restart the app
python main.py
Pre-push database wipe
The database is not committed to the repo (see .gitignore). Before pushing, run the purge script to clear all scraped listings while keeping CPU data intact:
python purge_db.py
It will show a summary and ask for confirmation before deleting anything.