84 lines
1.7 KiB
Markdown
84 lines
1.7 KiB
Markdown
# Scrapyard
|
|
|
|
A self-hosted aggregator for refurbished laptops. Scrapes multiple Swedish second-hand stores, enriches CPU specs from Intel ARK, and serves a filterable web UI.
|
|
|
|
---
|
|
|
|
## Requirements
|
|
|
|
- Python 3.11+
|
|
- Firefox (used by Playwright for JavaScript-heavy pages)
|
|
|
|
---
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
# 1. Clone the repo
|
|
git clone <repo-url>
|
|
cd scrapyard
|
|
|
|
# 2. Create and activate a virtual environment
|
|
python -m venv .venv
|
|
|
|
# Windows
|
|
.venv\Scripts\activate
|
|
|
|
# macOS / Linux
|
|
source .venv/bin/activate
|
|
|
|
# 3. Install dependencies
|
|
pip install -r requirements.txt
|
|
|
|
# 4. Install the Playwright Firefox browser
|
|
playwright install firefox
|
|
|
|
# 5. Start the app
|
|
python main.py
|
|
```
|
|
|
|
Open **http://localhost:5000** in your browser.
|
|
|
|
The database (`database.db`) is created automatically on first run. It ships with CPU spec data pre-populated — no scraping needed to browse those.
|
|
|
|
---
|
|
|
|
## Scraping data
|
|
|
|
Use the **Scraping** panel at the bottom of the sidebar in the UI, or trigger scrapers directly:
|
|
|
|
| Button | What it does |
|
|
|---|---|
|
|
| Scrape BilligTeknik | Crawls BilligTeknik listing and detail pages |
|
|
| Scrape Nuvoo | Crawls Nuvoo via the Shopify JSON API |
|
|
| Enrich CPUs | Fetches detailed CPU specs from Intel ARK |
|
|
|
|
Scrapers run in the background — the UI stays responsive while they work.
|
|
|
|
---
|
|
|
|
## Updating
|
|
|
|
```bash
|
|
# Pull latest changes
|
|
git pull
|
|
|
|
# Install any new dependencies
|
|
pip install -r requirements.txt
|
|
|
|
# Restart the app
|
|
python main.py
|
|
```
|
|
|
|
---
|
|
|
|
## Pre-push database wipe
|
|
|
|
The database is not committed to the repo (see `.gitignore`). Before pushing, run the purge script to clear all scraped listings while keeping CPU data intact:
|
|
|
|
```bash
|
|
python purge_db.py
|
|
```
|
|
|
|
It will show a summary and ask for confirmation before deleting anything.
|