Scraper guide

This project includes a Python scraper that collects officer data from stfc.space with Playwright. Follow this guide to run it on your machine and build your own dataset.

Prerequisites

uv (https://docs.astral.sh/uv/), which supplies Python 3.12
Node.js and pnpm (through mise) for the website
A terminal

Installation

# Clone the repository
git clone https://gitlab.com/nifty-little-tools/web-static/stfc-space-scraper.git
cd stfc-space-scraper/scripts/stfc_scraper

# Install the scraper and its dependencies (uv creates the environment)
uv sync

# Download the browser Playwright drives
uv run playwright install chromium

Running the scraper

Run it from the scripts folder: the scraper writes the officer images to officer/ under the folder it runs in, and the website looks for them, and for officers.csv, there.

cd ..
uv run --project stfc_scraper stfc-scraper --output officers.csv

Options

Flag	Description	Default
--output, -o	Output CSV path	officers.csv
--headless / --no-headless	Run without / with a visible browser window	Headless
--delay, -d	Delay between requests, in seconds	1.0
--page-start	First list page to scrape	1
--page-end	Last list page to scrape	14
--limit	Stop after this many officers	No limit
--lang	Language of the messages (en, es)	en

Updating the website

Convert the CSV and the images to what the website reads, then commit the result:

cd ../website
pnpm data
pnpm build

The scraper requests the public officer pages of stfc.space from your machine. Check the site's terms before scraping, and keep the delay between requests.