Extracting, Retrying, Persisting

use scraper::{Html, Selector};

let doc = Html::parse_document(&body);
let sel = Selector::parse("article h2 a").unwrap();
for el in doc.select(&sel) {
    println!("{} -> {:?}", el.text().collect::<String>(), el.value().attr("href"));
}

Parse the Selector once and reuse it — it compiles a CSS selector and is not free.

Retries. Network calls fail transiently. Retry on timeouts and 5xx with exponential backoff plus jitter; never retry a 4xx, which will fail identically forever.

Persist as you go. A scraper that holds everything in memory and writes at the end loses an hour of work to one panic. Append each result to a JSONL file the moment it arrives.

Be a good citizen. Check /robots.txt, set a real User-Agent with contact info, and keep concurrency low enough that you’d be comfortable explaining it to the site’s operator.