Concurrency and Rate Limiting

Sequential awaits are just slow synchronous code with extra syntax. Real concurrency comes from having many futures in flight:

use futures::stream::{self, StreamExt};

let bodies = stream::iter(urls)
    .map(|url| async move { reqwest::get(&url).await?.text().await })
    .buffer_unordered(8)      // at most 8 in-flight
    .collect::<Vec<_>>()
    .await;

buffer_unordered(8) is the whole rate limiter: it keeps eight requests running and starts a new one as each finishes. Compare with join_all, which starts everything at once — fine for ten URLs, hostile for ten thousand.

For a global limit across several tasks, use tokio::sync::Semaphore and hold the permit for the duration of the request.

Reuse one reqwest::Client across all requests. Building a new one per request throws away the connection pool and TLS session cache.