The web cache

Every HTTP and DNS lookup sqx makes is stored in a SQLite file on disk. The first query pays for the network; every query after that reads from the cache.

How it works

All http_*, html_* and dns_* functions, -f URLs, sqx prefetch and sqx import share one cache.
Calling five functions on the same URL downloads it once.
Entries are reused for 7 days (168h) by default, then fetched again.

Where the cache lives

System	Default location
Linux, FreeBSD	`~/.cache/sqx`
macOS	`~/Library/Caches/sqx`
Windows	`%LocalAppData%\sqx`

Use a separate cache for a project or in CI with -cache-db:

cat urls.txt | sqx -cache-db ./project-cache.db -i lines "SELECT url, http_status(url) FROM stdin"

Control freshness

You want to…	Use
Work without a network; a cache miss returns status `0` or `NULL`	`-offline`
Ignore the cache and fetch every URL again	`-force`
Treat entries older than an hour as stale	`-cache-ttl 1h`
Always refetch	`-cache-ttl 0s`
Change the request timeout	`-timeout 30s`
Keep larger response bodies	`-max-body-bytes 52428800`

Inspect the cache

sqx cache stats
sqx cache ls

stats shows URL and DNS counts, a status-code breakdown and the total body size. ls lists every cached URL with its status and timestamp.

The cache is just SQLite, so you can query it. The tables are http_cache and dns_cache:

sqx cache query "SELECT url, response_ms FROM http_cache ORDER BY response_ms DESC LIMIT 10"

Clear the cache

sqx cache clear -older-than 24h
sqx cache clear -all

As a safety measure, clear refuses to run unless you pass -all or -older-than.