@agntn/archives

Unified TypeScript interface for querying web archive providers. One API, multiple sources, consistent output.

• 🔍 Multiple providers - Wayback Machine, Archive-It, Archive.today, Common Crawl, Perma.cc, WebCite
• 🌳 Tree-shakable - providers are lazy-loaded via dynamic imports, bundle only what you use
• 📦 Caching built in - pluggable storage layer via unstorage with configurable TTL
• ⚡ Parallel queries - concurrency control, batching, automatic retries, configurable timeouts
• 🔧 Config files - supports archives.config.ts, .archives, and package.json via c12
• 🏷️ Fully typed - TypeScript definitions for all responses, options, and provider-specific metadata

pnpm add @agntn/archives

import { createArchive, providers } from "@agntn/archives";

const archive = createArchive(providers.wayback());
const response = await archive.snapshots("example.com", { limit: 100 });

if (response.success) {
  for (const page of response.pages) {
    console.log(page.url, page.timestamp, page.snapshot);
  }
}

Query all providers at once with providers.all() (excludes Archive-It because it needs a collection ID and Perma.cc because it needs an API key):

const archive = createArchive(providers.all());
const response = await archive.snapshots("example.com");

To pick specific providers, wrap them in Promise.all:

const archive = createArchive(
  Promise.all([providers.wayback(), providers.archiveToday(), providers.commoncrawl()]),
);

Perma.cc requires an API key and searches archives accessible to that account by exact submitted URL:

const archive = createArchive(providers.permacc({ apiKey: "YOUR_API_KEY" }));
const response = await archive.snapshots("https://example.com/page");

A bare domain such as example.com is normalized to https://example.com/. It does not match every path on that domain.

Archive-It queries one public collection at a time through its CDX/C API. Pass the numeric collection ID when creating the provider:

const archive = createArchive(providers.archiveIt({ collection: 4399 }));
const response = await archive.snapshots("archive-it.org");

Archive-It’s all-collections endpoint is temporarily blocked, so providers.archiveIt() requires a collection and is not included in providers.all().

snapshots() returns a response object with a success flag. If you prefer throwing on failure, use getPages():

// safe - check success flag yourself
const response = await archive.snapshots("example.com");

// throws on failure, returns pages array directly
const pages = await archive.getPages("example.com");

getPages() distinguishes runtime failures from structural ones. When every queried provider is unsupported for the operation (see below), it throws UnsupportedOperationError with the per-provider reasons attached:

import { UnsupportedOperationError } from "@agntn/archives";

try {
  const pages = await archive.getPages("example.com");
} catch (error) {
  if (error instanceof UnsupportedOperationError) {
    // error.providers: [{ provider, reason }, ...]
  } else {
    // generic Error: network failure, parse error, etc.
  }
}

You can add providers dynamically after creation:

const archive = createArchive(providers.wayback());
await archive.use(providers.archiveToday());
await archive.useAll([providers.commoncrawl(), providers.webcite()]);

@agntn/archives ships native extensions for OMP and Pi. Install the package directly from GitHub with the matching host:

omp install github:agntn/archives
pi install git:github.com/agntn/archives

Tools:

• archives — query archived snapshots for a domain or URL. Use provider="all" for broad coverage or provider="wayback" for a fast Wayback-only lookup.
• archives_providers — list built-in archive providers and Perma.cc API-key environment status.

Commands:

• /archive [domain-or-url] — search Wayback snapshots interactively and paste the selected snapshot URL into the editor.
• /archive-providers — show provider availability notes.

Every provider normalizes its output to the same shape:

interface ArchiveResponse {
  success: boolean;
  pages: ArchivedPage[];
  error?: string;
  unsupported?: boolean; // provider does not implement this operation
  unsupportedReason?: string;
  _meta?: ResponseMetadata;
  fromCache?: boolean;
}

interface ArchivedPage {
  url: string; // original URL
  timestamp: string; // ISO 8601
  snapshot: string; // direct link to the archived version
  _meta: Record<string, unknown>;
}

The _meta object on each page carries provider-specific fields. Wayback includes status and timestamp in its raw format. Common Crawl adds digest, mime, collection. Perma.cc has guid, title, created_by. Archive.today provides hash and raw_date.

Not every provider implements every operation. WebCite, for example, exposes no list-by-domain API — it only resolves snapshots by ID. When a provider cannot answer a call, it returns success: false with unsupported: true and a human-readable unsupportedReason, instead of fabricating data.

For multi-provider calls, the combined response surfaces unsupported providers under _meta.unsupportedProviders regardless of how the rest behaved. The top-level unsupported flag has stricter semantics:

Example:

const archive = createArchive(providers.all());
const response = await archive.snapshots("example.com");

response.pages; // results from Wayback, Archive.today, Common Crawl
response._meta?.unsupportedProviders;
// [{ provider: "webcite", reason: "WebCite has no list-by-domain API. ..." }]

To treat unsupported providers as a whole-call failure, check the top-level flag explicitly: if (!response.success && response.unsupported) { ... }.

Archives loads configuration through c12, which means you can configure it via config files, environment overrides, or package.json:

// archives.config.ts
export default {
  storage: {
    cache: true,
    ttl: 7 * 24 * 60 * 60 * 1000, // 7 days
    prefix: "archives",
  },
  performance: {
    concurrency: 3,
    batchSize: 20,
    timeout: 10_000,
    retries: 1,
  },
};

Environment-specific overrides work with $development, $production, and $test keys.

The caching layer is backed by unstorage, so any unstorage driver works:

import { configureStorage } from "@agntn/archives";
import fsDriver from "unstorage/drivers/fs";

await configureStorage({
  driver: fsDriver({ base: "./cache" }),
  ttl: 24 * 60 * 60 * 1000, // 1 day
});

Per-request cache control is also supported:

// skip cache for this request
await archive.snapshots("example.com", { cache: false });

Creates an archive client. Accepts a single provider, a Promise<ArchiveProvider>, or a Promise<ArchiveProvider[]>.

Returns:

• snapshots(domain, options?) - returns full ArchiveResponse with success flag
• getPages(domain, options?) - returns ArchivedPage[], throws on failure
• use(provider) - add a provider to the instance
• useAll(providers) - add multiple providers at once

All methods accept ArchiveOptions:

Options can be set at three levels: config file (global defaults), createArchive call (instance defaults), and individual method calls (per-request). Each level overrides the previous one.

• configureStorage(options?) - configure the cache driver and settings
• clearProviderStorage(provider) - clear cached responses for a specific provider
• storage - direct access to the underlying unstorage instance

Providers: Conifer (formerly Webrecorder)

Features: Page archiving API for creating archives, not just reading them

MIT