Keeper Network

How the Keeper Network Monitors Vaults#

Recovery on AeternumVault is not driven by a privileged operator watching the chain. It's driven by whoever calls triggerRecovery() on a wallet once that wallet's inactivity period has elapsed — and the contract places no restriction at all on who that can be.

In practice, today that's almost always the Aeternum Labs keeper bot, running as a public good (Phase 1 of the rollout, alongside the AeternumVault contract itself). This page is the deep dive into how it actually works: how it finds wallets that are due, how it double-checks that they're really due before spending any gas, and how it submits recovery transactions on-chain.

Three functions on the contract make this possible, each with a different role:

That last point is worth sitting with: the first two functions above are conveniences, not gates. Nothing on the contract requires anyone to call them before calling triggerRecovery(). A keeper that skipped validation entirely and just fired triggerRecovery() at every registered wallet on every cycle would still be perfectly safe — it would just be needlessly expensive, since most of those calls would hit the "not actually due yet" precondition and return without doing anything. The validation layer described below exists purely so the Aeternum Labs bot doesn't waste gas on calls it already knows will be no-ops.

The diagram above shows the two things that matter about this pipeline. First, it's a filter, not a scan: getDueVaults() asks the database for whatever's due right now, every cycle — there's no position to track, no "already scanned" portion of the registry, nothing to advance. Second, the ~12 second gap sits between cycles, not between steps within one — scanning, validating, and executing all happen back-to-back as fast as the database query and the on-chain calls allow, and the wait only starts once that's done.

The Keeper Cycle: Scan, Validate, Execute#

The Aeternum Labs keeper bot runs a continuous loop: scan, validate, execute, sleep, repeat. Each stage has a distinct cost profile.

// The main loop, simplified
while (running) {
  const dueWallets = await scan(db, publicClient, contractAddress, KEEPER_BATCH_SIZE);
 
  if (dueWallets.length > 0) {
    await execute(walletClient, publicClient, contractAddress, dueWallets);
  }
 
  await sleep(KEEPER_POLL_INTERVAL_MS);
}

One detail worth flagging: KEEPER_POLL_INTERVAL_MS is the gap after a cycle finishes, not a fixed clock tick. If a cycle has a lot of recoveries to execute, the actual time between scans stretches past 12 seconds. Under normal load this is imperceptible; it just means "roughly every 12 seconds" is a floor, not a guarantee.

Stage 1 — Scan (off-chain, free)#

The bot never queries the contract to find candidates. It queries its own database — populated in real time from on-chain events by a Ponder-based indexer, written to a Postgres database (Neon), and read via Drizzle:

// Layer 1: DB pre-filter
const rows = await getDueVaults(db, dbScanLimit); // dbScanLimit = KEEPER_BATCH_SIZE
const candidates = rows.map((r) => r.id as Address);

This costs nothing on-chain — it's a plain database query. It's also allowed to be slightly stale: the indexer can lag a few blocks behind the chain head, and that's fine, because nothing gets executed on the strength of this query alone.

Stage 2 — Validate (on-chain, batched)#

Every DB candidate gets re-checked directly against the contract before anything is submitted, to catch anything the database missed — a ping sent moments ago, for instance, that the indexer hasn't caught up to yet. This is a real on-chain read, but a cheap one, and all candidates are checked in a single eth_call via multicall rather than one RPC round-trip per wallet:

// Layer 2: on-chain validation
const results = await publicClient.multicall({
  contracts: candidates.map((wallet) => ({
    address: contractAddress,
    abi: AETERNUM_VAULT_ABI,
    functionName: "isRecoveryDue" as const,
    args: [wallet] as const,
  })),
  allowFailure: true,
});
 
const confirmed = candidates.filter((_, i) => results[i]?.status === "success" && results[i].result === true);

isRecoveryDue() is what gets called here — not getTriggerableVaultsBatch(). The two functions return conceptually similar information, but isRecoveryDue() is a single boolean check per wallet, which is exactly what's needed to validate a specific list of DB candidates; getTriggerableVaultsBatch() is built for a different job — scanning an index range of the registry without needing a pre-existing candidate list, useful for a keeper that has no indexer of its own.

This scanner is explicitly designed as a filter, not a source of truth. As its own source comments put it: it's "an efficiency layer, not a security boundary" — if the database or this validation pass ever drifts from actual on-chain state, _executeRecovery()'s own precondition checks are still the final word on whether any ETH moves. Stage 2 exists to save gas, not to provide safety; the safety comes from the contract regardless of what any keeper does or doesn't validate beforehand.

Stage 3 — Execute (on-chain, batched via Multicall3)#

Confirmed wallets aren't submitted as one triggerRecovery() transaction each. They're batched into as few transactions as possible via Multicall3's aggregate3 function, with allowFailure: true so one wallet's failed ETH transfer doesn't affect any other wallet bundled into the same transaction:

// Batched submission
const batches = chunk(wallets, MAX_CALLS_PER_TX); // MAX_CALLS_PER_TX = 120
 
for (const batch of batches) {
  const calls = batch.map((wallet) => ({
    target: contractAddress,
    allowFailure: true,
    callData: encodeFunctionData({
      abi: AETERNUM_VAULT_ABI,
      functionName: "triggerRecovery",
      args: [wallet],
    }),
  }));
 
  const estimatedGas = await publicClient.estimateContractGas({
    address: MULTICALL3_ADDRESS,
    abi: MULTICALL3_ABI,
    functionName: "aggregate3",
    args: [calls],
    account: walletClient.account,
  });
 
  const bufferedGas = (estimatedGas * 130n) / 100n; // 30% buffer
  const gasWithBuffer = bufferedGas > MAX_GAS_PER_TX ? MAX_GAS_PER_TX : bufferedGas;
 
  const hash = await walletClient.writeContract({
    address: MULTICALL3_ADDRESS,
    abi: MULTICALL3_ABI,
    functionName: "aggregate3",
    args: [calls],
    gas: gasWithBuffer,
  });
 
  await publicClient.waitForTransactionReceipt({ hash }); // awaited before the next batch
}

A few things worth understanding about this stage:

Why 30% gas buffer, not 0% or 100%. Testing surfaced a specific failure mode: eth_estimateGas's binary search doesn't always precisely account for EIP-150's 63/64ths gas-forwarding rule across the nested call chain here — Multicall3 calling into AeternumVault calling into a backup address's receive(). Underestimating caused individual sub-calls to run out of gas and revert on-chain even though the outer transaction succeeded. The 30% figure is an empirical margin against that, not a formal guarantee, and would need retuning if the contract's gas profile ever shifts materially.

Why a hard ceiling on top of the buffer. MAX_GAS_PER_TX = 20,000,000 is applied after buffering, as defense in depth against an anomalous estimate — an RPC glitch, or a backup address with a deliberately expensive receive() function — being submitted unchecked.

Why batches are submitted sequentially, not in parallel. Each batch awaits its transaction receipt before the next one is submitted. This keeps nonce ordering simple and avoids replacement-transaction edge cases, at the cost of some throughput — a cycle with 350 confirmed wallets takes three sequential transactions (120 + 120 + 110), one after another, rather than three in parallel.

What happens on failure. A failed batch — submission error, dropped transaction, gas estimation failure — is logged and does not abort any other batch in the same cycle. RecoveryExecuted, RecoveryFailed, and RecoveryAbandoned events are parsed out of each successful batch's receipt and logged per-wallet, so the outcome of every individual recovery is observable regardless of how many wallets were bundled together.

Two Separate Size Limits#

Just as the DB scan and the on-chain execution have different cost profiles, they're governed by two independently tunable limits:

These are deliberately decoupled. Raising KEEPER_BATCH_SIZE to pull more candidates per scan never silently grows the size of a single transaction — it just means more sequential transactions get submitted if more wallets turn out to be confirmed due.

MAX_CALLS_PER_TX's value of 120 isn't arbitrary — it's derived from measured gas cost: 120 calls at triggerRecovery's measured worst-case (90,988 gas) plus the 30% estimation buffer works out to roughly 14.2M gas, about 24% of a 60M block gas limit — comfortable headroom even under network congestion, while still keeping each transaction well clear of any single-transaction gas ceiling.

Stale Data Safety#

There's an unavoidable gap between when a wallet is confirmed due in Stage 2 and when its triggerRecovery() transaction actually confirms on-chain in Stage 3 — typically seconds, occasionally longer under load. A user could ping their vault, withdraw, or cancel recovery entirely in that window.

This is handled the same way regardless of how confident the keeper's own validation was: every precondition is checked again, from scratch, inside _executeRecovery(), independent of whatever motivated the call.

function _executeRecovery(address wallet) internal {
    RecoveryConfig storage config = s_configs[wallet];
 
    // Re-validate — stale entries are silently skipped
    if (!config.isActive) return;
    if (config.balance == 0) return;
    if (block.timestamp < config.lastActivity + config.inactivityPeriod) return;
 
    // ... proceed with recovery
}

If a wallet no longer qualifies, the call returns without moving any funds — no revert, no error, just a cheap no-op (the ~8,300 gas precondition-check floor rather than a full recovery execution). Because allowFailure: true isn't even needed to catch this case — the function doesn't revert in the first place — a stale entry inside a batch doesn't affect any other wallet in that same transaction.

Race Safety — Competing Keepers#

Because triggerRecovery() is permissionless, more than one caller can legitimately try to recover the same wallet — the Aeternum Labs bot and Gelato racing each other once Phase 2 is live, for instance, or all three keepers racing once Chainlink CRE joins in Phase 3. This resolves safely, and neither keeper needs to know the other exists, let alone communicate with it.

The first call to actually land executes recovery normally. Any subsequent call — whether it arrives in a separate transaction or happens to be bundled in the same Multicall3 batch — finds isActive == false and balance == 0, and silently returns without reverting or double-transferring funds.

This holds regardless of how the competing calls are batched, for a simple reason: the EVM executes every call within a transaction strictly sequentially. There's no concurrent storage access for this safety property to race against — each triggerRecovery() call fully completes its read-modify-write on the wallet's config and on the registry before the next call, whether internal to a Multicall3 batch or in the next block, ever begins.

Configuration Reference#

The keeper bot exposes a /health endpoint (port 3001, or PORT if set by the hosting platform) for uptime monitoring, and shuts down gracefully — finishing any in-progress cycle on SIGTERM before exiting, so a deploy or restart never interrupts a transaction mid-flight.

Running Your Own Keeper#

Because triggerRecovery() has no access control, nothing about the protocol requires using the Aeternum Labs bot specifically. Anyone can run their own keeper against a live AeternumVault deployment. At minimum, a keeper needs to:

An independent keeper running today would be operating alongside the Aeternum Labs bot rather than instead of it — both are just permissionless callers of the same function, and a wallet recovered by one is simply unavailable to the other by the time its call lands, see Race Safety above.

Scaling the Keeper Network#

Two different questions about scaling come up, and they have two different answers.

More throughput from a single keeper — horizontal signers#

Nothing about this architecture bottlenecks on discovery or validation. Discovery (Stage 1) is a database query — scaling a Postgres query over a larger table as the registry grows is a well-understood, off-the-shelf problem. Validation (Stage 2) is cheap enough (under 10,000 gas per wallet) to batch hundreds of candidates into a single eth_call at effectively zero real cost, regardless of how many candidates there are.

The actual constraint is on the execution side, and it isn't MAX_CALLS_PER_TX. 120 calls per transaction is a deliberately conservative ceiling with headroom to spare — it's not the number that limits how many recoveries the keeper can process per cycle. The real limit is that the keeper bot signs and submits every batch from a single private key, and batches from a single signer have to be submitted sequentially: each one waits for its transaction receipt before the next is sent, to keep nonce ordering simple. One signer means one batch in flight at a time, no matter how many wallets are confirmed due in a given cycle.

The path to scaling past that is horizontal, not vertical: run the keeper with multiple signing keys instead of one. Each key manages its own nonce sequence independently, so multiple batches can be in flight simultaneously — batch N from key A and batch N+1 from key B submitting in parallel rather than waiting on each other. Throughput scales roughly linearly with the number of active signers, without touching MAX_CALLS_PER_TX, without any change to the contract, and without any change to triggerRecovery() itself.

This works cleanly because of the same property that makes competing keepers safe in the first place, see Race Safety above. Every signer is just another independent, permissionless caller of triggerRecovery(). Multiple keys submitting concurrently is mechanically no different from multiple independent third-party keepers submitting concurrently — the contract doesn't distinguish between them, and doesn't need to. Adding signers is purely an operational change to the keeper bot's submission layer.

Multi-signer submission isn't built yet — this is the direction, not a shipped feature or a benchmarked result. The single-signer design comfortably covers Phase 1 volume; horizontal scaling is the planned response once real throughput data shows it's warranted, not a problem being solved preemptively.

More keepers over time — the phase rollout#

Separately from any single keeper's own throughput, resilience grows by adding entirely independent keepers over time, in step with the product roadmap. Phase 2 (Hybrid Account) brings Gelato online as a second, independently-operated keeper alongside the Aeternum Labs bot. Phase 3 (Multichain — independent deployments across Base, Arbitrum One, Optimism, BNB Chain, Polygon, and zkSync) adds Chainlink CRE as a multi-chain orchestrator: a single off-chain service running one independent workflow per chain, each scanning and submitting triggerRecovery transactions to that chain's own deployment — with no messaging or dependency of any kind between chains. It runs alongside the Aeternum Labs bot and Gelato on every chain, exactly as both already do today on Ethereum mainnet.

None of this requires any change to triggerRecovery(). Every keeper added at every phase is just another permissionless caller of the same function described throughout this page, not a new privileged role — the contract doesn't know or care whether a given call came from the Aeternum Labs bot, Gelato, Chainlink CRE, or an individual, and it doesn't need to.