De‐duplication

rustmailer edited this page 2026-05-21 09:09:14 +08:00

Table of Contents

• BLAKE3 content hashing
• Fjall blob store: immediate key-level dedup
• Tantivy index: periodic dedup task
• Grouping key
• Retention policy: keep the newest ingest_at
• Cascading attachment cleanup
• UIDVALIDITY-triggered mailbox rebuild
• How it works
• Reference-counted blob cleanup
• Edge cases
• Email moved between folders
• UIDVALIDITY reset
• Same email under different accounts
• EML import / SMTP ingestion
• Summary
• Dedup does not mean "store once"

Bichon handles deduplication through several independent mechanisms, each operating at a different point in the pipeline. Together they ensure that no matter how many times the same email is downloaded, moved across folders, or imported, only one canonical copy persists at the storage level.

BLAKE3 content hashing

Every deduplication decision starts with compute_content_hash() at crates/core/src/utils/mod.rs:350:

pub fn compute_content_hash(content: &[u8]) -> String {
    let hash = blake3::hash(content);
    hash.to_hex().to_string()
}

• Uses BLAKE3-256, producing a 64-character hex string.
• Computed over the full raw bytes of the email body for the message-level fingerprint.
• Computed over each attachment's raw bytes separately for per-attachment fingerprints.
• The hash is a stable identity for the content — it does not depend on IMAP UIDs or folder location.

Fjall blob store: immediate key-level dedup

Location: crates/core/src/store/blob.rs:58-87

This is the first line of defense — it fires immediately on every write to persistent blob storage.

flowchart LR
    A[New email / attachment] --> B[Compute BLAKE3 hash]
    B --> C{Fjall Keyspace<br/>contains_key?}
    C -->|No| D[Insert blob]
    C -->|Yes| E[Silently skip]

• BLOB_MANAGER manages two Fjall keyspaces: "email" and "attachments".
• Before every insert, it calls contains_key() on the content hash.
• If the hash already exists the write is a no-op — no overwrite, no error.
• This is fully automatic and requires no background task.

Net effect: if the same email is downloaded twice (e.g. it appears in two folders), Fjall stores only one copy of the raw bytes. Both index entries point to the same content hash key.

Tantivy index: periodic dedup task

Location: crates/core/src/store/tantivy/dedup.rs (733 lines)

A background task that runs every 12 hours, scanning the full-text index and removing duplicate documents.

Grouping key

Duplicates are identified by (mailbox_id, content_hash):

Mailbox A, hash 0xabc... → keep at most 1 copy
Mailbox B, hash 0xabc... → allowed (different mailbox)

Key design decisions:

• Only documents with the same content hash inside the same mailbox are considered duplicates.
• Cross-mailbox duplicates are preserved — a user may intentionally archive the same email into multiple folders.
• Documents belonging to different accounts are never deduplicated against each other (the task processes accounts one at a time).

Retention policy: keep the newest ingest_at

When duplicates are found, the copy with the highest ingest_at timestamp is kept; the rest are deleted.

The rationale is tied to UIDVALIDITY resets: after a server reassigns UIDs, the stale copy carries an outdated UID. If the stale copy were kept, UID-based incremental sync would see the new UIDs as missing and re-download emails already present. Keeping the most recently ingested copy (which bears the new UID) prevents this.

Cascading attachment cleanup

When a duplicate email is removed, its attachments are cleaned up from the attachment index:

1. Delete the duplicate email doc from the email index by f_id term
2. Delete attachment docs from the attachment index by f_envelope_id term (matching the removed email's f_id)
3. Commit both indexes per account

UIDVALIDITY-triggered mailbox rebuild

Location: crates/core/src/cache/imap/download/flow.rs:424-531

This is not deduplication in the traditional sense — it prevents data inconsistency by detecting stale data and purging it.

How it works

On every sync cycle, reconcile_mailboxes() compares the local and remote uid_validity for each mailbox:

1. UIDVALIDITY unchanged → perform incremental sync (fetch only UIDs > local max UID).
2. UIDVALIDITY changed → the mailbox data is considered invalid. The system:
   • Calls rebuild_mailbox_cache() or rebuild_mailbox_cache_by_date().
   • Deletes all existing envelope documents for the affected mailbox.
   • Re-downloads the entire mailbox from the server.

Reference-counted blob cleanup

Location: crates/core/src/store/tantivy/envelope.rs:775-851

When envelopes are deleted (whether by user action or UIDVALIDITY rebuild), the corresponding blobs are not deleted outright. Instead, the system checks whether any remaining document still references each content hash:

Delete envelopes
  └→ collect_content_hashes()   // gather every content_hash from the docs being deleted
       └→ cleanup_unused_content()
            ├→ For each email content hash:    Count query → refs == 0? → delete from Fjall
            └→ For each attachment content hash: Count query → refs == 0? → delete from Fjall

• Uses Tantivy's Count query to check whether each hash is still referenced by any surviving document.
• A blob is removed from Fjall only when no document references it.
• This guarantees that a blob shared by multiple index entries is not deleted when one of those entries is removed.

Edge cases

Email moved between folders

When an email is moved from Inbox to Archive:

1. The IMAP server presents the same message (same Message-ID / content) in both folders.
2. Bichon downloads it twice → two index documents in Tantivy (different mailbox_id).
3. Fjall stores only one copy of the blob (immediate key-level dedup).
4. The periodic dedup task does not merge them (different mailbox_id).
5. Both index documents point to the same blob → reference-counted cleanup protects the blob when either one is deleted.

UIDVALIDITY reset

1. The IMAP server changes UIDVALIDITY.
2. reconcile_mailboxes() detects the change → deletes stale data → performs a full re-download.
3. If detection fails (e.g. server bug), the periodic dedup task catches the resulting duplicates within the same mailbox and keeps the most recent ingest_at copy.

Same email under different accounts

Two accounts subscribed to the same mailing list receive identical content → never deduplicated against each other. Each user has an independent archive.

EML import / SMTP ingestion

• Import and SMTP paths use uid=0.
• They go through the same pipeline: BLAKE3 hash → Fjall immediate dedup → Tantivy periodic dedup.
• If the same email is later synced via IMAP (with a real UID), the periodic dedup task keeps the more recently ingested copy.

Summary

Mechanism	Where	When	Scope
BLAKE3-256 hashing	utils/mod.rs:350	Every ingestion	Per-byte
Fjall contains_key check	blob.rs:58-87	Immediate (every write)	Blob storage, global
Periodic index dedup	dedup.rs	Every 12 hours	Tantivy index, per (mailbox_id, content_hash)
UIDVALIDITY rebuild	flow.rs:451	Every sync cycle	Entire mailbox
Reference-counted blob cleanup	envelope.rs:813-851	On envelope deletion	Fjall blobs, global
Migration buffer dedup	migrate/store.rs:494	During migration	In-memory buffers

Dedup does not mean "store once"

A core design principle: the same email content may exist as multiple index entries across different mailboxes. The goals are:

1. Storage efficiency — the blob layer ensures identical content occupies disk space only once.
2. Index cleanliness — no duplicate entries within the same mailbox (no repeated search results).
3. Cross-mailbox fidelity — when a user files the same email into multiple folders, it remains searchable from each one.

This balances storage efficiency against user visibility: blob dedup saves space, while the index allows cross-mailbox coexistence so every folder perspective is complete.