- use a single Engine instance for email + attachment blobs
- add delete_batch, gc_if_needed, background flush to blob crate
- fix Entry.raw_size storing compressed length instead of original
- move fjall-dependent migration code from core to admin crate
- add STORAGE_VERSION file for layout version detection
- Replace bincode 3.0.0 (empty crate) with bincode_reloaded 3.1.10
- Add delete_batch for efficient grouped tombstone writes
- Add gc_if_needed to skip GC when no segment exceeds threshold
- Add flush() for lightweight fsync+meta checkpoint without compact
- Add Config::flush_interval_secs to spawn a background flush thread
- Remove per-put/per-delete fsync; persistence via background flush
- Split gc_segments into gc_prepare/gc_finish to reduce write lock hold time
- Make MBOX/PST upload size limits configurable via SETTINGS
(bichon_web_mbox_upload_limit_mb defaults to 1 GB,
bichon_web_pst_upload_limit_mb defaults to 2 GB)
When SMTP journaling ingests a message, parse_email() upserts the account's
INBOX MailBox row so the journaled envelope has a row to attach to. But it
built the row with uid_validity/highest_uid/uid_next = None and called
batch_upsert, which replaces the WHOLE row. The INBOX row id
(create_hash(account_id, "INBOX")) is the same id the IMAP sync maintains,
so every journaled delivery reset the IMAP-maintained uid_validity to None.
On the next reconcile, local_mailbox.uid_validity != Some(remote) is then
true, so the mailbox is treated as invalid and wiped + rebuilt. For a large,
UID-sparse INBOX whose rebuild gets interrupted, the local copy is silently
lost and never restored (the incremental fetch resumes past all existing
UIDs). See #297 for the full diagnosis and DB evidence.
Fix: only create the INBOX row when it does not already exist; otherwise
leave the IMAP-owned row untouched. The journaling path only needs the row
to exist so the envelope can attach to it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
update_envelope_tags lost f_body, f_from_text, f_to_text, f_cc_text,
f_bcc_text, f_attachment_name_text and f_attachment_name_exact because
they are not STORED and field_values() skipped them during delete+add.
Rebuild these from stored counterparts and the blob-store EML.
Previously, do_migrate_segment created a fresh NewIndexWriter (and
therefore a new Fjall Database) on every call, meaning the Fjall
database at bichon-storage/ was opened and closed once per segment.
This caused the migration to fail mid-way through (observed at segment
9/16) with:
Storage(InvalidTag(("ChecksumType", 171)))
Root cause: after segment N writes email blobs via Fjall's ingestion
API (start_ingestion / write / finish), those SSTables and KV-separated
blob files are flushed to disk and the Database is dropped. When segment
N+1 calls Database::builder(storage_dir).open(), Fjall must discover and
catalog all on-disk files produced by the previous segments. During that
discovery it reads SSTable or blob-file block headers and encounters a
ChecksumType discriminant byte (171 / 0xAB) that lsm-tree 3.1.4 does
not recognise, causing the fatal error.
The first N segments succeed because the cumulative set of ingested
SSTables stays small enough that Fjall does not need to read the
offending headers during reopen. Once enough data has accumulated the
reopen triggers a manifest or compaction read that exposes the mismatch.
Fix: open NewIndexWriter once, before the segment loop, and pass a
&mut reference into each do_migrate_segment call. finish_writers() is
called a single time after all segments complete. The Fjall Database
stays open for the entire migration and is never closed and reopened,
eliminating the incompatible-reopen path entirely.
