70 Commits
1.1.2 ... 1.5.3

Author SHA1 Message Date
rustmailer
6fcc9e0e8e bump to v1.5.3 2026-06-23 17:18:46 +08:00
rustmailer
220aa268c1 feat(imap): handle UIDVALIDITY changes via Message-ID comparison instead of full rebuild 2026-06-23 10:46:36 +08:00
rustmailer
e3fd9d2f29 fix: purge DedupCache entries on account/mailbox/envelope removal 2026-06-22 17:11:20 +08:00
rustmailer
41c3b84e65 feat(ui): persist account table sorting to localStorage 2026-06-21 12:50:21 +08:00
rustmailer
89557700ae Merge branch 'main' of https://github.com/rustmailer/bichon 2026-06-11 09:23:00 +08:00
rustmailer
2f5de48c6a fix(smtp): reject journaling attempts to non-local accounts 2026-06-11 09:22:57 +08:00
rustmailer
debb119d3d Merge pull request #298 from shadowdao/fix/smtp-inbox-uidvalidity-clobber-297
fix(smtp): don't clobber the IMAP-owned INBOX uid_validity on journal ingest (#297)
2026-06-11 09:10:01 +08:00
Josh
ce3f8944a3 fix(smtp): don't clobber the IMAP-owned INBOX uid_validity on journal ingest
When SMTP journaling ingests a message, parse_email() upserts the account's
INBOX MailBox row so the journaled envelope has a row to attach to. But it
built the row with uid_validity/highest_uid/uid_next = None and called
batch_upsert, which replaces the WHOLE row. The INBOX row id
(create_hash(account_id, "INBOX")) is the same id the IMAP sync maintains,
so every journaled delivery reset the IMAP-maintained uid_validity to None.

On the next reconcile, local_mailbox.uid_validity != Some(remote) is then
true, so the mailbox is treated as invalid and wiped + rebuilt. For a large,
UID-sparse INBOX whose rebuild gets interrupted, the local copy is silently
lost and never restored (the incremental fetch resumes past all existing
UIDs). See #297 for the full diagnosis and DB evidence.

Fix: only create the INBOX row when it does not already exist; otherwise
leave the IMAP-owned row untouched. The journaling path only needs the row
to exist so the envelope can attach to it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-10 11:36:38 -07:00
rustmailer
6f572b15ae deps: upgrade fjall to 3.1.5 to address NFS Bad file descriptor error #216 #242 #239 #231 #240 2026-06-09 01:33:17 +08:00
rustmailer
3bfc080258 bump to 1.5.1 2026-06-07 18:21:07 +08:00
rustmailer
c3a725770c fix: reconnect and retry IMAP batch on BrokenPipe/network errors 2026-06-07 18:18:37 +08:00
rustmailer
c736afffb0 fix: account deletion times out #291 2026-06-07 15:34:38 +08:00
rustmailer
62cb5264fd Add funding.json for project funding details 2026-06-06 15:56:47 +08:00
rustmailer
aebb94ee4e fix: preserve non-stored search fields when updating envelope tags
update_envelope_tags lost f_body, f_from_text, f_to_text, f_cc_text,
  f_bcc_text, f_attachment_name_text and f_attachment_name_exact because
  they are not STORED and field_values() skipped them during delete+add.
  Rebuild these from stored counterparts and the blob-store EML.
2026-06-05 09:13:25 +08:00
rustmailer
769630f9d7 Merge branch 'main' of https://github.com/rustmailer/bichon 2026-06-04 23:46:28 +08:00
rustmailer
368b18c45f bump to v1.5.0 2026-06-04 23:46:25 +08:00
rustmailer
42861f6cc9 Merge pull request #288 from Korov/fix/tencent-mail-uidvalidity
fix: Add fallback UIDVALIDITY support for non-compliant IMAP servers
2026-06-04 23:45:40 +08:00
rustmailer
327a3f39d9 Merge pull request #287 from fama/dedup-cache-fix
fix: open NewIndexWriter once across all migration segments
2026-06-04 10:24:22 +08:00
fama
427f7248d2 fix: open NewIndexWriter once across all migration segments
Previously, do_migrate_segment created a fresh NewIndexWriter (and
therefore a new Fjall Database) on every call, meaning the Fjall
database at bichon-storage/ was opened and closed once per segment.

This caused the migration to fail mid-way through (observed at segment
9/16) with:

  Storage(InvalidTag(("ChecksumType", 171)))

Root cause: after segment N writes email blobs via Fjall's ingestion
API (start_ingestion / write / finish), those SSTables and KV-separated
blob files are flushed to disk and the Database is dropped. When segment
N+1 calls Database::builder(storage_dir).open(), Fjall must discover and
catalog all on-disk files produced by the previous segments. During that
discovery it reads SSTable or blob-file block headers and encounters a
ChecksumType discriminant byte (171 / 0xAB) that lsm-tree 3.1.4 does
not recognise, causing the fatal error.

The first N segments succeed because the cumulative set of ingested
SSTables stays small enough that Fjall does not need to read the
offending headers during reopen. Once enough data has accumulated the
reopen triggers a manifest or compaction read that exposes the mismatch.

Fix: open NewIndexWriter once, before the segment loop, and pass a
&mut reference into each do_migrate_segment call. finish_writers() is
called a single time after all segments complete. The Fjall Database
stays open for the entire migration and is never closed and reopened,
eliminating the incompatible-reopen path entirely.
2026-06-03 15:10:03 -06:00
rustmailer
a2a51a2037 feat(imap): add message size check before download 2026-06-03 21:17:35 +08:00
rustmailer
4d783d5301 Update Cargo.lock 2026-05-31 06:05:50 +08:00
rustmailer
4172f11f00 Update Cargo.toml 2026-05-31 06:05:41 +08:00
rustmailer
257736a47b fix: add in-memory dedup cache to prevent duplicate emails before indexing
Use (account_id, mailbox_id, content_hash) as dedup key with time-based
  eviction to bound memory at ~50MB. Check happens before mail parsing,
  so duplicate emails skip all expensive work entirely.

  - Populate cache from Tantivy FAST columns on startup (7-day window)
  - Evict oldest 1/4 of entries when exceeding 300K capacity
  - Graceful degradation: populate failure → empty cache, still works
2026-05-31 06:04:58 +08:00
Lei Zhu
e8469da3bc fix: Add fallback UIDVALIDITY support for non-compliant IMAP servers
This commit adds support for IMAP servers that don't provide UIDVALIDITY,
such as Tencent Enterprise Mail (腾讯企业邮箱).

Changes:
- Added `generate_synthetic_uidvalidity()` function that creates a stable
  hash-based UIDVALIDITY from the mailbox name
- Modified `reconcile_mailboxes()` to use synthetic UIDVALIDITY when the
  server doesn't provide one
- Servers without UIDVALIDITY can now sync all mailboxes including system
  folders (Sent Messages, Drafts, Deleted Messages)
- Incremental sync is supported via the synthetic UIDVALIDITY
- Added warning logs to indicate when synthetic UIDVALIDITY is in use
- Updated mailbox metadata to store the resolved UIDVALIDITY

Fixes issues with:
- Tencent Enterprise Mail (腾讯企业邮箱)
- Other non-compliant IMAP servers
- Mailboxes that don't properly support UIDVALIDITY"
2026-05-30 16:32:06 +08:00
rustmailer
1346dd216a fix: "No body available" #262
When the request returns an empty body, choose to skip it and print the account ID and UID information, leaving it for the user to investigate themselves. Otherwise, the IMAP download process will be blocked by this.
2026-05-29 16:43:38 +08:00
rustmailer
d40ba90b54 fix: inline attachment detection and account-scoped export
- Treat MIME parts with Content-ID but no Content-Disposition as inline
  - Add account_ids filter to CLI export search to avoid pulling all accounts
  - Skip failed emails during export instead of aborting the entire batch
2026-05-29 16:13:19 +08:00
rustmailer
048d5f361c fix: Cant migrate with version >= 1.4.0 #277 2026-05-28 22:49:24 +08:00
rustmailer
4597df515a Update content.rs 2026-05-28 17:55:13 +08:00
rustmailer
f4be4a2e8c bump to 1.4.1 2026-05-28 17:49:18 +08:00
rustmailer
4a3c42c1eb fix: HTTP Error 500 Internal Server Error: Failed for 58344335-2e86-4009-979d-6da0331bff63 - Failed to export an email. Aborting process... #275 2026-05-28 17:47:10 +08:00
rustmailer
8fcb55320f Update README.md 2026-05-28 02:09:58 +08:00
rustmailer
86869ac848 Update release.yml 2026-05-26 20:11:49 +08:00
rustmailer
38453accc6 Update release.yml 2026-05-26 20:07:40 +08:00
rustmailer
a60b2c7dc4 Update release.yml 2026-05-26 20:02:55 +08:00
rustmailer
8e46c7a162 fix: use valid IMAP UID SEARCH instead of BEFORE in UID FETCH for incremental sync 2026-05-26 19:42:26 +08:00
rustmailer
1a615e1c45 bump to v1.4.0 2026-05-26 15:27:34 +08:00
rustmailer
83dd9cdd6b perf: optimize IMAP account fetch flow 2026-05-26 15:23:34 +08:00
rustmailer
f30cd66e00 feat: remove folder limit 2026-05-26 15:22:14 +08:00
rustmailer
4bd714a670 chore: remove custom global allocator 2026-05-26 15:20:59 +08:00
rustmailer
c0a63a1e3c feat: enhance autoconfig detection 2026-05-26 15:19:39 +08:00
rustmailer
04a022e850 add features endpoint 2026-05-24 20:31:58 +08:00
rustmailer
f9c2fc77ff feat(core): wire up attachment text extraction in IMAP sync pipeline 2026-05-24 19:52:28 +08:00
rustmailer
ec3e842bbb chore: add trace logging for duplicate email diagnosis #214 2026-05-24 17:27:01 +08:00
rustmailer
7311529908 bump to v1.3.0 2026-05-24 16:50:15 +08:00
rustmailer
0792bb546d perf: reduce tokio worker thread blocking to improve responsiveness on low-core machines
- Switch memdb durability from Full to Batch(100) with 10s flush worker
  - Offload BlobManager fjall writes to spawn_blocking
  - Wrap Tantivy commit operations in block_in_place
  - Flush memdb WAL on graceful shutdown
2026-05-24 14:34:16 +08:00
rustmailer
0be2670600 update 2026-05-24 02:37:22 +08:00
rustmailer
575f851cfb update 2026-05-24 02:27:52 +08:00
rustmailer
61430b72b0 feat(core): add ext module with EventBus and AttachmentTextExtractor traits 2026-05-24 02:04:28 +08:00
rustmailer
15c0cfc1d9 Update README.md 2026-05-23 15:57:39 +08:00
rustmailer
ea8c493374 Update .gitignore 2026-05-23 15:41:39 +08:00
rustmailer
005b1c2116 feat: added Cron scheduling for email downloads #211 2026-05-23 15:40:46 +08:00
rustmailer
d8b78b8010 update 2026-05-23 12:08:49 +08:00
rustmailer
36f3f19cdc fix(core): use email schema field for attachment hash lookup in cleanup_unused_content 2026-05-23 11:36:47 +08:00
rustmailer
9f3097df32 Merge pull request #259 from mmaudet/feat/search-by-server-timestamp
feat: filter and sort search-messages by a server-side timestamp
2026-05-23 11:04:54 +08:00
rustmailer
c075b9ef12 Merge pull request #258 from mmaudet/fix/self-heal-missing-content
fix: self-heal a missing content blob in download-message
2026-05-23 11:03:04 +08:00
rustmailer
21a7f7e9d5 Merge pull request #257 from mmaudet/fix/gc-blob-still-referenced
fix: prevent the dedup GC from deleting a still-referenced content blob
2026-05-23 10:58:34 +08:00
rustmailer
26c14fcaaf refactor(migrate): use searchable_segment_ids() instead of reader() for merge #261 2026-05-23 10:26:09 +08:00
Michel-Marie MAUDET
6873841ba4 fix(search): expose new SortBy variants via #[oai(rename)]
InternalDate and IngestAt were renamed for the wire with #[serde(rename)] only. SortBy derives poem_openapi::Enum, which does not honour serde attributes, so the REST deserializer exposed them under their Rust identifiers instead of the intended INTERNAL_DATE / INGEST_AT — inconsistent with the existing DATE/SIZE values and rejecting the documented names with HTTP 400. Add #[oai(rename = ...)] alongside the serde rename.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 23:14:01 +02:00
rustmailer
0dd81f599d update 2026-05-22 22:21:46 +08:00
Michel-Marie MAUDET
b3afc52a82 feat(search): filter and sort messages by server-side timestamps
POST /api/v1/search-messages previously filtered and sorted only on the
sender-controlled Date: header. Add support for two server-controlled
timestamps that are already indexed as FAST i64 fields:

- internal_date (IMAP INTERNALDATE)
- ingest_at (Bichon's archival time)

EmailSearchFilter gains internal_date_since/before and ingest_since/before
range bounds, mirroring the existing `since`/`before` Date: handling.
SortBy gains InternalDate and IngestAt variants (wire values INTERNAL_DATE
and INGEST_AT), mirroring the existing DATE/SIZE sort handling.

The envelope and attachment Tantivy schemas already declare these fields
as INDEXED | STORED | FAST, so no re-index or migration is required.
Attachments carry no IMAP INTERNALDATE, so the attachment search maps the
InternalDate sort to the attachment's own date field as a defined fallback.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 13:55:23 +02:00
Michel-Marie MAUDET
171a40d70f fix: self-heal missing content blob in download-message
The download-message REST endpoint returned 404 [30000] "Original email
content not found" whenever an indexed message's raw content blob was
missing from the blob store, leaving the message permanently
unrecoverable even though it still existed on the IMAP server.

Make the endpoint self-healing: when the content blob is absent, fetch
that one message on demand from the IMAP server via
UID FETCH <uid> (BODY.PEEK[]), repopulate the detached blob, and return
the content. The 404 is now only produced when the on-demand fetch
itself fails (mailbox gone, UID gone, connection failure, or the fetched
bytes no longer match the archived content_hash).

- Add ImapExecutor::fetch_single_message_body: examines the mailbox
  read-only and fetches one message by UID, reusing the existing
  BODY.PEEK[] fetch command.
- Add reattach_eml_content_self_healing / recover_message_blob in the
  envelope extractor: fast-path delegates to reattach_eml_content when
  the blob exists; otherwise recovers it via IMAP and re-stores the
  stripped EML + attachments through the existing blob queue.
- Make store::blob::get_reader async and route it through the
  self-healing path; update the single caller (download_message).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 13:46:40 +02:00
Michel-Marie MAUDET
895ea543a9 fix(core): commit/reload barrier before dedup GC reference count
cleanup_unused_content decides whether to delete a deduplicated blob by
running a Tantivy Count of envelopes referencing each content_hash. The
searcher it used reflected only the committed index state at the time it
was built, so an envelope that shared a content hash but was still
sitting uncommitted in the IndexWriter buffer (for example added by the
background ingest task before the delete operation acquired the writer
lock) was invisible to the count. The count read 0 and a
still-referenced blob was deleted, permanently 404ing that envelope's
download-message.

Pass the locked IndexWriter into cleanup_unused_content and fatal_commit
it immediately before creating the searcher. create_searcher already
reloads the reader, so the Count is now evaluated against a fully
committed, freshly-reloaded index state. The barrier is local to the GC
path and self-contained, independent of what the caller committed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 13:40:43 +02:00
rustmailer
6eca351994 update i18n 2026-05-21 23:52:26 +08:00
rustmailer
1f477eca65 bump to v1.2.0 2026-05-21 23:33:26 +08:00
rustmailer
a3cdc094e8 feat: Strip remote data from emails when viewed #54 2026-05-21 23:32:42 +08:00
rustmailer
95147a7824 bump to v1.1.3 2026-05-21 18:31:00 +08:00
rustmailer
b22811f78c fix: Transparent menu on iPhone #253 2026-05-21 18:29:03 +08:00
rustmailer
fd61d013a2 fix: Imported emails and UTF-8 folders missing #182 2026-05-21 17:57:43 +08:00
rustmailer
3a950e7591 fix: account name don't change when Update Account #248 2026-05-21 10:38:04 +08:00
rustmailer
f17820bfa8 fix: add missing attachment index cleanup logic 2026-05-21 08:45:30 +08:00
130 changed files with 11059 additions and 1641 deletions

4
.gitignore vendored
View File

@@ -1,4 +1,6 @@
/target
.vscode
.idea
config.toml
config.toml
node_modules
dedup_report.txt

95
Cargo.lock generated
View File

@@ -293,7 +293,7 @@ checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6"
[[package]]
name = "bichon-admin"
version = "1.1.2"
version = "1.5.3"
dependencies = [
"bichon-core",
"console",
@@ -301,7 +301,6 @@ dependencies = [
"indicatif",
"itertools",
"memdb",
"mimalloc",
"native_db",
"native_model",
"serde",
@@ -312,7 +311,7 @@ dependencies = [
[[package]]
name = "bichon-cli"
version = "1.1.2"
version = "1.5.3"
dependencies = [
"base64 0.22.1",
"bichon-core",
@@ -338,7 +337,7 @@ dependencies = [
[[package]]
name = "bichon-core"
version = "1.1.2"
version = "1.5.3"
dependencies = [
"async-imap",
"base64 0.22.1",
@@ -346,6 +345,7 @@ dependencies = [
"bytes 1.11.1",
"chrono",
"clap",
"cron",
"dashmap",
"deunicode",
"email_address",
@@ -396,7 +396,7 @@ dependencies = [
[[package]]
name = "bichon-server"
version = "1.1.2"
version = "1.5.3"
dependencies = [
"bichon-core",
"bichon-smtp",
@@ -404,7 +404,6 @@ dependencies = [
"email_address",
"governor",
"http",
"mimalloc",
"poem",
"poem-derive",
"poem-openapi",
@@ -421,7 +420,7 @@ dependencies = [
[[package]]
name = "bichon-smtp"
version = "1.1.2"
version = "1.5.3"
dependencies = [
"base64 0.22.1",
"bichon-core",
@@ -657,9 +656,9 @@ dependencies = [
[[package]]
name = "chrono"
version = "0.4.44"
version = "0.4.45"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c673075a2e0e5f4a1dde27ce9dee1ea4558c7ffe648f576438a20ca1d2acc4b0"
checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327"
dependencies = [
"iana-time-zone",
"js-sys",
@@ -875,6 +874,17 @@ version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "790eea4361631c5e7d22598ecd5723ff611904e3344ce8720784c93e3d83d40b"
[[package]]
name = "cron"
version = "0.15.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5877d3fbf742507b66bc2a1945106bd30dd8504019d596901ddd012a4dd01740"
dependencies = [
"chrono",
"once_cell",
"winnow 0.6.26",
]
[[package]]
name = "crossbeam-channel"
version = "0.5.15"
@@ -1006,9 +1016,9 @@ dependencies = [
[[package]]
name = "dashmap"
version = "6.1.0"
version = "6.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5041cc499144891f3790297212f32a74fb938e5136a14943f338ef9e0ae276cf"
checksum = "e6361d5c062261c78a176addb82d4c821ae42bed6089de0e12603cd25de2059c"
dependencies = [
"cfg-if",
"crossbeam-utils",
@@ -1285,9 +1295,9 @@ checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582"
[[package]]
name = "fjall"
version = "3.1.4"
version = "3.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b62b25b4d815ae178d7d9e4aa32ee59f072efd5431c736abede1e6ee13c8c453"
checksum = "038acd422d607e0eca09e093f299f9eccf9bd097554343d93746afff81a45113"
dependencies = [
"byteorder-lite",
"byteview",
@@ -1779,9 +1789,9 @@ checksum = "e9025058dae765dee5070ec375f591e2ba14638c63feff74f13805a72e523163"
[[package]]
name = "http"
version = "1.4.0"
version = "1.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e3ba2a386d7f85a81f119ad7498ebe444d2e22c2af0b86b069416ace48b3311a"
checksum = "6970f50e31d6fc17d3fa27329444bfa74e196cf62e95052a3f6fee181dba6425"
dependencies = [
"bytes 1.11.1",
"itoa",
@@ -2252,15 +2262,6 @@ version = "0.2.185"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "52ff2c0fe9bc6cb6b14a0592c2ff4fa9ceb83eea9db979b0487cd054946a2b8f"
[[package]]
name = "libmimalloc-sys"
version = "0.1.47"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2d1eacfa31c33ec25e873c136ba5669f00f9866d0688bea7be4d3f7e43067df6"
dependencies = [
"cc",
]
[[package]]
name = "linux-raw-sys"
version = "0.12.1"
@@ -2314,9 +2315,9 @@ checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154"
[[package]]
name = "lsm-tree"
version = "3.1.4"
version = "3.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e447ac67ff6aef4ec07fc19e507b219336cbba90a697c0dbeb1bf51b91536b67"
checksum = "8ef86c3c797c10eefcc73407c43ae48c19d4df686131a8334b2895a513e91df4"
dependencies = [
"byteorder-lite",
"bytes 1.11.1",
@@ -2465,15 +2466,6 @@ dependencies = [
"libc",
]
[[package]]
name = "mimalloc"
version = "0.1.50"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b3627c4272df786b9260cabaa46aec1d59c93ede723d4c3ef646c503816b0640"
dependencies = [
"libmimalloc-sys",
]
[[package]]
name = "mime"
version = "0.3.17"
@@ -3931,9 +3923,9 @@ dependencies = [
[[package]]
name = "serde_json"
version = "1.0.149"
version = "1.0.150"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "83fc039473c5595ace860d8c4fafa220ff474b3fc6bfdb4293327f1a37e94d86"
checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9"
dependencies = [
"itoa",
"memchr",
@@ -4123,18 +4115,18 @@ checksum = "97c9f5dd7ec5cc6d743f33fcb96de4eb91bb1cc51c5e0ba40cb285a9012043da"
[[package]]
name = "snafu"
version = "0.9.0"
version = "0.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d1d4bced6a69f90b2056c03dcff2c4737f98d6fb9e0853493996e1d253ca29c6"
checksum = "d1a012328be2e3f5d5f6f3218147ca02588cea4cb865e876849ab6debcf36522"
dependencies = [
"snafu-derive",
]
[[package]]
name = "snafu-derive"
version = "0.9.0"
version = "0.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "54254b8531cafa275c5e096f62d48c81435d1015405a91198ddb11e967301d40"
checksum = "5f103c50866b8743da9429b8a581d81a27c2d3a9c4ac7df8f8571c1dd7896eda"
dependencies = [
"heck",
"proc-macro2",
@@ -4286,9 +4278,9 @@ dependencies = [
[[package]]
name = "sysinfo"
version = "0.39.1"
version = "0.39.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a4deba334e1190ba7cb498327affa11e5ece10d26a30ab2f27fcf09504b8d8b6"
checksum = "14311e7e9a03114cd4b65eedd54e8fed2945e17f08586ae97ef53bc0669f9581"
dependencies = [
"libc",
"memchr",
@@ -4647,9 +4639,9 @@ dependencies = [
[[package]]
name = "tokio-socks"
version = "0.5.2"
version = "0.5.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0d4770b8024672c1101b3f6733eab95b18007dbe0847a8afe341fcf79e06043f"
checksum = "a7e2948f60dbe26b35f2c7fb74ac2854c1fddded0fe9d7548fcc674a246f7615"
dependencies = [
"either",
"futures-util",
@@ -5047,9 +5039,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
[[package]]
name = "uuid"
version = "1.23.1"
version = "1.23.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ddd74a9687298c6858e9b88ec8935ec45d22e8fd5e6394fa1bd4e99a87789c76"
checksum = "d258b83ceec21034727ecee8c382cfa6c3e133699b0742c64571814fb420c9f7"
dependencies = [
"getrandom 0.4.2",
"js-sys",
@@ -5611,6 +5603,15 @@ version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650"
[[package]]
name = "winnow"
version = "0.6.26"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e90edd2ac1aa278a5c4599b1d89cf03074b610800f866d4026dc199d7929a28"
dependencies = [
"memchr",
]
[[package]]
name = "winnow"
version = "0.7.15"

View File

@@ -6,29 +6,29 @@ members = [
"crates/server",
"crates/cli",
"crates/admin",
"crates/smtp",
]
resolver = "2"
[workspace.package]
version = "1.1.2"
version = "1.5.3"
edition = "2021"
[workspace.dependencies]
chrono = "0.4.44"
chrono = "0.4.45"
clap = { version = "4.6.1", features = ["derive", "env"] }
mimalloc = "0.1.50"
memdb = { path = "crates/memdb" }
itertools = "0.14.0"
ring = { version = "0.17.14", features = ["std"] }
serde = { version = "1.0.228", features = ["derive"] }
serde_json = "1.0.149"
serde_json = "1.0.150"
tokio = { version = "1.52.3", features = ["full"] }
tracing = "0.1.44"
tracing-appender = "0.2.3"
tracing-subscriber = { version = "0.3.23", features = ["env-filter", "json"] }
base64 = "0.22.1"
snafu = "0.9.0"
snafu = "0.9.1"
reqwest = { version = "0.12.24", default-features = false, features = [
"json",
"stream",
@@ -37,8 +37,8 @@ reqwest = { version = "0.12.24", default-features = false, features = [
"blocking",
"socks",
] }
tokio-socks = "0.5.2"
http = "1.4.0"
tokio-socks = "0.5.3"
http = "1.4.2"
regex = "1.12.3"
email_address = "0.2.9"
futures = "0.3.32"
@@ -52,7 +52,7 @@ tokio-rustls = { version = "0.26.4", default-features = false, features = [
timeago = "0.6.0"
oauth2 = { version = "5.0.0", features = ["reqwest-blocking"] }
url = { version = "2.5.8", features = ["serde"] }
sysinfo = "0.39.1"
sysinfo = "0.39.2"
num_cpus = "1.17.0"
rand = "0.10.1"
encoding_rs = "0.8.35"
@@ -73,7 +73,7 @@ time = { version = "0.3.47", features = [
rust-embed = "8.11.0"
murmur3 = "0.5.2"
urlencoding = "2.1.3"
dashmap = "6.1.0"
dashmap = "6.2.1"
gethostname = "1.1.0"
itoa = "1.0.18"
html2text = "0.17.1"
@@ -84,8 +84,8 @@ mail-send = "0.6.0"
rcgen = "0.14.8"
rustls-pemfile = "2.2.0"
blake3 = "1.8.5"
uuid = { version = "1.23.1", features = ["v4", "serde"] }
fjall = { version = "3.1.4", features = ["lz4", "metrics", "bytes_1"] }
uuid = { version = "1.23.2", features = ["v4", "serde"] }
fjall = { version = "3.1.5", features = ["lz4", "metrics", "bytes_1"] }
tracing-log = "0.2.0"
tokio-util = "0.7.18"
indicatif = "0.18.4"

102
README.md
View File

@@ -106,6 +106,10 @@
- **Admin Tooling**: Password reset for locked-out admins. Non-destructive v0.3.7 to v1.0 data migration.
- **API Token Management**: Create, list, and revoke long-lived API tokens for programmatic access.
- **SOCKS5 Proxy Management**: Configure and manage proxy profiles for routing IMAP traffic per account.
- **Scheduled Download**: Configure per-account download schedules using cron expressions. Run syncs at specific times or intervals — for example, nightly-only or business-hours-only archiving.
- **Remote Content Blocking**: External images and tracking pixels embedded in emails are blocked by default. Users can selectively allow remote content to load on a per-message basis from the WebUI.
- **Async Index Deduplication**: Duplicate detection in the search index is performed asynchronously, reducing write latency during high-throughput ingestion.
## Quick Start
@@ -445,7 +449,7 @@ Storage Layer │
└──────────────┘ └──────────────┘ └──────────────┘
```
- **memdb**: Key-value metadata store. Houses accounts, users, roles, OAuth2 configs, proxy settings, and system configuration. All operations wrapped in `tokio::spawn_blocking`.
- **memdb**: Key-value metadata store. Houses accounts, users, roles, OAuth2 configs, proxy settings, and system configuration.
- **Tantivy**: Full-text search indices with Zstd compression support. Two separate indices: envelope (email metadata + body text) and attachment (file metadata + extracted text). Batch-committed every 1,000 documents or 60 seconds.
- **Fjall**: LZ4-compressed LSM tree key-value store. Two keyspaces — `email_keyspace` and `attachments_keyspace`. Content-hash addressed (BLAKE3) with insert-time deduplication. Values larger than 1 KB stored as separate files (KV separation).
@@ -481,6 +485,79 @@ Tantivy Fjall memdb
- Manual sync via `POST /api/v1/accounts/:id/start-download`; cancel with `cancel-download`
- Busy-check prevents overlapping manual and automatic syncs on the same account
### Content Deduplication & Attachment Storage
```
┌──────────────────────────────────────────┐
│ Raw EML bytes │
└────────────────┬─────────────────────────┘
┌──────────────────────────────────────────┐
│ BLAKE3 → email_content_hash │
└────────────────┬─────────────────────────┘
┌──────────────────────────────────────────┐
│ MIME parse → Message │
└───────┬──────────────────┬──────────────┘
│ │
│ ┌────────────┘
│ │ detach attachments
│ │
▼ ▼
┌─────────────────┐ ┌──────────────────────────────┐
│ EMAIL BODY │ │ EACH ATTACHMENT │
│ │ │ │
│ Replace raw │ │ BLAKE3(decoded content) │
│ attachment │ │ → attachment_content_hash │
│ bytes with │ │ │
│ placeholder: │ │ Store raw undecoded bytes │
│ │ │ in Fjall attachments_ks │
│ <<BICHON_ │ │ (skip if hash exists) │
│ DETACH_HASH: │ │ │
│ xxx>> │ │ Extract text for indexing │
│ │ │ (PDF, DOCX, etc.) │
└───────┬─────────┘ └──────────────┬───────────────┘
│ │
▼ │
┌──────────────────────────────┐ │
│ Stripped EML stored in │ │
│ Fjall email_keyspace │ │
│ keyed by email_content_hash │ │
│ (skip if hash exists) │ │
└──────────────┬───────────────┘ │
│ │
▼ ▼
┌─────────────────────────────────────────────────┐
│ Tantivy full-text index │
│ envelope index · attachment index │
└─────────────────────────────────────────────────┘
═══════════════════════════════════════════════════════════════
Dedup layers
┌─────────────────────────────────────────────────────────────────┐
│ Fjall (insert-time) │
│ contains_key(hash)? → skip : store with LZ4 compression │
│ │
│ Tantivy (periodic, every 12 h) │
│ Group by (account, mailbox, content_hash) │
│ Keep latest ingest_at → soft-delete older copies │
│ Cascade-delete orphaned attachment index entries │
└─────────────────────────────────────────────────────────────────┘
Reconstruction
┌─────────────────────────────────────────────────────────────────┐
│ Fetch stripped EML by content_hash from Fjall │
│ Find <<BICHON_DETACH_HASH:xxx>> placeholders │
│ Replace each with raw attachment blob from Fjall │
│ Result → byte-identical original EML │
└─────────────────────────────────────────────────────────────────┘
```
Every ingested email is hashed with BLAKE3. Attachments are detached from the MIME tree, hashed independently (decoded content), and stored as raw undecoded bytes in Fjall's `attachments_keyspace`. The email body is patched with hash-based placeholders and stored in `email_keyspace`. Both keyspaces check for existing hashes before writing — identical content is never stored twice, regardless of which account or folder it arrives in. A periodic index dedup task (every 12 hours) scans Tantivy for duplicate `(account, mailbox, content_hash)` tuples, keeps the most recently ingested copy, and cascade-deletes orphaned attachment entries so UID-based incremental sync remains accurate. The original EML reconstructs byte-for-byte by swapping placeholders back with their attachment blobs.
## Storage & Backup
### Data Directory Layout
@@ -528,21 +605,21 @@ The WebUI is available in **18 languages**:
Language preference and UI theme are saved to your user profile and can be changed anytime from the WebUI settings.
## Data Migration (v0.3.7 → v1.0)
## Data Migration (v0.3.7 → v1.x)
Bichon v1.0 introduced a redesigned storage architecture:
Bichon v1.x introduced a redesigned storage architecture:
| Layer | v0.3.7 (Legacy) | v1.0 |
|-------|---------------|------|
| **Index** | Tantivy (shared) | Tantivy (separate envelope + attachment indices) |
| **Raw data** | Tantivy (inline) | Fjall (LZ4-compressed key-value store) |
| **Metadata** | Tantivy (shared) | memdb (dedicated embedded DB) |
| Layer | v0.3.7 (Legacy) | v1.x |
| :--- | :--- | :--- |
| **Index** | Tantivy (shared instance, no full attachments) | Tantivy (separate envelope + attachment indices) |
| **Raw data** | Tantivy (inline, stored in another Tantivy instance) | Fjall (LZ4-compressed LSM-tree key-value store) |
| **Metadata** | Native_DB (shared, disk-based DB powered by redb) | memdb (dedicated, in-house in-memory DB) |
If you ran Bichon prior to v1.0, migrate your data:
If you ran Bichon prior to v1.x, migrate your data:
```bash
./bichon-admin
# Select "Migrate Legacy v0.3.7 Storage to v1.0"
# Select "Migrate Legacy v0.3.7 Storage to v1.x"
```
> [!NOTE]
@@ -580,8 +657,8 @@ No. Bichon is an **archiver**, not an email client. The optional SMTP server **r
### What hardware does Bichon need?
- **Minimal:** 1 CPU core, 512 MB RAM
- **Recommended (100+ accounts, 200+ GB):** 4+ cores, 2+ GB RAM
- **Recommended:** 4+ CPU cores, 2+ GB RAM (sufficient for 10+ accounts and 200+ GB of archived data)
- Filesystem: use a mainstream Linux filesystem such as **ext4** or **XFS**; avoid network / virtual filesystems (NFS, VirtIO-FS) for all data directories
- Indices benefit from SSD storage; blob storage can use HDD
### How do I reset the admin password?
@@ -652,7 +729,6 @@ Feel free to open an [Issue](https://github.com/rustmailer/bichon/issues) or joi
| **Frontend** | React 18, TypeScript, Vite 6, ShadCN UI, TanStack Router/Query/Table |
| **Charts** | Recharts |
| **i18n** | i18next (18 languages) |
| **Allocator** | mimalloc |
| **Container** | Ubuntu 24.04, Docker |
## License

View File

@@ -1,2 +1,2 @@
base_url = "http://localhost:15630"
api_token = "WuqNC0g8yNle7CVnxcvjUwjN"
api_token = "eErI7WN3PtKeLwWAbIfSXCP6"

View File

@@ -17,5 +17,4 @@ serde_json.workspace = true
itertools.workspace = true
snafu.workspace = true
memdb.workspace = true
mimalloc = "0.1.50"
memdb.workspace = true

View File

@@ -18,7 +18,6 @@
use console::style;
use dialoguer::{theme::ColorfulTheme, Select};
use mimalloc::MiMalloc;
use crate::{migrate::handle_migration, reset::handle_reset_password};
@@ -26,8 +25,6 @@ pub mod meta;
pub mod migrate;
pub mod reset;
#[global_allocator]
static GLOBAL: MiMalloc = MiMalloc;
fn main() {
run_interactive();
@@ -43,7 +40,7 @@ async fn run_interactive() {
let main_options = vec![
"Reset Admin Password",
"Migrate Legacy v0.3.7 Storage to v1.0.x",
"Migrate Legacy v0.3.7 Storage to v1.x",
"Exit",
];

View File

@@ -246,11 +246,11 @@ impl From<AccountV3> for AccountModel {
capabilities: value.capabilities,
date_since: value.date_since,
date_before: value.date_before,
folder_limit: value.folder_limit,
download_folders: value.sync_folders,
account_type: value.account_type,
download_interval_min: value.sync_interval_min,
download_batch_size: value.sync_batch_size,
max_email_size_bytes: None,
known_folders: value.known_folders,
created_at: value.created_at,
updated_at: value.updated_at,
@@ -261,6 +261,8 @@ impl From<AccountV3> for AccountModel {
imap_quota_window: None,
imap_quota_bytes: None,
auto_download_new_mailboxes: None,
download_schedule: None,
deleting: false,
}
}
}
@@ -660,6 +662,7 @@ impl From<MailBox> for bichon_core::cache::imap::mailbox::MailBox {
unseen: value.unseen,
uid_next: value.uid_next,
uid_validity: value.uid_validity,
highest_uid: None,
}
}
}

View File

@@ -2,7 +2,7 @@ use std::path::{Path, PathBuf};
use bichon_core::migrate::{
count_eml_segments, do_migrate_segment, is_tantivy_index_dir,
store::{LegacyDirs, NewDirs},
store::{LegacyDirs, NewDirs, NewIndexWriter},
};
use console::style;
use dialoguer::{theme::ColorfulTheme, Confirm, Input};
@@ -11,7 +11,7 @@ use indicatif::{ProgressBar, ProgressStyle};
pub fn handle_migration(theme: &ColorfulTheme) {
println!(
"\n{}",
style("MIGRATION: Bichon v0.3.7 Storage Architecture → v1.0.x")
style("MIGRATION: Bichon v0.3.7 Storage Architecture → v1.x")
.bold()
.yellow()
);
@@ -20,7 +20,7 @@ pub fn handle_migration(theme: &ColorfulTheme) {
"{}",
style(
"This tool migrates data from the legacy v0.3.7 Tantivy-based storage \
architecture to the new v1.0.x \
architecture to the new v1.x \
separated index and Fjall-backed storage format."
)
.dim()
@@ -32,7 +32,7 @@ pub fn handle_migration(theme: &ColorfulTheme) {
"Legacy v0.3.7 architecture:\n\
• envelope metadata stored in Tantivy\n\
• message data stored in Tantivy\n\n\
New v1.0.x architecture:\n\
New v1.x architecture:\n\
• mail indexes stored in Tantivy\n\
• attachment indexes stored in Tantivy\n\
• raw message data stored in Fjall\n\
@@ -163,7 +163,7 @@ pub fn handle_migration(theme: &ColorfulTheme) {
println!("----------------------------------------");
println!(
"\n{} Checking legacy v0.x storage layout...",
"\n{} Checking legacy v0.3.7 storage layout...",
style("").yellow()
);
@@ -172,7 +172,7 @@ pub fn handle_migration(theme: &ColorfulTheme) {
println!(
"{} {}",
style("").green(),
style("Legacy v0.3.7 Tantivy-based storage detected. Migration to v1.0 is required.")
style("Legacy v0.3.7 Tantivy-based storage detected. Migration to v1.x is required.")
.yellow()
);
}
@@ -186,7 +186,7 @@ pub fn handle_migration(theme: &ColorfulTheme) {
println!(
"{}",
style(
"The selected directories may already be using the v1.0 storage architecture."
"The selected directories may already be using the v1.x storage architecture."
)
.dim()
);
@@ -288,12 +288,6 @@ pub fn handle_migration(theme: &ColorfulTheme) {
style(batch_size).cyan().bold()
);
println!(
"{} Using batch size: {}\n",
style("").green(),
style(batch_size).cyan().bold()
);
let legacy = LegacyDirs::new(index_path.clone(), data_path.clone());
let total_segments = match count_eml_segments(&legacy) {
Ok(n) => n,
@@ -332,6 +326,18 @@ pub fn handle_migration(theme: &ColorfulTheme) {
.progress_chars("#>-"),
);
let mut writer = match NewIndexWriter::open(NewDirs::new(
new_index_path.clone(),
new_data_path.clone(),
)) {
Ok(w) => w,
Err(e) => {
pb.finish_with_message(format!("{}", style("Migration failed.").red()));
eprintln!("\n{} {:?}", style("").red().bold(), e);
return;
}
};
let mut grand_total_migrated: usize = 0;
let mut grand_total_skipped: usize = 0;
@@ -343,7 +349,7 @@ pub fn handle_migration(theme: &ColorfulTheme) {
match do_migrate_segment(
batch_size,
legacy,
NewDirs::new(new_index_path.clone(), new_data_path.clone()),
&mut writer,
seg_idx,
|msg| {
if let Some(data) = msg.strip_prefix("TOTAL:") {
@@ -413,6 +419,13 @@ pub fn handle_migration(theme: &ColorfulTheme) {
pb.set_position((seg_idx + 1) as u64);
}
pb.set_message(style("Finalizing indexes...").dim().to_string());
if let Err(e) = writer.finish_writers() {
pb.finish_with_message(format!("{}", style("Migration failed.").red()));
eprintln!("\n{} {:?}", style("").red().bold(), e);
return;
}
pb.finish_with_message(format!(
"Migration finished. Total: {}, Skipped: {}",
grand_total_migrated, grand_total_skipped

View File

@@ -10,13 +10,17 @@ use crate::BichonCliConfig;
pub async fn search_messages(
client: &Client,
config: &BichonCliConfig,
account_ids: Option<std::collections::HashSet<u64>>,
page: u64,
page_size: u64,
) -> Option<DataPage<Envelope>> {
let url = format!("{}/api/v1/search-messages", config.base_url);
let payload = EmailSearchRequest {
filter: EmailSearchFilter::default(),
filter: EmailSearchFilter {
account_ids,
..Default::default()
},
page,
page_size,
sort_by: Some(SortBy::DATE),

View File

@@ -146,20 +146,23 @@ pub async fn handle_account_export(
let mut total_pages;
loop {
if let Some(batch) = search_messages(&client, config, current_page, page_size).await {
let account_ids = Some(std::collections::HashSet::from([account.id]));
if let Some(batch) = search_messages(&client, config, account_ids, current_page, page_size).await {
total_pages = batch.total_pages.unwrap();
pb.set_message(format!("Page {}/{}", current_page, total_pages));
for envelope in batch.items {
let success =
download_and_export_with_json_header(&client, config, envelope, &mut file)
download_and_export_with_json_header(&client, config, envelope.clone(), &mut file)
.await;
if !success {
pb.finish_with_message("Failed");
eprintln!(" ✘ Failed to export an email. Aborting process...");
return;
eprintln!(
" ✘ Failed to export email {}, skipping...",
envelope.id
);
continue;
}
pb.inc(1);
}

View File

@@ -60,3 +60,75 @@ pub fn determine_folder(labels_raw: &str) -> String {
}
}
}
#[cfg(test)]
mod tests {
use mail_parser::{HeaderValue, MessageParser};
use super::*;
fn parse_x_gmail_labels(raw_message: &[u8]) -> Option<String> {
// MessageParser::new() has an empty header_map so the hardcoded match at
// parsers/header.rs:76 treats ALL unknown headers as raw (no RFC 2047
// decoding). We need three things to get decoding:
// 1. A non-empty header_map (so the else branch runs)
// 2. default_header_text() so the fallback fn is parse_unstructured
// 3. OR register X-Gmail-Labels explicitly via header_text()
let message = MessageParser::new()
.with_minimal_headers()
.default_header_text()
.parse(raw_message)?;
let value: &HeaderValue<'_> = message.header("X-Gmail-Labels")?;
value.as_text().map(|s| s.to_string())
}
/// Construct a raw MIME message with RFC 2047 encoded X-Gmail-Labels,
/// parse it, and verify the header is correctly decoded.
fn build_email(x_gmail_labels: &str) -> Vec<u8> {
format!(
"From: sender@example.com\r\n\
To: recipient@example.com\r\n\
Subject: Test\r\n\
X-Gmail-Labels: {}\r\n\
\r\n\
Body text here.\r\n",
x_gmail_labels
)
.into_bytes()
}
#[test]
fn rfc2047_encoded_labels_are_decoded() {
// Exactly the format the user reported: French Gmail labels
let raw = build_email("=?UTF-8?Q?Corbeille?=, =?UTF-8?Q?Messages_archiv=C3=A9s?=");
let labels = parse_x_gmail_labels(&raw).expect("failed to parse X-Gmail-Labels");
// mail-parser decodes RFC 2047 header values during initial parsing.
// The decoded text should NOT contain raw =?UTF-8?Q?... sequences.
assert!(!labels.contains("=?UTF-8"), "labels still encoded: {labels:?}");
assert!(labels.contains("Corbeille"), "missing 'Corbeille': {labels:?}");
assert!(
labels.contains("archivés"),
"missing decoded 'archivés': {labels:?}",
);
// Full pipeline: decoded labels → determine_folder
let folder = determine_folder(&labels);
assert_eq!(folder, "Corbeille");
}
#[test]
fn plain_ascii_labels_passthrough() {
let raw = build_email("Inbox, Important");
let labels = parse_x_gmail_labels(&raw).expect("failed to parse X-Gmail-Labels");
assert_eq!(labels, "Inbox, Important");
assert_eq!(determine_folder(&labels), "Important");
}
#[test]
fn missing_x_gmail_labels_header() {
let raw = b"From: sender@example.com\r\nTo: r@example.com\r\n\r\nBody.\r\n";
let message = MessageParser::new().parse(raw.as_slice()).unwrap();
assert!(message.header("X-Gmail-Labels").is_none());
}
}

View File

@@ -28,7 +28,6 @@ use bichon_core::envelope::meta::{parse_bichon_metadata, BichonMetadata};
use console::style;
use dialoguer::{theme::ColorfulTheme, Input};
use dialoguer::{Confirm, Select};
use mail_parser::parsers::MessageStream;
use mail_parser::MessageParser;
use reqwest::Client;
@@ -161,7 +160,11 @@ pub async fn run_import(
continue;
}
let message = match MessageParser::new().parse(body) {
let message = match MessageParser::new()
.with_minimal_headers()
.default_header_text()
.parse(body)
{
Some(msg) => msg,
None => {
eprintln!(
@@ -181,15 +184,12 @@ pub async fn run_import(
}
let get_default_folder = || {
let gmail_labels = message.header_raw("X-Gmail-Labels").unwrap_or("INBOX");
let text_cow = MessageStream::new(gmail_labels.as_bytes())
.parse_unstructured()
.into_text();
let data: &str = match &text_cow {
Some(c) => c.as_ref(),
None => "INBOX",
};
determine_folder(data)
let labels = message
.header("X-Gmail-Labels")
.and_then(|h| h.as_text())
.map(|s| s.to_string())
.unwrap_or_else(|| "INBOX".to_string());
determine_folder(&labels)
};
let folder_name = if let Some(ref folder) = target_folder {

View File

@@ -69,5 +69,6 @@ tokio-util.workspace = true
whichlang = "0.1.1"
deunicode = "1.6.2"
scopeguard = "1.2.0"
cron = "0.15"
quick-xml = { version = "0.40.0", features = ["serialize"] }
hickory-resolver = "0.26.0-alpha.1"

View File

@@ -80,11 +80,12 @@ pub struct Account {
pub capabilities: Option<Vec<String>>,
pub date_since: Option<DateSince>,
pub date_before: Option<RelativeDate>,
pub folder_limit: Option<u32>,
pub download_folders: Option<Vec<String>>,
pub account_type: AccountType,
pub download_interval_min: Option<i64>,
pub download_batch_size: Option<u32>,
#[serde(default)]
pub max_email_size_bytes: Option<u64>,
pub known_folders: Option<BTreeSet<String>>,
pub created_at: i64,
pub updated_at: i64,
@@ -95,6 +96,9 @@ pub struct Account {
pub imap_quota_bytes: Option<u64>,
pub imap_quota_window: Option<QuotaWindow>,
pub auto_download_new_mailboxes: Option<bool>,
pub download_schedule: Option<String>,
#[serde(default)]
pub deleting: bool,
}
impl MemDbModel for Account {
@@ -124,15 +128,17 @@ impl Account {
created_at: utc_now!(),
updated_at: utc_now!(),
use_proxy: request.use_proxy,
folder_limit: request.folder_limit,
use_dangerous: request.use_dangerous,
pgp_key: request.pgp_key,
created_by: user_id,
download_batch_size: request.download_batch_size,
max_email_size_bytes: request.max_email_size_bytes,
date_before: request.date_before,
auto_download_new_mailboxes: request.auto_download_new_mailboxes,
imap_quota_bytes: request.imap_quota_bytes,
imap_quota_window: request.imap_quota_window,
download_schedule: request.download_schedule,
deleting: false,
})
}
@@ -220,14 +226,46 @@ impl Account {
pub async fn delete(account_id: u64) -> BichonResult<()> {
let account = Self::get(account_id)?;
if let Err(error) = Self::cleanup_account_resources_sequential(&account).await {
tracing::error!(
"[CLEANUP_ACCOUNT_ERROR] Account {}: failed to cleanup resources: {:#?}",
account_id,
error
);
return Err(error);
// Immediately stop scheduling to prevent new downloads
if matches!(account.account_type, AccountType::IMAP) {
SYNC_TASKS.stop(account.id).await?;
}
// Mark as deleting and disabled so frontend shows status and download tasks skip it
update_impl(
DB_MANAGER.db(),
&account_id.to_string(),
move |current: Account| {
let mut updated = current.clone();
updated.deleting = true;
updated.enabled = false;
Ok(updated)
},
)?;
// Spawn background cleanup — heavy work (Tantivy, attachments) runs off the request path
tokio::spawn(async move {
if let Err(error) = Self::cleanup_account_resources_sequential(&account).await {
tracing::error!(
"[CLEANUP_ACCOUNT_ERROR] Account {}: cleanup failed, reverting deleting flag: {:#?}",
account_id,
error
);
// Revert deleting flag so the user can retry (only if account record still exists)
let _ = update_impl(
DB_MANAGER.db(),
&account_id.to_string(),
move |current: Account| {
let mut updated = current.clone();
updated.deleting = false;
updated.enabled = true;
Ok(updated)
},
);
}
});
Ok(())
}
@@ -236,8 +274,8 @@ impl Account {
}
async fn cleanup_account_resources_sequential(account: &AccountModel) -> BichonResult<()> {
// Sync task already stopped in delete() before spawning this background task
if matches!(account.account_type, AccountType::IMAP) {
SYNC_TASKS.stop(account.id).await?;
DownloadState::delete(account.id)?;
}
OAuth2AccessToken::try_delete(account.id)?;
@@ -365,14 +403,8 @@ impl Account {
}
}
if let Some(folder_limit) = request.folder_limit {
new.folder_limit = Some(folder_limit);
}
if let Some(clear_folder_limit) = request.clear_folder_limit {
if clear_folder_limit {
new.folder_limit = None;
}
if let Some(account_name) = request.account_name {
new.account_name = Some(account_name);
}
if matches!(old.account_type, AccountType::IMAP) {
@@ -401,6 +433,10 @@ impl Account {
new.download_batch_size = Some(*download_batch_size);
}
if let Some(max_email_size_bytes) = request.max_email_size_bytes {
new.max_email_size_bytes = Some(max_email_size_bytes);
}
if let Some(use_proxy) = request.use_proxy {
new.use_proxy = Some(use_proxy);
}
@@ -435,6 +471,12 @@ impl Account {
if let Some(auto_download_new_mailboxes) = request.auto_download_new_mailboxes {
new.auto_download_new_mailboxes = Some(auto_download_new_mailboxes);
}
if let Some(download_schedule) = request.download_schedule {
new.download_schedule = Some(download_schedule);
}
if request.clear_download_schedule == Some(true) {
new.download_schedule = None;
}
new.updated_at = utc_now!();
Ok(new)
}

View File

@@ -16,6 +16,8 @@
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use std::str::FromStr;
use crate::account::entity::ImapConfig;
use crate::account::migration::{AccountModel, AccountType, QuotaWindow};
use crate::account::since::{DateSince, RelativeDate};
@@ -39,8 +41,6 @@ pub struct AccountCreateRequest {
pub date_since: Option<DateSince>,
pub date_before: Option<RelativeDate>,
pub account_type: AccountType,
#[cfg_attr(feature = "web-api", oai(validator(minimum(value = "100"))))]
pub folder_limit: Option<u32>,
#[cfg_attr(feature = "web-api", oai(validator(minimum(value = "10"))))]
pub download_interval_min: Option<i64>,
#[cfg_attr(
@@ -48,12 +48,14 @@ pub struct AccountCreateRequest {
oai(validator(minimum(value = "10"), maximum(value = "200")))
)]
pub download_batch_size: Option<u32>,
pub max_email_size_bytes: Option<u64>,
pub use_proxy: Option<u64>,
pub use_dangerous: bool,
pub pgp_key: Option<String>,
pub imap_quota_bytes: Option<u64>,
pub imap_quota_window: Option<QuotaWindow>,
pub auto_download_new_mailboxes: Option<bool>,
pub download_schedule: Option<String>,
}
impl AccountCreateRequest {
@@ -92,12 +94,15 @@ impl AccountCreateRequest {
))
}
}
if self.download_interval_min.is_none() {
if self.download_interval_min.is_none() && self.download_schedule.is_none() {
return Err(raise_error!(
"`sync_interval_min` is required for IMAP account type".into(),
"`sync_interval_min` or `download_schedule` is required for IMAP account type".into(),
ErrorCode::InvalidParameter
));
}
if let Some(ref schedule) = self.download_schedule {
validate_cron_expression(schedule)?;
}
}
AccountType::NoSync => {}
}
@@ -139,12 +144,6 @@ pub struct AccountUpdateRequest {
pub date_since: Option<DateSince>,
pub date_before: Option<RelativeDate>,
pub clear_date_range: Option<bool>,
/// Max emails to sync for this folder.
/// If not set, sync all emails.
/// otherwise sync up to `n` most recent emails (min 10).
#[cfg_attr(feature = "web-api", oai(validator(minimum(value = "100"))))]
pub folder_limit: Option<u32>,
pub clear_folder_limit: Option<bool>,
/// Configuration for selective folder (mailbox/label) synchronization
///
/// - For IMAP/SMTP accounts:
@@ -167,6 +166,7 @@ pub struct AccountUpdateRequest {
oai(validator(minimum(value = "10"), maximum(value = "200")))
)]
pub download_batch_size: Option<u32>,
pub max_email_size_bytes: Option<u64>,
/// Optional proxy ID for establishing the connection to external APIs (e.g., Gmail, Outlook).
/// - If `None` or not provided, the client will connect directly to the API server.
/// - If `Some(proxy_id)`, the client will use the pre-configured proxy with the given ID for API requests.
@@ -178,6 +178,8 @@ pub struct AccountUpdateRequest {
pub imap_quota_bytes: Option<u64>,
pub imap_quota_window: Option<QuotaWindow>,
pub auto_download_new_mailboxes: Option<bool>,
pub download_schedule: Option<String>,
pub clear_download_schedule: Option<bool>,
}
impl AccountUpdateRequest {
@@ -197,13 +199,6 @@ impl AccountUpdateRequest {
));
}
if self.clear_folder_limit == Some(true) && self.folder_limit.is_some() {
return Err(raise_error!(
"clear_folder_limit cannot be combined with folder_limit".into(),
ErrorCode::InvalidParameter
));
}
if self.clear_date_range == Some(true)
&& (self.date_since.is_some() || self.date_before.is_some())
{
@@ -230,11 +225,36 @@ impl AccountUpdateRequest {
));
}
}
if self.clear_download_schedule == Some(true) && self.download_schedule.is_some() {
return Err(raise_error!(
"clear_download_schedule cannot be combined with download_schedule".into(),
ErrorCode::InvalidParameter
));
}
if let Some(ref schedule) = self.download_schedule {
validate_cron_expression(schedule)?;
}
}
Ok(())
}
}
fn validate_cron_expression(expr: &str) -> BichonResult<()> {
if expr.trim().is_empty() {
return Err(raise_error!(
"download_schedule must not be empty".into(),
ErrorCode::InvalidParameter
));
}
cron::Schedule::from_str(expr).map_err(|e| {
raise_error!(
format!("Invalid cron expression '{}': {}", expr, e),
ErrorCode::InvalidParameter
)
})?;
Ok(())
}
#[derive(Clone, Debug, Default, Eq, PartialEq, Deserialize, Serialize)]
#[cfg_attr(feature = "web-api", derive(poem_openapi::Object))]
@@ -253,3 +273,34 @@ pub fn filter_accessible_accounts<'a>(
.cloned()
.collect()
}
#[cfg(test)]
mod test {
use super::validate_cron_expression;
#[test]
fn valid_cron_expressions() {
assert!(validate_cron_expression("0 0 0 * * *").is_ok()); // daily at midnight
assert!(validate_cron_expression("0 */5 * * * *").is_ok()); // every 5 minutes
assert!(validate_cron_expression("0 0 12 * * 1-5").is_ok()); // weekdays at noon
assert!(validate_cron_expression("0 30 4 1 * *").is_ok()); // 1st of month at 04:30
assert!(validate_cron_expression("0 0 * * * *").is_ok()); // every hour
}
#[test]
fn invalid_cron_expression_too_few_fields() {
assert!(validate_cron_expression("0 0 * *").is_err());
assert!(validate_cron_expression("* * * * *").is_err()); // 5 fields, needs seconds
}
#[test]
fn invalid_cron_expression_empty() {
assert!(validate_cron_expression("").is_err());
assert!(validate_cron_expression(" ").is_err());
}
#[test]
fn invalid_cron_expression_garbage() {
assert!(validate_cron_expression("not a cron").is_err());
}
}

View File

@@ -364,4 +364,43 @@ mod test {
};
assert!(e.validate().is_err());
}
// ── Sliding window tests ──────────────────────────────────────
#[test]
fn relative_date_calculate_returns_valid_format() {
let r = RelativeDate {
unit: Unit::Years,
value: 1,
};
let date_str = r.calculate_date().unwrap();
// Expect format like "26-May-2025"
assert!(date_str.len() > 5);
assert!(date_str.contains('-'));
}
#[test]
fn relative_date_one_year_ago_is_before_now() {
let r = RelativeDate {
unit: Unit::Years,
value: 1,
};
let date_str = r.calculate_date().unwrap();
let parsed = chrono::NaiveDate::parse_from_str(&date_str, "%d-%b-%Y").unwrap();
let today = chrono::Local::now().date_naive();
assert!(parsed < today, "1 year ago ({parsed}) should be before today ({today})");
}
#[test]
fn relative_date_one_day_ago_is_yesterday() {
let r = RelativeDate {
unit: Unit::Days,
value: 1,
};
let date_str = r.calculate_date().unwrap();
let parsed = chrono::NaiveDate::parse_from_str(&date_str, "%d-%b-%Y").unwrap();
let today = chrono::Local::now().date_naive();
let yesterday = today - chrono::Duration::days(1);
assert_eq!(parsed, yesterday, "1 day ago should be yesterday");
}
}

View File

@@ -40,11 +40,11 @@ pub struct AccountResp {
pub capabilities: Option<Vec<String>>,
pub date_since: Option<DateSince>,
pub date_before: Option<RelativeDate>,
pub folder_limit: Option<u32>,
pub download_folders: Option<Vec<String>>,
pub account_type: AccountType,
pub download_interval_min: Option<i64>,
pub download_batch_size: Option<u32>,
pub max_email_size_bytes: Option<u64>,
pub known_folders: Option<BTreeSet<String>>,
pub created_at: i64,
pub updated_at: i64,
@@ -57,6 +57,8 @@ pub struct AccountResp {
pub imap_quota_bytes: Option<u64>,
pub imap_quota_window: Option<QuotaWindow>,
pub auto_download_new_mailboxes: Option<bool>,
pub download_schedule: Option<String>,
pub deleting: bool,
}
impl AccountResp {
@@ -72,11 +74,11 @@ impl AccountResp {
capabilities: account.capabilities,
date_since: account.date_since,
date_before: account.date_before,
folder_limit: account.folder_limit,
download_folders: account.download_folders,
account_type: account.account_type,
download_interval_min: account.download_interval_min,
download_batch_size: account.download_batch_size,
max_email_size_bytes: account.max_email_size_bytes,
known_folders: account.known_folders,
created_at: account.created_at,
updated_at: account.updated_at,
@@ -93,6 +95,8 @@ impl AccountResp {
imap_quota_bytes: account.imap_quota_bytes,
imap_quota_window: account.imap_quota_window,
auto_download_new_mailboxes: account.auto_download_new_mailboxes,
download_schedule: account.download_schedule,
deleting: account.deleting,
}
}
}

View File

@@ -45,6 +45,10 @@ pub struct IncomingServer {
#[serde(rename = "socketType")]
pub socket_type: String,
pub username: String,
/// Authentication method from the XML, e.g. "OAuth2", "password-cleartext",
/// "password-encrypted", "GSSAPI", "NTLM". Absent in DNS SRV fallback.
#[serde(default)]
pub authentication: String,
}
#[derive(Debug, Clone, Default, PartialEq, Eq, Deserialize)]
@@ -137,6 +141,7 @@ async fn lookup_srv(domain: &str) -> Option<MailConfig> {
port: imap_port,
socket_type: "SSL".to_string(),
username: "%EMAILADDRESS%".to_string(),
authentication: String::new(),
}],
outgoing: vec![OutgoingServer {
protocol: "smtp".to_string(),
@@ -153,30 +158,37 @@ async fn lookup_srv(domain: &str) -> Option<MailConfig> {
// ---------------------------------------------------------------------------
/// Discover mail server configuration for a domain using the Thunderbird
/// autoconfig protocol (ISPDB) and DNS SRV fallback.
/// autoconfig protocol (ISPDB), DNS SRV, MX fallback, and finally guessing.
///
/// Probe order:
/// 1. `https://autoconfig.{domain}/mail/config-v1.1.xml`
/// 2. `https://{domain}/.well-known/autoconfig/mail/config-v1.1.xml`
/// 3. DNS SRV records (`_imaps._tcp` / `_submission._tcp`)
/// 4. Thunderbird central ISPDB (`https://autoconfig.thunderbird.net/v1.1/{domain}`)
/// 2. `http://autoconfig.{domain}/mail/config-v1.1.xml`
/// 3. `https://{domain}/.well-known/autoconfig/mail/config-v1.1.xml`
/// 4. `http://{domain}/.well-known/autoconfig/mail/config-v1.1.xml`
/// 5. DNS SRV records (`_imaps._tcp` / `_submission._tcp`)
/// 6. Thunderbird central ISPDB (`https://autoconfig.thunderbird.net/v1.1/{domain}`)
/// 7. MX lookup → ISPDB for MX domain
/// 8. MX lookup → ISP autoconfig for MX domain
/// 9. GuessConfig — probe common hostnames + ports
pub async fn fetch(domain: &str) -> BichonResult<MailConfig> {
let client = Client::builder()
.timeout(std::time::Duration::from_secs(10))
.build()
.map_err(|e| raise_error!(format!("{:#?}", e), ErrorCode::InternalError))?;
// 1. Try autoconfig subdomain
if let Some(config) = fetch_xml(
&client,
&format!("https://autoconfig.{domain}/mail/config-v1.1.xml"),
)
.await
// ── ISP autoconfig (HTTPS, then HTTP) ──────────────────────────
if let Some(config) =
fetch_xml(&client, &format!("https://autoconfig.{domain}/mail/config-v1.1.xml")).await
{
return Ok(config);
}
if let Some(config) =
fetch_xml(&client, &format!("http://autoconfig.{domain}/mail/config-v1.1.xml")).await
{
return Ok(config);
}
// 2. Try well-known path
// ── Well-known path (HTTPS, then HTTP) ─────────────────────────
if let Some(config) = fetch_xml(
&client,
&format!("https://{domain}/.well-known/autoconfig/mail/config-v1.1.xml"),
@@ -185,28 +197,100 @@ pub async fn fetch(domain: &str) -> BichonResult<MailConfig> {
{
return Ok(config);
}
if let Some(config) = fetch_xml(
&client,
&format!("http://{domain}/.well-known/autoconfig/mail/config-v1.1.xml"),
)
.await
{
return Ok(config);
}
// 3. Try DNS SRV records
// ── DNS SRV records ────────────────────────────────────────────
if let Some(config) = lookup_srv(domain).await {
return Ok(config);
}
// 4. Fall back to Thunderbird central database
if let Some(config) = fetch_xml(
&client,
&format!("https://autoconfig.thunderbird.net/v1.1/{domain}"),
)
.await
// ── Thunderbird central ISPDB ──────────────────────────────────
if let Some(config) =
fetch_xml(&client, &format!("https://autoconfig.thunderbird.net/v1.1/{domain}")).await
{
return Ok(config);
}
// ── MX fallback ────────────────────────────────────────────────
if let Some(config) = fetch_for_mx(&client, domain).await {
return Ok(config);
}
// ── GuessConfig ────────────────────────────────────────────────
if let Some(config) = crate::autoconfig::guess::guess_config(domain).await {
return Ok(config);
}
Err(raise_error!(
format!("No autoconfig found for domain: {domain}"),
ErrorCode::InternalError
))
}
/// DNS MX lookup → retry ISPDB and ISP autoconfig for the MX domain.
///
/// Many self-hosted domains have their MX pointed at Google, Microsoft, etc.
/// The MX domain's ISPDB entry covers the original domain.
async fn fetch_for_mx(client: &Client, domain: &str) -> Option<MailConfig> {
let mx_domain = lookup_mx_domain(domain).await?;
if mx_domain == domain.to_ascii_lowercase() {
return None; // same domain, already tried above
}
// Try ISPDB for the MX domain
if let Some(config) =
fetch_xml(client, &format!("https://autoconfig.thunderbird.net/v1.1/{mx_domain}")).await
{
return Some(config);
}
// Try ISP autoconfig for the MX domain (HTTPS then HTTP)
if let Some(config) =
fetch_xml(client, &format!("https://autoconfig.{mx_domain}/mail/config-v1.1.xml")).await
{
return Some(config);
}
if let Some(config) =
fetch_xml(client, &format!("http://autoconfig.{mx_domain}/mail/config-v1.1.xml")).await
{
return Some(config);
}
None
}
/// DNS MX lookup → extract the second-level domain of the first MX hostname.
async fn lookup_mx_domain(domain: &str) -> Option<String> {
let resolver = TokioResolver::builder(TokioConnectionProvider::default())
.ok()?
.build();
let lookup = resolver.mx_lookup(domain).await.ok()?;
let record = lookup.iter().next()?;
let mx_host = record.to_string().trim_end_matches('.').to_string();
// Extract a reasonable base domain from the MX hostname.
// E.g., "aspmx.l.google.com" → "google.com"
// "company.mail.protection.outlook.com" → "outlook.com"
extract_base_domain(&mx_host)
}
/// Extract the top two labels from a hostname as a rough base domain.
fn extract_base_domain(host: &str) -> Option<String> {
let parts: Vec<&str> = host.split('.').collect();
if parts.len() >= 2 {
Some(parts[parts.len() - 2..].join("."))
} else {
None
}
}
#[cfg(test)]
mod tests {
use super::*;

View File

@@ -0,0 +1,105 @@
//
// Copyright (c) 2025-2026 rustmailer.com (https://rustmailer.com)
//
// This file is part of the Bichon Email Archiving Project
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use crate::account::entity::Encryption;
use crate::autoconfig::client::{IncomingServer, MailConfig};
use crate::imap::client::Client;
use tracing::{debug, info};
/// A single host:port:encryption combination to probe.
struct Guess {
hostname: String,
port: u16,
encryption: Encryption,
socket_type: &'static str,
}
/// Generate candidates in the same order Thunderbird uses:
/// 1. imap.{domain} — most common
/// 2. mail.{domain} — fallback
/// 3. {domain} — bare domain (rare)
fn make_guesses(domain: &str) -> Vec<Guess> {
let hosts = [
format!("imap.{domain}"),
format!("mail.{domain}"),
domain.to_string(),
];
let mut guesses = Vec::with_capacity(hosts.len() * 2);
for host in &hosts {
guesses.push(Guess {
hostname: host.clone(),
port: 993,
encryption: Encryption::Ssl,
socket_type: "SSL",
});
guesses.push(Guess {
hostname: host.clone(),
port: 143,
encryption: Encryption::StartTls,
socket_type: "STARTTLS",
});
}
guesses
}
/// Try to open a connection, read the IMAP banner, and close.
/// Returns `true` if the server responds with an IMAP greeting.
async fn probe(hostname: &str, port: u16, encryption: &Encryption) -> bool {
match Client::connection(hostname, encryption, port, None, true).await {
Ok(_) => {
debug!("GuessConfig probe succeeded: {hostname}:{port} ({encryption:?})");
true
}
Err(e) => {
debug!("GuessConfig probe failed for {hostname}:{port}: {e:?}");
false
}
}
}
/// Thunderbird-style guessing: try common hostnames and ports, probing
/// each with a real TCP connection.
///
/// Returns the first working `MailConfig`, or `None` if nothing works.
pub async fn guess_config(domain: &str) -> Option<MailConfig> {
let guesses = make_guesses(domain);
info!("GuessConfig: trying {} candidates for {domain}", guesses.len());
for g in &guesses {
if probe(&g.hostname, g.port, &g.encryption).await {
info!(
"GuessConfig: found working IMAP at {}:{} ({})",
g.hostname, g.port, g.socket_type
);
return Some(MailConfig {
incoming: vec![IncomingServer {
protocol: "imap".to_string(),
hostname: g.hostname.clone(),
port: g.port,
socket_type: g.socket_type.to_string(),
username: "%EMAILADDRESS%".to_string(),
authentication: String::new(),
}],
outgoing: vec![],
});
}
}
None
}

View File

@@ -19,6 +19,7 @@
use crate::account::entity::Encryption;
use crate::autoconfig::client::{self, MailConfig};
use crate::autoconfig::entity::{MailServerConfig, ServerConfig};
use crate::autoconfig::oauth2_providers::lookup_oauth2;
use crate::autoconfig::CachedMailSettings;
use crate::error::code::ErrorCode;
use crate::error::BichonResult;
@@ -54,9 +55,17 @@ pub(crate) fn mail_config_to_server_config(config: &MailConfig) -> Option<MailSe
}
};
// Detect OAuth2 support: the XML <authentication> field and a known
// hostname → issuer mapping determine whether the provider supports OAuth2.
let oauth2 = if imap.authentication.eq_ignore_ascii_case("OAuth2") {
lookup_oauth2(&imap.hostname)
} else {
None
};
Some(MailServerConfig {
imap: ServerConfig::new(imap.hostname.clone(), port, encryption),
oauth2: None,
oauth2,
})
}

View File

@@ -24,7 +24,9 @@ use serde::{Deserialize, Serialize};
pub mod client;
pub mod entity;
pub mod guess;
pub mod load;
mod oauth2_providers;
#[cfg(test)]
mod tests;

View File

@@ -0,0 +1,151 @@
//
// Copyright (c) 2025-2026 rustmailer.com (https://rustmailer.com)
//
// This file is part of the Bichon Email Archiving Project
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use crate::autoconfig::entity::OAuth2Config;
/// Per-provider OAuth2 metadata, mirroring Thunderbird's `OAuth2Providers.sys.mjs`.
///
/// Each entry maps one or more IMAP hostname suffixes to a well-known OIDC issuer
/// and the IMAP-specific OAuth2 scopes.
struct Provider {
/// Suffixes matched case-insensitively against the end of the IMAP hostname.
host_suffixes: &'static [&'static str],
/// The OIDC issuer URL used by the provider.
issuer: &'static str,
/// OAuth2 scope(s) required for IMAP access.
scopes: &'static [&'static str],
}
const PROVIDERS: &[Provider] = &[
// Google
Provider {
host_suffixes: &["imap.gmail.com", ".gmail.com", ".googlemail.com"],
issuer: "https://accounts.google.com",
scopes: &["https://mail.google.com/"],
},
// Microsoft (Outlook / Office 365 / Hotmail / Live)
Provider {
host_suffixes: &[
"outlook.office365.com",
".outlook.com",
".hotmail.com",
".live.com",
".office365.com",
],
issuer: "https://login.microsoftonline.com/common/v2.0",
scopes: &[
"https://outlook.office365.com/IMAP.AccessAsUser.All",
"offline_access",
],
},
// Yahoo / AOL / ATT / Verizon
Provider {
host_suffixes: &[
"imap.mail.yahoo.com",
".yahoo.com",
".yahoodns.net",
".aol.com",
"imap.aol.com",
],
issuer: "https://login.yahoo.com",
scopes: &["mail-w"],
},
// Yandex
Provider {
host_suffixes: &["imap.yandex.ru", "imap.yandex.com", ".yandex.ru"],
issuer: "https://oauth.yandex.com",
scopes: &["imap:all"],
},
// Mail.ru
Provider {
host_suffixes: &["imap.mail.ru", ".mail.ru", ".bk.ru", ".list.ru", ".inbox.ru"],
issuer: "https://o2.mail.ru",
scopes: &["imap"],
},
// Fastmail
Provider {
host_suffixes: &["imap.fastmail.com", ".fastmail.com"],
issuer: "https://www.fastmail.com",
scopes: &[
"https://www.fastmail.com/dev/imap",
"offline_access",
],
},
// Comcast
Provider {
host_suffixes: &["imap.comcast.net", ".comcast.net"],
issuer: "https://oauth.xfinity.com",
scopes: &["https://email.comcast.net/"],
},
];
/// Try to find an OAuth2 provider that matches the given IMAP hostname.
///
/// Matching is case-insensitive and done by suffix: a hostname "imap.gmail.com"
/// matches the suffix ".gmail.com".
pub fn lookup_oauth2(hostname: &str) -> Option<OAuth2Config> {
let host = hostname.to_ascii_lowercase();
for provider in PROVIDERS {
if provider
.host_suffixes
.iter()
.any(|suffix| host.ends_with(&suffix.to_ascii_lowercase()))
{
return Some(OAuth2Config {
issuer: provider.issuer.to_string(),
scope: provider.scopes.iter().map(|s| s.to_string()).collect(),
auth_url: String::new(),
token_url: String::new(),
});
}
}
None
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_known_providers() {
let cases = [
("imap.gmail.com", Some("https://accounts.google.com")),
("imap.gmail.com", Some("https://accounts.google.com")),
("outlook.office365.com", Some("https://login.microsoftonline.com/common/v2.0")),
("imap.mail.yahoo.com", Some("https://login.yahoo.com")),
("imap.aol.com", Some("https://login.yahoo.com")),
("imap.yandex.ru", Some("https://oauth.yandex.com")),
("imap.mail.ru", Some("https://o2.mail.ru")),
("imap.fastmail.com", Some("https://www.fastmail.com")),
("imap.comcast.net", Some("https://oauth.xfinity.com")),
];
for (hostname, expected_issuer) in &cases {
let result = lookup_oauth2(hostname);
assert_eq!(
result.map(|c| c.issuer),
expected_issuer.map(|s| s.to_string()),
"failed for hostname: {hostname}"
);
}
}
#[test]
fn test_unknown_provider() {
assert!(lookup_oauth2("mail.my-company.example").is_none());
}
}

View File

@@ -195,6 +195,7 @@ fn make_imap_server(host: &str, port: u16, socket_type: &str) -> IncomingServer
port,
socket_type: socket_type.to_string(),
username: "%EMAILADDRESS%".to_string(),
authentication: String::new(),
}
}
@@ -241,6 +242,7 @@ fn convert_no_imap_only_pop3() {
port: 995,
socket_type: "SSL".to_string(),
username: "%EMAILADDRESS%".to_string(),
authentication: String::new(),
}],
outgoing: vec![],
};
@@ -266,6 +268,7 @@ fn convert_picks_imap_over_pop3() {
port: 995,
socket_type: "SSL".to_string(),
username: "%EMAILADDRESS%".to_string(),
authentication: String::new(),
},
make_imap_server("imap.example.com", 993, "SSL"),
],
@@ -284,6 +287,7 @@ fn convert_imaps_protocol_variant() {
port: 993,
socket_type: "SSL".to_string(),
username: "%EMAILADDRESS%".to_string(),
authentication: String::new(),
}],
outgoing: vec![],
};
@@ -300,9 +304,65 @@ fn convert_case_insensitive_protocol() {
port: 143,
socket_type: "STARTTLS".to_string(),
username: "%EMAILADDRESS%".to_string(),
authentication: String::new(),
}],
outgoing: vec![],
};
let result = mail_config_to_server_config(&config).expect("should recognize 'IMAP'");
assert_eq!(result.imap.host, "imap.example.com");
}
#[test]
fn convert_gmail_oauth2() {
let config = MailConfig {
incoming: vec![IncomingServer {
protocol: "imap".to_string(),
hostname: "imap.gmail.com".to_string(),
port: 993,
socket_type: "SSL".to_string(),
username: "%EMAILADDRESS%".to_string(),
authentication: "OAuth2".to_string(),
}],
outgoing: vec![],
};
let result = mail_config_to_server_config(&config).expect("should convert");
let oauth2 = result.oauth2.expect("Gmail should have OAuth2");
assert_eq!(oauth2.issuer, "https://accounts.google.com");
assert!(oauth2.scope.contains(&"https://mail.google.com/".to_string()));
}
#[test]
fn convert_outlook_oauth2() {
let config = MailConfig {
incoming: vec![IncomingServer {
protocol: "imap".to_string(),
hostname: "outlook.office365.com".to_string(),
port: 993,
socket_type: "SSL".to_string(),
username: "%EMAILADDRESS%".to_string(),
authentication: "OAuth2".to_string(),
}],
outgoing: vec![],
};
let result = mail_config_to_server_config(&config).expect("should convert");
let oauth2 = result.oauth2.expect("Outlook should have OAuth2");
assert!(oauth2.issuer.contains("microsoftonline"));
}
#[test]
fn convert_unknown_host_no_oauth2() {
// OAuth2 auth flag on an unknown hostname → no OAuth2 returned
let config = MailConfig {
incoming: vec![IncomingServer {
protocol: "imap".to_string(),
hostname: "mail.random-isp.example".to_string(),
port: 993,
socket_type: "SSL".to_string(),
username: "%EMAILADDRESS%".to_string(),
authentication: "OAuth2".to_string(),
}],
outgoing: vec![],
};
let result = mail_config_to_server_config(&config).expect("should convert");
assert!(result.oauth2.is_none(), "unknown hostname → no OAuth2 mapping");
}

View File

@@ -16,6 +16,11 @@
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use std::str::FromStr;
use chrono::{DateTime, Local, TimeZone, Utc};
use cron::Schedule;
use crate::{
utc_now,
{
@@ -48,11 +53,20 @@ pub async fn decide_next_download_task(
let should_start = match trigger_type {
TriggerType::Manual => true,
TriggerType::Scheduled => should_trigger_next_download(
state.last_trigger_at,
state.last_finished_at.unwrap_or(0),
account.download_interval_min.unwrap_or(60),
),
TriggerType::Scheduled => {
let now = utc_now!();
let cooldown_ok = now - state.last_finished_at.unwrap_or(0) > 60 * 1000;
if !cooldown_ok {
false
} else if let Some(ref schedule) = account.download_schedule {
should_trigger_scheduled(schedule, state.last_trigger_at)
} else {
should_trigger_next_download(
state.last_trigger_at,
account.download_interval_min.unwrap_or(60),
)
}
}
};
if should_start {
@@ -63,11 +77,80 @@ pub async fn decide_next_download_task(
}
}
fn should_trigger_next_download(
last_trigger_at: i64,
last_finished_at: i64,
sync_interval_min: i64,
) -> bool {
fn should_trigger_next_download(last_trigger_at: i64, sync_interval_min: i64) -> bool {
let now = utc_now!();
now - last_trigger_at > (sync_interval_min * 60 * 1000) && now - last_finished_at > 60 * 1000
now - last_trigger_at > (sync_interval_min * 60 * 1000)
}
fn should_trigger_scheduled(schedule_str: &str, last_trigger_at: i64) -> bool {
let schedule = match Schedule::from_str(schedule_str) {
Ok(s) => s,
Err(e) => {
tracing::warn!(
"Invalid cron expression '{}', falling back to no trigger: {}",
schedule_str,
e
);
return false;
}
};
// last_trigger_at is a UTC millis timestamp; convert to server local time
let last_utc = match Utc.timestamp_millis_opt(last_trigger_at) {
chrono::LocalResult::Single(dt) => dt,
_ => {
tracing::warn!("Invalid last_trigger_at timestamp: {}", last_trigger_at);
return false;
}
};
let last_dt: DateTime<Local> = last_utc.with_timezone(&Local);
let now = Local::now();
schedule
.after(&last_dt)
.next()
.map_or(false, |next| next <= now)
}
#[cfg(test)]
mod test {
use super::*;
#[test]
fn cron_every_minute_triggers_after_60s() {
// "0 * * * * *" = every minute at second 0. last_trigger 90s ago → should trigger
let now = Local::now();
let last_trigger = now.timestamp_millis() - 90_000;
assert!(should_trigger_scheduled("0 * * * * *", last_trigger));
}
#[test]
fn cron_daily_midnight_triggers_when_missed() {
// "0 0 0 * * *" = daily at midnight
// last_trigger was 25 hours ago → should trigger (we missed midnight)
let now = Local::now();
let last_trigger = now.timestamp_millis() - 25 * 60 * 60 * 1000;
assert!(should_trigger_scheduled("0 0 0 * * *", last_trigger));
}
#[test]
fn cron_daily_midnight_no_trigger_if_already_fired() {
// "0 0 0 * * *" = daily at midnight
// last_trigger was 1 minute ago → should NOT trigger
let now = Local::now();
let last_trigger = now.timestamp_millis() - 60_000;
assert!(!should_trigger_scheduled("0 0 0 * * *", last_trigger));
}
#[test]
fn invalid_cron_returns_false() {
assert!(!should_trigger_scheduled("invalid cron expression", 0));
}
#[test]
fn cron_every_hour_triggers() {
// "0 0 * * * *" = every hour at minute 0, second 0
// last_trigger was 61 minutes ago → should trigger
let now = Local::now();
let last_trigger = now.timestamp_millis() - 61 * 60 * 1000;
assert!(should_trigger_scheduled("0 0 * * * *", last_trigger));
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -17,24 +17,20 @@
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use crate::{
raise_error,
{
account::{
migration::AccountModel,
state::{DownloadState, DownloadStatus, FolderStatus},
},
cache::{
imap::{
download::flow::{
fetch_and_save_by_date, fetch_and_save_full_mailbox, FetchDirection,
},
mailbox::MailBox,
},
SEMAPHORE,
},
error::{code::ErrorCode, BichonResult},
store::tantivy::envelope::ENVELOPE_MANAGER,
account::{
migration::AccountModel,
state::{DownloadState, DownloadStatus, FolderStatus},
},
cache::{
imap::{
download::flow::{fetch_and_save_by_date, fetch_and_save_full_mailbox, FetchDirection},
mailbox::MailBox,
},
SEMAPHORE,
},
error::{code::ErrorCode, BichonResult},
raise_error,
store::tantivy::{attachment::ATTACHMENT_MANAGER, envelope::ENVELOPE_MANAGER},
};
use tokio_util::sync::CancellationToken;
@@ -91,9 +87,13 @@ pub async fn rebuild_cache(
continue;
}
};
match fetch_and_save_full_mailbox(&account, &mailbox, token.clone()).await {
Ok(_) => {}
Ok(new_highest_uid) => {
let mut updated = mailbox.clone();
updated.highest_uid = new_highest_uid;
MailBox::batch_upsert(&[updated])?;
}
Err(err) => {
has_error = true;
tracing::error!("Folder sync task failed: {:#?}", err);
@@ -173,7 +173,11 @@ pub async fn rebuild_cache_by_date(
match fetch_and_save_by_date(&account, date.as_str(), &mailbox, direction, token.clone())
.await
{
Ok(_) => {}
Ok(new_highest_uid) => {
let mut updated = mailbox.clone();
updated.highest_uid = new_highest_uid;
MailBox::batch_upsert(&[updated])?;
}
Err(err) => {
has_error = true;
tracing::error!("Folder sync task failed: {:#?}", err);
@@ -200,11 +204,13 @@ pub async fn rebuild_mailbox_cache(
local_mailbox: &MailBox,
remote_mailbox: &MailBox,
token: CancellationToken,
) -> BichonResult<()> {
) -> BichonResult<Option<u32>> {
ENVELOPE_MANAGER
.delete_mailbox_envelopes(account.id, vec![local_mailbox.id])
.await?;
ATTACHMENT_MANAGER
.delete_mailbox_attachments(account.id, vec![local_mailbox.id])
.await?;
if remote_mailbox.exists == 0 {
info!(
"Account {}: Mailbox '{}' has no emails on the remote server. The mailbox is empty, no envelopes to fetch.",
@@ -219,11 +225,11 @@ pub async fn rebuild_mailbox_cache(
FolderStatus::Success,
None,
)?;
return Ok(());
return Ok(None);
}
fetch_and_save_full_mailbox(account, remote_mailbox, token).await?;
Ok(())
let result = fetch_and_save_full_mailbox(account, remote_mailbox, token).await?;
Ok(result)
}
pub async fn rebuild_mailbox_cache_by_date(
@@ -233,10 +239,13 @@ pub async fn rebuild_mailbox_cache_by_date(
remote: &MailBox,
direction: FetchDirection,
token: CancellationToken,
) -> BichonResult<()> {
) -> BichonResult<Option<u32>> {
ENVELOPE_MANAGER
.delete_mailbox_envelopes(account.id, vec![local_mailbox_id])
.await?;
ATTACHMENT_MANAGER
.delete_mailbox_attachments(account.id, vec![local_mailbox_id])
.await?;
if remote.exists == 0 {
info!(
"Account {}: Mailbox '{}' has no emails on the remote server. The mailbox is empty, no envelopes to fetch.",
@@ -251,9 +260,9 @@ pub async fn rebuild_mailbox_cache_by_date(
FolderStatus::Success,
None,
)?;
return Ok(());
return Ok(None);
}
fetch_and_save_by_date(account, date, remote, direction, token).await?;
Ok(())
let result = fetch_and_save_by_date(account, date, remote, direction, token).await?;
Ok(result)
}

View File

@@ -56,6 +56,10 @@ pub struct MailBox {
/// The validity identifier for UIDs in this mailbox, used to ensure UID consistency across sessions.
/// If `None`, the IMAP server has not provided this information.
pub uid_validity: Option<u32>,
/// The highest UID that has been successfully downloaded and stored locally.
/// Used for incremental sync: next fetch starts from `highest_uid + 1`.
/// If `None`, a fallback query against the Tantivy index will be performed once.
pub highest_uid: Option<u32>,
}
impl MemDbModel for MailBox {

View File

@@ -113,6 +113,9 @@ impl AccountDownTask {
let account = AccountModel::get(account_id).ok();
match account {
Some(account) => {
if account.deleting {
return Ok(());
}
if !account.enabled {
let last = LAST_WARN_TIME.load(Ordering::Relaxed);
let now = utc_now!();
@@ -246,6 +249,10 @@ impl AccountDownTask {
}
};
if account.deleting {
return;
}
if let Err(e) = process_imap_download(&account, token_clone, TriggerType::Manual).await
{
error!("Manual download failed for {}: {:?}", account_id, e);

View File

@@ -32,12 +32,16 @@ impl DatabaseManager {
let db_path = &DATA_DIR_MANAGER.memdb_dir;
std::fs::create_dir_all(db_path).expect("Failed to create memdb data directory");
let db = MemDb::open_with(db_path, Durability::Full)
let db = MemDb::open_with(db_path, Durability::Batch { max_ops: 100 })
.expect("Failed to open memdb database");
// Start periodic snapshot worker (every 5 minutes)
db.start_snapshot_worker(Duration::from_secs(300));
// Start periodic flush worker (every 10 seconds) so buffered writes
// are flushed regularly and not only at the batch threshold.
db.start_flush_worker(Duration::from_secs(10));
DatabaseManager { db }
}
@@ -45,4 +49,12 @@ impl DatabaseManager {
pub fn db(&self) -> &MemDb {
&self.db
}
/// Flush any buffered WAL entries to disk. Must be called before shutdown
/// to avoid losing writes that haven't hit the batch threshold yet.
pub fn flush(&self) {
if let Err(e) = self.db.flush() {
eprintln!("[memdb] flush error on shutdown: {e}");
}
}
}

View File

@@ -16,14 +16,17 @@
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use crate::cache::imap::mailbox::MailBox;
use crate::common::AddrVec;
use crate::envelope::meta::parse_bichon_metadata;
use crate::envelope::utils::normalize_subject;
use crate::error::code::ErrorCode;
use crate::error::BichonResult;
use crate::imap::executor::ImapExecutor;
use crate::message::content::AttachmentInfo;
use crate::store::blob::{DetachedEmail, BLOB_MANAGER};
use crate::store::tantivy::attachment::ATTACHMENT_MANAGER;
use crate::store::tantivy::dedup_cache::DEDUP_CACHE;
use crate::store::tantivy::envelope::ENVELOPE_MANAGER;
use crate::store::tantivy::model::{AttachmentModel, EnvelopeWithAttachments};
use crate::utils::html::extract_text;
@@ -48,9 +51,17 @@ pub async fn extract_envelope_and_store_it(
.map(|d| d.timestamp_millis())
.unwrap_or(0);
let uid = fetch.uid.unwrap_or(0);
let body = fetch
.body()
.ok_or_else(|| raise_error!("No body available".into(), ErrorCode::InternalError))?;
let body = match fetch.body() {
Some(b) => b,
None => {
tracing::warn!(
account_id,
uid = fetch.uid,
"FETCH response has no body, skipping message"
);
return Ok(());
}
};
let size = fetch.size.unwrap_or(body.len() as u32);
extract_envelope_core(body, uid, size, internal_date, account_id, mailbox_id).await
}
@@ -87,7 +98,13 @@ async fn extract_envelope_core(
account_id: u64,
mailbox_id: u64,
) -> BichonResult<()> {
//The content hash of the original raw EML
let email_content_hash = compute_content_hash(body);
if DEDUP_CACHE.contains(account_id, mailbox_id, &email_content_hash) {
tracing::debug!("Duplicate email detected");
//println!("Duplicate email detected");
return Ok(());
}
let message: Message<'_> = MessageParser::new().parse(body).ok_or_else(|| {
raise_error!(
"Email header parse result is not available".into(),
@@ -194,33 +211,37 @@ async fn extract_envelope_core(
let attachment_docs: Vec<TantivyDocument> = attachments
.iter()
.filter(|a| !a.inline || a.content_id.is_none())
.map(|a| AttachmentModel {
id: Uuid::new_v4().to_string(),
envelope_id: envelope_id.clone(),
account_id,
account_email: None,
mailbox_id,
mailbox_name: None,
subject: subject.clone(),
content_hash: a.content_hash.clone(),
from: from.clone(),
date,
ingest_at: now,
size: a.size as u64,
ext: a.get_extension(),
category: a.get_category().to_string(),
content_type: a.file_type.clone(),
shard_id: 0,
text: None,
has_text: false,
is_ocr: false,
page_count: None,
is_indexed: false,
is_message: a.is_message,
name: a.filename.clone(),
tags: None,
auto_tags: None,
}).map(|a|a.into_document())
.map(|a| {
let has_text = a.extracted_text.is_some();
AttachmentModel {
id: Uuid::new_v4().to_string(),
envelope_id: envelope_id.clone(),
account_id,
account_email: None,
mailbox_id,
mailbox_name: None,
subject: subject.clone(),
content_hash: a.content_hash.clone(),
from: from.clone(),
date,
ingest_at: now,
size: a.size as u64,
ext: a.get_extension(),
category: a.get_category().to_string(),
content_type: a.file_type.clone(),
shard_id: 0,
text: a.extracted_text.clone(),
has_text,
is_ocr: a.extracted_is_ocr,
page_count: a.extracted_page_count.map(|n| n as u64),
is_indexed: has_text,
is_message: a.is_message,
name: a.filename.clone(),
tags: None,
auto_tags: None,
}
})
.map(|a| a.into_document())
.collect();
let envelope = Envelope {
@@ -245,7 +266,7 @@ async fn extract_envelope_core(
tags: (!final_tags.is_empty()).then_some(final_tags),
account_email: None,
mailbox_name: None,
content_hash: email_content_hash,
content_hash: email_content_hash.clone(),
};
// 'attachments' contains both regular and inline attachments
let ea = EnvelopeWithAttachments {
@@ -253,7 +274,16 @@ async fn extract_envelope_core(
attachments: Some(attachments),
};
let doc = ea.to_document(&body_text, 0)?;
tracing::debug!(
"[account {}][mailbox {}] extract: uid={} msg_id={} content_hash={}",
account_id,
mailbox_id,
uid,
&ea.envelope.message_id,
&ea.envelope.content_hash,
);
ENVELOPE_MANAGER.queue(doc).await;
DEDUP_CACHE.insert(account_id, mailbox_id, &email_content_hash);
for doc in attachment_docs {
ATTACHMENT_MANAGER.queue(doc).await;
}
@@ -386,42 +416,125 @@ pub async fn detach_and_store_attachments(
ranges.sort_by(|a, b| b.0.cmp(&a.0));
let mut attachments = Vec::with_capacity(ranges.len());
// Collect candidates for text extraction (non-inline, known document types).
struct TextCandidate {
content_hash: String,
file_type: String,
ext: String,
bytes: Vec<u8>,
}
let mut text_candidates: Vec<TextCandidate> = Vec::new();
for (raw_start, raw_end, att) in ranges {
// Step 2: Extract raw bytes and store them as standalone documents
let raw_bytes = &original_body[raw_start..raw_end];
// mail-parser may report attachment offsets past the body end for
// malformed messages; clamp the range to avoid a slice panic.
let body_len = original_body.len();
let raw_start = raw_start.min(body_len);
let raw_end = raw_end.min(body_len);
let range_valid = raw_start < raw_end;
// content hash is computed from the decoded attachment contents,
// which is always available regardless of raw offset validity.
let content_hash = compute_content_hash(att.contents());
attachments.push((content_hash.clone(), Bytes::copy_from_slice(raw_bytes)));//
if range_valid {
let raw_bytes = &original_body[raw_start..raw_end];
// The actual content stored in the blob is the raw undecoded data.
attachments.push((content_hash.clone(), Bytes::copy_from_slice(raw_bytes)));
// Step 3: Replace raw attachment content with a hash-based placeholder
let placeholder = format!("<<BICHON_DETACH_HASH:{}>>", &content_hash);
let p_bytes = placeholder.as_bytes();
stripped_eml.splice(raw_start..raw_end, p_bytes.iter().cloned());
// Replace raw attachment content with a hash-based placeholder
let placeholder = format!("<<BICHON_DETACH_HASH:{}>>", &content_hash);
stripped_eml.splice(raw_start..raw_end, placeholder.as_bytes().iter().cloned());
} else {
// Invalid range: store a zero-length blob so the consistency
// check passes; reattachment will log a warning for the missing
// blob data but won't panic.
attachments.push((content_hash.clone(), Bytes::new()));
}
let inline = att
.content_disposition()
.map(|d| d.is_inline())
.unwrap_or_else(|| att.content_id().is_some());
let file_type = att
.content_type()
.map(|ct| {
format!(
"{}/{}",
ct.c_type.as_ref(),
ct.c_subtype.as_deref().unwrap_or("")
)
})
.unwrap_or_else(|| "application/octet-stream".to_string());
let has_cid = att.content_id().is_some();
let ext = att
.attachment_name()
.and_then(|n| {
std::path::Path::new(&n)
.extension()
.and_then(|e| e.to_str())
.map(|s| s.to_ascii_lowercase())
})
.unwrap_or_default();
if !inline || !has_cid {
let decoded_len = att.contents().len();
if decoded_len <= crate::ext::text_extractor::MAX_EXTRACT_BYTES
&& crate::ext::text_extractor::should_try_extract(&file_type, &ext)
{
text_candidates.push(TextCandidate {
content_hash: content_hash.clone(),
file_type: file_type.clone(),
ext: ext.clone(),
bytes: att.contents().to_vec(),
});
}
}
let info = AttachmentInfo {
filename: att.attachment_name().map(|n| n.to_string()),
size: att.contents().len(),
inline: att
.content_disposition()
.map(|d| d.is_inline())
.unwrap_or(false),
file_type: att
.content_type()
.map(|ct| {
format!(
"{}/{}",
ct.c_type.as_ref(),
ct.c_subtype.as_deref().unwrap_or("")
)
})
.unwrap_or_else(|| "application/octet-stream".to_string()),
inline,
file_type,
content_id: att.content_id().map(|id| id.to_string()),
content_hash: content_hash.clone(),
is_message: att.is_message(),
extracted_text: None,
extracted_page_count: None,
extracted_is_ocr: false,
};
attachment_infos.push(info);
}
// Run text extraction in a single spawn_blocking batch.
if !text_candidates.is_empty() {
if let Ok(mut extracted_map) = tokio::task::spawn_blocking(move || {
let mut map: std::collections::HashMap<
String,
(String, Option<u32>, bool),
> = std::collections::HashMap::new();
for c in text_candidates {
if let Some(r) =
crate::ext::text_extractor::extract_text(&c.file_type, &c.ext, &c.bytes)
{
map.insert(c.content_hash, (r.text, r.page_count, r.is_ocr));
}
}
map
})
.await
{
for info in &mut attachment_infos {
if let Some((text, pages, is_ocr)) = extracted_map.remove(&info.content_hash) {
info.extracted_text = Some(text);
info.extracted_page_count = pages;
info.extracted_is_ocr = is_ocr;
}
}
}
}
// Step 4: Store the final stripped EML content
BLOB_MANAGER
.queue(DetachedEmail {
@@ -515,6 +628,114 @@ pub fn reattach_eml_content(
Ok((e.envelope, Bytes::from(restored_eml)))
}
/// Returns the raw EML for an indexed message, self-healing a missing content blob.
///
/// Behaves like [`reattach_eml_content`], but when the message's content blob is
/// absent from the blob store it fetches that single message on demand from the
/// IMAP server (`UID FETCH <uid> (BODY.PEEK[])`), persists it for future requests,
/// and returns it. If the on-demand fetch itself fails, the original "content not
/// found" error from [`reattach_eml_content`] is surfaced unchanged so the caller
/// still produces its 404.
pub async fn reattach_eml_content_self_healing(
account_id: u64,
envelope_id: String,
) -> BichonResult<(Envelope, Bytes)> {
let envelope = ENVELOPE_MANAGER
.get_envelope_by_id(account_id, &envelope_id)?
.ok_or_else(|| {
raise_error!(
format!(
"Envelope not found: account_id={} envelope_id={}",
account_id, &envelope_id
),
ErrorCode::ResourceNotFound
)
})?
.envelope;
// Fast path: the content blob is present, reuse the regular reattach logic.
if BLOB_MANAGER.get_email(&envelope.content_hash)?.is_some() {
return reattach_eml_content(account_id, envelope_id);
}
// The blob is missing. Try to recover it directly from the IMAP server.
match recover_message_blob(&envelope).await {
Ok(raw_body) => {
tracing::info!(
account_id,
envelope_id = %envelope_id,
uid = envelope.uid,
"Self-healed missing email content blob via on-demand IMAP fetch"
);
Ok((envelope, raw_body))
}
Err(e) => {
tracing::warn!(
account_id,
envelope_id = %envelope_id,
uid = envelope.uid,
error = %e,
"On-demand IMAP fetch for missing content blob failed; returning not-found"
);
Err(e)
}
}
}
/// Fetches one message from IMAP and re-stores its detached blob.
///
/// On success the freshly fetched raw RFC822 body is returned; it is also queued
/// (in detached form) into the blob store so subsequent requests hit the cache.
/// Fails if the message cannot be fetched, or if the fetched bytes do not match
/// the archived `content_hash` (the server-side message no longer matches what
/// Bichon archived, so it cannot be treated as a recovery of that blob).
async fn recover_message_blob(envelope: &Envelope) -> BichonResult<Bytes> {
let mailbox = MailBox::find_mailbox(envelope.account_id, envelope.mailbox_id)?
.ok_or_else(|| {
raise_error!(
format!(
"Mailbox not found: account_id={} mailbox_id={}",
envelope.account_id, envelope.mailbox_id
),
ErrorCode::ResourceNotFound
)
})?;
let mut session = ImapExecutor::create_connection(envelope.account_id).await?;
let result = ImapExecutor::fetch_single_message_body(
&mut session,
&mailbox.encoded_name(),
envelope.uid,
)
.await;
session.logout().await.ok();
let raw_body = result?;
let fetched_hash = compute_content_hash(&raw_body);
if fetched_hash != envelope.content_hash {
return Err(raise_error!(
format!(
"Fetched message does not match archived content: expected content_hash={} got={}",
envelope.content_hash, fetched_hash
),
ErrorCode::ImapUnexpectedResult
));
}
// Re-create the detached blob (stripped EML + attachments) so the missing
// blob is repopulated for future requests. The detached EML is queued under
// `fetched_hash`, which equals `envelope.content_hash`.
let message = MessageParser::new().parse(raw_body.as_slice()).ok_or_else(|| {
raise_error!(
"Failed to parse fetched email content".into(),
ErrorCode::InternalError
)
})?;
detach_and_store_attachments(&raw_body, &message, &fetched_hash).await;
Ok(Bytes::from(raw_body))
}
#[cfg(test)]
mod test {
use html2text::config;
@@ -562,4 +783,56 @@ mod test {
}
}
}
/// Verifies that [`super::detach_and_store_attachments`] does not panic
/// when mail-parser reports attachment offsets past the raw body length.
///
/// Regression test for: "range end index X out of range for slice of
/// length Y" panic caused by a malformed email whose attachment
/// `raw_end_offset` exceeded the actual body size.
#[tokio::test]
async fn detach_attachments_bounds_check() {
let raw = concat!(
"From: sender@example.com\r\n",
"To: recipient@example.com\r\n",
"Subject: Test\r\n",
"MIME-Version: 1.0\r\n",
"Content-Type: multipart/mixed; boundary=\"bnd\"\r\n",
"\r\n",
"--bnd\r\n",
"Content-Type: text/plain\r\n",
"\r\n",
"Hello\r\n",
"--bnd\r\n",
"Content-Type: application/octet-stream\r\n",
"Content-Disposition: attachment; filename=\"test.bin\"\r\n",
"\r\n",
"AAAAABBBBBCCCCCDDDDDEEEEEAAAAABBBBBCCCCCDDDDDEEEEE\r\n",
"--bnd--\r\n",
)
.as_bytes()
.to_vec();
let message = mail_parser::MessageParser::new()
.parse(&raw)
.expect("parse valid MIME message");
assert_eq!(message.attachment_count(), 1);
// Truncate the raw body so the attachment's raw_end_offset lies
// past the body end — exactly the scenario reported by users.
let truncated = &raw[..raw.len() - 20];
assert!(truncated.len() < raw.len());
// Must not panic.
let infos = super::detach_and_store_attachments(
truncated,
&message,
"test_content_hash",
)
.await;
// The attachment count must still match so the consistency check
// in reattach_eml_content doesn't fail later.
assert_eq!(infos.len(), 1);
}
}

View File

@@ -16,4 +16,12 @@ pub enum BichonError {
},
}
impl BichonError {
pub fn code(&self) -> ErrorCode {
match self {
BichonError::Generic { code, .. } => *code,
}
}
}
pub type BichonResult<T, E = BichonError> = std::result::Result<T, E>;

View File

@@ -0,0 +1,86 @@
//
// Copyright (c) 2025-2026 rustmailer.com (https://rustmailer.com)
//
// This file is part of the Bichon Email Archiving Project
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
// Event bus extension point.
//
// Community edition: NoopEventBus — all events are discarded.
// Pro edition: AuditEventBus — events are persisted to audit database.
// Enterprise edition: adds SIEM webhook to the same trait impl.
//
// The open-source server emits events at key points (login, view, delete, search).
// It never reads from the event bus — events are fire-and-forget.
use std::net::IpAddr;
use std::sync::{LazyLock, RwLock};
#[derive(Debug, Clone)]
pub enum Event {
EmailViewed {
email_id: String,
user: String,
ip: IpAddr,
},
EmailDeleted {
email_id: String,
user: String,
},
UserLoggedIn {
user: String,
ip: IpAddr,
},
UserCreated {
created_by: String,
new_user: String,
},
SearchPerformed {
query: String,
user: String,
},
SettingsChanged {
key: String,
user: String,
},
AttachmentDownloaded {
email_id: String,
content_hash: String,
user: String,
},
}
pub trait EventBus: Send + Sync {
fn emit(&self, event: Event);
}
/// Default — all events are discarded.
struct NoopEventBus;
impl EventBus for NoopEventBus {
fn emit(&self, _event: Event) {}
}
static EVENT_BUS: LazyLock<RwLock<Box<dyn EventBus>>> =
LazyLock::new(|| RwLock::new(Box::new(NoopEventBus)));
/// Called by Pro/Enterprise at startup to replace the noop default.
pub fn set_event_bus(bus: Box<dyn EventBus>) {
*EVENT_BUS.write().unwrap() = bus;
}
/// Fire-and-forget. Called by the server at key points.
pub fn emit(event: Event) {
EVENT_BUS.read().unwrap().emit(event);
}

View File

@@ -0,0 +1,29 @@
//
// Copyright (c) 2025-2026 rustmailer.com (https://rustmailer.com)
//
// This file is part of the Bichon Email Archiving Project
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
// Event bus extension point.
//
// Community edition: NoopEventBus — all events are discarded.
// Pro edition: AuditEventBus — events are persisted to audit database.
// Enterprise edition: adds SIEM webhook to the same trait impl.
//
// The open-source server emits events at key points (login, view, delete, search).
// It never reads from the event bus — events are fire-and-forget.
pub mod event_bus;
pub mod text_extractor;

View File

@@ -0,0 +1,75 @@
//
// Copyright (c) 2025-2026 rustmailer.com (https://rustmailer.com)
//
// This file is part of the Bichon Email Archiving Project
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
// Attachment text extraction extension point.
//
// Community edition: NoopExtractor — no attachments are text-indexed.
// Pro edition: PdfExtractor — extracts text from PDF, Word, etc.
//
// Used in: crates/core/src/envelope/extractor.rs
use std::sync::{LazyLock, RwLock};
pub struct ExtractedText {
pub text: String,
pub page_count: Option<u32>,
pub is_ocr: bool,
}
pub trait AttachmentTextExtractor: Send + Sync {
/// Returns None if this extractor doesn't handle the file type.
/// Returns Some(ExtractedText) if text was successfully extracted.
fn extract(&self, content_type: &str, ext: &str, bytes: &[u8]) -> Option<ExtractedText>;
}
/// Default — all attachments are skipped.
struct NoopExtractor;
impl AttachmentTextExtractor for NoopExtractor {
fn extract(&self, _ct: &str, _ext: &str, _bytes: &[u8]) -> Option<ExtractedText> {
None
}
}
static EXTRACTOR: LazyLock<RwLock<Box<dyn AttachmentTextExtractor>>> =
LazyLock::new(|| RwLock::new(Box::new(NoopExtractor)));
/// Called by Pro/Enterprise at startup to replace the noop default.
pub fn set_extractor(extractor: Box<dyn AttachmentTextExtractor>) {
*EXTRACTOR.write().unwrap() = extractor;
}
/// Attachments larger than this are skipped (10 MiB). Avoids excessive memory
/// and CPU cost for huge files whose text is rarely useful for search.
pub const MAX_EXTRACT_BYTES: usize = 10 * 1024 * 1024;
/// Quick pre-filter: returns true for file types where text extraction may
/// produce useful results. Avoids cloning attachment bytes for images, videos,
/// archives, etc. when no registered extractor would handle them.
pub fn should_try_extract(content_type: &str, ext: &str) -> bool {
matches!(
ext,
"pdf" | "doc" | "docx" | "xls" | "xlsx" | "ppt" | "pptx"
| "txt" | "rtf" | "odt" | "ods" | "odp"
) || content_type.starts_with("text/")
}
/// Called by the attachment pipeline during IMAP sync.
/// The caller should wrap this in spawn_blocking for CPU-bound extraction.
pub fn extract_text(content_type: &str, ext: &str, bytes: &[u8]) -> Option<ExtractedText> {
EXTRACTOR.read().unwrap().extract(content_type, ext, bytes)
}

View File

@@ -34,6 +34,25 @@ use std::ops::DerefMut;
use tokio::io::BufWriter;
use tracing::debug;
/// Classify an `io::Error` (from TLS stream I/O) for IMAP connection errors.
/// `UnexpectedEof` is treated as a network error because many servers skip
/// the TLS `close_notify` alert, causing rustls to emit this error when the
/// TCP connection is dropped normally.
fn classify_io_error(e: &std::io::Error) -> ErrorCode {
use std::io::ErrorKind;
matches!(
e.kind(),
ErrorKind::BrokenPipe
| ErrorKind::ConnectionReset
| ErrorKind::ConnectionAborted
| ErrorKind::TimedOut
| ErrorKind::UnexpectedEof
| ErrorKind::NotConnected
)
.then_some(ErrorCode::NetworkError)
.unwrap_or(ErrorCode::ImapCommandFailed)
}
#[derive(Debug)]
pub(crate) struct Client {
inner: ImapClient<Box<dyn SessionStream>>,
@@ -141,7 +160,7 @@ impl Client {
let _greeting = client
.read_response()
.await
.map_err(|e| raise_error!(format!("{:#?}", e), ErrorCode::ImapCommandFailed))?
.map_err(|e| raise_error!(format!("{:#?}", e), classify_io_error(&e)))?
.ok_or_else(|| {
raise_error!(
"Failed to read IMAP greeting — this usually indicates an incorrect encryption setting (SSL vs. STARTTLS). Your current setting is SSL.".into(),
@@ -171,7 +190,7 @@ impl Client {
let _greeting = client
.read_response()
.await
.map_err(|e| raise_error!(format!("{:#?}", e), ErrorCode::ImapCommandFailed))?
.map_err(|e| raise_error!(format!("{:#?}", e), classify_io_error(&e)))?
.ok_or_else(|| {
raise_error!(
"failed to read greeting".into(),
@@ -202,7 +221,7 @@ impl Client {
let _greeting = client
.read_response()
.await
.map_err(|e| raise_error!(format!("{:#?}", e), ErrorCode::ImapCommandFailed))?
.map_err(|e| raise_error!(format!("{:#?}", e), classify_io_error(&e)))?
.ok_or_else(|| {
raise_error!(
"Failed to read IMAP greeting — this usually indicates an incorrect encryption setting (SSL vs. STARTTLS). Your current setting is STARTTLS.".into(),

View File

@@ -17,8 +17,7 @@
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use crate::account::migration::AccountModel;
use crate::account::state::{DownloadState, FolderStatus};
use crate::cache::imap::download::flow::{generate_uid_sequence_hashset, DEFAULT_BATCH_SIZE};
use crate::account::state::{DownloadState, DownloadStatus, FolderStatus};
use crate::cache::imap::mailbox::MailBox;
use crate::envelope::extractor::extract_envelope_and_store_it;
use crate::error::code::ErrorCode;
@@ -28,11 +27,29 @@ use crate::{error::BichonResult, imap::manager::ImapConnectionManager};
use async_imap::types::Name;
use async_imap::Session;
use futures::TryStreamExt;
use std::collections::HashSet;
use std::collections::{HashMap, HashSet};
use tokio_util::sync::CancellationToken;
use tracing::info;
const BODY_FETCH_COMMAND: &str = "(UID INTERNALDATE RFC822.SIZE BODY.PEEK[])";
const SIZE_ONLY_FETCH: &str = "(UID RFC822.SIZE)";
fn classify_imap_error(e: &async_imap::error::Error) -> ErrorCode {
match e {
async_imap::error::Error::Io(io) => matches!(
io.kind(),
std::io::ErrorKind::BrokenPipe
| std::io::ErrorKind::ConnectionReset
| std::io::ErrorKind::ConnectionAborted
| std::io::ErrorKind::TimedOut
| std::io::ErrorKind::UnexpectedEof
)
.then_some(ErrorCode::NetworkError)
.unwrap_or(ErrorCode::ImapCommandFailed),
async_imap::error::Error::ConnectionLost => ErrorCode::NetworkError,
_ => ErrorCode::ImapCommandFailed,
}
}
pub struct ImapExecutor;
@@ -43,11 +60,11 @@ impl ImapExecutor {
let list = session
.list(Some(""), Some("*"))
.await
.map_err(|e| raise_error!(format!("{:#?}", e), ErrorCode::ImapCommandFailed))?;
.map_err(|e| raise_error!(format!("{:#?}", e), classify_imap_error(&e)))?;
let result = list
.try_collect::<Vec<Name>>()
.await
.map_err(|e| raise_error!(format!("{:#?}", e), ErrorCode::ImapCommandFailed))?;
.map_err(|e| raise_error!(format!("{:#?}", e), classify_imap_error(&e)))?;
Ok(result)
}
@@ -59,11 +76,11 @@ impl ImapExecutor {
session
.examine(mailbox_name)
.await
.map_err(|e| raise_error!(format!("{:#?}", e), ErrorCode::ImapCommandFailed))?;
.map_err(|e| raise_error!(format!("{:#?}", e), classify_imap_error(&e)))?;
let result = session
.uid_search(query)
.await
.map_err(|e| raise_error!(format!("{:#?}", e), ErrorCode::ImapCommandFailed))?;
.map_err(|e| raise_error!(format!("{:#?}", e), classify_imap_error(&e)))?;
Ok(result)
}
@@ -77,9 +94,18 @@ impl ImapExecutor {
session
.append(mailbox_name, flags, internaldate, content)
.await
.map_err(|e| raise_error!(format!("{:#?}", e), ErrorCode::ImapCommandFailed))
.map_err(|e| raise_error!(format!("{:#?}", e), classify_imap_error(&e)))
}
/// Fetches new mail for a mailbox.
///
/// When `before` is `Some(date)`, a two-step approach is used:
/// `UID SEARCH` to find matching UIDs (standard IMAP), then batch `UID FETCH`
/// for the specific UIDs. When `before` is `None`, a direct ranged
/// `UID FETCH {start}:*` is issued and results are streamed.
///
/// Returns `Ok(Some(max_uid))` with the highest UID fetched, or `Ok(None)`
/// if no new mail was found.
pub async fn fetch_new_mail(
session: &mut Session<Box<dyn SessionStream>>,
account: &AccountModel,
@@ -87,114 +113,221 @@ impl ImapExecutor {
start_uid: u64,
before: Option<&str>,
token: CancellationToken,
) -> BichonResult<()> {
) -> BichonResult<Option<u32>> {
assert!(start_uid > 0, "start_uid must be greater than 0");
let query = match before {
Some(date) => format!("UID {start_uid}:* BEFORE {date}"),
None => format!("UID {start_uid}:*"),
};
session
.examine(&mailbox.encoded_name())
.await
.map_err(|e| raise_error!(format!("{:#?}", e), classify_imap_error(&e)))?;
let uid_list = match Self::uid_search(session, &mailbox.encoded_name(), &query).await {
Ok(uid_list) => uid_list,
Err(e) => {
let err_msg = format!("UID search failed in [{}]: {:#?}", mailbox.name, e);
DownloadState::update_folder_progress(
account.id,
mailbox.name.clone(),
0,
0,
FolderStatus::Failed,
Some(err_msg.clone()),
)?;
DownloadState::append_session_error(account.id, err_msg)?;
return Err(e);
match before {
Some(date) => {
Self::fetch_new_mail_with_before(session, account, mailbox, start_uid, date, token)
.await
}
};
None => Self::fetch_new_mail_range(session, account, mailbox, start_uid, token).await,
}
}
let len = uid_list.len();
if len == 0 {
let msg = match before {
Some(date) => format!("No emails found before {}.", date),
None => "No new emails found.".into(),
};
/// Two-step approach for date-filtered incremental fetch: UID SEARCH first,
/// then batch UID FETCH for matching UIDs. Uses standard IMAP syntax that
/// works across all compliant servers.
async fn fetch_new_mail_with_before(
session: &mut Session<Box<dyn SessionStream>>,
account: &AccountModel,
mailbox: &MailBox,
start_uid: u64,
date: &str,
token: CancellationToken,
) -> BichonResult<Option<u32>> {
let query = format!("UID {start_uid}:* BEFORE {date}");
info!(
"[account {}][mailbox {}] fetch_new_mail: UID SEARCH {}",
account.id, mailbox.name, query
);
let results = session.uid_search(&query).await.map_err(|e| {
let err_msg = format!("UID SEARCH failed in [{}]: {:#?}", mailbox.name, e);
let _ = DownloadState::append_session_error(account.id, err_msg);
raise_error!(format!("{:#?}", e), classify_imap_error(&e))
})?;
if results.is_empty() {
DownloadState::update_folder_progress(
account.id,
mailbox.name.clone(),
0,
0,
FolderStatus::Success,
Some(msg),
Some("No new emails found.".into()),
)?;
return Ok(());
return Ok(None);
}
info!(
"[account {}][mailbox {}] {} envelopes need to be fetched",
account.id, mailbox.name, len
);
let mut uid_vec: Vec<u32> = uid_list.into_iter().collect();
let mut uid_vec: Vec<u32> = results.into_iter().collect();
uid_vec.sort();
let uid_batches = generate_uid_sequence_hashset(
uid_vec,
account.download_batch_size.unwrap_or(DEFAULT_BATCH_SIZE) as usize,
false,
);
let mut current_processed = 0u64;
let mut has_error_or_cancel = false;
for (index, batch) in uid_batches.into_iter().enumerate() {
let max_uid = uid_vec.last().copied();
let planned = uid_vec.len() as u64;
let batch_size = account.download_batch_size.unwrap_or(DEFAULT_BATCH_SIZE) as usize;
let uid_batches = generate_uid_sequence_hashset(uid_vec, batch_size);
DownloadState::update_folder_progress(
account.id,
mailbox.name.clone(),
planned,
0,
FolderStatus::Pending,
None,
)?;
let mut count = 0u64;
for batch in uid_batches {
if token.is_cancelled() {
break;
DownloadState::update_session_status(
account.id,
DownloadStatus::Cancelled,
Some("User stopped or system shutdown".to_string()),
)?;
DownloadState::update_folder_progress(
account.id,
mailbox.name.clone(),
planned,
count,
FolderStatus::Cancelled,
None,
)?;
return Err(raise_error!(
"Stream cancelled".into(),
ErrorCode::InternalError
));
}
match Self::uid_batch_retrieve_emails(
let processed = Self::uid_batch_retrieve_emails(
session,
account.id,
mailbox.id,
&batch.0,
account.max_email_size_bytes,
token.clone(),
)
.await
{
Ok(_) => {
current_processed += batch.1;
DownloadState::update_folder_progress(
account.id,
mailbox.name.clone(),
len as u64,
current_processed,
FolderStatus::Downloading,
None,
)?;
}
Err(e) => {
let err_msg = format!("Batch {} failed: {:#?}", index, e);
DownloadState::append_session_error(account.id, err_msg.clone())?;
DownloadState::update_folder_progress(
account.id,
mailbox.name.clone(),
len as u64,
current_processed,
FolderStatus::Failed,
Some(err_msg),
)?;
has_error_or_cancel = true;
break;
}
}
}
if !has_error_or_cancel {
.await?;
count += processed;
DownloadState::update_folder_progress(
account.id,
mailbox.name.clone(),
len as u64,
current_processed,
FolderStatus::Success,
planned,
count,
FolderStatus::Downloading,
None,
)?;
}
Ok(())
DownloadState::update_folder_progress(
account.id,
mailbox.name.clone(),
count,
count,
FolderStatus::Success,
None,
)?;
Ok(max_uid)
}
/// Direct ranged UID FETCH without date filtering. Streams results from
/// the server in a single IMAP round-trip.
async fn fetch_new_mail_range(
session: &mut Session<Box<dyn SessionStream>>,
account: &AccountModel,
mailbox: &MailBox,
start_uid: u64,
token: CancellationToken,
) -> BichonResult<Option<u32>> {
let uid_range = format!("{start_uid}:*");
info!(
"[account {}][mailbox {}] fetch_new_mail: direct UID FETCH {}",
account.id, mailbox.name, uid_range
);
let mut stream = session
.uid_fetch(&uid_range, BODY_FETCH_COMMAND)
.await
.map_err(|e| {
let err_msg = format!("UID FETCH failed in [{}]: {:#?}", mailbox.name, e);
let _ = DownloadState::append_session_error(account.id, err_msg);
raise_error!(format!("{:#?}", e), classify_imap_error(&e))
})?;
let mut count = 0u64;
let mut skipped = 0u64;
let mut max_uid: Option<u32> = None;
let size_limit = account
.max_email_size_bytes
.unwrap_or(DEFAULT_MAX_EMAIL_SIZE);
while let Some(fetch) = stream
.try_next()
.await
.map_err(|e| raise_error!(format!("{:#?}", e), classify_imap_error(&e)))?
{
if token.is_cancelled() {
tracing::info!("Account {}: fetch_new_mail stream interrupted.", account.id);
DownloadState::update_session_status(
account.id,
DownloadStatus::Cancelled,
Some("User stopped or system shutdown".to_string()),
)?;
return Err(raise_error!(
"Stream cancelled".into(),
ErrorCode::InternalError
));
}
let msg_size = fetch.size.unwrap_or(0) as u64;
if msg_size > 0 && msg_size > size_limit {
tracing::warn!(
account_id = account.id,
mailbox_id = mailbox.id,
uid = fetch.uid,
size = msg_size,
limit = size_limit,
"Skipping oversized email (streaming mode)"
);
skipped += 1;
continue;
}
if let Some(uid) = fetch.uid {
max_uid = Some(max_uid.unwrap_or(0).max(uid));
}
extract_envelope_and_store_it(fetch, account.id, mailbox.id).await?;
count += 1;
}
let total = count + skipped;
if total == 0 {
DownloadState::update_folder_progress(
account.id,
mailbox.name.clone(),
0,
0,
FolderStatus::Success,
Some("No new emails found.".into()),
)?;
} else {
DownloadState::update_folder_progress(
account.id,
mailbox.name.clone(),
total,
count,
FolderStatus::Success,
if skipped > 0 {
Some(format!("{skipped} email(s) skipped due to size limit"))
} else {
None
},
)?;
}
Ok(max_uid)
}
pub async fn batch_retrieve_emails(
@@ -205,48 +338,149 @@ impl ImapExecutor {
page: u64,
page_size: u64,
encoded_mailbox_name: &str,
desc: bool,
max_email_size_bytes: Option<u64>,
token: CancellationToken,
max_uid: &mut Option<u32>,
) -> BichonResult<usize> {
assert!(page > 0, "Page number must be greater than 0");
assert!(page_size > 0, "Page size must be greater than 0");
let (start, end) = if desc {
// Fetch messages starting from the newest (descending order)
let end = total.saturating_sub((page - 1) * page_size);
if end == 0 {
return Ok(0);
}
// Calculate start as end - page_size + 1 to avoid off-by-one errors
let start = end.saturating_sub(page_size - 1).max(1);
(start, end)
} else {
// Fetch messages starting from the oldest (ascending order)
let start = (page - 1) * page_size + 1;
if start > total {
return Ok(0);
}
// Calculate end, capped by the total number of messages
let end = (start + page_size - 1).min(total);
(start, end)
};
// Fetch messages starting from the oldest (ascending order).
let start = (page - 1) * page_size + 1;
if start > total {
return Ok(0);
}
let end = (start + page_size - 1).min(total);
let sequence_set = format!("{}:{}", start, end);
info!(
"Fetching mailbox '{}' messages: sequence {} (page {}, page_size {}, desc={})",
encoded_mailbox_name, sequence_set, page, page_size, desc
"Fetching mailbox '{}' messages: sequence {} (page {}, page_size {})",
encoded_mailbox_name, sequence_set, page, page_size
);
let mut stream = session
.fetch(sequence_set.as_str(), BODY_FETCH_COMMAND)
let limit = max_email_size_bytes.unwrap_or(DEFAULT_MAX_EMAIL_SIZE);
// PASS 1: fetch only SIZE to identify oversized messages
let acceptable_uids = {
let mut size_stream = session
.fetch(sequence_set.as_str(), SIZE_ONLY_FETCH)
.await
.map_err(|e| raise_error!(format!("{:#?}", e), classify_imap_error(&e)))?;
let mut uids: Vec<u32> = Vec::new();
while let Some(fetch) = size_stream
.try_next()
.await
.map_err(|e| raise_error!(format!("{:#?}", e), classify_imap_error(&e)))?
{
let uid = fetch.uid.unwrap_or(0);
let msg_size = fetch.size.unwrap_or(0) as u64;
if msg_size == 0 || msg_size <= limit {
uids.push(uid);
} else {
tracing::warn!(
account_id,
mailbox_id,
uid,
size = msg_size,
limit,
"Skipping oversized email"
);
}
}
uids
};
if acceptable_uids.is_empty() {
return Ok(0);
}
// PASS 2: fetch bodies only for acceptable UIDs
let filtered = compress_uid_list(acceptable_uids);
let mut body_stream = session
.uid_fetch(&filtered, BODY_FETCH_COMMAND)
.await
.map_err(|e| raise_error!(format!("{:#?}", e), ErrorCode::ImapCommandFailed))?;
.map_err(|e| raise_error!(format!("{:#?}", e), classify_imap_error(&e)))?;
let mut count = 0;
while let Some(fetch) = stream
while let Some(fetch) = body_stream
.try_next()
.await
.map_err(|e| raise_error!(format!("{:#?}", e), ErrorCode::ImapCommandFailed))?
.map_err(|e| raise_error!(format!("{:#?}", e), classify_imap_error(&e)))?
{
if token.is_cancelled() {
tracing::info!("Account {}: UID fetch stream interrupted.", account_id);
return Err(raise_error!(
"Stream cancelled".into(),
ErrorCode::InternalError
));
}
if let Some(uid) = fetch.uid {
*max_uid = Some((*max_uid).unwrap_or(0).max(uid));
}
extract_envelope_and_store_it(fetch, account_id, mailbox_id).await?;
count += 1;
}
Ok(count)
}
pub async fn uid_batch_retrieve_emails(
session: &mut Session<Box<dyn SessionStream>>,
account_id: u64,
mailbox_id: u64,
uid_set: &str,
max_email_size_bytes: Option<u64>,
token: CancellationToken,
) -> BichonResult<u64> {
let limit = max_email_size_bytes.unwrap_or(DEFAULT_MAX_EMAIL_SIZE);
// PASS 1: fetch only SIZE to identify oversized messages
let acceptable_uids = {
let mut size_stream = session
.uid_fetch(uid_set, SIZE_ONLY_FETCH)
.await
.map_err(|e| raise_error!(format!("{:#?}", e), classify_imap_error(&e)))?;
let mut uids: Vec<u32> = Vec::new();
while let Some(fetch) = size_stream
.try_next()
.await
.map_err(|e| raise_error!(format!("{:#?}", e), classify_imap_error(&e)))?
{
let uid = fetch.uid.unwrap_or(0);
let msg_size = fetch.size.unwrap_or(0) as u64;
if msg_size == 0 || msg_size <= limit {
uids.push(uid);
} else {
tracing::warn!(
account_id,
mailbox_id,
uid,
size = msg_size,
limit,
"Skipping oversized email"
);
}
}
uids
};
if acceptable_uids.is_empty() {
return Ok(0);
}
// PASS 2: fetch bodies only for acceptable UIDs
let filtered = compress_uid_list(acceptable_uids);
let mut body_stream = session
.uid_fetch(&filtered, BODY_FETCH_COMMAND)
.await
.map_err(|e| raise_error!(format!("{:#?}", e), classify_imap_error(&e)))?;
let mut count = 0u64;
while let Some(fetch) = body_stream
.try_next()
.await
.map_err(|e| raise_error!(format!("{:#?}", e), classify_imap_error(&e)))?
{
if token.is_cancelled() {
tracing::info!("Account {}: UID fetch stream interrupted.", account_id);
@@ -261,32 +495,56 @@ impl ImapExecutor {
Ok(count)
}
pub async fn uid_batch_retrieve_emails(
/// Fetches the raw RFC822 body of a single message by UID.
///
/// Selects (read-only) the given mailbox and issues `UID FETCH <uid> (BODY.PEEK[])`.
/// Used for on-demand self-healing when an indexed message's content blob is missing.
/// Returns the raw bytes, or an error if the message cannot be retrieved.
pub async fn fetch_single_message_body(
session: &mut Session<Box<dyn SessionStream>>,
account_id: u64,
mailbox_id: u64,
uid_set: &str,
token: CancellationToken,
) -> BichonResult<()> {
let mut stream = session
.uid_fetch(uid_set, BODY_FETCH_COMMAND)
encoded_mailbox_name: &str,
uid: u32,
) -> BichonResult<Vec<u8>> {
session
.examine(encoded_mailbox_name)
.await
.map_err(|e| raise_error!(format!("{:#?}", e), ErrorCode::ImapCommandFailed))?;
while let Some(fetch) = stream
.map_err(|e| raise_error!(format!("{:#?}", e), classify_imap_error(&e)))?;
let mut stream = session
.uid_fetch(uid.to_string(), BODY_FETCH_COMMAND)
.await
.map_err(|e| raise_error!(format!("{:#?}", e), classify_imap_error(&e)))?;
let fetch = stream
.try_next()
.await
.map_err(|e| raise_error!(format!("{:#?}", e), ErrorCode::ImapCommandFailed))?
{
if token.is_cancelled() {
tracing::info!("Account {}: UID fetch stream interrupted.", account_id);
return Err(raise_error!(
"Stream cancelled".into(),
ErrorCode::InternalError
));
}
extract_envelope_and_store_it(fetch, account_id, mailbox_id).await?;
}
Ok(())
.map_err(|e| raise_error!(format!("{:#?}", e), classify_imap_error(&e)))?
.ok_or_else(|| {
raise_error!(
format!("UID {uid} not found on IMAP server"),
ErrorCode::ResourceNotFound
)
})?;
let body = fetch
.body()
.ok_or_else(|| {
raise_error!(
format!("No body returned for UID {uid}"),
ErrorCode::ImapUnexpectedResult
)
})?
.to_vec();
// // Drain any remaining items so the stream is fully consumed before reuse.
// while stream
// .try_next()
// .await
// .map_err(|e| raise_error!(format!("{:#?}", e), classify_imap_error(&e)))?
// .is_some()
// {}
Ok(body)
}
pub async fn create_connection(
@@ -294,4 +552,250 @@ impl ImapExecutor {
) -> BichonResult<Session<Box<dyn SessionStream>>> {
ImapConnectionManager::build(account_id).await
}
/// Fetch UID → Message-ID mapping without downloading bodies.
/// `uid_set` is an IMAP sequence-set string (e.g. "1:100" or "1,3,5").
pub async fn fetch_uid_metadata(
session: &mut Session<Box<dyn SessionStream>>,
uid_set: &str,
token: CancellationToken,
) -> BichonResult<HashMap<u32, Option<String>>> {
let mut stream = session
.uid_fetch(uid_set, "(UID BODY.PEEK[HEADER])")
.await
.map_err(|e| raise_error!(format!("{:#?}", e), classify_imap_error(&e)))?;
let mut result = HashMap::new();
while let Some(fetch) = stream
.try_next()
.await
.map_err(|e| raise_error!(format!("{:#?}", e), classify_imap_error(&e)))?
{
if token.is_cancelled() {
return Err(raise_error!(
"Stream cancelled".into(),
ErrorCode::InternalError
));
}
let uid = fetch.uid.unwrap_or(0);
let msg_id = fetch.header().and_then(parse_message_id_header);
result.insert(uid, msg_id);
}
Ok(result)
}
}
pub const DEFAULT_BATCH_SIZE: u32 = 30;
pub const DEFAULT_MAX_EMAIL_SIZE: u64 = 100 * 1024 * 1024;
/// Compresses a sorted list of UIDs into an IMAP sequence-set string.
/// Consecutive UIDs become ranges (e.g. `1:5`), non-consecutive are
/// comma-separated (e.g. `1:5,10,12:15`).
pub fn compress_uid_list(nums: Vec<u32>) -> String {
if nums.is_empty() {
return String::new();
}
let mut sorted_nums = nums;
sorted_nums.sort();
let mut result = Vec::new();
let mut current_range_start = sorted_nums[0];
let mut current_range_end = sorted_nums[0];
for &n in sorted_nums.iter().skip(1) {
if n == current_range_end + 1 {
current_range_end = n;
} else {
if current_range_start == current_range_end {
result.push(current_range_start.to_string());
} else {
result.push(format!("{}:{}", current_range_start, current_range_end));
}
current_range_start = n;
current_range_end = n;
}
}
if current_range_start == current_range_end {
result.push(current_range_start.to_string());
} else {
result.push(format!("{}:{}", current_range_start, current_range_end));
}
result.join(",")
}
/// Splits a sorted list of unique UIDs into compressed sequence-set batches.
/// Returns `Vec<(sequence_set_string, batch_count)>`.
pub fn generate_uid_sequence_hashset(
unique_nums: Vec<u32>,
chunk_size: usize,
) -> Vec<(String, u64)> {
assert!(!unique_nums.is_empty());
let mut result = Vec::new();
let nums = unique_nums;
for chunk in nums.chunks(chunk_size) {
let size = chunk.len() as u64;
let compressed = compress_uid_list(chunk.to_vec());
result.push((compressed, size));
}
result
}
fn parse_message_id_header(header_bytes: &[u8]) -> Option<String> {
let header = std::str::from_utf8(header_bytes).ok()?;
for line in header.lines() {
if let Some(value) = line
.strip_prefix("Message-ID:")
.or_else(|| line.strip_prefix("Message-Id:"))
.or_else(|| line.strip_prefix("Message-id:"))
{
// mail_parser strips angle brackets, so we must do the same
// to ensure comparisons against the Tantivy index match.
let trimmed = value.trim();
let stripped = trimmed.strip_prefix('<').unwrap_or(trimmed);
let stripped = stripped.strip_suffix('>').unwrap_or(stripped);
if !stripped.is_empty() {
return Some(stripped.to_string());
}
}
}
None
}
#[cfg(test)]
mod test {
use super::*;
// ── compress_uid_list ──────────────────────────────────────────
#[test]
fn compress_empty() {
assert_eq!(compress_uid_list(vec![]), "");
}
#[test]
fn compress_single_uid() {
assert_eq!(compress_uid_list(vec![42]), "42");
}
#[test]
fn compress_consecutive_range() {
assert_eq!(compress_uid_list(vec![1, 2, 3, 4, 5]), "1:5");
}
#[test]
fn compress_mixed_ranges() {
assert_eq!(
compress_uid_list(vec![1, 2, 3, 5, 7, 8, 9, 10]),
"1:3,5,7:10"
);
}
#[test]
fn compress_gap_at_boundary() {
assert_eq!(compress_uid_list(vec![1, 2, 4, 5]), "1:2,4:5");
}
// ── generate_uid_sequence_hashset ──────────────────────────────
#[test]
fn batch_single_chunk() {
let batches = generate_uid_sequence_hashset(vec![1, 2, 3], 10);
assert_eq!(batches.len(), 1);
assert_eq!(batches[0].0, "1:3");
assert_eq!(batches[0].1, 3);
}
#[test]
fn batch_multiple_chunks() {
let batches = generate_uid_sequence_hashset(vec![1, 2, 3, 4, 5], 2);
assert_eq!(batches.len(), 3);
assert_eq!(batches[0].0, "1:2");
assert_eq!(batches[0].1, 2);
assert_eq!(batches[1].0, "3:4");
assert_eq!(batches[1].1, 2);
assert_eq!(batches[2].0, "5");
assert_eq!(batches[2].1, 1);
}
// ── parse_message_id_header ─────────────────────────────────────
#[test]
fn parse_standard_message_id() {
let header = b"Message-ID: <abc123@example.com>\r\n";
assert_eq!(
parse_message_id_header(header),
Some("abc123@example.com".into())
);
}
#[test]
fn parse_message_id_lowercase() {
let header = b"Message-Id: <foo@bar.com>\r\n";
assert_eq!(
parse_message_id_header(header),
Some("foo@bar.com".into())
);
}
#[test]
fn parse_message_id_extra_whitespace() {
let header = b"Message-ID: <spaces@test.com> \r\n";
assert_eq!(
parse_message_id_header(header),
Some("spaces@test.com".into())
);
}
#[test]
fn parse_empty_message_id_returns_none() {
let header = b"Message-ID: <>\r\n";
assert_eq!(parse_message_id_header(header), None);
}
#[test]
fn parse_missing_header_returns_none() {
let header = b"X-Custom: something\r\n";
assert_eq!(parse_message_id_header(header), None);
}
#[test]
fn parse_empty_body_returns_none() {
assert_eq!(parse_message_id_header(b""), None);
}
#[test]
fn parse_message_id_in_full_header() {
// The Message-ID line is in the middle, not at the start.
let header = b"From: sender@example.com\r\n\
Date: Thu, 01 Jan 2025 00:00:00 +0000\r\n\
Subject: test\r\n\
Message-ID: <mid@example.com>\r\n\
To: recipient@example.com\r\n\r\n";
assert_eq!(
parse_message_id_header(header),
Some("mid@example.com".into())
);
}
#[test]
fn parse_message_id_only_in_full_header() {
// Only a few headers, Message-ID is among them.
let header = b"From: a@b.com\r\nMessage-ID: <x@y.com>\r\n\r\n";
assert_eq!(parse_message_id_header(header), Some("x@y.com".into()));
}
#[test]
fn parse_message_id_no_brackets_still_works() {
let header = b"Message-ID: plain@example.com\r\n";
assert_eq!(
parse_message_id_header(header),
Some("plain@example.com".into())
);
}
}

View File

@@ -95,16 +95,40 @@ impl ImapConnectionManager {
pub async fn build(account_id: u64) -> BichonResult<Session<Box<dyn SessionStream>>> {
let account = AccountModel::get(account_id)?;
let client = match Self::create_client(&account).await {
Ok(client) => client,
Err(error) => {
error!(
"Failed to create IMAP {}'s client: {:#?}",
&account.email, error
);
return Err(error);
let account_email = account.email.clone();
let mut client = None;
for attempt in 0..3u32 {
match Self::create_client(&account).await {
Ok(c) => {
client = Some(c);
break;
}
Err(error) if error.code() == ErrorCode::NetworkError && attempt < 2 => {
warn!(
"IMAP connection attempt {}/3 to {} failed (network error), retrying...",
attempt + 1,
account_email
);
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
continue;
}
Err(error) => {
error!(
"Failed to create IMAP {}'s client: {:#?}",
account_email, error
);
return Err(error);
}
}
};
}
let client = client.ok_or_else(|| {
raise_error!(
format!("Failed to create IMAP {}'s client after 3 attempts", account_email),
ErrorCode::NetworkError
)
})?;
let mut session = match Self::authenticate(client, &account).await {
Ok(session) => session,

View File

@@ -0,0 +1,538 @@
//
// Copyright (c) 2025-2026 rustmailer.com (https://rustmailer.com)
//
// This file is part of the Bichon Email Archiving Project
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
//! A minimal scriptable IMAP server for integration testing.
//!
//! Each instance listens on a random localhost port and responds to a
//! pre-configured script of (expected_command, response) pairs. Commands
//! are matched by substring — the first matching pattern wins.
//!
//! # Example
//! ```ignore
//! let server = MockImapServer::new()
//! .greeting("* OK ready\r\n")
//! .respond("LOGIN", "A0 OK logged in\r\n")
//! .respond("CAPABILITY", "* CAPABILITY IMAP4rev1\r\nA0 OK done\r\n")
//! .respond("STATUS", "* STATUS INBOX (MESSAGES 10 UIDVALIDITY 42)\r\nA0 OK\r\n")
//! .respond("LOGOUT", "* BYE\r\nA0 OK\r\n")
//! .start()
//! .await;
//!
//! let (host, port) = server.addr();
//! // connect to host:port with Encryption::None
//! ```
use std::net::SocketAddr;
use std::sync::Arc;
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
use tokio::net::{TcpListener, TcpStream};
type Response = Vec<u8>;
pub struct MockImapServer {
greeting: Vec<u8>,
script: Vec<(String, Response)>,
}
impl MockImapServer {
pub fn new() -> Self {
Self {
greeting: b"* OK Mock IMAP server ready\r\n".to_vec(),
script: Vec::new(),
}
}
/// Set the greeting banner sent immediately after connection.
pub fn greeting(mut self, banner: impl Into<Vec<u8>>) -> Self {
self.greeting = banner.into();
self
}
/// Add a script step: when a client command *contains* `pattern` (case-insensitive),
/// respond with `response`. Steps are checked in insertion order.
pub fn respond(mut self, pattern: impl Into<String>, response: impl Into<Vec<u8>>) -> Self {
self.script.push((pattern.into(), response.into()));
self
}
/// Start the server on a random port. Returns a handle whose `addr()` gives
/// the `(host, port)` to connect to.
pub async fn start(self) -> MockImapServerHandle {
let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind");
let addr = listener.local_addr().expect("local_addr");
let server = Arc::new(self);
tokio::spawn(async move {
loop {
match listener.accept().await {
Ok((stream, _)) => {
let srv = server.clone();
tokio::spawn(async move {
srv.handle_connection(stream).await;
});
}
Err(_) => break,
}
}
});
MockImapServerHandle { addr }
}
async fn handle_connection(&self, mut stream: TcpStream) {
let (reader, mut writer) = stream.split();
let mut reader = BufReader::new(reader);
// Send greeting
if writer.write_all(&self.greeting).await.is_err() {
return;
}
let mut line = String::new();
loop {
line.clear();
match reader.read_line(&mut line).await {
Ok(0) => break, // EOF
Ok(_) => {}
Err(_) => break,
}
let tag = extract_tag(&line).unwrap_or("A0");
let matched = self.find_match(&line);
if let Some(response) = matched {
let substituted = substitute_tag(response, tag);
if writer.write_all(&substituted).await.is_err() {
break;
}
} else {
// Default: send tagged OK for commands we don't handle
let fallback = format!("{tag} OK done\r\n");
if writer.write_all(fallback.as_bytes()).await.is_err() {
break;
}
}
}
}
fn find_match(&self, line: &str) -> Option<&[u8]> {
let line_lower = line.to_lowercase();
for (pattern, response) in &self.script {
if line_lower.contains(&pattern.to_lowercase()) {
return Some(response);
}
}
None
}
}
impl Default for MockImapServer {
fn default() -> Self {
Self::new()
}
}
/// Handle to a running mock IMAP server. The server stops when this handle
/// is dropped.
pub struct MockImapServerHandle {
addr: SocketAddr,
}
impl MockImapServerHandle {
pub fn host(&self) -> String {
self.addr.ip().to_string()
}
pub fn port(&self) -> u16 {
self.addr.port()
}
}
fn extract_tag(line: &str) -> Option<&str> {
line.split_whitespace().next()
}
/// Replace `{TAG}` placeholders in `response` with `tag`.
fn substitute_tag(response: &[u8], tag: &str) -> Vec<u8> {
let placeholder = b"{TAG}";
if response.is_empty() || !contains_slice(response, placeholder) {
return response.to_vec();
}
let tag_bytes = tag.as_bytes();
let mut result = Vec::with_capacity(response.len());
let mut pos = 0;
while let Some(idx) = find_slice(&response[pos..], placeholder) {
result.extend_from_slice(&response[pos..pos + idx]);
result.extend_from_slice(tag_bytes);
pos += idx + placeholder.len();
}
result.extend_from_slice(&response[pos..]);
result
}
fn contains_slice(haystack: &[u8], needle: &[u8]) -> bool {
haystack.windows(needle.len()).any(|w| w == needle)
}
fn find_slice(haystack: &[u8], needle: &[u8]) -> Option<usize> {
haystack
.windows(needle.len())
.position(|w| w == needle)
}
// ============================================================
// Pre-built response helpers
// ============================================================
/// Build a tagged OK response.
pub fn ok(tag: impl AsRef<str>, msg: impl AsRef<str>) -> Vec<u8> {
format!("{} OK {}\r\n", tag.as_ref(), msg.as_ref()).into_bytes()
}
/// Build a STATUS response line.
pub fn status_response(
mailbox: &str,
messages: u32,
unseen: u32,
uid_next: u32,
uid_validity: Option<u32>,
) -> Vec<u8> {
let uv = uid_validity
.map(|v| format!(" UIDVALIDITY {v}"))
.unwrap_or_default();
let text = format!(
"* STATUS \"{mailbox}\" (MESSAGES {messages} UNSEEN {unseen} UIDNEXT {uid_next}{uv})\r\n"
);
// Clients expect a tagged response after the untagged STATUS line.
// We produce a generic OK that works for any tag.
let mut out = text.into_bytes();
out.extend_from_slice(b"{TAG} OK STATUS completed\r\n");
out
}
/// Build an EXAMINE response with mailbox data.
pub fn examine_response(
_mailbox: &str,
exists: u32,
uid_validity: u32,
uid_next: u32,
) -> Vec<u8> {
format!(
"* FLAGS (\\Seen \\Answered \\Flagged \\Deleted \\Draft)\r\n\
* OK [PERMANENTFLAGS ()]\r\n\
* {exists} EXISTS\r\n\
* 0 RECENT\r\n\
* OK [UIDVALIDITY {uid_validity}]\r\n\
* OK [UIDNEXT {uid_next}]\r\n\
* OK [HIGHESTMODSEQ 1]\r\n\
{{TAG}} OK [READ-ONLY] EXAMINE completed\r\n"
)
.into_bytes()
}
/// Build a UID SEARCH response for the given UID list.
pub fn uid_search_response(uids: &[u32]) -> Vec<u8> {
let uid_str = uids
.iter()
.map(|u| u.to_string())
.collect::<Vec<_>>()
.join(" ");
format!("* SEARCH {uid_str}\r\n{{TAG}} OK SEARCH completed\r\n").into_bytes()
}
/// Build a UID FETCH response returning full headers (for BODY[HEADER]).
/// Each entry: (uid, message_id)
pub fn uid_fetch_metadata_response(entries: &[(u32, &str)]) -> Vec<u8> {
let mut out = Vec::new();
for (uid, msg_id) in entries {
// Build a minimal header that contains the Message-ID line.
let header_data = format!(
"From: sender@example.com\r\n\
To: recipient@example.com\r\n\
Date: Thu, 01 Jan 2025 00:00:00 +0000\r\n\
Subject: test\r\n\
Message-ID: {msg_id}\r\n\r\n"
);
let header_len = header_data.len();
let line = format!(
"* {uid} FETCH (UID {uid} BODY[HEADER] {{{header_len}}}\r\n\
{header_data}\
)\r\n",
);
out.extend_from_slice(line.as_bytes());
}
out.extend_from_slice(b"{TAG} OK FETCH completed\r\n");
out
}
/// Build a UID FETCH RFC822 response with a full email body.
pub fn uid_fetch_rfc822_response(uid: u32, eml: &[u8]) -> Vec<u8> {
let header = format!(
"* {uid} FETCH (UID {uid} RFC822 {{{len}}}\r\n",
len = eml.len()
);
let mut out = header.into_bytes();
out.extend_from_slice(eml);
out.extend_from_slice(b")\r\n{TAG} OK FETCH completed\r\n");
out
}
/// A minimal RFC822 email fixture for testing.
pub fn minimal_eml(subject: &str, message_id: &str) -> Vec<u8> {
format!(
"From: sender@example.com\r\n\
To: recipient@example.com\r\n\
Subject: {subject}\r\n\
Message-ID: <{message_id}>\r\n\
Date: Thu, 01 Jan 2025 00:00:00 +0000\r\n\
MIME-Version: 1.0\r\n\
Content-Type: text/plain; charset=utf-8\r\n\
\r\n\
This is a test email: {subject}.\r\n"
)
.into_bytes()
}
// ============================================================
// Self-tests for the mock server itself
// ============================================================
#[cfg(test)]
mod tests {
use super::*;
async fn connect_and_read_greeting(host: &str, port: u16) -> String {
let mut stream = TcpStream::connect((host, port)).await.unwrap();
let (reader, _writer) = stream.split();
let mut reader = BufReader::new(reader);
let mut line = String::new();
reader.read_line(&mut line).await.unwrap();
line
}
async fn send_and_recv(host: &str, port: u16, cmd: &str) -> String {
let mut stream = TcpStream::connect((host, port)).await.unwrap();
let (reader, mut writer) = stream.split();
let mut reader = BufReader::new(reader);
// Read greeting
let mut line = String::new();
reader.read_line(&mut line).await.unwrap();
// Send command
writer.write_all(cmd.as_bytes()).await.unwrap();
writer.write_all(b"\r\n").await.unwrap();
// Read response (may be multi-line; read until tagged response)
let mut out = String::new();
loop {
line.clear();
reader.read_line(&mut line).await.unwrap();
out.push_str(&line);
if line.starts_with("A0") || line.starts_with("A1") {
break;
}
}
out
}
#[tokio::test]
async fn test_mock_greeting() {
let handle = MockImapServer::new().start().await;
let greeting = connect_and_read_greeting(&handle.host(), handle.port()).await;
assert!(greeting.starts_with("* OK"));
}
#[tokio::test]
async fn test_mock_scripted_response() {
let handle = MockImapServer::new()
.respond(
"LOGIN",
"A0 OK LOGIN completed\r\n",
)
.start()
.await;
let resp = send_and_recv(&handle.host(), handle.port(), "A0 LOGIN u p").await;
assert!(resp.contains("LOGIN completed"));
}
#[tokio::test]
async fn test_mock_fallback_on_unmatched() {
let handle = MockImapServer::new().start().await;
// Send a command that has no scripted response
let resp = send_and_recv(&handle.host(), handle.port(), "A0 NOOP").await;
assert!(resp.contains("OK done"), "unmatched command should get fallback OK");
}
#[tokio::test]
async fn test_status_response_helper() {
let resp = status_response("INBOX", 10, 2, 11, Some(42));
let text = String::from_utf8(resp).unwrap();
assert!(text.contains("MESSAGES 10"));
assert!(text.contains("UNSEEN 2"));
assert!(text.contains("UIDNEXT 11"));
assert!(text.contains("UIDVALIDITY 42"));
}
#[tokio::test]
async fn test_status_response_without_uidvalidity() {
let resp = status_response("INBOX", 10, 2, 11, None);
let text = String::from_utf8(resp).unwrap();
assert!(!text.contains("UIDVALIDITY"));
assert!(text.contains("MESSAGES 10"));
}
#[tokio::test]
async fn test_examine_response() {
let resp = examine_response("INBOX", 10, 42, 11);
let text = String::from_utf8(resp).unwrap();
assert!(text.contains("UIDVALIDITY 42"));
assert!(text.contains("10 EXISTS"));
}
#[tokio::test]
async fn test_uid_search_response() {
let resp = uid_search_response(&[1, 3, 5]);
let text = String::from_utf8(resp).unwrap();
assert!(text.contains("SEARCH 1 3 5"));
}
#[tokio::test]
async fn test_uid_fetch_metadata_response() {
let resp = uid_fetch_metadata_response(&[(1, "msg-a@x.com"), (2, "msg-b@x.com")]);
let text = String::from_utf8(resp).unwrap();
assert!(text.contains("Message-ID: msg-a@x.com"));
assert!(text.contains("Message-ID: msg-b@x.com"));
}
#[tokio::test]
async fn test_multiple_commands_in_sequence() {
let handle = MockImapServer::new()
.respond("LOGIN", "A0 OK LOGIN\r\n")
.respond("STATUS", status_response("INBOX", 5, 1, 6, Some(99)))
.respond("LOGOUT", "* BYE\r\nA0 OK\r\n")
.start()
.await;
let mut stream = TcpStream::connect((handle.host(), handle.port()))
.await
.unwrap();
let (reader, mut writer) = stream.split();
let mut reader = BufReader::new(reader);
// Read greeting
let mut buf = String::new();
reader.read_line(&mut buf).await.unwrap();
// LOGIN
writer.write_all(b"A0 LOGIN u p\r\n").await.unwrap();
buf.clear();
reader.read_line(&mut buf).await.unwrap();
assert!(buf.contains("LOGIN"));
// STATUS
writer
.write_all(b"A0 STATUS INBOX (MESSAGES UNSEEN UIDNEXT UIDVALIDITY)\r\n")
.await
.unwrap();
buf.clear();
// Read multi-line STATUS response (untagged line + tagged OK)
loop {
reader.read_line(&mut buf).await.unwrap();
if buf.contains("UIDVALIDITY 99") {
// Consume the tagged OK line that follows
buf.clear();
reader.read_line(&mut buf).await.unwrap();
break;
}
}
// LOGOUT
writer.write_all(b"A0 LOGOUT\r\n").await.unwrap();
buf.clear();
reader.read_line(&mut buf).await.unwrap();
assert!(buf.contains("BYE"));
}
#[tokio::test]
async fn test_tag_substitution_in_response() {
// Use {TAG} placeholder in the response and verify it gets the
// client's actual tag ("A5") substituted in.
let handle = MockImapServer::new()
.respond("LOGIN", "{TAG} OK LOGIN succeeded\r\n")
.start()
.await;
let mut stream = TcpStream::connect((handle.host(), handle.port()))
.await
.unwrap();
let (reader, mut writer) = stream.split();
let mut reader = BufReader::new(reader);
// Read greeting
let mut buf = String::new();
reader.read_line(&mut buf).await.unwrap();
// Send LOGIN with non-standard tag
writer.write_all(b"A5 LOGIN u p\r\n").await.unwrap();
buf.clear();
reader.read_line(&mut buf).await.unwrap();
assert!(
buf.contains("A5 OK LOGIN succeeded"),
"expected 'A5 OK LOGIN succeeded', got '{buf}'"
);
}
#[tokio::test]
async fn test_tag_substitution_multiple_placeholders() {
let handle = MockImapServer::new()
.respond("NOOP", "* 0 RECENT\r\n{TAG} OK NOOP done\r\n")
.start()
.await;
let mut stream = TcpStream::connect((handle.host(), handle.port()))
.await
.unwrap();
let (reader, mut writer) = stream.split();
let mut reader = BufReader::new(reader);
// Read greeting
let mut buf = String::new();
reader.read_line(&mut buf).await.unwrap();
// Send with tag "B99"
writer.write_all(b"B99 NOOP\r\n").await.unwrap();
// Read all lines
let mut all = String::new();
loop {
buf.clear();
reader.read_line(&mut buf).await.unwrap();
all.push_str(&buf);
if buf.starts_with("B99") {
break;
}
}
assert!(all.contains("* 0 RECENT\r\n"));
assert!(all.contains("B99 OK NOOP done\r\n"));
}
}

View File

@@ -26,3 +26,5 @@ pub mod session;
pub mod stats;
#[cfg(test)]
mod tests;
#[cfg(test)]
pub mod mock_server;

View File

@@ -105,6 +105,7 @@ impl ImportEmls {
unseen: None,
uid_next: None,
uid_validity: None,
highest_uid: None,
};
let mailbox_id = mailbox.id;
// Upsert the mailbox, creating it if it doesn't exist

View File

@@ -1,4 +1,5 @@
pub mod account;
pub mod ext;
pub mod admin;
pub mod autoconfig;
pub mod cache;

View File

@@ -157,8 +157,13 @@ async fn fetch_remote_with_progress(account_id: u64) -> BichonResult<Vec<MailBox
mailbox.account_id = account_id;
mailbox.id = create_hash(account_id, &mailbox.name);
// Use STATUS instead of EXAMINE: gets MESSAGES/UNSEEN/UIDNEXT/UIDVALIDITY
// without selecting the mailbox, avoiding context switches.
let mx = session
.examine(mailbox_name.as_str())
.status(
mailbox_name.as_str(),
"(MESSAGES UNSEEN UIDNEXT UIDVALIDITY)",
)
.await
.map_err(|e| raise_error!(format!("{:#?}", e), ErrorCode::ImapCommandFailed))?;
mailbox.exists = mx.exists;
@@ -205,8 +210,13 @@ pub async fn convert_names_to_mailboxes(
mailbox.account_id = account_id;
mailbox.id = create_hash(account_id, &mailbox.name);
// Use STATUS instead of EXAMINE: gets MESSAGES/UNSEEN/UIDNEXT/UIDVALIDITY
// without selecting the mailbox, avoiding context switches.
let mx = session
.examine(mailbox_name.as_str())
.status(
mailbox_name.as_str(),
"(MESSAGES UNSEEN UIDNEXT UIDVALIDITY)",
)
.await
.map_err(|e| raise_error!(format!("{:#?}", e), ErrorCode::ImapCommandFailed))?;
mailbox.exists = mx.exists;

View File

@@ -22,6 +22,7 @@ use crate::envelope::extractor::{extract_envelope_from_nested_message, reattach_
use crate::error::code::ErrorCode;
use crate::store::envelope::Envelope;
use crate::utils::compute_content_hash;
use crate::utils::html::block_remote_content;
use crate::{error::BichonResult, raise_error};
use mail_parser::{MessageParser, MimeHeaders};
//use poem_openapi::Object;
@@ -46,6 +47,14 @@ pub struct AttachmentInfo {
/// Hash of the content.
pub content_hash: String,
pub is_message: bool,
/// Text extracted from the attachment body (Pro/Enterprise feature).
/// Populated during IMAP sync; None for inline attachments and unsupported file types.
pub extracted_text: Option<String>,
/// Page count reported by the extractor, if any.
pub extracted_page_count: Option<u32>,
/// Whether the extracted text came from OCR.
#[serde(default)]
pub extracted_is_ocr: bool,
}
impl AttachmentInfo {
@@ -142,6 +151,9 @@ pub struct FullMessageContent {
pub html: Option<String>,
// all Attachments include inline attachments
pub attachments: Option<Vec<AttachmentInfo>>,
/// True when remote content (http/https URLs) was detected and stripped from html.
#[serde(default)]
pub has_remote_content: bool,
}
#[derive(Clone, Debug, Default, Eq, PartialEq, Deserialize, Serialize)]
@@ -155,11 +167,15 @@ pub struct FullNestedMessageContent {
pub attachments: Option<Vec<AttachmentInfo>>,
/// Metadata for the email envelope.
pub envelope: Envelope,
/// True when remote content (http/https URLs) was detected and stripped from html.
#[serde(default)]
pub has_remote_content: bool,
}
pub fn retrieve_email_content(
account_id: u64,
envelope_id: String,
block_remote: bool,
) -> BichonResult<FullMessageContent> {
AccountModel::check_account_exists(account_id)?;
let (envelope, eml) = reattach_eml_content(account_id, envelope_id)?;
@@ -190,7 +206,9 @@ pub fn retrieve_email_content(
content_type.c_subtype.as_deref().unwrap_or("")
);
let inline = disposition.map(|d| d.is_inline()).unwrap_or(false);
let inline = disposition
.map(|d| d.is_inline())
.unwrap_or_else(|| attachment.content_id().is_some());
if inline {
if let Some(html1) = html.as_deref() {
@@ -214,19 +232,31 @@ pub fn retrieve_email_content(
let is_message = attachment.is_message();
let content_hash = compute_content_hash(attachment.contents());
attachments.push(AttachmentInfo {
filename: filename.or(Some(content_hash.clone())), // Fallback to content_hash as the default filename if it is not provided.
filename: filename.or(Some(content_hash.clone())),
size: attachment.contents().len(),
inline,
file_type,
is_message,
content_hash,
content_id: attachment.content_id().map(Into::into),
extracted_text: None,
extracted_page_count: None,
extracted_is_ocr: false,
});
}
let mut has_remote_content = false;
if let Some(ref html_body) = html {
let filtered = block_remote_content(html_body);
has_remote_content = *html_body != filtered;
if block_remote {
html = Some(filtered);
}
}
Ok(FullMessageContent {
text,
html,
attachments: Some(attachments),
has_remote_content,
})
}
@@ -234,6 +264,7 @@ pub fn retrieve_nested_eml_content(
account_id: u64,
envelope_id: String,
content_hash: &str,
block_remote: bool,
) -> BichonResult<FullNestedMessageContent> {
let (_, eml) = reattach_eml_content(account_id, envelope_id)?;
let parent_message = MessageParser::default().parse(&eml).ok_or_else(|| {
@@ -273,7 +304,9 @@ pub fn retrieve_nested_eml_content(
for attachment in nested_message.attachments() {
let cid = attachment.content_id();
let disposition = attachment.content_disposition();
let is_inline = disposition.map(|d| d.is_inline()).unwrap_or(false);
let is_inline = disposition
.map(|d| d.is_inline())
.unwrap_or_else(|| cid.is_some());
if has_html && is_inline && cid.is_some() {
let content_id = cid.unwrap();
@@ -302,22 +335,133 @@ pub fn retrieve_nested_eml_content(
filename: attachment
.attachment_name()
.map(|n| n.to_string())
.or(Some(content_hash.clone())), // Fallback to content_hash as the default filename if it is not provided.
.or(Some(content_hash.clone())),
size: attachment.contents().len(),
inline: is_inline,
file_type,
content_hash,
is_message: attachment.is_message(),
content_id: cid.map(Into::into),
extracted_text: None,
extracted_page_count: None,
extracted_is_ocr: false,
});
}
let envelope = extract_envelope_from_nested_message(nested_message, account_id)?;
let mut has_remote_content = false;
if let Some(ref html_body) = html {
let filtered = block_remote_content(html_body);
has_remote_content = *html_body != filtered;
if block_remote {
html = Some(filtered);
}
}
Ok(FullNestedMessageContent {
text,
html,
attachments: Some(attachments),
envelope,
has_remote_content,
})
}
#[cfg(test)]
mod tests {
use super::*;
/// Simulates JSON written by a version before `extracted_text`, `extracted_page_count`,
/// and `extracted_is_ocr` were added to [`AttachmentInfo`]. Deserialization must
/// succeed and fill the missing fields with their defaults.
#[test]
fn attachment_info_backward_compat_no_extracted_fields() {
let old_json = r#"[
{
"file_type": "application/pdf",
"inline": false,
"filename": "report.pdf",
"size": 12345,
"content_id": null,
"content_hash": "abc123",
"is_message": false
},
{
"file_type": "image/png",
"inline": true,
"filename": "logo.png",
"size": 6789,
"content_id": "cid:logo@example.com",
"content_hash": "def456",
"is_message": false
}
]"#;
let attachments: Vec<AttachmentInfo> =
serde_json::from_str(old_json).expect("should deserialize legacy JSON");
assert_eq!(attachments.len(), 2);
// First attachment (regular file)
assert_eq!(attachments[0].file_type, "application/pdf");
assert!(!attachments[0].inline);
assert_eq!(attachments[0].filename.as_deref(), Some("report.pdf"));
assert_eq!(attachments[0].size, 12345);
assert_eq!(attachments[0].content_id, None);
assert_eq!(attachments[0].content_hash, "abc123");
assert!(!attachments[0].is_message);
// Fields added after the legacy format — must default correctly
assert_eq!(attachments[0].extracted_text, None);
assert_eq!(attachments[0].extracted_page_count, None);
assert!(!attachments[0].extracted_is_ocr);
// Second attachment (inline image with content-id)
assert_eq!(attachments[1].file_type, "image/png");
assert!(attachments[1].inline);
assert_eq!(attachments[1].filename.as_deref(), Some("logo.png"));
assert_eq!(attachments[1].size, 6789);
assert_eq!(attachments[1].content_id.as_deref(), Some("cid:logo@example.com"));
assert_eq!(attachments[1].content_hash, "def456");
assert!(!attachments[1].is_message);
assert_eq!(attachments[1].extracted_text, None);
assert_eq!(attachments[1].extracted_page_count, None);
assert!(!attachments[1].extracted_is_ocr);
}
/// Current struct must round-trip through serde_json without data loss.
#[test]
fn attachment_info_round_trip() {
let attachments = vec![
AttachmentInfo {
file_type: "text/html".into(),
inline: false,
filename: Some("page.html".into()),
size: 42,
content_id: None,
content_hash: "hash1".into(),
is_message: true,
extracted_text: Some("hello world".into()),
extracted_page_count: Some(1),
extracted_is_ocr: false,
},
AttachmentInfo {
file_type: "application/zip".into(),
inline: false,
filename: Some("archive.zip".into()),
size: 99999,
content_id: None,
content_hash: "hash2".into(),
is_message: false,
extracted_text: None,
extracted_page_count: None,
extracted_is_ocr: true,
},
];
let json = serde_json::to_string(&attachments).expect("serialize");
let round_tripped: Vec<AttachmentInfo> =
serde_json::from_str(&json).expect("deserialize");
assert_eq!(attachments, round_tripped);
}
}

View File

@@ -45,6 +45,14 @@ pub struct EmailSearchFilter {
pub bcc: Option<String>,
pub since: Option<i64>,
pub before: Option<i64>,
/// Lower bound (inclusive) on the IMAP server INTERNALDATE timestamp.
pub internal_date_since: Option<i64>,
/// Upper bound (inclusive) on the IMAP server INTERNALDATE timestamp.
pub internal_date_before: Option<i64>,
/// Lower bound (inclusive) on Bichon's archival (ingest) timestamp.
pub ingest_since: Option<i64>,
/// Upper bound (inclusive) on Bichon's archival (ingest) timestamp.
pub ingest_before: Option<i64>,
pub account_ids: Option<HashSet<u64>>,
pub mailbox_ids: Option<HashSet<u64>>,
pub min_size: Option<u64>,
@@ -64,6 +72,14 @@ pub enum SortBy {
#[default]
DATE,
SIZE,
/// Sort by the IMAP server INTERNALDATE timestamp.
#[serde(rename = "INTERNAL_DATE")]
#[cfg_attr(feature = "web-api", oai(rename = "INTERNAL_DATE"))]
InternalDate,
/// Sort by Bichon's archival (ingest) timestamp.
#[serde(rename = "INGEST_AT")]
#[cfg_attr(feature = "web-api", oai(rename = "INGEST_AT"))]
IngestAt,
}
#[derive(Debug, Clone, Default, Eq, PartialEq, Serialize, Deserialize)]

View File

@@ -4,7 +4,7 @@ use crate::{
error::{code::ErrorCode, BichonResult},
migrate::{
legacy::schema::SchemaTools,
store::{LegacyDirs, NewDirs, NewIndexWriter},
store::{LegacyDirs, NewIndexWriter},
},
raise_error,
settings::cli::SETTINGS,
@@ -121,7 +121,7 @@ fn is_dir_not_empty(path: &PathBuf) -> std::io::Result<bool> {
pub fn do_migrate_segment<F>(
batch_size: u32,
legacy: LegacyDirs,
new_dirs: NewDirs,
writer: &mut NewIndexWriter,
segment_index: usize,
mut on_progress: F,
) -> BichonResult<()>
@@ -226,8 +226,6 @@ where
drop(envelope_index);
// ── Phase 2: process EML docs, streaming one at a time ─────────────
let mut writer = NewIndexWriter::open(new_dirs)?;
let mut total_migrated = 0usize;
let mut total_skipped = 0usize;
@@ -308,7 +306,6 @@ where
chunk_start = chunk_end;
}
writer.finish_writers()?;
on_progress(&format!("DONE:{}:{}", total_migrated, total_skipped));
Ok(())
}

View File

@@ -86,13 +86,22 @@ pub fn detach_attachments_standalone(
for (raw_start, raw_end, att) in ranges {
let content_hash = compute_content_hash(att.contents());
blobs.push((
content_hash.clone(),
Bytes::copy_from_slice(&original_body[raw_start..raw_end]),
));
let body_len = original_body.len();
let raw_start = raw_start.min(body_len);
let raw_end = raw_end.min(body_len);
let range_valid = raw_start < raw_end;
let placeholder = format!("<<BICHON_DETACH_HASH:{}>>", &content_hash);
stripped_eml.splice(raw_start..raw_end, placeholder.as_bytes().iter().cloned());
if range_valid {
blobs.push((
content_hash.clone(),
Bytes::copy_from_slice(&original_body[raw_start..raw_end]),
));
}
if range_valid {
let placeholder = format!("<<BICHON_DETACH_HASH:{}>>", &content_hash);
stripped_eml.splice(raw_start..raw_end, placeholder.as_bytes().iter().cloned());
}
infos.push(AttachmentInfo {
filename: att.attachment_name().map(|n| n.to_string()),
@@ -100,7 +109,7 @@ pub fn detach_attachments_standalone(
inline: att
.content_disposition()
.map(|d| d.is_inline())
.unwrap_or(false),
.unwrap_or_else(|| att.content_id().is_some()),
file_type: att
.content_type()
.map(|ct| {
@@ -114,6 +123,9 @@ pub fn detach_attachments_standalone(
content_id: att.content_id().map(|id| id.to_string()),
content_hash,
is_message: att.is_message(),
extracted_text: None,
extracted_page_count: None,
extracted_is_ocr: false,
});
}
@@ -457,16 +469,10 @@ impl NewIndexWriter {
("attachment", &mut self.attachment_writer),
] {
if let Some(writer) = writer_opt.as_mut() {
let reader = writer
let seg_ids = writer
.index()
.reader()
.searchable_segment_ids()
.map_err(|e| raise_error!(format!("{:#?}", e), ErrorCode::InternalError))?;
let seg_ids: Vec<_> = reader
.searcher()
.segment_readers()
.iter()
.map(|r| r.segment_id())
.collect();
println!("merging {} {} segments...", seg_ids.len(), name);
if seg_ids.len() > 1 {
let _ = writer.merge(&seg_ids);

View File

@@ -18,7 +18,7 @@
use crate::{
common::signal::SIGNAL_MANAGER,
envelope::extractor::reattach_eml_content,
envelope::extractor::reattach_eml_content_self_healing,
error::{code::ErrorCode, BichonResult},
settings::dir::DATA_DIR_MANAGER,
};
@@ -68,7 +68,9 @@ impl BlobManager {
}
}
Err(e) => tracing::error!("Fjall email_ks error: {:?}", e),
_ => {}
Ok(true) => {
tracing::debug!("Email blob already exists (dedup): {}", &email_hash);
}
}
if let Some(attachments) = eml.attachments {
@@ -80,7 +82,9 @@ impl BlobManager {
}
}
Err(e) => tracing::error!("Fjall attach_ks error: {:?}", e),
_ => {}
Ok(true) => {
tracing::debug!("Attachment blob already exists (dedup): {}", &a_hash);
}
}
}
}
@@ -145,9 +149,18 @@ impl BlobManager {
res = receiver.recv() => {
match res {
Some(eml) => {
Self::process_detached_email(eml, &email_ks, &attach_ks);
let mut batch = vec![eml];
while let Ok(next_eml) = receiver.try_recv() {
Self::process_detached_email(next_eml, &email_ks, &attach_ks);
batch.push(next_eml);
}
let email_ks = email_ks.clone();
let attach_ks = attach_ks.clone();
if let Err(e) = tokio::task::spawn_blocking(move || {
for eml in batch {
Self::process_detached_email(eml, &email_ks, &attach_ks);
}
}).await {
tracing::error!("BlobManager: spawn_blocking join error: {:#?}", e);
}
}
None => {
@@ -158,16 +171,25 @@ impl BlobManager {
}
_ = shutdown.recv() => {
receiver.close();
let remaining = receiver.len();
let mut remaining = Vec::new();
while let Some(eml) = receiver.recv().await {
remaining.push(eml);
}
tracing::info!(
"BlobManager: Shutdown signal received. Processing {} remaining tasks...",
remaining
remaining.len()
);
while let Some(eml) = receiver.recv().await {
Self::process_detached_email(eml, &email_ks, &attach_ks);
if !remaining.is_empty() {
let email_ks = email_ks.clone();
let attach_ks = attach_ks.clone();
if let Err(e) = tokio::task::spawn_blocking(move || {
for eml in remaining {
Self::process_detached_email(eml, &email_ks, &attach_ks);
}
}).await {
tracing::error!("BlobManager: shutdown spawn_blocking join error: {:#?}", e);
}
}
tracing::info!("BlobManager: All remaining tasks processed. Closing Fjall.");
break;
}
@@ -185,7 +207,9 @@ impl BlobManager {
}
pub async fn queue(&self, email: DetachedEmail) {
let _ = self.sender.send(email).await;
if let Err(e) = self.sender.send(email).await {
tracing::error!("BlobManager channel closed, email lost: {:#?}", e);
}
}
pub fn get_email(&self, content_hash: &str) -> BichonResult<Option<Bytes>> {
@@ -226,7 +250,13 @@ impl BlobManager {
}
}
pub fn get_reader(account_id: u64, eid: String) -> BichonResult<Cursor<Bytes>> {
let (_, data) = reattach_eml_content(account_id, eid)?;
/// Returns a reader over the raw EML for an indexed message.
///
/// If the message's content blob is missing from the blob store, it is fetched
/// on demand from the IMAP server, persisted, and returned (self-healing). The
/// underlying "content not found" error is only surfaced if that on-demand
/// fetch itself fails.
pub async fn get_reader(account_id: u64, eid: String) -> BichonResult<Cursor<Bytes>> {
let (_, data) = reattach_eml_content_self_healing(account_id, eid).await?;
Ok(Cursor::new(data))
}

View File

@@ -38,8 +38,8 @@ use crate::{
store::tantivy::{
fatal_commit,
fields::{
F_ATTACHMENT_CATEGORY, F_ATTACHMENT_CONTENT_TYPE, F_ATTACHMENT_EXT, F_DATE, F_SIZE,
F_TAGS,
F_ATTACHMENT_CATEGORY, F_ATTACHMENT_CONTENT_TYPE, F_ATTACHMENT_EXT, F_DATE,
F_INGEST_AT, F_SIZE, F_TAGS,
},
model::{extract_senders, AttachmentModel},
schema::SchemaTools,
@@ -154,7 +154,11 @@ impl IndexManager {
"Tantivy: Reached threshold ({} docs), committing...",
pending_count
);
fatal_commit(&mut writer);
tokio::task::block_in_place(|| fatal_commit(&mut writer));
tracing::debug!(
"Tantivy attach: committed {} docs, pending reset to 0",
pending_count
);
pending_count = 0;
commit_interval.reset();
}
@@ -163,7 +167,7 @@ impl IndexManager {
tracing::info!("Tantivy: Receiver closed. Finalizing...");
if pending_count > 0 {
let mut writer = writer.lock().await;
fatal_commit(&mut writer);
tokio::task::block_in_place(|| fatal_commit(&mut writer));
}
break;
},
@@ -172,16 +176,19 @@ impl IndexManager {
_ = commit_interval.tick() => {
if pending_count > 0 {
let mut writer = writer.lock().await;
fatal_commit(&mut writer);
tracing::debug!(
"Tantivy attach: periodic commit ({} docs pending)",
pending_count
);
tokio::task::block_in_place(|| fatal_commit(&mut writer));
pending_count = 0;
tracing::debug!("Tantivy: Periodic commit finished.");
}
}
_ = shutdown.recv() => {
tracing::info!("Tantivy: Shutdown signal received. Performing final commit...");
if pending_count > 0 {
let mut writer = writer.lock().await;
fatal_commit(&mut writer);
tokio::task::block_in_place(|| fatal_commit(&mut writer));
}
tracing::info!("Tantivy: Shutdown cleanup complete.");
break;
@@ -864,6 +871,30 @@ impl IndexManager {
.map_err(|e| raise_error!(format!("{:#?}", e), ErrorCode::InternalError))?;
attachment_docs = size_docs.into_iter().map(|(_, addr)| addr).collect();
}
// Attachments carry no IMAP INTERNALDATE; fall back to the
// attachment's own date field so the sort remains well defined.
SortBy::InternalDate => {
let date_docs: Vec<(Option<i64>, DocAddress)> = searcher
.search(
&query,
&TopDocs::with_limit(page_size as usize)
.and_offset(offset as usize)
.order_by_fast_field(F_DATE, order),
)
.map_err(|e| raise_error!(format!("{:#?}", e), ErrorCode::InternalError))?;
attachment_docs = date_docs.into_iter().map(|(_, addr)| addr).collect();
}
SortBy::IngestAt => {
let ingest_at_docs: Vec<(Option<i64>, DocAddress)> = searcher
.search(
&query,
&TopDocs::with_limit(page_size as usize)
.and_offset(offset as usize)
.order_by_fast_field(F_INGEST_AT, order),
)
.map_err(|e| raise_error!(format!("{:#?}", e), ErrorCode::InternalError))?;
attachment_docs = ingest_at_docs.into_iter().map(|(_, addr)| addr).collect();
}
}
let mut result = Vec::new();

View File

@@ -255,11 +255,22 @@ fn dedup_account(
// uidvalidity, which is required for correct incremental sync.
entries.sort_by_key(|e| std::cmp::Reverse(e.ingest_at));
eprintln!(
"DEBUG Phase2: key={_key:?} kept={} deleting={}",
entries[0].email_id,
tracing::debug!(
"dedup: account={} mailbox={} hash={}: {} copies, keeping eid={} ingest_at={}, deleting {}",
account_id,
_key.0,
&_key.1,
entries.len(),
&entries[0].email_id,
entries[0].ingest_at,
entries.len() - 1
);
// eprintln!(
// "DEBUG Phase2: key={_key:?} kept={} deleting={}",
// entries[0].email_id,
// entries.len() - 1
// );
// Keep entries[0], soft-delete everything else via term query on f_id
for entry in &entries[1..] {
eprintln!(

View File

@@ -0,0 +1,641 @@
use std::collections::HashMap;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{LazyLock, Mutex};
use crate::store::tantivy::envelope::ENVELOPE_MANAGER;
use crate::store::tantivy::fields::{F_ACCOUNT_ID, F_CONTENT_HASH, F_INGEST_AT, F_MAILBOX_ID};
use crate::utc_now;
/// Max entries before evicting the oldest.
/// At ~152 bytes/entry, 300_000 ≈ 45 MB, within the 50 MB budget.
const MAX_ENTRIES: usize = 300_000;
/// Fraction of entries to keep when evicting (newest 3/4).
const KEEP_FRACTION_NUM: usize = 3;
const KEEP_FRACTION_DEN: usize = 4;
/// Populate only loads entries ingested within this window.
const POPULATE_WINDOW_MS: i64 = 7 * 24 * 60 * 60 * 1000; // 7 days
pub static DEDUP_CACHE: LazyLock<DedupCache> = LazyLock::new(DedupCache::new);
pub struct DedupCache {
entries: Mutex<HashMap<(u64, u64, String), i64>>,
max_entries: usize,
populated: AtomicBool,
}
impl DedupCache {
fn new() -> Self {
Self {
entries: Mutex::new(HashMap::new()),
max_entries: MAX_ENTRIES,
populated: AtomicBool::new(false),
}
}
#[cfg(test)]
fn new_for_test() -> Self {
Self {
entries: Mutex::new(HashMap::new()),
max_entries: MAX_ENTRIES,
populated: AtomicBool::new(true),
}
}
#[cfg(test)]
fn new_for_test_small(max_entries: usize) -> Self {
Self {
entries: Mutex::new(HashMap::new()),
max_entries,
populated: AtomicBool::new(true),
}
}
/// Returns true if this `(account_id, mailbox_id, content_hash)` triple
/// has already been seen.
///
/// On the very first call the cache is populated from the Tantivy index
/// FAST columns (only entries ingested within [`POPULATE_WINDOW_MS`]).
/// If that scan fails the cache starts empty and still operates correctly
/// for newly-arriving emails.
pub fn contains(&self, account_id: u64, mailbox_id: u64, hash: &str) -> bool {
self.ensure_populated();
let entries = self.entries.lock().unwrap();
entries.contains_key(&(account_id, mailbox_id, hash.to_string()))
}
/// Insert a triple into the cache after it has been queued for indexing.
///
/// Each entry is stamped with the current time. When the cache exceeds
/// [`MAX_ENTRIES`], the oldest entries are evicted, keeping the newest
/// `MAX_ENTRIES * 3/4`.
pub fn insert(&self, account_id: u64, mailbox_id: u64, hash: &str) {
let mut entries = self.entries.lock().unwrap();
let now = utc_now!();
entries.insert((account_id, mailbox_id, hash.to_string()), now);
if entries.len() > self.max_entries {
let keep = self.max_entries * KEEP_FRACTION_NUM / KEEP_FRACTION_DEN;
let mut vec: Vec<_> = entries.drain().collect();
// Sort descending by timestamp (newest first)
vec.sort_by(|a, b| b.1.cmp(&a.1));
for (k, v) in vec.into_iter().take(keep) {
entries.insert(k, v);
}
tracing::warn!(
"DedupCache evicted oldest entries, kept {}/{}",
entries.len(),
keep
);
}
}
// ── private ──────────────────────────────────────────────────────────────
fn ensure_populated(&self) {
if self.populated.load(Ordering::Acquire) {
return;
}
self.do_populate();
}
fn do_populate(&self) {
if self
.populated
.compare_exchange(false, true, Ordering::AcqRel, Ordering::Relaxed)
.is_err()
{
return;
}
let reader = match ENVELOPE_MANAGER.create_reader() {
Ok(r) => r,
Err(e) => {
tracing::warn!("DedupCache: failed to create reader for populate: {e}");
return;
}
};
let searcher = reader.searcher();
let cutoff = utc_now!() - POPULATE_WINDOW_MS;
let mut entries = self.entries.lock().unwrap();
for segment_reader in searcher.segment_readers() {
let account_col = match segment_reader.fast_fields().u64(F_ACCOUNT_ID) {
Ok(c) => c,
Err(_) => continue,
};
let mailbox_col = match segment_reader.fast_fields().u64(F_MAILBOX_ID) {
Ok(c) => c,
Err(_) => continue,
};
let hash_col = match segment_reader.fast_fields().str(F_CONTENT_HASH) {
Ok(Some(c)) => c,
_ => continue,
};
let ingest_col = match segment_reader.fast_fields().i64(F_INGEST_AT) {
Ok(c) => c,
Err(_) => continue,
};
let max_doc = segment_reader.max_doc();
for doc_id in 0..max_doc {
if segment_reader.is_deleted(doc_id) {
continue;
}
let ingest_at = ingest_col.values.get_val(doc_id);
if ingest_at < cutoff {
continue;
}
let account_id = account_col.values.get_val(doc_id);
let mailbox_id = mailbox_col.values.get_val(doc_id);
let hash_ord = hash_col
.ords()
.values_for_doc(doc_id as u32)
.next()
.unwrap_or(0);
let mut hash_buf = String::new();
if hash_col.ord_to_str(hash_ord, &mut hash_buf).is_err() {
continue;
}
entries.insert((account_id, mailbox_id, hash_buf), ingest_at);
}
}
tracing::info!(
"DedupCache populated with {} entries from index (cutoff {}d ago)",
entries.len(),
POPULATE_WINDOW_MS / (24 * 60 * 60 * 1000),
);
}
/// Remove all entries for a specific account.
pub fn remove_by_account(&self, account_id: u64) {
let mut entries = self.entries.lock().unwrap();
let before = entries.len();
entries.retain(|(aid, _, _), _| *aid != account_id);
let removed = before - entries.len();
if removed > 0 {
tracing::info!(
"DedupCache: removed {} entries for account {}",
removed,
account_id
);
}
}
/// Remove all entries for a specific mailbox (across all accounts).
pub fn remove_by_mailbox(&self, mailbox_id: u64) {
let mut entries = self.entries.lock().unwrap();
let before = entries.len();
entries.retain(|(_, mid, _), _| *mid != mailbox_id);
let removed = before - entries.len();
if removed > 0 {
tracing::info!(
"DedupCache: removed {} entries for mailbox {}",
removed,
mailbox_id
);
}
}
/// Remove a specific triple (most precise removal).
pub fn remove(&self, account_id: u64, mailbox_id: u64, hash: &str) {
let mut entries = self.entries.lock().unwrap();
if entries
.remove(&(account_id, mailbox_id, hash.to_string()))
.is_some()
{
tracing::debug!(
"DedupCache: removed specific entry ({}, {}, {})",
account_id,
mailbox_id,
hash
);
}
}
}
// ── Tests ─────────────────────────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
use crate::store::tantivy::fields::EmailFields;
use crate::store::tantivy::schema::SchemaTools;
use crate::store::tantivy::tokenizers::EuroTokenizer;
use std::fs;
use tantivy::{Index, TantivyDocument};
fn temp_dir(name: &str) -> std::path::PathBuf {
let dir = std::env::temp_dir()
.join("bichon-dedup-cache-test")
.join(name)
.join(uuid::Uuid::new_v4().to_string());
fs::create_dir_all(&dir).unwrap();
dir
}
// ── basic contains / insert ─────────────────────────────────────────────
#[test]
fn contains_after_insert() {
let cache = DedupCache::new_for_test();
assert!(!cache.contains(1, 10, "hash-aaa"));
cache.insert(1, 10, "hash-aaa");
assert!(cache.contains(1, 10, "hash-aaa"));
}
#[test]
fn different_hash_not_matched() {
let cache = DedupCache::new_for_test();
cache.insert(1, 10, "hash-aaa");
assert!(!cache.contains(1, 10, "hash-bbb"));
}
#[test]
fn different_account_not_matched() {
let cache = DedupCache::new_for_test();
cache.insert(1, 10, "hash-aaa");
assert!(!cache.contains(2, 10, "hash-aaa"));
}
#[test]
fn different_mailbox_not_matched() {
let cache = DedupCache::new_for_test();
cache.insert(1, 10, "hash-aaa");
assert!(!cache.contains(1, 20, "hash-aaa"));
}
#[test]
fn cross_account_allowed() {
let cache = DedupCache::new_for_test();
cache.insert(1, 10, "hash-aaa");
cache.insert(2, 10, "hash-aaa");
assert!(cache.contains(2, 10, "hash-aaa"));
assert!(cache.contains(1, 10, "hash-aaa"));
}
#[test]
fn cross_mailbox_allowed() {
let cache = DedupCache::new_for_test();
cache.insert(1, 10, "hash-aaa");
cache.insert(1, 20, "hash-aaa");
assert!(cache.contains(1, 20, "hash-aaa"));
assert!(cache.contains(1, 10, "hash-aaa"));
}
// ── time-based eviction ─────────────────────────────────────────────────
#[test]
fn eviction_keeps_newest() {
let cap = 100;
let cache = DedupCache::new_for_test_small(cap);
// Fill to exact capacity. Entry hash-0 is oldest.
for i in 0..cap {
cache.insert(1, 1, &format!("hash-{}", i));
std::thread::sleep(std::time::Duration::from_micros(100));
}
assert!(cache.contains(1, 1, "hash-0"));
assert!(cache.contains(1, 1, &format!("hash-{}", cap - 1)));
// One more triggers eviction
cache.insert(1, 1, "hash-overflow");
// Newest survives, oldest evicted
assert!(cache.contains(1, 1, "hash-overflow"));
assert!(cache.contains(1, 1, &format!("hash-{}", cap - 1)));
assert!(!cache.contains(1, 1, "hash-0"));
let keep = cap * KEEP_FRACTION_NUM / KEEP_FRACTION_DEN;
assert!(cache.entries.lock().unwrap().len() <= keep);
}
// ── memory bound ────────────────────────────────────────────────────────
#[test]
fn memory_bound_within_budget() {
let cache = DedupCache::new_for_test();
for i in 0..MAX_ENTRIES {
cache.insert(1, 1, &format!("{:064x}", i));
}
let entries = cache.entries.lock().unwrap();
assert_eq!(entries.len(), MAX_ENTRIES);
let capacity = entries.capacity();
// HashMap with (u64,u64,String) key + i64 value ≈ 112 + map overhead
let approx_bytes = capacity * (104 + 8 + 8);
let approx_mb = approx_bytes as f64 / (1024.0 * 1024.0);
println!(
"DedupCache: {} entries, {} buckets, ~{:.1} MB",
MAX_ENTRIES, capacity, approx_mb
);
assert!(
approx_mb < 55.0,
"memory estimate {:.1} MB exceeds 55 MB buffer",
approx_mb
);
}
// ── populate guard ──────────────────────────────────────────────────────
#[test]
fn populate_cas_is_idempotent() {
let cache = DedupCache::new_for_test();
assert!(cache.populated.load(Ordering::Acquire));
cache.ensure_populated();
assert!(cache.populated.load(Ordering::Acquire));
cache.do_populate();
}
// ── populate from test index ────────────────────────────────────────────
fn build_test_index() -> (Index, &'static EmailFields) {
let dir = temp_dir("populate");
let schema = SchemaTools::email_schema();
let fields = SchemaTools::email_fields();
let index = Index::create_in_dir(&dir, schema).unwrap();
index.tokenizers().register("euro", EuroTokenizer::new());
(index, fields)
}
fn add_email_doc(
fields: &EmailFields,
writer: &mut tantivy::IndexWriter,
account: u64,
mailbox: u64,
hash: &str,
ingest_at: i64,
) {
let mut doc = TantivyDocument::new();
doc.add_u64(fields.f_account_id, account);
doc.add_u64(fields.f_mailbox_id, mailbox);
doc.add_text(fields.f_content_hash, hash);
doc.add_i64(fields.f_ingest_at, ingest_at);
doc.add_text(fields.f_id, &uuid::Uuid::new_v4().to_string());
doc.add_text(fields.f_subject, "test");
doc.add_text(fields.f_body, "test body");
doc.add_u64(fields.f_uid, 1);
doc.add_i64(fields.f_date, 1);
doc.add_i64(fields.f_internal_date, 1);
doc.add_u64(fields.f_size, 100);
writer.add_document(doc).unwrap();
}
#[test]
fn populate_reads_all_docs_in_window() {
let (index, fields) = build_test_index();
let mut writer = index.writer_with_num_threads(1, 50_000_000).unwrap();
let recent = utc_now!();
add_email_doc(&fields, &mut writer, 1, 10, "hash-recent", recent);
add_email_doc(&fields, &mut writer, 2, 10, "hash-recent", recent);
add_email_doc(&fields, &mut writer, 1, 20, "hash-recent", recent);
writer.commit().unwrap();
drop(writer);
let reader = index.reader().unwrap();
let cache = DedupCache::new_for_test();
{
let searcher = reader.searcher();
let cutoff = utc_now!() - POPULATE_WINDOW_MS;
let mut entries = cache.entries.lock().unwrap();
entries.clear();
for segment_reader in searcher.segment_readers() {
let account_col = segment_reader.fast_fields().u64(F_ACCOUNT_ID).unwrap();
let mailbox_col = segment_reader.fast_fields().u64(F_MAILBOX_ID).unwrap();
let hash_col = segment_reader
.fast_fields()
.str(F_CONTENT_HASH)
.unwrap()
.unwrap();
let ingest_col = segment_reader.fast_fields().i64(F_INGEST_AT).unwrap();
let max_doc = segment_reader.max_doc();
for doc_id in 0..max_doc {
if segment_reader.is_deleted(doc_id) {
continue;
}
let ingest_at = ingest_col.values.get_val(doc_id);
if ingest_at < cutoff {
continue;
}
let account_id = account_col.values.get_val(doc_id);
let mailbox_id = mailbox_col.values.get_val(doc_id);
let hash_ord = hash_col
.ords()
.values_for_doc(doc_id as u32)
.next()
.unwrap_or(0);
let mut hash_buf = String::new();
hash_col.ord_to_str(hash_ord, &mut hash_buf).unwrap();
entries.insert((account_id, mailbox_id, hash_buf), ingest_at);
}
}
}
assert_eq!(cache.entries.lock().unwrap().len(), 3);
assert!(cache.contains(1, 10, "hash-recent"));
assert!(cache.contains(2, 10, "hash-recent"));
assert!(cache.contains(1, 20, "hash-recent"));
}
#[test]
fn populate_skips_old_entries() {
let (index, fields) = build_test_index();
let mut writer = index.writer_with_num_threads(1, 50_000_000).unwrap();
let recent = utc_now!();
let old = recent - POPULATE_WINDOW_MS - 60_000; // 1 minute past the window
add_email_doc(&fields, &mut writer, 1, 10, "hash-recent", recent);
add_email_doc(&fields, &mut writer, 1, 10, "hash-old", old);
writer.commit().unwrap();
drop(writer);
let reader = index.reader().unwrap();
let cache = DedupCache::new_for_test();
{
let searcher = reader.searcher();
let cutoff = utc_now!() - POPULATE_WINDOW_MS;
let mut entries = cache.entries.lock().unwrap();
entries.clear();
for segment_reader in searcher.segment_readers() {
let account_col = segment_reader.fast_fields().u64(F_ACCOUNT_ID).unwrap();
let mailbox_col = segment_reader.fast_fields().u64(F_MAILBOX_ID).unwrap();
let hash_col = segment_reader
.fast_fields()
.str(F_CONTENT_HASH)
.unwrap()
.unwrap();
let ingest_col = segment_reader.fast_fields().i64(F_INGEST_AT).unwrap();
let max_doc = segment_reader.max_doc();
for doc_id in 0..max_doc {
if segment_reader.is_deleted(doc_id) {
continue;
}
let ingest_at = ingest_col.values.get_val(doc_id);
if ingest_at < cutoff {
continue;
}
let account_id = account_col.values.get_val(doc_id);
let mailbox_id = mailbox_col.values.get_val(doc_id);
let hash_ord = hash_col
.ords()
.values_for_doc(doc_id as u32)
.next()
.unwrap_or(0);
let mut hash_buf = String::new();
hash_col.ord_to_str(hash_ord, &mut hash_buf).unwrap();
entries.insert((account_id, mailbox_id, hash_buf), ingest_at);
}
}
}
assert!(cache.contains(1, 10, "hash-recent"));
assert!(!cache.contains(1, 10, "hash-old"));
assert_eq!(cache.entries.lock().unwrap().len(), 1);
}
#[test]
fn populate_skips_deleted_docs() {
let (index, fields) = build_test_index();
let mut writer = index.writer_with_num_threads(1, 50_000_000).unwrap();
let recent = utc_now!();
add_email_doc(&fields, &mut writer, 1, 10, "hash-keep", recent);
add_email_doc(&fields, &mut writer, 1, 10, "hash-delete", recent);
writer.commit().unwrap();
let term = tantivy::Term::from_field_text(fields.f_content_hash, "hash-delete");
writer.delete_term(term);
writer.commit().unwrap();
drop(writer);
let reader = index.reader().unwrap();
let cache = DedupCache::new_for_test();
{
let searcher = reader.searcher();
let cutoff = utc_now!() - POPULATE_WINDOW_MS;
let mut entries = cache.entries.lock().unwrap();
entries.clear();
for segment_reader in searcher.segment_readers() {
let account_col = segment_reader.fast_fields().u64(F_ACCOUNT_ID).unwrap();
let mailbox_col = segment_reader.fast_fields().u64(F_MAILBOX_ID).unwrap();
let hash_col = segment_reader
.fast_fields()
.str(F_CONTENT_HASH)
.unwrap()
.unwrap();
let ingest_col = segment_reader.fast_fields().i64(F_INGEST_AT).unwrap();
let max_doc = segment_reader.max_doc();
for doc_id in 0..max_doc {
if segment_reader.is_deleted(doc_id) {
continue;
}
let ingest_at = ingest_col.values.get_val(doc_id);
if ingest_at < cutoff {
continue;
}
let account_id = account_col.values.get_val(doc_id);
let mailbox_id = mailbox_col.values.get_val(doc_id);
let hash_ord = hash_col
.ords()
.values_for_doc(doc_id as u32)
.next()
.unwrap_or(0);
let mut hash_buf = String::new();
hash_col.ord_to_str(hash_ord, &mut hash_buf).unwrap();
entries.insert((account_id, mailbox_id, hash_buf), ingest_at);
}
}
}
assert!(cache.contains(1, 10, "hash-keep"));
assert!(!cache.contains(1, 10, "hash-delete"));
}
// ── removal methods ─────────────────────────────────────────────────────
#[test]
fn remove_by_account_works() {
let cache = DedupCache::new_for_test();
cache.insert(1, 10, "hash-a1");
cache.insert(1, 20, "hash-a2");
cache.insert(2, 10, "hash-b1");
cache.insert(2, 30, "hash-b2");
cache.insert(1, 10, "hash-a3");
assert_eq!(cache.entries.lock().unwrap().len(), 5);
cache.remove_by_account(1);
let entries = cache.entries.lock().unwrap();
assert_eq!(entries.len(), 2);
assert!(!entries.contains_key(&(1, 10, "hash-a1".to_string())));
assert!(!entries.contains_key(&(1, 20, "hash-a2".to_string())));
assert!(!entries.contains_key(&(1, 10, "hash-a3".to_string())));
assert!(entries.contains_key(&(2, 10, "hash-b1".to_string())));
assert!(entries.contains_key(&(2, 30, "hash-b2".to_string())));
}
#[test]
fn remove_by_mailbox_works() {
let cache = DedupCache::new_for_test();
cache.insert(1, 10, "hash-1");
cache.insert(1, 20, "hash-2");
cache.insert(2, 10, "hash-3");
cache.insert(3, 20, "hash-4");
cache.insert(1, 10, "hash-5");
cache.remove_by_mailbox(10);
let entries = cache.entries.lock().unwrap();
assert_eq!(entries.len(), 2);
assert!(entries.contains_key(&(1, 20, "hash-2".to_string())));
assert!(entries.contains_key(&(3, 20, "hash-4".to_string())));
assert!(!entries.contains_key(&(1, 10, "hash-1".to_string())));
assert!(!entries.contains_key(&(2, 10, "hash-3".to_string())));
}
#[test]
fn remove_specific_triple_works() {
let cache = DedupCache::new_for_test();
cache.insert(1, 10, "hash-aaa");
cache.insert(1, 10, "hash-bbb");
cache.insert(2, 20, "hash-aaa");
assert!(cache.contains(1, 10, "hash-aaa"));
assert!(cache.contains(1, 10, "hash-bbb"));
assert!(cache.contains(2, 20, "hash-aaa"));
cache.remove(1, 10, "hash-aaa");
assert!(!cache.contains(1, 10, "hash-aaa"));
assert!(cache.contains(1, 10, "hash-bbb"));
assert!(cache.contains(2, 20, "hash-aaa"));
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -25,6 +25,7 @@ use crate::{
pub mod attachment;
pub mod dedup;
pub mod dedup_cache;
pub mod envelope;
pub mod fields;
pub mod filter;

View File

@@ -17,9 +17,63 @@
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use regex::Regex;
use std::panic;
use std::sync::LazyLock;
use tracing::error;
/// Removes remote content references from HTML email body.
///
/// Strips attributes that load content from http:// or https:// URLs,
/// keeping data: URIs and cid: references intact. Does NOT affect
/// navigation links (<a href>).
pub fn block_remote_content(html: &str) -> String {
let mut result = html.to_string();
// 1. Strip src, poster, data attributes with remote URLs.
// These always load content regardless of the tag.
static SRC_ATTR_RE: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r#"(?i)\s+(src|poster|data)\s*=\s*["'][^"']*(?:https?://|//)[^"']*["']"#).unwrap()
});
result = SRC_ATTR_RE.replace_all(&result, "").to_string();
// 2. Strip srcset attributes with remote URLs.
static SRCSET_ATTR_RE: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r#"(?i)\s+srcset\s*=\s*["'][^"']*(?:https?://|//)[^"']*["']"#).unwrap()
});
result = SRCSET_ATTR_RE.replace_all(&result, "").to_string();
// 3. Strip href on <link> tags (stylesheets), never <a> links.
static LINK_HREF_RE: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r#"(?i)(<link\b[^>]*)\s+href\s*=\s*["'][^"']*(?:https?://|//)[^"']*["']"#).unwrap()
});
result = LINK_HREF_RE.replace_all(&result, "$1").to_string();
// 4. Strip CSS url() references with remote URLs in inline styles.
static CSS_URL_RE: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r#"(?i)url\(\s*["']?\s*(?:https?://|//)[^)"'\s]*\s*["']?\s*\)"#).unwrap()
});
result = CSS_URL_RE.replace_all(&result, "").to_string();
// 5. Strip @import url(...) with remote URLs inside <style> blocks.
static IMPORT_URL_RE: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(
r#"(?i)@import\s+url\(\s*["']?\s*(?:https?://|//)[^)"'\s]*\s*["']?\s*\)\s*;"#,
)
.unwrap()
});
result = IMPORT_URL_RE.replace_all(&result, "").to_string();
// 6. Strip background attribute on <body> with remote URLs.
static BODY_BG_RE: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r#"(?i)(<body\b[^>]*)\s+background\s*=\s*["'][^"']*(?:https?://|//)[^"']*["']"#)
.unwrap()
});
result = BODY_BG_RE.replace_all(&result, "$1").to_string();
result
}
pub fn extract_text(html: String) -> String {
let result = panic::catch_unwind(|| {
html2text::config::plain()
@@ -88,4 +142,132 @@ mod tests {
let text = extract_text(html);
assert!(text.contains("Click here"));
}
mod block_remote {
use super::*;
#[test]
fn strips_img_src_http() {
let html = r#"<img src="https://tracker.example.com/pixel.gif" alt="x">"#;
let result = block_remote_content(html);
assert!(!result.contains("https://tracker.example.com"));
assert!(result.contains("alt=")); // other attrs preserved
}
#[test]
fn strips_img_src_protocol_relative() {
let html = r#"<img src="//tracker.example.com/pixel.gif">"#;
let result = block_remote_content(html);
assert!(!result.contains("//tracker.example.com"));
}
#[test]
fn preserves_data_uri() {
let html = r#"<img src="data:image/png;base64,ABC123" alt="embedded">"#;
let result = block_remote_content(html);
assert!(result.contains("data:image/png;base64,ABC123"));
}
#[test]
fn preserves_cid_reference() {
let html = r#"<img src="cid:abc123@example.com" alt="inline">"#;
let result = block_remote_content(html);
assert!(result.contains("cid:abc123@example.com"));
}
#[test]
fn preserves_anchor_href() {
let html = r#"<a href="https://example.com/page">Click</a>"#;
let result = block_remote_content(html);
assert!(result.contains(r#"href="https://example.com/page""#));
}
#[test]
fn strips_link_stylesheet_href() {
let html =
r#"<link rel="stylesheet" href="https://fonts.example.com/font.css">"#;
let result = block_remote_content(html);
assert!(!result.contains("https://fonts.example.com"));
assert!(result.contains("<link")); // tag preserved
}
#[test]
fn strips_script_src() {
let html = r#"<script src="https://evil.example.com/malware.js"></script>"#;
let result = block_remote_content(html);
assert!(!result.contains("https://evil.example.com"));
}
#[test]
fn strips_iframe_src() {
let html = r#"<iframe src="https://ads.example.com/banner"></iframe>"#;
let result = block_remote_content(html);
assert!(!result.contains("https://ads.example.com"));
}
#[test]
fn strips_css_url_in_style() {
let html = r#"<div style="background: url(https://tracker.example.com/bg.jpg)"></div>"#;
let result = block_remote_content(html);
assert!(!result.contains("https://tracker.example.com"));
}
#[test]
fn strips_css_url_protocol_relative() {
let html = r#"<div style="background: url(//tracker.example.com/bg.jpg)"></div>"#;
let result = block_remote_content(html);
assert!(!result.contains("//tracker.example.com"));
}
#[test]
fn strips_css_import() {
let html =
r#"<style>@import url("https://fonts.example.com/font.css");</style>"#;
let result = block_remote_content(html);
assert!(!result.contains("https://fonts.example.com"));
}
#[test]
fn strips_video_poster() {
let html = r#"<video poster="https://cdn.example.com/thumb.jpg"></video>"#;
let result = block_remote_content(html);
assert!(!result.contains("https://cdn.example.com"));
}
#[test]
fn strips_srcset() {
let html =
r#"<img srcset="https://cdn.example.com/img1.jpg 1x, https://cdn.example.com/img2.jpg 2x">"#;
let result = block_remote_content(html);
assert!(!result.contains("https://cdn.example.com"));
}
#[test]
fn strips_body_background() {
let html = r#"<body background="https://tracker.example.com/bg.jpg">"#;
let result = block_remote_content(html);
assert!(!result.contains("https://tracker.example.com"));
assert!(result.contains("<body"));
}
#[test]
fn handles_mixed_content() {
let html = r#"
<html>
<body>
<img src="https://spy.example.com/pixel.gif" width="1" height="1">
<img src="data:image/png;base64,OK123" alt="ok">
<a href="https://example.com/read-more">Read more</a>
<div style="background: url(https://tracker.example.com/bg.jpg) no-repeat"></div>
</body>
</html>"#;
let result = block_remote_content(html);
// Remote content gone
assert!(!result.contains("spy.example.com"));
assert!(!result.contains("tracker.example.com"));
// Safe content preserved
assert!(result.contains("data:image/png;base64,OK123"));
assert!(result.contains(r#"href="https://example.com/read-more""#));
}
}
}

View File

@@ -3,6 +3,10 @@ name = "bichon-server"
version.workspace = true
edition.workspace = true
[features]
default = ["embed-web"]
embed-web = ["dep:rust-embed"]
[dependencies]
bichon-core = { path = "../core", features = ["web-api"] }
@@ -17,7 +21,7 @@ poem-openapi = { version = "5.1.16", features = [
"swagger-ui",
"email",
] }
rust-embed = "8.11.0"
rust-embed = { version = "8.11.0", optional = true }
email_address.workspace = true
serde.workspace = true
serde_json.workspace = true
@@ -29,7 +33,6 @@ tracing.workspace = true
tokio.workspace = true
http.workspace = true
urlencoding.workspace = true
mimalloc.workspace = true
[dev-dependencies]
poem = { version = "3.1.12", features = ["test"] }

135
crates/server/src/lib.rs Normal file
View File

@@ -0,0 +1,135 @@
//
// Copyright (c) 2025-2026 rustmailer.com (https://rustmailer.com)
//
// This file is part of the Bichon Email Archiving Project
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
pub mod common;
pub mod error;
pub mod rest;
use std::sync::LazyLock;
use bichon_core::{
bichon_version,
cache::imap::task::SYNC_TASKS,
common::{rustls::BichonTls, signal::SignalManager},
context::{executors::BichonContext, Initialize},
database::manager::DB_MANAGER,
error::{code::ErrorCode, BichonResult},
logger,
migrate::check_data_status,
raise_error,
settings::{cli::SETTINGS, dir::DataDirManager},
store::{
blob::BLOB_MANAGER,
tantivy::{attachment::ATTACHMENT_MANAGER, envelope::ENVELOPE_MANAGER},
},
tasks::PeriodicTasks,
users::manager::UserManager,
};
use bichon_smtp::server::{start_smtp_server, SmtpServer};
use tracing::{error, info};
pub async fn run() -> BichonResult<()> {
logger::initialize_logging();
info!(
r#"
_ _ _
| | (_) | |
| |__ _ ___ | |__ ___ _ __
| '_ \ | | / __|| '_ \ / _ \ | '_ \
| |_) || || (__ | | | || (_) || | | |
|_.__/ |_| \___||_| |_| \___/ |_| |_|
"#
);
info!("Starting bichon-server");
info!("Version: {}", bichon_version!());
info!("Git: [{}]", env!("GIT_HASH"));
info!("GitHub: https://github.com/rustmailer/bichon");
match check_data_status() {
Ok(false) => {
error!("Incompatible data format detected.");
error!("Your data was created by an older version of Bichon and must be migrated before use.");
error!("Please stop the Bichon v0.3.7 service before migration.");
error!("Please run: bichon-admin");
error!("Documentation: https://github.com/rustmailer/bichon/wiki/Bichon-Data-Migration:-v0.3.7-%E2%86%92-v1.0");
return Err(raise_error!(
"Legacy data layout detected".into(),
ErrorCode::InternalError
));
}
Err(e) => {
error!("Failed to check data layout: {:#?}", e);
return Err(raise_error!(format!("{:#?}", e), ErrorCode::InternalError));
}
Ok(true) => {}
}
if let Err(error) = initialize().await {
eprintln!("{:?}", error);
return Err(error);
}
let periodic_tasks = PeriodicTasks::setup();
let mut smtp_service: Option<SmtpServer> = None;
if SETTINGS.bichon_enable_smtp {
info!("SMTP service is enabled, starting...");
match start_smtp_server().await {
Ok(server) => {
info!("SMTP server listening on: {}", server.smtp_addr);
smtp_service = Some(server);
}
Err(e) => {
error!("Failed to start SMTP server: {}", e);
return Err(raise_error!(format!("{:#?}", e), ErrorCode::InternalError));
}
}
} else {
info!("SMTP service is disabled by configuration.");
}
rest::start_http_server().await?;
periodic_tasks.shutdown().await;
if let Some(server) = smtp_service {
info!("Shutting down SMTP server...");
server.stop().await;
info!("SMTP server stopped.");
}
SYNC_TASKS.shutdown().await;
ENVELOPE_MANAGER.shutdown().await;
ATTACHMENT_MANAGER.shutdown().await;
BLOB_MANAGER.shutdown().await;
DB_MANAGER.flush();
info!("Bichon server stopped.");
Ok(())
}
async fn initialize() -> BichonResult<()> {
SignalManager::initialize().await?;
DataDirManager::initialize().await?;
UserManager::initialize().await?;
BichonTls::initialize().await?;
BichonContext::initialize().await?;
LazyLock::force(&BLOB_MANAGER);
LazyLock::force(&ENVELOPE_MANAGER);
LazyLock::force(&ATTACHMENT_MANAGER);
Ok(())
}

View File

@@ -16,205 +16,9 @@
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use std::sync::LazyLock;
use bichon_core::error::BichonResult;
use bichon_core::{
bichon_version,
cache::imap::task::SYNC_TASKS,
common::rustls::BichonTls,
context::{executors::BichonContext, Initialize},
error::{code::ErrorCode, BichonResult},
logger,
migrate::check_data_status,
raise_error,
settings::cli::SETTINGS,
store::{
blob::BLOB_MANAGER,
tantivy::{attachment::ATTACHMENT_MANAGER, envelope::ENVELOPE_MANAGER},
},
tasks::PeriodicTasks,
};
use bichon_smtp::server::{start_smtp_server, SmtpServer};
use mimalloc::MiMalloc;
use tracing::{error, info};
use bichon_core::{
common::signal::SignalManager, settings::dir::DataDirManager, users::manager::UserManager,
};
use crate::rest::start_http_server;
pub mod common;
pub mod error;
pub mod rest;
#[global_allocator]
static GLOBAL: MiMalloc = MiMalloc;
static LOGO: &str = r#"
_ _ _
| | (_) | |
| |__ _ ___ | |__ ___ _ __
| '_ \ | | / __|| '_ \ / _ \ | '_ \
| |_) || || (__ | | | || (_) || | | |
|_.__/ |_| \___||_| |_| \___/ |_| |_|
"#;
#[tokio::main]
async fn main() -> BichonResult<()> {
logger::initialize_logging();
info!("{}", LOGO);
info!("Starting bichon-server");
info!("Version: {}", bichon_version!());
info!("Git: [{}]", env!("GIT_HASH"));
info!("GitHub: https://github.com/rustmailer/bichon");
match check_data_status() {
Ok(false) => {
error!("Incompatible data format detected.");
error!("Your data was created by an older version of Bichon and must be migrated before use.");
error!("Please stop the Bichon v0.3.7 service before migration.");
error!("Please run: bichon-admin");
error!("Documentation: https://github.com/rustmailer/bichon/wiki/Bichon-Data-Migration:-v0.3.7-%E2%86%92-v1.0");
return Err(raise_error!(
"Legacy data layout detected".into(),
ErrorCode::InternalError
));
}
Err(e) => {
error!("Failed to check data layout: {:#?}", e);
return Err(raise_error!(format!("{:#?}", e), ErrorCode::InternalError));
}
Ok(true) => {}
}
if let Err(error) = initialize().await {
eprintln!("{:?}", error);
return Err(error);
}
let periodic_tasks = PeriodicTasks::setup();
let mut smtp_service: Option<SmtpServer> = None;
if SETTINGS.bichon_enable_smtp {
info!("SMTP service is enabled, starting...");
match start_smtp_server().await {
Ok(server) => {
info!("SMTP server listening on: {}", server.smtp_addr);
smtp_service = Some(server);
}
Err(e) => {
error!("Failed to start SMTP server: {}", e);
return Err(raise_error!(format!("{:#?}", e), ErrorCode::InternalError));
}
}
} else {
info!("SMTP service is disabled by configuration.");
}
start_http_server().await?;
periodic_tasks.shutdown().await;
if let Some(server) = smtp_service {
info!("Shutting down SMTP server...");
server.stop().await;
info!("SMTP server stopped.");
}
SYNC_TASKS.shutdown().await;
ENVELOPE_MANAGER.shutdown().await;
ATTACHMENT_MANAGER.shutdown().await;
BLOB_MANAGER.shutdown().await;
info!("Bichon server stopped.");
Ok(())
}
/// Initialize the system by validating settings and starting necessary tasks.
async fn initialize() -> BichonResult<()> {
SignalManager::initialize().await?;
DataDirManager::initialize().await?;
UserManager::initialize().await?;
BichonTls::initialize().await?;
BichonContext::initialize().await?;
LazyLock::force(&BLOB_MANAGER);
LazyLock::force(&ENVELOPE_MANAGER);
LazyLock::force(&ATTACHMENT_MANAGER);
Ok(())
}
#[cfg(test)]
mod tests;
#[cfg(test)]
mod api_tests {
use super::rest::api::create_openapi_service;
use poem::test::TestClient;
#[tokio::test]
async fn openapi_spec_json_is_served() {
let api_service = create_openapi_service();
let spec_endpoint = api_service.spec_endpoint();
let cli = TestClient::new(spec_endpoint);
let resp = cli.get("/").send().await;
resp.assert_status_is_ok();
let body = resp.json().await;
let obj = body.value().object();
assert!(obj.get_opt("openapi").is_some(), "missing openapi version");
assert!(obj.get_opt("info").is_some(), "missing info section");
assert!(obj.get_opt("paths").is_some(), "missing paths section");
}
#[tokio::test]
async fn openapi_spec_yaml_is_served() {
let api_service = create_openapi_service();
let spec_endpoint = api_service.spec_endpoint_yaml();
let cli = TestClient::new(spec_endpoint);
let resp = cli.get("/").send().await;
resp.assert_status_is_ok();
}
#[tokio::test]
async fn swagger_ui_is_served() {
let api_service = create_openapi_service();
let swagger = api_service.swagger_ui();
let cli = TestClient::new(swagger);
let resp = cli.get("/").send().await;
resp.assert_status_is_ok();
}
#[tokio::test]
async fn openapi_spec_lists_all_tag_groups() {
let api_service = create_openapi_service();
let spec_endpoint = api_service.spec_endpoint();
let cli = TestClient::new(spec_endpoint);
let resp = cli.get("/").send().await;
let body = resp.json().await;
let value = body.value();
let tag_names: Vec<&str> = value
.object()
.get("tags")
.array()
.iter()
.map(|v| v.object().get("name").string())
.collect();
assert!(
tag_names.contains(&"AccessToken"),
"missing AccessToken tag"
);
assert!(tag_names.contains(&"Attachment"), "missing Attachment tag");
assert!(tag_names.contains(&"AutoConfig"), "missing AutoConfig tag");
assert!(tag_names.contains(&"Account"), "missing Account tag");
assert!(tag_names.contains(&"System"), "missing System tag");
assert!(tag_names.contains(&"Mailbox"), "missing Mailbox tag");
assert!(tag_names.contains(&"OAuth2"), "missing OAuth2 tag");
assert!(tag_names.contains(&"Message"), "missing Message tag");
assert!(tag_names.contains(&"Import"), "missing Import tag");
assert!(tag_names.contains(&"Users"), "missing Users tag");
}
bichon_server::run().await
}

View File

@@ -121,6 +121,8 @@ impl MessageApi {
}
/// Fetches the content of a specific email.
/// Set `block_remote_content=true` to strip external images, scripts,
/// and other content loaded from http/https URLs.
#[oai(
path = "/message-content/:account_id/:envelope_id",
method = "get",
@@ -132,11 +134,18 @@ impl MessageApi {
account_id: Path<u64>,
/// The ID of the message to fetch.
envelope_id: Path<String>,
/// Block remote content (http/https URLs) from email body.
block_remote_content: Query<Option<bool>>,
context: WrappedContext,
) -> ApiResult<Json<FullMessageContent>> {
let account_id = account_id.0;
let block_remote = block_remote_content.0.unwrap_or(false);
context.require_permission(Some(account_id), Permission::DATA_READ)?;
Ok(Json(retrieve_email_content(account_id, envelope_id.0)?))
Ok(Json(retrieve_email_content(
account_id,
envelope_id.0,
block_remote,
)?))
}
/// Retrieves the content of an email embedded as an attachment.
@@ -152,15 +161,18 @@ impl MessageApi {
/// The ID of the message to fetch.
envelope_id: Path<String>,
content_hash: Query<String>,
block_remote_content: Query<Option<bool>>,
context: WrappedContext,
) -> ApiResult<Json<FullNestedMessageContent>> {
let account_id = account_id.0;
let block_remote = block_remote_content.0.unwrap_or(false);
context.require_permission(Some(account_id), Permission::DATA_READ)?;
let content_hash = content_hash.0.trim();
Ok(Json(retrieve_nested_eml_content(
account_id,
envelope_id.0,
content_hash,
block_remote,
)?))
}
@@ -213,7 +225,7 @@ impl MessageApi {
AccountModel::check_account_exists(account_id)?;
context.require_permission(Some(account_id), Permission::DATA_RAW_DOWNLOAD)?;
let envelope_id = envelope_id.0;
let reader = get_reader(account_id, envelope_id.clone())?;
let reader = get_reader(account_id, envelope_id.clone()).await?;
let body = Body::from_async_read(reader);
let attachment = Attachment::new(body)
.attachment_type(AttachmentType::Attachment)

View File

@@ -1,24 +1,11 @@
//
// Copyright (c) 2025-2026 rustmailer.com (https://rustmailer.com)
//
// This file is part of the Bichon Email Archiving Project
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
#[cfg(feature = "embed-web")]
mod inner {
use rust_embed::RustEmbed;
#[derive(RustEmbed)]
#[folder = "../../web/dist/"]
pub struct FrontEndAssets;
}
use rust_embed::RustEmbed;
#[derive(RustEmbed)]
#[folder = "../../web/dist/"]
pub struct FrontEndAssets;
#[cfg(feature = "embed-web")]
pub use inner::FrontEndAssets;

View File

@@ -16,51 +16,49 @@
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use crate::common::auth::ApiGuard;
use crate::common::error::ErrorCapture;
use crate::common::log::Tracing;
use crate::common::tls::rustls_config;
use crate::common::timeout::{Timeout, TIMEOUT_HEADER};
use crate::error::handler::error_handler;
use crate::rest::public::features::get_features;
use crate::rest::public::login::login;
use crate::rest::public::status::get_status;
use bichon_core::common::signal::SIGNAL_MANAGER;
use bichon_core::error::code::ErrorCode;
use bichon_core::error::BichonResult;
use bichon_core::raise_error;
use bichon_core::settings::cli::SETTINGS;
use super::error::ApiErrorResponse;
use crate::common::auth::ApiGuard;
use crate::common::timeout::{Timeout, TIMEOUT_HEADER};
use api::create_openapi_service;
use assets::FrontEndAssets;
use bichon_core::raise_error;
use http::{HeaderValue, Method};
use poem::endpoint::EmbeddedFilesEndpoint;
use http::Method;
use poem::listener::{Listener, TcpListener};
use poem::middleware::{CatchPanic, Compression, SetHeader};
use poem::{get, handler, post, IntoResponse};
use poem::{middleware::Cors, EndpointExt, Route, Server};
use poem::middleware::{CatchPanic, Compression, Cors};
use poem::{get, post, Endpoint, EndpointExt, Route, Server};
use public::oauth2::oauth2_callback;
use std::collections::HashSet;
use std::time::Duration;
#[cfg(feature = "embed-web")]
use {
assets::FrontEndAssets,
http::HeaderValue,
poem::{handler, endpoint::EmbeddedFilesEndpoint, IntoResponse},
poem::middleware::SetHeader,
};
pub mod api;
pub mod assets;
pub mod public;
pub type ApiResult<T, E = ApiErrorResponse> = std::result::Result<T, E>;
pub async fn start_http_server() -> BichonResult<()> {
let listener = TcpListener::bind((
SETTINGS.bichon_bind_ip.clone().unwrap_or("0.0.0.0".into()),
SETTINGS.bichon_http_port as u16,
));
let listener = if SETTINGS.bichon_enable_rest_https {
listener.rustls(rustls_config()?).boxed()
} else {
listener.boxed()
};
use super::error::ApiErrorResponse;
/// Build the community route tree. Pro/Enterprise servers can call this
/// and then add their own routes before passing the tree to the server.
pub fn build_routes() -> impl Endpoint {
let api_service = create_openapi_service()
.summary("A lightweight, high-performance Rust email archiver with WebUI");
@@ -79,7 +77,6 @@ pub async fn start_http_server() -> BichonResult<()> {
.with(Tracing);
let cors_origins: Option<HashSet<String>> = SETTINGS.bichon_cors_origins.clone();
let cors_origins: Vec<String> = cors_origins.unwrap_or_default().into_iter().collect();
let cors = Cors::new()
@@ -92,7 +89,6 @@ pub async fn start_http_server() -> BichonResult<()> {
}
cors_origins.iter().any(|o| o == origin)
})
//.allow_origins(cors_origins)
.allow_credentials(true)
.allow_methods(&[
Method::GET,
@@ -107,13 +103,6 @@ pub async fn start_http_server() -> BichonResult<()> {
.expose_headers(vec!["Accept"])
.max_age(SETTINGS.bichon_cors_max_age);
let cache_static = || {
SetHeader::new().overriding(
http::header::CACHE_CONTROL,
HeaderValue::from_static("max-age=86400"),
)
};
let app_logic = Route::new()
.nest("/api-docs/swagger", swagger)
.nest("/api-docs/redoc", redoc)
@@ -122,42 +111,43 @@ pub async fn start_http_server() -> BichonResult<()> {
.nest("/api-docs/spec.json", spec_json)
.nest("/api-docs/spec.yaml", spec_yaml)
.nest("/oauth2/callback", get(oauth2_callback))
.nest("/api/v1/features", get(get_features))
.nest("/api/status", get(get_status))
.nest("/api/login", post(login))
.nest_no_strip("/api/v1", open_api_route)
.nest_no_strip("/api/v1", open_api_route);
let app_logic = add_web_assets(app_logic);
Route::new()
.nest(&SETTINGS.bichon_base_url, app_logic)
.with(cors)
.with_if(SETTINGS.bichon_http_compression_enabled, Compression::new())
.with(CatchPanic::new())
}
#[cfg(feature = "embed-web")]
fn add_web_assets(route: Route) -> impl Endpoint {
let cache_static = || {
SetHeader::new().overriding(
http::header::CACHE_CONTROL,
HeaderValue::from_static("max-age=86400"),
)
};
route
.nest_no_strip(
"/assets",
EmbeddedFilesEndpoint::<FrontEndAssets>::new().with(cache_static()),
)
.at("/*", serve_index_with_base);
let route = Route::new()
.nest(&SETTINGS.bichon_base_url, app_logic)
.with(cors)
.with_if(SETTINGS.bichon_http_compression_enabled, Compression::new())
.with(CatchPanic::new());
let mut rx = SIGNAL_MANAGER.subscribe();
let shutdown_fut = async move {
let _ = rx.recv().await;
};
let server = Server::new(listener)
.name("Bichon Service")
.idle_timeout(Duration::from_secs(60))
.run_with_graceful_shutdown(
route.catch_all_error(error_handler),
shutdown_fut,
Some(Duration::from_secs(5)),
);
println!(
"Bichon Service is now running on port {}.",
SETTINGS.bichon_http_port
);
server
.await
.map_err(|e| raise_error!(format!("{:#?}", e), ErrorCode::InternalError))
.at("/*", serve_index_with_base)
}
#[cfg(not(feature = "embed-web"))]
fn add_web_assets(route: Route) -> Route {
route
}
#[cfg(feature = "embed-web")]
#[handler]
async fn serve_index_with_base() -> impl IntoResponse {
let mut html =
@@ -180,3 +170,38 @@ async fn serve_index_with_base() -> impl IntoResponse {
.content_type("text/html; charset=utf-8")
.body(html)
}
pub async fn start_http_server() -> BichonResult<()> {
let listener = TcpListener::bind((
SETTINGS.bichon_bind_ip.clone().unwrap_or("0.0.0.0".into()),
SETTINGS.bichon_http_port as u16,
));
let listener = if SETTINGS.bichon_enable_rest_https {
listener.rustls(rustls_config()?).boxed()
} else {
listener.boxed()
};
let route = build_routes();
let mut rx = SIGNAL_MANAGER.subscribe();
let shutdown_fut = async move {
let _ = rx.recv().await;
};
let server = Server::new(listener)
.name("Bichon Service")
.idle_timeout(Duration::from_secs(60))
.run_with_graceful_shutdown(
route.catch_all_error(error_handler),
shutdown_fut,
Some(Duration::from_secs(5)),
);
println!(
"Bichon Service is now running on port {}.",
SETTINGS.bichon_http_port
);
server
.await
.map_err(|e| raise_error!(format!("{:#?}", e), ErrorCode::InternalError))
}

View File

@@ -0,0 +1,36 @@
//
// Copyright (c) 2025-2026 rustmailer.com (https://rustmailer.com)
//
// This file is part of the Bichon Email Archiving Project
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use poem::{handler, web::Json, IntoResponse};
use serde::Serialize;
#[derive(Serialize)]
struct FeaturesResponse {
features: Vec<String>,
edition: &'static str,
version: String,
}
#[handler]
pub async fn get_features() -> impl IntoResponse {
Json(FeaturesResponse {
features: vec![],
edition: "community",
version: env!("CARGO_PKG_VERSION").to_string(),
})
}

View File

@@ -17,6 +17,7 @@
// along with this program. If not, see <http://www.gnu.org/licenses/>.
pub mod features;
pub mod login;
pub mod oauth2;
pub mod status;

View File

@@ -21,6 +21,7 @@ use std::net::SocketAddr;
use std::time::Duration;
use base64::{prelude::BASE64_STANDARD, Engine as _};
use bichon_core::account::migration::AccountType;
use bichon_core::cache::imap::mailbox::{Attribute, AttributeEnum};
use bichon_core::common::signal::SIGNAL_MANAGER;
use bichon_core::envelope::extractor::extract_envelope_from_smtp;
@@ -429,8 +430,20 @@ where
}
if is_allowed {
session.rcpt_to.push(account);
stream.write_all(b"250 OK\r\n").await?;
if !matches!(account.account_type, AccountType::NoSync) {
tracing::warn!(
"SMTP: Rejected journaling attempt to IMAP account <{}>",
addr
);
let err = format!(
"550 5.7.1 <{}>: Not a Bichon local account, journaling is not supported\r\n",
account.email
);
stream.write_all(err.as_bytes()).await?;
} else {
session.rcpt_to.push(account);
stream.write_all(b"250 OK\r\n").await?;
}
}
}
Ok(None) => {
@@ -615,25 +628,40 @@ async fn parse_email(data: &[u8], session: &Session) -> BichonResult<()> {
return Ok(());
}
};
let mailbox = MailBox {
id: create_hash(rcpt.id, "INBOX"),
account_id: rcpt.id,
name: "INBOX".into(),
delimiter: Some("/".to_string()),
attributes: vec![Attribute {
attr: AttributeEnum::Extension,
extension: Some("CreatedByBichon".into()),
}],
exists: 0,
unseen: None,
uid_next: None,
uid_validity: None,
};
let mailbox_id = mailbox.id;
let mailbox_id = create_hash(rcpt.id, "INBOX");
if let Err(e) = MailBox::batch_upsert(&[mailbox]) {
tracing::error!("SMTP: Failed to upsert mailbox for {}: {:?}", rcpt.email, e);
return Err(e.into());
// The INBOX row is owned by the IMAP sync, which maintains `uid_validity`,
// `highest_uid` and `uid_next` on it. `batch_upsert` replaces the *whole*
// row, so blindly upserting here (with those fields = None) clobbers the
// IMAP-maintained state back to None. The next reconcile then sees
// `uid_validity` change from Some -> None, treats the mailbox as invalid,
// and wipes + rebuilds it — silently losing the local copy of a large
// mailbox when that rebuild is interrupted (see #297).
//
// We only need the row to *exist* so the journaled envelope can attach to
// it, so create it only when it is missing and otherwise leave the
// IMAP-owned row untouched.
if MailBox::find_mailbox(rcpt.id, mailbox_id)?.is_none() {
let mailbox = MailBox {
id: mailbox_id,
account_id: rcpt.id,
name: "INBOX".into(),
delimiter: Some("/".to_string()),
attributes: vec![Attribute {
attr: AttributeEnum::Extension,
extension: Some("CreatedByBichon".into()),
}],
exists: 0,
unseen: None,
uid_next: None,
uid_validity: None,
highest_uid: None,
};
if let Err(e) = MailBox::batch_upsert(&[mailbox]) {
tracing::error!("SMTP: Failed to upsert mailbox for {}: {:?}", rcpt.email, e);
return Err(e.into());
}
}
extract_envelope_from_smtp(data, rcpt.id, mailbox_id)

58
funding.json Normal file
View File

@@ -0,0 +1,58 @@
{
"$schema": "https://fundingjson.org/schema/v1.1.0.json",
"version": "v1.0.0",
"entity": {
"type": "individual",
"role": "maintainer",
"name": "rustmailer",
"email": "rustmailer.git@gmail.com",
"phone": "",
"description": "I'm an indie developer and the sole maintainer of Bichon, a lightweight open-source email archiver built in Rust. I believe in privacy, data ownership, and the right to self-host your own digital life.",
"webpageUrl": {
"url": "https://github.com/rustmailer"
}
},
"projects": [
{
"guid": "bichon",
"name": "Bichon",
"description": "Bichon is a lightweight, high-performance, self-hosted email archiver built in Rust. It synchronizes emails from IMAP servers, indexes them for full-text search, and provides a clean WebUI and REST API for access.\n\nBichon requires no external database and runs as a single binary — making it easy to deploy and maintain. It supports multiple accounts, OAuth2, SOCKS5 proxy, scheduled sync, bulk import (EML/MBOX/PST), and multi-user RBAC.\n\nAs the sole maintainer, I develop and support Bichon in my personal time. With 1.8k GitHub stars and 327k+ Docker pulls, the project has grown well beyond a personal tool and is actively used by individuals and teams worldwide — including a real-world deployment archiving 1.15 million emails across 28 accounts (800 GB original data, compressed to 421 GB on disk).",
"webpageUrl": {
"url": "https://github.com/rustmailer/bichon"
},
"repositoryUrl": {
"url": "https://github.com/rustmailer/bichon"
},
"licenses": ["spdx:AGPL-3.0"],
"tags": ["email", "rust", "self-hosted", "archiver", "imap", "full-text-search", "privacy", "webui"]
}
],
"funding": {
"channels": [
{
"guid": "buymeacoffee",
"type": "payment-provider",
"address": "https://buymeacoffee.com/rustmailer",
"description": "Support via Buy Me a Coffee."
},
{
"guid": "bank",
"type": "bank",
"address": "",
"description": "Direct bank transfer also accepted. Please email rustmailer.git@gmail.com for details."
}
],
"plans": [
{
"guid": "maintainer-time",
"status": "active",
"name": "Maintainer Time",
"description": "Cover the cost of dedicated development and maintenance time for Bichon — including bug fixes, feature development, security updates, issue triage, and community support.",
"amount": 10000,
"currency": "USD",
"frequency": "yearly",
"channels": ["bank"]
}
]
}
}

View File

@@ -10,7 +10,10 @@
"preview": "vite preview",
"format:check": "prettier --check .",
"format": "prettier --write .",
"knip": "knip"
"knip": "knip",
"test": "vitest run",
"test:watch": "vitest",
"test:coverage": "vitest run --coverage"
},
"dependencies": {
"@emotion/react": "^11.14.0",
@@ -81,6 +84,9 @@
"@tanstack/react-query-devtools": "^5.62.3",
"@tanstack/router-devtools": "^1.86.1",
"@tanstack/router-plugin": "^1.86.0",
"@testing-library/jest-dom": "^6.9.1",
"@testing-library/react": "^16.3.2",
"@testing-library/user-event": "^14.6.1",
"@trivago/prettier-plugin-sort-imports": "^4.3.0",
"@types/file-saver": "^2.0.7",
"@types/js-cookie": "^3.0.6",
@@ -88,18 +94,22 @@
"@types/react": "^18.3.18",
"@types/react-dom": "^18.3.5",
"@vitejs/plugin-react-swc": "^3.7.2",
"@vitest/coverage-v8": "^4.1.7",
"autoprefixer": "^10.4.20",
"eslint": "^9.16.0",
"eslint-plugin-react-hooks": "^5.1.0",
"eslint-plugin-react-refresh": "^0.4.16",
"globals": "^15.13.0",
"jsdom": "^29.1.1",
"knip": "^5.41.1",
"msw": "^2.14.6",
"postcss": "^8.4.49",
"prettier": "^3.4.2",
"prettier-plugin-tailwindcss": "^0.6.9",
"tailwindcss": "^3.4.16",
"typescript": "~5.7.2",
"typescript-eslint": "^8.17.0",
"vite": "^6.0.11"
"vite": "^6.0.11",
"vitest": "^4.1.7"
}
}

1192
web/pnpm-lock.yaml generated

File diff suppressed because it is too large Load Diff

View File

@@ -128,10 +128,10 @@ export interface AccountModel {
capabilities?: string[];
date_since?: DateSelection;
date_before?: RelativeDate;
folder_limit?: number,
download_folders: string[];
download_interval_min?: number;
download_batch_size?: number;
max_email_size_bytes?: number;
created_by: number;
created_user_name: string;
created_user_email: string;
@@ -143,6 +143,8 @@ export interface AccountModel {
imap_quota_window?: QuotaWindow;
imap_quota_bytes?: number;
auto_download_new_mailboxes?: boolean;
download_schedule?: string;
deleting?: boolean;
}
export const download_state = async (account_id: number) => {

View File

@@ -64,7 +64,8 @@ export interface AttachmentInfo {
export interface MessageContentResponse {
text?: string;
html?: string;
attachments?: AttachmentInfo[]
attachments?: AttachmentInfo[];
has_remote_content?: boolean;
}
export interface NestedMessageContentResponse {
@@ -72,6 +73,7 @@ export interface NestedMessageContentResponse {
html?: string;
attachments?: AttachmentInfo[];
envelope: EmailEnvelope;
has_remote_content?: boolean;
}
export const getContent = (messageContent: MessageContentResponse): string | null => {
@@ -83,13 +85,25 @@ export const getContent = (messageContent: MessageContentResponse): string | nul
return null;
};
export const load_message = async (accountId: number, id: string) => {
const response = await axiosInstance.get<MessageContentResponse>(`api/v1/message-content/${accountId}/${id}`);
export const load_message = async (accountId: number, id: string, blockRemoteContent = false) => {
const params = new URLSearchParams();
if (blockRemoteContent) {
params.set('block_remote_content', 'true');
}
const qs = params.toString();
const url = `api/v1/message-content/${accountId}/${id}${qs ? '?' + qs : ''}`;
const response = await axiosInstance.get<MessageContentResponse>(url);
return response.data;
};
export const load_nested_message = async (accountId: number, id: string, content_hash: string) => {
const response = await axiosInstance.get<NestedMessageContentResponse>(`api/v1/nested-message-content/${accountId}/${id}?content_hash=${content_hash}`);
export const load_nested_message = async (accountId: number, id: string, content_hash: string, blockRemoteContent = false) => {
const params = new URLSearchParams({ content_hash });
if (blockRemoteContent) {
params.set('block_remote_content', 'true');
}
const response = await axiosInstance.get<NestedMessageContentResponse>(
`api/v1/nested-message-content/${accountId}/${id}?${params.toString()}`
);
return response.data;
};

View File

@@ -28,18 +28,54 @@ interface GithubLinkButtonProps {
title?: string;
}
const CACHE_KEY = "github_stars_cache";
const CACHE_TTL = 6 * 60 * 60 * 1000; // 6 hours
interface StarsCache {
stars: number;
fetchedAt: number;
}
function getCachedStars(repo: string): number | null {
try {
const raw = localStorage.getItem(`${CACHE_KEY}_${repo}`);
if (!raw) return null;
const cache: StarsCache = JSON.parse(raw);
if (Date.now() - cache.fetchedAt > CACHE_TTL) return null;
return cache.stars;
} catch {
return null;
}
}
function setCachedStars(repo: string, stars: number) {
try {
localStorage.setItem(
`${CACHE_KEY}_${repo}`,
JSON.stringify({ stars, fetchedAt: Date.now() })
);
} catch { }
}
export const GithubLinkButton: React.FC<GithubLinkButtonProps> = ({
href = "https://github.com/rustmailer/bichon",
repo = "rustmailer/bichon",
size = 18,
title = "View on GitHub",
}) => {
const [stars, setStars] = useState<number | null>(null);
const [stars, setStars] = useState<number | null>(() => getCachedStars(repo));
useEffect(() => {
if (stars !== null) return; // already have cached value, skip fetch
fetch(`https://api.github.com/repos/${repo}`)
.then(res => res.json())
.then(data => setStars(data.stargazers_count))
.then(data => {
const count = data.stargazers_count;
if (typeof count === "number") {
setStars(count);
setCachedStars(repo, count);
}
})
.catch(() => { });
}, [repo]);

View File

@@ -31,7 +31,7 @@ const EmailIframe: React.FC<EmailIframeProps> = ({ emailHtml, height }) => {
return (
<iframe
src={iframeSrc}
sandbox="allow-scripts"
sandbox=""
className="w-full border-none"
title="Email Content"
style={{ height: height ?? '4000px' }}

View File

@@ -202,7 +202,7 @@ const Sidebar = React.forwardRef<
<SheetContent
data-sidebar='sidebar'
data-mobile='true'
className='w-[--sidebar-width] bg-sidebar p-0 text-sidebar-foreground [&>button]:hidden'
className='w-[--sidebar-width] !bg-sidebar p-0 text-sidebar-foreground [&>button]:hidden'
style={
{
'--sidebar-width': SIDEBAR_WIDTH_MOBILE,

View File

@@ -0,0 +1,145 @@
import { describe, it, expect } from 'vitest'
import { getAccountSchema } from '../schema'
const t = (key: string) => key
const baseData = {
email: 'test@example.com',
imap: {
host: 'imap.example.com',
port: 993,
encryption: 'Ssl' as const,
auth: {
auth_type: 'Password' as const,
password: 'mypassword',
},
},
enabled: true,
use_dangerous: false,
download_interval_min: 60,
download_batch_size: 30,
auto_download_new_mailboxes: true,
}
describe('Account Schema - date_since validation', () => {
const schema = getAccountSchema(false, t)
it('accepts fixed date_since', () => {
const result = schema.safeParse({
...baseData,
date_since: { fixed: '2024-01-01' },
})
expect(result.success).toBe(true)
})
it('accepts relative date_since', () => {
const result = schema.safeParse({
...baseData,
date_since: { relative: { unit: 'Months', value: 6 } },
})
expect(result.success).toBe(true)
})
it('accepts undefined date_since', () => {
const result = schema.safeParse(baseData)
expect(result.success).toBe(true)
})
it('rejects relative date_since with value 0', () => {
const result = schema.safeParse({
...baseData,
date_since: { relative: { unit: 'Months', value: 0 } },
})
expect(result.success).toBe(false)
})
it('rejects relative date_since with negative value', () => {
const result = schema.safeParse({
...baseData,
date_since: { relative: { unit: 'Months', value: -1 } },
})
expect(result.success).toBe(false)
})
it('rejects relative date_since with non-integer value', () => {
const result = schema.safeParse({
...baseData,
date_since: { relative: { unit: 'Months', value: 1.5 } },
})
expect(result.success).toBe(false)
})
it('rejects fixed date_since with empty string', () => {
const result = schema.safeParse({
...baseData,
date_since: { fixed: '' },
})
expect(result.success).toBe(false)
})
})
describe('Account Schema - date_before validation', () => {
const schema = getAccountSchema(false, t)
it('accepts valid date_before', () => {
const result = schema.safeParse({
...baseData,
date_before: { unit: 'Days', value: 30 },
})
expect(result.success).toBe(true)
})
it('accepts undefined date_before', () => {
const result = schema.safeParse(baseData)
expect(result.success).toBe(true)
})
it('rejects date_before with value 0', () => {
const result = schema.safeParse({
...baseData,
date_before: { unit: 'Days', value: 0 },
})
expect(result.success).toBe(false)
})
})
describe('Account Schema - use_dangerous and enabled flags', () => {
const schema = getAccountSchema(false, t)
it('accepts use_dangerous: true', () => {
const result = schema.safeParse({ ...baseData, use_dangerous: true })
expect(result.success).toBe(true)
})
it('accepts enabled: false', () => {
const result = schema.safeParse({ ...baseData, enabled: false })
expect(result.success).toBe(true)
})
it('accepts auto_download_new_mailboxes: false', () => {
const result = schema.safeParse({
...baseData,
auto_download_new_mailboxes: false,
})
expect(result.success).toBe(true)
})
})
describe('Account Schema - missing required nested fields', () => {
const schema = getAccountSchema(false, t)
it('rejects missing imap entirely', () => {
const { imap, ...noImap } = baseData
const result = schema.safeParse(noImap)
expect(result.success).toBe(false)
})
it('rejects missing imap.auth', () => {
const { auth, ...noAuth } = baseData.imap
const result = schema.safeParse({
...baseData,
imap: noAuth,
})
expect(result.success).toBe(false)
})
})

View File

@@ -0,0 +1,332 @@
import { describe, it, expect } from 'vitest'
import { getAccountSchema, getAuthConfigSchema } from '../schema'
const t = (key: string) => key
const validAccountData = {
email: 'user@example.com',
imap: {
host: 'imap.example.com',
port: 993,
encryption: 'Ssl' as const,
auth: {
auth_type: 'Password' as const,
password: 'mypassword',
},
},
enabled: true,
use_dangerous: false,
download_interval_min: 60,
download_batch_size: 30,
auto_download_new_mailboxes: true,
}
describe('Account Form Schema', () => {
describe('email field', () => {
const schema = getAccountSchema(false, t)
it('rejects empty email', () => {
const result = schema.safeParse({ ...validAccountData, email: '' })
expect(result.success).toBe(false)
})
it('rejects invalid email format', () => {
const result = schema.safeParse({
...validAccountData,
email: 'not-an-email',
})
expect(result.success).toBe(false)
})
it('rejects email without @', () => {
const result = schema.safeParse({
...validAccountData,
email: 'username',
})
expect(result.success).toBe(false)
})
it('accepts valid email', () => {
const result = schema.safeParse(validAccountData)
expect(result.success).toBe(true)
})
})
describe('imap.host field', () => {
it('rejects empty IMAP host', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
imap: { ...validAccountData.imap, host: '' },
})
expect(result.success).toBe(false)
})
it('accepts valid hostname', () => {
const result = getAccountSchema(false, t).safeParse(validAccountData)
expect(result.success).toBe(true)
})
it('accepts IP address as host', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
imap: { ...validAccountData.imap, host: '192.168.1.1' },
})
expect(result.success).toBe(true)
})
})
describe('imap.port field', () => {
it('accepts port 993 (standard IMAP SSL)', () => {
const result = getAccountSchema(false, t).safeParse(validAccountData)
expect(result.success).toBe(true)
})
it('accepts port 143 (standard IMAP)', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
imap: { ...validAccountData.imap, port: 143 },
})
expect(result.success).toBe(true)
})
it('accepts port 0 (auto-detect)', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
imap: { ...validAccountData.imap, port: 0 },
})
expect(result.success).toBe(true)
})
it('rejects negative port', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
imap: { ...validAccountData.imap, port: -1 },
})
expect(result.success).toBe(false)
})
it('rejects port > 65535', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
imap: { ...validAccountData.imap, port: 99999 },
})
expect(result.success).toBe(false)
})
it('rejects non-integer port', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
imap: { ...validAccountData.imap, port: 993.5 },
})
expect(result.success).toBe(false)
})
})
describe('imap.encryption field', () => {
it('accepts Ssl', () => {
const result = getAccountSchema(false, t).safeParse(validAccountData)
expect(result.success).toBe(true)
})
it('accepts StartTls', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
imap: { ...validAccountData.imap, encryption: 'StartTls' },
})
expect(result.success).toBe(true)
})
it('accepts None', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
imap: { ...validAccountData.imap, encryption: 'None' },
})
expect(result.success).toBe(true)
})
it('rejects invalid encryption value', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
imap: { ...validAccountData.imap, encryption: 'TLS' },
})
expect(result.success).toBe(false)
})
})
describe('download_interval_min field', () => {
it('rejects value less than 10', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
download_interval_min: 5,
})
expect(result.success).toBe(false)
})
it('accepts value of exactly 10', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
download_interval_min: 10,
})
expect(result.success).toBe(true)
})
it('rejects non-integer value', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
download_interval_min: 30.5,
})
expect(result.success).toBe(false)
})
})
describe('download_batch_size field', () => {
it('rejects value less than 10', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
download_batch_size: 5,
})
expect(result.success).toBe(false)
})
it('rejects value greater than 200', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
download_batch_size: 500,
})
expect(result.success).toBe(false)
})
it('accepts value of exactly 10', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
download_batch_size: 10,
})
expect(result.success).toBe(true)
})
it('accepts value of exactly 200', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
download_batch_size: 200,
})
expect(result.success).toBe(true)
})
})
describe('account_name and login_name fields', () => {
it('accepts undefined account_name and login_name', () => {
const result = getAccountSchema(false, t).safeParse(validAccountData)
expect(result.success).toBe(true)
})
it('accepts provided account_name', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
account_name: 'My Work Email',
})
expect(result.success).toBe(true)
})
it('accepts provided login_name', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
login_name: 'username',
})
expect(result.success).toBe(true)
})
})
describe('download_schedule field', () => {
it('accepts undefined download_schedule', () => {
const result = getAccountSchema(false, t).safeParse(validAccountData)
expect(result.success).toBe(true)
})
it('accepts valid 6-field cron expression', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
download_schedule: '0 0 0 * * *',
})
expect(result.success).toBe(true)
})
it('accepts cron with */step syntax', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
download_schedule: '0 */30 8-17 * * 1-5',
})
expect(result.success).toBe(true)
})
it('rejects cron with too few fields', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
download_schedule: '0 0 *',
})
expect(result.success).toBe(false)
})
it('accepts empty string cron (treated as not set)', () => {
const result = getAccountSchema(false, t).safeParse({
...validAccountData,
download_schedule: '',
})
expect(result.success).toBe(true)
})
})
})
describe('Auth Config Schema (password validation)', () => {
describe('when creating (isEdit = false)', () => {
const schema = getAuthConfigSchema(false, t)
it('requires password when auth_type is Password', () => {
const result = schema.safeParse({
auth_type: 'Password',
password: '',
})
expect(result.success).toBe(false)
})
it('requires password when auth_type is Password and password undefined', () => {
const result = schema.safeParse({
auth_type: 'Password',
})
expect(result.success).toBe(false)
})
it('accepts valid password with Password auth', () => {
const result = schema.safeParse({
auth_type: 'Password',
password: 'mypassword',
})
expect(result.success).toBe(true)
})
it('does not require password when auth_type is OAuth2', () => {
const result = schema.safeParse({
auth_type: 'OAuth2',
})
expect(result.success).toBe(true)
})
})
describe('when editing (isEdit = true)', () => {
const schema = getAuthConfigSchema(true, t)
it('does not require password even with Password auth', () => {
const result = schema.safeParse({
auth_type: 'Password',
password: '',
})
expect(result.success).toBe(true)
})
it('accepts with undefined password', () => {
const result = schema.safeParse({
auth_type: 'Password',
})
expect(result.success).toBe(true)
})
})
})

View File

@@ -97,6 +97,10 @@ export function AccountDetailDrawer({ open, onOpenChange, currentRow }: Props) {
<span className="text-muted-foreground">{t('accounts.downloadBatchSize')}:</span>
<span>{currentRow.download_batch_size}</span>
</div>
<div className="flex flex-wrap items-center gap-2">
<span className="text-muted-foreground">{t('accounts.maxEmailSizeBytes')}:</span>
<span>{currentRow.max_email_size_bytes ? `${(currentRow.max_email_size_bytes / 1024 / 1024).toFixed(0)} MB` : t('accounts.maxEmailSizeBytesUnlimited')}</span>
</div>
<div className="flex flex-col gap-2">
<span className="text-muted-foreground">{t('accounts.capabilities')}:</span>
<code className="rounded-md bg-muted/50 px-2 py-1 text-sm border overflow-x-auto inline-block">
@@ -133,8 +137,8 @@ export function AccountDetailDrawer({ open, onOpenChange, currentRow }: Props) {
)}
</div>
<div className="flex flex-wrap items-center gap-2">
<span className="text-muted-foreground">{t('accounts.folderLimit')}:</span>
<span>{currentRow.folder_limit ? currentRow.folder_limit : t('accounts.notAvailable')}</span>
<span className="text-muted-foreground">{t('accounts.downloadSchedule')}:</span>
<span>{currentRow.download_schedule || t('accounts.notAvailable')}</span>
</div>
</div>
</CardContent>

View File

@@ -19,7 +19,6 @@
import { zodResolver } from '@hookform/resolvers/zod';
import * as React from 'react';
import { useForm } from 'react-hook-form';
import { z } from 'zod';
import { Button } from '@/components/ui/button';
import { Form } from '@/components/ui/form';
import { Dialog, DialogContent, DialogDescription, DialogFooter, DialogHeader, DialogTitle } from '@/components/ui/dialog';
@@ -35,112 +34,9 @@ import { ToastAction } from '@/components/ui/toast';
import { AxiosError } from 'axios';
import { useTranslation } from 'react-i18next';
import { cn } from "@/lib/utils";
import { getAccountSchema, type AccountFormValues } from './schema';
const encryptionSchema = z.union([
z.literal('Ssl'),
z.literal('StartTls'),
z.literal('None'),
]);
const authTypeSchema = z.union([
z.literal('Password'),
z.literal('OAuth2'),
]);
const getAuthConfigSchema = (isEdit: boolean, t: (key: string) => string) =>
z.object({
auth_type: authTypeSchema,
password: z.string().optional(),
}).refine(
(data) => {
if (data.auth_type === 'Password' && !isEdit) {
return !!data.password?.trim();
}
return true;
},
{
message: t('validation.passwordRequired'),
path: ['password'],
}
);
const getImapConfigSchema = (isEdit: boolean, t: (key: string) => string) =>
z.object({
host: z.string({ required_error: t('validation.imapHostRequired') }).min(1, { message: t('validation.imapHostCannotBeEmpty') }),
port: z.number().int().min(0, { message: t('validation.imapPortMustBePositive') }).max(65535, { message: t('validation.imapPortMustBeLessThan65536') }),
encryption: encryptionSchema,
auth: getAuthConfigSchema(isEdit, t),
use_proxy: z.number().optional(),
});
const getRelativeDateSchema = (t: (key: string) => string) => z.object({
unit: z.enum(["Days", "Months", "Years"], { message: t('accounts.selectUnit') }),
value: z.number({ message: t('accounts.enterValue') }).int().min(1, t('accounts.mustBeAtLeast1')),
});
const getDateSelectionSchema = (t: (key: string) => string) => z.union([
z.object({ fixed: z.string({ message: t('accounts.selectDate') }) }),
z.object({ relative: getRelativeDateSchema(t) }),
z.undefined(),
]);
export type Account = {
login_name?: string;
account_name?: string;
email: string;
imap: {
host: string;
port: number;
encryption: 'Ssl' | 'StartTls' | 'None';
auth: {
auth_type: 'Password' | 'OAuth2';
password?: string;
};
use_proxy?: number;
};
enabled: boolean;
use_dangerous: boolean;
date_since?: {
fixed?: string;
relative?: {
unit?: 'Days' | 'Months' | 'Years';
value?: number;
};
};
date_before?: {
unit?: 'Days' | 'Months' | 'Years';
value?: number;
};
folder_limit?: number;
download_interval_min: number;
download_batch_size: number;
auto_download_new_mailboxes: boolean;
};
const getAccountSchema = (isEdit: boolean, t: (key: string) => string) =>
z.object({
account_name: z.string().optional(),
login_name: z.string().optional(),
email: z.string({ required_error: t('validation.emailRequired') }).email({ message: t('validation.invalidEmail') }),
imap: getImapConfigSchema(isEdit, t),
enabled: z.boolean(),
use_dangerous: z.boolean(),
date_since: getDateSelectionSchema(t).optional(),
date_before: getRelativeDateSchema(t).optional(),
folder_limit: z
.number({ invalid_type_error: t('validation.folderLimitMustBeNumber') })
.int()
.min(100, { message: t('validation.folderLimitMustBeAtLeast100') })
.nullable()
.optional(),
download_interval_min: z.number({ invalid_type_error: t('validation.incrementalSyncMustBeNumber') }).int().min(10, { message: t('validation.incrementalSyncMustBeAtLeast10') }),
download_batch_size: z
.number({ invalid_type_error: t('validation.singleRequestBatchSizeMustBeNumber') })
.int()
.min(10, { message: t('validation.singleRequestBatchSizeTooSmall') })
.max(200, { message: t('validation.singleRequestBatchSizeTooLarge') }),
auto_download_new_mailboxes: z.boolean(),
});
export type Account = AccountFormValues;
type Step = {
id: `step-${number}`;
@@ -153,7 +49,7 @@ export type Steps = [...Step[]];
const getSteps = (t: (key: string) => string): Steps => [
{ id: "step-1", name: t('accounts.steps.emailAddress'), fields: ["email", "account_name"] },
{ id: "step-2", name: t('accounts.steps.imap'), fields: ["imap", "use_dangerous", "login_name"] },
{ id: "step-3", name: t('accounts.steps.syncPreferences'), fields: ["enabled", "date_since", "date_before", "folder_limit", "download_interval_min", "download_batch_size", "auto_download_new_mailboxes"] },
{ id: "step-3", name: t('accounts.steps.syncPreferences'), fields: ["enabled", "date_since", "date_before", "download_interval_min", "download_batch_size", "max_email_size_bytes", "auto_download_new_mailboxes", "download_schedule"] },
{ id: "step-4", name: t('accounts.steps.summary'), fields: [] },
];
@@ -183,10 +79,11 @@ const defaultValues: Account = {
use_dangerous: false,
date_since: undefined,
date_before: undefined,
folder_limit: undefined,
download_interval_min: 60,
download_batch_size: 30,
max_email_size_bytes: 100 * 1024 * 1024,
auto_download_new_mailboxes: true,
download_schedule: undefined,
};
const emptyImap: ImapConfig = {
@@ -205,6 +102,7 @@ const mapCurrentRowToFormValues = (currentRow: AccountModel): Account => {
}
return {
account_name: currentRow.account_name ?? undefined,
login_name: currentRow.login_name ?? undefined,
email: currentRow.email,
imap,
@@ -212,10 +110,11 @@ const mapCurrentRowToFormValues = (currentRow: AccountModel): Account => {
use_dangerous: currentRow.use_dangerous,
date_since: currentRow.date_since ?? undefined,
date_before: currentRow.date_before ?? undefined,
folder_limit: currentRow.folder_limit ?? undefined,
download_interval_min: currentRow.download_interval_min ?? 60,
download_batch_size: currentRow.download_batch_size ?? 30,
max_email_size_bytes: currentRow.max_email_size_bytes ?? 100 * 1024 * 1024,
auto_download_new_mailboxes: currentRow.auto_download_new_mailboxes ?? true,
download_schedule: currentRow.download_schedule ?? undefined,
};
};
@@ -294,18 +193,19 @@ export function AccountActionDialog({ currentRow, open, onOpenChange }: Props) {
use_dangerous: data.use_dangerous,
date_since: data.date_since,
date_before: data.date_before,
folder_limit: data.folder_limit,
download_interval_min: data.download_interval_min,
download_batch_size: data.download_batch_size,
max_email_size_bytes: data.max_email_size_bytes,
auto_download_new_mailboxes: data.auto_download_new_mailboxes,
download_schedule: data.download_schedule || null,
};
if (isEdit) {
const isAllMode = !data.date_since && !data.date_before;
const clear_folder_limit = !data.folder_limit;
const clear_download_schedule = !data.download_schedule && currentRow?.download_schedule;
updateMutation.mutate({
...commonData,
...(isAllMode ? { clear_date_range: true } : {}),
...(clear_folder_limit ? { clear_folder_limit: true } : {})
...(clear_download_schedule ? { clear_download_schedule: true } : {})
});
} else {
createMutation.mutate({ ...commonData, account_type: "IMAP" });

View File

@@ -106,9 +106,12 @@ export function useColumns(): ColumnDef<AccountModel>[] {
if (account_type === "NoSync") {
return <LongText className="text-center">n/a</LongText>
}
if (row.original.download_schedule) {
return <LongText className="text-center">{row.original.download_schedule}</LongText>
}
return <LongText className="text-center">{row.original.download_interval_min} min</LongText>
},
meta: { className: 'text-center max-w-[120px]' },
meta: { className: 'text-center max-w-[160px]' },
enableHiding: false,
},
{

View File

@@ -50,13 +50,16 @@ export function DataTableRowActions({ row }: DataTableRowActionsProps) {
const hasPermission = require_any_permission(['system:root', 'account:manage'], row.original.id);
const hasReadPermission = require_any_permission(['system:root', 'account:read_details'], row.original.id);
const isDeleting = row.original.deleting === true;
const canShowAnyAction =
!isDeleting && (
(hasPermission) ||
(account_type === 'IMAP' && hasPermission) ||
(account_type === 'IMAP' && hasReadPermission);
(account_type === 'IMAP' && hasReadPermission)
);
const showDownload = account_type === 'IMAP' && hasPermission;
const showDownload = !isDeleting && account_type === 'IMAP' && hasPermission;
const handleStartDownload = async () => {
try {
@@ -98,26 +101,6 @@ export function DataTableRowActions({ row }: DataTableRowActionsProps) {
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align='end' className='w-[220px]'>
{showDownload && (
<DropdownMenuItem onClick={handleStartDownload}>
{t('accounts.startDownload')}
<DropdownMenuShortcut>
<IconPlayerPlay size={16} />
</DropdownMenuShortcut>
</DropdownMenuItem>
)}
{showDownload && (
<DropdownMenuItem onClick={handleCancelDownload}>
{t('accounts.cancelDownload')}
<DropdownMenuShortcut>
<IconPlayerStop size={16} />
</DropdownMenuShortcut>
</DropdownMenuItem>
)}
{showDownload && <DropdownMenuSeparator />}
{hasPermission && <DropdownMenuItem
onClick={() => {
setCurrentRow(row.original)
@@ -169,6 +152,27 @@ export function DataTableRowActions({ row }: DataTableRowActionsProps) {
</DropdownMenuShortcut>
</DropdownMenuItem>}
{hasPermission && <DropdownMenuSeparator />}
{showDownload && (
<DropdownMenuItem onClick={handleStartDownload}>
{t('accounts.startDownload')}
<DropdownMenuShortcut>
<IconPlayerPlay size={16} />
</DropdownMenuShortcut>
</DropdownMenuItem>
)}
{showDownload && (
<DropdownMenuItem onClick={handleCancelDownload}>
{t('accounts.cancelDownload')}
<DropdownMenuShortcut>
<IconPlayerStop size={16} />
</DropdownMenuShortcut>
</DropdownMenuItem>
)}
{showDownload && <DropdownMenuSeparator />}
{hasPermission && <DropdownMenuItem
onClick={() => {
setCurrentRow(row.original)

View File

@@ -43,8 +43,8 @@ export function AccountDeleteDialog({ open, onOpenChange, currentRow }: Props) {
const queryClient = useQueryClient();
function handleSuccess() {
toast({
title: t('dialogs.accountDeleted'),
description: t('dialogs.accountDeletedDesc'),
title: t('dialogs.accountDeletionStarted'),
description: t('dialogs.accountDeletionStartedDesc'),
action: <ToastAction altText={t('common.close')}>{t('common.close')}</ToastAction>,
});

View File

@@ -77,7 +77,7 @@ export function EnableAction({ row }: DataTableRowActionsProps) {
<Switch
checked={row.original.enabled}
onCheckedChange={() => setOpen(true)}
disabled={!hasPermission || updateMutation.isPending}
disabled={!hasPermission || updateMutation.isPending || row.original.deleting}
/>
<ConfirmDialog
open={open}

View File

@@ -35,6 +35,9 @@ export function RunningStateCellAction({ row }: Props) {
const { setOpen, setCurrentRow } = useAccountContext()
const { require_any_permission } = useCurrentUser()
if (row.original.deleting) {
return <span className="text-xs text-muted-foreground italic">Deleting...</span>
}
let account_type = row.original.account_type;
if (account_type === "NoSync") {
return <span className="text-xs text-muted-foreground">n/a</span>

View File

@@ -0,0 +1,127 @@
import { z } from 'zod'
const encryptionSchema = z.union([
z.literal('Ssl'),
z.literal('StartTls'),
z.literal('None'),
])
const authTypeSchema = z.union([
z.literal('Password'),
z.literal('OAuth2'),
])
export const getAuthConfigSchema = (isEdit: boolean, t: (key: string) => string) =>
z
.object({
auth_type: authTypeSchema,
password: z.string().optional(),
})
.refine(
(data) => {
if (data.auth_type === 'Password' && !isEdit) {
return !!data.password?.trim()
}
return true
},
{
message: t('validation.passwordRequired'),
path: ['password'],
}
)
export const getImapConfigSchema = (isEdit: boolean, t: (key: string) => string) =>
z.object({
host: z
.string({ required_error: t('validation.imapHostRequired') })
.min(1, { message: t('validation.imapHostCannotBeEmpty') }),
port: z
.number()
.int()
.min(0, { message: t('validation.imapPortMustBePositive') })
.max(65535, { message: t('validation.imapPortMustBeLessThan65536') }),
encryption: encryptionSchema,
auth: getAuthConfigSchema(isEdit, t),
use_proxy: z.number().optional(),
})
const relativeDateSchema = (t: (key: string) => string) =>
z.object({
unit: z.enum(['Days', 'Months', 'Years'], {
message: t('accounts.selectUnit'),
}),
value: z
.number({ message: t('accounts.enterValue') })
.int()
.min(1, t('accounts.mustBeAtLeast1')),
})
const dateSelectionSchema = (t: (key: string) => string) =>
z
.object({
fixed: z
.string({ message: t('accounts.selectDate') })
.min(1, { message: t('accounts.selectDate') })
.optional(),
relative: relativeDateSchema(t).optional(),
})
.optional()
export const getAccountSchema = (isEdit: boolean, t: (key: string) => string) =>
z.object({
account_name: z.string().optional(),
login_name: z.string().optional(),
email: z
.string({ required_error: t('validation.emailRequired') })
.email({ message: t('validation.invalidEmail') }),
imap: getImapConfigSchema(isEdit, t),
enabled: z.boolean(),
use_dangerous: z.boolean(),
date_since: dateSelectionSchema(t).optional(),
date_before: relativeDateSchema(t).optional(),
download_interval_min: z
.number({
invalid_type_error: t('validation.incrementalSyncMustBeNumber'),
})
.int()
.min(10, {
message: t('validation.incrementalSyncMustBeAtLeast10'),
}),
download_batch_size: z
.number({
invalid_type_error: t(
'validation.singleRequestBatchSizeMustBeNumber'
),
})
.int()
.min(10, {
message: t('validation.singleRequestBatchSizeTooSmall'),
})
.max(200, {
message: t('validation.singleRequestBatchSizeTooLarge'),
}),
max_email_size_bytes: z
.number({
invalid_type_error: t('validation.maxEmailSizeMustBeNumber'),
})
.int()
.min(1 * 1024 * 1024, { message: t('validation.maxEmailSizeTooSmall') })
.max(100 * 1024 * 1024, { message: t('validation.maxEmailSizeTooLarge') }),
auto_download_new_mailboxes: z.boolean(),
download_schedule: z
.string()
.optional()
.refine(
(val) => {
if (!val || val.trim() === '') return true;
const fields = val.trim().split(/\s+/);
if (fields.length < 6) return false;
return true;
},
{ message: t('validation.invalidCronExpression') }
),
})
export type AccountFormValues = z.infer<
ReturnType<typeof getAccountSchema>
>

View File

@@ -48,6 +48,72 @@ import i18n from "@/i18n";
type SyncMode = 'all' | 'since_fixed' | 'since_relative' | 'before_relative';
type ScheduleMode = 'interval' | 'cron';
type CronMode = 'simple' | 'advanced';
type CronFrequency = 'daily' | 'weekly' | 'monthly';
interface CronSimpleState {
frequency: CronFrequency;
hour: number;
minute: number;
dayOfWeek: number;
dayOfMonth: number;
}
const DEFAULT_CRON_SIMPLE: CronSimpleState = {
frequency: 'daily',
hour: 0,
minute: 0,
dayOfWeek: 1,
dayOfMonth: 1,
};
function buildCronFromSimple(s: CronSimpleState): string {
switch (s.frequency) {
case 'daily':
return `0 ${s.minute} ${s.hour} * * *`;
case 'weekly':
return `0 ${s.minute} ${s.hour} * * ${s.dayOfWeek}`;
case 'monthly':
return `0 ${s.minute} ${s.hour} ${s.dayOfMonth} * *`;
}
}
function tryParseCronToSimple(cron: string): CronSimpleState | null {
const fields = cron.trim().split(/\s+/);
if (fields.length < 6) return null;
const sec = fields[0];
const min = fields[1];
const hour = fields[2];
const dom = fields[3];
const month = fields[4];
const dow = fields[5];
if (sec !== '0') return null;
if (month !== '*') return null;
const minuteVal = parseInt(min, 10);
const hourVal = parseInt(hour, 10);
if (isNaN(minuteVal) || isNaN(hourVal)) return null;
if (dom === '*' && dow === '*') {
return { frequency: 'daily', hour: hourVal, minute: minuteVal, dayOfWeek: 1, dayOfMonth: 1 };
}
if (dom === '*') {
const dowVal = parseInt(dow, 10);
if (!isNaN(dowVal)) {
return { frequency: 'weekly', hour: hourVal, minute: minuteVal, dayOfWeek: dowVal, dayOfMonth: 1 };
}
}
if (dow === '*') {
const domVal = parseInt(dom, 10);
if (!isNaN(domVal)) {
return { frequency: 'monthly', hour: hourVal, minute: minuteVal, dayOfWeek: 1, dayOfMonth: domVal };
}
}
return null;
}
export default function Step3() {
const { t } = useTranslation();
@@ -61,6 +127,31 @@ export default function Step3() {
return 'all';
});
const [scheduleMode, setScheduleMode] = useState<ScheduleMode>(() => {
if (current.download_schedule) return 'cron';
return 'interval';
});
const [cronMode, setCronMode] = useState<CronMode>(() => {
if (current.download_schedule && tryParseCronToSimple(current.download_schedule)) {
return 'simple';
}
if (current.download_schedule) return 'advanced';
return 'simple';
});
const [cronSimple, setCronSimple] = useState<CronSimpleState>(() => {
if (current.download_schedule) {
return tryParseCronToSimple(current.download_schedule) ?? DEFAULT_CRON_SIMPLE;
}
return DEFAULT_CRON_SIMPLE;
});
const updateCronFromSimple = (partial: Partial<CronSimpleState>) => {
const next = { ...cronSimple, ...partial };
setCronSimple(next);
setValue('download_schedule', buildCronFromSimple(next));
};
const handleModeChange = (mode: SyncMode) => {
setSyncMode(mode);
@@ -77,41 +168,263 @@ export default function Step3() {
}
};
const handleScheduleModeChange = (mode: ScheduleMode) => {
setScheduleMode(mode);
if (mode === 'interval') {
setValue("download_schedule", undefined);
} else {
setValue("download_interval_min", 60);
if (cronMode === 'simple') {
updateCronFromSimple(cronSimple);
}
}
};
return (
<div className="space-y-8">
<div className="grid grid-cols-1 md:grid-cols-2 gap-6">
<FormField
control={control}
name="download_interval_min"
render={({ field }) => (
<FormItem>
<FormLabel>{t('accounts.downloadInterval')}</FormLabel>
<FormControl>
<Input type="number" {...field} onChange={(e) => field.onChange(parseInt(e.target.value, 10))} />
</FormControl>
<FormMessage />
<FormDescription>
{t('accounts.downloadIntervalPlaceholder')}
</FormDescription>
</FormItem>
<div className="space-y-4">
<FormItem>
<FormLabel className="text-base font-semibold">{t('accounts.scheduleMode')}</FormLabel>
<FormDescription>
{t('accounts.scheduleModeDescription')}
</FormDescription>
<Select value={scheduleMode} onValueChange={(v) => handleScheduleModeChange(v as ScheduleMode)}>
<SelectTrigger className="w-full">
<SelectValue />
</SelectTrigger>
<SelectContent>
<SelectItem value="interval">{t('accounts.scheduleModeInterval')}</SelectItem>
<SelectItem value="cron">{t('accounts.scheduleModeCron')}</SelectItem>
</SelectContent>
</Select>
</FormItem>
<div className="grid grid-cols-1 md:grid-cols-2 gap-6">
{scheduleMode === 'interval' ? (
<FormField
control={control}
name="download_interval_min"
render={({ field }) => (
<FormItem>
<FormLabel>{t('accounts.downloadInterval')}</FormLabel>
<FormControl>
<Input type="number" {...field} onChange={(e) => field.onChange(parseInt(e.target.value, 10))} />
</FormControl>
<FormMessage />
<FormDescription>
{t('accounts.downloadIntervalPlaceholder')}
</FormDescription>
</FormItem>
)}
/>
) : (
<div className="space-y-3">
<div className="flex items-center gap-2">
{/* <FormLabel className="text-sm font-medium">{t('accounts.downloadSchedule')}</FormLabel> */}
<div className="flex items-center rounded-md border text-xs">
<button
type="button"
className={`px-2 py-1 rounded-l-md ${cronMode === 'simple' ? 'bg-primary text-primary-foreground' : 'text-muted-foreground hover:text-foreground'}`}
onClick={() => setCronMode('simple')}
>
{t('accounts.cronSimple')}
</button>
<button
type="button"
className={`px-2 py-1 rounded-r-md ${cronMode === 'advanced' ? 'bg-primary text-primary-foreground' : 'text-muted-foreground hover:text-foreground'}`}
onClick={() => setCronMode('advanced')}
>
{t('accounts.cronAdvanced')}
</button>
</div>
</div>
{cronMode === 'simple' ? (
<div className="flex flex-wrap items-end gap-3">
<FormItem className="w-[140px]">
<FormLabel className="text-xs">{t('accounts.cronFrequency')}</FormLabel>
<Select
value={cronSimple.frequency}
onValueChange={(v) => updateCronFromSimple({ frequency: v as CronFrequency })}
>
<SelectTrigger>
<SelectValue />
</SelectTrigger>
<SelectContent>
<SelectItem value="daily">{t('accounts.cronDaily')}</SelectItem>
<SelectItem value="weekly">{t('accounts.cronWeekly')}</SelectItem>
<SelectItem value="monthly">{t('accounts.cronMonthly')}</SelectItem>
</SelectContent>
</Select>
</FormItem>
<FormItem className="w-[80px]">
<FormLabel className="text-xs">{t('accounts.cronHour')}</FormLabel>
<Select
value={String(cronSimple.hour)}
onValueChange={(v) => updateCronFromSimple({ hour: parseInt(v, 10) })}
>
<SelectTrigger>
<SelectValue />
</SelectTrigger>
<SelectContent>
{Array.from({ length: 24 }, (_, i) => (
<SelectItem key={i} value={String(i)}>
{String(i).padStart(2, '0')}
</SelectItem>
))}
</SelectContent>
</Select>
</FormItem>
<span className="text-muted-foreground pb-2">:</span>
<FormItem className="w-[80px]">
<FormLabel className="text-xs">{t('accounts.cronMinute')}</FormLabel>
<Select
value={String(cronSimple.minute)}
onValueChange={(v) => updateCronFromSimple({ minute: parseInt(v, 10) })}
>
<SelectTrigger>
<SelectValue />
</SelectTrigger>
<SelectContent>
{[0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55].map((m) => (
<SelectItem key={m} value={String(m)}>
{String(m).padStart(2, '0')}
</SelectItem>
))}
</SelectContent>
</Select>
</FormItem>
{cronSimple.frequency === 'weekly' && (
<FormItem className="w-[140px]">
<FormLabel className="text-xs">{t('accounts.cronDayOfWeek')}</FormLabel>
<Select
value={String(cronSimple.dayOfWeek)}
onValueChange={(v) => updateCronFromSimple({ dayOfWeek: parseInt(v, 10) })}
>
<SelectTrigger>
<SelectValue />
</SelectTrigger>
<SelectContent>
<SelectItem value="1">{t('accounts.cronMonday')}</SelectItem>
<SelectItem value="2">{t('accounts.cronTuesday')}</SelectItem>
<SelectItem value="3">{t('accounts.cronWednesday')}</SelectItem>
<SelectItem value="4">{t('accounts.cronThursday')}</SelectItem>
<SelectItem value="5">{t('accounts.cronFriday')}</SelectItem>
<SelectItem value="6">{t('accounts.cronSaturday')}</SelectItem>
<SelectItem value="0">{t('accounts.cronSunday')}</SelectItem>
</SelectContent>
</Select>
</FormItem>
)}
{cronSimple.frequency === 'monthly' && (
<FormItem className="w-[90px]">
<FormLabel className="text-xs">{t('accounts.cronDayOfMonth')}</FormLabel>
<Select
value={String(cronSimple.dayOfMonth)}
onValueChange={(v) => updateCronFromSimple({ dayOfMonth: parseInt(v, 10) })}
>
<SelectTrigger>
<SelectValue />
</SelectTrigger>
<SelectContent className="max-h-[200px]">
{Array.from({ length: 28 }, (_, i) => i + 1).map((d) => (
<SelectItem key={d} value={String(d)}>
{d}
</SelectItem>
))}
</SelectContent>
</Select>
</FormItem>
)}
<div className="text-xs text-muted-foreground pb-2 font-mono">
= {buildCronFromSimple(cronSimple)}
</div>
</div>
) : (
<FormField
control={control}
name="download_schedule"
render={({ field }) => (
<FormItem>
<FormControl>
<Input
{...field}
value={field.value ?? ''}
placeholder={t('accounts.downloadSchedulePlaceholder')}
/>
</FormControl>
<FormDescription>
{t('accounts.downloadScheduleDescription')}
</FormDescription>
<FormMessage />
</FormItem>
)}
/>
)}
{cronMode === 'simple' && (
<FormDescription>{t('accounts.downloadScheduleDescription')}</FormDescription>
)}
<div className="text-xs text-muted-foreground mt-1 flex items-center gap-1">
<span>{t('accounts.cronTimezoneNote')}</span>
</div>
</div>
)}
/>
<FormField
control={control}
name="download_batch_size"
render={({ field }) => (
<FormItem>
<FormLabel>{t('accounts.downloadBatchSize')}</FormLabel>
<FormControl>
<Input type="number" {...field} onChange={(e) => field.onChange(parseInt(e.target.value, 10))} />
</FormControl>
<FormMessage />
<FormDescription>
{t('accounts.downloadBatchSizeDescription')}
</FormDescription>
</FormItem>
)}
/>
<FormField
control={control}
name="download_batch_size"
render={({ field }) => (
<FormItem>
<FormLabel>{t('accounts.downloadBatchSize')}</FormLabel>
<FormControl>
<Input type="number" {...field} onChange={(e) => field.onChange(parseInt(e.target.value, 10))} />
</FormControl>
<FormMessage />
<FormDescription>
{t('accounts.downloadBatchSizeDescription')}
</FormDescription>
</FormItem>
)}
/>
<FormField
control={control}
name="max_email_size_bytes"
render={({ field }) => {
const BYTES_PER_MB = 1024 * 1024;
return (
<FormItem>
<FormLabel>{t('accounts.maxEmailSizeBytes')}</FormLabel>
<FormControl>
<div className="flex items-center gap-2">
<Input
type="number"
placeholder={t('accounts.maxEmailSizeBytesPlaceholder')}
className="flex-1"
value={field.value ? field.value / BYTES_PER_MB : ''}
onChange={(e) => {
const parsed = parseInt(e.target.value, 10);
field.onChange(isNaN(parsed) ? parsed : parsed * BYTES_PER_MB);
}}
/>
<span className="text-sm text-muted-foreground whitespace-nowrap">MB</span>
</div>
</FormControl>
<FormMessage />
<FormDescription>
{t('accounts.maxEmailSizeBytesDescription')}
</FormDescription>
</FormItem>
);
}}
/>
</div>
</div>
<FormField
@@ -249,29 +562,6 @@ export default function Step3() {
/>
<hr className="my-4" />
<FormField
control={control}
name="folder_limit"
render={({ field }) => (
<FormItem>
<FormLabel>{t('accounts.folderLimit')}</FormLabel>
<FormDescription>{t('accounts.folderLimitDescription')}</FormDescription>
<FormControl>
<Input
type="number"
placeholder={t('accounts.folderLimitPlaceholder')}
value={field.value ?? ''}
onChange={(e) => {
const value = e.target.value;
field.onChange(value === '' ? null : Number(value));
}}
/>
</FormControl>
<FormMessage />
</FormItem>
)}
/>
</div>
);
}

View File

@@ -50,7 +50,11 @@ export default function Step4() {
return (
<div className="rounded-xl">
<Accordion type="multiple" defaultValue={['email', 'account_name', 'login_name', 'imap', 'date_since', 'folder_limit', 'sync_interval', 'sync_scope', 'sync_batch_size']}>
<Accordion type="multiple" defaultValue={[
'email', 'account_name', 'login_name', 'imap', 'date_since',
'max_email_size_bytes', 'sync_interval', 'sync_scope',
'sync_batch_size', 'download_schedule'
]}>
<AccordionItem key="email" value="email">
<AccordionTrigger className="font-medium capitalize text-gray-600">{t('accounts.email')}:</AccordionTrigger>
<AccordionContent>{summaryData.email}</AccordionContent>
@@ -154,11 +158,6 @@ export default function Step4() {
</AccordionItem>
<AccordionItem key="folder_limit" value="folder_limit">
<AccordionTrigger className="font-medium capitalize text-gray-600">{t('accounts.folderLimit')}:</AccordionTrigger>
<AccordionContent>{summaryData.folder_limit ?? t('accounts.notAvailable')}</AccordionContent>
</AccordionItem>
<AccordionItem key="sync_interval" value="sync_interval">
<AccordionTrigger className="font-medium capitalize text-gray-600">{t('accounts.downloadInterval')}:</AccordionTrigger>
<AccordionContent>{summaryData.download_interval_min} {t('accounts.minutes')}</AccordionContent>
@@ -169,6 +168,16 @@ export default function Step4() {
<AccordionContent>{summaryData.download_batch_size}</AccordionContent>
</AccordionItem>
<AccordionItem key="max_email_size_bytes" value="max_email_size_bytes">
<AccordionTrigger className="font-medium capitalize text-gray-600">{t('accounts.maxEmailSizeBytes')}:</AccordionTrigger>
<AccordionContent>{summaryData.max_email_size_bytes ? `${(summaryData.max_email_size_bytes / 1024 / 1024).toFixed(0)} MB` : t('accounts.maxEmailSizeBytesUnlimited')}</AccordionContent>
</AccordionItem>
<AccordionItem key="download_schedule" value="download_schedule">
<AccordionTrigger className="font-medium capitalize text-gray-600">{t('accounts.downloadSchedule')}:</AccordionTrigger>
<AccordionContent>{summaryData.download_schedule || t('accounts.notAvailable')}</AccordionContent>
</AccordionItem>
<AccordionItem key="auto_download_new_mailboxes" value="auto_download_new_mailboxes">
<AccordionTrigger className="font-medium capitalize text-gray-600">{t('accounts.autoDownloadNewMailboxes')}:</AccordionTrigger>
<AccordionContent>{summaryData.auto_download_new_mailboxes ? t('common.yes') : t('common.no')}</AccordionContent>

View File

@@ -17,7 +17,7 @@
// along with this program. If not, see <http://www.gnu.org/licenses/>.
import { useState } from 'react'
import { useEffect, useRef, useState } from 'react'
import {
ColumnDef,
ColumnFiltersState,
@@ -64,7 +64,19 @@ export function AccountTable({ columns, data }: DataTableProps) {
const [rowSelection, setRowSelection] = useState({})
const [columnVisibility, setColumnVisibility] = useState<VisibilityState>({})
const [columnFilters, setColumnFilters] = useState<ColumnFiltersState>([])
const [sorting, setSorting] = useState<SortingState>([])
const [sorting, setSorting] = useState<SortingState>(() => {
const saved = localStorage.getItem('bichon_accounts_sorting');
return saved ? JSON.parse(saved) : [];
})
// Persist sorting state to localStorage
const prevSortingRef = useRef(sorting);
useEffect(() => {
if (prevSortingRef.current !== sorting) {
localStorage.setItem('bichon_accounts_sorting', JSON.stringify(sorting));
prevSortingRef.current = sorting;
}
}, [sorting]);
const table = useReactTable({
data,
@@ -127,7 +139,7 @@ export function AccountTable({ columns, data }: DataTableProps) {
<TableRow
key={row.id}
data-state={row.getIsSelected() && 'selected'}
className='group/row'
className={row.original.deleting ? 'opacity-50' : 'group/row'}
>
{row.getVisibleCells().map((cell) => (
<TableCell

View File

@@ -55,6 +55,10 @@ export default function Accounts() {
const { data: accountList, isLoading } = useQuery({
queryKey: ['account-list'],
queryFn: list_accounts,
refetchInterval: (query) => {
const items = (query.state.data as { items?: { deleting?: boolean }[] })?.items;
return items?.some((item) => item.deleting) ? 5000 : false;
},
})
const hasAccounts = accountList != null && accountList.items.length > 0;

View File

@@ -19,7 +19,7 @@
import { useEffect, useState } from 'react';
import { useMutation } from '@tanstack/react-query';
import { Loader, Download, Trash2, MessageSquareMore, FileText, FileImage, FileVideo, FileArchive, FileSpreadsheet, FileCode, FileIcon, FileAudio, Upload } from 'lucide-react';
import { Loader, Download, Trash2, MessageSquareMore, FileText, FileImage, FileVideo, FileArchive, FileSpreadsheet, FileCode, FileIcon, FileAudio, Upload, ShieldCheck } from 'lucide-react';
import { Button } from '@/components/ui/button';
import { Separator } from '@/components/ui/separator';
@@ -121,6 +121,12 @@ export function MailMessageView({
const [nestedEmlFile, setNestedEmlFile] = useState<AttachmentInfo | null>(null);
const { getEmailById } = useMinimalAccountList();
const [threadOpen, setThreadOpen] = useState(false);
const [blockRemote, setBlockRemote] = useState(true);
const [hasRemoteContent, setHasRemoteContent] = useState(false);
const toggleBlockRemote = () => {
setBlockRemote((prev) => !prev);
};
const downloadAttachmentMutation = useMutation({
mutationFn: ({ content_hash }: { content_hash: string }) =>
@@ -137,12 +143,13 @@ export function MailMessageView({
});
const loadMessageMutation = useMutation({
mutationFn: () => load_message(envelope.account_id, envelope.id),
mutationFn: () => load_message(envelope.account_id, envelope.id, blockRemote),
onSuccess: (data) => {
setLoading(false);
setContent(getContent(data));
if (data.attachments) setAttachments(data.attachments);
setContentType(data.html ? 'Html' : 'Plain');
setHasRemoteContent(!!data.has_remote_content);
},
onError: (error: any) => {
setLoading(false);
@@ -154,10 +161,14 @@ export function MailMessageView({
},
});
useEffect(() => {
setBlockRemote(true);
}, [envelope.id]);
useEffect(() => {
setLoading(true);
loadMessageMutation.mutate();
}, [envelope.id]);
}, [envelope.id, blockRemote]);
const handleViewNestedEml = (attachment: AttachmentInfo) => {
@@ -376,6 +387,30 @@ export function MailMessageView({
</div>
)}
{showAttachments && <Separator className="mb-2" />}
{hasRemoteContent && (
<div className="flex items-center justify-between bg-muted border px-3 py-1.5 mb-3 text-xs">
<div className="flex items-center gap-1.5 min-w-0">
<ShieldCheck className="h-3.5 w-3.5 text-muted-foreground shrink-0" />
{blockRemote ? (
<span className="text-muted-foreground truncate">
{t('mail.remoteBlocked', 'To protect your privacy, Bichon has blocked remote content in this message.')}
</span>
) : (
<span className="text-muted-foreground truncate">
{t('mail.remoteShown', 'Remote content is now shown.')}
</span>
)}
</div>
<span
className="underline cursor-pointer hover:no-underline text-muted-foreground text-[11px] font-medium shrink-0 ml-2 select-none"
onClick={toggleBlockRemote}
>
{blockRemote
? t('mail.showRemoteContent', 'Show remote content')
: t('mail.blockRemoteAgain', 'Block again')}
</span>
</div>
)}
<div className="flex-1 overflow-auto">
{loading ? (
<div className="flex justify-center items-center py-8">

View File

@@ -150,7 +150,7 @@ export function NestedEmailDialog({ open, onOpenChange }: any) {
const { data, isLoading } = useQuery({
queryKey: ['nested-message', currentAttachment?.account_id!, currentAttachment?.envelope_id!, currentAttachment?.content_hash!],
queryFn: () => load_nested_message(currentAttachment?.account_id!, currentAttachment?.envelope_id!, currentAttachment?.content_hash!),
queryFn: () => load_nested_message(currentAttachment?.account_id!, currentAttachment?.envelope_id!, currentAttachment?.content_hash!, true),
enabled: open && !!currentAttachment,
});

View File

@@ -0,0 +1,81 @@
import { describe, it, expect } from 'vitest'
import { getFormSchema } from '../schema'
// Simple mock t function that returns the key
const t = (key: string, _options?: Record<string, any>) => key
describe('Login Form Schema', () => {
const schema = getFormSchema(t)
describe('username field', () => {
it('rejects empty username', () => {
const result = schema.safeParse({ username: '', password: 'abcd' })
expect(result.success).toBe(false)
if (!result.success) {
const usernameErrors = result.error.issues.filter(
(i) => i.path[0] === 'username'
)
expect(usernameErrors.length).toBeGreaterThan(0)
}
})
it('accepts valid username with password', () => {
const result = schema.safeParse({ username: 'admin', password: 'pass1234' })
expect(result.success).toBe(true)
})
it('accepts email as username', () => {
const result = schema.safeParse({
username: 'user@example.com',
password: 'mypassword',
})
expect(result.success).toBe(true)
})
})
describe('password field', () => {
it('rejects empty password', () => {
const result = schema.safeParse({ username: 'admin', password: '' })
expect(result.success).toBe(false)
if (!result.success) {
const passwordErrors = result.error.issues.filter(
(i) => i.path[0] === 'password'
)
expect(passwordErrors.length).toBeGreaterThan(0)
}
})
it('rejects password shorter than 4 characters', () => {
const result = schema.safeParse({ username: 'admin', password: 'ab' })
expect(result.success).toBe(false)
})
it('accepts password of exactly 4 characters', () => {
const result = schema.safeParse({
username: 'admin',
password: 'abcd',
})
expect(result.success).toBe(true)
})
it('accepts long password', () => {
const result = schema.safeParse({
username: 'admin',
password: 'a'.repeat(256),
})
expect(result.success).toBe(true)
})
})
describe('missing fields', () => {
it('rejects empty object', () => {
const result = schema.safeParse({})
expect(result.success).toBe(false)
})
it('rejects object with only username', () => {
const result = schema.safeParse({ username: 'admin' })
expect(result.success).toBe(false)
})
})
})

View File

@@ -0,0 +1,16 @@
import { z } from 'zod'
export const getFormSchema = (
t: (key: string, options?: Record<string, any>) => string
) =>
z.object({
username: z
.string()
.min(1, { message: t('validation.pleaseEnterUsernameOrEmail') }),
password: z
.string()
.min(1, { message: t('validation.pleaseEnterPassword') })
.min(4, { message: t('validation.passwordMinLength', { min: 4 }) }),
})
export type LoginFormValues = z.infer<ReturnType<typeof getFormSchema>>

View File

@@ -18,10 +18,10 @@
import { HTMLAttributes, useState } from 'react'
import { z } from 'zod'
import { useForm } from 'react-hook-form'
import { zodResolver } from '@hookform/resolvers/zod'
import { cn, toSearchParams } from '@/lib/utils'
import { getFormSchema, type LoginFormValues } from './schema'
import {
Form,
FormControl,
@@ -47,17 +47,6 @@ import { useTheme } from '@/context/theme-context'
type UserAuthFormProps = HTMLAttributes<HTMLDivElement>
const getFormSchema = (t: (key: string, options?: Record<string, any>) => string) =>
z.object({
username: z
.string()
.min(1, { message: t('validation.pleaseEnterUsernameOrEmail') }),
password: z
.string()
.min(1, { message: t('validation.pleaseEnterPassword') })
.min(4, { message: t('validation.passwordMinLength', { min: 4 }) }),
});
export function UserAuthForm({ className, ...props }: UserAuthFormProps) {
const [isLoading, setIsLoading] = useState(false)
const { setTheme } = useTheme();
@@ -68,7 +57,7 @@ export function UserAuthForm({ className, ...props }: UserAuthFormProps) {
const redirect = toSearchParams(search).get('redirect') || '/';
const formSchema = getFormSchema(t)
const form = useForm<z.infer<typeof formSchema>>({
const form = useForm<LoginFormValues>({
resolver: zodResolver(formSchema),
defaultValues: {
username: '',
@@ -81,7 +70,7 @@ export function UserAuthForm({ className, ...props }: UserAuthFormProps) {
retry: 0,
});
async function onSubmit(data: z.infer<typeof formSchema>) {
async function onSubmit(data: LoginFormValues) {
setIsLoading(true)
mutation.mutate(data, {

View File

@@ -0,0 +1,189 @@
import { describe, it, expect } from 'vitest'
import { getOAuth2Schema } from '../schema'
const t = (key: string) => key
describe('OAuth2 Form Schema', () => {
const schema = getOAuth2Schema(t)
const validData = {
client_id: 'my-client-id',
auth_url: 'https://accounts.example.com/o/oauth2/auth',
token_url: 'https://oauth2.example.com/token',
redirect_uri: 'https://myapp.example.com/oauth2/callback',
enabled: true,
}
describe('client_id field', () => {
it('rejects empty client_id', () => {
const result = schema.safeParse({ ...validData, client_id: '' })
expect(result.success).toBe(false)
})
it('accepts valid client_id', () => {
const result = schema.safeParse(validData)
expect(result.success).toBe(true)
})
})
describe('client_secret field', () => {
it('accepts undefined client_secret', () => {
const result = schema.safeParse(validData)
expect(result.success).toBe(true)
})
it('accepts provided client_secret', () => {
const result = schema.safeParse({
...validData,
client_secret: 'my-secret',
})
expect(result.success).toBe(true)
})
})
describe('auth_url field', () => {
it('rejects empty auth_url', () => {
const result = schema.safeParse({ ...validData, auth_url: '' })
expect(result.success).toBe(false)
})
it('rejects invalid URL format for auth_url', () => {
const result = schema.safeParse({
...validData,
auth_url: 'not-a-url',
})
expect(result.success).toBe(false)
})
it('accepts valid auth_url', () => {
const result = schema.safeParse(validData)
expect(result.success).toBe(true)
})
})
describe('token_url field', () => {
it('rejects empty token_url', () => {
const result = schema.safeParse({ ...validData, token_url: '' })
expect(result.success).toBe(false)
})
it('rejects invalid URL format for token_url', () => {
const result = schema.safeParse({
...validData,
token_url: 'not-a-url',
})
expect(result.success).toBe(false)
})
})
describe('redirect_uri field', () => {
it('rejects empty redirect_uri', () => {
const result = schema.safeParse({ ...validData, redirect_uri: '' })
expect(result.success).toBe(false)
})
it('rejects invalid URL format for redirect_uri', () => {
const result = schema.safeParse({
...validData,
redirect_uri: 'not-a-url',
})
expect(result.success).toBe(false)
})
})
describe('scopes field', () => {
it('accepts empty scopes array', () => {
const result = schema.safeParse({ ...validData, scopes: [] })
expect(result.success).toBe(true)
})
it('accepts valid scopes', () => {
const result = schema.safeParse({
...validData,
scopes: [{ value: 'https://mail.google.com/' }],
})
expect(result.success).toBe(true)
})
it('rejects scope with empty value', () => {
const result = schema.safeParse({
...validData,
scopes: [{ value: '' }],
})
expect(result.success).toBe(false)
})
})
describe('extra_params field', () => {
it('accepts empty extra_params array', () => {
const result = schema.safeParse({ ...validData, extra_params: [] })
expect(result.success).toBe(true)
})
it('accepts valid extra_params', () => {
const result = schema.safeParse({
...validData,
extra_params: [{ key: 'access_type', value: 'offline' }],
})
expect(result.success).toBe(true)
})
it('rejects param with empty key', () => {
const result = schema.safeParse({
...validData,
extra_params: [{ key: '', value: 'offline' }],
})
expect(result.success).toBe(false)
})
it('rejects param with empty value', () => {
const result = schema.safeParse({
...validData,
extra_params: [{ key: 'access_type', value: '' }],
})
expect(result.success).toBe(false)
})
})
describe('enabled field', () => {
it('accepts enabled: true', () => {
const result = schema.safeParse(validData)
expect(result.success).toBe(true)
})
it('accepts enabled: false', () => {
const result = schema.safeParse({ ...validData, enabled: false })
expect(result.success).toBe(true)
})
})
describe('description field', () => {
it('rejects description longer than 255 characters', () => {
const result = schema.safeParse({
...validData,
description: 'a'.repeat(256),
})
expect(result.success).toBe(false)
})
it('accepts description of exactly 255 characters', () => {
const result = schema.safeParse({
...validData,
description: 'a'.repeat(255),
})
expect(result.success).toBe(true)
})
})
describe('use_proxy field', () => {
it('accepts undefined use_proxy', () => {
const result = schema.safeParse(validData)
expect(result.success).toBe(true)
})
it('accepts numeric use_proxy', () => {
const result = schema.safeParse({ ...validData, use_proxy: 1 })
expect(result.success).toBe(true)
})
})
})

View File

@@ -17,7 +17,6 @@
// along with this program. If not, see <http://www.gnu.org/licenses/>.
import { z } from 'zod'
import { useFieldArray, useForm } from 'react-hook-form'
import { zodResolver } from '@hookform/resolvers/zod'
import { toast } from '@/hooks/use-toast'
@@ -53,115 +52,22 @@ import { AxiosError } from 'axios'
import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from '@/components/ui/select'
import useProxyList from '@/hooks/use-proxy'
import { useTranslation } from 'react-i18next'
const getParamSchema = (t: (key: string) => string) => z.object({
key: z.string({ required_error: t('oauth2.keyIsRequired') }).min(1, t('oauth2.keyCannotBeEmpty')),
value: z.string({ required_error: t('oauth2.valueIsRequired') }).min(1, t('oauth2.valueCannotBeEmpty')),
});
const paramSchema = z.object({
key: z.string({ required_error: 'Key is required' }).min(1, "Key cannot be empty"),
value: z.string({ required_error: 'Value is required' }).min(1, "Value cannot be empty"),
});
const getScopeSchema = (t: (key: string) => string) => z.object({
value: z.string({ required_error: t('oauth2.valueIsRequired') }).min(1, t('oauth2.valueCannotBeEmpty')),
});
const scopeSchema = z.object({
value: z.string({ required_error: 'Value is required' }).min(1, "Value cannot be empty"),
});
const extraparamSchema = z.record(z.string()).optional();
const authorizescopeSchema = z.array(z.string()).optional();
import { getOAuth2Schema, type OAuth2FormValues } from './schema'
function convertToExtraParamsSchema(
record: z.infer<typeof extraparamSchema>
): z.infer<typeof paramSchema>[] {
if (!record) {
return [];
}
return Object.entries(record).map(([key, value]) => ({
key,
value,
}));
record: Record<string, string> | undefined
): { key: string; value: string }[] {
if (!record) return []
return Object.entries(record).map(([key, value]) => ({ key, value }))
}
function convertToScopeSchema(authorizeScopes: z.infer<typeof authorizescopeSchema>): z.infer<typeof scopeSchema>[] {
if (!authorizeScopes || authorizeScopes.length === 0) {
return [];
}
return authorizeScopes.map((scope) => ({
value: scope,
}));
function convertToScopeSchema(
scopes: string[] | undefined
): { value: string }[] {
if (!scopes || scopes.length === 0) return []
return scopes.map((scope) => ({ value: scope }))
}
const getOAuth2Schema = (t: (key: string) => string) => z.object({
description: z.string().max(255, { message: t('oauth2.descriptionMustNotExceed255Characters') }).optional(),
client_id: z.string({
required_error: t('oauth2.clientIdIsRequired'),
}).min(1, { message: t('oauth2.clientIdCannotBeEmpty') }),
client_secret: z.string().optional(),
auth_url: z.string({
required_error: t('oauth2.authorizationUrlIsRequired'),
})
.min(1, { message: t('oauth2.authorizationUrlCannotBeEmpty') })
.url({ message: t('oauth2.invalidAuthorizationUrlFormat') }),
token_url: z.string({
required_error: t('oauth2.tokenUrlIsRequired'),
})
.min(1, { message: t('oauth2.tokenUrlCannotBeEmpty') })
.url({ message: t('oauth2.invalidTokenUrlFormat') }),
redirect_uri: z.string({
required_error: t('oauth2.redirectUriIsRequired'),
})
.min(1, { message: t('oauth2.redirectUriCannotBeEmpty') })
.url({ message: t('oauth2.invalidRedirectUriFormat') }),
scopes: z.array(getScopeSchema(t)).optional(),
extra_params: z.array(getParamSchema(t)).optional(),
enabled: z.boolean(),
use_proxy: z.number().optional(),
});
const oauth2Schema = z.object({
description: z.string().max(255, { message: "Description must not exceed 255 characters." }).optional(),
client_id: z.string({
required_error: "Client ID is required",
}).min(1, { message: "Client ID cannot be empty" }),
client_secret: z.string().optional(),
auth_url: z.string({
required_error: "Authorization URL is required",
})
.min(1, { message: "Authorization URL cannot be empty" })
.url({ message: "Invalid Authorization URL format" }),
token_url: z.string({
required_error: "Token URL is required",
})
.min(1, { message: "Token URL cannot be empty" })
.url({ message: "Invalid Token URL format" }),
redirect_uri: z.string({
required_error: "Redirect URI is required",
})
.min(1, { message: "Redirect URI cannot be empty" })
.url({ message: "Invalid Redirect URI format" }),
scopes: z.array(scopeSchema).optional(),
extra_params: z.array(paramSchema).optional(),
enabled: z.boolean(),
use_proxy: z.number().optional(),
});
export type OAuth2Form = z.infer<typeof oauth2Schema>;
interface Props {
currentRow?: OAuth2Entity
open: boolean
@@ -185,7 +91,7 @@ const defaultValues = {
export function ActionDialog({ currentRow, open, onOpenChange }: Props) {
const { t } = useTranslation()
const isEdit = !!currentRow
const form = useForm<OAuth2Form>({
const form = useForm<OAuth2FormValues>({
resolver: zodResolver(getOAuth2Schema(t)),
defaultValues: isEdit
? {
@@ -255,7 +161,7 @@ export function ActionDialog({ currentRow, open, onOpenChange }: Props) {
console.error(error);
}
const onSubmit = (values: OAuth2Form) => {
const onSubmit = (values: OAuth2FormValues) => {
if (!isEdit) {
if (!values.client_secret) {
form.setError('client_secret', {

View File

@@ -0,0 +1,56 @@
import { z } from 'zod'
const paramEntry = (t: (key: string) => string) =>
z.object({
key: z
.string({ required_error: t('oauth2.keyIsRequired') })
.min(1, t('oauth2.keyCannotBeEmpty')),
value: z
.string({ required_error: t('oauth2.valueIsRequired') })
.min(1, t('oauth2.valueCannotBeEmpty')),
})
const scopeEntry = (t: (key: string) => string) =>
z.object({
value: z
.string({ required_error: t('oauth2.valueIsRequired') })
.min(1, t('oauth2.valueCannotBeEmpty')),
})
export const getOAuth2Schema = (t: (key: string) => string) =>
z.object({
description: z
.string()
.max(255, { message: t('oauth2.descriptionMustNotExceed255Characters') })
.optional(),
client_id: z
.string({
required_error: t('oauth2.clientIdIsRequired'),
})
.min(1, { message: t('oauth2.clientIdCannotBeEmpty') }),
client_secret: z.string().optional(),
auth_url: z
.string({
required_error: t('oauth2.authorizationUrlIsRequired'),
})
.min(1, { message: t('oauth2.authorizationUrlCannotBeEmpty') })
.url({ message: t('oauth2.invalidAuthorizationUrlFormat') }),
token_url: z
.string({
required_error: t('oauth2.tokenUrlIsRequired'),
})
.min(1, { message: t('oauth2.tokenUrlCannotBeEmpty') })
.url({ message: t('oauth2.invalidTokenUrlFormat') }),
redirect_uri: z
.string({
required_error: t('oauth2.redirectUriIsRequired'),
})
.min(1, { message: t('oauth2.redirectUriCannotBeEmpty') })
.url({ message: t('oauth2.invalidRedirectUriFormat') }),
scopes: z.array(scopeEntry(t)).optional(),
extra_params: z.array(paramEntry(t)).optional(),
enabled: z.boolean(),
use_proxy: z.number().optional(),
})
export type OAuth2FormValues = z.infer<ReturnType<typeof getOAuth2Schema>>

View File

@@ -19,7 +19,7 @@
import { useEffect, useState } from 'react';
import { useMutation } from '@tanstack/react-query';
import { Loader, Download, Trash2, MessageSquareMore, FileText, FileImage, FileVideo, FileArchive, FileSpreadsheet, FileCode, FileIcon, FileAudio, Upload } from 'lucide-react';
import { Loader, Download, Trash2, MessageSquareMore, FileText, FileImage, FileVideo, FileArchive, FileSpreadsheet, FileCode, FileIcon, FileAudio, Upload, ShieldCheck } from 'lucide-react';
import { Button } from '@/components/ui/button';
import { Separator } from '@/components/ui/separator';
@@ -129,6 +129,12 @@ export function MailMessageView({
const [nestedEmlFile, setNestedEmlFile] = useState<AttachmentInfo | null>(null);
const { getEmailById } = useMinimalAccountList();
const [threadOpen, setThreadOpen] = useState(false);
const [blockRemote, setBlockRemote] = useState(true);
const [hasRemoteContent, setHasRemoteContent] = useState(false);
const toggleBlockRemote = () => {
setBlockRemote((prev) => !prev);
};
const downloadAttachmentMutation = useMutation({
mutationFn: ({ content_hash }: { content_hash: string }) =>
@@ -145,12 +151,13 @@ export function MailMessageView({
});
const loadMessageMutation = useMutation({
mutationFn: () => load_message(envelope.account_id, envelope.id),
mutationFn: () => load_message(envelope.account_id, envelope.id, blockRemote),
onSuccess: (data) => {
setLoading(false);
setContent(getContent(data));
if (data.attachments) setAttachments(data.attachments);
setContentType(data.html ? 'Html' : 'Plain');
setHasRemoteContent(!!data.has_remote_content);
},
onError: (error: any) => {
setLoading(false);
@@ -162,10 +169,14 @@ export function MailMessageView({
},
});
useEffect(() => {
setBlockRemote(true);
}, [envelope.id]);
useEffect(() => {
setLoading(true);
loadMessageMutation.mutate();
}, [envelope.id]);
}, [envelope.id, blockRemote]);
const handleViewNestedEml = (attachment: AttachmentInfo) => {
@@ -384,6 +395,30 @@ export function MailMessageView({
</div>
)}
{showAttachments && <Separator className="mb-2" />}
{hasRemoteContent && (
<div className="flex items-center justify-between bg-muted border px-3 py-1.5 mb-3 text-xs">
<div className="flex items-center gap-1.5 min-w-0">
<ShieldCheck className="h-3.5 w-3.5 text-muted-foreground shrink-0" />
{blockRemote ? (
<span className="text-muted-foreground truncate">
{t('mail.remoteBlocked', 'To protect your privacy, Bichon has blocked remote content in this message.')}
</span>
) : (
<span className="text-muted-foreground truncate">
{t('mail.remoteShown', 'Remote content is now shown.')}
</span>
)}
</div>
<span
className="underline cursor-pointer hover:no-underline text-muted-foreground text-[11px] font-medium shrink-0 ml-2 select-none"
onClick={toggleBlockRemote}
>
{blockRemote
? t('mail.showRemoteContent', 'Show remote content')
: t('mail.blockRemoteAgain', 'Block again')}
</span>
</div>
)}
<div className="flex-1 overflow-auto">
{loading ? (
<div className="flex justify-center items-center py-8">

View File

@@ -148,7 +148,7 @@ export function NestedEmailDialog({ open, onOpenChange, accountId, envelopeId, f
const { data, isLoading } = useQuery({
queryKey: ['nested-message', accountId, envelopeId, content_hash],
queryFn: () => load_nested_message(accountId, envelopeId, content_hash),
queryFn: () => load_nested_message(accountId, envelopeId, content_hash, true),
enabled: open && !!content_hash,
});

View File

@@ -0,0 +1,153 @@
import { describe, it, expect } from 'vitest'
import { profileSchema } from '../schema'
const t = (key: string) => key
describe('Profile Form Schema', () => {
const schema = profileSchema(t)
describe('username field', () => {
it('rejects empty username', () => {
const result = schema.safeParse({
username: '',
email: 'user@example.com',
password: '',
})
expect(result.success).toBe(false)
if (!result.success) {
const errors = result.error.issues.filter(
(i) => i.path[0] === 'username'
)
expect(errors.length).toBeGreaterThan(0)
}
})
it('rejects username shorter than 3 characters', () => {
const result = schema.safeParse({
username: 'ab',
email: 'user@example.com',
password: '',
})
expect(result.success).toBe(false)
})
it('accepts username of exactly 3 characters', () => {
const result = schema.safeParse({
username: 'abc',
email: 'user@example.com',
password: '',
})
expect(result.success).toBe(true)
})
it('rejects username longer than 32 characters', () => {
const result = schema.safeParse({
username: 'a'.repeat(33),
email: 'user@example.com',
password: '',
})
expect(result.success).toBe(false)
})
it('accepts username of exactly 32 characters', () => {
const result = schema.safeParse({
username: 'a'.repeat(32),
email: 'user@example.com',
password: '',
})
expect(result.success).toBe(true)
})
})
describe('email field', () => {
it('rejects empty email', () => {
const result = schema.safeParse({
username: 'validuser',
email: '',
password: '',
})
expect(result.success).toBe(false)
})
it('rejects invalid email format', () => {
const result = schema.safeParse({
username: 'validuser',
email: 'not-an-email',
password: '',
})
expect(result.success).toBe(false)
})
it('rejects email without domain', () => {
const result = schema.safeParse({
username: 'validuser',
email: 'user@',
password: '',
})
expect(result.success).toBe(false)
})
it('accepts valid email', () => {
const result = schema.safeParse({
username: 'validuser',
email: 'user@example.com',
password: '',
})
expect(result.success).toBe(true)
})
})
describe('password field', () => {
it('accepts empty password (keep current)', () => {
const result = schema.safeParse({
username: 'validuser',
email: 'user@example.com',
password: '',
})
expect(result.success).toBe(true)
if (result.success) {
// Empty password should be transformed to undefined
expect(result.data.password).toBeUndefined()
}
})
it('rejects password shorter than 8 characters when provided', () => {
const result = schema.safeParse({
username: 'validuser',
email: 'user@example.com',
password: 'short',
})
expect(result.success).toBe(false)
})
it('accepts password of exactly 8 characters', () => {
const result = schema.safeParse({
username: 'validuser',
email: 'user@example.com',
password: '12345678',
})
expect(result.success).toBe(true)
})
it('rejects password longer than 256 characters', () => {
const result = schema.safeParse({
username: 'validuser',
email: 'user@example.com',
password: 'a'.repeat(257),
})
expect(result.success).toBe(false)
})
it('transforms non-empty password to the string value', () => {
const result = schema.safeParse({
username: 'validuser',
email: 'user@example.com',
password: 'myNewPassword123',
})
expect(result.success).toBe(true)
if (result.success) {
expect(result.data.password).toBe('myNewPassword123')
}
})
})
})

View File

@@ -16,7 +16,6 @@
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
import { z } from 'zod'
import { useForm } from 'react-hook-form'
import { zodResolver } from '@hookform/resolvers/zod'
import { useMutation, useQueryClient } from '@tanstack/react-query'
@@ -42,41 +41,7 @@ import { Badge } from '@/components/ui/badge'
import { FileWithPreview } from '@/hooks/use-file-upload'
import AvatarUpload from './avatar-upload'
import { PermissionsDialog } from '../access/permissions-dialog'
const profileSchema = (t: (key: string) => string) => z.object({
username: z
.string({
required_error: t('settings.profile.validation.username.required'),
})
.min(3, {
message: t('settings.profile.validation.username.min'),
})
.max(32, {
message: t('settings.profile.validation.username.max'),
}),
email: z
.string({
required_error: t('settings.profile.validation.email.required'),
})
.email({
message: t('settings.profile.validation.email.invalid'),
}),
password: z
.string()
.min(8, {
message: t('settings.profile.validation.password.min'),
})
.max(256, {
message: t('settings.profile.validation.password.max'),
})
.or(z.literal(''))
.optional()
.transform((v) => (v ? v : undefined)),
})
export type ProfileFormValues = z.infer<ReturnType<typeof profileSchema>>
import { profileSchema, type ProfileFormValues } from './schema'
function fileToBase64(file: File): Promise<string> {
return new Promise((resolve, reject) => {

View File

@@ -0,0 +1,37 @@
import { z } from 'zod'
export const profileSchema = (t: (key: string) => string) =>
z.object({
username: z
.string({
required_error: t('settings.profile.validation.username.required'),
})
.min(3, {
message: t('settings.profile.validation.username.min'),
})
.max(32, {
message: t('settings.profile.validation.username.max'),
}),
email: z
.string({
required_error: t('settings.profile.validation.email.required'),
})
.email({
message: t('settings.profile.validation.email.invalid'),
}),
password: z
.string()
.min(8, {
message: t('settings.profile.validation.password.min'),
})
.max(256, {
message: t('settings.profile.validation.password.max'),
})
.or(z.literal(''))
.optional()
.transform((v) => (v ? v : undefined)),
})
export type ProfileFormValues = z.infer<ReturnType<typeof profileSchema>>

View File

@@ -0,0 +1,173 @@
import { describe, it, expect } from 'vitest'
import { proxyFormSchema } from '../schema'
describe('Proxy Form Schema', () => {
describe('url field - basic validation', () => {
it('rejects empty URL', () => {
const result = proxyFormSchema.safeParse({ url: '' })
expect(result.success).toBe(false)
})
it('accepts valid socks5 URL', () => {
const result = proxyFormSchema.safeParse({
url: 'socks5://127.0.0.1:1080',
})
expect(result.success).toBe(true)
})
it('accepts valid http URL', () => {
const result = proxyFormSchema.safeParse({
url: 'http://proxy.example.com:8080',
})
expect(result.success).toBe(true)
})
})
describe('url field - protocol validation', () => {
it('rejects https protocol', () => {
const result = proxyFormSchema.safeParse({
url: 'https://proxy.example.com:443',
})
expect(result.success).toBe(false)
if (!result.success) {
expect(
result.error.issues.some((i) =>
i.message?.includes('http:// or socks5://')
)
).toBe(true)
}
})
it('rejects ftp protocol', () => {
const result = proxyFormSchema.safeParse({
url: 'ftp://files.example.com',
})
expect(result.success).toBe(false)
})
it('rejects URL without protocol', () => {
const result = proxyFormSchema.safeParse({
url: '127.0.0.1:1080',
})
expect(result.success).toBe(false)
if (!result.success) {
expect(
result.error.issues.some((i) =>
i.message?.includes('Invalid URL format')
)
).toBe(true)
}
})
})
describe('url field - port validation', () => {
it('rejects port 0', () => {
const result = proxyFormSchema.safeParse({
url: 'socks5://127.0.0.1:0',
})
expect(result.success).toBe(false)
})
it('rejects port > 65535', () => {
const result = proxyFormSchema.safeParse({
url: 'socks5://127.0.0.1:99999',
})
expect(result.success).toBe(false)
})
it('accepts port 65535', () => {
const result = proxyFormSchema.safeParse({
url: 'socks5://127.0.0.1:65535',
})
expect(result.success).toBe(true)
})
it('accepts port 1', () => {
const result = proxyFormSchema.safeParse({
url: 'socks5://127.0.0.1:1',
})
expect(result.success).toBe(true)
})
it('defaults to port 1080 when no port specified', () => {
const result = proxyFormSchema.safeParse({
url: 'socks5://127.0.0.1',
})
expect(result.success).toBe(true)
})
})
describe('url field - hostname validation', () => {
it('accepts IP address hostname', () => {
const result = proxyFormSchema.safeParse({
url: 'socks5://192.168.1.1:1080',
})
expect(result.success).toBe(true)
})
it('accepts domain hostname', () => {
const result = proxyFormSchema.safeParse({
url: 'socks5://proxy.internal:1080',
})
expect(result.success).toBe(true)
})
it('rejects hostname with invalid characters', () => {
const result = proxyFormSchema.safeParse({
url: 'socks5://proxy_host:1080',
})
expect(result.success).toBe(false)
if (!result.success) {
expect(
result.error.issues.some((i) =>
i.message?.includes('Hostname contains invalid characters')
)
).toBe(true)
}
})
})
describe('url field - auth validation', () => {
it('rejects username without password', () => {
const result = proxyFormSchema.safeParse({
url: 'socks5://user@127.0.0.1:1080',
})
expect(result.success).toBe(false)
if (!result.success) {
expect(
result.error.issues.some((i) =>
i.message?.includes('Password cannot be empty')
)
).toBe(true)
}
})
it('rejects short password when username provided', () => {
const result = proxyFormSchema.safeParse({
url: 'socks5://user:short@127.0.0.1:1080',
})
expect(result.success).toBe(false)
if (!result.success) {
expect(
result.error.issues.some((i) =>
i.message?.includes('Password must be at least 8')
)
).toBe(true)
}
})
it('accepts valid auth credentials', () => {
const result = proxyFormSchema.safeParse({
url: 'socks5://user:password123@127.0.0.1:1080',
})
expect(result.success).toBe(true)
})
it('accepts URL without auth (no credentials)', () => {
const result = proxyFormSchema.safeParse({
url: 'socks5://127.0.0.1:1080',
})
expect(result.success).toBe(true)
})
})
})

View File

@@ -17,7 +17,6 @@
// along with this program. If not, see <http://www.gnu.org/licenses/>.
import { z } from 'zod'
import { useForm } from 'react-hook-form'
import { zodResolver } from '@hookform/resolvers/zod'
import { toast } from '@/hooks/use-toast'
@@ -47,75 +46,7 @@ import { Loader2 } from 'lucide-react'
import { add_proxy, update_proxy } from '@/api/system/api'
import { useTranslation } from 'react-i18next'
import { Proxy } from '@/api/system/api'
const proxyFormSchema = z.object({
url: z.string()
.min(1, "Proxy address cannot be empty")
.superRefine((value, ctx) => {
if (value.length === 0) {
return;
}
let url: URL;
try {
url = new URL(value);
} catch (e) {
ctx.addIssue({
code: z.ZodIssueCode.custom,
message: "Invalid URL format",
path: [],
});
return;
}
if (url.protocol !== 'socks5:' && url.protocol !== 'http:') {
ctx.addIssue({
code: z.ZodIssueCode.custom,
message: "URL must start with http:// or socks5://",
path: [],
});
}
if (!/^[a-zA-Z0-9\-\.]+$/.test(url.hostname)) {
ctx.addIssue({
code: z.ZodIssueCode.custom,
message: "Hostname contains invalid characters",
path: [],
});
}
const port = parseInt(url.port || '1080');
if (port <= 0 || port > 65535) {
ctx.addIssue({
code: z.ZodIssueCode.custom,
message: "Port must be between 1-65535",
path: [],
});
}
if (url.username && !url.password) {
ctx.addIssue({
code: z.ZodIssueCode.custom,
message: "Password cannot be empty when username is provided",
path: [],
});
} else if (url.password && url.password.length < 8) {
ctx.addIssue({
code: z.ZodIssueCode.custom,
message: "Password must be at least 8 characters",
path: [],
});
}
})
});
export type ProxyForm = z.infer<typeof proxyFormSchema>;
import { proxyFormSchema, type ProxyFormValues } from './schema'
interface Props {
@@ -140,7 +71,7 @@ export function ProxyActionDialog({ currentRow, open, onOpenChange }: Props) {
const { t } = useTranslation()
const isEdit = !!currentRow
const queryClient = useQueryClient();
const form = useForm<ProxyForm>({
const form = useForm<ProxyFormValues>({
resolver: zodResolver(proxyFormSchema),
defaultValues: isEdit
? mapCurrentRowToFormValues(currentRow)
@@ -186,7 +117,7 @@ export function ProxyActionDialog({ currentRow, open, onOpenChange }: Props) {
}
const onSubmit = (values: ProxyForm) => {
const onSubmit = (values: ProxyFormValues) => {
const url = values.url;
if (isEdit) {
updateMutation.mutate(url);

Some files were not shown because too many files have changed in this diff Show More