Pumpkin

Mili 8a29f1e95f Implement Bulk Loading/Writing for Chunks (#535 )

* feat: add linear chunk format with zstd lib dependency

* fixing a typo

* refactor: rename file handling methods for clarity and more lint fixes

* refactor: improve clarity on validation methods

* fix: invert version check in LinearFile header validation

* fix: remove leading dot from file path formatting in LinearChunkFormat

* refactor: improve error handling and method naming in LinearFile operations

* fix: ensure file is flushed after writing in LinearFile operations

* fix: lints and warnings

* fix: more lints

* refactor: streamline LinearFile structure and improve header management

* refactor: change chunks_headers to use Box for improved memory management and use exception to know if file exist

* refactor: update path parameter types to PathBuf for consistency and improved usability

* refactor: update path parameter types to Path avoiding uknown changes and adjust file truncation logic

* fix: avoid using the same path for different tests

* fix: fix some bytes sizes for the header based on the spec

* refactor: enable file truncation when opening for writing and simplify save method

* fix: removing IoSlices for windows issues.

* fix: clippy lints

* refactor: improve incompatible files cheking, some documentation and variable nameing

* feat: add file_format to ChunkConfig and implement format handling in Level

* feat: add error logging for invalid file headers and data size checks in LinearFile and tmp filenames on write

* feat: implement file locking mechanism for concurrent chunk read/write operations

* fix: bug at desserializing the chunks headers.

* feat: (Bulk API) refactor chunk reading and writing to handle multiple chunks at once

* refactor: streamline chunk reading and writing methods, improve chunk management logic

* feat: enhance chunk reading and writing to support bulk and improve concurrency

* feat: add file locking during chunk writing to ensure safe concurrent access

* fix: resolve some issues with the merge

* fix: update chunk header size calculation in linear chunk parsing

* refactor: optimize chunk processing and improve concurrency handling

* feat: implement chunk I/O management with file locking for safe concurrent access

* refactor: simplify public load method and fix clippy issues

* feature: add file cache for files with ongoing IO ops

* refactor: update LoadedData enum variants for consistency and clarity and include docs.

* refactor: restore compression enum and replace ChunkSerializingError with ChunkWritingError for improved error handling

* refactor: enhance chunk cache cleaning logic and improve logging during chunk fetching

* refactor: optimize chunk data handling with parallel processing and improve chunk fetching logic

* fix: lints and logging

* fix: correct typo and import format

* refactor: anvil code organization and llinear chunk reading code.

* fix logging issue and refactor some locks bindings

* refactor: reduce chunk cloning, ensure drop bindings/guards

* refactor: reduce some threading overhead when not needed

* fix: clippy lints

* refactor: simplify chunk data processing in Anvil and Linear serializers

* fix: clippy lints rust 1.84.1 to 1.85.0

* add file drop guard and rework some logic

* refactor: improve AnvilChunkFile and AnvilChunkData serialization logic and docs

* fix: typos

* feature: initial async implementation of  Bulk API

* fix: chunk writing with async read

* refactor: improve async handling and use asyn IO for reading files

* fix: sorry i delet a mod xd

* refactor: enhance chunk management with async locks and OneCells

* use async methods where avaliable and stream chunks instead of collecting them

* refactor: streamline chunk management by removing unnecessary mutex and optimizing cache cleanup

* Split data

* Move format related stuff from chunk

* refactor: decoupling Serializer from ChunkIO and reduce CPU usage by 25%-50% when flying as spectator at max speed

* refactor: imrpove CPU usage marking Heavy CPU tasks with tokio block_in_place and improve Docs/Comments

* refactor: improve linear format performance and imrpove some localize tokio::block_in_place for compression and uncompression in both formats, also for serialization and deserialization

* Remove parking_lot dependency from Cargo.toml

* feat: add futures crate for improved async handling and performance optimizations

* refactor: change some comments about blocking (not needed with futures::future::join_all) and zstd compression concerns

* refactor: remove unnecessary blocking in AnvilChunkFile methods for improved performance

* refactor: change receive_chunks parameter from slice to vector for improved flexibility

* fix: add timestamp tracking for chunk updates

* refactor: remove async_trait usage from chunk serializers

* fix: handle potential errors in chunk decompression and parsing

* fix: prevent parsing of empty chunks by updating condition for sector offset and count

* fix: improve dashmap reference management

* dont log if chunk does not exist and tweak help functions

* refactor: reduce CPU and Memory usage while improving Chunk Generation performance and reducing code complexity.

* fix: improve logging for file operations and ensure proper lock management during closure

* fix extra log

* fix: chunk generation threading issue (not using rayon spawn) and a emory leak in the level cache cleanup

* fix: handle expected chunk reading errors without logging as failures

* feat: add intial benchmark for chunk I/O performance and update criterion dependencies

* remove panic for bad chunk return

* refactor: comment out benchmark code for test without BulkAPI PR

* Refactor benchmarking code and remove unused chunk_io benchmark

* fix: remove dummy benchmark file and using paths/files instead of temp dirs

* refactor: update benchmark to re-instanciate the level on each iteration avoiding caching

* fix: clippy lints and add comments

* fix: correct chunk count logging in benchmarking tests

* refactor: replace hardcoded channel size with a constant, print seed used, add one more size to bench.

* use bytes for anvil and stream deserialization

* re-add streaming w/ capacities

* tweak rayon threads

* fix deadlock, max concurrency, box chunks early

* add parallel benchmarking for chunk read/write operations

* work on chunk batching

* add chunk batching

* fix chunk batching

* add comment to lru cache

* fix test

* fix spelling

---------

Co-authored-by: Mili <santiagoogle@hotmail.com>
Co-authored-by: kralverde <github@email.kralverde.dev>
Co-authored-by: suprohub <suprohub@gmail.com>

2025-03-06 08:26:52 -10:00