Skip to content

feat(universal_io): MultiUniversalRead trait and VecMultiUniversalRead implementation#8255

Open
generall wants to merge 7 commits intotracker-universal-iofrom
multi-file-universal-io
Open

feat(universal_io): MultiUniversalRead trait and VecMultiUniversalRead implementation#8255
generall wants to merge 7 commits intotracker-universal-iofrom
multi-file-universal-io

Conversation

@generall
Copy link
Member

@generall generall commented Mar 1, 2026

Adds the multi-source universal I/O interface and minimal mmap-based implementation as planned in docs/design/multi-file-universal-io-plan.md.

Changes

  • SourceId type and MultiUniversalRead<T> trait with read_batch_multi, optional source_len, populate, clear_ram_cache
  • UniversalIoError::InvalidSourceId variant
  • VecMultiUniversalRead<T, S>: holds Vec<S: UniversalRead<T>>, implements MultiUniversalRead, supports attach(source) -> SourceId
  • Unit test vec_multi_universal_read_batch_and_attach using MmapUniversal
  • segment: handle InvalidSourceId in From<UniversalIoError> for OperationError

No gridstore or other call-site changes in this PR; focus is interface + minimal impl only.

Made with Cursor

Cursor Agent and others added 7 commits March 1, 2026 11:59
…lRead impl

- Add SourceId, MultiUniversalRead<T> trait with read_batch_multi, source_len,
  populate, clear_ram_cache (latter two with default no-op).
- Add UniversalIoError::InvalidSourceId for invalid source id in batch reads.
- Add VecMultiUniversalRead<T, S>: minimal implementation over Vec<S: UniversalRead<T>>
  with attach(source) -> SourceId for adding sources at runtime.
- Add test vec_multi_universal_read_batch_and_attach using MmapUniversal.
- Handle InvalidSourceId in segment OperationError From<UniversalIoError>.

Implements the interface and minimal mmap-based implementation from
docs/design/multi-file-universal-io-plan.md (multi-source universal I/O).

Made-with: Cursor
… drop plan from PR

- Add universal_io/multi_universal_read.rs with SourceId, MultiUniversalRead,
  VecMultiUniversalRead and test; re-export from mod.rs.
- Remove docs/design/multi-file-universal-io-plan.md from the branch.

Made-with: Cursor
…tach, len, is_empty to trait

- MultiUniversalRead: remove default impls for populate() and clear_ram_cache();
  they are now required.
- Add to trait: new(), len(), is_empty() (default), attach() (default Err).
- Introduce associated type Source for attach; add AttachUnsupported<T>
  placeholder for impls that do not support dynamic attach.
- VecMultiUniversalRead: type Source = S; implement all trait methods.
- Re-export AttachUnsupported from universal_io.

Made-with: Cursor
…all impls

- Drop AttachUnsupported placeholder type and its UniversalRead/Send impls.
- Make attach() a required method on MultiUniversalRead (no default).
- Doc: all implementations must support attaching sources dynamically.
- Remove AttachUnsupported from re-exports.

Made-with: Cursor
…, add MultiUniversalWrite

MultiUniversalRead:
- Remove type Source; attach(path, options) opens by path and returns SourceId.
- new(options: OpenOptions) for creating an empty multi-source view.
- Move VecMultiUniversalRead to vec_multi_universal_read.rs.

MultiUniversalWrite (new):
- Trait: new(options), len(), is_empty(), attach(path, options),
  write_batch_multi((SourceId, offset, data)...), source_len, flusher(),
  populate(), clear_ram_cache().
- VecMultiUniversalWrite in vec_multi_universal_write.rs; flusher()
  runs all source flushers.

Re-export MultiUniversalWrite, VecMultiUniversalWrite from universal_io.

Made-with: Cursor
Make MultiUniversalWrite<T>: MultiUniversalRead<T> like UniversalWrite
extends UniversalRead. Remove duplicated methods (new, len, is_empty,
attach, source_len, populate, clear_ram_cache) from the write trait;
keep only write_batch_multi and flusher. VecMultiUniversalWrite now
impl MultiUniversalRead and MultiUniversalWrite separately.

Made-with: Cursor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant