MediumServingPython 3
Batch Inference Scheduler
Batch requests by size or oldest-wait deadline while preserving FIFO order.
35m2 sample tests3 hidden tests
Group request IDs into deterministic inference batches.
Requirements
- Define
BatchScheduler(max_batch_size, max_wait_ms, now_ms). add(request_id)appends a pending request.- If the batch reaches
max_batch_size, return the emitted batch list. - Otherwise return
None. flush_due()returns a batch if the oldest pending request has waited at leastmax_wait_ms.flush_all()returns all pending requests as one batch.- Empty flushes return
None. - Preserve FIFO order.
Example
python
1clock = {"t": 0}
2s = BatchScheduler(2, 50, now_ms=lambda: clock["t"])
3assert s.add("a") is None
4assert s.add("b") == ["a", "b"]Constraints
- Use injected time in milliseconds.
- Do not reorder requests.
- A full batch emits immediately.
Editor
Results
Run sample tests or submit all tests.