[WIP] Add support for ShareGpt4Video Dataset#494
Draft
Bslabe123 wants to merge 3 commits into
Draft
Conversation
k8s-ci-robot
added
the
do-not-merge/work-in-progress
Contributor
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: Bslabe123 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
k8s-ci-robot
added
needs-rebase
Bslabe123
force-pushed
the
sharegpt4video-dataset
branch
from
afb7694 to
11fcff9
Compare
k8s-ci-robot
added
cncf-cla: yes
Bslabe123
force-pushed
the
sharegpt4video-dataset
branch
7 times, most recently
from
359d031 to
c8c7ee7
Compare
k8s-ci-robot
added
needs-rebase
Bslabe123
force-pushed
the
sharegpt4video-dataset
branch
from
c8c7ee7 to
b87b4e2
Compare
k8s-ci-robot
removed
the
needs-rebase
Bslabe123
force-pushed
the
sharegpt4video-dataset
branch
from
b87b4e2 to
9f4a204
Compare
ShareGPT4Video is a gated HuggingFace video-captioning corpus (https://huggingface.co/datasets/ShareGPT4Video/ShareGPT4Video, ~1.5 TB total, 15-21 GB per source zip). The loader streams captions + keyframe indices and pulls source MP4 zips into a local cache directory via a background thread, so the hot request path never blocks on the network. Init blocks only until the first zip has been extracted so there is always at least one usable video when load generation begins; the on-disk pool grows over time as background downloads complete. Each request decodes the configured keyframes via PyAV, resizes to ``target_resolution``, re-encodes as PNG or JPEG, and emits the frames as ``PreEncodedFramesVideoSpec`` (one ``image_url`` block per frame at a single insertion point) — compatible with VLMs that don't accept ``video_url`` natively. MP4 representation is explicitly rejected for v1. ``GatedHFDataset`` mixin in ``inference_perf/datagen/gated_hf_dataset.py`` resolves the HF access token (config field → ``HF_TOKEN`` → ``HUGGING_FACE_HUB_TOKEN``) and wraps ``load_dataset`` so other gated loaders (MMMU) can reuse the pattern. This commit also folds in two adjacent changes that the loader and its parent issue rely on: - ``config.py``: ``SessionReplayConfig`` base extracted from ``OTelTraceReplayConfig``, picking up ``inject_random_session_id``, ``duplicate_sessions_target``, ``max_wait_ms``, ``hf_dataset_path`` on OTel and ``max_model_len`` on ``ConversationReplayConfig``. - ``chat.py``: tool-call wire support (``tool_calls`` / ``tool_call_id`` on ``ChatMessage``, ``tool_definitions`` → ``tools`` on the request payload, JSON-Schema normalisation for vLLM/xgrammar compatibility). Addresses kubernetes-sigs#475.
- ``tests/optional/manual/sharegpt4video/cases/{png,jpeg}_frames/``
— Kubernetes manual e2e cases. Each case deploys a Qwen3-VL vLLM
server and runs an inference-perf Job that hits ShareGPT4Video
through the gated HuggingFace dataset. ``hf-secret`` is expected
in the ``default`` namespace; the orchestrator copies it into the
per-case namespace before applying the workload.
- ``tests/optional/manual/sharegpt4video/e2e_test.sh`` — orchestrator
that iterates every case folder, runs each in its own namespace,
extracts the lifecycle reports, and verifies non-zero successes /
zero failures.
- ``e2e/tests/test_multimodal_sim.py::test_sharegpt4video_against_sim``
— Python e2e test against llm-d-inference-sim. Skipped by default;
requires ``SHAREGPT4VIDEO_CACHE_DIR`` to be pre-populated (the
loader does not auto-download in this test) and ``HF_TOKEN``.
Pinned to small resolution / 4 frames so the run stays fast.
Bslabe123
force-pushed
the
sharegpt4video-dataset
branch
from
9f4a204 to
276376c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DO NOT REVIEW