regenerate system prompts per stage in conversation replay

[zetxqx]

Fixes: #479

This PR updates the ConversationReplayDataGenerator to automatically regenerate system prompts at the start of every new benchmark stage.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: zetxqx
Once this PR has been reviewed and has the lgtm label, please assign arangogutierrez for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[achandrasekar]

@alonh @LoganVegnaSHOP @Navjot10 Any thoughts on the behavior here?

[alonh]

This looks good overall — the cross-stage regeneration approach makes sense.

The seed derivation in load_lazy_data uses f"{self.cr_config.seed}stage{data.stage_idx}" without any per-conversation component. When dynamic_system_prompt_len is configured, each conversation originally gets a unique suffix at init time. But on stage transitions, all conversations derive the same RNG from the same seed string. Wouldn't that cause all conversations to collapse to the same prompt, losing the per-conversation uniqueness?

Nit: the new field is called stage_idx but the rest of the codebase uses stage_id — would be nice to keep consistent.

[zetxqx]

This looks good overall — the cross-stage regeneration approach makes sense.

The seed derivation in load_lazy_data uses f"{self.cr_config.seed}stage{data.stage_idx}" without any per-conversation component. When dynamic_system_prompt_len is configured, each conversation originally gets a unique suffix at init time. But on stage transitions, all conversations derive the same RNG from the same seed string. Wouldn't that cause all conversations to collapse to the same prompt, losing the per-conversation uniqueness?

Nit: the new field is called stage_idx but the rest of the codebase uses stage_id — would be nice to keep consistent.

Good point

1. updated the logic, now the implementation make sure only the shared prompt will be regenerated for different stages. the dynamic prompt will just be reused. So the uniqueness is protected now.
2. change stage_idx to stage_id