Add timeout to standard pilot fetch

[peter941221]

Summary

Fix StandardPilot.JobGetAvailable so a stalled fetch does not hang a producer indefinitely.

Problem

producer.dispatchWork intentionally strips cancellation from the work context before fetching jobs so an in-flight fetch is allowed to complete during shutdown:

• producer.go:744-766

That is reasonable, but StandardPilot.JobGetAvailable forwarded directly to exec.JobGetAvailable with no timeout at all:

• rivershared/riverpilot/standard_pilot.go:18-22

This meant a stalled driver call could block a standard-pilot producer forever. The pro pilot already applies per-attempt fetch timeouts, so the standard pilot was the outlier.

Change

Add a 10-second timeout inside StandardPilot.JobGetAvailable before calling the driver.

This keeps the existing shutdown semantics intact:

• fetches still ignore parent cancellation from dispatchWork
• but they are now bounded, so a wedged DB call eventually returns instead of freezing the producer forever

The timeout is local to the standard pilot so there is no driver SQL change and no producer state-machine change.

Testing

• added rivershared/riverpilot/standard_pilot_test.go
• covered MaxToLock <= 0 no-op behavior
• covered a hung JobGetAvailable call timing out with context.DeadlineExceeded
• covered parent cancellation still winning when the incoming context is already canceled

Verification

Locally verified with:

• GOPROXY=https://goproxy.cn,direct GOSUMDB=off go test ./rivershared/riverpilot -count=1

Closes #1026.

[brandur]

Thanks!

@bgentry Any strong opinions on how you want to handle this one? Another option is to just put the timeout in producer.go in dispatchWork:

func (p *producer) dispatchWork(workCtx context.Context, count int, fetchResultCh chan<- producerFetchResult) {
	// This intentionally removes any deadlines or cancellation from the parent
	// context because we don't want it to get cancelled if the producer is asked
	// to shut down. In that situation, we want to finish fetching any jobs we are
	// in the midst of fetching, work them, and then stop. Otherwise we'd have a
	// risk of shutting down when we had already fetched jobs in the database,
	// leaving those jobs stranded. We'd then potentially have to release them
	// back to the queue.
	ctx := context.WithoutCancel(workCtx)

	// Maximum size of the `attempted_by` array on each job row. This maximum is
	// rarely hit, but exists to protect against degenerate cases.
	const maxAttemptedBy = 100

	jobs, err := p.pilot.JobGetAvailable(ctx, p.exec, p.state, &riverdriver.JobGetAvailableParams{
		ClientID:       p.config.ClientID,
		MaxAttemptedBy: maxAttemptedBy,
		MaxToLock:      count,
		Now:            p.Time.NowOrNil(),
		Queue:          p.config.Queue,
		ProducerID:     p.id.Load(),
		Schema:         p.config.Schema,
	})
	if err != nil {
		fetchResultCh <- producerFetchResult{err: err}
		return
	}

	fetchResultCh <- producerFetchResult{jobs: jobs}
}

That might be better in the way that not every pilot needs to remember to bring its own context cancellations. That said, maybe in this case we might want a longer cancellation for the pro pilot so it'd make sense to break up the two.

[bgentry]

Another more robust option is to do fetches within a transaction that sets a statement_timeout at the DB level. This is far more robust because it's not prone to accidentally stranding jobs, although the extra round trips may impact throughput. As long as the Go side waits for db_timeout + margin it'd be super unlikely to strand jobs. We basically do this on the Pro side already due to extra logic running within fetches. Thoughts @brandur?

[brandur]

@bgentry WFM. It'd probably be worth re-running the benchmark on the branch to verify no major degradation in performance, but given the fetch queries are relatively few compared to everything else, hopefully there wouldn't be.

You were previously against use of statement_timeout with a context cancellation though weren't you? You're saying there'd still be both here right?

[peter941221]

@bgentry @brandur
I put it on StandardPilot first because I wasn’t sure yet whether ProPilot should inherit the same timeout semantics.
If both pilots are really meant to have the same JobGetAvailable cancellation policy, then I agree dispatchWork is probably the cleaner home for it since it makes the timeout automatic instead of something each pilot has to remember.
My hesitation was exactly the case you called out: if ProPilot wants a meaningfully longer timeout, pushing it up immediately may be the wrong abstraction. I can take a closer look at the pro path and either move this up to dispatchWork or keep it split if the semantics are intentionally different.

[brandur]

@peter941221 Yep, makes sense. Thanks.

[peter941221]

1. I’m going to keep this PR scoped to the StandardPilot timeout.

2. I don’t want to move the timeout up to dispatchWork or switch this branch to a transaction + statement_timeout design until I’ve checked whether the Pro fetch path is meant to keep different timeout semantics.

3. If we do want one policy across both pilots, I’d rather handle that as a separate follow-up with a benchmark around the extra round trips.

[brandur]

Thx. Gonna pull this in and make a few tweaks on top.