Skip to content

Conversation

@aldenh-viam
Copy link
Contributor

See ticket for investigation findings.

@viambot viambot added the safe to test This pull request is marked safe to test from a trusted zone label Nov 5, 2025
func (svc *webService) closeStreamServer() {
svc.streamServerMu.Lock()
defer svc.streamServerMu.Unlock()
if err := svc.streamServer.Close(); err != nil {
Copy link
Contributor Author

@aldenh-viam aldenh-viam Nov 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: svc.streamServer.Close() acquires svc.streamServer.mu.

If I move the svc.streamServerMu.Lock()/Unlock() to line 37 instead and only guard the nil assignment, there will be another race on:
Read @ robot/web/stream.(*Server).refreshVideoSources(): server.robot
Write @ robot/web/stream.(*Server).Close(): server.activeBackgroundWorkers.Wait()

@dgottlieb
Copy link
Member

Seems reasonable to me. I defer to @cheukt who definitely knows the calls/lifetimes of these methods/objects better than myself.

Copy link
Member

@benjirewis benjirewis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was hesitant to throw another mutex in here, but I do see how the one you've added does something that's different than streamServer.mu. Thanks! Just a thought on naming + documentation.

robot/web/web.go Outdated
isRunning bool
webWorkers sync.WaitGroup
modWorkers sync.WaitGroup
streamServerMu sync.Mutex
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Could we call this streamServerInitCloseMu or something that helps differentiate from streamServer.mu? And maybe leave a comment above this mutex saying that it synchronizes the web service's concurrent access to the streamServer object?

@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 6, 2025
defer svc.streamServerInitCloseMu.Unlock()
// May be nil if this is being run after closeStreamServer.
if svc.streamServer != nil {
return svc.streamServer.AddNewStreams(svc.cancelCtx)
Copy link
Member

@cheukt cheukt Nov 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should streamServer ever be nil if the web server is up? feels like that'd be bad since that would be we won't be able to get video streams

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a test would be to get the web server in such a state that the streamServer is nil during a reconfigure and see if we can still access webrtc video streams (UI Live camera streams, for example)

@dgottlieb dgottlieb removed their request for review November 12, 2025 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

safe to test This pull request is marked safe to test from a trusted zone

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants