One faulty GPU worker failed to process requests

Partial outageResolvedVirtual Try-OnStarted March 28, 2025 at 5:01 PM UTCLasted 8 hours and 9 minutes

Resolved March 29, 2025 at 1:10 AM UTC
- One GPU worker went into a bad state, it was restarted and returned to normal operation
- A mechanism to detect and auto-restart such states was developed and deployed.
Investigating March 28, 2025 at 5:01 PM UTC
Some of the requests to the nightly endpoint (experimental in app) are returning errors

Updates