Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 29 September, 2025 10:53

Incident End: 29 September, 2025 11:42

Duration: 49 minutes

GWS Affected Services and Features:

Workspace Gemini and NotebookLM

Regions/Zones: Global

Description: Various AI products experienced elevated 500 (internal) errors due to the rollout of a faulty change to production. Google will complete a full IR in the following days that will provide a full root cause.

Customer Impact:

Customers affected by this issue may have observed up to a 100% of requests failing with '500' errors during the impacted time, between 10:53 and 11:42 US/Pacific on 29 September 2025.

Additional details:

A faulty change was rolled out to the control plane of the canonical Machine Learning (ML) serving stack in one cluster. Traffic routers that received their configuration from the faulty cluster assumed that some models were deleted and returned errors to the clients. Due to the error, the faulty control plane cluster had a significantly lower load, which resulted in more routers getting their configuration from the faulty cluster. This exacerbated the impact of the change.

The bad change was rolled back and all the errors returned by ML serving were reduced to pre-fault levels, mitigating customer impact.

Summary:

WorkSpace NotebookLM customers may experience elevated errors for online users

Description:

The issue with Workspace NotebookLM has been mitigated as of Monday, 2025-09-29 11:49 PDT.

Based on preliminary investigation, the root cause of the issue was due to a recent change, which has been rolled back.

Customer Symptoms:

Customers are experiencing elevated "INTERNAL_ERROR" with error code 500s

Summary:

NotebookLM customers may experience elevated errors for online users

Description:

Our engineering team has identified the cause of the issue and a mitigation has been put in place. We are currently monitoring and believe we are seeing full recovery.

Customer Symptoms:

Customers are experiencing elevated "INTERNAL_ERROR" with error code 500s