Google Cloud Storage RetryHelper Failure

I have numerous applications which use Google Cloud Storage buckets to save files, maintain backups, etc. Most of these applications are Java apps using the official Google Cloud Storage library.

Yesterday one of these applications had difficulty accessing Cloud Storage and printed out the following error report:

The GCS library attempted to access Cloud Storage a total of 6 times; this image shows the last failure and the final “giving-up” error.

Here’s a copy of the error in text form:

com.google.appengine.tools.cloudstorage.RetryHelper doRetry: 
    RetryHelper(4.114 s, 6 attempts, com.google.appengine.tools.cloudstorage.GcsServiceImpl): 
    Attempt 6 failed, sleeping for 3140 ms: java.io.IOException: Response code 503, 
    retryable: Request: POST https://storage.googleapis.com/[bucket_url]/[file_name]
x-goog-resumable: start
x-goog-api-version: 2
Content-Type: image/jpg

no content

Response: 503 with 19 bytes of content
Content-Length: 19
Date: Thu, 01 Aug 2013 [time]
Server: HTTP Upload Server Built on Jul 21 2013 19:20:38 (1374459638)
Content-Type: text/html; charset=UTF-8
X-Google-Cache-Control: remote-fetch
Via: HTTP/1.1 GWA
Service Unavailable

And after 6 failures attempting to access the Google Cloud Storage bucket, the library reports an error and stops:

.<stderr>: IO Exception: RetryHelper(4.115 s, 6 attempts, 
    com.google.appengine.tools.cloudstorage.GcsServiceImpl): 
    Too many failures, giving up

Fortunately this seemed to be only a transient problem; the next upload to GCS succeeded.