Occasionally applications – even the best behaved applications – will get the error “Deadline exceeded while waiting for HTTP response from URL.”
Generally, this means that the web service you’re trying to connect to is down or slow. If the service is down, then you can continuously retry your URL fetches by queuing them up within a task.
If the web service is slow, then you have an alternative: setting the read and connect timeouts to a longer timeout point. By default, App Engine expects that an URL fetch will take – at most – 5 seconds. That’s 5 seconds to connect to the web service (resolve DNS and so forth), send the request data, allow the web service to process the request, and finally retrieve any response sent back. For the vast majority of applications, that’s more than enough. The popular web APIs such as Twitter, Facebook, Google, etc all process and return requests in much less than 5 seconds.
However, a slow or malfunctioning web service may take longer than 5 seconds to respond to a query. If your app is downloading a large amount of data (more than a few MB) you may also go past this limit. To tell App Engine to wait for a longer period of time, use this code (url_connection represents a HttpURLConnection object):
url_connection.setReadTimeout(milliseconds_to_wait_for_read);
url_connection.setConnectTimeout(milliseconds_to_wait_for_connect);
Remember that the time to wait is denoted in milliseconds, so do the appropriate conversions (for example, if you wanted the connection to wait 30 seconds, you would put 30000 milliseconds).