URLFetch User Agent

When an application makes an URLFetch request, App Engine adds the following text as the User-Agentheader:

AppEngine-Google; (+http://code.google.com/appengine; 

Even if the application sets a custom user agent header, App Engine will append the above text to the header.

This can be annoying because there are some servers and services that rate limit based on the user agent. If there is a human reviewing the request logs, it can be confusing to see a stream of largely-identical user agent strings.

It’s good practice to set a descriptive user agent for all URL fetches. It’s even better if you can write your user agent with App Engine’s required text in mind. For instance, consider writing user agent headers like this one: App Engine Example Crawler hosted by. When App Engine appends its required text to the end of this, the receiving server will see an user agent of:

App Engine Example Crawler hosted by
AppEngine-Google; (+http://code.google.com/appengine; 

This user agent header looks cleaner, neater, and is easier for a human to understand.

Here is the above in code form:

String user_agent = "App Engine Example Crawler";
user_agent += " hosted by ";//After this, GAE will append the identifier.
connection.setRequestProperty("User-Agent", user_agent);

The connection variable represents a java.net.HttpURLConnection object.

Setting The Content Type In Java

In most cases servlets are used to generate and serve HTML pages. However, servlets can serve any type of data including images, plain text, PDFs, spreadsheets, Javascript code, etc. To do this, the servlet must declare the type of content being served to the web browser. This declaration is called the content type or the media type of the file.

Here’s how to set the content type of a servlet’s response. The variable resp represents a javax.servlet.http.HttpServletResponse object:


Put the appropriate content type in the content_type variable. Some common content types are text/html(for HTML pages), text/plain (plain text), application/javascript (JS code), application/vnd.ms-excel (Excel spreadsheets), image/jpeg (JPEG images), application/pdf (PDFs); the list goes on and on. If you need to figure out the appropriate content type for your data, look it up on the Wikipedia Internet Media Types list.

Logging API Example

Here’s an example of how to use the Logging API to inspect an application’s logs. This function extracts all logs dated within the last hour, averages the execution time of all requests, and records how many requests resulted in errors (in other words recorded a FATAL level log report).

The String this function returns will look like this:

In the last 1 hour, requests took an average of 
451255 microseconds (451 ms) to execute. 0 errors reported.

This function is entirely self-contained; you don’t need to pass in any variables or set any global variables. However it needs to be run within an App Engine environment, either production or development.

public String doLogExam() {
     * For this example, we'll look into this application's logs, pull out the last few logs, 
     * and calculate the average length of servlet requests. 
     * We'll also examine each line of the log and see if there are any errors reported.
    //Access the log service, pull out logs, and grab an Iterator over the logs.
    LogService log = LogServiceFactory.getLogService();
    //Get all logs starting from 1 hour ago.
    long end_time = (new java.util.Date()).getTime();
    long start_time = end_time - (1000 * 60 * 60 * 1);
    LogQuery q = LogQuery.Builder.withDefaults().includeAppLogs(true).startTimeMillis(start_time).endTimeMillis(end_time);
    Iterator<RequestLogs> log_iterator = log.fetch(q).iterator();
    //Holds the sum of the execution time of all HTTP requests; we'll divide this by the 
    //num_of_logs_counted to get the average execution time.
    long execution_time_microseconds_sum = 0;
    //Number of log lines that are reporting errors.
    int num_of_error_log_lines = 0;
    //Number of logs that we pulled out of the LogService
    int num_of_logs_counted = 0;
    //Iterate over each log.
    while (log_iterator.hasNext()) {
        //Each request_log represents a single HTTP request.
        RequestLogs request_log = log_iterator.next();
        //Retrieve the execution time of this request, and add it to our sum variable.
        long execution_time_microseconds = request_log.getLatencyUsec();
        execution_time_microseconds_sum = execution_time_microseconds_sum + execution_time_microseconds;
        //Pull out any lines in this request log, and examine them to see 
        //if they report an error.
        List<AppLogLine> log_line_list = request_log.getAppLogLines();
        for (int i = 0; i < log_line_list.size(); i++) {
            AppLogLine app_log_line = log_line_list.get(i);
            LogService.LogLevel line_level = app_log_line.getLogLevel();
            //If this log line's reporting level is classified as fatal 
            //(causing the request to fail), record it.
            if (LogService.LogLevel.FATAL.equals(line_level)) {
        }//end looping through each line of the request log
    }//End looping through each request log
    long avg_execution_time_microsec = (execution_time_microseconds_sum / num_of_logs_counted);
    long avg_execution_time_millisec = avg_execution_time_microsec / 1000;
    String comment_text = "In the last 1 hour, requests took an average of ";
    comment_text += avg_execution_time_microsec + " microseconds (" + avg_execution_time_millisec;
    comment_text += " ms) to execute. " + num_of_error_log_lines + " errors reported.";
    return comment_text;

Remember to import the following classes:

import java.util.Iterator;
import java.util.List;
import com.google.appengine.api.log.AppLogLine;
import com.google.appengine.api.log.LogQuery;
import com.google.appengine.api.log.LogService;
import com.google.appengine.api.log.LogServiceFactory;
import com.google.appengine.api.log.RequestLogs;

Checking For A Twitter Follow

This brief code example checks to see whether user_to_check (a Twitter username as a String) is following the Twitter account twitter_username. The boolean is_user_following will be true if there is a follow.

The twitter object represents a twitter4j.Twitter class preconfigured with authentication details.

Relationship to_other_user = twitter.showFriendship(twitter_username, user_to_check);
boolean is_user_following = to_other_user.isTargetFollowingSource();

App Engine Production Upgrading To 1.8.4

App Engine is currently upgrading all applications to the 1.8.4 runtime. Here’s how the upgrade process looks on a free tier application.

On the main administration screen, this notice appears:

Clicking the Details link brings you to the Instances page, where this shows up:

The first instance loaded under the 1.8.4 runtime was launched on August 25, 14:43 PDT:

Geolocating A Tweet In Java

Here’s a demonstration of how to use the twitter4j library to post a tweet and set a location from where it was posted.

This example requires a twitter4j.twitter object (the twitter variable) to be preconfigured with authentication tokens. Replace the text Orlando, FL with a string describing the user’s location, and the GPS coordinates with your user’s coordinates.

    StatusUpdate status = new StatusUpdate(tweet_text);
     * We'll add location information for this tweet. 
     * Here, we tell Twitter that this tweet was made from Orlando, FL.
     * The GPS coordinates here are for Walt Disney's Hollywood Studios theme park.
    status.placeId("Orlando, FL");
    GeoLocation geolocate = new GeoLocation(28.355269, -81.559390);
    //And finally, post this tweet.
    try {
    catch (TwitterException e) {
        System.err.println("Problem accessing Twitter. Exception message: " + e.getMessage());

Remember the appropriate import:

import twitter4j.*;

The Cron Log

The following is an example of a request originating from a cron:

It’s easy to tell this request originates from cron because (1) the originating IP is (which is a reserved IP; cron requests come only from that address) and (2) the queue name is listed as cron .