Bulk Adding Headers To An URL Fetch Request

A quick code example: how to easily add headers to an URL Fetch request.

First, create a java.util.Hashtable:

Hashtable<String, String> request_headers;
request_headers = new Hashtable<String, String>();

Put the headers you want into this hashtable. The keys and values of this hashtable will become the header names and values in the fetch request.

When you’re configuring the URL Fetch request, use the code below to add in all the headers:

Enumeration<String> set_header_keys = request_headers.keys();
while (set_header_keys.hasMoreElements()) {
    String key = set_header_keys.nextElement();
    String value = request_headers.get(key);
    connection.setRequestProperty(key, value);
}

The connection variable represents a java.net.HttpURLConnection object.

URLFetch User Agent

When an application makes an URLFetch request, App Engine adds the following text as the User-Agentheader:

AppEngine-Google; (+http://code.google.com/appengine; 
appid: YOUR_APPLICATION_ID_HERE)

Even if the application sets a custom user agent header, App Engine will append the above text to the header.

This can be annoying because there are some servers and services that rate limit based on the user agent. If there is a human reviewing the request logs, it can be confusing to see a stream of largely-identical user agent strings.

It’s good practice to set a descriptive user agent for all URL fetches. It’s even better if you can write your user agent with App Engine’s required text in mind. For instance, consider writing user agent headers like this one: App Engine Example Crawler hosted by. When App Engine appends its required text to the end of this, the receiving server will see an user agent of:

App Engine Example Crawler hosted by
AppEngine-Google; (+http://code.google.com/appengine; 
appid: YOUR_APPLICATION_ID_HERE)

This user agent header looks cleaner, neater, and is easier for a human to understand.

Here is the above in code form:

String user_agent = "App Engine Example Crawler";
user_agent += " hosted by ";//After this, GAE will append the identifier.
connection.setRequestProperty("User-Agent", user_agent);

The connection variable represents a java.net.HttpURLConnection object.

Setting The Content Type In Java

In most cases servlets are used to generate and serve HTML pages. However, servlets can serve any type of data including images, plain text, PDFs, spreadsheets, Javascript code, etc. To do this, the servlet must declare the type of content being served to the web browser. This declaration is called the content type or the media type of the file.

Here’s how to set the content type of a servlet’s response. The variable resp represents a javax.servlet.http.HttpServletResponse object:

resp.setContentType(content_type);

Put the appropriate content type in the content_type variable. Some common content types are text/html(for HTML pages), text/plain (plain text), application/javascript (JS code), application/vnd.ms-excel (Excel spreadsheets), image/jpeg (JPEG images), application/pdf (PDFs); the list goes on and on. If you need to figure out the appropriate content type for your data, look it up on the Wikipedia Internet Media Types list.

Logging API Example

Here’s an example of how to use the Logging API to inspect an application’s logs. This function extracts all logs dated within the last hour, averages the execution time of all requests, and records how many requests resulted in errors (in other words recorded a FATAL level log report).

The String this function returns will look like this:

In the last 1 hour, requests took an average of 
451255 microseconds (451 ms) to execute. 0 errors reported.

This function is entirely self-contained; you don’t need to pass in any variables or set any global variables. However it needs to be run within an App Engine environment, either production or development.

public String doLogExam() {
    /**
     * For this example, we'll look into this application's logs, pull out the last few logs, 
     * and calculate the average length of servlet requests. 
     * We'll also examine each line of the log and see if there are any errors reported.
     */
    //Access the log service, pull out logs, and grab an Iterator over the logs.
    LogService log = LogServiceFactory.getLogService();
    //Get all logs starting from 1 hour ago.
    long end_time = (new java.util.Date()).getTime();
    long start_time = end_time - (1000 * 60 * 60 * 1);
    LogQuery q = LogQuery.Builder.withDefaults().includeAppLogs(true).startTimeMillis(start_time).endTimeMillis(end_time);
    Iterator<RequestLogs> log_iterator = log.fetch(q).iterator();
    //Holds the sum of the execution time of all HTTP requests; we'll divide this by the 
    //num_of_logs_counted to get the average execution time.
    long execution_time_microseconds_sum = 0;
    //Number of log lines that are reporting errors.
    int num_of_error_log_lines = 0;
    //Number of logs that we pulled out of the LogService
    int num_of_logs_counted = 0;
    //Iterate over each log.
    while (log_iterator.hasNext()) {
        //Each request_log represents a single HTTP request.
        RequestLogs request_log = log_iterator.next();
        num_of_logs_counted++;
        //Retrieve the execution time of this request, and add it to our sum variable.
        long execution_time_microseconds = request_log.getLatencyUsec();
        execution_time_microseconds_sum = execution_time_microseconds_sum + execution_time_microseconds;
        //Pull out any lines in this request log, and examine them to see 
        //if they report an error.
        List<AppLogLine> log_line_list = request_log.getAppLogLines();
        for (int i = 0; i < log_line_list.size(); i++) {
            AppLogLine app_log_line = log_line_list.get(i);
            LogService.LogLevel line_level = app_log_line.getLogLevel();
            //If this log line's reporting level is classified as fatal 
            //(causing the request to fail), record it.
            if (LogService.LogLevel.FATAL.equals(line_level)) {
                num_of_error_log_lines++;
            }
        }//end looping through each line of the request log
    }//End looping through each request log
    long avg_execution_time_microsec = (execution_time_microseconds_sum / num_of_logs_counted);
    long avg_execution_time_millisec = avg_execution_time_microsec / 1000;
    String comment_text = "In the last 1 hour, requests took an average of ";
    comment_text += avg_execution_time_microsec + " microseconds (" + avg_execution_time_millisec;
    comment_text += " ms) to execute. " + num_of_error_log_lines + " errors reported.";
    return comment_text;
}

Remember to import the following classes:

import java.util.Iterator;
import java.util.List;
import com.google.appengine.api.log.AppLogLine;
import com.google.appengine.api.log.LogQuery;
import com.google.appengine.api.log.LogService;
import com.google.appengine.api.log.LogServiceFactory;
import com.google.appengine.api.log.RequestLogs;

Checking For A Twitter Follow

This brief code example checks to see whether user_to_check (a Twitter username as a String) is following the Twitter account twitter_username. The boolean is_user_following will be true if there is a follow.

The twitter object represents a twitter4j.Twitter class preconfigured with authentication details.

Relationship to_other_user = twitter.showFriendship(twitter_username, user_to_check);
boolean is_user_following = to_other_user.isTargetFollowingSource();

Geolocating A Tweet In Java

Here’s a demonstration of how to use the twitter4j library to post a tweet and set a location from where it was posted.

This example requires a twitter4j.twitter object (the twitter variable) to be preconfigured with authentication tokens. Replace the text Orlando, FL with a string describing the user’s location, and the GPS coordinates with your user’s coordinates.

    StatusUpdate status = new StatusUpdate(tweet_text);
    /**
     * We'll add location information for this tweet. 
     * Here, we tell Twitter that this tweet was made from Orlando, FL.
     * The GPS coordinates here are for Walt Disney's Hollywood Studios theme park.
     */
    status.placeId("Orlando, FL");
    GeoLocation geolocate = new GeoLocation(28.355269, -81.559390);
    status.setLocation(geolocate);
    status.setDisplayCoordinates(true);
    //And finally, post this tweet.
    try {
        twitter.updateStatus(status);
    }
    catch (TwitterException e) {
        System.err.println("Problem accessing Twitter. Exception message: " + e.getMessage());
    }

Remember the appropriate import:

import twitter4j.*;

Adding A Cookie To A Response In Java

Here’s a basic example of how to set a cookie into a servlet response.

Cookie_name and cookie_value is the name and value for the cookie to store. The variable resp represents a javax.servlet.http.HttpServletResponse reference. Notice setMaxAge(int expiry) – this method sets how long the cookie should last before it expires in seconds. In this example I set a value of 2 weeks (multiplying the following together: 60 seconds, 60 minutes, 24 hours, 7 days, 2 weeks).

Cookie cookie = new Cookie(cookie_name, cookie_value);
cookie.setMaxAge(2 * 7 * 24 * 60 * 60);//Two weeks to expire, in seconds.
resp.addCookie(cookie);

Remember to add the import (the package includes the Cookie class and HttpServletResponse ):

import javax.servlet.http.*;

Checking To See If An Application Is In Production

Sometimes an application needs to be able to tell if it’s in production (deployed on App Engine servers) or in development (deployed on the dev appserver on a developer’s local machine). You can test this by querying the SystemProperty class. Here’s a code example:

String runtime_env = SystemProperty.environment.value().toString(); 
if (runtime_env.equalsIgnoreCase("Production")) {
    //This application is in production. Do something special.
}
else {
    //This application is NOT in production (perhaps the dev appserver)
}

Remember to import the appropriate class:

import com.google.appengine.api.utils.SystemProperty;

Measuring Elapsed Time: System.nanoTime()

When I need to measure elapsed time (for example, timing how long an API call takes) I prefer to use System.nanoTime() instead of using the usual method of (new Date()).getTime().

The key idea to remember is that nanoTime() doesn’t return a timestamp – it represents a measurement of time from some arbitrary point (so for example nanoTime() could return a negative figure). This helps when writing self-documenting code: a variable storing a nanoTime() value can only be used for elapsed time comparisons, not timestamp recording.

Here’s an example of measuring elapsed time with System.nanoTime(). First, record the start time:

/**
 * Records the start time 
 * using System.nanoTime()
 */
Long start_time;
//Record the start time.
start_time = System.nanoTime();

Insert some time-consuming operation here, or another long-running call. Place this code where you want to end the timer:

//Calculate how long we've been running in nanoseconds.
Long diff_time = System.nanoTime() - start_time;

The variable diff_time stores the number of nanoseconds that has elapsed during the timer. Now suppose you wanted to throw an exception if the timer took more than 30 seconds (30 billion nanoseconds); here’s the example code:

//We set the maximum time in nanoseconds, multiplied by milliseconds, 
//multiplied by seconds.
Long MAXIMUM_TIME_OPEN = new Long(1000000L * 1000 * 30);
//If the runtime of this operation is longer than the time we've set, 
//throw an Exception.
if (diff_time > MAXIMUM_TIME_OPEN) {
    throw new IOException("Timeout has been exceeded.");
}

To keep the code easy to understand, we’re showing how the maximum time amount is computed: there are 1 million (1,000,000) nanoseconds in a millisecond, multiplied by 1 thousand milliseconds in a second (1,000), multiplied by 30 seconds.

Creating A Server In Java On Compute Engine

Compute Engine is a terrific hosting platform for applications – not only HTTP-based applications, but for servers running all types of services. To run these services an application has to listen for incoming connections using a server socket. Here’s how to do that in Java.

First, bind a server socket to a port. Here we’re binding it to the SMTP-reserved port 25. The port binding will fail if there’s already a server bound to that port.

//The server socket that handles all incoming connections.
ServerSocket server_socket = null;
//Bind the server socket to port 25. Fail and exit 
//if we fail to bind.
try {
    server_socket = new ServerSocket(25);
    System.out.println("OK: Bound to port 25.");
} 
catch (IOException e) {
    System.err.println("ERROR: Unable to bind to port 25.");
    e.printStackTrace();
    System.exit(-1);
}

Now wait for an incoming connection to accept (you can put the following code into a while loop if you want to accept multiple connections):

//Now we need to wait for a client to connect to us.
//Accept() blocks until it receives a new connection.
try {
    //A new client has connected.
    Socket client_socket = server_socket.accept();
    System.out.println("Accepted!");
    //Hand off the socket for further handling.
    //Do something here with the socket.
} 
catch (IOException e) {
    System.err.println("Failed to accept incoming connection:" + e.getMessage());
    e.printStackTrace();
    System.exit(-1);
}

From here, handle the socket as needed.

Remember to import the following classes:

import java.io.IOException;
import java.net.ServerSocket;
import java.net.Socket;