Generate SHA1 Hash

Many web applications need to generate a SHA1 hash for verifying file integrity. If you use the Apache Commons libraries, the DigestUtils class offers convenience methods for generating hashes.

If you don’t use Apache Commons, the following function will generate SHA1 hashes for you. Simply pass the string to calculate the hash from into the function ( to_be_sha1 ); the function will then return the SHA1 hash. If an error is encountered (such as the passed-in string being null), a RuntimeException will be thrown.

/**
 * Calculates a SHA1 hash from a provided String. If 
 * to_be_sha1 is null, a RuntimeException will be thrown.
 * 
 * @param to_be_sha1 String to calculate a SHA1 hash from.
 * @return A SHA1 hash from the provided String.
 */
public static String generateSHA1(String to_be_sha1) {
    String sha1_sum = "";
    //if the provided String is null, throw an Exception.
    if (to_be_sha1 == null) {
        throw new RuntimeException("There is no String to calculate a SHA1 hash from.");
    }
    try {
        MessageDigest digest = MessageDigest.getInstance("SHA1");
        byte[] array = digest.digest(to_be_sha1.getBytes("UTF-8"));
        StringBuffer collector = new StringBuffer();
        for (int i = 0; i < array.length; i++) {
            collector.append(Integer.toString((array[i] & 0xff) + 0x100, 16).substring(1));
        }
        sha1_sum = collector.toString();
    }//end try
    catch (NoSuchAlgorithmException e) {
        throw new RuntimeException("Could not find a SHA1 instance: " + e.getMessage());
    }
    catch (UnsupportedEncodingException e) {
        throw new RuntimeException("Could not translate UTF-8: " + e.getMessage());
    }
    return sha1_sum;
}//end generateSHA1

Add these imports as well:

import java.io.UnsupportedEncodingException;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;

Generating A JAR Application For Compute Engine

Packaging Java applications for Compute Engine is slightly different than for App Engine. The code, plus any associated libraries and resources, needs to be built into a single JAR file. Here’s how to do that in Eclipse.

First, go to File – Export:

Select Java – Runnable JAR file from the options presented:

This screen sets up the JAR options. The Launch Configuration option sets the class file to run when the JAR is executed (in other words, the file containing the main(String[] args) function to start the application). Export Destination sets the directory to store the finished JAR file in.

Press Finish when you’re done. You can now use gcutil to upload the JAR file to your Compute Engine machine and run it using the standard java command (remember to install a JRE first).

URL Path To Array In Golang

Here’s a short Go function that converts the URL path into an array; it’s useful if you want to implement REST style queries into a simple application.

func parseComponents(r *http.Request) []string {
    //Creates an App Engine Context - required to access App Engine specific services
    c := appengine.NewContext(r)
    //The URL that the user queried.
    path := r.URL.Path
    path = strings.TrimSpace(path)
    //Log the URL received:
    c.Infof("URL: ", path)
    //Cut off the leading and trailing forward slashes, if they exist.
    //This cuts off the leading forward slash.
    if strings.HasPrefix(path, "/") {
        path = path[1:]
    }
    //This cuts off the trailing forward slash.
    if strings.HasSuffix(path, "/") {
        cut_off_last_char_len := len(path) - 1
        path = path[:cut_off_last_char_len]
    }
    //We need to isolate the individual components of the path.
    components := strings.Split(path, "/")
    return components
}

Useful Commands In Google Compute Engine

I’m currently building a Java application on top of a Compute Engine machine. Here’s a few commands I find useful:

Authorizing Access To A Project & Setting A Default Project:

gcutil auth –project=projectname

gcutil getproject –project=projectname –cache_flag_values

Open Up Firewall Port:

gcutil addfirewall name-of-firewall –description=“description of firewall” –allowed=“tcp:portname

For instance, the following firewall rule opens up port 25 and names the firewall smtp:

gcutil addfirewall smtp –description=“Allow Incoming Mail” –allowed=“tcp:25”

Upload To A Server: gcutil –project=projectname push servername local-path remote-path

SSH To A Server: gcutil –project=projectname ssh servername

Installing Java: sudo aptitude install openjdk-7-jre-headless

Running A Java JAR: sudo java -jar application-file.jar

Free Tier Performance

Two weeks ago I posted a picture of a free tier application serving close to 52,000 requests in 1 day. Soon after that, the same application exceeded that by processing over 53,000 requests within a single day. Here’s a few pictures demonstrating the capabilities of GAE’s free tier:

First off, here’s a picture of the Current Load screen:

A graph of milliseconds/request over the same time period:

The instances used:

And finally requests/second:

App Engine’s free tier is generally enough to host applications during development, testing, and low-level production traffic. With that said, it’s always a good idea to activate billing if you believe that your application will be receiving more than small amounts of traffic.

Reading In A File With Java

Here’s a short code example demonstrating how to read in a file within your application. This snippet assumes the file data.json is within your application’s root directory (change as necessary). The contents of the file are read into the variable json – insert your own processing code after the while code block. If the file cannot be found, or if there are other difficulties in reading from the file, an IOException will be thrown.

try {
    InputStream is = context.getResourceAsStream("/data.json");
    BufferedReader reader = new BufferedReader(new InputStreamReader(is));  
    String json = "";
    String line = "";   
    while (line != null) {
        json += line;
        line = reader.readLine();
    }   
    //Do something with the read-in data within the json variable.
}
catch (IOException e) {
    throw new RuntimeException("Unable to read in file.");
}

Inspecting Task Queue Logs

Here’s a sample view of a request from the Task Queue service. The log includes the name of the queue (in this case default), the task’s name (the long number starting with 1636460), and the instance that handled the request (the blue text on the bottom).

An important note: a quick and easy way to figure out if a request comes from the Task Queue is to look at the request’s IP address. All Task Queue requests come from the IP 0.1.0.2.

Generating MD5 Hashes In Java

Web applications frequently need to generate hashes for comparing data, verifying file integrity, generating keys, etc. If your application uses the Apache Commons libraries, you can use the included DigestUtils convenience class to create hashes. If you don’t use the Apache libraries, then you need an alternate way to generate MD5 hashes.

The below code snippet is a simple function that generates a MD5 hash from a given string. It’s simple to use: simply pass the string to be hashed into the function (to_be_md5), and the hash is returned. This function will throw a RuntimeException if an error is encountered (for example, if the passed-in String is null).

/**
 * Calculate a MD5 hash from the provided String. If the 
 * provided String is null, this method will throw a 
 * RuntimeException.
 * 
 * @param to_be_md5 A String to calculate a MD5 hash from.
 * @return A MD5 hash calculated from the provided String.
 * @throws RuntimeException If an error was encountered during calculating.
 */
public static String generateMD5(String to_be_md5) {
    String md5_sum = "";
    //If the provided String is null, then throw an Exception.
    if (to_be_md5 == null) {
        throw new RuntimeException("There is no string to calculate a MD5 hash from.");
    }
    try {
        MessageDigest md = MessageDigest.getInstance("MD5");
        byte[] array = md.digest(to_be_md5.getBytes("UTF-8"));
        StringBuffer collector = new StringBuffer();
        for (int i = 0; i < array.length; ++i) {
            collector.append(Integer.toHexString((array[i] & 0xFF) | 0x100).substring(1, 3));
        }
        md5_sum = collector.toString();
    }//end try
    catch (NoSuchAlgorithmException e) {
        throw new RuntimeException("Could not find a MD5 instance: " + e.getMessage());
    }
    catch (UnsupportedEncodingException e) {
        throw new RuntimeException("Could not translate UTF-8: " + e.getMessage());
    }
    return md5_sum;
}//end generateMD5

Don’t forget to import the following classes:

import java.io.UnsupportedEncodingException;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;

Retrieving All Recently Created Entities

Searching the datastore using date-based entity properties can often be confusing to new App Engine developers. Here’s how to search for all recently created entities.

First, each entity has to record the time it was created. Add the following line of code when constructing your entities (entity represents the Entity object being created):

entity.setProperty("add_date", new Date());//Records the date this entity was added.

Here’s the code for the query. This code example searches for all entities created within the last 24 hours with a kind of mail. The resulting entities are placed into the variable results (which is a standard java.util.List object).

/**
 * Creates a Date object that represents a point in time 
 * 24 hours ago.
 */
Date current_date = new Date();//The current date.
long current_date_in_ms = current_date.getTime();//The current date in milliseconds.
//Take the current date in milliseconds, and subtract 24 hours worth 
//of milliseconds (1000 milliseconds per second, times 60 seconds per 
//minute, times 60 minutes per hour, times 24 hours in a day).
long one_day_ago_in_ms = current_date_in_ms - (1000 * 60 * 60 * 24);
Date one_day_ago = new Date(one_day_ago_in_ms);//Create a new Date object representing one day ago.
//Search all entities with the kind "mail". 
//We create a Filter that requires that the add_date property be greater 
//than the Date we computed above.
Query q = new Query("mail");
Query.Filter date_filter = new Query.FilterPredicate("add_date", Query.FilterOperator.GREATER_THAN_OR_EQUAL, one_day_ago);
q.setFilter(date_filter);
PreparedQuery pq = DatastoreServiceFactory.getDatastoreService().prepare(q);
List<Entity> results = pq.asList(FetchOptions.Builder.withDefaults());

 

Error Code 202

On high traffic App Engine applications, you may occasionally see a request fail with error code 202. You’ll see the following text in your logs:

A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. (Error code 202)

The request log will look similar to the picture below:

Error code 202 is an internal App Engine failure code: an unknown error occurred within App Engine or its associated services, not from your application. This is not an error that your application can catch or recover from, nor is the application at fault for this error. The current request will fail, but the client is free to retry the request. If this request is from the task queue service, the task queue will automatically retry the request if you set the appropriate settings while creating the task.

A note about error codes: the code 202 relates to App Engine; it is NOT a HTTP status code. When a request encounters this internal error, it fails with the HTTP status code 500 (as you can see from the above picture).