App Engine Logos

Badge logos (sometimes called button logos) are used to advertise a website’s server software or platform. Here’s a few examples:

Apache Badge Logo

Apple Xserve:

PHP:

Personally, I always like seeing badge logos because they give an interesting insight into that website: which stack is powering the website, the preferred coding language of the webmaster, etc.

Here’s logos for App Engine:

For reference, these are the equivalent badge logos for Microsoft Azure and AWS:

 

Google Cloud Storage RetryHelper Failure

I have numerous applications which use Google Cloud Storage buckets to save files, maintain backups, etc. Most of these applications are Java apps using the official Google Cloud Storage library.

Yesterday one of these applications had difficulty accessing Cloud Storage and printed out the following error report:

The GCS library attempted to access Cloud Storage a total of 6 times; this image shows the last failure and the final “giving-up” error.

Here’s a copy of the error in text form:

com.google.appengine.tools.cloudstorage.RetryHelper doRetry: 
    RetryHelper(4.114 s, 6 attempts, com.google.appengine.tools.cloudstorage.GcsServiceImpl): 
    Attempt 6 failed, sleeping for 3140 ms: java.io.IOException: Response code 503, 
    retryable: Request: POST https://storage.googleapis.com/[bucket_url]/[file_name]
x-goog-resumable: start
x-goog-api-version: 2
Content-Type: image/jpg

no content

Response: 503 with 19 bytes of content
Content-Length: 19
Date: Thu, 01 Aug 2013 [time]
Server: HTTP Upload Server Built on Jul 21 2013 19:20:38 (1374459638)
Content-Type: text/html; charset=UTF-8
X-Google-Cache-Control: remote-fetch
Via: HTTP/1.1 GWA
Service Unavailable

And after 6 failures attempting to access the Google Cloud Storage bucket, the library reports an error and stops:

.<stderr>: IO Exception: RetryHelper(4.115 s, 6 attempts, 
    com.google.appengine.tools.cloudstorage.GcsServiceImpl): 
    Too many failures, giving up

Fortunately this seemed to be only a transient problem; the next upload to GCS succeeded.

Creating A Google Cloud Storage Bucket

Creating a Google Cloud Storage bucket is relatively simple. Just follow these steps:

Go to https://cloud.google.com/console‎ and log in. You’ll see the following page. Create a project or click on an already-existing project (in this picture, Fact and Invalid are pre-existing projects).

Click on the Cloud Storage link in the next screen:

Now press the New Bucket button:

A prompt will open asking you to name your new bucket.

Your new bucket will be created. You can upload new files and folders to the bucket by clicking the Upload button in the middle of the screen:

Generate SHA1 Hash

Many web applications need to generate a SHA1 hash for verifying file integrity. If you use the Apache Commons libraries, the DigestUtils class offers convenience methods for generating hashes.

If you don’t use Apache Commons, the following function will generate SHA1 hashes for you. Simply pass the string to calculate the hash from into the function ( to_be_sha1 ); the function will then return the SHA1 hash. If an error is encountered (such as the passed-in string being null), a RuntimeException will be thrown.

/**
 * Calculates a SHA1 hash from a provided String. If 
 * to_be_sha1 is null, a RuntimeException will be thrown.
 * 
 * @param to_be_sha1 String to calculate a SHA1 hash from.
 * @return A SHA1 hash from the provided String.
 */
public static String generateSHA1(String to_be_sha1) {
    String sha1_sum = "";
    //if the provided String is null, throw an Exception.
    if (to_be_sha1 == null) {
        throw new RuntimeException("There is no String to calculate a SHA1 hash from.");
    }
    try {
        MessageDigest digest = MessageDigest.getInstance("SHA1");
        byte[] array = digest.digest(to_be_sha1.getBytes("UTF-8"));
        StringBuffer collector = new StringBuffer();
        for (int i = 0; i < array.length; i++) {
            collector.append(Integer.toString((array[i] & 0xff) + 0x100, 16).substring(1));
        }
        sha1_sum = collector.toString();
    }//end try
    catch (NoSuchAlgorithmException e) {
        throw new RuntimeException("Could not find a SHA1 instance: " + e.getMessage());
    }
    catch (UnsupportedEncodingException e) {
        throw new RuntimeException("Could not translate UTF-8: " + e.getMessage());
    }
    return sha1_sum;
}//end generateSHA1

Add these imports as well:

import java.io.UnsupportedEncodingException;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;

Generating A JAR Application For Compute Engine

Packaging Java applications for Compute Engine is slightly different than for App Engine. The code, plus any associated libraries and resources, needs to be built into a single JAR file. Here’s how to do that in Eclipse.

First, go to File – Export:

Select Java – Runnable JAR file from the options presented:

This screen sets up the JAR options. The Launch Configuration option sets the class file to run when the JAR is executed (in other words, the file containing the main(String[] args) function to start the application). Export Destination sets the directory to store the finished JAR file in.

Press Finish when you’re done. You can now use gcutil to upload the JAR file to your Compute Engine machine and run it using the standard java command (remember to install a JRE first).

URL Path To Array In Golang

Here’s a short Go function that converts the URL path into an array; it’s useful if you want to implement REST style queries into a simple application.

func parseComponents(r *http.Request) []string {
    //Creates an App Engine Context - required to access App Engine specific services
    c := appengine.NewContext(r)
    //The URL that the user queried.
    path := r.URL.Path
    path = strings.TrimSpace(path)
    //Log the URL received:
    c.Infof("URL: ", path)
    //Cut off the leading and trailing forward slashes, if they exist.
    //This cuts off the leading forward slash.
    if strings.HasPrefix(path, "/") {
        path = path[1:]
    }
    //This cuts off the trailing forward slash.
    if strings.HasSuffix(path, "/") {
        cut_off_last_char_len := len(path) - 1
        path = path[:cut_off_last_char_len]
    }
    //We need to isolate the individual components of the path.
    components := strings.Split(path, "/")
    return components
}

Useful Commands In Google Compute Engine

I’m currently building a Java application on top of a Compute Engine machine. Here’s a few commands I find useful:

Authorizing Access To A Project & Setting A Default Project:

gcutil auth –project=projectname

gcutil getproject –project=projectname –cache_flag_values

Open Up Firewall Port:

gcutil addfirewall name-of-firewall –description=“description of firewall” –allowed=“tcp:portname

For instance, the following firewall rule opens up port 25 and names the firewall smtp:

gcutil addfirewall smtp –description=“Allow Incoming Mail” –allowed=“tcp:25”

Upload To A Server: gcutil –project=projectname push servername local-path remote-path

SSH To A Server: gcutil –project=projectname ssh servername

Installing Java: sudo aptitude install openjdk-7-jre-headless

Running A Java JAR: sudo java -jar application-file.jar

Free Tier Performance

Two weeks ago I posted a picture of a free tier application serving close to 52,000 requests in 1 day. Soon after that, the same application exceeded that by processing over 53,000 requests within a single day. Here’s a few pictures demonstrating the capabilities of GAE’s free tier:

First off, here’s a picture of the Current Load screen:

A graph of milliseconds/request over the same time period:

The instances used:

And finally requests/second:

App Engine’s free tier is generally enough to host applications during development, testing, and low-level production traffic. With that said, it’s always a good idea to activate billing if you believe that your application will be receiving more than small amounts of traffic.

Reading In A File With Java

Here’s a short code example demonstrating how to read in a file within your application. This snippet assumes the file data.json is within your application’s root directory (change as necessary). The contents of the file are read into the variable json – insert your own processing code after the while code block. If the file cannot be found, or if there are other difficulties in reading from the file, an IOException will be thrown.

try {
    InputStream is = context.getResourceAsStream("/data.json");
    BufferedReader reader = new BufferedReader(new InputStreamReader(is));  
    String json = "";
    String line = "";   
    while (line != null) {
        json += line;
        line = reader.readLine();
    }   
    //Do something with the read-in data within the json variable.
}
catch (IOException e) {
    throw new RuntimeException("Unable to read in file.");
}

Inspecting Task Queue Logs

Here’s a sample view of a request from the Task Queue service. The log includes the name of the queue (in this case default), the task’s name (the long number starting with 1636460), and the instance that handled the request (the blue text on the bottom).

An important note: a quick and easy way to figure out if a request comes from the Task Queue is to look at the request’s IP address. All Task Queue requests come from the IP 0.1.0.2.