Posted on

Utility Functions To Save To Google Cloud Storage

Here are short utility functions used to save to Google Cloud Storage in a Java App Engine application.

These snippets require the Java Cloud Storage library ( com.google.appengine.tools.cloudstorage.GcsService ). The bucket string is the name of the bucket, no gs:// required. folder_path is the subdirectory you want to save the file under. It should end with a forward slash. For example this/is/a/subdirectory/ . If you do not want a subdirectory, pass an empty string (“”). filename is the filename of the file you want to save, while contents is the data being saved.

There are two functions here. The top is intended to quickly save simple text/html/xml/etc files, while the bottom is intended to save binary files such as images, executables, and so forth. The bottom function includes a filetype argument, which should be filled with the appropriate MIME type. For example: image/png, image/jpg, application/octet-stream.

Reminder: GCS innately views files as individual objects. When you specify a subfolder, the filename for the object itself includes the folder path. When you read the file back from Google cloud storage, you need to include the folder path: for example this/is/a/subdirectory/example.png .

	public static void saveStringToGCS(String bucket, String folder_path, String filename, String contents) throws IOException {
		GcsService storage_service = GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance());
		GcsFilename to_file = new GcsFilename(bucket, folder_path + filename);
		GcsFileOptions file_options = GcsFileOptions.getDefaultInstance();
		storage_service.createOrReplace(to_file, file_options, ByteBuffer.wrap(contents.getBytes(StandardCharsets.UTF_8)));
	}
	
	public static void saveByteBufferToGCS(String bucket, String folder_path, String filename, ByteBuffer contents, String filetype) throws IOException {
		GcsService storage_service = GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance());
		GcsFilename to_file = new GcsFilename(bucket, folder_path + filename);
		GcsFileOptions file_options = (new GcsFileOptions.Builder()).mimeType(filetype).build();
		storage_service.createOrReplace(to_file, file_options, contents);
	}
Posted on

A HTTP GET Example Using The Apache HTTPClient Library

Here’s a quick example of using the Apache HTTPClient library to issue a simple GET request. The URL you’re connecting to is in endpoint, and the contents of the URL can be read from the BufferedReader response_reader.

		HttpGet get_request = new HttpGet(endpoint);
		get_request.addHeader(HttpHeaders.USER_AGENT, "Example Reader Service");

		HttpResponse http_response = HttpClientBuilder.create().build().execute(get_request);
		content_type_header = http_response.getFirstHeader("Content-Type").getValue();
		
		int response_code = http_response.getStatusLine().getStatusCode();

		if (response_code != 200) {
			throw new IOException("Response code is not 200 OK. Response code: " + response_code + " : " + endpoint.toString());
		}

		//Pull out text response
		InputStream response_is = http_response.getEntity().getContent();
		BufferedReader response_reader = new BufferedReader(new InputStreamReader(response_is, "UTF-8"));

Posted on

Mapping Unicode Punctuation To ASCII Punctuation

The following code maps a bunch of unicode punctuation to their more normal ASCII counterparts. For example, the Unicode character LEFT DOUBLE QUOTATION MARK is mapped to the ASCII double quotation mark: “. 

		Hashtable<String, String> map = new Hashtable<String, String>();
		map.put(new String(Character.toChars(0x00AB)), new String("\""));
		map.put(new String(Character.toChars(0x00AD)), "-");
		map.put(new String(Character.toChars(0x00B4)), "\'");
		map.put(new String(Character.toChars(0x00BB)), "\"");
		map.put(new String(Character.toChars(0x00F7)), "/");
		map.put(new String(Character.toChars(0x01C0)), "|");
		map.put(new String(Character.toChars(0x01C3)), "!");
		map.put(new String(Character.toChars(0x02B9)), "\'");
		map.put(new String(Character.toChars(0x02BA)), "\"");
		map.put(new String(Character.toChars(0x02BC)), "\'");
		map.put(new String(Character.toChars(0x02C4)), "^");
		map.put(new String(Character.toChars(0x02C6)), "^");
		map.put(new String(Character.toChars(0x02C8)), "\'");
		map.put(new String(Character.toChars(0x02CB)), "`");
		map.put(new String(Character.toChars(0x02CD)), "_");
		map.put(new String(Character.toChars(0x02DC)), "~");
		map.put(new String(Character.toChars(0x0300)), "`");
		map.put(new String(Character.toChars(0x0301)), "\'");
		map.put(new String(Character.toChars(0x0302)), "^");
		map.put(new String(Character.toChars(0x0303)), "~");
		map.put(new String(Character.toChars(0x030B)), "\"");
		map.put(new String(Character.toChars(0x030E)), "\"");
		map.put(new String(Character.toChars(0x0331)), "_");
		map.put(new String(Character.toChars(0x0332)), "_");
		map.put(new String(Character.toChars(0x0338)), "/");
		map.put(new String(Character.toChars(0x0589)), ":");
		map.put(new String(Character.toChars(0x05C0)), "|");
		map.put(new String(Character.toChars(0x05C3)), ":");
		map.put(new String(Character.toChars(0x066A)), "%");
		map.put(new String(Character.toChars(0x066D)), "*");
		map.put(new String(Character.toChars(0x200B)), " ");
		map.put(new String(Character.toChars(0x2010)), "-");
		map.put(new String(Character.toChars(0x2011)), "-");
		map.put(new String(Character.toChars(0x2012)), "-");
		map.put(new String(Character.toChars(0x2013)), "-");
		map.put(new String(Character.toChars(0x2014)), "-");
		map.put(new String(Character.toChars(0x2015)), "-");
		map.put(new String(Character.toChars(0x2016)), "|");
		map.put(new String(Character.toChars(0x2017)), "_");
		map.put(new String(Character.toChars(0x2018)), "\'");
		map.put(new String(Character.toChars(0x2019)), "\'");
		map.put(new String(Character.toChars(0x201A)), ",");
		map.put(new String(Character.toChars(0x201B)), "\'");
		map.put(new String(Character.toChars(0x201C)), "\"");
		map.put(new String(Character.toChars(0x201D)), "\"");
		map.put(new String(Character.toChars(0x201E)), "\"");
		map.put(new String(Character.toChars(0x201F)), "\"");
		map.put(new String(Character.toChars(0x2032)), "\'");
		map.put(new String(Character.toChars(0x2033)), "\"");
		map.put(new String(Character.toChars(0x2034)), "\'");
		map.put(new String(Character.toChars(0x2035)), "`");
		map.put(new String(Character.toChars(0x2036)), "\"");
		map.put(new String(Character.toChars(0x2037)), "\'");
		map.put(new String(Character.toChars(0x2038)), "^");
		map.put(new String(Character.toChars(0x2039)), "<");
		map.put(new String(Character.toChars(0x203A)), ">");
		map.put(new String(Character.toChars(0x203D)), "?");
		map.put(new String(Character.toChars(0x2044)), "/");
		map.put(new String(Character.toChars(0x204E)), "*");
		map.put(new String(Character.toChars(0x2052)), "%");
		map.put(new String(Character.toChars(0x2053)), "~");
		map.put(new String(Character.toChars(0x2060)), " ");
		map.put(new String(Character.toChars(0x20E5)), "\\");
		map.put(new String(Character.toChars(0x2212)), "-");
		map.put(new String(Character.toChars(0x2215)), "/");
		map.put(new String(Character.toChars(0x2216)), "\\");
		map.put(new String(Character.toChars(0x2217)), "*");
		map.put(new String(Character.toChars(0x2223)), "|");
		map.put(new String(Character.toChars(0x2236)), ":");
		map.put(new String(Character.toChars(0x223C)), "~");
		map.put(new String(Character.toChars(0x2264)), "<");
		map.put(new String(Character.toChars(0x2265)), ">");
		map.put(new String(Character.toChars(0x2266)), "<");
		map.put(new String(Character.toChars(0x2267)), ">");
		map.put(new String(Character.toChars(0x2303)), "^");
		map.put(new String(Character.toChars(0x2329)), "<");
		map.put(new String(Character.toChars(0x232A)), ">");
		map.put(new String(Character.toChars(0x266F)), "#");
		map.put(new String(Character.toChars(0x2731)), "*");
		map.put(new String(Character.toChars(0x2758)), "|");
		map.put(new String(Character.toChars(0x2762)), "!");
		map.put(new String(Character.toChars(0x27E6)), "[");
		map.put(new String(Character.toChars(0x27E8)), "<");
		map.put(new String(Character.toChars(0x27E9)), ">");
		map.put(new String(Character.toChars(0x2983)), "{");
		map.put(new String(Character.toChars(0x2984)), "}");
		map.put(new String(Character.toChars(0x3003)), "\"");
		map.put(new String(Character.toChars(0x3008)), "<");
		map.put(new String(Character.toChars(0x3009)), ">");
		map.put(new String(Character.toChars(0x301B)), "]");
		map.put(new String(Character.toChars(0x301C)), "~");
		map.put(new String(Character.toChars(0x301D)), "\"");
		map.put(new String(Character.toChars(0x301E)), "\"");
		map.put(new String(Character.toChars(0xFEFF)), " ");
Posted on

Delete Old Entities – Java Datastore

This is an ultra-simplified example of how to delete old entities from the App Engine Datastore. The first 3 lines of code retrieves the current date, then subtracts 60 days from the current time (the multiplication converts days to milliseconds). DATE_PROPERTY_ON_ENTITY is the date property on the entity – when first writing the entity to the datastore, add the current date as a property. ENTITY_KIND is the entity kind we’re deleting.

		//Calculate 60 days ago.
		long current_date_long = (new Date()).getTime();
		long past_date_long = current_date_long - (1000 * 60 * 60 * 24 * 60);
		Date past_date = new Date(past_date_long);
		
		DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
		Query.Filter date_filter = new Query.FilterPredicate("DATE_PROPERTY_ON_ENTITY", Query.FilterOperator.LESS_THAN_OR_EQUAL, past_date);
		Query date_query = new Query("ENTITY_KIND").setFilter(date_filter);
		PreparedQuery date_query_results = datastore.prepare(date_query);
		
		Iterator<Entity> iterate_over_old_entities = date_query_results.asIterator();
		
		while (iterate_over_old_entities.hasNext()) {
			Entity old_entity = iterate_over_old_entities.next();
			
			System.out.println("Deleting: " + old_entity.getProperties());
			
			datastore.delete(old_entity.getKey());
		}

Note that is a simplified function – it’s useful if you have a handful of entities that need deleting, but if you have more than a handful, you should convert to using datastore cursors and paging through entities to delete.

Posted on

IOException: tmpFile.renameTo Failed

On rare occasions, the Google App Engine dev server displays the following error:

The important part of the error is this text:

java.io.IOException: tmpFile.renameTo(classfile) failed

This exception crops up whenever an app file (in this case a JSP file) is currently being accessed by another program. If you see this exception, double check to ensure that the named file isn’t being accessed by another program.

If this error persists, close down and reopen Eclipse and the development app server – the file may have been left open from a previous run.

Posted on

Accessing Google Cloud SQL From App Engine

Accessing Google Cloud SQL from an App Engine application is relatively straightforward. To start, an application must first specify a host name for the Cloud SQL servers.

For example, here’s the host name for PHP applications:

:/cloudsql/[Google-Cloud-Project-Name]:[Cloud-SQL-Instance-Name]

Java accesses Cloud SQL through a special JDBC driver. Here’s the proper host name for it:

jdbc:google:mysql://[Google-Cloud-SQL-Project-Name]:[Cloud-SQL-Instance-Name]/[Database-Name]

Secondly, Cloud SQL must whitelist incoming connections from permitted App Engine applications. To do this, open up the Google Cloud console and select the project you’re using. Then press the Cloud SQL option on the left hand navigation bar:

Click the New Instance button:

On the bottom of the form there’s an option to whitelist named App Engine applications. Type in the application ID of the App Engine application using the database:

Click the Confirm button to finish setting up the database.

Posted on

AccessControlException Resulting From RuntimePermission: modifyThreadGroup

While using certain libraries on App Engine, you may encounter the following exception notice:

javax.servlet.ServletContext log: 
    Exception while dispatching incoming RPC call
threw an unexpected exception: 
    java.security.AccessControlException: 
    access denied (java.lang.RuntimePermission modifyThreadGroup)

If you see this exception message, your application or (more frequently the case) a library is attempting to create a new thread. App Engine doesn’t allow frontend instances to spawn threads, so any attempt to start up a thread will result in AccessControlExceptions.

However, App Engine does allow backend threads: threads which run within backend instances. If your application absolutely needs to run threads, run the threading component within a backend or a backend module.

Posted on

DatastoreFailureException: Internal Error Exception Messages

On rare occasions, an application may spawn errors similar to the below:

com.google.appengine.api.datastore.DatastoreFailureException: 
    internal error.
at com.google.appengine.api.datastore.DatastoreApiHelper.translateError     (DatastoreApiHelper.java:50)
at com.google.appengine.api.datastore.DatastoreApiHelper$1.convertException     (DatastoreApiHelper.java:70)

As the exception message suggests, this log indicates that the datastore encountered an internal error while handling the datastore operation. Unfortunately, there’s nothing that a developer can do to fix this error since it’s an internal App Engine issue.

Generally this type of exception fixes itself sooner or later; if it persists, file an issue at the App Engine bug tracker: https://code.google.com/p/googleappengine/issues/entry?template=Production%20issue

Posted on

Datastore Query Cursor: ExQ

While using JDO or another datastore helper library, applications may encounter a value of ExQ when requesting a query cursor. This value means that there are no more results (entities) to display for that query, and consequently that there are no more query pages to return.

All applications using query cursors should check for an invalid or end of query results marker, and handle this in an application-specific way. A query cursor string that is null or empty (zero length) generally denotes the end of query results. The cursor value ExQ can be handled in the same way.

A word of warning: some third-party datastore helper libraries spawn an exception or error condition when applications attempt to access invalid cursors, or attempt to request a cursor when end of results has already been reached. When using these libraries, it’s important to fully read the documentation and understand how these libraries interact with the App Engine datastore.

Posted on

Mapping Servlets To App Engine XMPP Requests

Here’s an example of how to configure servlets to accept XMPP requests. The XML below should be added to the web.xml file in the /war/WEB-INF directory.

This XML snippet configures three servlets: ChatServlet (handles inbound XMPP chat messages), PresenceServlet (handles requests for the application’s presence information, and presence info for other clients) and SubscribeServlet (when other XMPP users grant the application access to their presence information).

<!-- Handles the chat part of XMPP. -->
<servlet>
    <servlet-name>Chat</servlet-name>
    <servlet-class>com.example.xmpp.ChatServlet</servlet-class>
</servlet>
<servlet-mapping>
    <servlet-name>Chat</servlet-name>
    <url-pattern>/_ah/xmpp/message/chat/</url-pattern>
</servlet-mapping>
<!-- Handles the presence part of XMPP. -->
<servlet>
    <servlet-name>Presence</servlet-name>
    <servlet-class>com.example.xmpp.PresenceServlet</servlet-class>
</servlet>
<servlet-mapping>
    <servlet-name>Presence</servlet-name>
    <url-pattern>/_ah/xmpp/presence/*</url-pattern>
</servlet-mapping>  
<!-- Handles the subscription part of XMPP. -->
<servlet>
    <servlet-name>Subscribe</servlet-name>
    <servlet-class>com.example.xmpp.SubscribeServlet</servlet-class>
</servlet>
<servlet-mapping>
    <servlet-name>Subscribe</servlet-name>
    <url-pattern>/_ah/xmpp/subscription/*</url-pattern>
</servlet-mapping>