Firestore Add Document – Python

A simple example of adding a document to the Firestore:

    from google.cloud import firestore
    
    #Make a note in the firestore
    firestore_client = firestore.Client()
    doc_ref = firestore_client.collection(u'CollectionsNameHere').add({
        u'propertyone': u'thisisavalue',
        u'propertytwo': 2,
    })
    logging.warning(doc_ref[1].id)

.add returns a tuple of the date that the document was written into the firestore, and a document reference to the written document. .add lets Firestore create a document ID to the written document: to figure out what the ID is, the last line accesses the tuple and pulls out the document ID.

An item that bit me: the official documentation for Firestore states that .add and .doc().set() are equivalent. That’s true, but the returns from both of those functions are different.

Documentation from Google showing .add and .doc.set are the same operation.

.add returns a tuple of the document creation time and then a reference to the added document. However, .set returns a WriteResult, not a tuple. The WriteResult contains the time the document was updated/set in the property update_time. Make sure to use the correct function as necessary.

Set Documentation

Set Documentation - Firestore
WriteResult documentation - Firestore.

Add Documentation

Add documentation - Firestore

Missing Logs – Google Logs Viewer

I opened a new GCP project to host a Python application when I hit a problem – my logging.info() and logging.warn() statements weren’t showing up in my logs. Then I realized the standard error and standard out streams weren’t selected in logging!

If you’re missing log information, make sure to select the correct streams in the second dropdown box, as in below:

Screenshot of logging, selecting stderr and stdout streams.
Screenshot of logging, selecting stderr and stdout streams.

PhantomJSCloud Error: Invalid Character ‘u’ – Python

I use PhantomJSCloud to take screenshots of web apps. While writing new code, I noticed this error coming back from the server:

{  
   "name":"HttpStatusCodeException",
   "statusCode":400,
   "message":"Error extracting userRequest. innerException: JSON5: invalid character 'u' at 1:1",
   "stack":[  
      "no stack unless env is DEV or TEST, or logLevel is DEBUG or TRACE"
   ]
}

The problem came because the request wasn’t JSON-encoding the object; the fix looks like this (using the requests library):

    post_data_string = json.dumps(python_object_here, sort_keys=True, indent=3, separators=(',', ': '))
    r = requests.post('https://phantomjscloud.com/api/browser/v2/', data=post_data_string)

Turning A Date Into A Folder Path – Node

This short code segment turns today’s date into a folder path. For example, today’s date of April 2, 2019 would be turned into 2019/04/02. I use it for categorizing date related documents into appropriate folders within GCS.

var datestamp_obj = new Date( (new Date()).getTime() + (-6) * 3600 * 1000 );
var datestamp_slash = datestamp_obj.getFullYear() + "/";
    datestamp_slash += datestamp_obj.getMonth().toString().padStart(2, "0") + "/" + datestamp_obj.getDate().toString().padStart(2, "0");
    

Using Google Protobuf Timestamp In PHP

This is more of a documentary post because I haven’t seen documentation on Google’s Timestamp class anywhere.

Google’s libraries – in particular, the GCP libraries for datastore/tasks/etc – use the Google/Protobuf/Timestamp class to represent time. Timestamp is a simple wrapper around the number of seconds since UNIX epoch, in the UTC timezone. For example here is how to create a Timestamp reflecting the current date and time, plus 2 minutes into the future (120 seconds):

use Google\Protobuf\Timestamp;

    $future_time_seconds = time() + 120;
    $future_timestamp = new Timestamp();
    $future_timestamp->setSeconds($future_time_seconds);
    $future_timestamp->setNanos(0);

There are equivalent classes and functions for Python/Java/Go/other languages that Google Cloud supports.

Using the Timestamp class – especially setting up future dates – is necessary for configuring my favorite Google Cloud service: Cloud Tasks. A Cloud Task can accept a future date to run at, thereby giving you a way to queue up and delay execution of an activity. For example, see the below screenshot: I’ve created 3 tasks 20 seconds ago, yet they’re set for a future execution 3 minutes 45 seconds from now:

Creating A WordPress Blog Slug Part 2 – Python

In a previous post, I posted a sample NodeJS function to assemble a WordPress blog slug. I ended up rewriting part of the larger application (and the function itself) in Python.

In the below function, source is a blog title string, and it returns a slug suitable for use in a blog URL.

def generate_slug(source):
    i = 0
    source = source.lower().strip()
    allowed = "abcdefghijklmnopqrstuvwxyz1234567890"
    slug = ""
    while i < (len(source) - 0):
        single_letter = source[i:i+1]
        if single_letter in allowed:
            slug += single_letter
        elif not slug.endswith("-"):
            #letter is not allowed
            #check that the slug doesn't already end in a dash
            slug += "-"
        i = i + 1
    return slug

Utility Functions To Save To Google Cloud Storage

Here are short utility functions used to save to Google Cloud Storage in a Java App Engine application.

These snippets require the Java Cloud Storage library ( com.google.appengine.tools.cloudstorage.GcsService ). The bucket string is the name of the bucket, no gs:// required. folder_path is the subdirectory you want to save the file under. It should end with a forward slash. For example this/is/a/subdirectory/ . If you do not want a subdirectory, pass an empty string (“”). filename is the filename of the file you want to save, while contents is the data being saved.

There are two functions here. The top is intended to quickly save simple text/html/xml/etc files, while the bottom is intended to save binary files such as images, executables, and so forth. The bottom function includes a filetype argument, which should be filled with the appropriate MIME type. For example: image/png, image/jpg, application/octet-stream.

Reminder: GCS innately views files as individual objects. When you specify a subfolder, the filename for the object itself includes the folder path. When you read the file back from Google cloud storage, you need to include the folder path: for example this/is/a/subdirectory/example.png .

	public static void saveStringToGCS(String bucket, String folder_path, String filename, String contents) throws IOException {
		GcsService storage_service = GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance());
		GcsFilename to_file = new GcsFilename(bucket, folder_path + filename);
		GcsFileOptions file_options = GcsFileOptions.getDefaultInstance();
		storage_service.createOrReplace(to_file, file_options, ByteBuffer.wrap(contents.getBytes(StandardCharsets.UTF_8)));
	}
	
	public static void saveByteBufferToGCS(String bucket, String folder_path, String filename, ByteBuffer contents, String filetype) throws IOException {
		GcsService storage_service = GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance());
		GcsFilename to_file = new GcsFilename(bucket, folder_path + filename);
		GcsFileOptions file_options = (new GcsFileOptions.Builder()).mimeType(filetype).build();
		storage_service.createOrReplace(to_file, file_options, contents);
	}

A HTTP GET Example Using The Apache HTTPClient Library

Here’s a quick example of using the Apache HTTPClient library to issue a simple GET request. The URL you’re connecting to is in endpoint, and the contents of the URL can be read from the BufferedReader response_reader.

		HttpGet get_request = new HttpGet(endpoint);
		get_request.addHeader(HttpHeaders.USER_AGENT, "Example Reader Service");

		HttpResponse http_response = HttpClientBuilder.create().build().execute(get_request);
		content_type_header = http_response.getFirstHeader("Content-Type").getValue();
		
		int response_code = http_response.getStatusLine().getStatusCode();

		if (response_code != 200) {
			throw new IOException("Response code is not 200 OK. Response code: " + response_code + " : " + endpoint.toString());
		}

		//Pull out text response
		InputStream response_is = http_response.getEntity().getContent();
		BufferedReader response_reader = new BufferedReader(new InputStreamReader(response_is, "UTF-8"));