Tips, tricks, and thoughts about Google, AdWords, Google Cloud Platform, and all its subsidiaries. Not affiliated with or sponsored by Google.
This short code segment turns today’s date into a folder path. For example, today’s date of April 2, 2019 would be turned into 2019/04/02. I use it for categorizing date related documents into appropriate folders within GCS.
var datestamp_obj = new Date( (new Date()).getTime() + (-6) * 3600 * 1000 );
var datestamp_slash = datestamp_obj.getFullYear() + "/";
datestamp_slash += datestamp_obj.getMonth().toString().padStart(2, "0") + "/" + datestamp_obj.getDate().toString().padStart(2, "0");
This is more of a documentary post because I haven’t seen documentation on Google’s Timestamp class anywhere.
Google’s libraries – in particular, the GCP libraries for datastore/tasks/etc – use the Google/Protobuf/Timestamp class to represent time. Timestamp is a simple wrapper around the number of seconds since UNIX epoch, in the UTC timezone. For example here is how to create a Timestamp reflecting the current date and time, plus 2 minutes into the future (120 seconds):
use Google\Protobuf\Timestamp;
$future_time_seconds = time() + 120;
$future_timestamp = new Timestamp();
$future_timestamp->setSeconds($future_time_seconds);
$future_timestamp->setNanos(0);There are equivalent classes and functions for Python/Java/Go/other languages that Google Cloud supports.
Using the Timestamp class – especially setting up future dates – is necessary for configuring my favorite Google Cloud service: Cloud Tasks. A Cloud Task can accept a future date to run at, thereby giving you a way to queue up and delay execution of an activity. For example, see the below screenshot: I’ve created 3 tasks 20 seconds ago, yet they’re set for a future execution 3 minutes 45 seconds from now:
In a previous post, I posted a sample NodeJS function to assemble a WordPress blog slug. I ended up rewriting part of the larger application (and the function itself) in Python.
In the below function, source is a blog title string, and it returns a slug suitable for use in a blog URL.
def generate_slug(source):
i = 0
source = source.lower().strip()
allowed = "abcdefghijklmnopqrstuvwxyz1234567890"
slug = ""
while i < (len(source) - 0):
single_letter = source[i:i+1]
if single_letter in allowed:
slug += single_letter
elif not slug.endswith("-"):
#letter is not allowed
#check that the slug doesn't already end in a dash
slug += "-"
i = i + 1
return slugHere are short utility functions used to save to Google Cloud Storage in a Java App Engine application.
These snippets require the Java Cloud Storage library ( com.google.appengine.tools.cloudstorage.GcsService ). The bucket string is the name of the bucket, no gs:// required. folder_path is the subdirectory you want to save the file under. It should end with a forward slash. For example this/is/a/subdirectory/ . If you do not want a subdirectory, pass an empty string (“”). filename is the filename of the file you want to save, while contents is the data being saved.
There are two functions here. The top is intended to quickly save simple text/html/xml/etc files, while the bottom is intended to save binary files such as images, executables, and so forth. The bottom function includes a filetype argument, which should be filled with the appropriate MIME type. For example: image/png, image/jpg, application/octet-stream.
Reminder: GCS innately views files as individual objects. When you specify a subfolder, the filename for the object itself includes the folder path. When you read the file back from Google cloud storage, you need to include the folder path: for example this/is/a/subdirectory/example.png .
public static void saveStringToGCS(String bucket, String folder_path, String filename, String contents) throws IOException {
GcsService storage_service = GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance());
GcsFilename to_file = new GcsFilename(bucket, folder_path + filename);
GcsFileOptions file_options = GcsFileOptions.getDefaultInstance();
storage_service.createOrReplace(to_file, file_options, ByteBuffer.wrap(contents.getBytes(StandardCharsets.UTF_8)));
}
public static void saveByteBufferToGCS(String bucket, String folder_path, String filename, ByteBuffer contents, String filetype) throws IOException {
GcsService storage_service = GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance());
GcsFilename to_file = new GcsFilename(bucket, folder_path + filename);
GcsFileOptions file_options = (new GcsFileOptions.Builder()).mimeType(filetype).build();
storage_service.createOrReplace(to_file, file_options, contents);
}Here’s a quick example of using the Apache HTTPClient library to issue a simple GET request. The URL you’re connecting to is in endpoint, and the contents of the URL can be read from the BufferedReader response_reader.
HttpGet get_request = new HttpGet(endpoint);
get_request.addHeader(HttpHeaders.USER_AGENT, "Example Reader Service");
HttpResponse http_response = HttpClientBuilder.create().build().execute(get_request);
content_type_header = http_response.getFirstHeader("Content-Type").getValue();
int response_code = http_response.getStatusLine().getStatusCode();
if (response_code != 200) {
throw new IOException("Response code is not 200 OK. Response code: " + response_code + " : " + endpoint.toString());
}
//Pull out text response
InputStream response_is = http_response.getEntity().getContent();
BufferedReader response_reader = new BufferedReader(new InputStreamReader(response_is, "UTF-8"));The following code maps a bunch of unicode punctuation to their more normal ASCII counterparts. For example, the Unicode character LEFT DOUBLE QUOTATION MARK is mapped to the ASCII double quotation mark: “.
Hashtable<String, String> map = new Hashtable<String, String>();
map.put(new String(Character.toChars(0x00AB)), new String("\""));
map.put(new String(Character.toChars(0x00AD)), "-");
map.put(new String(Character.toChars(0x00B4)), "\'");
map.put(new String(Character.toChars(0x00BB)), "\"");
map.put(new String(Character.toChars(0x00F7)), "/");
map.put(new String(Character.toChars(0x01C0)), "|");
map.put(new String(Character.toChars(0x01C3)), "!");
map.put(new String(Character.toChars(0x02B9)), "\'");
map.put(new String(Character.toChars(0x02BA)), "\"");
map.put(new String(Character.toChars(0x02BC)), "\'");
map.put(new String(Character.toChars(0x02C4)), "^");
map.put(new String(Character.toChars(0x02C6)), "^");
map.put(new String(Character.toChars(0x02C8)), "\'");
map.put(new String(Character.toChars(0x02CB)), "`");
map.put(new String(Character.toChars(0x02CD)), "_");
map.put(new String(Character.toChars(0x02DC)), "~");
map.put(new String(Character.toChars(0x0300)), "`");
map.put(new String(Character.toChars(0x0301)), "\'");
map.put(new String(Character.toChars(0x0302)), "^");
map.put(new String(Character.toChars(0x0303)), "~");
map.put(new String(Character.toChars(0x030B)), "\"");
map.put(new String(Character.toChars(0x030E)), "\"");
map.put(new String(Character.toChars(0x0331)), "_");
map.put(new String(Character.toChars(0x0332)), "_");
map.put(new String(Character.toChars(0x0338)), "/");
map.put(new String(Character.toChars(0x0589)), ":");
map.put(new String(Character.toChars(0x05C0)), "|");
map.put(new String(Character.toChars(0x05C3)), ":");
map.put(new String(Character.toChars(0x066A)), "%");
map.put(new String(Character.toChars(0x066D)), "*");
map.put(new String(Character.toChars(0x200B)), " ");
map.put(new String(Character.toChars(0x2010)), "-");
map.put(new String(Character.toChars(0x2011)), "-");
map.put(new String(Character.toChars(0x2012)), "-");
map.put(new String(Character.toChars(0x2013)), "-");
map.put(new String(Character.toChars(0x2014)), "-");
map.put(new String(Character.toChars(0x2015)), "-");
map.put(new String(Character.toChars(0x2016)), "|");
map.put(new String(Character.toChars(0x2017)), "_");
map.put(new String(Character.toChars(0x2018)), "\'");
map.put(new String(Character.toChars(0x2019)), "\'");
map.put(new String(Character.toChars(0x201A)), ",");
map.put(new String(Character.toChars(0x201B)), "\'");
map.put(new String(Character.toChars(0x201C)), "\"");
map.put(new String(Character.toChars(0x201D)), "\"");
map.put(new String(Character.toChars(0x201E)), "\"");
map.put(new String(Character.toChars(0x201F)), "\"");
map.put(new String(Character.toChars(0x2032)), "\'");
map.put(new String(Character.toChars(0x2033)), "\"");
map.put(new String(Character.toChars(0x2034)), "\'");
map.put(new String(Character.toChars(0x2035)), "`");
map.put(new String(Character.toChars(0x2036)), "\"");
map.put(new String(Character.toChars(0x2037)), "\'");
map.put(new String(Character.toChars(0x2038)), "^");
map.put(new String(Character.toChars(0x2039)), "<");
map.put(new String(Character.toChars(0x203A)), ">");
map.put(new String(Character.toChars(0x203D)), "?");
map.put(new String(Character.toChars(0x2044)), "/");
map.put(new String(Character.toChars(0x204E)), "*");
map.put(new String(Character.toChars(0x2052)), "%");
map.put(new String(Character.toChars(0x2053)), "~");
map.put(new String(Character.toChars(0x2060)), " ");
map.put(new String(Character.toChars(0x20E5)), "\\");
map.put(new String(Character.toChars(0x2212)), "-");
map.put(new String(Character.toChars(0x2215)), "/");
map.put(new String(Character.toChars(0x2216)), "\\");
map.put(new String(Character.toChars(0x2217)), "*");
map.put(new String(Character.toChars(0x2223)), "|");
map.put(new String(Character.toChars(0x2236)), ":");
map.put(new String(Character.toChars(0x223C)), "~");
map.put(new String(Character.toChars(0x2264)), "<");
map.put(new String(Character.toChars(0x2265)), ">");
map.put(new String(Character.toChars(0x2266)), "<");
map.put(new String(Character.toChars(0x2267)), ">");
map.put(new String(Character.toChars(0x2303)), "^");
map.put(new String(Character.toChars(0x2329)), "<");
map.put(new String(Character.toChars(0x232A)), ">");
map.put(new String(Character.toChars(0x266F)), "#");
map.put(new String(Character.toChars(0x2731)), "*");
map.put(new String(Character.toChars(0x2758)), "|");
map.put(new String(Character.toChars(0x2762)), "!");
map.put(new String(Character.toChars(0x27E6)), "[");
map.put(new String(Character.toChars(0x27E8)), "<");
map.put(new String(Character.toChars(0x27E9)), ">");
map.put(new String(Character.toChars(0x2983)), "{");
map.put(new String(Character.toChars(0x2984)), "}");
map.put(new String(Character.toChars(0x3003)), "\"");
map.put(new String(Character.toChars(0x3008)), "<");
map.put(new String(Character.toChars(0x3009)), ">");
map.put(new String(Character.toChars(0x301B)), "]");
map.put(new String(Character.toChars(0x301C)), "~");
map.put(new String(Character.toChars(0x301D)), "\"");
map.put(new String(Character.toChars(0x301E)), "\"");
map.put(new String(Character.toChars(0xFEFF)), " ");
A slug in WordPress is the part of the URL that references the blog title. For example, if the URL looks like example.com/2019/this-is-my-test-post, then this-is-my-test-post would be a slug. Typically a slug is all letters and numbers, with any other character being replaced by a dash.
The below code fragment is a simple JS function to reduce a blog title down to a slug – you may want to add some additional code to guarantee a certain slug length.
var generateSlug = function generateSlug(title) {
var allowable_characters = "abcdefghijklmnopqrstuvwxyz1234567890";
//Builds our own slug
title = title.trim().toLowerCase();
var title_array = title.split("");
var outbound_title = " ";
for ( var i = 0; i < title_array.length; i++) {
var title_char = title_array[i];
if ( allowable_characters.indexOf(title_char) != -1 ) {
outbound_title = outbound_title + title_char;
}
else if ( outbound_title.split("").pop() != "-" ) {
outbound_title = outbound_title + "-";
}
}
console.log("Generated slug: " + outbound_title);
return outbound_title.trim();
}//end generateSlug
The following code example pulls out the last 5 published WordPress posts by their ID, extracts the first ID (which is the last published post ID) and then extracts the post’s tags into an array.
$return_tag_array is a list of the last published post’s tags.
//Get list of past published posts, up to 5
$args = array(
'numberposts' => 5,
'offset' => 0,
'category' => 0,
'orderby' => 'post_date',
'order' => 'DESC',
'include' => '',
'exclude' => '',
'meta_key' => '',
'meta_value' =>'',
'post_type' => 'post',
'post_status' => 'publish',
'suppress_filters' => true
);
$published_posts = wp_get_recent_posts( $args, ARRAY_A );
//Pull off the top most post, & retrieve the ID
$last_published_post = reset($published_posts);
$last_published_post_id = $last_published_post["ID"];
//TAGS
//Use the post ID to pull off tags
$tags = wp_get_post_tags($last_published_post_id);
$return_tag_array = array();
foreach ($tags as $tag) {
$tag_name = $tag->name;
array_push($return_tag_array, $tag_name);
}I’ve been writing a custom WordPress plugin to accept emailed posts and insert them into a WordPress blog. One of the difficult issues I’ve encountered is that WordPress deals with tags and categories in different ways: it’s relatively easy to add and remove tags. It takes a bit of roundabout work to do the same with categories.
The below code fragment takes a string array of categories ( $to_post_categories ) and sets them onto a post (the ID of the post is in $created_post_id ).
//Set post categories
/**
* WP doesn't allow us to set categories as easily as tags. With tags we can simply declare
* the names of the tags and be done with it. WP requires that we pass in the ID of the category,
* and if the category doesn't exist, we need to create it. So we loop through the array of
* category names, see if they exist (and if so, collect the ID) and if they don't exist,
* create the category and map the ID to the post.
**/
foreach ($to_post_categories as $category_name) {
$category_id = get_category_by_slug($category_name);
if ($category_id == false) {
//Category doesn't exist, create it
$category_id = wp_create_category($category_name);
}
//category_id now holds ID of right category or recently created category.
//Add category to post
wp_set_post_categories($created_post_id, array($category_id), true);
}//end looping through post categories sent to usHere’s a short code example in Python to iterate through a folder’s ( thisisafolder ) contents within Google Cloud Storage (GCS). Each filename can be accessed through blobi.name – in the below code sample, we print it out and test whether it ends with .json.
Remember that folders don’t actually exist on GCS, but a folder-like structure can be created by prefixing filenames with the folder name and the forward slash character ( / ).
client = storage.Client()
bucket = client.get_bucket("example-bucket-name")
blob_iterator = bucket.list_blobs(prefix="thisisafolder",client=client)
#iterate through and print out blob filenames
for blobi in blob_iterator:
print(blobi.name)
if blobi.name.endswith(".json"):
#do something with blob that ends with ".json"