From the “my blog is actually my code backup store” department, this is a simple function I use on Google Cloud to accept a base64-encoded zip file, then unzip it all to a Google Cloud Storage bucket.
import functions_framework
import base64
from google.cloud import storage
import zipfile
import os
import datetime, pytz
@functions_framework.http
def hello_http(request):
#pull out the base64 encoded data and decode it.
request_bytes = base64.b64decode(request.get_data())
print(len(request_bytes))
print(request.data)
print(request.content_length)
print(request.method)
print(request.method)
print("request load: " + str(len(request_bytes)))
#generate working directory prefix name
datetime_string = datetime.datetime.now(pytz.timezone("US/Central")).isoformat()
print("Current Chicago Date-Time: %s" % (datetime_string))
directory_prefix = datetime_string[:19].replace(":", "-")
#dump to GCS
gcs_client = storage.Client()
gcs_bucket = gcs_client.get_bucket("bucket name goes here")
file_blob = storage.Blob("/zipped/" + directory_prefix + ".zip", gcs_bucket)
file_blob.upload_from_string(request_bytes, content_type="application/zip", client=gcs_client)
#dump to temporary directory within functions
with open("/tmp/" + directory_prefix + ".zip", "wb") as file:
file.write(request_bytes)
file.flush()
file.close()
#extract zipfile
zip_file = zipfile.ZipFile("/tmp/" + directory_prefix + ".zip")
zip_file.extractall("/tmp/local/unzip/" + directory_prefix + "/")
#go through extracted files
for file_name in os.listdir("/tmp/local/unzip/" + directory_prefix + "/"):
print(str(file_name))
file_blob = storage.Blob("/open/" + directory_prefix + "/" + file_name + "", gcs_bucket)
file_blob.upload_from_filename("/tmp/local/unzip/" + directory_prefix + "/" + file_name + "", client=gcs_client)
print("end")
return str(len(request_bytes))
You may want to alter the date reference (the US/Central note) but otherwise it’s a small and efficient tool for moving data where I can easily reference it later by date.
I always love pointing out fun errors – where I define “fun error” as an error that is intermittent/happens rarely in the context of regular operation in the application. Those errors are always the most fun to fix.
Today I was poking through some of my toy applications running on Google Cloud when I saw this:
And the text only:
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
._end_unary_response_blocking ( /layers/google.python.pip/pip/lib/python3.7/site-packages/grpc/_channel.py:1003 ) - Jan 23, 2024 22 hours ago -
DeadlineExceeded: 504 Deadline Exceeded
.error_remapped_callable ( /layers/google.python.pip/pip/lib/python3.7/site-packages/google/api_core/grpc_helpers.py:81 ) - Jan 23, 2024 22 hours ago
Hmm – so an error occurred 22 hours ago, that last reoccurred almost 4 months ago (Jan 23, 2024). Doesn’t sound very important. But just for the laughs, let’s dig in.
Of the two errors, I know that the first one (InactiveRPCError) is most likely due to a connection being unable to complete. Not a giant problem, happens all the time in the cloud – servers get rebooted, VMs get shuffled off to another machine, etc. Not a serious problem. The deadline exceeded one concerns me though because I know this application connects to a bunch of different APIs and does a ton of time consuming operations, and I want to make sure that everything is back to normal.
So here’s the view of the error page:
So I know that the error is somewhere communicating with Google services since the error pops up in the google api core library. Let’s hop on over to logging and find the stack trace – I’ve redacted a line that doesn’t mean anything to the purpose of this post:
Traceback (most recent call last):
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/app.py", line 2070, in wsgi_app
response = self.full_dispatch_request()
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/app.py", line 1515, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/app.py", line 1513, in full_dispatch_request
rv = self.dispatch_request()
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/app.py", line 1499, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
[REDACTED]
File "/srv/main.py", line 331, in launch_task
task_creation_results = client.create_task(parent=queue_prop, task=task)
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/cloud/tasks_v2/services/cloud_tasks/client.py", line 2203, in create_task
metadata=metadata,
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
return wrapped_func(*args, **kwargs)
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
return func(*args, **kwargs)
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 81, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded
If you missed the culprit in the above text, let me help you out: the call to the Google Task Queue service on line 331 of my application ended up exceeding Google’s deadline, and threw up the exception from Google’s end. Perhaps there was a transient infrastructure issue, perhaps task queue was under maintenance, perhaps it was just bad luck.
File "/srv/main.py", line 331, in launch_task
task_creation_results = client.create_task(parent=queue_prop, task=task)
There’s really nothing to be done here, other than maybe catching the exception and trying again. I will point out that the task queue service is surprisingly resilient: out of tens/hundreds of thousands of task queue calls over the past 5 months that this application has performed, only 2 tasks (one in January 2024, one yesterday) have failed to enqueue. More importantly, my code is functioning as intended and I can mark this issue as Resolved or at least Muted.
Now honestly, this is a my bad sort of situation. If this was a real production app there should be something catching the exception. But since this is a toy application, I absolutely am fine tolerating the random and thankfully very rare failures in task queue.
This is more of a documentary post because I haven’t seen documentation on Google’s Timestamp class anywhere.
Google’s libraries – in particular, the GCP libraries for datastore/tasks/etc – use the Google/Protobuf/Timestamp class to represent time. Timestamp is a simple wrapper around the number of seconds since UNIX epoch, in the UTC timezone. For example here is how to create a Timestamp reflecting the current date and time, plus 2 minutes into the future (120 seconds):
use Google\Protobuf\Timestamp;
$future_time_seconds = time() + 120;
$future_timestamp = new Timestamp();
$future_timestamp->setSeconds($future_time_seconds);
$future_timestamp->setNanos(0);
There are equivalent classes and functions for Python/Java/Go/other languages that Google Cloud supports.
Using the Timestamp class – especially setting up future dates – is necessary for configuring my favorite Google Cloud service: Cloud Tasks. A Cloud Task can accept a future date to run at, thereby giving you a way to queue up and delay execution of an activity. For example, see the below screenshot: I’ve created 3 tasks 20 seconds ago, yet they’re set for a future execution 3 minutes 45 seconds from now:
Today is a rather large fragment demonstrating how to post to Google PubSub. While there are libraries to handle this, I prefer to understand the low-level process so debugging is easier.
Note that this fragment is designed to run on App Engine, as it relies on the App Identity service to pull the credentials required to publish to PubSub. You only need to set up 3 variables: $message_data, which should be a JSON-encodable object, NAMEOFGOOGLEPROJECT, which is the name of the Google project containing the pubsub funnel you want to publish to, and NAMEOFPUBSUB which is the pubsub funnel name.
It isn’t required, but it is good practice to customize the User-Agent header below. I have it set to Publisher, but a production service should have it set to an appropriate custom name.
When installing a new Django app on App Engine, I forgot to set the hosts header, and found this error:
The problem here is that I didn’t set the allowed hosts header in the settings file. To fix this, go to settings.py file and set ALLOWED_HOSTS to list a wildcard, like so:
A screenshot of allowed_hosts including a wildcard.
Warning: A wildcard is only acceptable because App Engine’s other security policies are protecting the application. If you move the Django site to another host, make sure you change out the wildcard to the appropriate domain/domains.
Here’s an example of how to use the Users API in Go. This example checks to see if the current user is logged in; if not, the user is given a link to log in. If the user is already logged in, then it creates and prints a link to log out.
//Creates an App Engine Context - required to access App Engine services.
c := appengine.NewContext(r)
//Acquire the current user
user := appengineuser.Current(c)
if user == nil {
url, _ := appengineuser.LoginURL(c, "/")
fmt.Fprintf(w, `<a href="%s">Sign in</a>`, url)
} else {
url, _ := appengineuser.LogoutURL(c, "/")
fmt.Fprintf(w, `Welcome, %s! (<a href="%s">sign out</a>)`, user, url)
}
You can search for specific users by using the filter labels function of App Engine logging. Here’s a short example: the following filter searches for all users matching the substring inny :
The search finds the following request logs, where the user vinnyapp (my Gmail account) has logged in:
An improperly configured YAML file may show the error Error Parsing YAML File: Mapping Values Are Not Allowed Here . This error is demonstrated below:
Here is an example YAML file that causes this error:
application:application-id
version:1
runtime:php
api_version:1
threadsafe:true
#Error happens at line 7 below, even though the incorrect lines are above.
handlers:
- url: /example
Even though the YAML parser reports the error at line 7, the actual incorrect lines are above that point: lines 1 – 5 are missing the space character between the colon and the value. If you encounter this error, make sure that the key: value pairs are separated by 1 colon and 1 space character, as shown below:
While using App Engine’s Mail API, some applications may encounter the following error:
This error means that the application attempted to send email with a non-whitelisted from address.
To send email from App Engine, applications must declare a sending address matching one of the following: a registered administrator of the application, the Google user account of the currently-logged-in user, or an email address of the form:
[any string]@[Application ID].appspotmail.com
For most purposes, using the appspotmail string as a from address is perfectly fine. To generate this sending address, you can use App Engine’s environment variables to collect the application ID. For example, here’s how to do it in Java:
For applications that need to send email originating from their custom domain, register a Google Apps account with the address you want to use, then register it as an administrator of the application.
This exception crops up whenever an app file (in this case a JSP file) is currently being accessed by another program. If you see this exception, double check to ensure that the named file isn’t being accessed by another program.
If this error persists, close down and reopen Eclipse and the development app server – the file may have been left open from a previous run.
