Searching The Content Of A Web Page: The intext: and allintext: Search Operators

Among the less useful of operators are the intext: and allintext: search operators. As the title says, these operators require that the given word(s) show up in the content of a web page. For example, if you searched for intext:stock (no space between intext: and the searched keyword), the returned web pages would have the word stock as part of the web page:

intext:stock

Similarly, if you searched for allintext:stock dis, you would get web pages with the words stock and dis within their text content:

allintext:stock dis

While these operators are important to remember, they’re not as useful as their intitle/allintitle/inurl/allinurl counterparts. In the vast majority of cases, skipping the intext: search function and searching on the same key words would result in the same, or largely the same, search results as using the operators.

Delete Old Entities – Java Datastore

This is an ultra-simplified example of how to delete old entities from the App Engine Datastore. The first 3 lines of code retrieves the current date, then subtracts 60 days from the current time (the multiplication converts days to milliseconds). DATE_PROPERTY_ON_ENTITY is the date property on the entity – when first writing the entity to the datastore, add the current date as a property. ENTITY_KIND is the entity kind we’re deleting.

		//Calculate 60 days ago.
		long current_date_long = (new Date()).getTime();
		long past_date_long = current_date_long - (1000 * 60 * 60 * 24 * 60);
		Date past_date = new Date(past_date_long);
		
		DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
		Query.Filter date_filter = new Query.FilterPredicate("DATE_PROPERTY_ON_ENTITY", Query.FilterOperator.LESS_THAN_OR_EQUAL, past_date);
		Query date_query = new Query("ENTITY_KIND").setFilter(date_filter);
		PreparedQuery date_query_results = datastore.prepare(date_query);
		
		Iterator<Entity> iterate_over_old_entities = date_query_results.asIterator();
		
		while (iterate_over_old_entities.hasNext()) {
			Entity old_entity = iterate_over_old_entities.next();
			
			System.out.println("Deleting: " + old_entity.getProperties());
			
			datastore.delete(old_entity.getKey());
		}

Note that is a simplified function – it’s useful if you have a handful of entities that need deleting, but if you have more than a handful, you should convert to using datastore cursors and paging through entities to delete.

PHP Post To PubSub

Today is a rather large fragment demonstrating how to post to Google PubSub. While there are libraries to handle this, I prefer to understand the low-level process so debugging is easier.

Note that this fragment is designed to run on App Engine, as it relies on the App Identity service to pull the credentials required to publish to PubSub. You only need to set up 3 variables: $message_data, which should be a JSON-encodable object, NAMEOFGOOGLEPROJECT, which is the name of the Google project containing the pubsub funnel you want to publish to, and NAMEOFPUBSUB which is the pubsub funnel name.

It isn’t required, but it is good practice to customize the User-Agent header below. I have it set to Publisher, but a production service should have it set to an appropriate custom name.

use google\appengine\api\app_identity\AppIdentityService;

//Build JSON object to post to Pubsub

$message_data_string = base64_encode(json_encode($message_data));

$single_message_attributes = array ("key" => "iana.org/language_tag",
    "value" => "en",
);

$single_message = array ("attributes" => $single_message_attributes,
    "data" => $message_data_string,
);
$messages = array ("messages" => $single_message);

//Post to Pubsub

$url = 'https://pubsub.googleapis.com/v1/projects/NAMEOFGOOGLEPROJECT/topics/NAMEOFPUBSUB:publish';

$pubsub_data = json_encode($messages);

syslog(LOG_INFO, "Pubsub Message: " . $pubsub_data);

$access_token = AppIdentityService::getAccessToken('https://www.googleapis.com/auth/pubsub');

$headers = "accept: */*\r\n" .
    "Content-Type: text/json\r\n" .
    "User-Agent: Publisher\r\n" .
    "Authorization: OAuth " . $access_token['access_token'] . "\r\n" .
    "Custom-Header-Two: custom-value-2\r\n";

$context = [
    'http' => [
        'method' => 'POST',
        'header' => $headers,
        'content' => $pubsub_data,
    ]
];
$context = stream_context_create($context);
$result = file_get_contents($url, false, $context);

syslog(LOG_INFO, "Returning from PubSub: " . $result);

Serializing A Java Object To Google Cloud Storage – Java

A quick code example today: serializing a Java object to Google Cloud storage. write_object stands for the object being written. This code depends on the App Engine libraries for Java, and the Google Cloud Storage libraries.

	GcsService gcs_service = GcsServiceFactory.createGcsService();
	GcsFilename gcs_filename = new GcsFilename(AppIdentityServiceFactory.getAppIdentityService().getDefaultGcsBucketName(), "subfolder_within_bucket" + "/" + "filename.extension");
	GcsFileOptions gcs_options = GcsFileOptions.getDefaultInstance();
	GcsOutputChannel output = gcs_service.createOrReplace(gcs_filename, gcs_options);
	ObjectOutputStream oos = new ObjectOutputStream(Channels.newOutputStream(output));
	oos.writeObject(write_object);
	oos.flush();
	oos.close();
	output.close();

Listening To Music Offline With Google Drive

Do you need to transfer text/music/pictures from your desktop/laptop PC to your iPhone? Do you need these files available to look/listen to even when your iPhone can’t get a signal?

I frequently need to transfer audio files/music from my laptop and listen to them on my iPhone, even in areas that don’t have cell reception. Fortunately, Google Drive offers the ability to mark files as available offline – to download the files to the iPhone’s local memory so they’re available at all times.

To do this, first use your PC to upload files to Google Drive. Then on the iPhone, open up the Google Drive app, find the audio file, and click on the three period symbol (inside the purple box) below:

Google Drive iOS app. Click on the three dot symbol.

A context menu will pop up below:

Google Drive file context menu.

Use your finger to pull the menu up (towards the top of your phone). You’ll see the full menu. Where it says Available Offline, pull the switch to the right until you see blue.

Use the available offline switch to mark the file as available at all times.

You’re done! A prompt should show up, where Drive acknowledges the offline request:

The notice that pops up when the file will be available offline.

To make sure the file is fully downloaded, leave the Google Drive app open a moment before you close it out.

DisallowedHost – Invalid HTTP_HOST Header “example.appspot.com”

When installing a new Django app on App Engine, I forgot to set the hosts header, and found this error:



The problem here is that I didn’t set the allowed hosts header in the settings file. To fix this, go to settings.py file and set ALLOWED_HOSTS to list a wildcard, like so:

ALLOWED_HOSTS = ['*']

You can see this replicated in Google’s own Django example: https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/appengine/standard/django/mysite/settings.py#L46 (see screenshot below).

A screenshot of allowed_hosts including a wildcard.

Warning: A wildcard is only acceptable because App Engine’s other security policies are protecting the application. If you move the Django site to another host, make sure you change out the wildcard to the appropriate domain/domains.

Golang Users API Example

Here’s an example of how to use the Users API in Go. This example checks to see if the current user is logged in; if not, the user is given a link to log in. If the user is already logged in, then it creates and prints a link to log out.

//Creates an App Engine Context - required to access App Engine services.
c := appengine.NewContext(r)
//Acquire the current user
user := appengineuser.Current(c)
if user == nil {
    url, _ := appengineuser.LoginURL(c, "/")
    fmt.Fprintf(w, `<a href="%s">Sign in</a>`, url)
} else {
    url, _ := appengineuser.LogoutURL(c, "/")
    fmt.Fprintf(w, `Welcome, %s! (<a href="%s">sign out</a>)`, user, url)
}

Searching For User Request Logs In App Engine

You can search for specific users by using the filter labels function of App Engine logging. Here’s a short example: the following filter searches for all users matching the substring inny :

The search finds the following request logs, where the user vinnyapp (my Gmail account) has logged in:

Error Parsing YAML File: Mapping Values Are Not Allowed Here

An improperly configured YAML file may show the error Error Parsing YAML File: Mapping Values Are Not Allowed Here . This error is demonstrated below:

Here is an example YAML file that causes this error:

application:application-id
version:1
runtime:php
api_version:1
threadsafe:true
#Error happens at line 7 below, even though the incorrect lines are above.
handlers:
- url: /example

Even though the YAML parser reports the error at line 7, the actual incorrect lines are above that point: lines 1 – 5 are missing the space character between the colon and the value. If you encounter this error, make sure that the key: value pairs are separated by 1 colon and 1 space character, as shown below:

application: application-id

Mail Service Error: Sender Is Not An Authorized Email Address

While using App Engine’s Mail API, some applications may encounter the following error:

This error means that the application attempted to send email with a non-whitelisted from address.

To send email from App Engine, applications must declare a sending address matching one of the following: a registered administrator of the application, the Google user account of the currently-logged-in user, or an email address of the form:

[any string]@[Application ID].appspotmail.com

For most purposes, using the appspotmail string as a from address is perfectly fine. To generate this sending address, you can use App Engine’s environment variables to collect the application ID. For example, here’s how to do it in Java:

String application_id = SystemProperty.applicationId.get();
String sender = "donotreply@" + application_id + ".appspotmail.com";

For applications that need to send email originating from their custom domain, register a Google Apps account with the address you want to use, then register it as an administrator of the application.