Setting Up Sendgrid To Receive Mail

I was setting up a new application to use SendGrid’s inbound parse email function, so here’s some quick documentation. In the Sendgrid dashboard, go under Settings > Inbound Parse:

Sendgrid's settings menu holds the inbound parse option.

Then click on the top blue button: Add host & URL.

Inbound parse screen on Sendgrid. Click the top blue button to continue adding inbound options for your email.

Fill in the screen that comes up with the proper domain, and subdomain (the subdomain is optional). The destination URL is where Sendgrid will POST the email to.

At the domain registrar, set up the proper MX record. Look up the appropriate documentation based on the registrar you use – this is how it looks like on GoDaddy:

Screenshot of the proper MX record on GoDaddy.

In your application, set up a handler to answer the SendGrid request: in the screenshot example above, the handler was located at /inboundmailwebhook/. Any inbound mail gets POSTed as regular form data, which most frameworks can handle automatically.

Mail Service Error: Sender Is Not An Authorized Email Address

While using App Engine’s Mail API, some applications may encounter the following error:

This error means that the application attempted to send email with a non-whitelisted from address.

To send email from App Engine, applications must declare a sending address matching one of the following: a registered administrator of the application, the Google user account of the currently-logged-in user, or an email address of the form:

[any string]@[Application ID].appspotmail.com

For most purposes, using the appspotmail string as a from address is perfectly fine. To generate this sending address, you can use App Engine’s environment variables to collect the application ID. For example, here’s how to do it in Java:

String application_id = SystemProperty.applicationId.get();
String sender = "donotreply@" + application_id + ".appspotmail.com";

For applications that need to send email originating from their custom domain, register a Google Apps account with the address you want to use, then register it as an administrator of the application.

Setting SPF For Your Domain

Sender Policy Framework (SPF) is a way to validate outgoing mail; it essentially allows a domain to say, “only these named servers are allowed to send mail under my name; any other servers attempting to do so may be malicious or may be sending spam.” If you send mail from your domain, it’s important to set SPF rules so receiving domains know that your mail is valid and isn’t spam.

To create your SPF record, visit the SPF website and figure out the appropriate SPF record for your domain. Then place it as a TXT record in your domain’s DNS.

As an example, my domain sends no mail so the appropriate SPF record is:

v=spf1 -all

If you have NameCheap as your domain registrar, here’s how to set an SPF record. First, log in and click the link All Host Records:

Put in the following settings:

Host Name: @
IP Address: v=spf1 -all
Record Type: TXT
TTL: 1800

Here’s how it looks like on the administration console:

If you use a different domain registrar there should be similar options. If not, contact your registrar for the appropriate steps to take.

Receiving Mail In Java

Receiving email is a bit harder than sending it. First, we need to inform App Engine that this application is allowed to receive mail. In the /war/WEB-INF/ directory there is a file marked appengine-web.xml. Write the below line into that file:

<inbound-services> <service>mail</service> </inbound-services>

Now we need to map a servlet to handle all of the incoming email. Go into web.xml (in the same directory) and put in the following lines:

<servlet>
    <servlet-name>ReceiveMail</servlet-name>
    <servlet-class>com.example.ReceiveMailServlet</servlet-class>
</servlet>
<servlet-mapping>
    <servlet-name>ReceiveMail</servlet-name>
    <url-pattern>/_ah/mail/*</url-pattern>
</servlet-mapping>

This informs App Engine that there is a servlet called com.example.ReceiveMailServlet (modify the name to match your code), and it is responsible for handling all incoming email (it handles all requests directed to /_ah/mail/ which is where App Engine sends the mail).

In the servlet handling the incoming email, paste this doPost() function:

public void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException {   

try {
    Properties prop = new Properties();
    Session session = Session.getDefaultInstance(prop, null); 
    //May throw a MessagingException if incoming message is malformed.
    MimeMessage message = new MimeMessage(session, req.getInputStream());
}
catch (MessagingException e) {
    System.out.println("This message was malformed. Stopping.");
}

}//end doPost

From here, we can retrieve the from address and the subject with a few lines:

String from = message.getFrom()[0].toString();//Get the first From address listed.
String subject = message.getSubject();

The content of the email can be extracted using the getContent() function of MimeMessage.

Accessing POP3 & IMAP From AppEngine

A quick note about accessing IMAP and POP3 servers from AppEngine: you need to include the JavaMail libraries and import everything from javax.mail and javax.mail.internet.

If you need to streamline the amount of JARs bundled with your application, you can also select individual protocol provider JARs.

Here’s example code to extract emails from a POP3 store:

    Properties properties = new Properties();
    properties.put("mail.host", host);
    properties.put("mail.store.protocol", "pop3s");
    properties.put("mail.pop3s.auth", "true");
    properties.put("mail.pop3s.port", port);
    Session session = Session.getDefaultInstance(properties);
    Store store = session.getStore();
    //With a POP3 store available, connect to the given account.
    store.connect(host, user, password);
    //Open up the inbox folder, and give ourselves read/write privilegese.
    Folder folder = store.getFolder("inbox");
    folder.open(Folder.READ_WRITE);
    //Collect messages.
    Message[] messages = folder.getMessages();
    System.out.println(messages.length);
    for (int i = 0; i < messages.length; i++) {
        //Extract message.
        MimeMessage message = (MimeMessage)messages[i];

From here, you can extract the from address and the subject line from the MimeMessage. Remember that the Sockets API requires that you have billing enabled on your application.

Sending Mail With Golang

Previously I published a Java code example using the low level Mail API to send a message to the registered admins of an application. Here’s sample code for a Golang application to send mail to app admins. C stands for an appengine.Context reference.

If you want to send mail to an arbitrary user, and not an admin, you can uncomment the To line and change SendToAdmins() to Send().

application_id := appengine.AppID(c)
separation_point := strings.Index(application_id, ":")
if separation_point > -1 {
    application_id = application_id[separation_point:]
}

//Create the message struct
msg := &mail.Message{
    Sender:  "donotreply@" + application_id + ".appspotmail.com",
    //To:    []string{"To-User <[email protected]>"},
    Subject: subject,
    Body:    email_body,
}
c.Infof("Sending message: %v", msg)

//Send an email to admins
err := mail.SendToAdmins(c, msg)
if err != nil {
    c.Errorf("Unable to send email: %v", err)
}

Sending Mail With The Java Low Level API

Code example on how to send mail using the Java low level API. This code generates the appropriate From address based on your application’s ID.

MailService.Message message = new MailService.Message();
message.setSubject(subject);
message.setTextBody(body);
String application_id = SystemProperty.applicationId.get();
String sender = "donotreply@" + application_id + ".appspotmail.com";
message.setSender(sender);
MailServiceFactory.getMailService().sendToAdmins(message);

Google Compute Engine

Google’s Compute Engine just released into General Availability, and I’ve been testing it out the last couple of days.

The one thing that blows me away is how reliably fast even the low-end instances are. I provisioned and set up a f1-micro instance – it runs great and quite consistently. That’s in sharp contrast to Amazon’s micro instance which is limited to “bursty” processing; there are spikes where processing goes quickly, then the CPU gets throttled and the instance grinds to a near-halt.

I’m considering building a mail app on top of GCE – so far, everything looks great. GCE even allows inbound SMTP connections (although unfortunately no outbound SMTP connections).

Receiving Email in Golang

I’m in the middle of writing a Java application on App Engine to receive mail, and I decided to look up on how to do it in Go. It’s shockingly easy, just a few lines of code (r represents http.Request):

    c := appengine.NewContext(r)
    defer r.Body.Close()
    msg, err := mail.ReadMessage(r.Body)

And that’s it. You can extract headers and the mail message body from the Message struct. It’s quite pleasant to use, and surprisingly fast at parsing email.