Gmailpush

Gmailpush is Node.js library for handling Gmail API push notifications using Google APIs Node.js Client.

Note: Gmailpush is not affiliated with Gmail.

Features

• Fields such as from, to, subject and body parsed from original message's payload
• Filter by types of history, e.g. messageAdded, labelRemoved
• Filter by label ids, e.g. INBOX, UNREAD
• Automatic renewal of mailbox watch request
• Uses JSON file to store each user's Gmail API history id and watch request expiration

Prerequisites

Gmail API

• OAuth2 client ID and client secret (how to get)
• Access token for user's Gmail data (how to get, quickstart)

Google Cloud Pub/Sub

• Pub/Sub topic and subscription (how to create) with push endpoint URL being set (how to set)

Installation

npm

$ npm install gmailpush

yarn

$ yarn add gmailpush

Example

Request

const express = require('express');
const Gmailpush = require('gmailpush');

const app = express();

// Initialize with OAuth2 config and Pub/Sub topic
const gmailpush = new Gmailpush({
  clientId: '12345abcdefg.apps.googleusercontent.com',
  clientSecret: 'hijklMNopqrstU12vxY345ZA',
  pubsubTopic: 'projects/PROJECT_NAME/topics/TOPIC_NAME'
});

const users = [
  {
    email: '[email protected]',
    token: {
      access_token: 'ABcdefGhiJKlmno-PQ',
      refresh_token: 'RstuVWxyzAbcDEfgh',
      scope: 'https://www.googleapis.com/auth/gmail.readonly',
      token_type: 'Bearer',
      expiry_date: 1543210123451
    }
  }
];

app.post(
  // Use URL set as Pub/Sub Subscription endpoint
  '/pubsub-push-endpoint',
  // Parse JSON request payload
  express.json(),
  (req, res) => {
    // Acknowledge Gmail push notification webhook
    res.sendStatus(200);

    // Get Email address contained in the push notification
    const email = gmailpush.getEmailAddress(req.body);

    // Get access token for the Email address
    const token = users.find((user) => user.email === email).token;

    gmailpush
      .getMessages({
        notification: req.body,
        token
      })
      .then((messages) => {
        console.log(messages);
      })
      .catch((err) => {
        console.log(err);
      });
  }
);

app.listen(3000, () => {
  console.log('Server listening on port 3000...');
});

Response

[
  {
    id: 'fedcba9876543210',
    threadId: 'fedcba9876543210',
    labelIds: [ 'CATEGORY_PERSONAL', 'INBOX', 'UNREAD', 'IMPORTANT' ],
    snippet: 'this is body',
    historyId: '987654321',
    historyType: 'labelAdded',
    internalDate: '1546300800000',
    date: 'Tue, 1 Jan 2019 00:00:00 +0000',
    from: { name: 'user', address: '[email protected]' },
    to: [ { name: 'user1', address: '[email protected]' } ],
    subject: 'this is subject',
    bodyText: 'this is body\r\n',
    bodyHtml: '<div dir="ltr">this is body</div>\r\n',
    attachments: [
      {
        mimeType: 'image/jpeg',
        filename: 'example.jpg',
        attachmentId: 'abcdef0123456789',
        size: 2,
        data: <Buffer ff ff ff ff>
      }
    ],
    payload: {
      partId: '',
      mimeType: 'multipart/alternative',
      filename: '',
      headers: [Array],
      body: [Object],
      parts: [Array]
    },
    sizeEstimate: 4321
  }
]

Initialization

new Gmailpush(options)

Usage

const Gmailpush = require('gmailpush');

const gmailpush = new Gmailpush({
  clientId: 'GMAIL_OAUTH2_CLIENT_ID',
  clientSecret: 'GMAIL_OAUTH2_CLIENT_SECRET',
  pubsubTopic: 'GMAIL_PUBSUB_TOPIC',
  prevHistoryIdFilePath: 'gmailpush_history.json'
});

options object

clientId (required) string

Gmail API OAuth2 client ID. Follow this instruction to create.

clientSecret (required) string

Gmail API OAuth2 client secret. Follow this instruction to create.

pubsubTopic (required) string

Google Cloud Pub/Sub API's topic. Value should be provided as 'projects/PROJECT_NAME/topics/TOPIC_NAME'. Used to call watch(). Follow this instruction to create.

prevHistoryIdFilePath string

File path for storing emailAddress, prevHistoryId and watchExpiration.

• emailAddress: Email address of a user for whom Gmail push notification messages are sent.

• prevHistoryId: Gmail API's push notification messages are not real messages but contain historyId which is the latest history id as of the time they are sent. To retrieve real messages, one needs to request for history of changes to the user's mailbox since a certain history id. But historyId in the push notification message cannot be used for that certain history id because it is the latest one after which no changes have been made. So Gmailpush stores historyId from the push notification message for later use when next push notification message is received. Similarly the first push notification since installing Gmailpush could not be turned into messages but an empty array because the history id used for the first getMessages() is the latest one.

• watchExpiration: Google Cloud Pub/Sub API requires calling watch() at least every 7 days. Otherwise push notification will be stopped. Whenever Gmailpush is initialized, it calls watch() to extend expiration for 7 days. And Gmailpush stores watch expiration so that schedulers like node-schedule can use it to call watch() before expiration.

Methods like getMessages(), getMessagesWithoutAttachment() and getNewMessage will automatically create a file using prevHistoryIdFilePath if the file doesn't exist.

Default is 'gmailpush_history.json' and its content would be like:

[
  {
    "emailAddress": "[email protected]",
    "prevHistoryId": 9876543210,
    "watchExpiration": 1576543210
  },
  {
    "emailAddress": "[email protected]",
    "prevHistoryId": 1234567890,
    "watchExpiration": 1576543211
  }
]

API

getMessages(options)

Gets Gmail messages which have caused change to history since prevHistoryId which is the historyId as of the previous push notification and is stored in prevHistoryIdFilePath.

Messages can be filtered by options. For example, messages in the following usage will be an array of messages that have INBOX label in their labelIds and have added IMPORTANT label to their labelIds.

The first call of this method for a user will result in an empty array as returned value and store prevHistoryId in gmailpush_history.json. (also create the file if not exists)

When a Gmail user is composing a new message, every change the user has made to the draft (even typing a single character) causes two push notifications, i.e. messageDeleted type for deletion of the last draft and messageAdded type for addition of current draft.

Usage

const messages = await gmailpush.getMessages({
  notification: req.body,
  token,
  historyTypes: ['labelAdded'],
  addedLabelIds: ['IMPORTANT'],
  withLabelIds: ['INBOX']
});

options object

notification (required) object

An object which is JSON-parsed from Gmail API's push notification message. Push notification messages should be JSON-parsed using JSON.parse() or middleware like express.json() or body-parser before passed to notification.

Below is an example notification object:

{
  message: {
    // This is the actual notification data, as base64url-encoded JSON.
    data: 'eyJlbWFpbEFkZHJlc3MiOiJ1c2VyMUBnbWFpbC5jb20iLCJoaXN0b3J5SWQiOiI5ODc2NTQzMjEwIn0=',
    // This is a Cloud Pub/Sub message id, unrelated to Gmail messages.
    message_id: '1234567890',
  },
  subscription: 'projects/PROJECT_NAME/subscriptions/SUBSCRIPTION_NAME'
}

And JSON.parse(Buffer.from(notification.message.data, 'base64').toString()) would result in the following object:

{
  emailAddress: '[email protected]',
  historyId: '9876543210'
}

token (required) object

Gmail API OAuth2 access token for user's Gmail data which has the following form:

{
  access_token: 'USER1_ACCESS_TOKEN',
  refresh_token: 'USER1_REFRESH_TOKEN',
  scope: 'USER1_SCOPE',
  token_type: 'USER1_TOKEN_TYPE',
  expiry_date: 'USER1_EXPIRY_DATE'
}

historyTypes string[]

Specifies which types of change to history this method should consider. There are four types of change.

• messageAdded: Messages have been added to mailbox, e.g. sending or receiving an Email.
• messageDeleted: Messages have been deleted from mailbox, e.g. deleting an Email in Trash.
• labelAdded: Messages have added label ids, e.g. TRASH would be added when delete an Email.
• labelRemoved: Messages have removed label ids, e.g. UNREAD would be removed when read an unread Email.

Elements in historyTypes will be OR-ed. Default is ['messageAdded', 'messageDeleted', 'labelAdded', 'labelRemoved'].

addedLabelIds string[]

Used with labelAdded history type to specify which added label ids to monitor. Elements will be OR-ed. If not provided, Gmailpush won't filter by addedLabelIds. User-generated labels have label ids which don't match their label names. To get label id for user-generated label, use getLabels().

removedLabelIds string[]

Used with labelRemoved history type to specify which removed label ids to monitor. Elements will be OR-ed. If not provided, Gmailpush won't filter by removedLabelIds. User-generated labels have label ids which don't match their label names. To get label id for user-generated label, use getLabels().

withLabelIds string[]

Specifies which label ids should be included in labelIds of messages this method returns. Elements will be OR-ed. If not provided, Gmailpush won't filter by withLabelIds. withLabelIds would filter out any messages with messageDeleted type of history because they don't have labelIds. User-generated labels have label ids which don't match their label names. To get label id for user-generated label, use getLabels(). withLabelIds and withoutLabelIds cannot contain the same label id.

withoutLabelIds string[]

Specifies which label ids should not be included in labelIds of messages this method returns. Elements will be OR-ed. If not provided, Gmailpush won't filter by withoutLabelIds. withoutLabelIds would not filter out messages with messageDeleted type of history because they don't have labelIds to be filtered. User-generated labels have label ids which don't match their label names. To get label id for user-generated label, use getLabels(). withLabelIds and withoutLabelIds cannot contain the same label id.

Return object[]

An array of message objects. If there is no message objects that satisfy criteria set by options, an empty array will be returned.

Gmail API sends push notifications for many reasons of which some are not related to the four history types, i.e. messageAdded, messageDeleted, labelAdded and labelRemoved. In those cases this method will return an empty array.

If prevHistoryId for a user doesn't exist in gmailpush_history.json, calling this method for the user will result in an empty array.

If the messages have attachments, data of the attachments is automatically fetched and appended as Buffer instance. Alternatively you can use getMessagesWithoutAttachment() which returns messages without attachment data.

For messages that would have been deleted before requested, return value for those messages would have no material properties and look like this:

{
  id: 'fedcba9876543210',
  historyType: 'messageDeleted',
  notFound: true, // Indicates that Gmail API has returned "Not Found" or "Requested entity was not found." error
  attachments: [] // Exists only for internal purpose
}

In message object, from, to, cc, bcc, subject, date, bodyText and bodyHtml are present only when original message has them.

If parsing originator/destination headers like From, To, Cc and Bcc has failed, raw values will be assigned to from, to, cc and bcc, respectively. For example, value of To header in the message.payload.headers seems to be truncated if it has more than a certain number (about 9,868) of characters. In that case, the last one in the list of recipient Email addresses might look like the following and not be parsed:

// message.payload.headers:
[
  {
    name: 'To',
    value: '[email protected] <[email protected]>, user2@'
  }
]

// message.to:
[
  {
    name: '[email protected]',
    address: '[email protected]'
  },
  {
    name: 'user2@',
    address: 'user2@'
  }
]

From header can have multiple Email addresses theoretically. But Gmailpush assumes that From header has a single Email address.

getMessagesWithoutAttachment(options)

Same as getMessages() except that elements of attachments don't have data.

Usage

const messages = await gmailpush.getMessagesWithoutAttachment({
  notification: req.body,
  token,
  historyTypes: ['messageAdded']
});

Options object

Same as those of getMessages().

Return object[]

Same as that of getMessages() except that elements of attachments don't have data.

Example attachments in return

[
  {
    mimeType: 'image/jpeg',
    filename: 'example.jpg',
    attachmentId: 'abcdef0123456789',
    size: 2
  }
]

getAttachment(message, attachment)

Gets attachment data as Node.js Buffer. getMessages() is actually wrapper of getMessagesWithoutAttachment() and getAttachment().

message object

Message object returned by getMessagesWithoutAttachment(), of which id will be used to call gmail.users.messages.attachments.get().

attachment object

Attachment object in the above message object, of which attachmentId will be used to call gmail.users.messages.attachments.get().

Return Buffer

Buffer instance of attachment data.

getNewMessage(options)

Gets only a new Email received at inbox. This method is implementation of getMessages() with the following options:

{
  historyTypes: ['messageAdded'],
  withLabelIds: ['INBOX'],
  withoutLabelIds: ['SENT']
}

Usage

const message = await gmailpush.getNewMessage({
  notification: req.body,
  token
});

options object

notification (required) object

Same as that of getMessages().

token (required) object

Same as that of getMessages().

Return object | null

Message object which is the first element of array returned by getMessages(). Gmailpush assumes that the array is either one-element or empty array. If there is no message object that satisfies criteria set by options, null will be returned.

getEmailAddress(notification)

Gets Email address from a push notification.

Usage

const email = gmailpush.getEmailAddress(req.body);

notification (required) object

Same as that of getMessages().

Return string

Email address.

Example return

'[email protected]'

getLabels(notification, token)

Gets a list of labels which can be used to find label ids for user-generated labels because user-generated labels' id is not same as their name.

Usage

const labels = await gmailpush.getLabels(req.body, token);

notification (required) object

Same as that of getMessages().

token (required) object

Same as that of getMessages().

Return object[]

Array of label objects.

Example return

[
  {
    id: 'INBOX',
    name: 'INBOX',
    messageListVisibility: 'hide',
    labelListVisibility: 'labelShow',
    type: 'system'
  },
  {
    id: 'Label_1',
    name: 'Invoice',
    messageListVisibility: 'show',
    labelListVisibility: 'labelShow',
    type: 'user',
    color: { textColor: '#222222', backgroundColor: '#eeeeee' }
  }
]

License

MIT