Forever Chat Format

Why

Because in the world where our communication is siloed into Google Hangouts, Facebook, iMessage, WhatsApp, etc etc it's getting hard to hold onto conversations for archival purposes. First thing we need is a universal format that we can export those proprietary chats into that's easy to understand, and that can be relied upon.

Basic Format

Output should be an array of hashes, and each object in the array is a JSON object defined as follows:

Chat Object Definition

Name | Type | Description ----------- | ------- | ------------- sha | string | sha1 of [sender, receiver, date, text, service] sender | string | The handle of the person sending the message receiver | string | The handle of the person receiving the message is_from_me | boolean | true if you sent the message send_error | boolean | true if the message didn't send date | ISO-8601 datetime | date message was sent/received by sender service | string | Service message was sent using. All lowercase, dasherized. 'imessage', 'google-hangouts', 'facebook-messenger', 'sms' date_read | ISO-8601 datetime | (optional) date message was sent/read by receiver date_delivered | ISO-8601 datetime | (optional) date message was delivered by receiver participants | array | (see definition below) An array of strings of handles involved in the conversations. Used for grouping. For a two person conversation, this has two entries in it. Sender and receiver are always here. message_text | string | text representation of the message message_segments | array | (see definition below) ordered segments of content as it appears in the message attachments | array | (see definition below) array of attachments associated_sha | string | sha of message this message is in response to. Generally used for reactions _debug | object | (optional) debug information to help trace back to the source of the data

Message Segment Definition

Name | Type | Possible Values | Description ----------- | ------- | ----------------- | ------------- type | string | text, link, file | base segment type file_type | string | null, or a mime_type | only valid when type == file path | string | url or file path |
text | string | | the rendered name of the link, or the text content for the text, or optionally

Text Segment

{"type": "text", "text": "Why hello there"}

Link Segment

{"type": "link", "text": "Whoa, unreal", "path": "http://latlmes.com/technology/why-we-need-a-new-chat-format-1"}

File Segment

{"type": "file", "file_type": "image/png", "path": "/path/to/image.png"}

Reaction Segment

{"type": "reaction", "reaction_type": "laughed"}

Attachments

Array of attachments in the order they were received. Each attachment has the following properties. This information should also be made available in message_segments.

Name | Type
----------- | -------
path | string
type | string

Usage

The included tests check whether some json is valid forever chat format.

var foreverJson    = importer("test.json"); // done by an exporter

var {validate, runTests} = require('forever-chat-format-tests');

runTests(foreverJson); // runs mocha tests on data

The last step of your exporter should be to pass your exported data to the prepare function. It will check that the data conforms to the forever-chat-format spec and then return a hash with two keys: messages, and validations. Messages is all the data your exporter returned, and validations are what you see below:

"validations": {
  "version":           1.2 // forever chat version
  "checkedCount":      100 // number of records checked
  "errorCount":        0   // number of records with errors
  "recordsWithErrors":
  "ruleResults": { /* each rule keyed by name with errorCount and "erroredRecords" */
    "unique-sha": {
      "description":    "Each record should have a unique sha",
      "errorCount":     0,
      "erroredRecords": []
    }

    ...
  }
}

Testing

var foreverJson   = importer("test.json"); // done by an exporter
var {validate, runTests} = require('forever-chat-format-tests');
runTests(foreverJson);