live-source-gdoc

Plug-in for live-source which handles Google Docs

Provides a live-source interface to Google docs, so that you can be notified quickly (<2s) when they change. Also includes write-back capabilities to change the doc, if the installation's user account has been granted edit permission to the doc.

Strategy

At the moment we just poll the HTML-export view of the doc, every 1.5s, since the Google stated rate limit is 1.0s.

As soon as we run into some larger docs, that's going to get very painful and we should switch to polling the Drive API for the doc's last-modify time or version id.

We should also use the webhook to be notified of changes, but that only works when changes are infrequent, so we'll still need to poll for a time (to be determined experimentally -- I'm guessing 15m) after the webhook fires.

I did some experiments with running the google docs app within puppeteer, and it works in terms of detecting changes very quickly and not using a lot of bandwidth, but it uses a ton of memory on the server, which would make have many docs open at once impractical. In our particular security model, access control under puppeteer is manageable (we're not running as the end user). I was also hoping to spot the internal API the app uses to be notified of changes, but I couldn't find it. I'll be embarassed but pleased if someone can find it, or even find a documented way to get a fast feed of changes.

Auth

I don't have a deep understanding of Google authentication and Google APIs, but this seems to work.

We have three kinds of tokens which you'll have to understand if you're installing this. Google's documentation starts at https://developers.google.com/docs/api/how-tos/authorizing.

Client Credentials

As a developer, you get this file from Google when you set up a new API-using project, at https://console.developers.google.com/apis/credentials.

The steps are something like:

1. Possibly sign in using a developer account, not your usual account
2. Make a new project https://console.developers.google.com/projectcreate. End users will see the name in a few situations.
3. Enable the Drive and Docs APIs. Go to https://console.developers.google.com/apis/library and search for "drive" and "docs" and click the ENABLE button on each one.
4. Credentials >> Create Credentials >> OAuth client ID
5. Configured Consent Screen. This wont actually be seen by the public, but you need to enter stuff to get past the web form. Domain example.org seems to work. Their forms are bit weird - you need to press Enter to properly save the value of a field.
6. Application type: Web Application
7. Name doesn't matter
8. Authorized JS origins and redirect URIs: http://localhost:3000
9. Ignore the screen giving you the credentials
10. click the Download JSON link
11. Rename it to _google_client_secret.json in your working directory.

It should look something like this:

{ web:
   { client_id:
      '442101cllhn3ddupov5g4beverktnn3.apps.googleusercontent.com',
     project_id: 'my-great-new-app',
     auth_uri: 'https://accounts.google.com/o/oauth2/auth',
     token_uri: 'https://oauth2.googleapis.com/token',
     auth_provider_x509_cert_url: 'https://www.googleapis.com/oauth2/v1/certs',
     client_secret: 'thisisntmyrealsecretofcourse',
     redirect_uris:
      [ 'http://localhost:3000' ],
     javascript_origins:
      [ 'http://localhost:3000' ] } }

One User's Authorization

If you're going to use the write-back feature, you'll need edit authorization for Drive and Docs as some robot identity. This could be a service account associated with the client, or a normal user account you make for this purpose. These might look different to users when they look at their docs "share" info.

For the "normal" user account, do this, after getting client credentials above:

$ node authorize.js

This which will open a browser for you to authenticate as the chosen user and confirm authorization. Since the app isn't verified you'll have to ignore the warnings, clicking "Advanced" and "Go to (whatever you named the app)". Confirm "Allow" on both Drive and Docs, and then one more final Allow.

This should write _google_bearer_token.json, which should look something like this:

{ access_token: 'yaksdjfkasjfk;wjireuiweuriopweuriopqwueiopujkjasdklfjasdkfjakl;sdjfkl;asdjfkljasdkl;fjasl;kdjfl;kasdjfl;A',
  refresh_token: '1/sdkfjasdjf;askdjf;kasjd;fkjasd;lfjas;lkdj',
  scope: 'https://www.googleapis.com/auth/documents https://www.googleapis.com/auth/drive',
  token_type: 'Bearer',
  expiry_date: 1562733524156 } 

You should now find test-create.js works. You might want to change it to using your email address instead of mine, though.

Alternatively, back at the Google APIs console, you can create a service account credential, See https://developers.google.com/identity/protocols/OAuth2ServiceAccount and https://github.com/googleapis/google-auth-library-nodejs and then you can do:

$ export GOOGLE_APPLICATION_CREDENTIALS=_service_auth.json 
$ node servicetest.js
$ node test-service-create.js

... which uses a promise-based API, which we should probably switch to anyway, to handle renewals and such.

End User Identification

When we're modifying Google Docs on behalf of users, we need to make sure we only modify document D1 for user Alice if Alice has rights to modify D1.

We have two ways to determine this:

1. Give Alice a random string to put in the document, and see if the string appears there.

2. Have Alice go through Google Sign-In and give us the resulting token, which we can verify. That's what we do now.

Mysteries

Why does identify-user.js work for one my projects but not another? Only reason I can think of is that the one that works just uses localhost, while the other uses an https real domain before localhost. Or now it's not working for either. Weird.

Why does the user profile have different property names with one project vs another? Like totally different strings.