River5

River5 is a river-of-news RSS aggregator in JavaScript running in Node, written by Dave Winer.

How River5 works

When it starts up, River5 reads files in the lists folder at the top level of the River5 folder.

Each list contains a set of URLs of feeds. The list files can be straight text, a JSON array of URLs of feeds, or a traditional OPML subscription list, with extensions .txt, .json and .opml. River5 ignores files in the lists folder that are not in one of these formats.

The feeds pointed to from the lists can be in RSS 0.9x, 1.0 or 2.0, or Atom 1.0.

Every 15 minutes, River5 reads all the feeds you're subscribed to, and routes new items to corresponding files in the rivers folder at the top level of the River5 folder. You're subscribed to a feed if it appears in one or more of your lists. The river files are a documented form of JSONP that we've been using since River3, in 2010.

To read the output of River5 on the machine it's running on, go to the home page. Assuming your river is running on the default port, 1337, you would go to this address to read the rivers. http://localhost:1337/. If it's running on a public server, just replace localhost with the domain name or IP address of the server.

Pretty much everything in this narrative is configurable.

Requirements

A current Node.js installation.

The machine running River5 ahould able to access the Internet.

It is possible to run the software on a non-public machine. Of course, you won't be able to access the home page remotely, or receive realtime notifications of feed updates. You'll also have to find a way to upload the output files in the rivers folder to a place where they can be accessed, if you want others to be able to access your river.

If you don't have Node.js installed on your machine, here are step by step instructions for installing Node.js on a Macintosh.

How to install

1. Download the folder from the repository to your local computer.

2. From the command line, go to this folder.

3. npm install

4. node river5.js

If it's working properly, you should see tons of stories scroll by in the console. River5 is reading each of the feeds in the lists for the first time. When you see things settle down, then you should open the home page of the RIver5 website, at http://localhost:1337/ and see the result of the first feed readings.

At first there will be a lot of old stories, but as the rivers run for hours and days and into weeks, you'll get the flow. It's a good way to read news.

Examples of lists

I have included several files in the lists folder to help you get started, so something actually happens the first time you run River5. You can edit, consolidate or delete them, as you wish.

1. myJsonFeeds.json -- a list of URLs of feeds in the JSON format that River5 understands.

2. myOpmlFeeds.opml -- a list of URLs of feeds in the standard OPML format for subscription lists.

3. myTxtFeeds.txt -- a list of URLS in a text file.

4. nyt.opml -- a list of a New York Times news feeds.

5. hn.opml -- a list with just one feed in it, the Hacker News firehose. It's a good list for testing RIver5 because it updates so frequently.

To view your rivers

Go to http://localhost:1337/.

Configuring

There's only one way to configure it, via the config.json file in the same folder as the river5.js app.

The example config.json in the folder sets the max number of items in a river to 300.

All of the config.json options and their default values are listed on this page.

For River4 users

River5 does not support S3 storage, so if that's how you're running your rivers, you should continue to use River4.

1. If you're running out of the filesystem, then River5 can replace River4.

2. Follow the installation instructions above, and before you launch river5.js, remove the files from the lists folder, and replace them with the lists from your River4 installation.

3. It will take a while before your rivers re-populate, so I recommend doing it in parallel with your existing River4 install.

You may want to review the Configuration settings page, there are options for setting new things, and in some cases the defaults have changed.

Where's the code?

The heart of River5 is in a Node package in the lib folder, called davereader.

To upgrade to the latest version of the software npm update at the command line in the river5 folder.

river5.js itsef is a very small file and by design almost never changes.

Examples of working rivers

1. Podcatch.com subscribes to my friends' podcast feeds.

2. mlbriver.com and nbariver.com collect news for two sports I love.

3. I have a River panel on the Scripting News home page.

4. My rivers page puts all the news I care about on a single tabbed page, with panels for the NYT, Washington Post, Guardian, Movies, Tech, baseball and basketball.

Other docs

1. The road to River5, reviews the previous versions of my river-of-news software going back to 1999, with a special focus on the transition from River4 to River5.

2. The Hello World of Rivers shows you how to create a standalone river, one that people can read without accessing your RIver5 server.

3. My blog post announcing River5.

Updates

v0.50 -- 5/18/17 by DW

Major release of River5, but in a way not much changed.

The lib folder is gone. The two items that were in the folder have been published using NPM, the official Node.js software distribution system. This means that they can be used in projects other than River5.

river5.js has been changed to use the new version of feedtools, called davereader (the name feedtools was already taken). It should work exactly as before.

This change will make new feed-based software possible. I've written about that on the reader site, and in a blog post.

Otherwise the changes should be totally transparent to River5 users. The software should behave exactly as it did before. Not much actually changed, it's just been reorganized to make it more useful in more situations.

Future updates will be logged in the Updates section of the reader site.

To update to the new version, follow the same instructions as for previous releases.

v0.49 -- 4/18/17 by DW

Support for podcasting. New config values, described on the config docs page.

Added buildRiver callbacks.

v0.48 -- 3/29/17 by DW

Changes for Electric River. Added three new callbacks to config that allow us to do things a bit more efficiently because the reading and configuring app is running in the same process as the server. Wrote a howto doc for people with experience running River5. Added flBuildEveryFiveSeconds to config, if true, we build changed rivers every five seconds instead of once a minute, the default. Building rivers became a lot more efficient in the change from River4 so it made sense to allow them to be rebuilt more frequently.

v0.47 -- 2/9/17 by DW

Fixed two problems reported by Andrew Shell. Added some feeds to the default set up, and configured the default home page so it only has three panels and they have intelligent labels, using the relatively new homePage configuration option. Fixed a problem with NYT feeds and cookies. We now set the User-Agent header on requests. Apparently some servers won't respond unless this header is present.

v0.46 -- 11/16/16 by DW

Refreshed the admin app, relocated it, and fixed a couple of entry-points it depends on that didn't work with CORS.

v0.45 -- 3/14/16 by DW

Fixed a bug that prevented addToRiver callbacks from working correctly. When you made a change to an item in a river it wasn't actually being reflected back in the river.

As a result of this fix, River4 callbacks should now work in River5.

v0.44 -- 2/22/16 by DW

Fixed a bug that prevented text file lists from working. It would only subscribe to the last feed in the list.

v0.43 -- 2/4/16 by DW

You can now configure the panels on the server's home page.

The new configuration feature is documented in a new section on the configuration page.

v0.42 -- 2/4/16 by DW

We accept two new formats for files in the lists folder -- straight text and JSON, in addition to the traditional OPML subscription lists.

If a file in the lists folder has a .txt extension, we assume it's just a series of lines with the URLs of feeds. It's the simplest of all the list formats.

If it has a .json format, we look for a JSON file that's just an array of URLs.

Created a new top-level lists folder containing example lists in each of the three supported formats.

v0.41 -- 2/3/16 by DW

We now maintain a new data structure for each list, riverData.json, in the same folder as the listInfo.json file.

It contains an array of river items that are part of this river. To build the riverjs file, all we have to do is loop over that array.

It's kept in memory while River5 is running, so now building a river on reads a file from the disk the first time a river is built or an item is added to it.

One of the first bennies of rebuilding the river software, it's now quite a bit faster at building rivers, and it's no longer a function of how dense the river is. In previous versions, rivers that don't update often would require us to go back a long ways into the calendar structure to build the river.

Now the calendar structure is just an archive, it's no longer used to build anything. It could be made optional.

Questions, comments?

Please post a note on the River5 mail list.