npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

nodebb-plugin-import

v0.4.12

Published

Import your forum data to nodebb

Downloads

166

Readme

nodebb-plugin-import

Import your old forum data to nodebb | a one time use plugin

THIS PLUGIN ONLY SUPPORTS:

NodeBB v1.12.1

but you can upgrade after the import is done, make sure you follow the upgrade docs

git clone https://github.com/NodeBB/NodeBB.git
cd NodeBB
git checkout v1.12.1
npm install
node app --setup # i recommend using mongo over redis.
./nodebb start

# .. do the import...
# .. then when you're done and happy

git checkout v2.0.0
# or some other version higher than v1.12.1
./nodebb upgrade

If you want to import to an older nodebb version, you can, just checkout older versions of the plugin

Screenshots

screen shot 2015-12-09 at 8 50 35 pm screen shot 2015-12-09 at 8 51 06 pm screen shot 2015-12-09 at 8 51 27 pm screen shot 2015-12-09 at 8 51 40 pm screen shot 2015-12-09 at 8 53 02 pm screen shot 2015-12-09 at 8 53 12 pm

If you want a higher revision, import to the supported one, then just follow the upgrade procedure here https://docs.nodebb.org/en/latest/upgrading/index.html

Usage

Install it and activate it from the NodeBB Admin Panel, or

npm install nodebb-plugin-import

then re-build nodebb

./nodebb build

then run nodebb

node app

# or
./nodebb start

DISABLE ALL OTHER PLUGINS especially any DB indexer and Markdown

Activate it, then visit http://localhost:4567/admin/plugins/import (or whatever URL your instance lives on)

Source forums support:

Keep in mind that some Exporters may not be compatible or updated will all versions. File an issue when it's not.

Can't find the exporter you need?

File an issue, request it. We usually would want a DB Dump and some time to write one out.

Or even better, write your own, see write-my-own-exporter

Future versions support

I will try to keep supporting future NodeBB versions, since it's still very young and I'm a fan, but you need to submit an issue with all the details (NodeBB version, issue etc..), and I will help as fast as I can, or a pull request if you find an issue or a missing feature

Imported, now what?

Once the importer is done, 4 Files will be available for you to download (depending on your config, they may not be persisted for too long, so download them as soon as the import is done)

  • redirect.map.json Which is a map (which you would have configured beforhand snapshot) of all the old URLs and their corresponding new URLs if you want to redirect them correctly. This map is designed to work with RedirectBB which I wrote specifically for this purpose, but theoretically, you can write your own, or use an nginx Map Module or whatever else.
  • redirect.map.csv same data as the json, but in csv, probably what you need for the nginx map module, (example here, you probably need to find/replace all commas with a space and add a semi-colon at the end of each line, the latter you can just do when you setup the redirection template, just add the semicolon there, before downloading)

here's a sample regexy template setup that works with nginx, (still gotta replace commas with space(s)) screen shot 2016-02-12 at 5 14 22 pm

cp redirect.map.csv redirect.map
sed -i -e 's/,/    /g' redirect.map # replace each comma by 4 spaces
# you might also need to add a semi-colon ';' at the end of each line
# if you didn't do it when you setup the redirect templates

set up you nginx config

 map $request_uri $new {
       include /usr/share/nginx/html/redirect.map;
 }

Depending on how large your map file is, you might need to increase the limit

 server {
       # ....
      map_hash_max_size 123456789; # bytes
       # ....
 }
  • users.csv, which is just list of of all of the imported users, emails, oldId, newId, joindateTimeStamp, and most importantly, their new passwords (if you have configured the importer to generate passwords for you - i highly recommend against that, let them reset their passwords). Anyways, you can use this CSV file with this tool to blast an email to all of your users telling them what happened. http://akhoury.github.io/pages/mandrill-blast
  • users.json same data as the csv, but in a json format.

Redis Note

you may not need to do that: I didn't when I migrated over 350k records, I had a decent machine. (Ubuntu 12.04, 8GB Memory, 4 Cores, 80GB SSD Disk)

Since the importer will be hitting the database constantely, with almost 0 interval, I would add these config to the bottom of your redis.conf file, to disable some stuff and make redis more responsive, but less safe, then after the migration is complete, you must, before you kill your redis server, redis-cli save to synchronously write the data to disk, then remove these extra configs and restart your redis server. If you're a redis guru, you don't need my help, but take a look at it anyway and let me know where I went wrong :)

# NODEBB-PLUGIN-IMPORT TEMPORARY SETTINGS

# disabling saving !!!!
# then manually run 'redis-cli bgsave' after migration is done
save ""

stop-writes-on-bgsave-error no
rdbcompression no
rdbchecksum no
appendonly no
appendfsync no
no-appendfsync-on-rewrite yes
hz 100
aof-rewrite-incremental-fsync yes

Mongo Note

You should not need to do the same thing for redis, since Mongo immediately persists to disk.

Markdown Note

NodeBB prefers using Markdown as the content language format, and since most Web 1.0 forums use either straight out HTML or BB Code, there is a config option called "convert" which you can set to either "html-to-md" or "bbcode-to-md" and while importing, the importer will convert the following:

  • Users signatures
  • Topics Content
  • Topics Title
  • Categories Names
  • Categories Descriptions
  • Posts Content

If you are importing already 'markdownified' content, just don't set the convert option, or select "Don't convert" to skip the conversion, also if you are importing some other format, feel free to submit a pull request or open an issue, if there is a Node Module to it, or if there is some pre-built JS "function" to convert the content, I'll add it

Some common issues

  • After converting your content, you should restart the server, NodeBB caches the content recently read in memory.
  • EMFILE too many open files error, try disabling the server logs, there is an option for that
  • Segmentation fault error, along with disabling server logs, try the Redis note, then file an issue, I'll help you through it.
  • Error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. see the redis note, it might help
  • if NodeBB <= 0.5.1 hangs, similar to this issue, disable the Markdown plugin
  • if you get an uncaughtException: ER_WRONG_FIELD_WITH_GROUP: Expression #4 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'somedatabase.somecolumn' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by you will need to temporarily do this https://stackoverflow.com/a/35729681

Test

~~pfffffft~~ ~~soon~~ ....