Storacle [alpha]

Storacle is a decentralized file storage based on the spreadable protocol.

There is an article here with an explanation.

const Node = require('storacle').Node;

(async () => {  
  try {
    const node = new Node({
      port: 4000,
      hostname: 'localhost'
    });
    await node.init();
  }
  catch(err) {
    console.error(err.stack);
    process.exit(1);
  }
})();

const Client = require('storacle').Client;

(async () => {  
  try {
    const client = new Client({
      address: 'localhost:4000'
    });
    await client.init();

    // Store our file
    const hash = await client.storeFile('./my-file');

    // Get the direct file link
    const link = await client.getFileLink(hash);

    // Create the requested file link
    const requestedLink = client.createRequestedFileLink(hash);
    
    // Remove the file
    await client.removeFile(hash);
  }
  catch(err) {
    console.error(err.stack);
    process.exit(1);
  }
})();

The example above shows the simplest usage of the library. But the server can be flexibly configured.

Browser client

You can also use the client in a browser. Look at the description of the spreadable library. In window you have window.ClientStoracle instead of window.ClientSpreadable. The prepared file name is storacle.client.js.

How to use it via the command line

Look at the description of the spreadable library. You only need to change everywhere spreadable word to storacle.

How it works

Nodes interact via the spreadable mechanism. The file can be added to the network through any node. Files are saved entirely, not splitted into parts. After saving you get a hash of the file. With this hash you can later get it again, delete it or do something else. If possible, links to files are cached so if you work with the same file and in some other cases you will receive it immediately without re-traversing the network. For better reliability files can be duplicated. How exactly you can customize yourself. By default, each file tends to have its copies in amount of Math.ceil(Math.sqrt(networkSize)).

What are the limitations

The number of files on one node is limited by the file system. The speed of receiving / saving in the network files is limited by spreadable protocol.

What are the requirements

Look at the spreadable requirements.

Where to use it

1. Wherever files need to be stored decentralized

For example, you can create a network that stores books, invite new members, gather a large collection of books, and share links with others. Instead of books there could be anything else.

2. For own needs

Storing files of your own projects, websites, etc. The network can be made private and you will not need to figure out how to organize the file storage.

3. Serverless solutions

Since the library is written in javascript, you can receive / send / work with files in the browser and do not use server code at all. In some cases, it can be very convenient.

Node configuration

When you create an instance of the node you can pass options below. Only specific options of this library are described here, without considering the options of the parent classes.

• {integer} [request.clientStoringConcurrency=20] - the maximum number of simultaneous client storing requests per endpoint.

• {number|string} [request.fileStoringNodeTimeout="2h"] - file storing timeout.

• {number|string} [request.cacheTimeout=250] - file cache link check timeout.

• {number|string} [storage.dataSize="45%"] - amount of space for storing files. If indicated in percent, the calculation will be based on the maximum available disk space.

• {number|string} [storage.tempSize="45%"] - amount of space for temporary files. If indicated in percent, the calculation will be based on the maximum available disk space.

• {number|string} [storage.tempLifetime="2h"] - temporary files holding period.

• {integer} [storage.autoCleanSize=0] - amount of space that should always be free. If indicated in percent, the calculation will be based on storage.dataSize.

• {object} [file] - section that responds for a single file settings.

• {number|string} [file.maxSize="40%"] - maximum size of one file. If indicated in percent, the calculation will be based on the maximum available disk space.

• {number|string} [file.minSize=0] - minimum size of one file. If indicated in percent, the calculation will be based on the maximum available disk space.

• {integer|string} [file.preferredDuplicates="auto"] - preferred number of file copies on the network. If indicated in percent, the calculation will be based on the network size. If the option is "auto" it will be calculated as Math.ceil(Math.sqrt(networkSize)).

• {number|string} [file.responseCacheLifetime="7d"] - period of file caching after giving it to the client.

• {string[]} [file.mimeWhitelist=[]] - whitelist for filtering a file by mime type.

• {string[]} [file.mimeBlacklist=[]] - blacklist for filtering a file by mime type.

• {string[]} [file.extWhitelist=[]] - whitelist for filtering a file by its extension.

• {string[]} [file.extBlacklist=[]] - blacklist for filtering a file by its extension.

• {object|false} [file.linkCache] - file link caching transport options.

• {integer} [file.linkCache.limit=50000] - maximum cache links.

• {number|string} [file.linkCache.lifetime="1d"] - cache link holding period.

• {number|string} [task.cleanUpStorageInterval="30s"] - storage cleanup task interval.

• {number|string} [task.cleanUpTempDirInterval="20s"] - temporary folder cleanup task interval.

• {number|string} [task.calculateStorageInfoInterval="3s"] - storage information calculaion task interval.

Client configuration

When you create an instance of the client you can pass options below. Only specific options of this library are described here, without considering the options of the parent classes.

• {number|string} [request.fileStoringTimeout="2.05h"] - file storing timeout.

• {number|string} [request.fileGettingTimeout="1h"] - file getting timeout.

• {number|string} [request.fileRemovalTimeout="10s"] - file removal timeout.

• {number|string} [request.fileLinkGettingTimeout="10s"] - file link getting timeout.

Client interface

async Client.prototype.storeFile() - add file to the network.

• {string|fse.ReadStream|Buffer|Blob} file - any file
• {object} [options] - storing options
• {number} [options.timeout] - storing timeout

async Client.prototype.getFileLink() - get the file link by the hash.

• {string} hash - file hash
• {object} [options] - getting options
• {number} [options.timeout] - getting timeout

async Client.prototype.getFileLinks() - get all file links by the hash.

• {string} hash - file hash
• {object} [options] - getting options
• {number} [options.timeout] - getting timeout

async Client.prototype.getFileToBuffer() - download the file and return the buffer.

• {string} hash - file hash
• {object} [options] - getting options
• {number} [options.timeout] - getting timeout

async Client.prototype.getFileToPath() - download the file and write it to the specified path.

• {string} hash - file hash
• {object} [options] - getting options
• {number} [options.timeout] - getting timeout

async Client.prototype.getFileToBlob() - download the file and return the blob. For browser client only.

• {string} hash - file hash
• {object} [options] - getting options
• {number} [options.timeout] - getting timeout

async Client.prototype.removeFile() - Remove the file by hash.

• {string} hash - file hash
• {object} [options] - removal options
• {number} [options.timeout] - removal timeout

Client.prototype.createRequestedFileLink() - сreate a requested file link. This is convenient if you need to get the link without doing any asynchronous operations at the moment.

• {string} hash - file hash
• {object} [options] - options

Contribution

If you face a bug or have an idea how to improve the library, create an issue on github. In order to fix something or add new code yourself fork the library, make changes and create a pull request to the master branch. Don't forget about tests in this case. Also you can join the project on github.