@tableland/jeti
v2.0.0
Published
An IPFS and generic plugin framework for inserting & retrieving data with the @tableland/sdk
Downloads
28
Readme
@tableland/jeti
An IPFS and generic plugin framework for inserting & retrieving data with the @tableland/sdk
Table of Contents
Background
The @tableland/jeti
library builds on top of the @tableland/sdk
, allowing you to easily add IPFS data to Tableland and read the underlying data from Tableland. Simply import the library, connect to the Tableland network, and you are ready to start creating, updating, and reading table data.
JETI (JavaScript Extension for Tableland Integrations) is also designed to be extensible. You can create your own custom plugins/processors that transform data inserted into tables or retrieved from tables. For example, the IPFS plugin processes data and inserts a CID into a cell, and when data is read, it will "see" the CID and fetch/transform the underlying content within the query response. You can do whatever you'd like with an implementation using the creatorProcessor
method.
Install
Installation is easy using npm or yarn. An ES bundle is also available for those operating purely in a browser environment.
npm i @tableland/jeti
Usage
Before getting started, you'll need to set up and have an IPFS node running. For example, IPFS Desktop exposes the API on http://127.0.0.1:5001
by default; you'll then need to pass host/port information to connect to the IPFS node, which are endpoints defined in the IPFS HTTP API.
For the full documentation, see the docs page: here.
Set up Tableland database
First, connect and create a table. You'll want to make sure @tableland/sdk
is installed as well, and this example shows ethers for setting up a signer with a private key on a local-only Local Tableland network running.
import { Database } from "@tableland/sdk";
import { Wallet, getDefaultProvider } from "ethers";
// Set up signer
const privateKey =
"59c6995e998f97a5a0044966f0945389dc9e86dae88c7a8412f4603b6b78690d";
const wallet = new Wallet(privateKey);
// To avoid connecting to the browser wallet (locally, port 8545).
// For example: "https://polygon-mumbai.g.alchemy.com/v2/YOUR_ALCHEMY_KEY"
const provider = getDefaultProvider("http://127.0.0.1:8545");
const signer = wallet.connect(provider);
// Connect to database & create table
const db = new Database({ signer });
const { meta: create } = await db
.prepare(`CREATE TABLE my_table (id integer primary key, val text);`)
.run();
await create.txn?.wait();
const [tableName] = create.txn?.names ?? [];
console.log(tableName); // my_table_31337_2
Pin to IPFS
In this example, using pinToLocal
as a tagged template will insert the proper values in your statement, while simultaneously uploading files to your remote pinning services.
Using the resolve
function of pinToLocal
fetches the data from IPFS into the result set.
// Existing imports
// For pinning to a local IPFS node
import { pinToLocal, skip } from "@tableland/jeti";
// Database/table setup steps from above
// Set up pinner for string templating
const localPinner = pinToLocal({
host: "127.0.0.1",
port: 5001,
protocol: "http",
});
// Define content and process SQL string
const contentToPin = "Hello world"; // A string, or a file buffer (Uint8Array)
const sql = await localPinner`insert into ${skip(
tableName
)} (val) values ('${contentToPin}');`; // Converts non-skipped variable to CID
console.log(sql);
// Insert the transformed data with a CID
const { meta: insert } = await db.prepare(sql).all();
await insert.txn?.wait();
When you read the raw results, it'll contain the CID that was inserted, unless to resolve
the results with the pinner:
const { results } = await db.prepare(`SELECT * FROM ${tableName}`).all();
console.log(results);
// [
// {
// id: 1,
// val: 'bafybeiabfiu2uipule2sro2maoufk2waokktnsbqp5gvaaod3y44ouft54'
// }
// ]
const resultsWithCIDsResolved = await localPinner.resolve(results, ["val"]);
console.log(resultsWithCIDsResolved);
// [
// {
// id: 1,
// val: 'Hello world'
// }
// ]
The pinToProvider
follows nearly the same steps, except is assumes you've configured a remote pinning provider on your IPFS node. For example, if you've configured Pinata, you can use the pinToProvider
template to pin to Pinata.
Truncate
Table cells have a max limit of 1 KB (1024 bytes), so the IPFS plugin is especially useful for data larger than that. However, if you're okay with lossy data that gets cut at 1024 bytes, the truncate
plugin can be used:
import { truncate } from "@tableland/jeti";
// 1025 letter 'a's, i.e., 1025 bytes is one over the limit
const longString = new Array(1026).join("a"); // First value is `undefined`, so it will be skipped
const sql =
await truncate`INSERT INTO my_table_31337_2 (val) values ('${longString}')`;
Since the input string was one byte over the limit, it will be truncated to 1024 bytes. When you resolve it, the original string will not be returned—an ellipses is appended to represent the truncation occurred:
const detruncated = await truncate.resolve([{ val: `${longString}` }], ["val"]);
console.log(detruncated); // Truncated input at 1024 bytes plus `...`
Create your own
Creating your own processor requires two function to be created: one for transforming inputs before data is created, and one for transforming the data after it is read. For example, if you wanted to add a value to the end of a string before it is inserted, and remove it when it is read, you could do the following:
import { createProcessor } from "@tableland/jeti";
function addValue(value: string): PrepareResult {
const add = (input: string) => {
return input + value;
};
const remove = (input: string) => {
return input.replace(value, "");
};
return createProcessor(add, remove);
}
Then, set up the processor and insert data:
const processor = addValue(" world");
const originalValueOne = "Hello";
const originalValueTwo = "Hello again";
const sql =
await processor`INSERT INTO my_table_31337_2 (val) VALUES ('${originalValueOne}'), ('${originalValueTwo}');`;
console.log(sql);
// INSERT INTO my_table_31337_2 VALUES ('Hello world'), ('Hello again world');
The processor is designed to work with the parameter and response types from the Tableland SDK. But, to emulate it's functionality, we can do the following and resolve the results:
const rawResults = [
{
id: 1,
val: "Hello world",
},
{
id: 2,
val: "Hello again world",
},
];
const unprocessedResults = await processor.resolve(rawResults, ["val"]);
console.log(unprocessedResults);
// [
// {
// id: 1,
// val: "Hello",
// },
// {
// id: 2,
// val: "Hello again",
// },
// ];
Development
Get started by cloning, installing, building, and testing the project:
git clone https://github.com/tablelandnetwork/jeti.git
cd jeti
npm install
npm run build
npm test
Contributing
PRs accepted.
Small note: If editing the README, please conform to the standard-readme specification.
License
MIT AND Apache-2.0, © 2021-2023 Tableland Network Contributors