Description

Link-Metadata used to get metadata from internet URLs (htmls, images, media..). It also returns security policy information (Cross-Origin Resource Sharing, Content Security Policy) and checks them with Google Safe Browsing (Web Risk).

Installation

$ npm install @bedrock-api/link-metadata

Usage

Link-Metadata is TypeScript-friendly and comes with both CommonJS and ESM module support.

const lmd = require('@bedrock-api/link-metadata');

or

import lmd from '@bedrock-api/link-metadata';

First, create a new user if you don't have one:

const firstName = 'name';
const lastName = 'surname';
const email = '[email protected]';
const username = 'username';
const password = 'password';

const reply = await lmd.createUser(firstName, lastName,
    email, username, password);

console.log(reply);

Second, log in and create a new access key. If you already have one you don't need to login, just use the key.

const response = await lmd.login(username, password);

if (response.error) {
    console.error(response.error);
    return;
}

const key = response.accessKey;

Getting a metadata for the given URL:

const metadata = await lmd.getMetadata(url, key);

API

login(username, password)

Used to get first instance of access key. Returns KeyResponse:

interface AccessKey {
    accessId: string;
    accessSecret: string;
}

interface KeyResponse {
    username?: string;
    accessKey?: AccessKey;
    error?: string;
}

If an error occurs, it is returned in the corresponding field.

createKey(key)

Issue a new key (using current). All keys are the same in terms of functionality, there is no 'primary' key. You may create as much keys as you want. Returns KeyResponse.

getKeys(key)

List all available keys. Returns KeysResponse:

interface KeysResponse {
    username?: string;
    accessKeys?: AccessKey[];
    error?: string;
}

revokeKey(key)

Revokes a key (the one that passed in argument). Useful when a key becomes compromised. Returns a string.

getMetadata(url, key)

To obtain metadata for the URL. Needs an access key (token) to be provided. Returns MetadataResponse structure.

interface MetadataResponse {
    url: string; // The original request url
    status: number | null; // Request status code. Null in cases where url is invlaid and the like. Never make the network request

    type: string | null; // The type of the content. This should be a static list (enum). This may be base on
    // the file such as video. Or more specific like article/book/etc. Full list to be determined.
    contentType: string | null; // HTML mime time from the http request
    contentEncoding: string | null; // HTML content encoding from the http request

    audio: Audio | null; // If the file is an audio file, audio details will be here.
    file: File | null; // If the file is not any of the other supported types, what details we have will be here.
    html: HTML | null; // If the file is an html file, html details will be here.
    image: Image | null; // If the file is an image file, image details will be here.
    video: Video | null; // If the file is an video file, video details will be here.
    xml: XML | null; // If the file is an xml file, xml details will be here.

    // errors
    error: number | null; // Number indicates there was an error and is the error code
    reason: string | null; // Reason string for error
    warnings: string[] | null; // A list of warnings/errors while processing. See [Warnings](#warnings) section for some examples

    // Web Risk
    googleSafeBrowsing: GoogleSafeBrowsing | null;
}

reportBadData(url, reason)

Reports issues with the metadata (not sufficient, irrelevant, etc). The report should consist of the url of issue and an optional reason message. This helps us to get feedback and improve our service.

const response = await lmd.reportBadData('https://www.youtube.com/watch?v=2mNwZHcoW_o', 'wrong language');

What it does

Our service helps you extract metadata without manually processing it on your end. You can detect malware, get all data in one place and store frequently used websites in cache.

SDK installation

Create an empty project in your IDE. Run the following in terminal:

npm install @bedrock-api/link-metadata

Run the following (don't forget to insert your credentials):

import { AccessKey, MetadataResponse, login, createKey, getKeys, revokeKey, getMetadata } from '@bedrock-api/link-metadata';

const username = 'User';
const password = 'password';

async function main() {
    let key: AccessKey;
    let key2: AccessKey;
    let meta: MetadataResponse;

    // Logging in and creating a new AccessKey
    const response = await login(username, password);
    if (response.error) return;
    key = response.accessKey;

    // Getting a metadata for the given URL
    const url = 'https://www.npmjs.com';
    meta = await getMetadata(url, key);
    console.log(meta);

    // Creating additional key (optional)
    const reply = await createKey(key);
    if (reply.error) return;
    key2 = reply.accessKey;

    // Getting a list of current keys
    const keys = await getKeys(key);
    if (keys.error) return;
    console.log(keys);

    // Deleting created keys
    await revokeKey(key);
    await revokeKey(key2);
}

main();

Usage

Log in and try to get a sample response by running this:

const lmd = require('@bedrock-api/link-metadata');

const username = 'User';
const password = 'password';

async function main() {
    // Logging in and creating a new AccessKey
    let response = await lmd.login(username, password);
    const key = response.accessKey;

    // Getting a metadata for the given URL
    const url = 'https://www.npmjs.com';
    response = await lmd.getMetadata(url, key);
    console.log(response);

    // Creating additional key
    response = await lmd.createKey(key);
    const key2 = response.accessKey;

    // Getting a list of current keys
    response = await lmd.getKeys(key);
    console.log(response);

    // Deleting both created keys
    await lmd.revokeKey(key);
    await lmd.revokeKey(key2);
}

main();

Why use an SDK?

Using SDK allows you to get metadata without the need to encrypt your credentials.

Using the service without SDK

In order to use the service without an SDK installation, you have to use https://www.base64encode.org/ to generate the encoded key. Provide your secret to the service in this format:

accessId:accessSecret

Then proceed in Postman by using the key you get from Base64 in Headers section of Authorization.

When you're authorized, the request can be made by adding a URL after url= parameter:

http://localhost:3000/SpinItOn?url=https://shorturl.at/AT089

Sample response

Let's say your name is John Doe and you're maintaining a startup that collects all news connected to the election system. You're running the link to the news article through our service and this would be the response that you get:

{
       "url": "https://shorturl.at/AT089",
    "status": 200,
    "type": "html",
    "contentType": "text/html; charset=utf-8",
    "contentEncoding": null,
    "audio": null,
    "file": null,
    "html": {
        "url": "https://www.foxnews.com/politics/pains-pump-could-shift-midterm-outcome-election-won-lost-pump",
        "size": 135327,
        "title": "Pain at the pump could shift midterm outcome: 'This election is likely won or lost at the pump' | Fox News",
        "description": "Gas prices are back on the rise just weeks before the midterm elections, a development that could decide which party takes control of the House and the Senate this fall.",
        "author": "Aubrie Spady",
        "publisher": "Fox News",
        "language": null,
        "languageAlternates": [],
        "websiteName": "Fox News",
        "image": {
            "url": "https://a57.foxnews.com/static.foxnews.com/foxnews.com/content/uploads/2022/03/1200/675/gas-pumps-3.jpg?ve=1&tl=1",
            "size": 127729,
            "format": "jpg",
            "height": 675,
            "width": 1200,
            "security": {
                "cors": {
                    "Access-Control-Allow-Origin": "*",
                    "Access-Control-Allow-Methods": "GET,HEAD",
                    "Access-Control-Allow-Headers": "*",
                    "Access-Control-Max-Age": "86400"
                },
                "csp": {},
                "x-frame-options": null
            }
        },
        "video": null,
        "audio": null,
        "favicon": {
            "url": "https://static.foxnews.com/static/orion/styles/img/fox-news/favicons/favicon.ico",
            "size": 10990,
            "format": "ico",
            "height": 48,
            "width": 48,
            "security": {
                "cors": {
                    "Access-Control-Allow-Origin": "*",
                    "Access-Control-Allow-Methods": "GET,HEAD,POST",
                    "Access-Control-Allow-Headers": "*",
                    "Access-Control-Max-Age": "86400"
                },
                "csp": {},
                "x-frame-options": null
            }
        },
        "security": {
            "cors": {
                "Access-Control-Allow-Origin": "*",
                "Access-Control-Allow-Methods": "GET,HEAD",
                "Access-Control-Allow-Headers": "*",
                "Access-Control-Max-Age": "86400"
            },
            "csp": {},
            "x-frame-options": "DENY"
        }
    },
    "image": null,
    "video": null,
    "xml": null,
    "error": null,
    "reason": null,
    "warnings": null,
    "googleSafeBrowsing": {
        "isUrlSafe": true,
        "result": {}
    }
}

Fields content

You should expect the following fields that contain metadata. If there’s no content - that might happen due to lack of such data on the original website.

• URL name - the link you’ve run to get metadata
• Status - request result code (standardized: ex. 404 error)
• Type - type of content we’ve got following your link
• Image - shows if the link contains any images and provides the link to it
• ContentType - the type of content that is used in this link
• ContentEncoding - ?
• Audio - shows if the link contains any audio files and provides the link to it
• File - shows if the link contains any files and provides the link to it
• HTML - shows the metadata on the requested HTML page
• Size - shows the size of the content
• Format - shows the format of the content
• Height - shows height of the image
• Width - shows width of the image
• Video - shows if the link contains an video files and provides the link to it
• Xml - shows if the link contains an XML file and provides the link to it
• Error - critical errors that interfere with running the request
• Reason - the explanation of the error
• Warnings - non-critical yet important notes regarding your request
• GoogleSafeBrowsing - shows if the website you’re running has potential dangers such as malware, viruses e.t.c.