wikidata-jskos

v0.3.1

Published

2 years ago

Access Wikidata in JSKOS format

Downloads

0High
0Medium
0Low

nichtich

wikidata jskos skos

Wikidata JSKOS

Access Wikidata in JSKOS format

This node module provides a web service, a command line client, and a library to access Wikidata in JSKOS format. The data includes Wikidata items as concepts and concept schemes (read) and mappings between Wikidata and other authority files (read and write).

Background

Wikidata is a large knowledge base with detailed information about all kinds of entities. Mapping its data model to JSKOS data format allows simplified reuse of Wikidata as authority file. This implementation is used in the Cocoda web application but it can also be used independently.

The mapping between Wikidata and JSKOS format includes:

Wikidata items expressed as authority records (JSKOS Concepts)
Selected Wikidata items covering information about authority files (JSKOS Concept Schemes)
Selected Wikidata statements linking Wikidata to other authority files (JSKOS Mappings)

In addition a search service is provided for selecting a Wikidata item with typeahead.

Editing Wikidata mapping statements to other authority files requires authentification via OAuth. The following authority files have been tested succesfully:

Basisklassifikation (BK)
Regensburg Classification (RVK)
Integrated Authority File (GND)
Nomisma
Iconclass

Other systems (not including DDC) may also work but they have not been converted to JSKOS yet, so they are not provided for browsing in Cocoda.

Install

Clone and Install

git clone https://github.com/gbv/wikidata-jskos.git
cd wikidata-jskos
npm install

Optionally make the command line tool wdjskos available:

npm link

Configuration

You can customize the application settings via a configuration file, e.g. by providing a generic config.json file and/or a more specific config.{env}.json file (where {env} is the environment like development or production). The latter will have precendent over the former, and all missing keys will be defaulted with values from config.default.json.

Please consult config.default.json for possible configuration options. Some notes:

To use a custom Wikibase instance, you can set the subkeys of the wikibase property. Both instance and sparqlEnpoint are necessary. By default, Wikidata is used.
wikidata-jskos supports saving, editing, and deleting mappings in Wikidata. To enable this, you will need to provide auth.algorithm and auth.key (algorithm and key to decode the JWT), as well as oauth.consumer_key and oauth.consumer_secret (for your registered OAuth consumer).
Please provide a baseUrl when used in production. If no baseUrl is provided, http://localhost:${port}/ will be used.

Usage

Run Server

For development serve with hot reload and auto reconnect at http://localhost:2013/:

npm run start

Deployment

For deployment there is a config file to use with pm2:

pm2 start ecosystem.config.json

Web Service

An instance is available at https://coli-conc.gbv.de/services/wikidata/. The service provides selected endpoints of JSKOS API.

Authentication

The following endpoints require an authenticated user:

Authentication works via a JWT (JSON Web Token). The JWT has to be provided as a Bearer token in the authentication header, e.g. Authentication: Bearer <token>. It is integrated with login-server and the JWT is required to have the same format as the one login-server provides. Specifically, the OAuth token and secret for the user need to be provided as follows:

{
  "user": {
    "identities": {
      "wikidata": {
        "oauth": {
          "token": "..",
          "token_secret": "..."
        }
      }
    }
  }
}

There are more properties in the JWT, but those are not used by wikidata-jskos. Note that the JWT needs to be signed with the respective private key for the public key provided in the configuration. Also, the OAuth user token and secret need to come from the same OAuth consumer provided in the config.

GET /status

Returns a JSKOS API status object. See JSKOS Server for details.

GET /concept

Look up Wikidata items as JSKOS Concepts by their entity URI or QID.

URL Params
uri=[uri] URIs for concepts separated by |.
language or languages: comma separated list of language codes.
Success Response
JSON array of JSKOS Concepts

Only some Wikidata properties are mapped to JSKOS fields. The result also contains broader links determined by an additional SPARQL request.

GET /mappings

Look up Wikidata mapping statements as JSKOS Concept Mappings between Wikidata items (query parameter from) and external identifiers (query parameter to). At least one of both parameters must be given.

URL Params
from=[uri|notation] specify the source URI or notation
to=[uri|notation] specify the target URI or notation
fromScheme=[uri|notation] only show mappings from this concept scheme (URI or notation)
toScheme=[uri|notation] only show mappings to this concept scheme (URI or notation)
language or languages enables inclusion of entity labels. A comma separated list of language codes is used as preference list.
mode=[mode] specify the mode for from, to, one of and (default) and or
direction=forward|backward|both searches mappings from from to to (default), reverse, or in both directions
limit=[number] maximum number of mappings to return (not fully implemented)
offset=[number] start number of mappings to return (not fully implemented)

Concept Schemes are identified by BARTOC IDs (e.g. http://bartoc.org/en/node/430`).

Success Response
JSON array of JSKOS Concept Mappings
Examples
?from=http://www.wikidata.org/entity/Q42
?to=http://d-nb.info/gnd/119033364

Mapping relation types (P4390) are respected, if given, see for example mapping from Wikidata to http://d-nb.info/gnd/7527800-5.

GET /mappings/voc

Look up Wikidata items with Wikidata properties for authority control as JSKOS Concept Schemes with used for mappings. These schemes need to have a BARTOC-ID (P2689), and be subject item (P1629) of an external identifier property with statements P1921 (URI template) and P1793 (regular expression).

URL Params
None.
Success Response
JSON array of JSKOS Concept Schemes

GET /mappings/:_id

Returns a specific mapping for a Wikidata claim/statement.

Success Response
JSKOS object for mapping.
Error Response
If no claim with _id could be found, it will return a 404 not found error.

Sample Call

curl https://coli-conc.gbv.de/services/wikidata/mappings/Q11351-9968E140-6CA7-448D-BF0C-D8ED5A9F4598

{
  "uri": "http://localhost:2013/mappings/Q11351-9968E140-6CA7-448D-BF0C-D8ED5A9F4598",
  "identifier": [
    "http://www.wikidata.org/entity/statement/Q11351-9968E140-6CA7-448D-BF0C-D8ED5A9F4598",
    "urn:jskos:mapping:content:2807c55eac85ed8c0c9254ff04b457f89b801ac9",
    "urn:jskos:mapping:members:daafcd8580e6f0304f0b1cee024f65f04da98a3c"
  ],
  "to": {
    "memberSet": [
      {
        "uri": "http://rvk.uni-regensburg.de/nt/VK",
        "notation": [
          "VK"
        ]
      }
    ]
  },
  "type": [
    "http://www.w3.org/2004/02/skos/core#exactMatch"
  ],
  "fromScheme": {
    "uri": "http://bartoc.org/en/node/1940",
    "notation": [
      "WD"
    ]
  },
  "toScheme": {
    "uri": "http://bartoc.org/en/node/533",
    "notation": [
      "RVK"
    ]
  },
  "from": {
    "memberSet": [
      {
        "uri": "http://www.wikidata.org/entity/Q11351",
        "notation": [
          "Q11351"
        ]
      }
    ]
  },
  "@context": "https://gbv.github.io/jskos/context.json"
}

POST /mappings

Saves a mapping in Wikidata. Requires authentication.

Note that if an existing mapping in Wikidata is found with the exact same members, that mapping will be overwritten by this request.

Success Reponse
JSKOS Mapping object as it was saved in Wikidata.

PUT /mappings/:_id

Overwrites a mapping in Wikidata. Requires authentication.

Success Reponse
JSKOS Mapping object as it was saved in Wikidata.

DELETE /mappings/:_id

Deletes a mapping from Wikidata. Requires authentication.

Success Reponse
Status 204, no content.

GET /suggest

OpenSearch Suggest endpoint for typeahead search.

Command line tool

The command line client wdjskos provides roughly the same commands as accessible via the web service.

Mapping schemes are cached in the caching directory of wikidata-cli.

wdjskos concept

Look up Wikidata items as JSKOS Concepts.

wdjskos concept Q42

wdjskos mappings

Look up JSKOS Concept Mappings.

wdjskos mappings Q42 | jq .to.memberSet[].uri
wdjskos mappings - http://viaf.org/viaf/113230702

A single hyphen (-) can be used to nullify argument from or to, respectively. Mappings can be limited to a target scheme. These are equivalent:

wdjskos --scheme P227 mappings Q42
wdjskos --scheme 430 mappings Q42
wdjskos --scheme http://bartoc.org/en/node/430 mappings Q42

wdjskos schemes

Look up JSKOS Concept Schemes with Wikidata properties for authority control.

wdjskos find

Search a Wikidata item by its names and return OpenSearch Suggestions response.

wdjskos mapping-item

Convert a JSKOS mapping to a Wikidata item.

wdjskos mapping-item mapping.json
wdjskos --simplfiy mapping-item mapping.json

API

The node library can be used to convert Wikidata JSON format to JSKOS (mapEntity) and to convert JSKOS mappings to Wikidata JSON format (mapMapping).

mapEntity

jskos = wds.mapEntity(entity)

Entity data can be retrieved via Wikidata API method wbgetentities and from Wikidata database dumps. See JavaScript libraries wikidata-sdk and wikidata-filter for easy access and processing.

Map selected parts of a Wikidata entity

All methods return a JSKOS item.

jskos = wds.mapIdentifier(entity.id)
// { uri: "http://www.wikidata.org/entity/...", notation: [ "..." ] }

jskos = wds.mapLabels(entity.labels)
// { prefLabel: { ... } }

jskos = wds.mapAliases(entity.aliases)
// { altLabel: { ... } }

jskos = wds.mapDescriptions(entity.descriptions)
// { scopeNote: { ... } }

jskos = wds.mapSitelinks(entity.sitelinks)
// { occurrences: [ { ... } ], subjectOf: [ { url: ... }, ... ] }

jskos = wds.mapClaims(entity.claims)
// ...

// convert claims with mapping properties
jskos = wds.mapMappingClaims(claims)

jskos = wds.mapInfo(entity)
// ...

Map simplified Wikidata entities

Each method has a counterpart to map simplified Wikidata entities.

jskos = wds.mapSimpleEntity(entity)
jskos = wds.mapSimpleIdentifier(entity.id)
jskos = wds.mapSimpleLabels(entity.labels)
...

mapMapping

Convert a JSKOS mapping into a Wikidata claim. Only respects JSKOS fields from, to, uri, and type (if given) and only supports 1-to-1 mappings from a single Wikidata item to a concept in another concept scheme.

this is work in progress!

Contributing

See https://github.com/gbv/wikidata-jskos.

For debugging set environment variable DEBUG, also possible via file .env:

DEBUG=wds:*

See lib/utils/debug.js for details.

License

MIT License