teeft

Présentation

Ce plugin propose une série d'instructions pour extraire des mots-clés d'un texte français ou anglais en utilisant l'algorithme Teeft.

C'est le paquet officiel qui fait suite à l'expérimentation ezs-teeftfr.

Bibliographie

Cuxac P., Kieffer N., Lamirel J.C. : SKEEFT: indexing method taking into account the structure of the document. 20th Collnet meeting, 5-8 Nov 2019, Dalian, China.

Installation

npm install @ezs/core
npm install @ezs/teeft

Flux

Le principe est de décomposer le flux en une série d'instructions ayant chacune ses propres paramètres (voir usage).

Voici la séquence typique d'instructions qui permet de lire les fichiers .txt d'un répertoire et de les envoyer aux tokenizers de phrase, puis de mots; ensuite on procède à un étiquetage grammatical, puis on extrait les termes, et on les filtre (par fonction grammaticale), on enlève les nombres, on supprime les mots vides, on calcule les fréquences des tokens, puis leur spécificité, enfin, on les filtre suivant leur fréquence.

[ "/path/to/a/directory/of/documents" ] ->

[TeeftListFiles]
pattern = *.txt

--> [ "/path1", "path2", ... ] -->

[TeeftGetFilesContent]

--> [ { path, content }, ... ] -->

[TeeftSentenceTokenize]

--> [ { path, sentences: [ "sentence", ... ] }, ... ] -->

[TeeftTokenize]

--> [ { path, sentences: [ ["token", ... ], ...] }, ... ] -->

[TeeftNaturalTag]

--> [  { path, sentences: [ [
  {
    token: "token",
    tag: [ "tag", ...]
  }, ...
 ], ... ] }, ... ]


[TeeftExtractTerms]
nounTag = NOM
adjTag = ADJ

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length
  },
  {
    term: "multiterm",
    frequency,
    length
  }, ...
 ] }, ... ]

[TeeftFilterTags]
tags = NOM
tags = ADJ

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length
  },
  {
    term: "multiterm",
    frequency,
    length
  }, ...
 ] }, ... ]

[TeeftRemoveNumbers]

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length
  },
  {
    term: "multiterm",
    frequency,
    length
  }, ...
 ] }, ... ]

[TeeftStopWords]

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length
  },
  {
    term: "multiterm",
    frequency,
    length
  }, ...
 ] }, ... ]

[TeeftRemoveShortTerms]

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length
  },
  {
    term: "multiterm",
    frequency,
    length
  }, ...
 ] }, ... ]

[TeeftRemoveLongTerms]

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length
  },
  {
    term: "multiterm",
    frequency,
    length
  }, ...
 ] }, ... ]

[TeeftRemoveWeirdTerms]

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length
  },
  {
    term: "multiterm",
    frequency,
    length
  }, ...
 ] }, ... ]

[TeeftSumUpFrequencies]

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length
  },
  {
    term: "multiterm",
    frequency,
    length
  }, ...
 ] }, ... ]

[TeeftSpecificity]
sort = true

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length,
    specificity,
  },
  {
    term: "multiterm",
    frequency,
    length,
    specificity
  }, ...
 ] }, ... ]

[TeeftFilterMonoFreq]

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length,
    specificity,
  },
  {
    term: "multiterm",
    frequency,
    length,
    specificity
  }, ...
 ] }, ... ]

[TeeftFilterMultiSpec]

--> [  { path, terms:  [
  {
    term: "monoterm",
    tag: [ "tag", ...],
    frequency,
    length,
    specificity,
  },
  {
    term: "multiterm",
    frequency,
    length,
    specificity
  }, ...
 ] }, ... ]

[dump]
indent = true

usage

Table of Contents

• TeeftExtractTerms
• TeeftFilterMonoFreq
• TeeftFilterMultiSpec
• TeeftFilterTags
• TeeftGetFilesContent
• TeeftListFiles
• TeeftNaturalTag
• TeeftRemoveLongTerms
• TeeftRemoveNumbers
• TeeftRemoveShortTerms
• TeeftRemoveWeirdTerms
• TeeftSentenceTokenize
• TeeftSpecificity
• TeeftStopWords
• TeeftSumUpFrequencies
• TeeftTokenize
• TeeftToLowerCase

TeeftExtractTerms

• See: https://github.com/istex/sisyphe/blob/master/src/worker/teeft/lib/termextractor.js

Take an array of objects { path, sentences: [token, tag: ["tag"]]}. Regroup multi-terms when possible (noun + noun, adjective + noun, etc.), and computes statistics (frequency, etc.).

Use lang or nounTag, adjTag, but not lang and the others at the same time. lang is enough to set nounTag and adjTag.

Parameters

• lang string language of the terms to extract (en or fr) (optional, default 'fr')
• nounTag string noun tag (NOM in French, NN in English) (optional, default 'NOM')
• adjTag string adjective tag (ADJ in French, JJ in English) (optional, default 'ADJ')

Examples

[{
   path: '/path/1',
   sentences:
   [[
     { token: 'elle', tag: ['PRO:per'] },
     { token: 'semble', tag: ['VER'] },
     { token: 'se', tag: ['PRO:per'] },
     { token: 'nourrir', tag: ['VER'] },
     {
       token: 'essentiellement',
       tag: ['ADV'],
     },
     { token: 'de', tag: ['PRE', 'ART:def'] },
     { token: 'plancton', tag: ['NOM'] },
     { token: 'frais', tag: ['ADJ'] },
     { token: 'et', tag: ['CON'] },
     { token: 'de', tag: ['PRE', 'ART:def'] },
     { token: 'hotdog', tag: ['UNK'] }
   ]]
}]

Returns any same as input, with term replacing token, length, and frequency

TeeftFilterMonoFreq

Filter the data, keeping only multiterms and frequent monoterms.

Minimal frequency (minFrequency parameter) has a default value automatically computed from the number of tokens in the document.

Parameters

• multiLimit Number threshold for being a multiterm (in tokens number) (optional, default 2)
• minFrequency Number minimal frequency to be taken as a frequent term (optional, default 7)

TeeftFilterMultiSpec

Filter multiterms to keep only multiterms which specificity is higher than multiterms' average specificity.

TeeftFilterTags

Filter the text in input, by keeping only adjectives and names

Parameters

• lang string? Language to set tags (en or fr)
• tags string? Tags to keep (ex: ADJ, NOM)

TeeftGetFilesContent

Take an array of file paths as input, and returns a list of objects containing the path, and the content of each file.

Returns [{path: string, content: string}] Array of { path, content }

TeeftListFiles

Take an array of directory paths as input, a pattern, and returns a list of file paths matching the pattern in the directories from the input.

Parameters

• pattern String pattern for files (ex: "*.txt") (optional, default "*")

Returns [String] an array of file paths

TeeftNaturalTag

POS Tagger from natural

French pos tagging using natural (and LEFFF resources)

Take an array of documents (objects: { path, sentences: [[]] })

Yield an array of documents (objects:

{
     path, sentences: [
         [{
             token: "token",
             tag: [ "tag", ... ]
         },
         ...]
     ]
}

)

Parameters

• lang string language of the text to tag (possible values: fr, en) (optional, default 'en')

Examples

[{
     path: "/path/1",
     sentences: [{ "token": "dans",      "tag": ["prep"] },
                 { "token": "le",        "tag": ["det"]  },
                 { "token": "cadre",     "tag": ["nc"] },
                 { "token": "du",        "tag": ["det"] },
                 { "token": "programme", "tag": ["nc"] }
                 },
     ]
 }]

TeeftRemoveLongTerms

Remove long terms from documents (longer than 50 characters). Documents must have a terms key, containing an array of objects with a term key of type string..

Yields an array of documents with the same structure.

Input:

[{
  "path": "/path/to/file.txt",
  "terms": [{ "term": "this very long term should really be removed 678901" },
            { "term": "abcd" }]
}]

Output:

[{
  "path": "/path/to/file.txt",
  "terms": [{ "term": "abcd" }]
}]

TeeftRemoveNumbers

Remove numbers from the terms of documents (objects { path, terms: [{ term, ...}] }).

Yields an array of documents with the same structure.

TeeftRemoveShortTerms

Remove short terms from documents (shorter than 3 characters). Documents must have a terms key, containing an array of objects with a term key of type string..

Yields an array of documents with the same structure.

Input:

[{
  "path": "/path/to/file.txt",
  "terms": [{ "term": "a" }, { "term": "abcd" }]
}]

Output:

[{
  "path": "/path/to/file.txt",
  "terms": [{ "term": "abcd" }]
}]

TeeftRemoveWeirdTerms

Remove terms with too much non-alphanumeric characters. Documents must have a terms key, containing an array of objects with a term key of type string..

Yields an array of documents with the same structure.

Input:

[{
  "path": "/path/to/file.txt",
  "terms": [{ "term": "αβɣδ" }, { "term": "abcd" }]
}]

Output:

[{
  "path": "/path/to/file.txt",
  "terms": [{ "term": "abcd" }]
}]

TeeftSentenceTokenize

Segment the data into an array of documents (objects { path, content }).

Yield an array of documents (objects { path, sentences: []})

TeeftSpecificity

Take documents (with a path, an array of terms, each term being an object { term, frequency, length[, tag] }).

Process objects containing frequency, add a specificity to each object, and remove all object with a specificity below average specificity (except when filter is false).

Can also sort the objects according to their specificity, when sort is true.

Parameters

• lang string language to take into account (optional, default "en")
• filter Boolean filter below average specificity (optional, default true)
• sort Boolean sort objects according to their specificity (optional, default false)

TeeftStopWords

Filter the text in input, by removing stopwords in token

Parameters

• lang string language of the stopwords (en or fr) (optional, default 'en')

TeeftSumUpFrequencies

Sums up the frequencies of identical lemmas from different chunks.

TeeftTokenize

• See: http://yomguithereal.github.io/talisman/tokenizers/words

Extract tokens from an array of documents (objects { path, sentences: [] }).

Yields an array of documents (objects: { path, sentences: [[]] })

Warning: results are surprising on uppercase sentences, use TeeftToLowerCase

TeeftToLowerCase

Transform strings to lower case.

Parameters

• path Array<string> path to the property to modify (optional, default [])