LML-LIB

This library is intended to share the same code between Scratch and Snap! for computing the ML model from data coming from learningml-editor, even without needed to use the learningml-editor, that is, data (text, numbers and images) can be added and ML model can be generated directly by using blocks inside Scratch/Snap! thanks to this library.

The library expose function from three libraries:

• lml_message_protocol
  • lmlRequest
• lml_labeled_data_manager
  • LabeledDataManagerService
• lml_algorithms
  • FeatureExtractorText,
  • FeatureExtractorNumbers,
  • FeatureExtractorImage,
  • FeatureExtractor,
  • MlAlgorithmNeuralNetwork,
  • MlAlgorithmKNN,
  • MlAlgorithm

lml_message_protocol

This library only expose the function lmlRequest(operation, args). This function is intended to communicate with learningml-editor. The communication is performed through the web API function BroadcastChannel.

The first argument (operation) defines the operation to be carried out in the learningml-editor window, while the second one (args) is an array with the arguments needed by the operation.

The application learningml-editor implement a service called LmlMessageProtocolService wich allows to catch the requests made from lmlRequest(operation, args), compute the operation and returns the result by means of the channel opened trough BroadcastChannel object.

The following operation are allowed:

• classify_text
• confidence_text
• classify_full_text
• classify_image
• confidence_image
• classify_full_image
• classify_numerical
• confidence_numerical
• classify_full_numerical

This library is only used when Scratch/Snap! is used in combination with learningml-editor. Indeed, this is the original way to use LearningML. However, with the changes incorporated to Scratch and Snap! when incorporating lml-lib, both programming editor can be used to generate ML models in a independent way, that is, without the need to use learningml-editor.

lml_labeled_data_manager

This library expose an only function: LabeledDataManagerService, which is a reimplementation of the service with the same name implemented in learningml-editor. In a future is intented to replace the code of such service by this library. This way, all the application composing LML: learningml-editor, lml-scratch and lml-snap, will share the same code.

The function LabeledDataManagerService is used to build javascript objects which are instanciated like this:

let lmlLabeledDataManagerService = new lmlLib.LabeledDataManagerService();

This object acts as an parameter storage, and is used by all the application of the LML suite (learningml-editor, lml-scratch and lml-snap) in order to:

• configure the algorithms needed to extract features and build ML models (see next section lml_algorithms)
• sincronize the data between the learningml-editor and programing editors when the later are used in combination with the former.
• Add/remove entries to/from datasets.

In order to sincronize with learningml-editor parameters, LabeledDataManagerService use the localStorage web API object.

In order to maintain the same data both in the learningml-editor and programming editor, a BroadcastChannel web API object is used.

The parameters required and defined trough setter and getter methods.

A tipical use of LabeledDataManagerService in the code of a programming editor is shown in the following code snippet:

let lmlLabeledDataManagerService = new lmlLib.LabeledDataManagerService();

lmlLabeledDataManagerService.loadFromLocalStorage();
lmlLabeledDataManagerService.loadLabelsWithData();
lmlLabeledDataManagerService.loadLabels();

• lmlLabeledDataManagerService.loadFromLocalStorage() loads all the parameters needed to build the ML model.
• lmlLabeledDataManagerService.loadLabelsWithData() loads all the training data related with their labels.
• lmlLabeledDataManagerService.loadLabels() loads all the labels. Although this may seem redundant, to have the list of labels without their associated data is more convenient when using the extractor features and ML algorithms libraries.

It is important to note that, this three lines can be called even when the programming editor is used independenly without learningml-editor. In this case, the sincronization won't be performed, since it isn't needed.

- lml_algorithms

This library is intended to perform all the tasks needed to build ML algorithm. It's based on tensorflow.js and it's composed by two types of functions. The first type are feature extractor, that is, they receive as input some texts, images or numbers to be transformed in a tensor which will be the input of the second type of functions. This second type of funcitions implement several ML algorithm from which ML models are built.

The following snippet of code shows how to build a ML model

let labeledDataManager = new lmlLib.LabeledDataManagerService();
lmlLabeledDataManagerService.loadFromLocalStorage();
lmlLabeledDataManagerService.loadLabelsWithData();
lmlLabeledDataManagerService.loadLabels();

let fe = new lmlLib.FeatureExtractor(labeledDataManager);
let al = new lmlLib.MlAlgorithm(labeledDataManager);
labeledDataManager.changeState("train");
let info =  fe.extractDatasetFeatures().then(features => {
    labeledDataManager.changeState("end");
    return al.train(features);
});

The object labeledDataManager has all the information needed to build the model: labels, example data, type of algorithm to apply (KNN or Neural Networks right now), type of model (text, image or numbers), etcetera.

Once the ML model is built, it can be used to classify as follows:

let result = fe.extractInstanceFeatures(data).then(feature => {
    return al.classify(feature);
})

Where data is a text string, a HTMLImageElement or a array of numbers represented as a CSV text string.

The use of these libraries can be seen in the file src/extensions/scratch3_learningml/index.js of lml-scratch-vm project.

How to build for browser

In order to be used with lml-snap, this library must be broserified.

This can be done as follow:

# npm run browserify

This will create a file lml-lib.js. Once created, it must be copied into lml-snap project (src/lml folder).