Google News
logo
ml5.js Interview Questions
ml5.js a friendly machine learning JavaScript library for the Web, aims to make machine learning approachable for a broad audience of artists, creative coders, and students.

The library provides access to machine learning algorithms and models in the browser, building on top of TensorFlow.js with no other external dependencies.

Ml5.js is inspired by Processing and P5.js, whose goal is to empower people of all interests and backgrounds to learn how to program and make creative work with code.
 
However, to get started with machine learning, one needs advanced understanding of math and programming.
 
And we'd like to make this process easier so that machine learning can be something that everyone can learn, understand and explore freely.
It provides immediate access to pre-trained models in the browser and we can also build and train our own neural networks in the browser from scratch. 

* Ml5.js also provides friendly API to get access to more human-readable results and draw those results on the canvas with, for example, p5.js.

* We can create a neural network that classifies RGB values into common color names.
 
* With ml5.js, we can load the data, create model, train it and run the model.
 
* Ml5 has a wide collection of image, sound and text-based models with a variety of applications, such as detecting objects, human bodies, hand poses and faces, generating text, images and drawings, implementing image translations, classifying audios, detecting pitch and analyzing words and sentences.
 
* Ml5.js also provides NeuralNetwork, FeatureExtractor, KNNClassifier and KMeans as helper functions.
Step 1 : Call your ml5 function. 
const myClassifier = await ml5.imageClassifier(‘MobileNet’);
 
Step 2 : Apply your ml5 function - e.g. to an image, video, or text. 
const results = await myClassifier.classify(myCatImage);
 
Step 3 : Do something with the results.
// An array of objects with “label” and “confidence”
// [ { label: ‘cat’, confidence: 0.74 } ]
console.log(results);
 
We can run a model in the browser with ml5.js in three simple steps.
 
First, create a model.
 
Secondly, ask the model to classify or predict something based on a input, like an image or a text.
 
And step three, getting the results.
 
It also has great integration with p5.js, a JavaScript library for creating graphics and animations in the browser, which makes it easier to get inputs from webcam or microphones and also to show the outputs with canvas, image or audio.
Step 1: Define your variables
Here we define our variables that we will assign our classifier and image to.
// Initialize the Image Classifier method with MobileNet. A callback needs to be passed.
let classifier;

// A variable to hold the image we want to classify
let img;
 
Step 2: Load your imageClassifier and image
Use p5's preload() function to load our imageClassifier model and our bird image before running the rest of our code. Since machine learning models can be large, it can take time to load. We use preload() in this case to make sure our imageClassifier and image are ready to go before we can apply the image classification in the next step.
function preload() {
  classifier = ml5.imageClassifier('MobileNet');
  img = loadImage('images/bird.png');
}

Step 3: Setup, classify, and display
In p5.js we use the setup() function for everything in our program that just runs once. In our program, we use the setup() function to:
 
* create a canvas to render our image
* call .classify() on our classifier to classify our image
* render the image to the canvas

You will notice that the .classify() function takes two parameters: 1. the image you want to classify, and 2. a callback function called gotResult. Let's look at what gotResult does.
function setup() {
  createCanvas(400, 400);
  classifier.classify(img, gotResult);
  image(img, 0, 0);
}
 
Step 4: Define the gotResult() callback function
The gotResult() function takes two parameters: 1. error, and 2. results. These get passed along to gotResult() when the .classify() function finishes classifying the image. If there is an error, then an error will be logged. If our classifier manages to recognize the content of the image, then a result will be returned.
 
 
In the case of our program, we create a div that displays the label and the confidence of the content of the image that has been classified. The nf() function is a p5 function that formats our number to a nicer string.
// A function to run when we get any errors and the results
function gotResult(error, results) {
  // Display error in the console
  if (error) {
    console.error(error);
  } else {
    // The results are in an array ordered by confidence.
    console.log(results);
    createDiv(`Label: ${results[0].label}`);
    createDiv(`Confidence: ${nf(results[0].confidence, 0, 2)}`);
  }
}
You've just made a simple machine learning powered program that :
 
* takes an image,
* classifies the content of that image, and
* displays the results all in your web browser!

Not all of our examples are structured exactly like this, but this provides a taste into how ml5.js is trying to make machine learning more approachable. Try using different images and seeing what kinds of things get returned.
A big part of the ml5 project is to create lots of examples as launching points for all your creative project ideas. You can find all of the ml5.js examples at our Github page: ml5-library. If you start to explore these examples you can see how the different ml5 functions are used to accomplish different outcomes. We try our best to keep the examples as simple as possible so you can easily start to build your ideas on top of them.
 
You can check out all the examples running:
 
* on this massive list
* on the p5 web editor
In p5.js, callbacks are passed as arguments to functions that often perform some asynchronous operation. For example, p5.js defines the loadJSON() function as the following:
loadJSON('http//example.com/data.json', (results) => {
  // Do something with the results
});
 
Notice that the results from the callback in p5.js are given as the only argument to the function. There is no error argument in the callback.
 
ml5.js, on the other hand, uses a pattern referred to as an error-first callback :
 
With this pattern, a callback function is passed to the method as an argument. When the operation either completes or an error is raised, the callback function is called with the Error object (if any) passed as the first argument. If no error was raised, the first argument will be passed as null. Taken from the Node.js documentation
 
For example if you are using the imageClassifier() method, you will need to construct it in the following way:
// Pass a callback function to constructor
const classifier = ml5.imageClassifier('MobileNet', (err, model) => {
  console.log('Model Loaded!');
});

// Make a prediction with the selected image and pass a callback function with two arguments
classifier.predict(image, (err, results) => {
  // Check for errors. If no errors, then do something with the results
});
 
Error first callbacks is a convention common to many JavaScript libraries that we have chosen to adopt. The language JavaScript itself does not enforce this pattern. Keep in mind that most ml5.js methods and functions are asynchronous (machine learning models can take significant amounts of time to process inputs and generate outputs!). You will need to use the error-first callback pattern if you want to use callbacks.
ml5.js also supports Promises. If no callback is provided to any asynchronous function then a Promise is returned.
 
With Promises, the image classification example can be used in the following way :
// No callback needs to be passed to use Promises.
ml5
  .imageClassifier('MobileNet')
  .then(classifier => classifier.predict(image))
  .then((results) => {
    // Do something with the results
  });
Create your own neural network and train it in the browser with the ml5.neuralNetwork. Collect data to train your neural network or use existing data to train your neural network in real-time. Once it is trained, your neural network can do classification or regression tasks.
In general the steps for using the ml5.neuralNetwork look something like : 
 
Step 1 : load data or create some data
Step 2 : set your neural network options & initialize your neural network
Step 4 : add data to the neural network
Step 5 : normalize your data
Step 6 : train your neural network
Step 7 : use the trained model to make a classification
Step 8 : do something with the results

The below examples are quick
 
Creating data in real-time
// Step 1: load data or create some data 
const data = [
  {r:255, g:0, b:0, color:'red-ish'},
  {r:254, g:0, b:0, color:'red-ish'},
  {r:253, g:0, b:0, color:'red-ish'},
  {r:0, g:255, b:0, color:'green-ish'},
  {r:0, g:254, b:0, color:'green-ish'},
  {r:0, g:253, b:0, color:'green-ish'},
  {r:0, g:0, b:255, color:'blue-ish'},
  {r:0, g:0, b:254, color:'blue-ish'},
  {r:0, g:0, b:253, color:'blue-ish'}
];

// Step 2: set your neural network options
const options = {
  task: 'classification',
  debug: true
}

// Step 3: initialize your neural network
const nn = ml5.neuralNetwork(options);

// Step 4: add data to the neural network
data.forEach(item => {
  const inputs = {
    r: item.r, 
    g: item.g, 
    b: item.b
  };
  const output = {
    color: item.color
  };

  nn.addData(inputs, output);
});

// Step 5: normalize your data;
nn.normalizeData();

// Step 6: train your neural network
const trainingOptions = {
  epochs: 32,
  batchSize: 12
}
nn.train(trainingOptions, finishedTraining);

// Step 7: use the trained model
function finishedTraining(){
  classify();
}

// Step 8: make a classification
function classify(){
  const input = {
    r: 255, 
    g: 0, 
    b: 0
  }
  nn.classify(input, handleResults);
}

// Step 9: define a function to handle the results of your classification
function handleResults(error, result) {
    if(error){
      console.error(error);
      return;
    }
    console.log(result); // {label: 'red', confidence: 0.8};
}
Loading Existing Data :
External data : "data/colorData.json"
{
  "entries": [
    {"r":255, "g":0, "b":0, "color":"red-ish"},
    {"r":254, "g":0, "b":0, "color":"red-ish"},
    {"r":253, "g":0, "b":0, "color":"red-ish"},
    {"r":0, "g":255, "b":0, "color":"green-ish"},
    {"r":0, "g":254, "b":0, "color":"green-ish"},
    {"r":0, "g":253, "b":0, "color":"green-ish"},
    {"r":0, "g":0, "b":255, "color":"blue-ish"},
    {"r":0, "g":0, "b":254, "color":"blue-ish"},
    {"r":0, "g":0, "b":253, "color":"blue-ish"}
  ]
}
 
In your JavaScript : "script.js"
// Step 1: set your neural network options
const options = {
  dataUrl: "data/colorData.json",
  task: 'classification',
  inputs:['r', 'g', 'b'],
  outputs:['color'],
  debug: true
}

// Step 2: initialize your neural network
const nn = ml5.neuralNetwork(options, dataLoaded);

// Step 3: normalize data and train the model
function dataLoaded(){
  nn.normalizeData();
  trainModel();
}

// Step 4: train the model
function trainModel(){
  const trainingOptions = {
    epochs: 32,
    batchSize: 12
  }
  nn.train(trainingOptions, finishedTraining);
}

// Step 5: use the trained model
function finishedTraining(){
  classify();
}

// Step 6: make a classification
function classify(){
  const input = {
    r: 255, 
    g: 0, 
    b: 0
  }
  nn.classify(input, handleResults);
}

// Step 7: define a function to handle the results of your classification
function handleResults(error, result) {
    if(error){
      console.error(error);
      return;
    }
    console.log(result); // {label: 'red', confidence: 0.8};
}
For your reference, a few typical uses are showcased below :
Example 1 :
const options = {
  inputs: 1,
  outputs: 1,
  task: 'regression'
}
const nn = ml5.neuralNetwork(options)
 
Example 2 : loading data as a csv
const options = {
  dataUrl: 'weather.csv',
  inputs: ['avg_temperature', 'humidity'],
  outputs: ['rained'],
  task: 'classification'
}
const nn = ml5.neuralNetwork(options, modelLoaded)
 
Example 3 : loading data as a json
/**
The weather json looks something like:
{"data": [
  {"xs": {"avg_temperature":20, "humidity": 0.2}, "ys": {"rained": "no"}},
  {"xs": {"avg_temperature":30, "humidity": 0.9}, "ys": {"rained": "yes"}}
] }
* */
const options = {
  dataUrl: 'weather.json',
  inputs: ['avg_temperature', 'humidity'],
  outputs: ['rained'],
  task: 'classification'
}
const nn = ml5.neuralNetwork(options, modelLoaded)
 
Example 4 : specifying labels for a blank neural network
const options = {
  inputs: ['x', 'y'],
  outputs: ['label'],
  task: 'classification',
};
const nn = ml5.neuralNetwork(options);
 
Example 5 : creating a convolutional neural network for image classification by setting task: imageClassification.
const IMAGE_WIDTH = 64;
const IMAGE_HEIGHT = 64;
const IMAGE_CHANNELS = 4;
const options = {
  task: 'imageClassification',
  inputs:[IMAGE_WIDTH, IMAGE_HEIGHT, IMAGE_CHANNELS],
  outputs: ['label']
}
const nn = ml5.neuralNetwork(options);
There are a number of ways to initialize the ml5.neuralNetwork. Below we cover the possibilities :
 
* Minimal Configuration Method
* Defining inputs and output labels as numbers or as arrays of labels
* Loading External Data
* Loading a pre-trained Model
* A convolutional neural network for image classification tasks
* Defining custom layers
Minimal Configuration Method : If you plan to create data in real-time, you can just set the type of task you want to accomplish ('regression' | 'classification') and then create the neuralNetwork. You will have to add data later on, but ml5 will figure the inputs and outputs based on the data your add.
 
const options = {
  task: 'regression' // or 'classification'
}
const nn = ml5.neuralNetwork(options)
Defining inputs and output labels as numbers or as arrays of labels : If you plan to create data in real-time, you can just set the type of task you want to accomplish ('regression' | 'classification') and then create the neuralNetwork. To be more specific about your inputs and outputs, you can also define the names of the labels for your inputs and outputs as arrays OR the number of inputs and outputs. You will have to add data later on. Note that if you add data as JSON, your JSON Keys should match those defined in the options. If you add data as arrays, make sure the order you add your data match those given in the options.
 
As arrays of labels : 
const options = {
  task: 'classification' // or 'regression'
  inputs:['r', 'g','b'],
  outputs: ['color']
}
const nn = ml5.neuralNetwork(options)
 
As numbers :
const options = {
  task: 'classification' // or 'regression'
  inputs: 3, // r, g, b
  outputs: 2 // red-ish, blue-ish
}
const nn = ml5.neuralNetwork(options)
Loading External Data : You can initialize ml5.neuralNetwork specifying an external url to some data structured as a CSV or a JSON file. If you pass in data as part of the options, you will need to provide a callback function that will be called when your data has finished loading. Furthermore, you will need to specify which properties in the data that ml5.neuralNetwork will use for inputs and outputs.
const options = {
    dataUrl: 'data/colorData.csv'
    task: 'classification' // or 'regression'
    inputs: ['r', 'g','b'], // r, g, b
    outputs: ['color'] // red-ish, blue-ish
}

const nn = ml5.neuralNetwork(options, dataLoaded)

function dataLoaded(){
  // continue on your neural network journey
  nn.normalizeData();
  // ...
}
Loading a pre-trained Model : If you've trained a model using the ml5.neuralNetwork and saved it out using the ml5.neuralNetwork.save() then you can load in the model, the weights, and the metadata.
const options = {
    task: 'classification' // or 'regression'
  }
  const nn = ml5.neuralNetwork(options);
  
  const modelDetails = {
    model: 'model/model.json',
    metadata: 'model/model_meta.json',
    weights: 'model/model.weights.bin'
  }
  nn.load(modelDetails, modelLoaded)

  function modelLoaded(){
    // continue on your neural network journey
    // use nn.classify() for classifications or nn.predict() for regressions
  }
A convolutional neural network for image classification tasks : You can use convolutional neural networks in the ml5.neuralNetwork by setting the task:"imageClassification".
const IMAGE_WIDTH = 64;
const IMAGE_HEIGHT = 64;
const IMAGE_CHANNELS = 4;
const options = {
  task: 'imageClassification',
  inputs:[IMAGE_WIDTH, IMAGE_HEIGHT, IMAGE_CHANNELS],
  outputs: ['label']
}
const nn = ml5.neuralNetwork(options);
Defaults : By default the ml5.neuralNetwork has simple default architectures for the classification, regression and imageClassificaiton tasks.
 
default classification layers :
layers:[
  {
    type: 'dense',
    units: this.options.hiddenUnits,
    activation: 'relu',
  },
  {
    type: 'dense',
    activation: 'softmax',
  },
];
 
default regression layers :
layers: [
  {
    type: 'dense',
    units: this.options.hiddenUnits,
    activation: 'relu',
  },
  {
    type: 'dense',
    activation: 'sigmoid',
  },
];
 
default imageClassification layers :
layers = [
  {
    type: 'conv2d',
    filters: 8,
    kernelSize: 5,
    strides: 1,
    activation: 'relu',
    kernelInitializer: 'varianceScaling',
  },
  {
    type: 'maxPooling2d',
    poolSize: [2, 2],
    strides: [2, 2],
  },
  {
    type: 'conv2d',
    filters: 16,
    kernelSize: 5,
    strides: 1,
    activation: 'relu',
    kernelInitializer: 'varianceScaling',
  },
  {
    type: 'maxPooling2d',
    poolSize: [2, 2],
    strides: [2, 2],
  },
  {
    type: 'flatten',
  },
  {
    type: 'dense',
    kernelInitializer: 'varianceScaling',
    activation: 'softmax',
  },
];
 
Defining Custom Layers : You can define custom neural network architecture by defining your layers in the options that are passed to the ml5.neuralNetwork on initialization.
 
A neural network with 3 layers :
const options = {
  debug: true,
  task: 'classification',
  layers: [
    {
      type: 'dense',
      units: 16,
      activation: 'relu'
    },
    {
      type: 'dense',
      units: 16,
      activation: 'sigmoid'
    },
    {
      type: 'dense',
      activation: 'sigmoid'
    }
  ]
};
const nn = ml5.neuralNetwork(options);
property description datatype
.callback the callback to be called after data is loaded on initialization function
.options the options for how the neuralNetwork should be configured on initialization object
.neuralNetwork the neuralNetwork class where all of the tensorflow.js model operations are organized class
.neuralNetworkData the neuralNetworkData class where all of the data handling operations are organized class
.neuralNetworkVis the neuralNetworkVis class where all of the tf-vis operations are organized class
.data The property that stores all of the training data after .train() is called class
.ready set to true if the model is loaded and ready, false if it is not. boolean
method description
.addData() adds data to the neuralNetworkData.data.raw array
.normalizeData() normalizes the data stored in neuralNetworkData.data.raw and stores the normalized values in the neuralNetwork.data.training array
.train() uses the data in the neuralNetwork.data.training array to train your model
.predict() for regression tasks, allows you to make a prediction based on an input array or JSON object.
.predictMultiple() for regression tasks, allows you to make a prediction based on an input array of arrays or array of JSON objects.
.classify() for classification tasks, allows you to make a classification based on an input array or JSON object.
.classifyMultiple() for classification tasks, allows you to make classifications based on an input array of arrays or array of JSON objects.
.saveData() allows you to save your data out from the neuralNetworkData.data.raw array
.loadData() allows you to load data previously saved from the .saveData() function
.save() allows you to save the trained model
.load() allows you to load a trained model
You can use neural networks to recognize the content of images. ml5.imageClassifier() is a method to create an object that classifies an image using a pre-trained model.
 
It should be noted that the pre-trained model provided by the example below was trained on a database of approximately 15 million images (ImageNet). The ml5 library accesses this model from the cloud. What the algorithm labels an image is entirely dependent on that training data -- what is included, excluded, and how those images are labeled (or mislabeled).
// Initialize the Image Classifier method with MobileNet
const classifier = ml5.imageClassifier('MobileNet', modelLoaded);

// When the model is loaded
function modelLoaded() {
  console.log('Model Loaded!');
}

// Make a prediction with a selected image
classifier.classify(document.getElementById('image'), (err, results) => {
  console.log(results);
});
Initialize :
const classifier = ml5.imageClassifier(model, ?video, ?options, ?callback);
 
Parameters : 

model : REQUIRED. A String value of a valid model OR a url to a model.json that contains a pre-trained model. Case insensitive. Models available are: 'MobileNet', 'Darknet' and 'Darknet-tiny','DoodleNet', or any image classification model trained in Teachable Machine. Below are some examples of creating a new image classifier:

* mobilenet :
const classifier = ml5.imageClassifier('MobileNet', modelReady);
 
* Darknet :
const classifier = ml5.imageClassifier('Darknet', modelReady);
 
* DoodleNet :
const classifier = ml5.imageClassifier('DoodleNet', modelReady);
 
* Custom Model from Teachable Machine :
const classifier = ml5.imageClassifier('path/to/custom/model.json', modelReady);
 
video : OPTIONAL. An HTMLVideoElement

callback : OPTIONAL. A function to run once the model has been loaded. If no callback is provided, it will return a promise that will be resolved once the model has loaded.

options : OPTIONAL. An object to change the defaults (shown below). The available options are :
{
  version: 1,
  alpha: 1.0,
  topk: 3,
};
.video
* Object. HTMLVideoElement if given in the constructor. Otherwise it is null.
 
.model
* Object. The image classifier model specified in the constructor.
 
.modelName
* String. The name of the image classifier model specified in the constructor
 
.modelUrl
* String. The absolute or relative URL path to the input model.
.classify()

*
Given an image or video, returns an array of objects containing class names and probabilities
 
If you DID NOT specify an image or video in the constructor...
classifier.classify(input, ?numberOfClasses, ?callback);
 
If you DID specify an image or video in the constructor...
classifier.classify(?numberOfClasses , ?callback);
 
Inputs :
 
* input : HTMLImageElement | ImageData | HTMLCanvasElement | HTMLVideoElement. NOTE: Videos can also be added in the constructor and then do not need to be specified again as an input.

* numberOfClasses : Number. The number of classes you want to return.

* callback : Function. A function to handle the results of .segment(). Likely a function to do something with the segmented image.

Outputs :
 
* Object : Returns an array of objects. Each object contains {label, confidence}.
PoseNet is a machine learning model that allows for Real-time Human Pose Estimation.
 
PoseNet can be used to estimate either a single pose or multiple poses, meaning there is a version of the algorithm that can detect only one person in an image/video and one version that can detect multiple persons in an image/video.
 
The original PoseNet model was ported to TensorFlow.js by Dan Oved. 

Quickstart
const video = document.getElementById('video');

// Create a new poseNet method
const poseNet = ml5.poseNet(video, modelLoaded);

// When the model is loaded
function modelLoaded() {
  console.log('Model Loaded!');
}
// Listen to new 'pose' events
poseNet.on('pose', (results) => {
  poses = results;
});
 
Initialize : There are a couple ways to initialize ml5.poseNet.
// Initialize with video, type and callback
const poseNet = ml5.poseNet(?video, ?type, ?callback);
// OR Initialize with video, options and callback
const poseNet = ml5.poseNet(?video, ?options, ?callback);
// OR Initialize WITHOUT video. Just options and callback here
const poseNet = ml5.poseNet(?callback, ?options);
 
Parameters : 
video : OPTIONAL. Optional HTMLVideoElement input to run poses on.
 
type : OPTIONAL. A String value to run single or multiple estimation. Changes the detectionType property of the options. Default is multiple.
 
callback : OPTIONAL. A function that is called when the model is loaded.
 
options : OPTIONAL. A object that contains properties that effect the posenet model accuracy, results, etc.
{
  architecture: 'MobileNetV1',
  imageScaleFactor: 0.3,
  outputStride: 16,
  flipHorizontal: false,
  minConfidence: 0.5,
  maxPoseDetections: 5,
  scoreThreshold: 0.5,
  nmsRadius: 20,
  detectionType: 'multiple',
  inputResolution: 513,
  multiplier: 0.75,
  quantBytes: 2,
};
As written by the developers of BodyPix :
 
"Bodypix is an open-source machine learning model which allows for person and body-part segmentation in the browser with TensorFlow.js. In computer vision, image segmentation refers to the technique of grouping pixels in an image into semantic areas typically to locate objects and boundaries. The BodyPix model is trained to do this for a person and twenty-four body parts (parts such as the left hand, front right lower leg, or back torso). In other words, BodyPix can classify the pixels of an image into two categories: 1) pixels that represent a person and 2) pixels that represent background. It can further classify pixels representing a person into any one of twenty-four body parts."

Quickstart
const bodypix = ml5.bodyPix(modelReady);

function modelReady() {
  // segment the image given
  bodypix.segment(img, gotResults);
}

function gotResults(error, result) {
  if (error) {
    console.log(error);
    return;
  }
  // log the result
  console.log(result.backgroundMask);
}
 
Usage
Initialize : 
const bodyPix = new ml5.bodyPix(?video, ?options, ?callback);
Parameters :

video : OPTIONAL. An HTMLVideoElement

callback : REQUIRED. A function to run once the model has been loaded.

options : OPTIONAL. An object to change the defaults (shown below). The available options are:
{
  multiplier: 0.75, // 1.0, 0.75, or 0.50, 0.25
  outputStride: 16, // 8, 16, or 32, default is 16
  segmentationThreshold: 0.5, // 0 - 1, defaults to 0.5
  palette: {
    leftFace: {
      id: 0,
      color: [110, 64, 170],
    },
    rightFace: {
      id: 1,
      color: [106, 72, 183],
    },
    rightUpperLegFront: {
      id: 2,
      color: [100, 81, 196],
    },
    rightLowerLegBack: {
      id: 3,
      color: [92, 91, 206],
    },
    rightUpperLegBack: {
      id: 4,
      color: [84, 101, 214],
    },
    leftLowerLegFront: {
      id: 5,
      color: [75, 113, 221],
    },
    leftUpperLegFront: {
      id: 6,
      color: [66, 125, 224],
    },
    leftUpperLegBack: {
      id: 7,
      color: [56, 138, 226],
    },
    leftLowerLegBack: {
      id: 8,
      color: [48, 150, 224],
    },
    rightFeet: {
      id: 9,
      color: [40, 163, 220],
    },
    rightLowerLegFront: {
      id: 10,
      color: [33, 176, 214],
    },
    leftFeet: {
      id: 11,
      color: [29, 188, 205],
    },
    torsoFront: {
      id: 12,
      color: [26, 199, 194],
    },
    torsoBack: {
      id: 13,
      color: [26, 210, 182],
    },
    rightUpperArmFront: {
      id: 14,
      color: [28, 219, 169],
    },
    rightUpperArmBack: {
      id: 15,
      color: [33, 227, 155],
    },
    rightLowerArmBack: {
      id: 16,
      color: [41, 234, 141],
    },
    leftLowerArmFront: {
      id: 17,
      color: [51, 240, 128],
    },
    leftUpperArmFront: {
      id: 18,
      color: [64, 243, 116],
    },
    leftUpperArmBack: {
      id: 19,
      color: [79, 246, 105],
    },
    leftLowerArmBack: {
      id: 20,
      color: [96, 247, 97],
    },
    rightHand: {
      id: 21,
      color: [115, 246, 91],
    },
    rightLowerArmFront: {
      id: 22,
      color: [134, 245, 88],
    },
    leftHand: {
      id: 23,
      color: [155, 243, 88],
    },
  },
};​
The U-Net is a convolutional neural network that was developed for biomedical image segmentation at the Computer Science Department of the University of Freiburg, Germany.[1] The network is based on the fully convolutional network [2] and its architecture was modified and extended to work with fewer training images and to yield more precise segmentations.
 
UNET allows you to segment an image.
 
The ml5 unet face allows you to remove, for example, the background from video of the upper body of person.
 
Quickstart
// load your model...
const uNet = ml5.uNet('face');

// assuming you have an HTMLVideo feed...
uNet.segment(video, gotResult);

function gotResult(error, result) {
  // if there's an error return it
  if (error) {
    console.error(error);
    return;
  }
  // log your result
  console.log(result);
}
 
Usage
Initialize : 
const unet = ml5.uNet(model, ?callback);
Parameters : 

* model : A string to the path of the JSON model.

* callback : Optional. A callback function that is called once the model has loaded. If no callback is provided, it will return a promise that will be resolved once the model has loaded.
Handpose is a machine-learning model that allows for palm detection and hand-skeleton finger tracking in the browser. It can detect a maximum of one hand at a time and provides 21 3D hand keypoints that describe important locations on the palm and fingers.
Handpose

Quickstart :
let predictions = [];
const video = document.getElementById('video');

// Create a new handpose method
const handpose = ml5.handpose(video, modelLoaded);

// When the model is loaded
function modelLoaded() {
  console.log('Model Loaded!');
}

// Listen to new 'hand' events
handpose.on('hand', results => {
  predictions = results;
});
 
Usage :
Initialize
You can initialize ml5.handpose with an optional video, configuration options object, or a callback function.
const handpose = ml5.handpose(?video, ?options, ?callback);
Parameters :
* video : OPTIONAL. Optional HTMLVideoElement input to run predictions on.
 
* options : OPTIONAL. A object that contains properties that effect the Handpose model accuracy, results, etc. See documentation on the available options in TensorFlow's Handpose documentation.
const options = {
flipHorizontal: false, // boolean value for if the video should be flipped, defaults to false
maxContinuousChecks: Infinity, // How many frames to go without running the bounding box detector. Defaults to infinity, but try a lower value if the detector is consistently producing bad predictions.
detectionConfidence: 0.8, // Threshold for discarding a prediction. Defaults to 0.8.
scoreThreshold: 0.75, // A threshold for removing multiple (likely duplicate) detections based on a "non-maximum suppression" algorithm. Defaults to 0.75
iouThreshold: 0.3, // A float representing the threshold for deciding whether boxes overlap too much in non-maximum suppression. Must be between [0, 1]. Defaults to 0.3.
}
* callback : OPTIONAL. A function that is called once the model has loaded.
Facemesh is a machine-learning model that allows for facial landmark detection in the browser. It can detect multiple faces at once and provides 486 3D facial landmarks that describe the geometry of each face. Facemesh works best when the faces in view take up a large percentage of the image or video frame and it may struggle with small/distant faces.
Facemesh
The ml5.js Facemesh model is ported from the TensorFlow.js Facemesh implementation.

Quickstart :
let predictions = [];
const video = document.getElementById('video');

// Create a new facemesh method
const facemesh = ml5.facemesh(video, modelLoaded);

// When the model is loaded
function modelLoaded() {
  console.log('Model Loaded!');
}

// Listen to new 'face' events
facemesh.on('face', results => {
  predictions = results;
});
 
Usage :
Initialize :
You can initialize ml5.facemesh with an optional video, configuration options object, or a callback function.
const facemesh = ml5.facemesh(?video, ?options, ?callback);
Parameters :
* video : OPTIONAL. Optional HTMLVideoElement input to run predictions on.
 
* options : OPTIONAL. A object that contains properties that effect the Facemesh model accuracy, results, etc. See documentation on the available options in TensorFlow's Facemesh documentation.
const options = {
flipHorizontal: false, // boolean value for if the video should be flipped, defaults to false
maxContinuousChecks: 5, // How many frames to go without running the bounding box detector. Only relevant if maxFaces > 1. Defaults to 5.
detectionConfidence: 0.9, // Threshold for discarding a prediction. Defaults to 0.9.
maxFaces: 10, // The maximum number of faces detected in the input. Should be set to the minimum number for performance. Defaults to 10.
scoreThreshold: 0.75, // A threshold for removing multiple (likely duplicate) detections based on a "non-maximum suppression" algorithm. Defaults to 0.75.
iouThreshold: 0.3, // A float representing the threshold for deciding whether boxes overlap too much in non-maximum suppression. Must be between [0, 1]. Defaults to 0.3.
}
* callback : OPTIONAL. A function that is called once the model has loaded.
ml5.js has created an API to face-api.js that allows you to access face and face landmark detection.
Face API
The ml5.js implementation of face-api does not support expressions, age or gender estimation.
 
Quickstart :
const detectionOptions = {
  withLandmarks: true,
  withDescriptors: false,
};
// Initialize the magicFeature
const faceapi = ml5.faceApi(detectionOptions, modelLoaded);

// When the model is loaded
function modelLoaded() {
  console.log('Model Loaded!');

  // Make some sparkles
  faceapi.detect(myImage, (err, results) => {
    console.log(results);
  });
}
 
Usage :
Initialize :
const faceapi = ml5.faceApi(videoOrOptionsOrCallback, optionsOrCallback?, callback?);
Parameters :
* videoOrOptionsOrCallback : REQUIRED. Notice there is no question mark in front of the input.
 
* optionsOrCallback : OPTIONAL. Notice the ? indicates an optional parameter.
 
* callback : OPTIONAL. A description of some kind of object with some properties. Notice the ? indicates an optional parameter.
{
  withLandmarks: true,
  withDescriptors: true,
  minConfidence: 0.5,
  MODEL_URLS: {
    Mobilenetv1Model: 'https://raw.githubusercontent.com/ml5js/ml5-data-and-models/main/models/faceapi/ssd_mobilenetv1_model-weights_manifest.json',
    FaceLandmarkModel: 'https://raw.githubusercontent.com/ml5js/ml5-data-and-models/main/models/faceapi/face_landmark_68_model-weights_manifest.json',
    FaceLandmark68TinyNet: 'https://raw.githubusercontent.com/ml5js/ml5-data-and-models/main/models/faceapi/face_landmark_68_tiny_model-weights_manifest.json',
    FaceRecognitionModel: 'https://raw.githubusercontent.com/ml5js/ml5-data-and-models/main/models/faceapi/face_recognition_model-weights_manifest.json',
  },
};
Style Transfer is a machine learning technique that allows to transfer the style of one image into another one. This is a two step process, first you need to train a model on one particular style and then you can apply this style to another image.

Quickstart :
// Create a new Style Transfer Instance
const style = ml5.styleTransfer('data/myModel/', modelLoaded);

// When the model is loaded
function modelLoaded() {
  console.log('Model Loaded!');
}
// Grab a img element and generate a new image.
style.transfer(document.getElementById("img"), function(error, result) {
  img.src = result.src;
});
 
Usage :
Initialize :
const styletransfer = ml5.styleTransfer(model, ?callback);
// OR
const styletransfer = ml5.styleTransfer(model, ?video, ?callback);
Parameters :
* model : The path to Style Transfer model.
* video : Optional. A HTML video element or a p5 video element.
* callback : Optional. A function to be called once the model is loaded. If no callback is provided, it will return a promise that will be resolved once the model has loaded.
Image : Image of drawing converted to the Pokemon character, Pikachu using Pix2Pix trained on Pikachu images.
 
Description : Image-to-image translation with conditional adversarial nets, or pix2pix, is a machine learning technique developed by Isola et al that learns how to map input images to output images.
Pix2Pix
The pix2pix model works by training on pairs of images such as building facade labels to building facades, and then attempts to generate the corresponding output image from any input image you give it. 
 
Quickstart :
// Create a pix2pix model using a pre trained network
const pix2pix = ml5.pix2pix('models/customModel.pict', modelLoaded);

// When the model is loaded
function modelLoaded() {
  console.log('Model Loaded!');
}

// Transfer using a canvas
pix2pix.transfer(canvas, (err, result) => {
  console.log(result);
});
Usage :
Initialize :
const styleTransfer = ml5.pix2pix(model, ?callback);
Parameters :

* model : REQUIRED. The path for a valid model.

* callback : OPTIONAL. A function to run once the model has been loaded. If no callback is provided, it will return a promise that will be resolved once the model has loaded.
You can use neural networks to generate new content. A Generative Adversarial Network (GAN) is a machine learning architecture where two neural networks are adversaries competing. One neural network is a "generator", it makes new images. The other is a "discriminator" and tries to guess if the image is "fake" (made by the generator) or "real" (from the training data). Once the discriminator can no longer guess correctly, the model is trained! A DCGAN is a Deep Convolutional Generative Adversarial Network.
DCGAN
ml5.js provides a few default pre-trained models for DCGAN, but you may consider training your own DCGAN to generate images of things you're interested in.
SketchRNN is a recurrent neural network model trained on millions of doodles collected from the Quick, Draw! game. The SketchRNN model can create new drawings (from a list of categories) based on an initial path.
SketchRNN
Quickstart :
// Create a new SketchRNN Instance
const model = ml5.sketchRNN('cat', modelReady);

// When the model is loaded
function modelReady() {
  console.log('SketchRNN Model Loaded!');
}
// Reset the model's current stat
model.reset();
// Generate a new stroke
model.generate(gotSketch);

function gotSketch(err, result) {
  // Do something with the result
}
Usage :

* Initialize :
const sketchrnn = ml5.sketchRNN(model, ?callback);
* Parameters :

* model : The name of the model to use.

* callback : Optional. A function to be called once the model is loaded. If no callback is provided, it will return a promise that will be resolved once the model has loaded.
The ml5.soundClassifier() allows you to classify audio. With the right pre-trained models, you can detect whether a certain noise was made (e.g. a clapping sound or a whistle) or a certain word was said (e.g. Up, Down, Yes, No). At this moment, with the ml5.soundClassifier(), you can use your own custom pre-trained speech commands or use the the "SpeechCommands18w" which can recognize "the ten digits from "zero" to "nine", "up", "down", "left", "right", "go", "stop", "yes", "no", as well as the additional categories of "unknown word" and "background noise"."

Quickstart :
// Options for the SpeechCommands18w model, the default probabilityThreshold is 0
const options = { probabilityThreshold: 0.7 };
const classifier = ml5.soundClassifier('SpeechCommands18w', options, modelReady);

function modelReady() {
  // classify sound
  classifier.classify(gotResult);
}

function gotResult(error, result) {
  if (error) {
    console.log(error);
    return;
  }
  // log the result
  console.log(result);
}
Usage
Initialize :
const soundclassifier = ml5.soundClassifier(?model, ?options, ?callback)
By default the soundClassifier will start the default microphone.
 
Parameters :
* model : Optional. Model name or URL path to a model.json. Here are some options:
 
* SpeechCommands18w : loads the 18w speech commands
const classifier = ml5.soundClassifier('SpeechCommands18w', modelReady);
* Custom model made in Google's Teachable Machine :
const classifier = ml5.soundClassifier('path/to/model.json', modelReady);
* callback : Optional. A function to run once the model has been loaded.
 
* options : Optional. An object describing a model accuracy and performance. The available parameters are:
{
  probabilityThreshold: 0.7, // probabilityThreshold is 0
};
A pitch detection algorithm is a way of estimating the pitch or fundamental frequency of an audio signal. This method allows to use a pre-trained machine learning pitch detection model to estimate the pitch of sound file.

Quickstart :
const audioContext = new AudioContext();
// const MicStream = MicStream
const pitch = ml5.pitchDetection(
  './model/',
  audioContext,
  MicStream,
  modelLoaded,
);

// When the model is loaded
function modelLoaded() {
  console.log('Model Loaded!');
}

pitch.getPitch((err, frequency) => {
  console.log(frequency);
});
Usage :
Initialize :
const detector = ml5.pitchDetection(model, audioContext, stream, callback);
Parameters :

* model : REQUIRED. The path to the trained model. Only CREPE is available for now. Case insensitive.

* audioContext : REQUIRED. The browser audioContext to use.

* stream MediaStream : REQUIRED. The media stream to use.

* callback : Optional. A callback to be called once the model has loaded. If no callback is provided, it will return a promise that will be resolved once the model has loaded.
RNN and LSTMs (Long Short Term Memory networks) are a type of Neural Network architecture useful for working with sequential data (like characters in text or the musical notes of a song) where the order of the that sequence matters. This class allows you run a model pre-trained on a body of text to generate new text.

Quickstart :
// Create the character level generator with a pre trained model
const rnn = ml5.charRNN('models/bolaño/', modelLoaded);

// When the model is loaded
function modelLoaded() {
  console.log('Model Loaded!');
}

// Generate content
rnn.generate({ seed: 'the meaning of pizza is' }, (err, results) => {
  console.log(results);
});
Usage :

Initialize
const charrnn = ml5.charRNN(model, ?callback);
Parameters :

* model : REQUIRED. An absolute or relative path to the charRNN model files.

* callback : OPTIONAL. A callback to be called once the model has loaded. If no callback is provided, it will return a promise that will be resolved once the model has loaded.
Sentiment is a model trained to predict the sentiment of any given text. The default model, currently 'moviereviews', is trained using IMDB reviews that have been truncated to a maximum of 200 words, only the 20000 most used words in the reviews are used.
 
Quickstart :
// Create a new Sentiment method
const sentiment = ml5.sentiment('movieReviews', modelReady);

// When the model is loaded
function modelReady() {
  // model is ready
  console.log('Model Loaded!');
}

// make the prediction
const prediction = sentiment.predict(text);
console.log(prediction);
Usage :
Initialize
const sentiment = ml5.sentiment(model, ?callback);
Parameters

* model : REQUIRED. Defaults to 'moviereviews'. You can also use a path to a manifest.json file via a relative or absolute path.

* callback :
OPTIONAL. A callback function that is called once the model has loaded. If no callback is provided, it will return a promise that will be resolved once the model has loaded.
word2vec has been disabled : We've intentionally disabled the word2vec function after recognizing it has the potential to produce harmful outputs while using the pre-trained model files included in our examples. We'll consider reenabling the word2vec function along with changes to address these issues in a future release of ml5.js
 
Description : Word2vec is a group of related models that are used to produce word embeddings. This method allows you to perform vector operations on a given set of input vectors.
 
Quickstart :
// Create a new word2vec method
const wordVectors = ml5.word2vec('data/wordvecs.json', modelLoaded);

// When the model is loaded
function modelLoaded() {
  console.log('Model Loaded!');
}

// Find the closest word to 'rainbow'
wordVectors.nearest('rainbow', (err, results) => {
  console.log(results);
});
Usage :
Initialize
const word2vec = ml5.Word2Vec(model, ?callback);
Parameters

* model : A string to the path of the JSON model.

* callback : Optional. A callback function that is called once the model has loaded. If no callback is provided, it will return a promise that will be resolved once the model has loaded.
The ml5 utilities are handy functions that make your life easier when working with data, images, etc.
 
Usage : 

Methods :
.flipImage() : Flips an image or video input horizontally and returns the flipped image. Handy for mirroring an image or video.
const flippedImage = ml5.flipImage(input);
 
Inputs :
 
* input : Optional. A HTMLVideoElement | p5 video element | HTMLImageElement.

Outputs :
 
* Object : Returns a flipped image.

Example :
 
Assuming you're using ml5 with p5.js :
<html>
  <meta charset="UTF-8" />
  <title>flipImage</title>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.9.0/p5.min.js"></script>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.9.0/addons/p5.dom.min.js"></script>
  <script src="http://localhost:8080/ml5.js" type="text/javascript"></script>
  <body>
    <script>
      let video;
      function setup() {
        createCanvas(640, 480);
        video = createCapture(VIDEO);
        video.size(640, 480);
        video.hide();
      }

      function draw() {
        const flippedVideo = ml5.flipImage(video);
        image(flippedVideo, 0, 0, width, height);
      }
    </script>
  </body>
</html>

Sources : Ml5.js, W3C, more..