Google News
logo
ml5.js Interview Questions
Facemesh is a machine-learning model that allows for facial landmark detection in the browser. It can detect multiple faces at once and provides 486 3D facial landmarks that describe the geometry of each face. Facemesh works best when the faces in view take up a large percentage of the image or video frame and it may struggle with small/distant faces.
Facemesh
The ml5.js Facemesh model is ported from the TensorFlow.js Facemesh implementation.

Quickstart :
let predictions = [];
const video = document.getElementById('video');

// Create a new facemesh method
const facemesh = ml5.facemesh(video, modelLoaded);

// When the model is loaded
function modelLoaded() {
  console.log('Model Loaded!');
}

// Listen to new 'face' events
facemesh.on('face', results => {
  predictions = results;
});
 
Usage :
Initialize :
You can initialize ml5.facemesh with an optional video, configuration options object, or a callback function.
const facemesh = ml5.facemesh(?video, ?options, ?callback);
Parameters :
* video : OPTIONAL. Optional HTMLVideoElement input to run predictions on.
 
* options : OPTIONAL. A object that contains properties that effect the Facemesh model accuracy, results, etc. See documentation on the available options in TensorFlow's Facemesh documentation.
const options = {
flipHorizontal: false, // boolean value for if the video should be flipped, defaults to false
maxContinuousChecks: 5, // How many frames to go without running the bounding box detector. Only relevant if maxFaces > 1. Defaults to 5.
detectionConfidence: 0.9, // Threshold for discarding a prediction. Defaults to 0.9.
maxFaces: 10, // The maximum number of faces detected in the input. Should be set to the minimum number for performance. Defaults to 10.
scoreThreshold: 0.75, // A threshold for removing multiple (likely duplicate) detections based on a "non-maximum suppression" algorithm. Defaults to 0.75.
iouThreshold: 0.3, // A float representing the threshold for deciding whether boxes overlap too much in non-maximum suppression. Must be between [0, 1]. Defaults to 0.3.
}
* callback : OPTIONAL. A function that is called once the model has loaded.
ml5.js has created an API to face-api.js that allows you to access face and face landmark detection.
Face API
The ml5.js implementation of face-api does not support expressions, age or gender estimation.
 
Quickstart :
const detectionOptions = {
  withLandmarks: true,
  withDescriptors: false,
};
// Initialize the magicFeature
const faceapi = ml5.faceApi(detectionOptions, modelLoaded);

// When the model is loaded
function modelLoaded() {
  console.log('Model Loaded!');

  // Make some sparkles
  faceapi.detect(myImage, (err, results) => {
    console.log(results);
  });
}
 
Usage :
Initialize :
const faceapi = ml5.faceApi(videoOrOptionsOrCallback, optionsOrCallback?, callback?);
Parameters :
* videoOrOptionsOrCallback : REQUIRED. Notice there is no question mark in front of the input.
 
* optionsOrCallback : OPTIONAL. Notice the ? indicates an optional parameter.
 
* callback : OPTIONAL. A description of some kind of object with some properties. Notice the ? indicates an optional parameter.
{
  withLandmarks: true,
  withDescriptors: true,
  minConfidence: 0.5,
  MODEL_URLS: {
    Mobilenetv1Model: 'https://raw.githubusercontent.com/ml5js/ml5-data-and-models/main/models/faceapi/ssd_mobilenetv1_model-weights_manifest.json',
    FaceLandmarkModel: 'https://raw.githubusercontent.com/ml5js/ml5-data-and-models/main/models/faceapi/face_landmark_68_model-weights_manifest.json',
    FaceLandmark68TinyNet: 'https://raw.githubusercontent.com/ml5js/ml5-data-and-models/main/models/faceapi/face_landmark_68_tiny_model-weights_manifest.json',
    FaceRecognitionModel: 'https://raw.githubusercontent.com/ml5js/ml5-data-and-models/main/models/faceapi/face_recognition_model-weights_manifest.json',
  },
};
Style Transfer is a machine learning technique that allows to transfer the style of one image into another one. This is a two step process, first you need to train a model on one particular style and then you can apply this style to another image.

Quickstart :
// Create a new Style Transfer Instance
const style = ml5.styleTransfer('data/myModel/', modelLoaded);

// When the model is loaded
function modelLoaded() {
  console.log('Model Loaded!');
}
// Grab a img element and generate a new image.
style.transfer(document.getElementById("img"), function(error, result) {
  img.src = result.src;
});
 
Usage :
Initialize :
const styletransfer = ml5.styleTransfer(model, ?callback);
// OR
const styletransfer = ml5.styleTransfer(model, ?video, ?callback);
Parameters :
* model : The path to Style Transfer model.
* video : Optional. A HTML video element or a p5 video element.
* callback : Optional. A function to be called once the model is loaded. If no callback is provided, it will return a promise that will be resolved once the model has loaded.
Image : Image of drawing converted to the Pokemon character, Pikachu using Pix2Pix trained on Pikachu images.
 
Description : Image-to-image translation with conditional adversarial nets, or pix2pix, is a machine learning technique developed by Isola et al that learns how to map input images to output images.
Pix2Pix
The pix2pix model works by training on pairs of images such as building facade labels to building facades, and then attempts to generate the corresponding output image from any input image you give it. 
 
Quickstart :
// Create a pix2pix model using a pre trained network
const pix2pix = ml5.pix2pix('models/customModel.pict', modelLoaded);

// When the model is loaded
function modelLoaded() {
  console.log('Model Loaded!');
}

// Transfer using a canvas
pix2pix.transfer(canvas, (err, result) => {
  console.log(result);
});
Usage :
Initialize :
const styleTransfer = ml5.pix2pix(model, ?callback);
Parameters :

* model : REQUIRED. The path for a valid model.

* callback : OPTIONAL. A function to run once the model has been loaded. If no callback is provided, it will return a promise that will be resolved once the model has loaded.
You can use neural networks to generate new content. A Generative Adversarial Network (GAN) is a machine learning architecture where two neural networks are adversaries competing. One neural network is a "generator", it makes new images. The other is a "discriminator" and tries to guess if the image is "fake" (made by the generator) or "real" (from the training data). Once the discriminator can no longer guess correctly, the model is trained! A DCGAN is a Deep Convolutional Generative Adversarial Network.
DCGAN
ml5.js provides a few default pre-trained models for DCGAN, but you may consider training your own DCGAN to generate images of things you're interested in.
SketchRNN is a recurrent neural network model trained on millions of doodles collected from the Quick, Draw! game. The SketchRNN model can create new drawings (from a list of categories) based on an initial path.
SketchRNN
Quickstart :
// Create a new SketchRNN Instance
const model = ml5.sketchRNN('cat', modelReady);

// When the model is loaded
function modelReady() {
  console.log('SketchRNN Model Loaded!');
}
// Reset the model's current stat
model.reset();
// Generate a new stroke
model.generate(gotSketch);

function gotSketch(err, result) {
  // Do something with the result
}
Usage :

* Initialize :
const sketchrnn = ml5.sketchRNN(model, ?callback);
* Parameters :

* model : The name of the model to use.

* callback : Optional. A function to be called once the model is loaded. If no callback is provided, it will return a promise that will be resolved once the model has loaded.
The ml5.soundClassifier() allows you to classify audio. With the right pre-trained models, you can detect whether a certain noise was made (e.g. a clapping sound or a whistle) or a certain word was said (e.g. Up, Down, Yes, No). At this moment, with the ml5.soundClassifier(), you can use your own custom pre-trained speech commands or use the the "SpeechCommands18w" which can recognize "the ten digits from "zero" to "nine", "up", "down", "left", "right", "go", "stop", "yes", "no", as well as the additional categories of "unknown word" and "background noise"."

Quickstart :
// Options for the SpeechCommands18w model, the default probabilityThreshold is 0
const options = { probabilityThreshold: 0.7 };
const classifier = ml5.soundClassifier('SpeechCommands18w', options, modelReady);

function modelReady() {
  // classify sound
  classifier.classify(gotResult);
}

function gotResult(error, result) {
  if (error) {
    console.log(error);
    return;
  }
  // log the result
  console.log(result);
}
Usage
Initialize :
const soundclassifier = ml5.soundClassifier(?model, ?options, ?callback)
By default the soundClassifier will start the default microphone.
 
Parameters :
* model : Optional. Model name or URL path to a model.json. Here are some options:
 
* SpeechCommands18w : loads the 18w speech commands
const classifier = ml5.soundClassifier('SpeechCommands18w', modelReady);
* Custom model made in Google's Teachable Machine :
const classifier = ml5.soundClassifier('path/to/model.json', modelReady);
* callback : Optional. A function to run once the model has been loaded.
 
* options : Optional. An object describing a model accuracy and performance. The available parameters are:
{
  probabilityThreshold: 0.7, // probabilityThreshold is 0
};
A pitch detection algorithm is a way of estimating the pitch or fundamental frequency of an audio signal. This method allows to use a pre-trained machine learning pitch detection model to estimate the pitch of sound file.

Quickstart :
const audioContext = new AudioContext();
// const MicStream = MicStream
const pitch = ml5.pitchDetection(
  './model/',
  audioContext,
  MicStream,
  modelLoaded,
);

// When the model is loaded
function modelLoaded() {
  console.log('Model Loaded!');
}

pitch.getPitch((err, frequency) => {
  console.log(frequency);
});
Usage :
Initialize :
const detector = ml5.pitchDetection(model, audioContext, stream, callback);
Parameters :

* model : REQUIRED. The path to the trained model. Only CREPE is available for now. Case insensitive.

* audioContext : REQUIRED. The browser audioContext to use.

* stream MediaStream : REQUIRED. The media stream to use.

* callback : Optional. A callback to be called once the model has loaded. If no callback is provided, it will return a promise that will be resolved once the model has loaded.
RNN and LSTMs (Long Short Term Memory networks) are a type of Neural Network architecture useful for working with sequential data (like characters in text or the musical notes of a song) where the order of the that sequence matters. This class allows you run a model pre-trained on a body of text to generate new text.

Quickstart :
// Create the character level generator with a pre trained model
const rnn = ml5.charRNN('models/bolaño/', modelLoaded);

// When the model is loaded
function modelLoaded() {
  console.log('Model Loaded!');
}

// Generate content
rnn.generate({ seed: 'the meaning of pizza is' }, (err, results) => {
  console.log(results);
});
Usage :

Initialize
const charrnn = ml5.charRNN(model, ?callback);
Parameters :

* model : REQUIRED. An absolute or relative path to the charRNN model files.

* callback : OPTIONAL. A callback to be called once the model has loaded. If no callback is provided, it will return a promise that will be resolved once the model has loaded.
Sentiment is a model trained to predict the sentiment of any given text. The default model, currently 'moviereviews', is trained using IMDB reviews that have been truncated to a maximum of 200 words, only the 20000 most used words in the reviews are used.
 
Quickstart :
// Create a new Sentiment method
const sentiment = ml5.sentiment('movieReviews', modelReady);

// When the model is loaded
function modelReady() {
  // model is ready
  console.log('Model Loaded!');
}

// make the prediction
const prediction = sentiment.predict(text);
console.log(prediction);
Usage :
Initialize
const sentiment = ml5.sentiment(model, ?callback);
Parameters

* model : REQUIRED. Defaults to 'moviereviews'. You can also use a path to a manifest.json file via a relative or absolute path.

* callback :
OPTIONAL. A callback function that is called once the model has loaded. If no callback is provided, it will return a promise that will be resolved once the model has loaded.

Sources : Ml5.js, W3C, more..