Latest revision as of 15:54, 13 September 2022

Computed Measurement

Frequently, incoming metrics represent raw values that are read from the sensing device (eg. door open status, door closed status). To be able to derive better insights and to formulate more complex rules, it is useful to be able to calculate new metrics that are derived from the raw metrics. We refer to these metrics as derived metrics. The set of derived metrics and associated tags form the computed measurement.

Similar to measurement, a computed measurement comprises of:

timestamp
tags
metrics (derived)

Following from the above example, a computed measurement could compute the duration that a door remains opened by taking the timestamp between a door open and door closed metric.

At core of a computed measurement is a computation script that takes in input measurements, performs logic and calculations, and outputs a set of derived metrics as a computed measurement.

Computed Measurement Overview

To use a computed measurement, you have to:

Choose the input measurements
Choose the output metrics
Write a script that calculates the output metrics from the input measurements' metrics

The computed measurement script will be initialized immediately after the computed measurement is created and is active until the computed measurement is deleted. Senfi will execute the script whenever measurement data from the selected input measurements arrive.

Design

The first thing to do when designing a computed measurement is to decide what are the inputs and what to output. A computed measurement can specify multiple measurements as input. Metrics and tags in those measurement will then be available to the function that is responsible for generating the output.

Input Measurements

Any measurement that is available to you in the CMS can be used as an input measurement for your computed measurement.

You should choose only the measurements with metrics that are needed for you to calculate your computed measurement as your input measurements.

Output Measurements

Any data that you want to create with your computed measurement should be packaged in an output measurement. An output measurement has the same composition (metrics, tags, timestamp) as other measurements in Senfi.

When designing the output measurement metrics and tags, should consider the following:

Metrics: The metrics that you want and how to calculate them from your input measurement(s).
Tags: What tags that allow you to differentiate computed measurements derived from different sensors. You can re-use the tags from input measurements or specify different tags depending on the outcome of your computed measurement's computation.
Timestamp: Whether to use the input measurement's timestamp, or the time the computed measurement was output.

Implementation

To generate computed measurements, you will write a script that can process your selected input measurements, perform any calculations or logic necessary, and output the computed measurement data.

The scripting language used to author the computed measurement scripts is JavaScript.

The computed measurement script consists of two functions:

Initialization: Sets up the script
Computation: Processes data from input measurements, performs calculations, and output your computed measurement using the publishMeasurement function.

Both the initialization and computation functions must be present in your script.

init(): Initialization Function

The script initialization function can be used to initialized any data structures you need for logic and state management of your script. It is executed only once when your script is created, modified, or after a system maintenance in which services are restarted.

This is the script initialization function template:

/**
 * @name: init
 * @description: Perform one-time initialization of your script
 * param {string}
 **/
export async function init() {
  // Perform initialization of script here
  // TODO
}

You may choose to leave this function empty if your computed measurement does not need to keep track of data across multiple or consecutive input measurements, such as summation of metric values or checking the difference between a pairs of input measurements.

export async function init() {
  // Initialization not required
}

If your computed measurement needs to keep track of data across multiple or consecutive input measurements, you should create the appropriate data structures here.

You can also declare global variables. For example:

// Declare global variable for a summation value
let sum;

export async function init() {
  // Initialize sum to zero
  sum = 0;
}

compute(): Computation Function

The script computation function is executed every time a new measurement data, or batch of measurement data, from your specified input measurement arrives. In this function, you will read the data from those input measurements, performs any logic or calculations required, and output your computed measurement data.

This is the script computation function template:

/**
 * @name: compute
 * @description: Perform computation on input measurements. This function is called when new measurements arrive
 * @param {string} measurement  - The input measurement name
 * @param {Array.<object>} data - The measurement array data
 * @param {boolean} isBacklog   - true if the data is backlog data. false otherwise
 **/
export function compute(measurement, data, isBacklog) {
  // Perform calculation on data here
  // TODO

  // Example output
  const outputMeasurement = {
    metric1: 1,
    metric2: true,
    metric3: 1,
  };

  // Output computed measurement. See API docs below
  publishMeasurement(outputMeasurement);
}
 
// Output API function
/**
 * @function publishMeasurement
 * @description: User-invoked function to output a computed measurement
 * @param {object} outputMeasurement - Object containing output metrics (required tags, optional tags, measurement metrics)
 * @param {string} outputMeasurement.lsid - [Example] lsid tag (for lifts)
 * @param {string} outputMeasurement.country - [Example] country tag
 * @param {number} outputMeasurement.metric1 - [Example] metric1
 * @param {boolean} outputMeasurement.metric2 - [Example] metric2
 * @param {integer} outputMeasurement.metric3 - [Example] metric3
 **/

Do note that data is an array of measurement data, so typically you would loop through the data input to perform calculation tasks on each variable like this:

export function compute(measurement, data, isBacklog) {    
  // Loop through the array of incoming measurements
  for (let i = 0; i < data.length; i++) {
    const inputMeasurement = data[i];
 
    // Perform calculation and output
    // ...
  }
  // Output
  publishMeasurement(...); 
  
}

At the end of your computations, once you have constructed the output data for the computed measurement, you should call publishMeasurement(), with the output data, to output your computed measurement. This will inform the script engine to ingest your computed measurement into Senfi.

Note: The computation function does not return or output a computed measurement. You must call publishMeasurement() in order to output a computed measurement data to Senfi.

publishMeasurement(): Script measurement output

The script publishMeasurement function is a pre-defined function call that you should invoke whenever you want to output your computed measurement.

You must invoke the publishMeasurement function with a valid measurement data (e.g. all declared metrics and tags) for your computed measurement. Note the output measurement will automatically be attributed to the computed measurement name you have chosen in the CMS. It is not possible for the script to output another computed measurement.

Testing

You can test your computed measurement in the CMS when you are creating or editing the computed measurement.

Your testing data should be identical in format and values to the actual data from your input measurements. You can test with single or multiple sets of data.

The test data format is in JSON, as an Array of Objects of the form:

{
  "measurement":
  "data": []
}

where measurement is the name of the input measurement, and data is a list of one or more measurement data from the input measurement following the Senfi data message format.

Example of sending 1 set of measurement data from the measurement temperature_v1. The script computation function will be executed once during the test:

[
  {
    "measurement": "temperature_v1",
    "data": [{
      "tm_source": xxxxxxxxxx,
      "site_id": xxxxxxxx,
      "tag1": "xxxxxxxx",
      "tag2": "xxxxxxxx",
      "temp": xxxxxxxx,
    }]
  }
]

Abbreviated example of sending a batch of 3 sets of measurement data from the measurement temperature_v1. The script computation function will be executed once during the test:

[
  {
    "measurement": "temperature_v1",
    "data": [{
      ...
      "temp": xxxxxxxx,
    },{
      ...
      "temp": xxxxxxxx,
    },{
      ...
      "temp": xxxxxxxx,
    }]
  }
]

Abbreviated example of sending 3 sets of measurement data from the measurement temperature_v1. The script computation function will be executed 3 times during the test:

[
  {
    "measurement": "temperature_v1",
    "data": [{
      ...
      "temp": xxxxxxxx,
    }]
  },
  {
    "measurement": "temperature_v1",
    "data": [{
      ...
      "temp": xxxxxxxx,
    }]
  },
  {
    "measurement": "temperature_v1",
    "data": [{
      ...
      "temp": xxxxxxxx,
    }]
  }
]

If your script performs computation with multiple input measurements, you can specify multiple measurements. In this example, the computation function will be executed twice, once for temperature_v1 and once for humidity_v1:

[
  {
    "measurement": "temperature_v1",
    "data": [{
      ...
      "temp": xxxxxxxx,
    }]
  },
  {
    "measurement": "humidity_v1",
    "data": [{
      ...
      "humidity": xxxxxxxx,
    }]
  }
]

Execution

After you create or edit your computed measurement in the CMS, your computed measurement script will be compiled and the initialization and computation functions called:

When you finish creating the script: The initialization function will be run
When you finish editing the script: The initialization function will be run
When data from each of your input measurements arrives: The computation function will be run.

Note: If many measurements from the same input measurement arrive within a very short period of time, the measurements from that input measurement may be batched together. For example, if your input measurement input_measurement_1 sends data 10 times per second, you may receive 30 sets of measurements from input_measurement_1 every 3 seconds.

Note: If you have multiple input measurements, each invocation of the computation function will be for only one input measurement. For example, if you have the following input measurements input_measurement_1 and input_measurement_2, expect the computation function to be run twice, once for input_measurement_1 and another for input_measurement_1, for each set of data.

Errors and Debugging

Errors that occur during the initialization or execution of your script will show up in the Debugger in the Developer tab in the CMS.

Console messages logged by your script also appear in the Debugger.

Examples

Example 1: Temperature Scale Conversion

This example shows a simple usage of computed measurement.

Imagine you have a temperature sensor that sends raw temperature values in degrees Fahrenheit (℉) and you want to show the temperature in degrees Celsius (°C) instead. This is the temperature measurement from the temperature sensor that you have as input, where temperatureF is the temperature data in Fahrenheit (℉):

 {
   "tm_source": xxxxxxxxxx,
   "site_id": xxxxxxxx,
   "tag1": "xxxxxxxx",
   "tag2": "xxxxxxxx",
   "temperatureF": xxxxxxxxx
 }

And this is the computed measurement that you want to output, where temperatureC is the temperature metric in °C. You will retain the input measurement's tags in order to correctly attribute the computed measurement to the original temperature sensor:

 {
   "tm_source": xxxxxxxxxx,
   "site_id": xxxxxxxx,
   "tag1": "xxxxxxxx",
   "tag2": "xxxxxxxx",
   "temperatureC": xxxxxxxxx
 }

To convert the temperatureF temperature metric in ℉ into temperatureC in °C, you would use the ℉ to °C formula:

temperatureC = (temperatureF − 32) * 5/9;

As you are just performing conversion when data arrives, there is no need to perform any initialization tasks or declare any global variables. So you can leave the initialization function empty:

export async function init() {
  // No initialization tasks required for this simple example
}

Putting everything together, the full script would be similar to:

/**
 * @name: init
 * @description: Perform one-time initialization of your script
 * param {string}
 **/
export async function init() {
  // No initialization tasks required for this simple example
} 
  
/**
 * @name: compute
 * @description: Perform computation on input measurements. This function is called when new measurements arrive
 * @param {string} measurement - The input measurement name
 * @param {Array.<object>} data - The measurement array data
 **/
export function compute(measurement, data, isBacklog) {    
  // Loop through the array of incoming measurements,
  // and convert 'temperatureF' to 'temperatureC'
  for (let i = 0; i < data.length; i++) {
    const inputMeasurement = data[i];
    const temperatureF = inputMeasurement.temperatureF;
 
    // Calculate the temperature in C
    const temperatureC = (temperatureF − 32) * 5/9; 
   
    const outputMeasurement = {
      "tm_source": inputMeasurement.tm_source, // if not set, tm_source will be taken as the time when the output is produced
      "site_id": inputMeasurement.site_id,     
      "tag1": inputMeasurement.tag1,
      "tag2": inputMeasurement.tag2,
      "temperatureC": temperatureC,
    };  
  
    // Output computed measurement
    publishMeasurement(outputMeasurement);
  }
}
  
/**
 * @function publishMeasurement
 * @description: User-invoked function to output a computed measurement
 * @param {Object} outputMeasurement - Object containing output metrics (required tags, optional tags, measurement metrics)
 * @param {string} outputMeasurement.tm_source - Timestamp (Same as input measurement)
 * @param {string} outputMeasurement.site_id - Site Id (Same as input measurement)
 * @param {string} outputMeasurement.tag1  - Tag1 (Same as input measurement)
 * @param {number} outputMeasurement.tag2 - Tag2 (Same as input measurement)
 * @param {boolean} outputMeasurement.temperatureC - Calculated temperature in C
**/

Example 2: Lift Total Floors Moved

This example shows a more complex usage of computed measurement.

Imagine you have a lift controller that sends floor position values (i.e. 1 when the lift car is at the 1st Floor; 2 when it is at the 2nd floor, etc). You want to use the lift car position data to calculate the cumulative total number of floors it has moved.

For example, if the lift was previously reported to be at the 1st Floor and is now reported to be at the 5th Floor, the total number of floors moved is 5 - 1 = 4. If it is next reported to be at the 7th Floor, the new total number of floors moved is (7 - 5) + 4 = 6. Etc

This is the (abbreviated) measurement lift_controller_v1 from the lift controller that you have as input, where lsid is the lift ID tag, and position is the lift car position data:

 {
   "tm_source": xxxxxxxxxx,
   "site_id": xxxxxxxx,
   "lsid": "xxxxxxxx",
   "position": xxxxxxxxx
 }

And this is the computed measurement that you want to output, where floorsMoved is to cumulative total number of floors moved. You will retain the input measurement's tags in order to correctly attribute the computed measurement to the original lift:

 {
   "tm_source": xxxxxxxxxx,
   "site_id": xxxxxxxx,
   "lsid": "xxxxxxxx",
   "floorsMoved": xxxxxxxxx
 }

As you want to accumulate the total number of floors moved over time, and not just performing per-measurement-data calculations, you will declare a global variable totalFloorsMoved for this purpose:

let totalFloorsMoved;

Your computation function will perform calculation across consecutive lift_controller_v1 input measurements in the computation function. You will declare a global variable prevFloors to store the previous value of the floorsMoved metric, so the next time the computation function is executed on the next lift_controller_v1 input measurement, you can perform the calculation:

let prevFloors;

You can also use the API functions getStoredValue and setStoredValue to store accumulated values to persistent storage. This is so you will not lose the accumulated data if the Senfi undergoes system maintenance where computed measurements are terminated and reinitialized.

For example, to store calculated total floors moved:

let totalFloorsMoved;
// ...
totalFloorsMoved = 9000; // After calculations...
await setStoredValue('floorsMoved', totalFloorsMoved); // Store it!

and to read previously calculated and stored total floors moved:

let _totalFloorsMovedVal = await getStoredValue('floorsMoved'); // Retrieve it!
// _totalFloorsMovedVal = 9000

Thus, the completed initialization function would be:

let totalFloorsMoved;
let prevFloor;
 
/**
 * @name: init
 * @description: Perform one-time initialization of your script
 * param {string}
 **/
export async function init() {
  // Perform initialization of script here
  let _totalFloorsMovedVal = await getStoredValue('floorsMoved');
  let _prevFloorVal = await getStoredValue('prevFloor');
  
  // Return value from getStoredValue is a JSON string. 
  // We need to convert it back to the actual values (an integer) 
  // or null if no value was previously stored
  let _totalFloorsMoved = JSON.parse(_totalFloorsMovedVal);
  let _prevFloor = JSON.parse(_prevFloorVal);  
 
  if (_totalFloorsMoved == null) {
    totalFloorsMoved = 0; // This is the first time we run the script. Set to 0
  } else {
    totalFloorsMoved= _totalFloorsMoved ; // Set to the previously stored value
  }
 
  if (_prevFloor == null) {
    prevFloor = 0; // This is the first time we run the script. Set to 0
  } else {
    prevFloor = _prevFloor ; // Set to the previously stored value
  }
}

Now we need to write the computation logic:

Upon a new batch input measurements, for each individual measurement, calculate the absolute number of floors moved since the last measurement: floors moved = abs(current position - previous position)
Add the absolute number of floors moved to the cumulative total number of floors moved: total floors moved += floors moved
Construct and output the computed measurement
Store current position as previous position, for the next iteration

The computation function would be:

/**
 * @name: compute
 * @description: Perform computation on watching measurements. This function is called when new measurements arrive
 * @param {string} measurement - The source measurement name
 * @param {Array.<object>} data - The measurement array data
 **/
export function compute(measurement, data, isBacklog) {
  // Perform calculation on data here
  for (let i = 0; i < data.length; i++) {
    const d = data[i];

    // On the very first measurement where prevFloor is 0, set prevFloor to current position
    if (prevFloor == 0) {
      prevFloor = d.positon;
    }

    // Calculate the difference in floors moved
    const floorsMoved = Math.abs(d.position - prevFloor) + totalFloorsMoved;
 
    // Constuct the output measurement
    const outputMeasurement = {
      site_id: d.site_id,
      lsid: d.lsid,
      floorsMoved: floorsMoved 
    }; 
 
    // Output computed measurement
    publishMeasurement(outputMeasurement);

    // Store current values
    prevFloor = d.position;
    totalFloorsMoved = floorsMoved;
  }
 
  // Persist global values, to survive script restart
  await setStoredValue('floorsMoved', totalFloorsMoved);
  await setStoredValue('prevFloor', prevFloor);
}