How to bring your own data format¶

Introduction¶

This document describes the steps to integrate the MindConnect MQTT broker with the Insights Hub. The key focus of this feature is to enable the ingestion of custom-formatted JSON data into the Insights Hub platform.

This integration empowers you to seamlessly ingest data from your MQTT clients directly into Insights Hub platform, with minimal manual configuration. This streamlines the process of bringing MQTT-based data sources into the Insights Hub by leveraging the platform's advanced analytics and visualization capabilities.

Architecture¶

How to ingest Custom Data via API¶

Topic Registration & Mapping¶

Creating Custom Topics¶

To initiate the process of ingesting custom data, create custom topics on MQTT broker using the POST /customTopicRegistrations API endpoint. This API allows you to register your desired topics. However, it requires specific parameters for successful registration.

Example:

    POST /customTopicRegistrations
    Content-Type: application/json

    {
    "mqttClientID": "<tenant>_Client001",
    "topic": "firstfloor/<tenant>_Client001/timeseries"
    }

Example Request Body¶

    {
    "mqttClientId": "connint6_cdmdemo",
    "topic": "connint6_cdmdemo/OEFw5vE1R8WnL3hM"
    }

Example Response¶

    {
    "id": "O1HX9GVJ2M35SJA63QEMJC53YH",
    "mqttClientId": "connint6_cdmdemo",
    "topic": "connint6_cdmdemo/OEFw5vE1R8WnL3hM",
    "eTag": "O"
    }

Creating Custom Data Mappings¶

After creating custom topics, establish data mappings for your assets using the POST /customTopicRegistrations/{id}/mappings API endpoint. This step binds the MQTT client data to the assets.

• id: The {id} field on URL should be taken from the response body of the create custom topic request (POST /customTopicRegistrations). It uniquely identifies the created custom registration.

    POST /customTopicRegistrations/{id}/mappings
    Content-Type: application/json
    {
    "assetId": "SomeAssetId",
    "aspectName": "SomeAspectName",
    "variableName": "Temperature",
    "variableValueExpression": "[?propertyId=='lorawan-v1–0–8ivhglxiqhw0-hiu-temperature-1urbh5bx6n'].value | [*]",
    "variableTimestampExpression": "[?propertyId=='lorawan-v1–0–8ivhglxiqhw0-hiu-temperature-1urbh5bx6n'].timestamp | [*]",
    "timeConverter": "fromEpoch"
    }

• assetId: The unique identifier of the asset.
• aspectName: The name of the aspect associated with the asset.
• variableName: The name of the variable being mapped.
• variableValueExpression: JMESPath expression defining the transformation rule for the variable value in the customer's JSON format.
• variableTimestampExpression: JMESPath expression defining the transformation rule for the variable timestamp in the customer's JSON format.
• timeConverter: Specifies the method for converting timestamps (Currently we are only supporting 2 time format Epoch & ISO 8601).

Provide the necessary details such as asset id, aspect name, and variable name from Insights Hub, along with JMESPath expressions defining the transformation rules for the variable and timestamp in the customer's JSON format as well as timeConverter.

Example Request Body¶

    {
    "assetId": "7db400264f0d46b69f78e5660fe54d1c",
    "aspectName": "cdmDemoAspectType",
    "variableName": "temperature",
    "variableValueExpression": "[?propertyId=='Distance'].value | [*]",
    "variableTimestampExpression": "[?propertyId=='Distance'].timestamp | [*]",
    "timeConverter": "fromEpoch"
    }

Example Response¶

    {
    "id": "866d1c94-9b35-4b02-b7e9-380a086e2763",
    "assetId": "7db400264f0d46b69f78e5660fe54d1c",
    "aspectName": "temperature",
    "variableName": "cdmDemoAspectType",
    "variableValueExpression": "[?propertyId=='Distance'].value | [*]",
    "variableTimestampExpression": "[?propertyId=='Distance'].timestamp | [*]",
    "timeConverter": "fromEpoch"
    }

Data Ingestion through MindConnect MQTT¶

Once both the custom topic creation and data mappings are successfully completed, you can start ingesting data through the topic you are defined with your own Json. Before ingestion, ensure that the connection with MindConnect MQTT broker is established. For information on establishing quick connection with MindConnect MQTT broker, refer to Obtaining Auto-Generated Agent Certificate.

Except where otherwise noted, content on this site is licensed under the Development License Agreement.