DQMH Generic Networking Module

,

Joerg Hampel is a guest blogger. As the owner of Hampel Software Engineering and CLA and LabVIEW Champion, his biggest interest lies in team development best practices.

Ever wanted to use the API of a software module in an application when the module itself is actually running in another application on another device, connected via network? No? Well, imagine these scenarios:

Use cases

Remote control of headless applications

One of the three pillars of my company‘s business is using LabVIEW Real-Time and LabVIEW FPGA to build embedded, network-enabled control devices and device drivers. Most of these applications tend to run as headless, stand-alone units, but all of them need configuration, visualisation or user input from time to time.

We always start out with a Real-Time application that has its own user interface for operating the application in its early stages (and even longer, sometimes). When it’s time to start developing the Windows application that should connect remotely, one often wishes to be able to reuse at least part of the code that’s already running and proved to be working. Even better than reusing would be using, right?

Remote access of (shared) functionality

There is an abundance of examples where you would want or need to access a shared resource remotely:

  • A piece of hardware you need to talk to from a device running LabVIEW Real-Time, but with only drivers for Windows available
  • A service (like a database) living in its own network segment, and IT allows access only from one gateway computer, but you need access from multiple clients
  • A target that you select during runtime, such as when you are having a shared software control panel for multiple devices

In all of these cases, wouldn’t it be great if you didn’t have to worry about where the functionality will be executed? Wouldn’t it be great to just drop the DQMH requests into your block diagram and just use the module’s functions?

Solution

use the API of a DQMH module even if the module itself is running on a different, distributed system

The DQMH Generic Networking Module is a DQMH Singleton module altered to allow for (near) zero-coupled networking functionality. The motivation behind this implementation is to use the API of a DQMH module even if the module itself is running on a different, distributed system. Both the caller and the callee are absolutely oblivious of whether they are running on the same or on different systems.

Forward requests

The DQHM Generic Networking Module can be made a proxy. By loading a NET-TX clone module and redirecting communication, all requests are sent via TCP to another DQHM Generic Networking Module running in another application and/or another PC accessible via TCP/IP.

Process remote requests

The DQHM Generic Networking Module can be made to accept messages via network. By loading a NET-RX clone module, the module accepts messages from other DQHM Generic Networking Modules running in another application and/or on another PC via TCP/IP communication.

Implementation

1. Generalize communication

In order to be able to communicate in a generic way, we need to separate the message data and its type information from the framework’s communication mechanisms. This allows us to send and receive messages without knowing about their actual contents, and that’s the prerequisite for separating the module’s actual use code from the networking code.

All notifications for requests are changed to variant datatype. This allows for generic access to the notifier reference. The actual datatype of the event’s notification lives inside the variant.

Notifier

Helper VIs take care of the Notifier reference management:  HSE_DQMH_Add Notifier To Variant.vi is used in the event handling loop to store the notifier reference as an attribute of the message variant, and HSE_DQMH_Get Notifier From Variant.vi is used in the message handling loop to extract the notifier reference from the message variant.

Set and get the notifier reference

2. Modify Message Queue

A new MessageQueue class inherits from Delacor_Lib_QMH_Message Queue.lvclass, and overwrites the Message Queue to contain additional values:

  • member variable “relay via network?”
  • member variables for network configuration (ip address, port, timeout)
DQMH-GenNet Dequeue Message.vi
DQMH-GenNet Dequeue Message.vi

3. Network Communication

A DQMH Generic Networking module doesn’t have to know about how its messages are forwarded, received or answered.

When configured to forward requests, it dynamically loads a NET-TX clone module, which handles the actual forwarding:

DQMH Generic Networking: Load NET-TX module
DQMH Generic Networking: Load NET-TX module

When configured to receive remote requests, it dynamically loads a NET-RX module, which takes care of receiving messages and forwarding them to the module itself:

DQMH Generic Networking: Load NET-TR module
DQMH Generic Networking: Load NET-TR module

Example: Database Communication

For the following description, let’s imagine we have to talk to a database. We assume that we have two applications: One application is running locally, the other remotely (on another computer), and they are connected to each other via TCP. Only the remote application has access to the database, the local application cannot reach it due to network limitations.

Both applications make use of the same DQMH database module (DB module), but they use it differently:

  • The local application is only using the API of the module. The local DB module forwards all requests to the remote application and also receives replies from the remote application for events with responses.
  • The remote application is oblivious to the fact that its module is used by other applications. The remote DB module serves requests from other applications exactly the same way as requests from within the application it’s running in.

Preconditions

  • The local module is configured to relay messages via network
    • during configuration, a NetworkSender (NET-TX) module is started dynamically
    • the calling module’s message queue object is handed to the NET-TX Start Module.vi
  • The remote module is configured to receive networked messages
    • during configuration, a NetworkReceiver (NET-RX) module is started dynamically
    • the NetworkReceiver creates a TCP Listener and opens a port for incoming messages
    • the calling module’s message queue object is handed to the NET-RX Start Module.vi

Execution Sequence

  1. Local: DQMH Request Event with Notification (eg. Execute Query ( select * from table ))
  2. Local DB module: DQMH_Event_Main sends the message data as variant
    • Notification refnum (datatype variant) is set as an attribute of the message data variant
  3. Local DB module: Delacor_lib_QMH_Dequeue_Message.vi knows that the local module is configured to relay messages and thus calls the case “Send via Network”
    • checks message (string) against blacklist of messages that are not forwarded (eg ‘Exit’)
    • The original message (string) is appended as attribute to the message data (variant)
  4. Local DB module: The case “Send via Network” calls the “Send via Network” request event of the NetworkSender (NET-TX) module
    • The message data (variant) is the payload of the NET-TX’s “Send via Network” request event
  5. Local NetworkSender module: DQMH_MSG_CASES “Send via Network”
    • The original message is flattened to string and sent via TCP/IP
    • TCP read waits for response
  6. Remote NetworkReceiver module: DQMH_EVENT_HELPER spawns a reentrant TCP service VI for each client that connects
    • reads the data from the network and unflattens it to variant
    • If the variant contains an attribute “notifier” (the original one from the local requester), creates a new notifier and overwrites the invalid one in the the received variant
    • reads the message name from the variant attribute
    • sends the message directly via the caller’s message queue to the Remote DB module’s MHL
    • Wait for notification
  7. Remote DB module: DQMH_MSG_CASES
    • executes the called case
    • reads the notifier reference from the variant attribute
    • sends the notification from TCP service VI
  8. Remote NetworkReceiver module: TCP service VI
    • receives the notification from DB.DQMH_MSG_CASES
    • sends the variant that was sent with the notification back via TCP
  9. Local NetworkSender module: DQMH_MSG_CASES “Send via Network”
    • reads data from TCP, which is unflattened to variant
    • If the original message variant contained an attribute “notifier”, the received data is sent via the original notifier refnum back to the request VI
DQMH Generic Networking - Schema
DQMH Generic Networking – Schema

Where to go from here?

If you want to give it a try yourself, you can find the source code on gitlab.com. Pull the repo, check out the example, and be sure to let me know what went well for you and what not so much. Just be aware that this is still a work-in-progress. The code is based on DQMH 4.0, so don’t forget to update your DQMH to the latest version if you haven’t done so already.

“Generic Networking Singleton Module” https://gitlab.com/hampel-soft/dqmh

I’m also very much looking forward to hearing other people’s thoughts on the whole idea and on our implementation so far. So please feel free to leave your feedback in the comments section or contact me directly at joerg@hampel-soft.com. I’d really appreciate it.

Happy generic networking,
Joerg Hampel

12 responses to “DQMH Generic Networking Module”

  1. […] Fabiola De la Cueva (Fab): Not natively, but Joerg Hampell has created a Networking Module template http://delacor.com/dqmh-generic-networking-module/ […]

  2. ted Avatar
    ted

    Joerg,
    I am study your code for a while. Now I can send the message success between the client and server,I use client as a DAQ,server responsible send the configure and control message to it, what if I want to update the data stream from the client to server.Do you know how to realize it
    ? Looking for your reply.

    1. joerg.hampel Avatar
      joerg.hampel

      Teddy, the way our GenNet modules work at the moment, it is a “one way road”. The GenNet client is able to send messages to another module via network, and the GenNet server can receive and answer messages via network, but the server cannot send messages to the client on its own.

      If I understand correctly, you’ve configured a DAQ module to be a GenNet client (the party that sends messages) and another module as a GenNet server (the party that receives messages). In that scenario, the client would have to send a “give me my configuration” request which the server then could answer with the actual configuration. As soon as the client (the DAQ module) starts acquiring, it could send messages with the measurement data to the GenNet server.

      1. teddy Avatar
        teddy

        Joerg,
        Yes! I want build a distributed system(1 server: N clients),clients can be seen as multiple DAQ module,only communicate between server and client. I was use the AMC(UDP) to responsible the messages interact between server and
        clients and use the STM(TCP/IP) to responsible to bring the test data stream back to the server.
        But now I want use the DQMH as the fundamental framework to re-program, it struggle me for a while because I still do not know how to combine DQMH with TCP/IP communication.Do you get any ideas?
        Thanks for your reply!

        1. joerg.hampel Avatar
          joerg.hampel

          Teddy, there are various ways to accomplish what you‘re after. Our Generic Networking modules only cover the TCP/IP part so far. We‘re working on a GenNet UDP implementation which will cater for broadcast scenarios, but that won‘t be finished anytime soon.

          In the meantime, you could use multiple GenNet modules for both directions. Or you could just implement the DAQ streaming yourself by using the TCP or UDP primitives in LabVIEW. If you search on the internet, there are plenty of examples and even some whitepapers for that.

          Sorry I can‘t be of more help.

          1. teddy Avatar
            teddy

            Joerg,
            Really appreciate your patient answer, I have one more question: after GenNet-server received configuration data from GenNet-client,then the DAQ module received the configuration data and start acquire.It is perfectly work.

            After test start some time,if I need to see the continuously
            data from DAQ module,so I send a “give me the real data” request to GenNet-server ,is the the GenNet-server capable to continuously send the measurement data with the messages back to the GenNet-client?I tried add with a Help Loop to repeat this ‘do something with result’ state,but it not work,I must have something misunderstand in your code somewhere.

            Please bare with my poor understanding and explanation.

          2. joerg.hampel Avatar
            joerg.hampel

            Teddy, I am not sure I understand your architecture yet. Do you want to acquire data from multiple DAQ modules and collect it in one place? Or do you want to use the data from one DAQ module in various places?

            after GenNet-server received configuration data from GenNet-client, then the DAQ module received the configuration data and start acquire

            Is the DAQ module actually (using) a GenNet client?

          3. teddy Avatar

            Joerg,
            I want to acquire data from multiple DAQ modules and collect it in one place. I think the DAQ modules are GenNet-clients, because DAQ modules need to send the measurement data to the GenNet-server if the server want. In the sever side, I need listen the data from individual DAQ modules.
            I am sorry I made this confused in last two comments.

          4. joerg.hampel Avatar
            joerg.hampel

            I’m afraid that there is no other way with our GenNet modules than to just send all data, message after message. The “collector” has no way of telling the DAQ modules to start or stop sending data. The DAQ modules need to send all data and leave it to the collector to decide wether to display or discard it. Technically, you would create a request, similar to how you send the “give me my configuration data” (which then returns the config data to the DAQ module in the reply). Anyhow, that should work just fine right out of the box.

            You could also create a separate, proprietary channel (TCP or UDP) with the LabVIEW networking primitives. You’d use the GenNet mechanism for control messages, but stream the actual data over a separate and different network connection.

  3. MD Titoo Avatar
    MD Titoo

    Excellent work!

  4. Antoine Avatar

    Hi Joerg, this module looks really nice!
    Thanks a lot for sharing it, I’ll be playing around with it.

    1. joerg.hampel Avatar
      joerg.hampel

      Hey Antoine, great to hear that you’re looking into it. I hope you can make it work for your use-case.

Leave a Reply to joerg.hampel Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.