SOCKET.IO Mendix Implementation
Introduction
About six months ago this question was asked on the Mendix forum. It involved real time communication from server to client and how to accomplish that in a Mendix app. This question has been asked many times before on the forum and the solution that was always suggested involved using a timer to send a request to the database every x number of seconds to check if any of the data has changed. The issues with this method were that the client was responsible for figuring out if any of the data on the current user’s page has changed as well as multiple unnecessary calls to the database would have to be made. The ideal solution would be to call the database only if data has changed, but how do we accomplish this? It wasn’t until Eric posted his answer about working on a POC using the Pusher service to utilize WebSocket’s in order to establish server to client communication, which would tell the client to trigger a microflow to update the user’s screen with the most up to date data. A solution that solved the original question of how to establish real-time communication between server and client in a Mendix application. So, did we make it? Was this problem finally solved? This led to the birth of the official pusher widget in the app store and seemed like a great solution for any project that wants their data to be real time. In Eric’s answer he mentioned that “managing a WebSockets connection also goes against the principle of a Mendix 7 runtime being stateless, so I suspect you'd have a tough time horizontally scaling with this solution. For all of these reasons, I think it's best if your WebSocket connections are made via a non-Mendix server (whether that's a service like Pusher or a server you host using something like Socket.io).” Until that point, I had never heard of Socket.io and was very curious. Being a fan of anything open source, I decided to jump headfirst into exploring a solution that is very similar to Eric’s but instead utilizes Socket.io instead of Pusher.
Let’s walkthrough what I came up with and talk about where it can go.
Goal
The goal would be to call a rest service from a microflow that tells the client to trigger a microflow or nanoflow to update the data that the current user is viewing.
Where to start?
The first question I was faced with was “where to start?” I knew from the pusher implementation I needed a separate server that would interact with the Mendix application, and I needed to deploy an application that uses the Socket.io library. Researching this library, I came across the Socket.io website that provides a great example of how to create a chat application. At first, I thought- fantastic, this would be easier than I thought. After building my chat application I gained a good understanding of how the library worked, and solved the piece of the puzzle of server to client communication, but now on to figuring out how to get our Mendix application to interact with a node js application.
I knew that this tutorial was only half of what I needed, and it was back to the drawing board. I needed to create a node js application, I needed it to be deployed to a separate server, and I needed a way for an external server to interact with it. If I wanted to server to server communication, then what better way to do that then with Rest Services.
Next I needed to determine how to expose a Rest Service for our Mendix application to interact with. After a lot of googling, I came across the Feathers Library. This was by far the most promising library I came across since I started researching Socket.io. This library would be able to create my node js application, handle building all of my rest service endpoints, and setup Socket.io for me. On top of that, the library could handle setting up my authentication, and horizontal scaling-all of which led me to choose Feathers for facilitating this POC.
Feathers
To build out the Feathers application-I started with a quick google search which brought up several different tutorials on how to use Feathers and Socket.io, the one I followed was this one. (It might be helpful to walk through this tutorial to get a better understanding of the Feathers library before continuing on through this guide). The following steps walk through my process for creating the Feathers application:
If you haven’t already, install node JS (node JS can be found here) on your computer and open up a command prompt.
1) First install the feathers cli globally (if its not installed)
2) Install yeoman (if its not installed already)
cd Desktop mkdir feathers-demo cd feathers-demo
3) Move to your desktop and create a folder to put your app in.
4) Then use the generator to create your application and fill out all the details it asks.
Node JS output
5) Next we need to create a service for our Mendix application service to interact with. We can do that using the feathers service command.
Fill out the next questions asking what type of database do you want to use and what your services name will be. I used an in-memory database for this POC. Another option is to use the NeDB database type (which is what a lot of the other tutorials suggested) or mongoose DB. Then name your service and name the path, this is what I named everything:
Feathers service generator output
Once the service is created use the command “npm start”, and test out the rest service. The app is running on http:localhost:3030 and the services database can be viewed at http:localhost:3030/microflow
I used postman to send a post request to make sure it was working. Now let’s quickly setup the rest service call in our Mendix app.
Postman testt
Microflow Service Database
And when I accessed http:localhost:3030/microflow I could see my new record in my database (along with a previous test).
Our service can create and update records in our database. The Feathers documentation has four different service events that can be used in combination with this. This will come in handy when we work on connecting the client side of our Mendix app to our feathers app.
Side note: by default, the Feathers app will only interact with authenticated users. In the src folder there is a file called “channels.js” and on line 47 change “authenticated” to “anonymous.” Eventually this will get changed back when authorization is implemented.
Mendix App
Now that we have our Feathers app built, let’s quickly create a Mendix project to test with. I’m using Mx version 7.23.5 but any Mx7 version will work. Create your app using the blank app template and create an entity in the domain model with an attribute to test with.
Next add a datagrid to the homepage and use the modeler to generate a new/edit page.
Now let’s implement our Rest Service that we created previously. The json structure we are going to use is:
{ "text": "hello world!" }There is one attribute of text and the response will return our text attribute along with a unique id.
{
"text": "hello world!",
"id": 1
}
This will come in handy later.
Create a json structure for the request/response and the respective import/export mappings. Once that is completed, create the initial microflow to send a test message.
Once you test and confirm this is working as expected, it is time to move on to connecting the front end of our Mendix app to our feathers app.
Mendix Widget?
Reviewing the Socket.io and Feathers documentation, the code to connect our client to our Feathers app should be very minimal. The only caveat is that we need the Socket.io library loaded into our front end, and the only way to do that is to create a custom widget.
For our custom widget we want to be able to easily define the URL to our Feathers app, the service we are connecting to, and define a microflow to trigger when an event is triggered. When we create our widget let’s make sure to have those three items to be easily configurable.
Creating a Mendix Widget
If you’ve never created a widget before I recommend following the custom widgets learning path to get a good idea whats happening in the next few steps.
The best way to create a widget is to use the widget generator, but let’s take this a step further and use the ES6 generator so our code comes out a little cleaner and it will be easier to include the Socket.io library. The generator can be found here on github (thanks Jelte!), but we can interact with it using our node.js command prompt.
npm install -g yo npm install -g generator-mendix-widget
1) Open up a new command prompt and install the generator globally.
yo mendix-widget
2) Then run the command (don’t forget to cd to your desktop and create a folder for this widget)
and then answer all of the questions that are asked. This is what I entered:
Mendix Widget Generator Output
npm install socket.io npm install widget-base-helpers –save
3) Now let’s install the socket.io library as well as the widget base helpers.
Once that’s finished, we can open our widget in a code editor, I like to use visual studio code but any editor will work. Navigate to the src folder and find the .xml file where we can define our three parameters. For me, it’s called “socketexample.xml”.
Then remove the default dummykey parameter and use this code below:
<properties>
<property key="service" type="string" required="false" defaultValue="">
<caption>service name</caption>
<category>Details</category>
<description>The service setup in your node js application</description>
</property>
<property key="url" type="string" required="false" defaultValue="">
<caption>service url</caption>
<category>Details</category>
<description>The url to your node js application</description>
</property>
<property key="microflow" type="microflow" required="false" defaultValue="">
<caption>Microflow</caption>
<category>Data</category>
<description>The url to your node js application</description>
<returnType type="Void" />
</property>
</properties>
Next move on to the .js file to start writing the code for the widget. The first thing we want to do is import the socket.io client library, and then connect to the feathers app. From the tutorial on the socket.io website, the code they use is this:
<script src="/socket.io/socket.io.js"></script> <script> var socket = io(); </script>
Looking at the Socket.io documentation we can pass a URL parameter instead of having the io function default to localhost. We can also define our transport type to use websockets instead of polling.
Then we need to look at what types of events are available for our connection to listen too. Referring to the feathers documentation, there is a created event that we can connect too.
socket.on(serviceName + ' created', function(){ } );That way when a new record is created in our database all of our connected clients can be informed with the exact payload that was sent to the Feathers app, and we can execute our own custom logic. In this case we would trigger our configured microflow.
The one set back to this plan is that all clients will be notified if a new record is created in our Feathers app, but we don’t want any data to be refreshed that doesn’t need to be. The way I worked around this was when a record was created in my feathers app, the text attribute of the payload contained the object’s Mendix GUID (the object that was updated), then my client side code would pull the Mendix GUID of the context object and compare the two GUID’s to see if we need to execute the microflow to refresh our data. If the GUID’s didn’t match then the microflow isn’t executed and the user’s data is not updated or refreshed. In this case I’m not sure what the best practice would be, would we have a service for each entity type in our app? Or would we figure out a way to only transmit to specific clients that are registered to one service? Let me know in the comments what you think. In terms of this POC this approach will help us reach our goal of calling a rest service from a microflow that tells the client to trigger a microflow.
Now to move on to how we do this in our Mendix widget. In the .js file I added the code to connect to the Feathers app, and to listen to the created event in the update function. Then I made sure to disconnect the connection in the uninitiliaze() function.
The below output is an example of your code can look like:
import {
defineWidget,
log,
runCallback,
} from 'widget-base-helpers';
import io from 'socket.io-client';
export default defineWidget('socketExample', false, {
socket: null,
constructor() {
this.log = log.bind(this);
this.runCallback = runCallback.bind(this);
},
postCreate() {
log.call(this, 'postCreate', this._WIDGET_VERSION);
},
update(obj, callback) {
//the object the widget is in
this._contextObj = obj;
//connect to the feathers app using websockets as the transport method
this.socket = io(this.url, {transports: ['websocket'], upgrade: false});
const serviceName = this.service;
const microflow = this.microflow;
//listen to the created event for the service that is configured.
this.socket.on(serviceName + ' created', function(message){
log.call(this, 'socket received something: ' + message, this._WIDGET_VERSION);
//guid that was sent in the json request to the feathers app
const messageGuid = message.text;
//conext object guid
const objGuid = obj.getGuid();
//do the guids match? if so execute our microflow
if (objGuid === messageGuid) {
mx.data.action({
params: {
applyto: "selection",
actionname: microflow,
guids: [objGuid],
},
origin: this.mxform,
callback: function(response) {
// no MxObject expected
},
error: function(error)
,
});
}
} );
log.call(this, this.socket.connected, this._WIDGET_VERSION);
if(callback)
,
uninitialize() {
//close the connection
this.socket.close();
log.call(this, 'Uninitialized? ' + this.socket.disconnected, this._WIDGET_VERSION);
},
});
Once that’s done you can build your widget using the command “gulp build.” A copy of the widget will be in the dist folder, and you can move that to your Mendix project.
Testing our widget
Now it’s time to test what we built. Make a slight change to the microflow that you originally built to call our rest service. Add a parameter for the object that you want to use, and then use the community commons java action to get the GUID of the object. Then use that value with your export mapping when calling your rest service.
Then if you haven’t done so already, create new/edit pages for the datagrid on the homepage. Now we can put our new widget on this new/edit page and configure it.
Side note: add the content-type: application/json header to your rest call.
Part 1 widget configuration
Part 2 Widget Configuration
The refresh data microflow has a parameter for the entity object and uses the change object with the setting “refresh in client” set to yes.
Example Refresh Data Microflow
Then we can either replace the default save button with a microflow that commits the entity object, calls the rest service, and closes the page, or we can add an after-commit event handler to call our rest service. Either will work for our case.
Now let’s run our app and open it up in the browser. First create an object to test with. To make sure that the widget is working you can open the dev tools (I use chrome) and enable debugging for the socket.io library. Run this command in the console:
Then click refresh to make sure that your client is connected to our Feathers app. You should see something similar to this:
Navigating the SOCKET.IO Debug Messages
What we want to see here is that the transport is open from the Socket.io-client.
And once that object is created, open up the app again in an incognito browser or a separate browser. In both browsers open the same record. In the first browser make a change to the record and hit save, in the second browser watch how the data is updated without touching the screen.
Conclusion
With all of this built what would be some potential next steps?
· Authentication - Adding authentication to secure our API’s would be a good idea. Feathers supports using JWT for authentication so we can take advantage of the JWT module in the Appstore.
· Scaling - Currently our Feathers app is running on a single thread but Feathers supports horizontal scaling so we can utilize workers to handle multiple requests at once.
· Events – I think some time should be set aside researching the best practice around sending updates to clients from our Feathers app. Right now, we are using one service and every update is sent to every client. We handled this is in our Mendix widget but I’m not sure of the performance impact of this is. A possible solution would to utilize multiple services since they are easy to create.
· Java action – I think our sub microflow that calls our rest service should be replaced with a java action. That way any object can be passed as a parameter to the java action rather than creating a new sub-microflow for every entity type.
Real word applications?
· Any dashboard that you want to be real time without having a user constantly refresh their page or click a button to refresh their data.
· For complex forms that get worked on by multiple people, a user can send updates to a form while another user is filling it out (reloading the data using mxreloadwithstate() client api).
· Locking Mechanism – Sometimes you are faced with a requirement to create a “locked” state for a record. This can be used to update the status of a record on a user’s page without having to constantly refresh the page.
· Chat Application – I made a chat room with this POC, you can check it out here.
If you can think of any other applications for this let me know in the comments.
The example code for everything we just built can be found on my github. Follow these links if you want to check it out:
Widget Example: https://github.com/austinmcnicholas/SocketIO-blog-widget-example
Node App Example: https://github.com/austinmcnicholas/SocketIO-blog-node-app
Mendix Test App Example: https://github.com/austinmcnicholas/SocketIO-Test-App