When I started the Internet of Things research at Progress back in 2015, the resources on the Internet were very limited. The IoT landscape was still in its initial stages and a lot of big players like Amazon and Microsoft had just announced their strategies and IoT platforms.
2lemetry was acquired by Amazon to build their IoT PaaS. 2lemetry was one of the leading IoT platform vendor back then. Carriots is another such platform, IBM had their Bluemix IoT as an attempt to compete in the IoT space and so on, Microsoft IoT was still very new.
Goals of Internet of Things Research
The main motivation behind this research was to find if there is a place for Progress products in the IoT landscape. The primary goal of this project was to gain some insight of the IoT ecosystem, to understand more about what IoT is and how it fits into the bigger picture.
Checking out the possibility of becoming an IoT enabler, possibly in the solutions space was another goal of this project. To define a technology stack (including Progress products and third party open source projects) which can help clients build their vertical specific solutions.
The Real Challenge of such Research Projects
At the time, this was pretty new and people were just talking about it, publishing some white papers and outlining their expectations. It was not easy to agree on some concrete objectives and patterns at least not using the information on the public internet.
A lot of articles on the web talk about the IoT Maturity Model Framework. Gartner divides it into five levels, Level 1 through Level 5. Below is an image from Gartner’s website.
This itself is the biggest challenge when it comes to doing something with IoT. Imagine a situation where we made all our devices capable of transmitting all the data and we also build a solution to store the data. The real challenge is to know what the data means for our business, what can we do to improvise or benefit ourselves from so much of data.
Another challenge is the non-standardization of frameworks or technologies. This is a side effect of our lack of understanding due to nascent stage of the technology.
Few other challenges which adds up to the list are
Non availability of concrete case studies
Multiple skillsets required due to various pieces involved, we will talk about this in detail in coming sections.
With the challenges mentioned above, there is a lot to be done just to have a basic understanding around IoT. If I have to define a simple lifecycle of a IoT solution in the current scenario, it would look something similar to the below image
There are many parts to an IoT solution and they are spread across the complete spectrum of the available technologies around us. A simplified component level diagram may look like the one below. The diagram is taken from Google.
At the bottom lies the devices layer which generates the raw data. This data then travels upwards to a gateway, then a network and up to the IoT platform (also known as the Infrastructure Services). This is the place where the data starts making sense and it is fed to various data processing systems for real time event processing, analytics and many more.
Each IoT solution is built on top of data flowing through these layers and at the end we arrive at the solutions in the top layer (mentioned here as IoT Applications layer).
As a part of the initial study, we looked into few IoT use cases across different verticals. I put here a few of them to demonstrate the possibilities of IoT as a technology.
Disney uses a wearable in form of a wrist band which is the unique identity of the user in the theme park. This band is detected by the sensors in the park, restaurants, hotel rooms etc. Users can access their rooms, eat at restaurants and pay, book a ride and do many more things just using this band.
Virgin Atlantic is preparing for highly connected planes. The concept is to turn the airplane into an IoT device which can transmit data pertaining to fuel consumption, aircraft conditions, passenger information, emergency conditions and a lot more. This helps in diagnostics and a lot of measures can be deployed to address the concerns raised during the flight. In long term the data might help airlines to co-ordinate better and offer state of the art service.
General Electric is working with wearable technologies in the Industrial automation space. The workers can wear devices which can assess the state of assets and their functioning. It can prove to be a wonderful step in protecting human workers by raising early alarms in emergency situations.
Connected cars and self-driven cars are admirable developments. Volvo plans to bring in these features in their cars. Also, they have planned the in car delivery service where goods and packages are delivered directly into the car.
These are few use cases from different industries and as a first step they definitely make a lot of sense. However, these solutions will evolve over time and make IoT more meaningful.
The General Trend
We looked at many more use cases and learnt about the similarities or let say the trend or core functionalities which an IoT solution may need. Here is a diagrammatic representation of our findings.
Areas of interest
As discussed above an IoT solution is a complex system and we had to wisely choose the parts we were interested in. Being a software product organization it was evident that looking into the hardware and network protocols won’t make much sense to our research. The majority of our research was therefore focused around the system when data is received by the infrastructure services layer.
Here, we also skip the device identity and management which can be solved using a secure device registry. We were more interested in the following:
- Data Model
- Service Endpoint
- Event Processing Service
- Big Data Analytics Service
- User Interface Service
- Application Level Security Service
The data generated by different devices can be quite different structurally. One good idea is to define a standard message format which can be super imposed over the raw message and hence it would generate one canonical format irrespective of device types.
The major network protocols which are considered to be industry norms for an IoT solution are MQTT and HTTP. Although we did some research on MQTT but to simplify and not get into the details, let us just expose an HTTP endpoint for receiving the data in any format possible. Node.js was a good choice just for being lightweight, high throughput, event driven, non-blocking framework.
A nodejs endpoint would receive the message and convert the message into a canonical message format. This would standardize the message and hence, the post processing of the message would be much easier task. It perhaps won’t need any special handling for various device types.
Event Processing, Big Data Analytics, UI and Security
Let us discuss about these later in the post once we have a clarity of the IoT architecture.
IoT is a data intensive solution and processing the amount of data generated by millions of devices within short span of time is a challenge. Many designs and architectures have been proposed to handle this work load efficiently. Lambda architecture is one such architecture.
According to Wikipedia, lambda architecture is a data processing architecture designed to handle massive data load and facilitate both stream and batch processing. Alternatively, you can also look at the Kappa Architecture which is a simplification of lambda architecture.
For more details on lambda architecture you can check out this website. The below diagram is the basis of lambda architecture.
The incoming data is fed to a speed layer and a batch layer. The speed layer works off the stream and doesn’t typically maintain a data store. It is the real time event processing engine and answers queries in real time.
However, the batch layer maintains a master set of data which is immutable and then the data serves as the source for service layer which performs big data analytics.
The service layer and the speed layer together feed the User Interface and various external systems. Of course we have twisted this as per our convenience in the research.
The City Tourism Use Case
With enough insight into the IoT world, and to build upon our knowledge we started building a prototype with all our assumptions.
The problem in hand is to build an IoT application for the tourism industry. This was a big use case and hence, we precisely trimmed the problem statement to make it easier to build. As a part of this, we decided to demonstrate real time event processing and the big data analytics.
The detailed blog entry for this project is present here. I would like to add the architecture of the project below:
This is our implementation of the Lambda architecture. An endpoint receives the data and feeds it to two layers, the speed layer at the top and the batch layer at the bottom. Queries from the UI or any consumer system are answered by the service layer. The above diagram combines both the layers.
We specifically used Kafka for the MQ. The event processing is mostly done using Spark Streaming jobs. The batch layer contains a timestamp database (InfluxDB) which works as the master database and Spark is used for Data Analytics.
The service layer which is not clearly visible here contains the Business Logic Processing System and Business Rules for some more complex business logic. We used Rollbase for this part. Even the monitoring systems were build using Rollbase dashboards.
Just to put down all what you might need to create your own IoT solution, here is a complete stack. Of course this consists of few Progress products and services and many open source products.
What about AWS IoT, IBM BlueMix, Microsoft IoT and Carriots?
Now when I search the web, I find a lot of IoT platform vendors who are offering a lot of solutions. At the beginning of my research, the ones which I tried were AWS IoT, Carriots and IBM BlueMix.
AWS IoT (former 2lemetry)
This is really easy to setup and start with. The AWS IoT offers an SDK which to connect, authenticate and exchange messages between devices and the platform. It supports both MQTT and HTTP protocols for communication and AWS Lambda for rules. A shadow device in AWS IoT platform represents a real device in the world. A queue which is spcific to the device receives the data from it.
The AWS Lambda rules are used to filter relevant data from the queue and push it to any data store or API of your choice.
Carriots is another IoT platform which provides similar functionalities. You can create devices make them communicate, write rules and take decisions. Publish results to APIs and devices. And of course they also offer monitoring of the complete system. They are equally easy to work with. It becomes tough because you need to communicate with many third parties in the cloud if you want to save your data in some database or do some lightning fast stream processing etc. In case of AWS it was easy because AWS offers an array of products in their cloud and everything follows a similar development experience.
They claim to offer everything in the same way as AWS IoT, but my development experience has not been very pleasant. You need to create device specific applications and deploy it on the cloud. This platform requires you to write the complete application yourself unlike the other two. This seems to have improved over the last year.
There is a lot of use cases and stories published about Microsoft IoT this year. However, I didn’t get a chance to look into the platform.
Interesting Use Cases
Apart from what I mentioned in previously, there are many more use cases in case you want to try your hands on. The general idea is to tap data, analyze and improvise. May be take decisions to enhance productivity and user experiences.
Here is a link to industry use cases which Microsoft claims to run on their IoT infrastructure. Apart from that I came across some interesting use cases in the consumer segment.
- Parking Management System
- Presence Detection and Promotional Advertisements
It has been a wonderful and successful research which met the primary goals. Understanding the data and making sense out of it is still the biggest challenges. We are overwhelemed by the huge amount of data which has been generated in a short span of time. The maturity model framework from Gartner is perhaps right that many of us are between Level 2 and Level 3.
May be a concrete customer use case and real world data would solidify our understanding. It would also instill more confidence for building concrete IoT solutions.
For any further suggestions or help please contact me at Dharmendra Prasad