We all have been witnessing a powerful trend where more and more devices are able to communicate via the Internet, and this growing trend is considered as a part of the Internet of the future. In industrial processes these devices are capable of streaming real-time process data. Data is ubiquitous in businesses today, and the volume and speed of incoming data are constantly increasing. However, collecting, processing, and analyzing such streaming data workloads presents a unique set of architectural challenges. It becomes even more challenging when all this must be done in real-time, or near real-time.
In this series of blog posts, I will demonstrate how I architected, and implemented an end-to-end serverless application that collects, processes, analyzes, and presents data by using various AWS services.
The project uses the AWS Serverless Application Model (AWS SAM) to deploy the application in your account. To setup the application, please follow the instructions in README.md file located in GitHub repository.
Note that some of the services are not part of AWS Free Tier, and will incur costs as you run the application.
Example Application Overview: ACME Industries Inc.
ACME Industries is a fictional industrial company which has one facility that runs short 10 minute processes as part of its overall production process. There are two types of frontend applications that allow employees to view data emitted by various sensors during running process:
- Admin application – allows users to view sensor data, issue commands, and configure sensor types.
- Operator application – allows users to only view sensor data, and issue commands.
Admin application
The admin application runs on localhost, and when users navigate to it, they see the welcome page:
By clicking on the “‘LOGIN” button, user is navigated to the login page:
Note that if this is a first-time user, she will need to enter the temporary password that was issued to her, and then enter her own password when prompted. The detailed instructions on how to create user accounts, and add them to a Cognito user pool are provided in the Readme.md file.
After successful login, user is navigated to the main dashboard:
The top section of the dashboard displays the overall state of the facility:
- Status provides the current status of the production process. It can be in a number of states: IDLE, RUNNING, PAUSED, STOPPED, COMPLETING, COMPLETE, IDLE-OFLINE, STOPPED-OFFLINE, COMPLETE-OFFLINE. I will provide more details on each of the states later in this series.
- Total runtime (min) is the total time in minutes of the process run. Currently it is hardwired to 10 minutes.
- % Complete shows the completed percentage of the running process.
The middle section of the dashboard has two tables:
- Realtime Data shows real-time values for each sensor.
- Stats by latest minute shows sensor data statistics for the last 30 seconds, and it gets updated every 30 seconds.
The bottom section of the dashboard also has two tables:
- Daily Stats shows cumulative data stats from the start of the running process.
- Completed Process Stats has a dropdown which lists all completed processes. User can select one from the list to see sensor data stats for the selected process.
To launch a new process, user clicks on “LAUNCH FACILITY” button:
- The “Status” panel changes to “RUNNING”, user can pause, or stop the running process
- The “Realtime Data” table starts showing the simulated sensor data almost immediately
- The “Stats by latest minute” table refreshes sensor stats in the last 30 seconds window
- The “Daily Stats” table refreshes sensor stats about every minute. Note that this refresh time depends on overall volume of data and the configuration of Kinesis Firehose data buffering timeout
- After running a few processes to completion, the “Completed Process Stats” panel will show a list of completed processes:
Commands, actions and states
There are several commands that user can issue when running a facility process. Each of the commands changes the state of the facility. The table below shows various commands that user can issue from the UI and the corresponding state that the facility/process transitions to:
Below I provided several snapshots of the top section of the dashboard with command buttons and corresponding facility statuses:
In addition to operational commands, user can navigate away from the dashboard by either selecting the “CONFIGURE” or “LOGOUT” buttons on the application bar, or just reloading the page, and that will put the facility offline. The offline state of the facility will depend on the state it was in before going offline:
Note that going offline while in an active state like RUNNING or COMPLETING can only happen if user reloads the page. These offline states are particularly important to the Operator application as it stops receiving sensor data updates from the Admin node.
Configuration page
Users can navigate to the configuration page by clicking on the “CONFIGURE” button on the application bar:
The configuration page allows users to set the most basic properties for each sensor type: min, max, and total number. By default, there are three sensor types – temperature, pressure, and humidity – with the following attribute values for each type:
- Min value – 0
- Max value – 100
- Total number – 5
Users can modify each of the values by selecting sensor type in the table, enter new value, and then hit Enter key. The “Save” button gets enabled, and user can save the updated attribute values:
Please note that, since the sensor data simulator is built into the application, it runs in the browser, and doesn’t scale well. I tested it with up to 15 instances for each sensor type (45 total). I decided to address overall scalability issues of the architecture in the next series of articles. Also, to better understand the data flow in the application, I suggest to keep the overall number of configured instances small (15 total or less , for example). This will also help keeping the cost of running the application low.
Operator Application
The Operator application allows facility operators to view the same operational data on the dashboard as well as to issue the same commands as from the Admin application. Note that operators don’t have access to the configuration page.
Unlike the Admin application which runs on a single localhost node, the Operator application is deployed to CloudFront CDN. Its responsive UI design allows operator access from various types of devices: desktops, laptops, or mobile devices:
Here are a few snapshots from an iPhone that demonstrate responsive UI of the Operator application:
Architecture overview
The project uses several AWS services: Amazon Cognito, AWS IoT Core, Amazon Kinesis, AWS Lambda, Amazon DynamoDB, Amazon S3, Amazon API Gateway, and Amazon CloudFront:
- Amazon Cognito provides authentication, authorization, and user management. It consists of two main components: user pools and identity pools. Employees that operate ACME Industries facility are added to user pool in order for them to be able to sign in. The Identity pool enables authenticated employee access to other AWS services.
- AWS IoT Core lets large number of client applications (connected devices) securely interact with cloud applications through publish-subscribe capabilities.
- Amazon Kinesis is a fully managed service which makes it easy to collect, analyze, and process streaming data. Kinesis Data Streams is a scalable real-time data streaming service. Kinesis Data Firehose makes it easy to capture, transform, and load batches of streaming data into data stores like Amazon S3 in near real-time.
- AWS Lambda enables building event-driven functions that work with Kinesis, S3, and DynamoDB to perform custom processing of streaming data, writing data into DynamoDB table, or publish data to an IoT topic.
- DynamoDB is a highly scalable and performant NoSQL database. ACME Industries application writes statistics of cumulative data for each sensor of a running process into a database table. It also stores statistics for each sensor of a completed process.
- Amazon S3 is used by Kinesis Firehose to store batches of streaming sensor data of a running process.
- AWS API Gateway exposes REST API which allows both Admin and Operator clients to query for a list of completed processes as well as sensor data stats for a selected process.
- Amazon CloudFront is a content delivery network (CDN) that is globally distributed and massively scaled. The Operator application can be deployed to CloudFront. The setup instructions are provided in the Readme.md file.
The backend architecture of ACME Industries application is presented in the following two diagrams:
There are two data collection layers, and four distinct functional areas. Each area supports a specific type of data functionality used by both Admin and Operator frontend applications:
- Ingestion layer – in the current implementation the production facility is considered an IoT device which sends data every second for each sensor to an IoT topic. An IoT rule is configured to route this sensor data to Kinesis Data Streams.
- Near real-time data collection layer – in ACME Industries application, Kinesis Data Firehose is configured as a consumer of Kinesis Data Streams. It is used to process the incoming batches of sensor data records, and to store them in S3 bucket. The data then used to calculate cumulative stats of a running process, as well as sensor data stats of a completed process.
- Real-time sensor data – a Lambda function is configured as a consumer of Kinesis Data Streams. It sorts out sensor data by process ID from the payload, and then publishes data to an IoT topic.
- Latest minute data analytics – sensor data statistics are calculated for every 30 seconds of streaming data per running process. This data functionality is implemented in two different ways just to demonstrate two different options of implementing this use case:
- (a) Lambda function is configured as another consumer of Kinesis Data Streams. It receives sensor data every second. The tumbling window feature of Lambda allows invocations from Kinesis Data Streams to pass state between invocations. In ACME Industries, the window is configured to be 30 seconds, and it allows the function to buffer the aggregated sensor data between invocations. The function calculates statistics, and publishes results to an IoT topic at the end of the tumbling window.
- (b) Using Kinesis Data Analytics for SQL Applications as one of the consumers for the data stream. The Kinesis data analytics application – ‘sensor-stats’ – continuously reads streaming data. The application executes time-based windowed query in a 30 second tumbling window.
- Cumulative daily stats – after Kinesis Data Firehose stores an incoming batch of sensor data in S3 bucket, it triggers a Lambda function to merge the new sensor data batch of a running process into an S3 bucket of cumulative daily data. As a result of an update, a Lambda function is invoked to calculate cumulative statistics for each sensor which then are saved to a DynamoDB table. A DynamoDB stream invokes a Lambda function to publish these new cumulative statistics back to the frontend via an IoT topic.
- Completed process stats – the following steps describe the overall implementation:
- Similar to how the cumulative daily stats are merged and stored in S3 bucket, the same Lambda function first checks for records with “complete” events in sensor data batch of a running process.
- The “complete” event tells the Lambda function that the process is complete, and the merged data is saved in historical data S3 bucket.
- As a result of S3 object update, a Lambda function is invoked to calculate statistics for each sensor which then are saved to a DynamoDB table.
- A DynamoDB stream invokes a Lambda function to publish completed process ID and facility ID back to the frontend via an IoT topic.
- The frontend updates its list of completed processes.
- The frontend can query for sensor stats of a selected process via API Gateway.
Deploying ACME Industries application
The Readme.md file has detailed deployment instructions for the example application. The code project in the GitHub contains separate directories based on the functional areas of the application: ingestion layer, Kinesis consumer data functionality, as well as the Admin and Operator frontend applications.
Please note that such services as Kinesis Data Streams and Kinesis Data Firehose are not included in AWS Free Tier, and will incur costs when running the example application. By having relatively small number of simulated sensor instances (not more than 15, for example), will keep the costs low.
It is also important to follow the cleanup instructions after you are done working with the application. For example, forgetting to cleanup Amazon CloudWatch log files could be costly if you have large number of sensor instances running, and have logging enabled in the code.
Conclusion
In this post I introduced the ACME Industries application for processing, analyzing, and viewing streaming sensor data. I described the two frontends that allow users to view various data functionalities. I also provided an overview of the application architecture and its deployment.
Part 2 covers the ingestion layer for streaming sensor data with AWS IoT Core and Amazon Kinesis Data Streams.