Engineering IoT Solutions with Go

KNOWLEDGE TO POWER THE BUSINESS OF IOT

NEW: IoT PLATFORM RFx TEMPLATE  - CLICK HERE

An early marketing tagline we used at ClearBlade was mainframe to mobile. We had started the process of building our patented, powerful platform that allow real-time access to enterprise systems of record. Fast forward a few years, and we are at a point now where large companies can no longer afford to sit on the sidelines while younger competitors innovate around them. They need developer tools that allow them to take advantage of the latest connected technologies, no matter what their IT architecture looks like.

 

ClearBlade led the way starting in the summer of 2013 to build a platform powerful enough to support enterprise connectivity in all the ways it might manifest - from the Internet of Things (IoT) to the enterprise systems of record. While we considered a number of different tools for the job, Go quickly revealed itself as the only option for a performance-oriented platform addressing these emerging use cases for the enterprise. We were unsure about betting on a new language, but the decision has already paid off in ways that we didn’t expect.

 

Let’s back up a bit, though. After looking across all the end-user requirements that would be desirable in the platform, we began to define the architecture around three core pillars :

 

Data - the ability to store data in a scalable cloud; or connect to existing databases

 

Messaging - the ability to instantly communicate across multiple apps and platforms

 

Code - the ability to write logic in a scalable cloud, or call into existing service layers

 

While Python was our initial choice for building out the data module, it quickly began to fall short of the messaging demands we were asking of it. The parallelism support of Python was not adequate to meet the cloud demands of flexibility, scalability and performance.  We just couldn’t risk performance running Python to handle hundreds of thousands of messages per second that are exchanged in the mobile world for enterprises and government agencies.

 

We needed to build the messaging module around a language that had both out-of-the-box concurrency and parallelism, could be deployed in a cloud environment, had simple code base management, had the ability to run tests, could be easily documented, and, of course, was capable of extreme performance. Strong typing and the ability to profile were also nice to haves.

 

Our next look was at Node.js. It has strong performance metrics and many of the apps in our portfolio run on Javascript. In the end, though, it was our familiarity with Javascript that turned us away. We just couldn’t bear having our core back end maintained in a language with a weak type system, and code that easily turns into complex callback soup.

 

We also looked at Erlang. Not the sexiest language these days, but it does have some strong technical merit when it came to concurrency and stability. Ultimately, Erlang just didn’t make the cut. First our internal team wasn’t readily skilled in functional programming, and the ramp up time was an investment we couldn’t afford.  Secondly the enterprise deployment was a challenge.  Trying to convince a Fortune 500 company to deploy Erlang was an argument we weren’t ready to make.

 

Scala was the language that we were sure would be a natural fit. It’s used by Twitter, so its ability to scale as a cloud language is proven. Many developers even consider Scala to be the next Java, which would make it easy to adopt in enterprise environments. The overall size and feature bloat of Scala made the development and enablement costs too high for us.  Add to that the cost of virtualizing those virtual machines and it became a bit daunting.

 

So we finally looked to Go, a emerging language that we were hesitant to consider at first. It seemed to meet every one of our requirements, though, so we took a risk and decide to dive in. We wrote our message broker implementing the MQTT protocol, the most popular choice for low power M2M communications that epitomize the Internet of Things use cases, for our product.

 

What we ended up with exceeded all of our expectations. Our Go-based messaging module, built in just a few months, easily handles 1 million packets per second on a single node. Deployment time is near instant and the memory profile is extremely low, especially compared to Javascript.

 

We were so impressed with Go, in fact, that we used it for the entire platform. We wrote an RPC module to pass information between modules from REST inputs, wrote an analytics module to capture details about every interaction within the system, and recoded the data module from Python to Go. All of this was done by ClearBlade’s engineering team in under five months. It turns out our fears about working in a new language were totally unwarranted.

 

Best of all, the data module that we rewrote in Go now handles requests 5x faster than our pilot implementation. Sometimes we make lucky decisions, sometimes we make educated decisions. We took care of the lucky part - and encourage you to make an educated decision to look at Go.