eclipse_newsletter/2015/october/content/en_article3.php - gerrit/www.eclipse.org/community - Git at Google

 <?php
 /*******************************************************************************
  * Copyright (c) 2015 Eclipse Foundation and others.
  * All rights reserved. This program and the accompanying materials
  * are made available under the terms of the Eclipse Public License v1.0
  * which accompanies this distribution, and is available at
  * http://eclipse.org/legal/epl-v10.html
  *
  * Contributors:
  *    Eric Poirier (Eclipse Foundation) - Initial implementation
  *******************************************************************************/
 ?>

 <h1 class="article-title"><?php echo $pageTitle; ?></h1>

     <p>If you are as old as I am, and lived in Britain in 1981, you might remember advertisements for a product called "The Last One".  It promised
 	to be the last piece of software you would ever need to buy, as it would write all your programs for you.  Needless to say, its claims
 	were overblown -- I've written plenty of programs since then, all without its aid. (If you want to know more see
 	<a target="_blank" href="https://en.wikipedia.org/wiki/The_Last_One_%28software%29">this page</a>).</p>

 	<p>When I was asked to write an MQTT client library for ARM's mbed platform (<a target="_blank" href="https://www.mbed.com/en/">mbed.com</a>)
 	last year, it brought "The Last One" to my mind, because I've been writing MQTT tools and libraries in C for the past 10 years or more,
 	for IBM, and lately for the Eclipse Paho project.  On the whole, I like that work, but writing yet another C MQTT library was starting to
 	get a bit old.  And I could see that after mbed, there were other embedded devices and operating systems for which an MQTT solution would
 	be needed.  So rather than writing a client library specifically for mbed, I decided to see if I could write a really portable client library,
 	that would work wherever anyone needed it to, but still cater for just about everyone's requirements.</p>

 	<img alt="arcom viper" src="/community/eclipse_newsletter/2015/october/images/arcomviper.png" class="img-responsive" />
 	<p><i>Arcom Viper - one of the first embedded computers to run MQTT, using Java</i></p><br/>

 	<p>The standard C client libraries in Paho have a lineage that stretches back to the early days of MQTT in 2000.  A colleague of mine,
 	Ian Harwood, wrote a portable client library for IBM, called IA93.  The major drawback as I saw it was that the API it presented
 	was not very user friendly.  There was a lot of bit manipulation and buffer mangling going on.  When another colleague of mine,
 	Dave Locke, suggested I write a new C client to go along with my then new MQTT server (Really Small Message Broker - RSMB), I thought I
 	would try to make it look like a user friendly Java API.  But what I also did was write it primarily for Linux and Windows, as it
 	seemed that even embedded devices were consolidating on those operating systems.  And I also threw in the kitchen sink functionally,
 	partly because a committee was involved in defining the requirements.  For mbed and similar platforms, we needed a different approach
 	-- minimalist and portable.</p>


 	<p>I used the Paho mailing list to help me determine what criteria I should adhere to.  As usual on that mailing list, I got some very helpful
 	advice, which resulted in these principles:</p>

 	<ol>
 		<li>No dynamic memory allocation.  Embedded operating systems and devices sometimes have idiosyncratic memory models which make porting
 		difficult.  In any case, we would like to make memory use as predictable as possible.</li>

 		<li>No specific networking calls.  TCP stacks in the embedded world are not standardized - we'd like to copy with any of them without
 		changing the core client code.</li>

 		<li>No operating system specific calls.  No mallocs or TCP calls in the core code obviously helps with this.  We should try to use
 		ANSI standard library calls only, and even minimize the use of them.</li>
 	</ol>

 			<img alt="arcom viper" src="/community/eclipse_newsletter/2015/october/images/arm.jpg" class="img-responsive" />
 	<p><i>ARM mbed IoT Starter Kit for IBM Internet of Things Foundation - using the Paho embedded MQTT client</i></p><br/>

 	<p>To make the libraries as flexible as possible, I decided to structure them into 3 layers.</p>

 	<ol>
 		<li>The lowest level very simple layer, called MQTTPacket, simply serializes data structures into MQTT wire format.  The resulting
 		buffer is ready to be written directly to the network.  It can also accomplish the opposite: deserializing the MQTT wire format back
 		into data structures.  There are some helper functions for reading from and writing to the network, which I originally meant to be
 		just examples, but have become a more integral part of the package since then.  MQTTPacket is very small and simple, and limited
 		in its function.</li>

 		<li>A synchronous, blocking API layer, called MQTTClient, using MQTTPacket underneath.  This was first written in C++, as that is
 		the default language on the mbed platform.  Each MQTT call, like subscribe, or publish, waits for the MQTT protocol exchange to
 		complete before returning.  Thus it is easy to write an application - you don't have to worry about synchronization.  Again,
 		ease of use is paramount in the mbed environment.  Network and OS-specific API calls for timing are relegated to two replaceable
 		classes, so that porting to a different platform does not touch the core code.  Another colleague of mine, Allan Stockdill-Mander,
 		translated this layer into C, for a Texas Instruments environment where C++ was not well supported.  This C version is now called
 		MQTTClient-C.</li>

 		<li>An asynchronous, non-blocking API layer, called MQTTAsync, also built on top of MQTTPacket.  This needs a background thread and
 		syncronization facilities to work, both of which are OS specific and placed in replaceable classes.  As of the time of writing, I
 		have not completed this level, primarily I think because MQTTPacket and MQTTClient seemed to satisfy most people's needs. I'm open
 		to any offers to help me complete this work :-)</li>
 	</ol>

 	<p>Now these libraries in their different forms have been ported to Linux, mbed and Arduino platforms amongst others, and are used in a
 	variety of SDKs, including the recent Amazon Web Services IoT embedded SDK and IBM's IoT Foundation embedded toolkit.  I still have
 	quite a bit of work to do to complete my vision, including getting full support for FreeRTOS and other popular embedded operating systems.
 	However I'm very happy that my original goals seem to be being fulfilled, and that I may not have to write another totally new MQTT client again!</p>

 	<p>For more information, see <a href="https://www.eclipse.org/paho/clients/c/embedded/">https://www.eclipse.org/paho/clients/c/embedded/</a>.
 	If you would like to contribute any work on Paho, please check out the contributing guide at
 	<a href="http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.c.git/about/">http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.c.git/about/</a>.</p>

 <div class="bottomitem">
   <h3>About the Authors</h3>

   <div class="row">
     <div class="col-sm-12">
       <div class="row">
         <div class="col-sm-8">
           <img class="img-responsive"
         src="/community/eclipse_newsletter/2015/october/images/iancraggs.jpg"
         alt="Ian Craggs" />
         </div>
         <div class="col-sm-16">
           <p class="author-name">
         Ian Craggs <br />
         <a target="_blank" href="http://www.ibm.com/iot">IBM</a>
       </p>
       <ul class="author-link list-inline">
         <li><a class="btn btn-small btn-warning" target="_blank" href="http://modelbasedtesting.co.uk/">Blog</a></li>
         <li><a class="btn btn-small btn-warning" target="_blank" href="https://twitter.com/icraggs">Twitter</a></li>
         <?php //echo $og; ?>
       </ul>
         </div>
       </div>
     </div>
   </div>
 </div>
	<?php
	/*******************************************************************************
	* Copyright (c) 2015 Eclipse Foundation and others.
	* All rights reserved. This program and the accompanying materials
	* are made available under the terms of the Eclipse Public License v1.0
	* which accompanies this distribution, and is available at
	* http://eclipse.org/legal/epl-v10.html
	*
	* Contributors:
	* Eric Poirier (Eclipse Foundation) - Initial implementation
	*******************************************************************************/
	?>

	<h1 class="article-title"><?php echo $pageTitle; ?></h1>

	<p>If you are as old as I am, and lived in Britain in 1981, you might remember advertisements for a product called "The Last One". It promised
	to be the last piece of software you would ever need to buy, as it would write all your programs for you. Needless to say, its claims
	were overblown -- I've written plenty of programs since then, all without its aid. (If you want to know more see
	<a target="_blank" href="https://en.wikipedia.org/wiki/The_Last_One_%28software%29">this page</a>).</p>

	<p>When I was asked to write an MQTT client library for ARM's mbed platform (<a target="_blank" href="https://www.mbed.com/en/">mbed.com</a>)
	last year, it brought "The Last One" to my mind, because I've been writing MQTT tools and libraries in C for the past 10 years or more,
	for IBM, and lately for the Eclipse Paho project. On the whole, I like that work, but writing yet another C MQTT library was starting to
	get a bit old. And I could see that after mbed, there were other embedded devices and operating systems for which an MQTT solution would
	be needed. So rather than writing a client library specifically for mbed, I decided to see if I could write a really portable client library,
	that would work wherever anyone needed it to, but still cater for just about everyone's requirements.</p>

	<img alt="arcom viper" src="/community/eclipse_newsletter/2015/october/images/arcomviper.png" class="img-responsive" />
	<p><i>Arcom Viper - one of the first embedded computers to run MQTT, using Java</i></p><br/>

	<p>The standard C client libraries in Paho have a lineage that stretches back to the early days of MQTT in 2000. A colleague of mine,
	Ian Harwood, wrote a portable client library for IBM, called IA93. The major drawback as I saw it was that the API it presented
	was not very user friendly. There was a lot of bit manipulation and buffer mangling going on. When another colleague of mine,
	Dave Locke, suggested I write a new C client to go along with my then new MQTT server (Really Small Message Broker - RSMB), I thought I
	would try to make it look like a user friendly Java API. But what I also did was write it primarily for Linux and Windows, as it
	seemed that even embedded devices were consolidating on those operating systems. And I also threw in the kitchen sink functionally,
	partly because a committee was involved in defining the requirements. For mbed and similar platforms, we needed a different approach
	-- minimalist and portable.</p>


	<p>I used the Paho mailing list to help me determine what criteria I should adhere to. As usual on that mailing list, I got some very helpful
	advice, which resulted in these principles:</p>

	<ol>
	<li>No dynamic memory allocation. Embedded operating systems and devices sometimes have idiosyncratic memory models which make porting
	difficult. In any case, we would like to make memory use as predictable as possible.</li>

	<li>No specific networking calls. TCP stacks in the embedded world are not standardized - we'd like to copy with any of them without
	changing the core client code.</li>

	<li>No operating system specific calls. No mallocs or TCP calls in the core code obviously helps with this. We should try to use
	ANSI standard library calls only, and even minimize the use of them.</li>
	</ol>

	<img alt="arcom viper" src="/community/eclipse_newsletter/2015/october/images/arm.jpg" class="img-responsive" />
	<p><i>ARM mbed IoT Starter Kit for IBM Internet of Things Foundation - using the Paho embedded MQTT client</i></p><br/>

	<p>To make the libraries as flexible as possible, I decided to structure them into 3 layers.</p>

	<ol>
	<li>The lowest level very simple layer, called MQTTPacket, simply serializes data structures into MQTT wire format. The resulting
	buffer is ready to be written directly to the network. It can also accomplish the opposite: deserializing the MQTT wire format back
	into data structures. There are some helper functions for reading from and writing to the network, which I originally meant to be
	just examples, but have become a more integral part of the package since then. MQTTPacket is very small and simple, and limited
	in its function.</li>

	<li>A synchronous, blocking API layer, called MQTTClient, using MQTTPacket underneath. This was first written in C++, as that is
	the default language on the mbed platform. Each MQTT call, like subscribe, or publish, waits for the MQTT protocol exchange to
	complete before returning. Thus it is easy to write an application - you don't have to worry about synchronization. Again,
	ease of use is paramount in the mbed environment. Network and OS-specific API calls for timing are relegated to two replaceable
	classes, so that porting to a different platform does not touch the core code. Another colleague of mine, Allan Stockdill-Mander,
	translated this layer into C, for a Texas Instruments environment where C++ was not well supported. This C version is now called
	MQTTClient-C.</li>

	<li>An asynchronous, non-blocking API layer, called MQTTAsync, also built on top of MQTTPacket. This needs a background thread and
	syncronization facilities to work, both of which are OS specific and placed in replaceable classes. As of the time of writing, I
	have not completed this level, primarily I think because MQTTPacket and MQTTClient seemed to satisfy most people's needs. I'm open
	to any offers to help me complete this work :-)</li>
	</ol>

	<p>Now these libraries in their different forms have been ported to Linux, mbed and Arduino platforms amongst others, and are used in a
	variety of SDKs, including the recent Amazon Web Services IoT embedded SDK and IBM's IoT Foundation embedded toolkit. I still have
	quite a bit of work to do to complete my vision, including getting full support for FreeRTOS and other popular embedded operating systems.
	However I'm very happy that my original goals seem to be being fulfilled, and that I may not have to write another totally new MQTT client again!</p>

	<p>For more information, see <a href="https://www.eclipse.org/paho/clients/c/embedded/">https://www.eclipse.org/paho/clients/c/embedded/</a>.
	If you would like to contribute any work on Paho, please check out the contributing guide at
	<a href="http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.c.git/about/">http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.c.git/about/</a>.</p>

	<div class="bottomitem">
	<h3>About the Authors</h3>

	<div class="row">
	<div class="col-sm-12">
	<div class="row">
	<div class="col-sm-8">
	<img class="img-responsive"
	src="/community/eclipse_newsletter/2015/october/images/iancraggs.jpg"
	alt="Ian Craggs" />
	</div>
	<div class="col-sm-16">
	<p class="author-name">
	Ian Craggs <br />
	<a target="_blank" href="http://www.ibm.com/iot">IBM</a>
	</p>
	<ul class="author-link list-inline">
	<li><a class="btn btn-small btn-warning" target="_blank" href="http://modelbasedtesting.co.uk/">Blog</a></li>
	<li><a class="btn btn-small btn-warning" target="_blank" href="https://twitter.com/icraggs">Twitter</a></li>
	<?php //echo $og; ?>
	</ul>
	</div>
	</div>
	</div>
	</div>
	</div>