Monday, December 22, 2014

More new stuff - IBM Cloudant

Looking at IBM Cloudant..

Cloudant is a nosql database.  I don't know what that is exactly, so let's figure that out first.

It turns out that instead of storing data in rows and tables, as we would in a relational database, everything is stored as JSON objects in a NoSql database.

So, my Map Markers would fit nicely into this paradigm.    This is what JSON data looks like.  Everything is more or less self documenting.    This represents 6 records.  Each record has a type, and id, a lat, a lng, and a time.  I should be able to just poke them into a Cloudant database.

{"type":"marker","id":"User_38","lat":3.8908132995967E+1,"lng":-7.881591796875E+1,"time":1418995553937}
{"type":"marker","id":"User_38","lat":3.61733569352216E+1,"lng":-7.894775390625E+1,"time":1418995554521}
{"type":"marker","id":"User_38","lat":3.4016241889667E+1,"lng":-8.033203125E+1,"time":1418995555218}
{"type":"marker","id":"User_10","lat":4.16893222599704E+1,"lng":-6.6785888671875E+1,"time":1418995570282}
{"type":"marker","id":"User_26","lat":4.19247597193397E+1,"lng":-6.81836791522801E+1,"time":1418863150094}
{"type":"marker","id":"User_26","lat":4.19247597193397E+1,"lng":-6.81836791522801E+1,"time":1418863150096}


Cloudant is more than a NoSQL database.  It is industrial strength, cloud based, highly reliable, clusterable.  And it's more or less free if your monthly costs are low enough.

It is built on top of CouchDB, and supports Mongo Style queries.  It also has a Lucene search capability, Geospacial queries, etc.  So it's worth looking at, over and above what CouchDB provides.

We'll need to take a look.

IBM's Internet of Things Foundation

I'm always amazed at what IBM offers.  IBM hosts MQTT services in the cloud.  We just happen to have been discussing MTQQ recently, so this topic fits right in.

IoT Foundation is IBM's Cloud Based MQTT service - https://developer.ibm.com/iot/recipes/improvise-application-development/

And IoT Foundation is available on BlueMix.  So maybe we'll start there.

Logging into BlueMix, I'll add the IoT Service to my existing application, MQTTMaps.  that's convenient because I'm emitting MQTT messages anyway.

When I do that, BlueMix asks if I want to rehost in order to take advantage of IoT... Sure...

IoTF seems to be about Devices and about Organizations.  It wants me to register each of them.  Let's see...

My Device is an MQTTMap Marker, and there might be different kinds in the future depending on the mouse click.

It provides some predefined devices from major manufacturers:

  • ARM mbed device
  • Texas Instruments BeagleBone with Sensor Tag device
  • Texas Instruments SimpleLink CC3200 device
  • Intel Galileo device
  • Raspberry Pi Model B device
  • Arduino Uno device
  • Connect my own registered device
  • Connect other devices  
That's fine.  I think I'll be using Connect Other Devices.

And then BlueMix wants me to create a Node-Red application.

After that it says that I can visualize my devices... so I want to see how that's done...

And then Analytics and Hadoop... 

There's some cool stuff here...

I need to get started...

Thursday, December 18, 2014

Getting started with BlueMix

BlueMix is IBM's devOps in the cloud.  We could call it full featured and not be far off.  There are lots of starting points, templates for projects that can jump start your development.

You might want to have Git and CloudFoundry installed on your machine before you start.  Ant and Maven might be helpful too.

The initial view into BlueMix that I want to talk about is the dev Dashboard.  This shows your apps.  We'll start with a clean slate.  I have no apps and no services yet.  I'm going to start by clicking on Create An App.  From there I select a Runtime.  I'll use the WebSphere Liberty Java. Then it asks for an app name.  I'll call this MQTTMaps.  And my host access URL will be MQTTMaps.mybluemix.net.  Then I'll click Create.

Now the Dashboard view has changed.  It shows me my MQTTMaps application,  I can get back to the dev Dashboard by using the Back to Dashboard button at the top left.

Now it tells me I can add a service, which we might do in a bit.  I can also add a git.  That will give me access to the source code via a git repository.  Mow I can edit code.  But first, let's run the out-of-the-box app and see what it does.  So I go to http://mqttmaps.mybluemix.net/ and in there it shows me a link to some documentation.  If I go back to the dashboard page, there's a View Quick Start menu option.  It tells me how to get started, how to download my app starter kit, install CloudFoundry, and do some basic cloudfoundry steps.  Then it says to make edits and build a war file, and then deploy the war file.  So we'll try that.

I'll just copy in my web content files from the MQTT demo I've been working on,

Next, find Ant, and set ANT_HOME, and then JAVA_HOME has to point to a jdk. copy a command prompt, cd to my project and run ant, and it looks like it's built.

Now let's push... using cf

give it time to restart...

and... yeah... perfect... I will have to use this to demo to the group tomorrow.,






Wednesday, December 17, 2014

Getting Started with Decision Insights

Decision Insights is included in the new IBM ODM Advanced V8.7.  I've down loaded the tar file and uncompressed it to my product install drive in the DSI87 folder.  I'll point my install manager there and install to C:\IBM\DI87

This blog entry is just about installing and starting it up in relation to the previous posts about the demo I'm building.

When that's complete I'll want to edit the server.xml file located at:
C:\IBM\DI87\runtime\wlp\usr\servers\cisDev\server.xml and the bootstrap.properties in the same folder,  and modify some ports so I don't conflict with other stuff I have running.

http.port=19081
https.port=19444
ia.listenerPort=12810

And maybe change the default username and password. :)

And then also uncomment and give it the path to the MQ library it needs:
<variable
name="wmqJmsClient.rar.location"
value="C:/Program Files (x86)/IBM/WebSphere MQ/java/lib/jca/wmq.jmsra.rar" />


Then we'll back up to the server bin folder and issue a server start cisDev command.  It starts in a few seconds.  Sweet!

A look at the logs and I see that the server starts with a minimum of warnings.
1) It wants a password to create a keystore  I'll have to change admin to something else...
2) The HostName is not fully qualified.

But aside from those we have a clean start!






Getting to know WebSphere MQ

Now that I have MQ queues set up to collect the traffic from my MQTT clients, and I have a simple IIB application that stores that data to a file system, my next step is to connect WebSphere Application Server 8.5.5 to MQ so it can subscribe to topics and queues.

But when I go to the WAS admin console and try to create an JMS Activation Spec, I find that WAS cannot connect to MQ.   Why is that?

Admin Console says:  A connection could not be made to WebSphere MQ for the following reason: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2035' ('MQRC_NOT_AUTHORIZED').

So first we need to look in the MQ logs.  There are three of them.  They are located here:

C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\IB9QMGR\errors
C:\Program Files (x86)\IBM\WebSphere MQ\Qmgrs\@SYSTEM\errors
C:\Program Files (x86)\IBM\WebSphere MQ\errors

The first one has pay dirt.

12/17/2014 18:59:26 - Process(8804.3) User(MUSR_MQADMIN) Program(amqrmppa.exe)
                      Host(ADMINIB-GC3H285) Installation(Installation1)
                      VRMF(7.5.0.1) QMgr(IB9QMGR)
                   
AMQ9777: Channel was blocked

EXPLANATION:
The inbound channel 'SYSTEM.DEF.SVRCONN' was blocked from address '127.0.0.1'
because the active values of the channel matched a record configured with
USERSRC(NOACCESS). The active values of the channel were 'CLNTUSER(DeveloperHat)'.
ACTION:
Contact the systems administrator, who should examine the channel
authentication records to ensure that the correct settings have been
configured. The ALTER QMGR CHLAUTH switch is used to control whether channel
authentication records are used. The command DISPLAY CHLAUTH can be used to
query the channel authentication records.

--
So, as I recall, there was some security feature that was turned on by default in MQ 7.1 and I haven't played with MQ in this way since 7.0 I think, maybe earlier.

First we have to remember how to run MQ commands.  Do that by going to the MQ bin folder and running runmqsc QMGR, where QMGR is my queue manager name.  That opens a command window.

And then, for now, turn off channel security, which of course is not recommended for anything but your own personal test server, by issuing the command:  ALTER QMGR CHLAUTH(DISABLED)

--
And now WAS 8.5.5 can define Activation Specs and Factories to its heart's content.










MQTT to Message Broker IIB

So now that I have MQTT working on a WebSphere MQ Queue Manager, and once you figure out how to do it it is simple.  The next step is to subscribe to those MQTT topics in order to have queues read from them.

The gist of it is that you create a new local topic first, using the MQ Explorer, When you create that topic, the topic string in the topic has to match the topics that are being published to MQTT.  My MQTT clients in this demo are publishing to the topic /World.  So if I want to set up an MQ Topic that mirrors that, I just say its topic string is /World. No need for a subscription between MQTT and the MQ Topic.  MQ just magically maps between the two.

Then you create a subscription to that WebSphere MQ topic from a local queue, and then MQ has access to that queue.

The Message Broker App I've written reads the queue and write to a file.  It looks like this, just a queue input node and a file writer output node, tied together.  I set the message parser to JSON just in case I start to process messages in some later part of this.






Tuesday, December 16, 2014

Getting Comfortable with BlueMix

So today I wanted to take my existing web application / demo that I've been working on, and move it up to BlueMix.  I currently have the project stored in an eclipse workspace on my hard drive as an eclipse project.  What I want to do is store that project in BlueMix.

The first step is to create a project in BlueMix.  There are a few ways to do this.  You can start with a pre-built template.  Or you can just create a project.  A pre-built template has some advantages, but also has a learning curve.  You have to understand what the template is providing and how to modify it to meet your needs.

So I'll go with an empty project.  From BlueMix. Click on My Stuff. From there click on Create Project.  This will create an empty project and give you a link to a git repository.

Back on the workstation, go to your eclipse project folder.  From the command line do a 'git init', and then a 'git add -all', and finally a 'git remote bluemix {add the url to your bluemix repository}'.  Then 'git commit', and 'git push bluemix master', and your project will be up in JazzHub where you can share it with others.

At this point though, there's no build process for your project up in BlueMix.  That means, while all of your stuff is technically up there, BlueMix does not know how to build and deploy it.  Hmmm... that's a problem for another day.  

MQTT on WebSphere MQ

I wanted to use MQTT hosted on WebSphere MQ for my project.  It is an add on. Although it is shipped with the product, it is not installed by default.  So, the first step is to fire up the MQ installer Setup.exe and select "Maintain or upgrade an existing instance", and add the two MQTT components.

After that, you need to run a script to create an MQTT queue manager, and a couple of channels.  One for MQTT traffic, the other for HTTP WebSocket traffic.  They provide a sample file, but it does not include the HTTP WebSocket channel.  So, you'll have to add that.

The Sample file is located here:  C:\Program Files (x86)\IBM\WebSphere MQ\mqxr\samples
and it is called SamplesMQM.bat  or .sh if you're of that ilk.

Copy it, and add these lines at the bottom:
rem Create (and start) a channel
echo DEFINE CHANNEL('WebSocket') CHLTYPE(MQTT) PORT(8000) PROTOCOL(HTTP) MCAUSER('Guest') | runmqsc %QMGR%

While you're in there you can give your QMGR a better name.

Run that batch file, fire up your MQ Explorer, Start your Queue Manager, and you're ready to rock some MQTT traffic.


Working with Git

So today I decided that I want to get with Git, the distributed software version control system that's taken over by storm.

The easiest way was to download Git from their git-scm.com.  They have great tutorials, and you're up and running in minutes.  I wish I had this back when I was doing development.  It has certainly helped with my simple projects this week.

IBM has their Git repository at jazzhub.  Google has theirs.  That makes it easy.  Both IBM BlueMix and Google GCloud, when you create a new project, give you the link to the Git repository for that project.  After that all you have to do is clone it, write code on your machine, commit, and push your code up to the cloud and after a build, it's working.

eGit is a nice add on for eclipse, but you have to watch eclipse versions.  I had some trouble with a missing Apache HttpClient, so it took a little work.  But basically you don't really need it, as long as you're OK with command line.

So, Hourray for Git.

Monday, December 15, 2014

Demo on Blue Mix

I decided to try BlueMix today.  Actually I've used it before to develop Rule Services for some of our demos and proof of concepts.  Today I'm going to see if I can take what I did for the MQTTMaps demo and have it run in BlueMix.  I'm guessing it is not too hard.

1) Get a BlueMix Account - they're free for the first some number of days, so that was easy.  BlueMix.com

2) Pick a starting point - Since this is going to be a web app I'll start WebSphere Liberty Profile.

3) Add Git to the project - this basically gives me a way to develop the code on my machine, sync it with what's in the cloud, and run it in a matter of seconds.  It uses jazzhub, IBM's version of git.  So I'll do a git clone https://hub.jazz.net/git/ibmdthompson55/MQTTMaps in to my local file system.

4) I need something called Cloud Foundry... something new to learn...

Google Maps with MQTT Demo

I wanted to write a quick demo, where I could have everyone in the audience click locations on a map, and send those locations to a central server where I could process them and show aggregations, analytics, event processing, and some aspects of mobile.

The basic idea is to show a map of the U.S.  Then let my customers/friends/coworkers/wife click on it.  Each click, I will tell them, represents a transaction at a store.  My back end server will collect all of the clicks as map locations from all of the people in the room.  And from that information I will start to draw interesting stuff on the projector, from my computer.

To start with I will need some place to host the application, and for that I hope to use IBM's Cloud, possibly BlueMix or Maybe on my desktop using some flavor of WAS.

I'll need a source repository, and for that I hope to use Git.

I'll need to display maps on a browser, and for that I'll use Google Map API.

To gather all of those mouse clicks I will use MQTT, which can scale up to hundreds of thousands of clicks per second, if it is run on IBM hardware.

I'm going to add AngularJS, and Twitter BootStrap, as I will probably need them at some point.

WebSphere Liberty App Server --
This is just a great App Server.  It starts in a couple seconds, and it has full fidleity with its big brother WAS ND, should I ever what high availability and failover.  I downloaded it from WASDev, and configured a server tu run on port 19080.  In my eclipse tooling (I'm actually using IBM ODM Rule Designer for this because I happen to have it already set up) I just create a dynamic web project, and do an export of the WAR file to the DropIns folder of WebSphere Liberty.  It deploys immediately.

Map Pinner --
I started with Google Maps in an html file. Really simple, they provide a lot of examples.  You need to sign up for a API_KEY so they can track your map usage.

Relatively quickly I was able to put up a map that responds to mouse clicks, and draw markers where the mouse was clicked.  Next I just have to send those events to the server, and from there feed them into the magic software layer.  So far, because everything is client side.  HTML5. I just have all of this in my index.html.


To set markers, you need to add an event listener to the map, and in the callback function, that's where you place the markers, and call MQTT with their location.

I added approximately the following JavaScript  

var mapOptions = {
center: { lat: 41.748789, lng: -72.543329 },
zoom: 8 };

var map = new google.maps.Map(document.getElementById('map-canvas'),
mapOptions);

google.maps.event.addListener(map, 'click', function(event) {
placeMarker(event.latLng);
});

function placeMarker(location) {
var marker = new google.maps.Marker({
position: location,
title: "Marker #"+markerCounter ,
map: map
});
}

MQTT --

I'm using the opensource Paho MQTT client, which was originally developed by IBM.  There's a decent code sample, which I'll just add to the init code and then send location information in the placeMarker function.

So I just copy and paste the Paho file into a javascript include called mqttws31.js  and set some init variables.  I'll use localhost for now.  And the functions onConnectionLost and onMessageArrived are just straight out of the sample code.  I'll need to include those as well.

        var hostname = "localhost";
        var port = 8000; // the mqtt port is 1883, but we'll be using websockets to get there on 8000
        var clientid = "mapDemo_" + parseInt(Math.random() * 100, 10);
     
        //

        client = new Paho.MQTT.Client(hostname, Number(port), clientid);
        client.onConnectionLost = onConnectionLost;
        client.onMessageArrived = onMessageArrived;

MQTT Server --

I happen to have IBM MQ 7.5.0.1 on my desktop already, part of my IBM Integration Bus installation, so I'll use that.  I need to modify my MQ install in order to add MQTT functionality.  This is well documented in the Knowledge Center and takes a few minutes.  Acquiring WebSphere MQ requires licensing, beyond what a casual web developer might have access to.

But once you've done that, there's just a simple script to create an MQTT and WebSocket ports, MQ apps can access the data in the MQTT queues, which is kind of neat.

If you don't need to integrate with things outside of MQTT, then HiveMQ is a simple, no-cost for limited use, MQTT server.  Pull it down, unzip it, edit the configuration.properties files and set websockets.enabled=true.  Then start it from the bin folder.  Could not be simpler.  You still need to configure the WEbSocket port though, but it's simple.

Map Listener --
Again, this is a simple Google Maps API, running in html5.  The difference between the this and the Map Pinner, is that it listens for MQTT messages from the Map Pinner, and drops pins in it's own map, taking input from everyone who's dropping pins.  It makes a nice little MQTT demo.

Map Analytics --
Once we're getting a stream of map data we can do some Map Analytics.

Map Event Rules --
Once we're getting a stream of map data we can do some Map Event Rules, and looks for events that take place outside for the norm.

More to come...


 


















Put your Android Phone into Developer Mode

I am working on an HTML5 app.  It works on my browser but doesn't work on my phone.  What ame I to do?  Simple, set the phone to developer mode and connect it to a remote debugger on my laptop.

In the settings, scroll down to the software section.  From there find the build number.  Tap on that 5 times.  Eventually a dialog will pop up telling you you are a developer.  Got back to settings page and you'll find the Developer{} section is at your disposal.