Robotic Expertise and Messaging APIs

Messaging providers set the stage for people to work together with programmable robots utilizing the identical gadgets we already use to speak with one another. That type of interplay feels a little bit like magic, but it surely’s magic that anybody who codes can conjure. To indicate you what I imply, we have to have a look at Misty’s Photograph Sales space ability, which Misty demoed at Twilio SIGNAL 2019.

When this ability runs, you may ship an SMS to ask your Misty robotic to take your image. When Misty will get your textual content, she stops what she’s doing, turns to have a look at you, and snaps your portrait with the digital camera in her visor. She then sends that image proper again to your telephone by way of MMS.

To indicate how all this works, this text breaks down what occurs behind-the-visor of Misty’s Photograph Sales space ability, out of your finger touching ship to your portrait touchdown again in your gadget. It examines the hyperlinks between every service the ability combines, and it explains just a few vital blocks of the JavaScript code that controls Misty’s response.

Let’s begin by Twilio Autopilot, the service your textual content visits first.

SMS -> Twilio Autopilot

To get Misty’s consideration, the person texts a telephone quantity hooked as much as Twilio’s Autopilot service. In case you’re new to Autopilot, right here’s how Twilio describes it:

Autopilot is a conversational AI platform to construct omnichannel bots and digital assistants with pure language understanding and full programmability with Autopilot Actions.


Whenever you textual content this particular quantity, Twilio forwards your message to the Programmable Messaging channel related to a singular Autopilot “bot”. Every Autopilot bot you create can carry out a number of “duties” (a.ok.a. actions that set off when your bot receives a sure type of message). You customise duties within the Twilio Console to program what they need to do when triggered.

For the photograph sales space ability, the core perform of our Autopilot bot lives in a job referred to as we name take_picture . We “practice” our bot to set off the take_picture job when it receives one of many following phrases:

Screen Shot at PM

The subsequent step is to program what the bot does when the take_picture job triggers. Potential actions embrace sending a message again to the person, listening for responses, or amassing and storing data. Or, if the built-in actions aren’t sufficient, you may program duties to redirect to different providers. That’s what we do within the Photograph Sales space ability. When the take_picture job triggers, it calls a redirect to run a Twilio Perform.

Autopilot -> Twilio Perform

Twilio Capabilities are serveless capabilities for dealing with inbound Twilio communications. These capabilities are a fast solution to introduce the communications you’re processing in Twilio to the remainder of the online (a medium of communication through which robots like Misty are exceptionally well-versed).

The Twilio Perform we use in our Photograph Sales space ability does just a few issues. First, it assigns the person’s telephone quantity to a variable. Then, it makes use of the Twilio API to answer to the person with an SMS: [••] Time to Pose . Lastly, it POSTs the person’s telephone quantity (and the character of the motion they’ve requested for) to the PubNub channel Misty is listening to for brand new messages. (Extra element on this within the subsequent part).

Screen Shot at AM

The Twilio Perform code for the Photograph Sales space ability appears to be like one thing like this:

exports.handler = perform(context, occasion, callback) { // Saves telephone quantity to contact variable var contact = occasion.UserIdentifier || "19294421336"; // Sends SMS to person const replySms = {"actions": [{"say": "[••] Time to Pose"}]}; const replyErrorSms = {"actions": [{"say": "[••] Oops an error occured, Might you please attempt once more.."}]}; const axios = require('axios') // The PubNub URL contains publish/subscribe keys, a // channel identify (much like the identify of a chatroom), // and a shopper identify (a singular identify figuring out this // gadget within the PubNub channel). axios.put up('<publish-key>/<subscribe-key>/0/<channel-name>/0?retailer=0&uuid=<client-name>', { 'phNumber': contact, 'sort': 'photograph' }) .then((res) => { console.log(res); callback(null, replySms); }) .catch((error) => { console.log(error); callback(null, replyErrorSms); });

With the Twilio Perform and Autopilot Bot arrange, we’re prepared to have a look at PubNub, the service that notifies Misty to take an image.

Twilio Perform -> PubNub

Hacking a robotic to react to an SMS is fairly cool. Even cooler? When that robotic responds with hardly any latency. That’s the place PubNub is available in.

PubNub supplies a real-time messaging API that builders can leverage by way of HTTP communication protocols, permitting for fast communication between every kind of machines. Whenever you use PubNub’s messaging API, you create an information channel — type of like a chatroom for gadgets — that a number of gadgets (like Twilio servers and robots) can subscribe and publish messages to.

Whereas PubNub supplies SDKs for a number of totally different languages, we get alongside simply effective within the Photograph Sales space ability with fundamental HTTP requests. Whenever you create a brand new “app” in PubNub, you get distinctive API Keys for publishing and subscribing to that app. To publish information (as we do within the Twilio Perform above), we ship a POST request to:<publish-key>/<subscribe-key>/0/<channel-name>/0?retailer=0&uuid=<client-name>

You’ll discover that the PubNub URL contains publish/subscribe keys, a channel identify (type of just like the identify of a chatroom), and a shopper identify (a singular identify that identifies this gadget within the PubNub channel). After we ship this request, we go alongside a JSON physique with the message we need to publish. In our case, that message resembles:

{ 'phNumber': contact, 'sort': 'photograph'

You possibly can learn extra about this within the PubNub developer docs, however the high-level view is that this request publishes a message to the PubNub app we created for the Photograph Sales space ability. Exterior gadgets (like a programmable robotic) which might be listening to that app can then learn these messages and make use of them on their very own.


PubNub -> Misty

Earlier than we have a look at the robotic’s ability code, it’s useful to know what we imply by a ability. Within the Misty robo-verse, a ability is your JavaScript code, working regionally on the robotic. Every ability requires a JavaScript code file, with the JavaScript Misty executes when the ability runs, and a JSON meta file, with different details about the ability.

Misty’s on-board JavaScript API supplies strategies for subscribing to information from sensors and different occasions, and also you outline the callbacks for dealing with this information in your ability code. This API additionally contains strategies for instructions like shifting the robotic, utilizing her sensors, enjoying sounds, and sending internet requests. That final bit is how Misty will get information from PubNub within the Photograph Sales space ability.

For a quick response, the robotic working the Photograph Sales space ability must be powered on and working the code earlier than anybody sends her a textual content. Whereas the ability runs, Misty listens for brand new SMS notifications by usually sending requests to our PubNub channel. We do that in our code by way of the misty.SendExternalRequest() methodology from the Misty JavaScript API.

Every request Misty sends will day out if it doesn’t get a response after twenty seconds, and there’s no assure that somebody will ship Misty a message inside that body of time. We work round this in our ability by pairing our subscription request with a request to publish an empty message to the PubNub channel, which runs on a loop to maintain the strains of communication open. When our Twilio bot forwards a message to PubNub, Misty returns it to our ability and passes it into the _pubNubSubscribe() callback perform.

Within the JavaScript file for our Photograph Sales space ability, that code appears to be like one thing like this (you may as well discover an instance of simply the extracted PubNub performance on GitHub):

// Calls the keepActive() perform each 15 seconds
misty.RegisterTimerEvent("keepActive", 15000, true); // Sends a publish request to work round timeouts perform _keepActive() { misty.SendExternalRequest("POST", "<publish-key>/<subscribe-key/0/<channel-name>/myCallback", null, null, "{}", false, false, "", "software/json");
} // Will get the message Twilio sends to PubNub and passes
// the response into the _pubNubSubscribe callback perform
misty.SendExternalRequest("GET", "<subscribe-key>/<channel-name>/0/0?uuid=<client-id>", null, null, "{}", false, false, "", "software/json","_pubNubSubscribe"); // Extracts the telephone quantity and runs the perform that
// has Misty take an image
perform _pubNubSubscribe(information) { outputExt(information.End result.ResponseObject.Knowledge);

The outputExt() perform (proven above) extracts the telephone quantity and the worth of the sort parameter from the message the Twilio Perform sends. Misty shops the telephone quantity and checks that the worth of sort is the same as photograph. Whether it is, she runs a block of code that has her transfer her head (and digital camera) to face the person, change her show picture, and play sounds to let the person know what’s occurring. Right here’s an instance of how that may look:

if (information != [] && information.sort == 'photograph') { // Saves the person's contact data misty.Set("contact", (information.phNumber).toString(), false); // Adjustments show picture, units head place, and performs // sounds to indicate she's taking an image misty.DisplayImage("DefaultEyes_SystemCamera.jpg"); misty.Pause(100); misty.PlayAudio("DefaultSounds_Awe3.wav", 100); misty.Set("pictureMode", true, false); misty.MoveHeadPosition(0, 0, 0, 45); misty.Pause(3000); misty.DisplayImage("DefaultEyes_SystemFlash.jpg"); misty.ChangeLED(255, 255, 255); misty.PlayAudio("DefaultSounds_SystemCameraShutter.wav", 100); // Snaps a portrait! By default, this methodology passes // a base64-encoded string with the picture information for the // image into the _TakePicture() callback perform. misty.TakePicture("Photobooth", 375, 812, false, true); misty.Pause(200); misty.DisplayImage("DefaultEyes_SystemCamera.jpg"); misty.ChangeLED(140, 0, 255); misty.Pause(500); misty.DisplayImage("DefaultEyes_Joy2.jpg"); }

Misty -> Imgur

Whenever you code Misty to take footage, you may go base64-encoded strings of the image information into callback capabilities for extra processing. By default, these callback capabilities use the identical identify because the misty.TakePicture() methodology, prefixed with an underscore: _TakePicture(). In our ability, we use this _TakePicture() callback perform to go the bottom64-encoded string with our image information into an uploadImage() perform. This callback resembles the next:

perform _TakePicture(information) {
var base64String = information.End result.Base64;

After we name the uploadImage() perform, Misty posts the image to a personal Imgur album. There are a number of image-sharing providers we may use to host these footage, however Imgur’s API does two issues that make it very best for the Photograph Sales space Ability. Factor One: it accepts base64-encoded strings, and Factor Two: it returns the URL for the uploaded picture within the response physique. By passing this returned URL again into Twilio’s MMS API, Misty can ship the image on to the one who requested for it.

The code for managing this within the Photograph Sales space ability appears to be like one thing like this:

perform uploadImage(imageData) { // Units up the JSON physique for importing the image var jsonBody = { 'picture': imageData, 'sort' : 'base64', 'album': '<album-name> }; // Uploads the image to a personal album; then, passes Imgur // response information into the _imageUploadResponse() callback misty.SendExternalRequest("POST", "", "Bearer", "<bearer-token>", JSON.stringify(jsonBody), false, false, "", "software/json", "_imageUploadResponse");
perform _imageUploadResponse(responseData) { // Saves the URL misty.Set("imageLink", JSON.parse(responseData.End result.ResponseObject.Knowledge).information.hyperlink, false); // Runs the code to ship the image sendPicture();

Misty -> MMS

With the image in our Imgur album, only one step stays: getting that image into the telephone that desires it. That, too, occurs by means of the Twilio API. In our JavaScript ability code, we use the SendPicture() perform to put up a request that features the contact data of the one who despatched the unique textual content, together with the URL that hyperlinks to their uploaded picture.

After we name the sendPicture() perform, Misty sends a request to the Twilio API, which drops the picture on the given URL into our person’s messaging inbox. It goes a bit like this:

perform sendPicture() { misty.Debug("Sending Picture to Person"); // Units up thee JSON physique for the Twilio SMS API. // Consists of the telephone variety of the recipient and // the URL for his or her {photograph} on Imgur var jsonBody = { 'Physique': '[••] Greetings from Misty!', 'From': '<number-to-send-from>', 'To': misty.Get("contact"), 'MediaUrl': misty.Get("imageLink") }; // Sends a request to the Twilio API with our account // credentials to ship the image to the one who requested for it var credentials = "<base64-encoded-Twilio-credentials>" misty.SendExternalRequest("POST", "<account-id>/Messages.json", "Primary", credentials, JSON.stringify(jsonBody), false, false, "", "software/x-www-form-urlencoded");

To Recap

On this put up, we mentioned tips on how to hyperlink Twilio and PubNub’s messaging APIs, Imgur’s photo-sharing providers, and robot-icizied JavaScript code to construct a Photograph Sales space ability for Misty. After we use this ability:

  1. Somebody sends a message to our Twilio telephone quantity
  2. Twilio passes the message to our Twilio Autopilot bot
  3. Our Autopilot bot reads the message, identifies the duty, and redirects to our Twilio Perform
  4. The Twilio Perform posts the person’s telephone quantity to our PubNub channel
  5. Misty, who’s been working the Photograph Sales space ability all of the whereas, pulls down the message from PubNub
  6. Misty repositions her digital camera and takes an image
  7. Misty uploads the image to a personal Imgur album and calls out to the Twilio API to ship it as an MMS to our person

Sending your robotic a textual content is a fairly sociable solution to ask it to do one thing. When the robotic replies with an image of its favourite individual? That’s downright chummy.


Leave a Reply

Next Post

Bose broadcasts Transportable House Speaker with 360-degree sound

Sat Aug 24 , 2019
For those who’re out there for a premium moveable Bluetooth speaker, Bose’s newest launch ought to curiosity you. Bose Company has introduced the Transportable House Speaker, which, because the title suggests, is a transportable battery-powered speaker with inbuilt assist for digital assistants resembling Google Assistant and Amazon Alexa. Accessible in […]
Wordpress Social Share Plugin powered by Ultimatelysocial