Switch Controled Assistive Technology Mouse Using Feather 32u4 and OLED Display

In the ATMakers.org Facebook page we’ve been discussing the development of an assistive technology device we are preliminarily calling APHID. It’s my belief that at least one version of our APHID device will need some sort of display. Here is a proof of concept of the device using a feather 32u4, OLED display and three switches. It does every kind of mouse control that you could imagine as well as the limited keyboard commands. This device is connected through USB but we could also make a Bluetooth version.

The Ultimate Remote is Dead. Long Live the Ultimate Remote 2.0

In late 2015 I built a piece of assistive technology that I called my Ultimate Remote. The device was an infrared remote for controlling my TV and cable box. It also had infrared mouse control for my computers. There were also limited keyboard commands mostly arrow keys, enter key, backspace and some control keys for doing cut-and-paste. Finally it was a Bluetooth device for doing accessibility switch control on my iPhone. I wrote about it in this blog post from January 2016.

The Ultimate Remote Control and Why I Built It


The core of it was an Adafruit Micro BLE device which was discontinued shortly after I purchased it. Adafruit replaced it with the Adafruit Feather 32u4 BLE. The Micro BLE also had an ATMega32u4 which is one of my favorite 8-bit processors.

The display was a monochrome 1.3 inch 128×64 OLED graphic display. It also contained one of my infrared I/O boards although it only did output. I didn’t really need to read any IR codes. On the end of a long wire were three lever micro switches that I would hold in my right hand to control the device.

The device served me well for well over a year. It was critical to me during my recent hospital stay where I was on a ventilator and could not talk. I used the switch control to type notepad messages on my iPhone and communicate with the nurses and doctors. You can read about that adventure here.

Pray That They Listen to the Man with No Voice

The micro switches have always been the weakest link in my devices. I have to use switches that have a feather touch to them and that means they are very fragile. The switches get knocked around quite a bit and so it was inevitable that one of them would break. About a week ago one of the switches failed and we had to replace them.

I had planned for many months to rebuild the entire device. I had used up all of the program memory in the device and could not add any new features or any new IR commands. I did not have codes installed for my Blu-ray player and there wasn’t any room left. In fact one time I had tried to recompile the code and because one of the included libraries had been updated, the code wouldn’t fit anymore. I don’t know if it was the Bluetooth BLE library or the graphics display library but something changed. I had to go through my code and try to free up some space by eliminating some error messages or shortening other messages. I finally got it to recompile but the writing was on the wall that I needed to upgrade.

The obvious choice was to use a new Adafruit Feather M0 BLE. Instead of the traditional 8-bit 32u4 running at 16 MHz with 32K flash memory and 2.5K RAM, I would have a 32-bit ARM Cortex M0+ running at 48 MHz, with 256K flash memory and 32K of RAM. The main problem was that my infrared library IRLib did not support these newer 32-bit ARM processors.

I had just recently spent weeks researching the new processor and converting my infrared library to support the newer chips. The timing and frequency modulation portions of my IR code are extremely dependent on the internal timers of the processor and that is very hardware dependent. I had to learn a whole new system of timers and PWM frequency control to rewrite the code. Fortunately I got it running just in time.

Although I had the infrared code working on the new processor, and had a Feather M0 BLE available to build a new device, it should not have been necessary to rush the new device into construction. All we had to do was repair the old remote by replacing the micro switches. I already had another set of switches assembled in anticipation of building the new remote. All we had to do was cut the cable on the old one, splice in the new cable and everything would be fine.

Dad decided that rather than having a stiff, unsightly splice in the middle of my cable, he would open up the box, unsolder the old cable and solder on the new cable. He ended up completely disassembling everything to get at the wires that needed replacing. I appreciated that he would go to that trouble even though this device was going to get replaced probably in a month or two. Unfortunately this was a bad decision.



Above are photos of the interior of my original ultimate remote. As you can see the wiring is pretty complicated. Unfortunately I only have about four different colors of wires available so we had to use the same color wires for different purposes. For example there are multiple green wires used for different purposes. After replacing the cable to the micro switches, dad tried to plug everything back in the way it was. We plugged in the device and I tried pushing the buttons but nothing would happen. In the course of trying to figure out what was wrong he touched one of the infrared LEDs and discovered that it was very nearly becoming more red than infra. It was too hot to touch. Although we did not get a visit from the infamous “Blue Smoke Monster”, if we had left it connected very long we would’ve had at least smoke and possibly fire as well.

It didn’t take us long to find out that the culprit was 2 green wires that had been crossed. We fixed the wiring and plugged it back in. No heat this time. The device worked intermittently for about five minutes and then quit working altogether. It was obvious that we had burned out the IR LED at least and possibly the transistors driving them.

Fortunately I had sufficient parts to build a new IR output board so we spent the next afternoon building it and installing it. It would not work either! One of the problems we were facing was that we had assembled and disassembled the box many times. I had been using small gauge stranded wire with silicone insulation. That is very flexible and made it easy to route the wires in a tiny box. However several of these wires were soldered into through hole locations on the circuit boards. Right at the point where they are soldered in, they are extremely susceptible to breaking if they are bent back and forth too many times. A better solution would have been to put a header pin in the hole. Then we could solder the stranded wire onto the pin and cover it with a piece of heat shrink tubing. At this point it was too late to do that.

We tried repeatedly to diagnose the problem with the new IR board but every time we fixed one thing, something else would break. There was also the possibility that we had damaged the Adafruit Micro BLE board itself. As I mentioned earlier, that board has been discontinued so there was no possibility of replacing it. After two full afternoons of working on it, I decided to throw in the towel and put all of our efforts into building the completely new device that had been planning for months.

I had all of the necessary parts. The new device would have a much larger 2.4 inch TFT color graphics display instead of the 1.3 inch monochrome OLED. This device also features a resistive touchscreen however I don’t have any use for that feature. It also includes a slot for an SD memory card. I may come up with a future use for that.

One of the nice things about the Feather Wing TFT board is that the feather board plugs into a socket in the backside. You don’t have to run wires from the main processor to the display board. However one of the disadvantages is that there is only one power and one ground pin on the device. So I was going to have to cut up a little piece of prototype board to make a power and a ground bus. This would bring in power from the outside, connected to the Feather and display boards, and run power to the infrared board. Similarly I needed ground wires to all of those parts plus a ground wire for the micro switches. We also needed to solder a jumper so that I can turn the backlight of the display off and on. There is a solder pad available but you have to jumper it to one of your Feather pins.

Although the Feather boards have a built-in battery connector and a battery charging circuit, I decided to power the device from an external 5v battery source. My old Ultimate Remote was a 5v device throughout while the Feather system runs on 3.3v. In the old system I had a short cable with a barrel jack running from the remote into a battery pack I call a Printy Boost. The Printy Boost is a device which I designed for the Adafruit Learning System as seen here. The Printy Boost also provides extended battery life to power my iPhone. So rather than have a separate battery for the remote and for the iPhone backup power I decided I would stick with the old system and run a cable from the Printy Boost into the remote just like I did before. Rather than connect to the +3.3v pin I connected to me “USB” pin which was the same as powering it through the USB cable at 5v.

There are some differences between the Feather TFT board and the old monochrome OLED graphics board so I had to tinker with the software to get things to run. Most of it was compatible because Most of the Boards operate on the Adafruit Graphics library but there are still differences. Once I had display software converted I tried using the device. Unfortunately it didn’t work again! You would think I would be more careful about crossed wires after the previous fiasco. It turns out we had the infrared I/O board wired backwards. I had drawn the wiring diagram looking at the front side of the IR board. That means on the right 2 pins are power and ground. Then moving to the left you skip one pin and the next one is the IR output. However the way the board is oriented in the device, the backside of the board is facing upwards. We wired it with power and ground on the right but that was wrong. Fortunately this just meant that the power lines were going to the receiver pins which were not being used in this application. So nothing burned out. We reversed the wiring and everything worked fine.

The final step was to design a 3D printed enclosure for the device. That took another afternoon or so.

We mounted the TFT display into the lid of the box using black nylon plastic screws and nuts out of this kit sold by Adafruit. They are really handy because they are selected to fit in the 0.1 inches diameter mounting holes used in most Adafruit boards. When that box of screws was first added to the Adafruit catalog I knew I would need them someday and purchased it right away. They work perfectly so it was a great purchase.

The rest of the box is held together by 5/8 inch sheet metal screws. I like using sheet metal screws because they have a pointy end that taps really well into 3D printed PLA plastic. Here are some more photos of the completed project.



There are still lots of software tweaks I have to implement. It takes longer to erase a 320×240 color display than it does to erase a 128×64 monochrome display. The monochrome device required you to call a “display” method to update the display after writing to it. The new color device updates as you write to it. The result is you can see the screen update where the old one would update instantly. I think the updates actually slow down the entire process a little bit. So I’m going to have to optimize the code so that it only updates the screen when absolutely necessary and only does it in small pieces. In the old system it was easy to just erase everything and redraw it from scratch every time but that won’t work very well in the new system.

Of course I also have to add all the features I’ve been wanting to add but didn’t have sufficient memory under the old system. I have to add all the codes for my Blu-ray player and there are some additional keyboard codes that I want to add. I may end up implementing an entire keyboard system so that I can type anything using IR codes. I’ve developed a special protocol for my IR library that allows me to use any mouse or keyboard commands possible. I’ve only been using a fraction of that capability.

One other difference between the old and new system involves the power output of the IR board. When I originally designed my infrared I/O board I ran the LEDs with no current limiting resistors. Because the LEDs are only intermittently running (assuming you don’t cross your wires), it’s safe to put more than 1 amp through them. But my experience is that sometimes USB power can’t supply enough power when that current spikes during transmission. So I’ve added some 33 ohms current limiting resistors in line with the LEDs on the latest version of my infrared I/O board. The old device did not have these resistors but it ran well because it was powered by a battery pack rather than a USB plug. Now that I’ve worked with the new remote I’m realizing it doesn’t have the power of the old one. I’m going to try shorting across those resistors and see if it helps.

I still have fond memories of my original Ultimate Remote. It served me well for over a year and was literarily a lifesaver while I was in the hospital. But I’m also looking forward to the new things I will be able to do with the new improved Ultimate Remote 2.0.

Afterward: After I completed the project I presented it on the weekly Adafruit Show-and-Tell. I was the first guest in the video below.

IRLib 2.02 Increases Support for SAMD 21

A few weeks ago we released IRLib 2.01 with preliminary support for SAMD 21 processors such as used in Arduino Zero and Adafruit Feather M0 boards. Then initial release had many hardcoded values. For example output was only available on pin 9 and the timer interrupt for the 50 µs receiver class was hardcoded to use TC3.

In our latest release IRLib 2.02 you now can use any available PWM pin on your platform for output by editing values in IRLibProtocols/IRLibSAMD21.h

You can now also choose to use TC4 or TC5 to drive the hardware interrupt for the 50 µs receiver class.

Both sending and receiving use GCLK0. Note that typically GCLK0-GCLK3 are reserved for internal Arduino infrastructure use however we are using GCLK0 with its default clock source of 48 MHz and a divisor of 1. Because we are not modifying those defaults, it is safe to use. That allows you complete freedom to use GCLK4-GCLK7 for other purposes.

The code has been tested on Arduino Zero, Arduino M0 Pro, and Adafruit Feather M0 BLE boards successfully.

The code is available now on GitHub at https://github.com/cyborg5/IRLib2

Announcing the Release of IRLib2

We are pleased to announce a new version of IRLib library for receiving, decoded, and sending infrared signals using an Arduino. This represents months of work and is a major restructuring of the code to make it easier to include only those protocols that your application actually uses. Unfortunately the restructuring is so significant that it is not backwards compatible with our original IRLib1. It is however relatively easy to update your applications to use the new libraries. We will continue to leave IRLib 1.x in its original repository on GitHub. We have created a new repository https://github.com/cyborg5/IRLib2 for IRLib2. There will be no new updates to the original version of the code. Because we have changed the name to IRLib2 it is possible to have both versions available in your Arduino libraries folder at the same time without conflict.

Also included is a new feature called “auto resume” which ensures that you can continue receiving a second frame of data while processing the first frame.

The users manual has been expanded to 117 pages and not only includes the complete reference to all classes, methods, and variables that we had before, it also contains a new tutorial section which thoroughly explains all of the sample sketches. And finally we have the long-awaited tutorial on how to create new protocols for the library. You will also want to check out Appendix A which explains the changes between IRLib1 and IRLib2 as well as my rationale for the restructuring and the need to make it no longer backwards compatible.

The users manual is available in Microsoft Word, Adobe PDF, and EPUB formats in the GitHub repository. We will no longer be maintaining online HTML version of the documentation because it was too difficult to split into individual webpages and maintain it for this blog. However the supplied formats should be sufficient for all users.

Although we have thoroughly tested the code, it technically is version beta 0.1. If no major bugs are reported in a month or two we will upgrade it to official version 2.0 release.

We can’t wait to see what all of you will do with IRLib2. Please let us know how you are using the code.

The Ultimate Remote Control and Why I Built It

I’ve just completed building a new piece of adaptive technology that includes infrared TV remote, infrared mouse control, and Bluetooth switch control of my iPhone. It’s the combination of a project that has been in development nearly 3 years. Here’s the story of the solutions I had before I built this remote, why they no longer serve me, and how the new device solves a variety of problems. It all started with a simple wooden stick…

 

The History of the Stick

In many of my previous blog posts I’ve discussed how I built custom TV/cable remote controls so that I can watch TV and use my laptop computer in bed. However I’ve not discussed how I use these devices when I’m not in bed. That process has recently had to evolve significantly so I thought I would chronicle the history of my use of remote controls while sitting in my wheelchair.

Of course when I had good use of my hands, I would just pick up a remote and push the buttons like anyone else. But at some point, I don’t recall when years ago, my dad built a little aluminum bracket that mounts on the front of my wheelchair control box. We attach a regular universal learning remote to the bracket with some Velcro. I would then push the buttons using a stick in my mouth. That particular stick system has gone through evolutions of its own.

Johnny Carson used to have a comedy routine called “Dickie the stick”. It was a commercial for a toy that had 1000 uses. In fact it was just a wooden broom handle. He would say with a little imagination you could make it into anything. He would throw it across the stage and say “look it’s an airplane”. His pitch man character made it sound like it was the greatest toy ever made but it was just a stick. Actually that commercial wasn’t far off for me. For almost my whole life I’ve carried around a tool that was nothing more than a big long stick. I used it to push elevator buttons, open doors, and grab things that I couldn’t reach. Someday I’ll do a blog just about my big stick. But for this particular blog will just talk about the small wooden stick that I used for typing and pushing buttons. This particular stick has been a wonderful piece of adaptive equipment for me. As you will see it’s been a big challenge to deal with the fact that I can no longer use it (oops… that was a spoiler).

Over the years my disability has progressed. I used to have pretty good use of my hands so I could type on the computer, handle books and papers, even feed myself. I lost most of that in my early twenties. When I could no longer type on a keyboard using my hands I discovered an alternative way to type. I would prop up the computer keyboard on an easel so that the keys were facing me. I used a very long pencil or dowel rod in my right hand, put my elbow up on the armrest or control box, and I would poke at the keys. In the early days of 8-bit computers I would wire in in a couple of extra pushbuttons on the end of a long wire connected to the Shift and Control keys. I would hold those buttons in my left hand. By the time we got to MS-DOS and later Windows they had a built-in feature called “sticky keys”. Many people think that’s what you get from watching too much Internet porn but this is something different. It is a software driver that turns your Shift, Control and Alt keys into toggle keys. For example you push shift and the next key that you type will be shifted. If you push shift twice it holds the shift key down. Everything remains shifted until you hit it again to unlock it. Here is an old photo of me typing on the keyboard this way.

Me checking my stock portfolio using stick and keyboard on easel. Circa 2000

Me checking my stock portfolio using stick it keyboard on easel. Circa 2000

The photo shows a keyboard that was very near and dear to me. It was made by Gateway Computers and was called the “Gateway Any-Key 124 keyboard”. It had 2 sets of function keys, the traditional ones across the top and an identical set down the left side. It also had a special built in hardware keyboard remapping function. You could reprogram any of the 124 keys to send any remapped keypress that you wanted. It also would allow you to create macros so that one keypress would generate a string of multiple keystrokes. Because it was difficult for me to reach the arrow keys way off to the right side of the keyboard, I reprogrammed the top row of function keys into arrow keys. If I needed a function key I would use the ones down the left side of the keyboard.

You will also notice in the photo just above the keys is a mini trackball. It was the only way that I could operate the mouse. That particular model was small enough that it was handy to just mount on the keyboard with some Velcro. I could poke at it with my stick to operate it. The great thing about that particular model was that it included not only a right-click and a left-click button but it included a drag button. It was sort of like “sticky keys” for mouse buttons. You would toggle it off and on if you needed to drag. It would not right-click drag but you rarely have to do that. The problem with that particular trackball was it would wear out eventually. In the end I ended up buying three or four of them at once for about $50 each and keeping them in my closet just in case. The original Gateway keyboard had a 9 pin D serial connector. I was able to later purchase them with a round PS/2 style serial connector but they never did make one with USB. Fortunately I found some serial to USB adapter devices and was able to continue to do use this keyboard many years beyond its normal life expectancy.

When using a laptop, sometimes I was able to prop up the entire laptop on an easel so that the keys were facing me. I had to be careful when purchasing a new laptop because some of them do not allow the display to open a full 180°. Had I tried to prop up one of those, the display would’ve been facing the floor not to mention how difficult it would be to keep it on the easel. Even when I did put it up on the easel, it was very precarious. Eventually I started using the special Gateway keyboard on the laptop as well by just plugging it into the serial port or USB port with adapter later on. Because the keyboard had been discontinued and I was worried it might break, I even stocked up on those by purchasing a spare on eBay. I see here that they are still being sold on eBay. I still have a couple of them in the closet today. Maybe I should get rid of the old keyboards and make some money.

Vintage Gateway 2000 programmable 124 key keyboard available on eBay.

Vintage Gateway 2000 programmable 124 key keyboard available on eBay.

 

Enter the Dragon

At some point I supplemented all of this by using speech recognition software. I began using Dragon NaturallySpeaking when it was at version 7. I don’t remember what year that was. The current version is 14. Typically I would use my stick for most of what I did but if I had something long to type like a long email or a blog entry I would use the dictation. I didn’t realize that over a period of a couple of years I began using the Dragon more and more and would use the stick less. I recall one weekend we went to the cabin at Cordry Lake and although I had packed my laptop and extra keyboard, I had forgotten to pack the easel to stand up keyboard. So I had to just use Dragon alone the entire weekend. I was surprised to realize I didn’t really miss using the keyboard and trackball. It was soon after that that I resigned himself to using speech recognition exclusively.

Initially the stick in my hand was only used for typing. When I wasn’t at the computer typing on something, I would let go of the stick and leave it somewhere by the computer. Then one day back in the late 1980s as I was driving my wheelchair into my office, my hand slipped off of the joystick of the wheelchair and I crashed into a bookcase. Over the days that followed I begin to discover my arm had weakened to the point where I could no longer keep it steady on the joystick. After a little experimenting, I came up with a system where I would put the stick in my mouth. I would also hold that in my hand and the joystick as well. The mouth stick would steady my hand on the joystick of the wheelchair. Here is my typical driving position.

My typical driving position using the stick in my mouth to steady my hand on the wheelchair joystick. June 2015.

 

Can’t Touch This

Eventually I expanded my remote bracket on the wheelchair to make room for an iPod touch. If you’re not familiar with it, the iPod touch is sort of like an iPhone without the phone part. It plays music, video, games, and does Wi-Fi Internet access. I started out with the iPod touch model 1 shortly after it came out. The problem was the stick that I was using was a wooden dowel rod with a glob of silicone rubber on the tip so that it would not slip. Even if I turn the stick around and use the wood end, the iPod uses a capacitive touch system. The wooden stick would not activate it. I needed something metallic. After much experimenting I discovered that the metallic piece had to be a reasonable diameter rather than a sharp point. For example we tried just touching a key or a metal knitting needle but that didn’t work. It had to be flat and make flat contact with the screen. That was a problem because as you reach to the top or the bottom of the screen, the stick makes contact at a different angle. So we had to come up with something flexible.

We finally came up with a piece of foam rubber with a hole in it so that you could put the stick into it. On the front of it we glued a small flat piece of metal. I would’ve thought that would be sufficient but it still wasn’t enough electrical capacitance. We attached a very thin ground wire to the metal tip and it worked fine. Having a wire attached to it also had other positive effects. For one if I dropped it, it didn’t go anywhere because it was tethered to the bracket. Also I can hold on to the wire in my right hand to steady the stick as I drag across the touch screen. Here is a photo of me using the stick to operate an android phone that later replaced the iPod touch. If you look closely you can see the thin red ground wire that I’m holding my hand.

Operating a smartphone and TV remote using a stick in my mouth.

We built a little receptacle on the side of the mounting bracket to hold the tip when I wasn’t using it as seen in this photo.

Small receptacle on the side of the mounting bracket holds the foam rubber tip when not in use.

Years ago I had upgraded the iPod touch version 1 to a version 4 but eventually it became obsolete. It would not run the latest version of iOS. Although there was an iPod version 5, the iPhone was already up to version 6 and I knew the minute I bought an iPod version 5 they would come out with 6. I waited and waited for various Apple product announcements. All of the rumor websites kept predicting that an iPod 6 was coming soon but it still hasn’t. A Google search on the phrase “android alternative to iPod touch” led to several articles suggested purchasing a cheap unlocked android phone. If you did not activate it, you could still use it for apps, Wi-Fi, games, music video etc. So that’s when I did. The image above shows the Motorola Moto G second-generation that I purchased. It is unlocked GSM phone which runs the latest version of android. It only cost about $180 and was a really good deal. I wasn’t sure I really needed a phone but once I was to the point where I couldn’t drive my wheelchair anymore, I could not drive up to get to the land line speakerphone that I usually use. I ended up activating it after all. I got a really cheap pay-as-you-go plan from ting.com that is costing me $12-$15 per month.

One of the problems with the android phone is it is larger than the old iPod touch. That means that as I reach from the top to the bottom of the screen, the angle of the stick varies more. It was more difficult to get a little flat metal tip to keep in contact with the surface. I came up with an interesting adaptation to solve that problem. We took a small piece of conductive foam such as used to protect IC chips from static electricity. We cut it in a little rounded hemispherical shape and glued it on to the little metal plate on the end of the stick. So no matter what angle I used, there was still sufficient surface touching the screen. Here is a photo comparing the size of the new android phone on the left and the iPod touch 4 on the right.

My comparatively tiny iPod touch 4 next to my newer Moto G android phone.


tips

Here is a close-up showing the flat tip on the right that I used for the iPod and the one on the left is the new rounded tip use for the android phone.

That whole system worked really well for several years. It got me through hundreds of levels of Angry Birds and above level 1000 on Candy Crush as well as allowed me to look up countless facts on IMDb, post to Facebook, and check the weather radar anytime I felt like it.

 

I.R. An Expert

The stick isn’t the only piece of adaptive technology relevant to the story. A lot of what I do is assisted by infrared remote control such as the ones used by TV, cable, DVD etc. It was about three years ago that I got back into tinkering with electronics when I purchased my first Arduino microcontroller. I became interested in infrared remote controls and that led to the building of the TV/cable remote that I use in bed. The whole journey was documented elsewhere in this blog. I came up with another use for the Arduino when they introduced the Arduino Leonardo model. It is capable of emulating a mouse or keyboard when plugged into the USB port on the computer. So I built a little box that would pick up signals from my TV remote and it would create mouse movements as well as some keystrokes most specifically the arrow keys. I could switch back and forth between mouse mode and keyboard mode. Because I don’t have a cable box in my office, I use various codes from the cable box to control this device. However when I built another one for my laptop that I often use in the bedroom, I had to pick a different set of codes for that one so that it didn’t change channels. The one for the laptop used code from my Blu-ray player. I never use the laptop in the living room while watching Blu-ray so that was okay.

While Dragon dictation software does have mouse controls, they are not very flexible. For example you cannot use the mouse scroll wheel, it is difficult to shift drag and you cannot right-click drag at all. But pushing the TV remote with my stick, I can do all of that and more with the Arduino Leonardo and an IR receiver.

My infrared remotes are based on a library of code written by a guy named Ken Shirriff which I later rewrote to make a little more flexible. I published the code on GitHub and on this blog. My version has been so popular, I’m considered somewhat of an expert on IR remotes. It has been really satisfying to get emails from other people who have built their own IR remotes for their elderly or disabled friends and family. One guy build a remote for his nearly blind grandfather using my code. His version has very large pushbuttons and plays back soundbites when you push the button. It includes specialized buttons that automatically jump to some of his grandfather’s favorite channels such as news or sports. The guy just recorded his own voice to play back but I think it would’ve been better to have James Earl Jones say “THIS is CNN!” or the ESPN Sports Center jingle “Duh da dant, duh da dant”.

 

RIP The Stick

It’s kind of interesting that the stick saved me from two separate situations where I was very emotional over lost ability. When I couldn’t type anymore using my hands I thought it meant the end of using the computer altogether. The stick let me keep typing. Then when I couldn’t drive my wheelchair anymore, the stick let me keep driving. It also had other uses such as pushing TV remote buttons and operating an iPod or touchscreen smartphone. Unfortunately I met another one of those crossroads moments where I can’t do what I used to do. The stick is not going to bail me out. In fact because I could no longer get the stick into position that I needed, I’m not able to use it to operate the remote, the phone, or to assist me with driving. Here’s a brief video that shows how difficult it was for me recently to be able to get the stick into position.

That video was recorded back in June 2015. As I write this in January 2016 things have gotten worse. For the past couple of months I haven’t driven my wheelchair all. I couldn’t get the stick into position without help, so anytime I wanted to go somewhere, dad would have to get into position and even once I was there sometimes it was too difficult to drive. So in recent weeks he just pushes me everywhere. The new wheelchair I’ll be getting should resolve some of that. You can see other videos and blog entries about my quest to get new wheelchair.

 

Arduino to the Rescue

Now that I am no longer able to use the stick, I had to come up with something else. I had been anticipating this problem for many years especially when it came to using the TV remote. After using my specialized Arduino powered remote in bed with just a few pushbuttons, I realized it was much easier to use than pushing the buttons with my stick while sitting in the wheelchair. This latest remote that I purchased has very tiny buttons and unless you get the tip of the stick in exactly the right place they would not push. So over two years ago, I began working on building an Arduino powered TV remote to replace my usual universal remote.

I started out with an Arduino Leonardo and a 4 x 20 character LCD screen. This was going to be the ultimate remote that not only would control the TV and the computer mouse, it might also be used as a wireless infrared keyboard. That’s why I needed the bigger display. I decided I would put four or five pushbuttons on it to scroll through the menus. I would still using my mouth stick but I would have the options of attaching micro switches that I would hold in my hand. We actually built an early prototype of the device and it was pretty cool but it was a little bit impractical. It was rather large and heavy to be mounted on bracket that I had been using. I wasn’t sure what I was going to do to provide power to it. Also the various types of pushbuttons that I tried worked very well. If I used micro switches on the end of a wire, I had to put my arm in an awkward position in order to be able to use them. This particular device didn’t go completely unused. I programmed it with special codes for a toy helicopter that used in infrared remote as well as a toy dinosaur. I’ve already documented those projects in other blog posts and videos. I never did get around to using it as a TV remote because I couldn’t come up with the right buttons. Also I didn’t really need it, I could still pushbuttons with the mouth stick. Keep in mind this project started almost 3 years ago when I was in better shape. Here’s a photo of the prototype was a large 4 x 20 LCD display that I never did get fully working.

The Leonardo-based IR remote with 20×4 character LCD. I never finished it.


5_switches

Here are the 5 micro switches connected to the Arduino Leonardo remote that I could have used as an alternative to the pushbuttons on the device itself. I used these switches to control the helicopter and the toy dinosaur. It turns out that 5 switches were too many. I could not comfortably hold them and I could not hold them if my arm was in a position where I could drive the wheelchair with the joystick. Now that I’m no longer using my hand for the joystick, I found a position where I could comfortably hold 3 of these switches with a little adaptation which I will show you later.

 

Internet of Things in the Interim

Pinoccio

Pinoccio Wi-Fi Development Board

In the interim, I found other ways to control my TVs but this time over the Internet. Using the old Pinoccio Wi-Fi module and a webpage I was able to control the living room cable box from my adjacent office by clicking on a webpage. I have an HDMI splitter connected to the living room cable box. One output goes to the living room TV where my dad watches. The other output goes to a very long HDMI cable that goes through the wall into a TV in my office. There I can watch the same thing as dad is watching the living room. Of course he falls asleep in the recliner when doesn’t fast-forward through the commercials. Being able to control the living room cable box remotely is very useful. I also set up a similar system between the cable box/DVR in my bedroom and ran it into the office although I use RF coaxial cable for that link which makes it standard definition only. Still it’s very useful. I also made the webpage sensitive to keyboard presses which meant that I could use my voice control to operate it. For example if I would say “Press P” using Dragon dictation, the webpage would push the play button. The arrow keys were especially useful when browsing through the on-screen guide. I could simply speak the Dragon command “move right 6” or “page down” and it would scroll around the on-screen program guide.

Arduino Yun

Arduino Yun

The Pinoccio platform was unreliable and eventually discontinued. You can read about my love/hate relationship with that elsewhere. I replaced it with the Arduino Yun which not only made it a Wi-Fi compatible Arduino platform but it had a built in advanced processor running a version of Linux that could serve the webpages rather than hosting them on my PC. The Arduino Yun can be reprogrammed by Wi-Fi which is very useful. I can upgrade the software anytime I want. The unit in the living room is stuck underneath an end table next to my dad’s chair. The one in my bedroom is up on a bookshelf. In recent months as I’ve lost the ability to use my regular remote and stick I even added a third Arduino Yun in the office which is only used to turn the office TV off and on and control the volume. Each has its own webpage and creates its own IR codes specific to the devices in that room. I also have webpage control to a Kodi/XBMC media server running on raspberry pi. So I can control 2 different cable boxes, three different televisions, and three different raspberry pi media players all using webpages designed to be compatible with voice control.

 

Chris helped Christopher. Christopher helps Chris.

I still miss being able to push remote buttons. Once I realized I really do need a cell phone for safety purposes since occasionally dad leaves me alone while doing errands, I had to come up with a way to control the phone again now that I could no longer use my stick to operate the android phone. I could give up playing Angry Birds. You can play Candy Crush on a Facebook webpage. But I really need the phone. Thanks to a young man named Christopher Hills who lives in Australia, I discovered that my move from iOS to android was a mistake. He has cerebral palsy but he doesn’t let it stop him from pursuing his hobby and vocation of making videos. He is somewhat of an expert in adaptive technology. He posted a YouTube video embedded below in which he describes new accessibility features in iOS version 9. It allows you to use one or more pushbuttons to access nearly every function of the phone. Of course it also works for iPad tablets. The pushbuttons are connected to the phone or tablet over a Bluetooth connection.

Seeing that video was a very emotional experience for me. For one it was the realization that I could continue to use a smart phone after all. But he also reminded me of my late friend Christopher Lee. He was a friend who had very severe cerebral palsy. I had built an accessible computer for him back in the early 1980s. My late friend could not push a button but he could make a clicking sound with his tongue. Of course there was no speech recognition back in those days because of eight bit computer just wasn’t powerful enough to do it. I will write a whole other series of blog posts about my friendship with him and the things that we went through to get him computer access. It’s amazing that 30 years ago Chris Young was building accessible computer for a guy with cerebral palsy named Christopher Lee. Now a guy named Christopher Hills with cerebral palsy extremely similar to the severity of my friend Christopher Lee is demonstrating to me how I can continue to access a smartphone. It all came full circle. Here’s a link to the video demonstrating the switch control features of iOS 9.

 

Just-In-Time Technology

It’s a bit creepy that the technology that I need, has been developing just in time for me to use it. For example the mouse control via Dragon dictation software is usable but awkward. Once I got into Arduino I wanted to find a way to emulate a mouse or keyboard. While I was investigating various hard ways to do that, they released the new Arduino Leonardo which made it incredibly easy. Just as the Pinoccio platform was shutting down I discovered the Arduino Yun which in the long run was a better solution. And just as I was losing the ability to use a smart phone, iOS 9 was being released with unprecedented powerful switch control features. Then another “just-in-time” technology came along. The electronic supplier that I use, Adafruit Industries, is a phenomenal organization designing and selling maker products around the world. They are where I buy all of my electronic parts and I show off my projects on their weekly video chat “show-and-tell”. They had recently begun slowing various modules for Bluetooth control. The new Bluetooth 4.0 also known as Bluetooth Low Energy or BLE makes it very easy to build gadgets that communicate with computers, tablets, or smart phones via Bluetooth. That was exactly what I needed if I was going to use wireless switch control.

Adaptive equipment suppliers market Bluetooth devices especially made for switch and/or joystick control at a cost of up to $500. There was another model for about $150 that wasn’t nearly as capable but would work with a couple of push buttons. However I could purchase a handful of parts from Adafruit and I could build one for under $75. By building it myself I could customize the software however I wanted. I could incorporate whatever other functions I wanted besides the iOS switch control including making it an infrared remote TV, cable and mouse control.

Step-by-step all of the technology that I needed was appearing exactly when I needed it. They say a coincidence is when God creates a miracle and nobody notices. I was noticing things falling into place.

 

Goodbye Android… Back to iOS

There was switch control the older versions of iOS however my old iPod touch only worked up to iOS 5 or 6 I forget which. Those earlier versions of switch control left a lot to be desired. Until I saw that Christopher Hills video, I had no idea how capable it had become. Latest android version also introduced switch control but it wasn’t nearly as powerful or useful as iOS 9. It was obvious I was going to have to switch back to iOS.

I recently purchased an iPad for my nephew that he needed for school. Before delivering it to him, I tinkered with it using an Adafruit Bluefruit Micro BLE module and prove that it could communicate with iOS 9. That startled the issue for me. Although it cost me a fortune, I purchased an unlocked iPhone 6. Really didn’t need the 6s or the 6 plus versions. I could still keep my cheap pay-as-you-go plan that’s costing me less than $15 per month. I very rarely am away from Wi-Fi so I don’t need to pay $40 per month for unlimited data. My dad retired his old flip phone and inherited my android phone. He’s having a lot of fun with it learning how to use smartphone features he’s never had before.

 

Finally the Ultimate Remote “The IRBLE”

Since I was going to be building an electronic device to operate the iPhone, it was time to also incorporate TV/cable remote capability as well. Technology had advanced that things were smaller and more capable than when I first started building the big display Arduino Leonardo remote three years ago. The Arduino Micro BLE has the same ATmega 32u4 processor as the Leonardo. Additionally it has Bluetooth and it is only slightly larger than a USB thumb drive. Instead of the huge LCD display that only display 4 lines of 20 characters, Adafruit now sells an OLED graphic display that is only 1.3 inches across but has 128 x 64 pixels of resolution. Because I no longer had to keep my arm in a particular position to drive the wheelchair, I could place my arm in a position that made it easy to hold the micro switches to activate the device. I now have a 3-D printer which allowed me to make a little gadget that attaches to the micro switches that makes it easier to position them in my hand securely. The only remaining problem was how to get power to device. I had solved that problem a few months ago by designing the Printy Boost battery pack which uses a LiPoly battery, an Adafruit charging module and a 3-D printed case of my own design. Click here to see the tutorial I wrote for the Adafruit Learning System showing you have to make your own Printy Boost battery pack. I had already been using this battery pack to supplement on my android or iPhone. I just needed to run a little cable from the battery pack over to the new remote.

I have decided to call it “IRBLE” (pronounced the same as herbal) which is an combination of IR for infrared and BLE for Bluetooth Energy. Here are a series of photos of the project.

This is me holding three micro switches in my right hand. The orange plastic is a 3-D printed adapter that lets me hold the buttons in the proper orientation.

3_switches

Here are the Adafruit Micro BLE board, the Adafruit 1.3 inch OLED, and my custom-designed infrared output board that I use for all my projects.

The Adafruit Micro BLE measures about 2" long.

The Adafruit Micro BLE measures about 2″ long.


oled_irio

Here is the 3-D printed case will I made for the new device. Below that are some screen grabs from Blender 3-D when I designed the box.

case
cad1
cad2

Here’s the thing all wired up. It looks a mess but it works. Thanks to dad and his excellent wiring skills.

parts

Here are the parts in the case.

assembled

Here is the backside of the assembled device showing the infrared LEDs protruding out of the case. These transmit the signals to the TV or cable box.

ir_led

You can also see the Velcro tabs is to attach it to my bracket on the wheelchair. Between the tabs is a tiny hole which we drilled in the back of the case so I could reach the reset button. A couple of times during the programming I had a glitch and nearly bricked the device. Fortunately if you hit the reset button at just the right moment and initiate and upload it will work again. I was worried I would not be able to recover it After one of the glitches. Believe it or not even though this board was only introduced a few months ago, it has already been replaced by a new model. Adafruit has a new line of development boards called “Feather” that come in a variety of configurations. If I had ruined my board I already had purchased one of the newer Feather models but I would’ve had to redesign the wiring harness and the 3-D printed enclosure. Fortunately I was able to recover it.

Finally here is the final product mounted on my bracket next to my iPhone 6.

final

You can see on the side of the mounting practice holding the old capacitive touch foam rubber tip for my stick. I probably will never use it again but there is one final thing for which I might use the stick. When playing Candy Crush on a timed level I cannot work the mouse with my voice control or IR remote quickly enough to finish the level. I can only do those levels with the stick in my mouth. But other than that, I won’t be needing the stick anymore.

The TV in my living room, bedroom, and office are all Samsung and use the same codes even though they are different models. The cable box in my bedroom in the living room uses the same codes. In addition to being able to control all of that, and the mouse and arrow keys on my desktop PC, also programmed in the kitchen TV and the surroundsound amplifier in the living room. As I’m writing this, I still need to program in codes for my Blu-ray player. I’m a little bit worried I might run out of space on the Arduino. It is only has 2K of RAM memory and 32 K flash memory. I only have about 150 bytes of RAM remaining and I’ve used 94% of the flash memory. Fortunately Adafruit also has a Feather board that uses ARM Cortex M0 processor. https://www.adafruit.com/products/2995

It runs at 48 MHz, as 256 K of flash memory, and 32 K of RAM memory. I would have to rewrite my infrared code library but I intend to do that anyway.

Among the things I can do besides make phone calls and run most apps, I have an app that gives me remote control of the Kodi media players on my raspberry pi. Of course I could modify the design of the remote to use a Wi-Fi add-on and control the raspberry pi easier than through the phone. Adafruit does not yet make more than has both Wi-Fi and Bluetooth but my guess is by the time I got around that, they will have one. Possibly a Wi-Fi add-on for the Bluetooth board or a Bluetooth add-on for a Wi-Fi board. For now I got a remote do absolutely everything that I need it to do.

We can finally retire my last stick.

 

Epilog: Recently I was hospitalized and had to be put on a ventilator. The only way I could effectively communicate with doctors and nurses was to use this Bluetooth device to type messages into the notepad application on my iPhone. You can read a multipart blog about my hospital adventure titled “Pray That They Listen to the Man with No Voice“. Also I’ve written a tutorial for the Adafruit Learning System describing how to build a Bluetooth device for iOS switch control. You can read it here.

One more item… After over a year of faithful service my Ultimate Remote gave up the ghost. Here is an article about the replacement unit that I built.

The Ultimate Remote is Dead. Long Live the Ultimate Remote 2.0

R.I.P. Pinoccio… It Was Fun While It Lasted

Pinoccio

Pinoccio Wi-Fi Development Board

I’m a tiny bit sad to report the death of Pinoccio. Note that the missing “H” is deliberate. I’m not talking about Pinocchio the beloved character from children’s literature. I’m talking about “Pinoccio” (without the “H”) open-source semi-Arduino-compatible hardware and software platform. Actually to say that I’m reporting it is a bit of yesterday’s news. It was actually reported here in a blog post on their website dated March 30, 2015. The reason I didn’t see that blog post right away is that I had pretty much given up on the system several months earlier.

Here’s a bit of history… In late 2012 or early 2013 I became aware of this Indiegogo campaign titled “Pinoccio – A Complete Ecosystem Building of the Internet of Things” It was a small 1″ x 2″ Arduino compatible board using a chip similar to the one used by the Arduino Mega. This variety had a built in mesh radio such as used by the X-Bee system. It had a built-in LiPo battery, an RGB LED, a temperature sensor and a Wi-Fi backpack. You could connect to one of the units called a Lead Scout via Wi-Fi and then it would communicate to the other units over mesh radio which were known as Field Scouts. I quickly signed up at the $149 support level which would get me a Lead Scout and a Field Scout. Although the units were a bit pricey, if you added up what it would cost to buy an Arduino Mega, battery, charger circuit, X-Bee radio, and Wi-Fi module it seemed like a bargain at the time. They were seeking $60,000 in funding and end up raising over $105,000. Put this in proper context note that this was several months before Spark Core appeared on Kickstarter. Note that Spark Core has recently rebranded as Particle.io.

Had Pinoccio made its original delivery date of July 2013 they would’ve beaten everyone to market and would’ve been a major player in the Internet of Things movement. There were plans for a smaller version without the battery and sensors that would’ve been easier to embed into production products. There were plans for a variety of backpacks or shields including the one I was most anxious to see… infrared LED and receiver. My goal was to use it to create an Internet capable IR remote. I wanted to create an inexpensive alternative to the RedEye IR Remote from ThinkFlood. And it was a good thing I was planning such because ThinkFlood was shutting down as seen here.

My Pinoccho boards upon arrival.

My Pinoccho boards upon arrival.

The folks from Pinoccio quickly learned the hard lessons of crowd sourced funding. Lesson number one is inevitably “If you’ve got a great idea it’s easier to raise money then it is to bring a product to market.” I won’t bother to recount the litany of things that caused multiple delays but the bottom line is that backers did not get their units until April 2014 a full 10 months past the initial estimated delivery. The reasons were the usual things: finding suppliers, finding someone to make the boards, setting up the business itself. It seemed that the main reason/excuse for delays was that “We want to get the software right.” Here is my blog post which gave my initial positive reaction to the devices when they finally arrived.

The software that we waited so long to finally get was indeed quite extensive. There was an Internet portal called hq.pinocc.io that allowed you to do many things without even writing any code. And it included an extensive script language that you could write small programs directly from the HQ application without having to use the Arduino IDE or do any C++ coding. They wanted you to be able to plug the thing in and do something with it right away. That ability is one of the key things that led to the success of the original Arduino. With a built in RGB LED and temperature sensor, you could do more than just blank the LED on pin 13 like you can with Arduino out-of-the-box.

However if you wanted to write your own applications, the learning curve at least for me was very steep. I had no experience in connecting devices to the web. I did know quite a bit of JavaScript programming, CSS, and DOM HTML programming which was useful. I had a little bit of experience with PHP however that did not apply in this case. What I did not have any idea how to do what is called “asynchronous communication” with Web servers.

I was taught as a programmer from day one that execution of a program started at the top of the main module and went step-by-step through the module sequentially unless modified by the if/then/else or some sort of loop structure. But asynchronous coding is something significantly different. You make a request to a Web server and within that request, you send it the code that you want to execute when it finishes the job. That seems simple enough. I figured it would call the code when it was ready and then move on to the next step. But it doesn’t work that way. You move on to the next step immediately after making the request and you have no idea when it’s going to complete that request and invoke your callback function. I would make a request from the server for a piece of information. My callback function would tell the server where to put that information in my program. Then I would try to display that information on the webpage. But it was never there. When I would insert debugging code to see why it wasn’t there, it was there. It reminded me of the paranoid poem about the little man on the stair.

“Last night I saw on the stair
The little man who wasn’t there.
He wasn’t there again today.
Oh how I wish he’d go away.”

I was being haunted by the returned data that wasn’t there 🙂

What I didn’t understand was that this callback function wasn’t a normal subroutine where you call it, it executes, and when it returns you carry on. You have to carry on immediately after the request and you don’t know how long it’s going to take for the Web server to return your data. So you have to put something in the callback function that tells your main program “Hey he finally got what you asked for. You can use it now. It’s really here.”

When you are collecting several pieces of data in a row and you don’t want to proceed until you’ve got them all, the nested callback functions can get pretty hairy. It took me a long time to figure out what I was doing wrong. It also took a lot of patience on the part of the people from Pinoccio in their support forums to walk me through my inadequacies is particular area of programming. I owe them a tremendous debt of thanks. (Humorous note here… When I dictated the previous sentence using my dictation software, it misunderstood me and said “I/O them…”. How appropriate for a “.io” system). Anyway they were most gracious and patient with me.

I did my best to pay back their kindness. I blogged everything I could about the device. I showed it off on the Adafruit Google+ Hangout Show-and-Tell. I blogged about how to get around some SSL certificate problems that cropped up along the way. And I did my best to educate them in the hardware requirements necessary to make an infrared backpack for the Pinoccio system. And ultimately I did get a working Internet-of-Things infrared remote that I used to operate my TV and cable box.

My cable box is located in the living room but I have an HDMI cable that runs into my adjacent office so that I can watch the same thing my dad is watching and living room. I can get on a webpage in the office and control the cable box in the living room. I used it every day for several months and it was a real godsend. I succeeded in replacing my defunct and unsupported RedEye remote.

I included Pinoccio support in my next public release of my infrared library IRLib. I began cleaning up my asynchronous webpage code and preparing to publish it to share with the world everything I have learned so that the next novice who came along would have a clear and simple tutorial to follow I would not have to bother the people in the support forum to learn the things I have learned the hard way.

One of the things that bothered me about the entire system was that it was totally dependent upon the website hq.pinocc.io servers were maintained by the Pinoccio organization. On occasion, their server would crash and I would be stuck with a useless system. I asked them what do we do to remove that dependence. The old RedEye remote system was accessed directly to the local IP address of the device itself. It also included its own Web server in the device. In contrast the Pinoccio system required me to make my own webpage to control the device (as great as HQ was it wasn’t sufficient). That meant I had to install a Web server on my PC. I can only control the system from that PC unless I uploaded the webpage to cyborg5.com which I didn’t want to do. Setting up WampServer on my PC was much easier than I anticipated. That allowed me to host my controlling webpage with no problem.

Furthermore my webpage could not communicate with the device directly. It had to go through their servers. My controlling webpage went to a URL on their server and their server connected back to the device itself. The whole point of Pinoccio, for me at least, was to replace the defunct out of business RedEye Remote. What was I going to do if Pinoccio ever folded? They assured me that they were working on a system that would allow direct connections without going through their server system. And in the end they did so.. They developed a system where you could plug one Scout into a PC and have it communicate directly with the other Scouts using the mesh radio. You didn’t need the Wi-Fi at all. They called it a “Bridge Scout”. While that was of suited my purpose, still wouldn’t have given me all of the capabilities the device should have had. There supposedly is a way to set up one’s own server that would duplicate their part of the infrastructure. I believe them when they tell me that. However doing that myself would required another steep learning curve that I didn’t really feel like attempting. It would require more than I can get out of WampServer. It involved node.JS and a variety of other things that were beyond me.

Unfortunately somewhere along the way the device got a bit glitchy. There were constant software updates some of which I kept up with and some I did not. After not doing updates for several months, I finally decided to do an update in hopes of fixing the glitches. The end result was my device quit working altogether. I spent a couple of weeks presuming that the problem was an adverse interaction between my infrared code and their systems software. It involved going onto GitHub and figuring out which version of the software I had been using that was working somewhat well and then individually applying each and every update that had occurred over the period of several months. I finally found the patch that was causing my code to crash. In the end it was not an adverse reaction between my infrared code and their system. I was able to whittle it down to an example that did not include my IR code at all.

Somewhere in the midst of all this, I ended up “bricking” my lead scout. Somehow during a crash, the boot loader was overwritten. They talked me through a system wherein you could use one Scout as an ISP to restore the boot loader on a broken Scout. Initially that process did not work because I tried to do it with the Wi-Fi backpack still installed. Somewhere along the way while adding and removing the Wi-Fi backpack without disconnecting the power, I permanently damaged the Wi-Fi backpack. That meant that the so-called “Ecosystem for the Internet of Things” was no longer Internet capable. A replacement Wi-Fi backpack would cost $79. I was able to get the system running somewhat using the bridge Scout they described where you had to keep one unit plugged into a PC to communicate with another unit. However I had become frustrated with the whole thing and needed to get away from it for a while.

arduino_yunI ended up ordering an Arduino Yun from Adafruit.com. I immediately fell in love with the device. It consists of a powerful microprocessor running a minimal version of Linux and an Arduino Leonardo in one device. The communication between the Leonardo and the Linux was well defined and easy to use. The Linux portion did include its own Web server. You could also update the onboard Arduino sketch and the associated webpage using the latest Arduino IDE and do so wirelessly for Wi-Fi. That ability to update wirelessly was something that Pinoccio had promised from the beginning but never did fully implement at least from a users perspective. They did allow you to update system software wirelessly but not user programs at the Arduino level.

Although I did have the Pinoccio experience under my belt along with the experience of doing asynchronous callbacks to Web servers, setting up my IR remote on the Arduino Yun took me only a day and a half rather than the weeks and weeks of frustration that I had with Pinoccio. One of the things that was probably the nail in the coffin of Pinoccio for me personally was the fact that I could buy an Arduino Yun with its embedded Linux, Wi-Fi, Arduino Leonardo, and built-in Web server with no external infrastructure needed for about $75. That was four dollars less than the Wi-Fi backpack alone for Pinoccio.

With the success of modules like the CC3000, Particle a.k.a. Spark Core, and the amazing new and inexpensive ESP8266 there was no hope for Pinoccio. They might have salvaged it by ditching their own Wi-Fi backpack and making a backpack using the ESP8266.

My intent had been to take a break from Pinoccio after I got the other system up and running. I still felt like I owed them those tutorial blogs to help other Pinoccio users not have to go through the blood sweat and tears that I’ve been through. But I got distracted with other projects and never was able to do it. Now of course there is no more Pinoccio. Their servers are shutting down at the end of the month and anyone who hasn’t figured out how to launch their own server or who is content with using bridge mode is going to be out of luck.

Ironically my old RedEye Remote has in some ways outlasted the Pinoccio. Even ThinkFlood shut down over two years ago, you could still use their device as long as you didn’t need to update the software. I’ve been using mine as an Internet remote to control my bedroom TV and cable box in the same way I’m using the Arduino Yun for the living room. However I recently purchased a new TV in the bedroom and I cannot update the RedEye with the new codes. I had to order another Arduino Yun.

I had a bit of a scare because the Yun has been out of stock at Adafruit for a couple of months now. Fortunately I found one at my second favorite supplier SparkFun.com. I’m pretty sure that the shortage is a consequence of the trademark disputes between Arduino.cc and their previous hardware manufacturing company which has now rebranded itself as Arduino.org. Is my belief that Arduino.cc should and will win the dispute. And I’m absolutely ecstatic over the new announcement that Adafruit will become the US manufacturer for Arduino.cc. I’m confident that both Arduino and Adafruit will be around for a very long time.

In the end I think I’m going to miss Pinoccio. I was quite fond of the little fellow and a bit disappointed he never really achieved his full potential. I don’t regret any of it however. I learned a lot of useful programming. And I learned some lessons about crowd source funding. And it serves as a cautionary tale about Internet of Things systems that are too closely linked to their infrastructure. I think that as IoT continues to grow that’s going to be an important lesson for everyone.

Zoomer Dino Controlled by Arduino IR Remote

satb100

One of the hottest toys this Christmas is the Zoomer Dino. It is a remote-controlled self balancing two wheel robot dinosaur with an infrared remote. I got one for my 10-year-old nephew for Christmas and after playing around with it I had to get one of my own. I have been thinking about building Arduino powered two wheel balancing robot from scratch but here was one that works right out of the box. I’m not sure if it has a gyro or accelerometer to keep it balanced. He rolls around on his own making noises including burps, farts, and laughs. You can also take over control with an infrared remote. The remote has a joystick and three pushbuttons. The joystick moves the head up, down, left, right. If you hold in the “run” button while moving the joystick he rolls around under your control rather than moving the head. There also is a button for “chomp” which makes his jaw chomp open and closed. Additionally there is a “angry” button which makes his eyes glow red and he thrashes around angrily.

Zoomer Dino and IR remote

Zoomer Dino and IR remote

As with my IR controlled to a helicopter before, I had to see if I could reverse engineer the protocol it was using for the remote control. It was one of the most challenging IR reverse engineering projects I’ve ever encountered. I finally had to go out and purchase one for myself because I had to wrap up the original one for my nephew.

I used my IRLib Arduino Library for decoding and encoding IR signals. Once you push a button to get things started, the remote puts out a continuous signal that is the neutral position of the joystick with no buttons pushed. I came up with a raw dump of the timing values using my IRrecvDump sample sketch. Here are the results.

This is a quite typical set of signals used by many protocols. Only the timings and number of bits are unique. We have a header that consists of about 9200 mark and 4800 space followed by 25 data bits and a closing mark. The data marks run about 500 or 550. The data spaces are either about 650 or 1800. This variable spacing is the most common way to encode zeros and ones. We’ll call the short spaces ones and the long spaces as ones. I created a custom decoder object as follows.

Because the remote sends signals continuously, this routine only reports the initially received value and then anytime the value changes to something different. It outputs 6 hex digits but note that there are 25 data bits being received. I then plugged the hex values into an Excel spreadsheet, converted them to decimal, and then parsed out the individual bits to try to deduce a bit pattern. I tried leaving the joystick alone when pushing the 3 buttons individually. Then I tried moving the joystick full throw forward, backwards, left, and right. The results are shown in the table below. You may click the table image to enlarge it.

Table 1-Bit patterns from Dino remote.

Table 1-Bit patterns from Dino remote.

In all samples, the first three bits were always zero as was the last bit. It was pretty obvious that after the initial zero bits, there was a five bit field that was the left/right position of the joystick with the left position having a value of zero, the center as 16, and the right was 31. This was followed by another five bit field that was the forward/backwards joystick position with zero to the back, 16 in the center and 31 full forward. The next three bits were obviously the three buttons. There were eight additional bits that seem to have no rhyme or reason to them.

Here is a revised version of the receiving code that parses out the bits into their individual fields.

#include
#include

class IRdecodeDINO: public virtual IRdecodeBase
{
public:
bool decode(void);
void ParseFields(void);
void DumpResults(void);
unsigned JoyX, JoyY, Parity;
char Chomp, Angry, Run;
};
#define DINO_HEAD_MARK 9200
#define DINO_HEAD_SPACE 4800
#define DINO_ZERO_MARK 500
#define DINO_ZERO_SPACE 650
#define DINO_ONE_MARK 500
#define DINO_ONE_SPACE 1800
#define DINO_SAMPLES 54
#define DINO_BITS 25

#define DINO (LAST_PROTOCOL+1)
void IRdecodeDINO::ParseFields(void) {
long Temp = value>>1;
Parity = Temp & 0x0ffL; Temp= Temp>>8;
Run = Temp & 0x1L; Temp= Temp>>1;
Angry = Temp & 0x1L; Temp= Temp>>1;
Chomp = Temp & 0x1L; Temp= Temp>>1;
JoyY = Temp & 0x1fL; Temp= Temp>>5;
JoyX = Temp & 0x1fL;
};
void IRdecodeDINO::DumpResults(void) {
ParseFields();
Serial.print(F("Decoded DINO: Value:")); Serial.print(value, HEX);
Serial.print ("\tJX:"); Serial.print(JoyX,DEC);
Serial.print ("\tJY:"); Serial.print(JoyY,DEC);
Serial.print ("\tCh:"); Serial.print(Chomp,BIN);
Serial.print (" Ag:"); Serial.print(Angry,BIN);
Serial.print (" Run:"); Serial.print(Run,BIN);
Serial.print (" Parity:"); Serial.println(Parity,HEX);
};
bool IRdecodeDINO::decode(void) {
IRLIB_ATTEMPT_MESSAGE(F("DINO"));
if(!decodeGeneric(DINO_SAMPLES,DINO_HEAD_MARK, DINO_HEAD_SPACE,
0, DINO_ZERO_MARK, DINO_ONE_SPACE, DINO_ZERO_SPACE)) return false;
decode_type= static_castDINO;
return true;
};

IRdecodeDINO My_Decoder;

int RECV_PIN =11;

IRrecv My_Receiver(RECV_PIN);
long Previous;

void setup()
{
Previous=0;
Serial.begin(9600);while (! Serial);delay(1000);
Serial.println(F("Send a code from your dino remote and we will decode it."));
My_Receiver.enableIRIn(); // Start the receiver
}
void loop() {
if (My_Receiver.GetResults(&My_Decoder)) {
My_Decoder.decode();
if(My_Decoder.decode_type == DINO) {
if(My_Decoder.value!=Previous) My_Decoder.DumpResults();
Previous= My_Decoder.value;
}
else Serial.println("Unknown");
My_Receiver.resume();
}
}

But what to do about those parity bits? I had to presume they were some sort of checksum or other data verification field. Normally when capturing and decoding IR signals I really don’t care what the data represents internally. I point a TV remote at my receiver circuit, capture the hex value, and then re-create that value using my IR transmitter. I could have just quit now and build an application on the Arduino that would transmit button pushes and fixed joystick positions but it would be nice to be able to use any joystick position that I wanted from 0- 31 in either X or Y directions in any combination. But to do that I needed to figure out how to compute those additional eight bits of data.

Most consumer-electronics protocols use a simple checksum of adding the previous data fields together. Sometimes you take the 1’s complement of the data and repeat it. Sometimes you use bitwise XOR to combine the fields. I spent three days playing around with various versions of my Excel spreadsheet analyzing lots of bit patterns captured from the remote and could not deduce the pattern. I asked a couple of online acquaintances if they had any suggestions but they’re only input was to try checksum, XOR etc. A cyclical redundancy check or CRC was another option but that would be hard to reverse engineer and they generally are not used for such short streams of data. Keep in mind were only talking about 13 bits of actual data if you don’t count the leading and trailing zeros or the eight check bits.

At one point I remembered there was something called a Hamming error correction code. I had learned about them in college in my CS 484 class with Dr. Judith Gersting over 25 years ago. I barely understood how they worked back then and I’ve not used them ever since so the chances of me being able to figure it out on my own were represented by the three initial zeros in the data stream 🙂 Dr. Judith Gersting was not my only mentor in college. Her husband Dr. John Gersting told us on the first day of programming class that he was not going to teach us programming. He was going to teach us to teach ourselves programming because the minute we walked out the door, everything would probably be obsolete anyway. He literally handed us a textbook that he wrote and told us to go teach ourselves the course. I guess I was going to have to do some Google searches and teach myself everything I needed to know about Hamming codes.

As it turns out there is no one single way to create an error detecting/correcting code. There are entire websites devoted to multitudinous ways and they all use notation that I didn’t understand. I did find one website from UMass linked here.

http://www.ecs.umass.edu/ece/koren/FaultTolerantSystems/simulator/Hamming/HammingCodes.html

One thing I have learned about Hamming codes was that they typically only add a vary few extra bits. Using the calculator linked above if I put in a 16-bit value it would only add five additional bits. I was getting eight additional bits on just 13 bits of data. But the one thing I did learn from that page was the way you calculate those extra parity bits. There is a matrix that tells you which of the data bits to add together to get each of the individual parity bits. I really had to do was figure out which data bits contributed to each parity bit. I had already noticed that if you push the “run button” which I have listed as “B0”, that parity bits P2,P1,P0 all inverted from their normal position. Similarly the “angry button” which is “B1” toggles parity bits P3,P2,P1. Similarly the “chomp button” labeled “B2” toggles P4,P3,P2.

So I made a new set of columns in the spreadsheet to compute the parity bits. I begin by saying…


P0=B0
P1=B0+B1
P2=B0+B1+B2
P3=B2+B1
P4=B2

Additionally the center position of the joystick had only a single bit set, the highest order bit of the five bit field which was value 16. By comparing those to the joystick positions which were all of the way left or on the way back (which consisted of all zeros) I could figure out which parity bits changed based solely on X4 and Y4. Quite by chance I also found had captured samples with X positions of exactly 8, 4, and 2. So that told me which parity bits were based upon X3, X2, X1, and X0. As I updated the formulas for calculating each parity bit, I compared the computed results to actual capture values for every hex value I’d captured. I had a list of over 30 values that I had dumped from simply wiggling the joystick around. Eventually I was able to fill in the blanks and came up with the following formulas.

P0=X4+X0+Y4+Y3+B0
P1=X4+X1+Y3+B1+B0
P2=X4+X2+X0+Y3+B2+B1+B0
P3=X3+X1+Y4+Y0+B2+B1
P4=X4+X0+Y1+Y0+B2
P5=X3+X1+Y2+Y1+Y0
P6=X4+X2+Y3+Y2+Y1
P7=X3+Y4+Y3+Y2

Comparing my computed results to all my captured results I was convinced I had completely deduced how the parity bits were calculated. When you put those results in a table as follows, there is a definite pattern to them. It looks a lot like the patterns that had been seeing on various websites.

Table 2-Parity calculation matrix.

Table 2-Parity calculation matrix.

Finally I was able to create a sending routine that allowed me to test all of this out. This is a somewhat stripped-down version of what I ultimately came up with. You upload the sketch, call up the serial monitor, and type various characters into the monitor. The sketch then interprets those as commands and sends the proper hex code to the IR transmitter.

#include
#include

class IRsendDINO: public virtual IRsendBase
{
public:
void send(unsigned long data);
};
#define DINO_HEAD_MARK 9200
#define DINO_HEAD_SPACE 4800
#define DINO_ZERO_MARK 500
#define DINO_ZERO_SPACE 650
#define DINO_ONE_MARK 500
#define DINO_ONE_SPACE 1800
#define DINO_SAMPLES 54
#define DINO_BITS 25

#define DINO (LAST_PROTOCOL+1)

void IRsendDINO::send(unsigned long data) {
sendGeneric(data,DINO_BITS,DINO_HEAD_MARK, DINO_HEAD_SPACE,
DINO_ONE_MARK, DINO_ZERO_MARK,DINO_ONE_SPACE, DINO_ZERO_SPACE, 38, true, 0);
delay(27);//Average gap was 26700 us
};

IRsendDINO My_Sender;

int RECV_PIN =11;

IRrecv My_Receiver(RECV_PIN);
char Mult, Normal, Chars;
#define COUNT 7

void SendMultiple(unsigned long Data, char Count) {
for(char i=0;i<(Count);i++) {My_Sender.send(Data);}; }; void setup() { Serial.begin(9600);;delay(2000);while (! Serial);delay(2000); Serial.println(F("Enter one of the following letters A,C,U,D,L,R,F,B,<,>,N,0"));
Normal=1;
};
void loop() {
char Cmd;
if (Serial.available()>0) {
switch (Cmd) {
case 'A': SendMultiple(0x2105a0, 9); break; //anger
case 'C': SendMultiple(0x201984, 6); break; //chomp
case 'U': SendMultiple(0x21f162, COUNT); break; //head up
case 'D': SendMultiple(0x2000ae, COUNT); break; //head down
case 'L': SendMultiple(0x010112, COUNT); break; //head left
case 'R': SendMultiple(0x3f003a, COUNT); break; //head right
case 'F': SendMultiple(0x21f36c, COUNT); break; //forward
case 'B': SendMultiple(0x2002a0, COUNT); break; //backwards
case '<': SendMultiple(0x01031c, 5); break; //spin left case '>': SendMultiple(0x3f0234, 5); break; //spin right
case 'N': Normal=1; break; //send neutral commands
case '0': Normal=0; break;//send wiggle commands
}
} else {//if no serial data then send either normal play mode or sit still
if(Normal) {
//Un-comment next line to have Dino do normal behavior.
// My_Sender.send(0x2101bc); //Comment out to turn off continuous transmission
} else { //wiggle slightly in an attempt to set still
My_Sender.send(0x21132c);
My_Sender.send(0x20f27e);
}
}
}

The actual remote sends continuous streams of data even when you aren’t doing anything. If you don’t touch any buttons or move the joystick for about one minute it eventually shuts down. If you are not pushing a button and the joystick is in the neutral position, the dinosaur engages in various preprogrammed behaviors. I needed a way to get him to sit still while I’m typing in the next string of letters. If you select “Normal” mode. It doesn’t send any signals. If you press a “0” it continuously sends a tiny forward and backwards code which basically has him sit in one place. He wiggles a tiny bit and make some noises but he doesn’t go off on his own. Then when you type other letters he goes in the direction you want him to. You can look at the “switch statement” to see which characters perform which functions. I send multiple transmissions of each code. The default number of signals is defined in COUNT so it was easy to change the default. I have a more advanced version of the sketch which causes him to move at various speeds, shake his head repeatedly, travel in zigzag motions etc. I also added commands to make him go forward and right or forward and left simultaneously. That way he doesn’t spin in place, he turns in a broad curve. In the demo video below I use that feature to make him travel in an oval-shaped pattern.

Know that all of the driving patterns I put him through in the video below could be done with the actual remote. You’ll have to trust me that they were created using an Arduino and my software.

My ultimate goal is to port my code to my pinoccio board which includes Wi-Fi and mesh radio. That way I can control him through a webpage similar to this. I will end up with an “Internet of things” remote control dinosaur. The pinoccio has a LiPo battery and is very small. I might just velcro it with an IR LED to the back of the dinosaur and not worry about maintaining a line of sight from my transmitter. I will post more details when and if I ever get that working.

Here is where I demonstrated the project on the Adafruit weekly Show-and-Tell on Google Plus Hangouts.

IRLib Updated to Version 1.5

We are pleased to announce that IRLib has been updated to version 1.5. IRLib is a library for Arduino-based microcontrollers that allows for the receiving, decoding, and sending of infrared signals. These new changes are a step forward in making the library less hardware platform dependent so that it can be more easily used by a variety of microcontrollers. These changes include…

  • New bit-bang option for PWM output frequency setting. Now can use any output pin with no hardware timers. Note:bit-bang output not as accurate as timer-based frequency selection.
  • Major rewrite of IRLibTimer.h to facilitate bit-bang. Separated hardware timer selection and specification into sending and receiving sections in order to implement bit-bang.
  • New IRfrequency class for detecting input frequencies. Previously was a stand-alone special sketch but now is a class including a DumpResults method.
  • New IRfreq and IRrecvDumpFreq illustrate simultaneous detection of frequency and pattern with both an IR learner and IR receiver is connected.
  • New #define USE_IRRECV define can be commented out to completely remove the interrupt driven IRrecv receiver class from the library. This resolves ISR conflicts with other libraries such as Tone() when not using IRrecv class.
  • New #define USE_DUMP define can be commented out to disable DumpResults methods. Saves code space when these methods are not needed.
  • Revised user manuals to document new features and correct previous documentation problems.

Note: The included user manual update is not yet available on the website at http://tech.cyborg5.com/irlib/docs/ but it will be updated shortly. New user manual is available with the library itself as a Microsoft Word .docx file as well as PDF and EPUB versions.
This library is available on GitHub at . For more information on this library see http://tech.cyborg5.com/irlib/

Pinoccio Internet of Things IR Remote Demo

satb100Here is a YouTube video demonstrating how I created an internet of infrared remote using the new Pinoccio platform. The Pinoccio is an Arduino compatible board based on the ATmega256RFR2 chip. It uses a Wi-Fi backpacked and built-in 802.15.4 mesh radio.

As mentioned in the video I have not yet released my source code for the Pinoccio port of my IRLib but I will do so very soon. The source code is now available on GitHub here. Read more about Pinoccio (and yes it really is spelled without an “h”) at https://pinocc.io. And my IRLib at http://tech.cyborg5.com/irlib/

Update: I also did a live demonstration of this project on the Adafruit Industries Google+ Hangout weekly Show-and-Tell which you can see here…

At some point I will post an extensive tutorial on this project.

Pinoccio Wireless Arduino Compatible Board: First Impressions

My Pinocchio boards arrived today in the mail. By the way I’m going to continue to call them Pinocchio rather than it to spelling Pinoccio simply because it’s easier to dictate using my voice control software wherein I do not need to go back and delete the “h” each time I type the word. For an explanation of the name you can see their FAQ here.

This board started out as a crowdsource funded project on indiegogo.com. It was billed as “A Complete Eco-System for Building the Internet of Things”. It consists of an ATmega256RFr2 microprocessor, a USB interface, a temperature sensor, an RGB LED, and a LiPo battery. The mesh radio built into the chip is a 2.4 GHz transceiver supposedly capable of ZigBee and IEEE 802.15.4 transmission although I’ve not yet seen anything about how to communicate with X-Bee or ZigBee modules from other manufacturers.

The package I ordered as part of my funding sent me two modules, one of them with a Wi-Fi shield or backpack as they call them. They call the boards “Scouts” so the one with the Wi-Fi shield is the “Lead Scout” and those without Wi-Fi are called “Field Scouts”. As a milestone perk I also got a prototyping shield and a small USB cable for charging and configuring. Here are some photos of the devices as shipped. (Click the images for larger view.)

Everything I Got

Everything I Got


Topside view: Lead Scout with Wi-Fi on top. Field Scout below.

Topside view: Lead Scout with Wi-Fi on top. Field Scout below.


Bottom view showing battery.

Bottom view showing battery.


The units use a micro USB cable. The first one I tried was a mini USB with an adapter on it. The adapter was too large because it got in the way of the Wi-Fi shield. I finally found a cable in my pile of cables. I wanted to be able to plug in both of them at once but as it turns out that may be a bad idea.

There was a card included that says to get started I should open a chrome browser and go to hq.pinocc.io and download their chrome app. I thought about browsing around the website and reading a bunch of documentation but one of the things the designers have said is that it ought to be easy to get started. They spent a lot of time working on the software and the front end for this thing rather than just creating the hardware and sending them out. In fact the original deadline for shipping was last July and here is the following April. So I thought I would give it the acid test and play dumb and see what I can find.

The app loaded just fine and I got a screen that told me to click to add a Scout control panel. They call the control panel “HQ” The first thing it said I had to do was to downloading Windows driver and install it. They tell you to plug in the device first and then it would tell me that no driver was available. Then it suggested I go to Device Manager via Control Panel in Windows and then I would find the device under “Ports (COM & LPT)”. It was not there it was under “Other devices” which is no big deal. I downloaded the driver and unzipped it into a folder. I would’ve preferred a self extracting installer but that was no big deal. When I tried to install the driver, Windows for bid me because it was not signed. No mention of that fact in any of their documentation so minus a few points for that problem as well. This was a Windows 8.1 64 bit machine. I had had trouble loading Arduino drivers on my laptop for the same reason and I found a website that explains how to permanently turn off that feature. I don’t know why I had not had the similar problem with Arduino drivers on my desktop Windows 8.1 PC. If you are interested… here is the website that tells you how to do it. That went relatively well.

Then I went to the process of configuring the lead scout. It took me several attempts and some email exchanges with their technical support people. Eventually it did start working but I’m not really sure what we did to fix it. It seemed to me that the steps they outline history followed in an exact precise manner.

Scouts in a mesh are configured together in something called a “troop”. So the first thing you have to do is give it a troop name. I chose “The Collective” (as an homage to Star Trek’s Borg). Then you name your lead Scout. I chose “Picard”. It then searches for Wi-Fi signals and asks you which Wi-Fi it should try to connect to. You are the Wi-Fi from the list and enter your password. There’s also a manual option if it cannot detect your Wi-Fi. When you completed that a screen pops up telling you that it succeeded adding your lead Scout to the Scout HQ. It then tells you to unplug the device so that it will reset and reconnect to your Wi-Fi. After that you click on the button that says “Done”. Apparently they are very picky about the order. You cannot click done until you’ve unplug the device and give it time to reset. Even then I had difficulty getting it to work right. In the end the only reliable way I could configure it was to unplug it and turn its power switch off and on. Then wait a few seconds before clicking done. Oh by the way when you first plug it in, the HQ searches for it and tells you that he could not find the device and that you probably forgot to turn it on. But it doesn’t tell you where the switch is. It’s not obvious. Anyway after unplugging, cycling the power switch, waiting a few seconds, and then clicking the done button it seemed to work okay.

You go to the same process for the field Scout although it does not prompt you for Wi-Fi information since it doesn’t have Wi-Fi.

The HQ app gives you the ability to turn on the RGB LED and change colors. It tells you the percent your battery is charged any gives you the temperature from the temperature sensor. I took my field Scout outside to see what the outside temperature was. Unfortunately it was raining so we had to put it in a plastic bag when sitting it out on the back porch. I don’t know how accurate the temperature was but it did change to a reasonable value. There are also controls for configuring the several I/O pins. There’s also a command line for sending for sending script commands. Here are some screen grabs showing the two scouts.

HQ control panel for the lead scout "Picard"

HQ control panel for the lead scout “Picard”


HQ control panel for field scout "Seven-of-Nine"

HQ control panel for field scout “Seven-of-Nine”

There are variety of script commands that you can use on the command line. Here is a link to the reference page.

The next thing I tried to do was use the Arduino IDE. You have to download a beta version because the current IDE does not support that chip. Again is a zip file rather than an installer. I already had the current production Arduino IDE installed on this machine. I took a chance and just copied the new files over the old ones. Apparently that was not the right thing to do because when I tried to compile an example sketch I get a “compiler error” with no error message whatsoever. I will probably go back and uninstall original Arduino IDE and delete all the files and try to reinstall the beta version from scratch. But that’s a task for tomorrow.

It’s going to be a steep learning curve to do anything really useful with this. Of course my ultimate goal is to port my IRLib and use it to make an “Internet-of-Things” infrared remote. Let’s going to take lots of time with data sheets and schematics. It looks like as powerful system but are not a long way to go in order to get it to do what I wanted to do.