Topics: Spinal Cord Injury, Assistive Technology Devices, Accessibility, Jabberwocky

Using a Smartphone Touch-Free with Jabberwocky

Access to technology is a key part of living an independent life. And for those living with high quadriplegia from a spinal cord injury (tens of thousands in the U.S. alone), this usually means finding a different strategy to access technology than the touchscreens and keyboards that most devices seem to rely on at first glance.

Fortunately, there have been several recent developments in touch-free access, on inexpensive and widely available hardware. For mobile devices, in particular, there are now options for total device control based on either head tracking or voice, for both Android and iOS.

Head Tracking

For supported devices on Android, our team built Jabberwocky Touch-Free Accessibility, a free app to control your entire device based on head movement. You have access to email, games, internet, video, and all existing Android apps. It uses the device’s front-facing camera and Google’s Augmented Reality (AR) technology to track head movement and display a cursor where your nose is pointing on the screen. Open and close your mouth to perform a tap at the location of the cursor. To swipe or perform complex real-time gestures, open your mouth and drag the cursor across the screen with head movement, and then close your mouth to end the gesture.

Jabberwocky Touch-Free Accessibility is currently available for free on the Android platform, for any device that supports Google Play Services for AR. We chose the Android platform for our total accessibility app because Google embraces third-party assistive technology developers by allowing them to build Accessibility Services, apps with elevated permissions necessary to control the entire device.

Apple, by contrast, has a restrictive security model that prevents third-party app assistive technology developers from building an app that controls the entire device. Unfortunately, this policy, which attempts to restrict the reach of malicious developers, dramatically reduces the capability of teams outside of Apple to contribute to their accessibility solutions. For example, we were successful in building a head-tracking-based web browser, which enables email, shopping, YouTube, etc., but not an app that controls the entire device.

As of iOS 13, Apple does offer its head-tracking accessibility service on devices with a TrueDepth camera (iPhone X and later, iPad Pro 2018 series). For more information, head to this webpage and search for “head tracking”. There are some caveats. For example, using a facial gesture such as a smile to click on the screen causes the cursor to jump around, resulting in misclicks. Try it out for yourself and see what you think.

Voice Control

For controlling your device with your voice, there have been dramatic improvements even within the last year or so. Both Android and iOS now offer a first-party solution: Google Voice Access for Android and Voice Control for iOS. These go well beyond the now-limited capabilities of something like Siri in terms of unlocking the whole device. You can click on elements of the screen, scroll, dictate, and use every aspect of the phone with your voice.

 

 

What's best for me?

Ultimately you will decide what’s best for you, ideally after trying out the options. In general, the “pros” for head tracking are:

  • simplicity,
  • finer-grained control and efficiency, and 
  • better suitability for games and browsing.

The biggest “cons” for head tracking are:

  • it requires the use of the neck, sometimes more than a user might have available, and
  • It is “always-on” (though usually, there is a simple gesture or action to disable/enable)

Some “pros” for voice control are:

  • speaking is already a learned physical action,
  • basic commands like starting an app are super-easy, and 
  • it has great integration with speech-to-text. 

“Cons” for voice control are:

  • its inefficiency for some tasks (like clicking on a specific part of the screen), 
  • the high learning curve on the different voice commands, and
  • privacy in some cases (if speaking in public).

Both head tracking and voice control can behave worse in less-than-ideal environments (bright lighting for head tracking and ambient noise for voice control). But it is surprising to see how much both are becoming resilient to distracting signals.

At the end of the day, find what works for you. One of my friends became very effective at playing Words With Friends on his phone using a stick attached to his forearm. No solution is perfect, but hopefully, now you have a few more options to check out!

Learn more about Jabberwocky for Android:

en_badge_web_generic

 

 

 

 

Aaron Chavez

Written by Aaron Chavez

Hi, I’m Aaron Chavez, co-founder and CEO of Swiftable. We’re the world leader in head tracking for mobile devices and we’re on a mission to enable independence through assistive technology that is incredibly intuitive, effective and fun. Previously, I served as Vice President of Engineering and Chief Scientist at AlchemyAPI, a Denver-based artificial intelligence company acquired by IBM Watson in 2015. I live in Arvada, CO with my wife, three children, and a cat. If you’ve got a question, send me an email!

See what Swope, Rodante Has Done for Others With Catastrophic Injuries

Louis Tontodonato's Story
A Story of Hope - Louis Tontodonato

“What made me most excited was to be able to help my family, being able to pull the burden out of them that they’ve been struggling with. That’s what made me feel the best. Everything else to come, is to come. And we’re going to live life how we have been, just not going to have to worry about buying the supplies that I need.”

Todd Cabral's Story
Motorcyclist Sustains Massive Injuries - Swope, Rodante Client Testimonial

"As I got better, which I did in the future, I think I remember researching them [Swope, Rodante P.A.] myself and found that they are very good at what they do. "