tl;dr I made an Arduino device that can be hooked up via USB to a computer that you wish to control and connected via Bluetooth to a controlling computer. Go to the “in action” section for a demo.

I control my computer by voice these days via a combination of VoiceCode + Dragon Dictate and some other supporting hardware/software.
A few weeks ago I was working on some servers which didn’t have a network connection yet forcing me to type by keyboard.
In these sorts of situations it is nice to be able to continue using my setup but to be able to do things on other machines.

Enter the replay keyboard device I put together with an Arduino.
You simply connect it to a computer you would like to replay commands (control) on, connect your computer via Bluetooth, then type away!

You can find all of the software here:

replay keys

Some of my use cases are:

• Controlling servers when there is no network (yay for failures!).
• Setting up new computers/devices that require keyboard input.
• Pair programming/working with others on their computer.
• Screwing with friends/enemies.

What follows is a high level summary of how the device works.

Hardware

• Arduino Micro
  • I used the Micro to use the Keyboard library which is only available on Atmega32u4 chipsets (Micro, Leonardo, Duo). This will NOT work on the Uno style chipsets.
  • The Micro is also nice because you can simultaneously use Serial1 (Bluetooth) and Serial (terminal output for debugging).
  • The Arduino is nice because it is lightweight and most modern operating systems have appropriate drivers already. This means you can plug it into any machine without having to install and write another driver. I imagine you can do this with many other types of devices (like a Raspberry Pi) but the Arduino starts up within seconds and it is easy to develop software for.
• KEDSUM Arduino Wireless Bluetooth Transceiver Module Slave 4Pin Serial + DuPont Cable (basic version)
  • In principle, I think any Bluetooth transceiver should work.
  • My main gripe with this one is that the documentation isn’t that great and some of the commands didn’t work that were advertised in the documentation.

Setup

There really isn’t too much to this. Connect the 5V directly to VCC, ground to ground, and TX (Arduino) to RX (Bluetooth) and RX (Arduino) to TX (Bluetooth).
The Arduino Micro is a good choice because it has an TX/RX (Serial1) pins that are isolated from the USB TX/RX (Serial) communication.
In theory, I think this should work with SoftwareSerial, but it didn’t work for me.
Maybe someone knows why this didn’t work?
This Bluetooth device works over SoftwareSerial with the Uno, for what it is worth.

Limitations/challenges

• The Arduino and Bluetooth transceiver do not have error correction built into communication, so in some sense they aren’t completely reliable. Unless what you are doing mission-critical, it hasn’t really been an issue for me. In principle, this is something you could develop.
• The stream that is sent over Serial is only a byte which poses some challenges.

Software

The general procedure is as follows:

1. A key is typed inside the window on the source device.
2. The key is logged and encoded using a specification.
3. The encoded sequence of characters is sent byte-by-byte over Bluetooth to the Arduino hooked up to the device you wish to replay keys on.
4. The Arduino receives the sequence of characters and presses them on the worker device.

All of the communication is done using one byte ASCII since this is easy to do with an Arduino.

Logger (source device)

This program is essentially a key logger.
It is a python program running a Tkinter GUI that listens to all of the keystrokes while the window is active.
Originally, I had planned on using ncurses but quickly realized that many of the keystrokes that I would like to send would not be captured.
Additionally, because it is running in a terminal, sometimes weird things happen to modifier keys.

The encoding for a key combination is as follows:

[N][optional modifiers][key]

where:

• N is the number of keys pressed at one time.
• optional modifiers is a set of modifiers. See logger.py for possibilities
• key the key being pressed

For example, the encoding for ctrl + shift + a would be 3<s>a.

Limitations

Since this is a program in user space, certain keystrokes cannot be captured, such as operating system shortcuts (e.g. cmd + tab).
Of course, you could bind aliases to such things.
For example, I bind ctrl + 0 to cmd + space since I rarely ever need to type ctrl + 0.

Arduino (replay device)

This program is a typical Arduino program with setup and loop functions (see replay_arduino).
The main loop waits until it sees a byte coming into the Bluetooth serial stream.
It then simply reads the number of keystrokes to expect, then parses the keys for that specific command.
It then presses all of the keys one-by-one until the combination has been completed, then releases them.
This all happens very quickly, possibly under a millisecond.

The only complication that I encountered was bytes being read at incorrect times due to the buffer not being filled.
Basically, if there is nothing in the stream, then the Arduino will simply execute the command while not reading anything from the stream and go forward.
The way that I dealt with this is by implementing a spinlock until there is something available in the stream (see safe_read()).

In action

Here is a demo:

Future work

The Arduino Micro has the ability to send mouse coordinates.
It would be cool to replay all of the mouse movements/commands.

I haven’t used it enough to notice issues with dropping packets, however, I imagine that at some point I will experience this.
If it ever becomes a major issue, it would be nice to implement some sort of error correction and recovery scheme.