Smarter smartmirror

So I also decided to build myself a smartmirror. However, I want it to provide a little more functionality than just displaying some information and telling me that I’m beautiful. Here is the finished build:

And here is a video of the leap-motion-control in action:

I want to place it in my bathroom, because that’s the only place where I actually spend some time in front of the mirror. I do want some controls, but I do not want to touch buttons or the mirror itself, so I chose a leap motion controller. Below I will detail some of the steps I went through in building this thing.

Update: Speech recognition and the bloody-mary-protocol.

Hardware

Screen

I chose a 28″ screen (SamsungT28D310ES 71,1 cm) which offers all needed interfaces and should be slim enough for my build. It has been stripped out of the plastic case, but there is a small metal-frame that I left attached (works).

Computer

First I wanted to use a Raspberry Pi, but it is too slow for any real work and the Leap SDK is not available for the ARM platform. Instead I used a Intel Celeron G1610T (DualCore 2,3GHz) 8GB DDRIII, 500GB HDD, HDMI, (H61MV) e.Mini which I ordered some years ago from amazon

Frame

I chose to do a custom build with wood from the hardware-store and paint it white.

We painted three layers of waterproof glaze, then spraypainted it with white finish.

Mirror

After about 4 tries with some 2-way-sheeting and a nightmare of bubbles and scratches, I ordered a solid glass 2-way mirror from myspiegel.de. (In the meantime, 10 days later, nearly all of the bubbles are gone and it really looks like a mirror).

Software

The webpage can be found in this git-repo: https://github.com/TobiasWeis/smartmirror, the rest of the scripts are not compiled to a package yet, but have to be copied from this page.

Chromium-browser, some javascript,php and python

To autostart chrome in kiosk-mode and load the smartmirror-webpage, create ~/.config/autostart/chromium.desktop:

For the basic framework I took the code from https://github.com/Montellese/home-smart-mirror.git, which is an awesome clean and structured framework that already provided some of the functionality I wanted (time, live and forecast weather, icons). I extended it by writing my own videoplayer-, motion-sensing and bitcoin-module and integrated mousetrap to map keypresses to functions.

Input: Leap motion controller

First, the hardware to mount it: I chose to go with a headmounted setup to prevent water and other stuff on the device and to get it out of the way (you can find the scad- and stl-files here: http://www.thingiverse.com/thing:1454096 if you want to print it yourself):

For my feeling, the classification rate of more sophisticated gestures it pretty lame, even after the calibration. I chose to just use the hand-position and put a threshold on x-movement to detect left- or right-swipes – classification rate 100%! Using pykeyboard xdotools, I translate those swipes to keypresses for the browser. Also, a short info-sound is played as soon as the hand is recognized as such by the leap-device.

To make it autostart with the xserver, I created a file called ~/.config/autostart/leap.desktop (I am running gnome) with the following content:

LibCec and cec-client

Some TVs are able to receive commands over a CEC-interface, like turning on and off. Sadly, my graphics-card does not support this, so I ordered a Pulse Eight CEC-Adapter for USB. This is a USB-bridge with which you are able to “inject” CEC-commands into the HDMI-stream. With linux, turning on and off the TV (well, standby), is just a matter of:

echo "as 1" | cec-client -s

or

echo "standby 0" | cec-client -s

Motion sensing: Arduino with a PIR-sensor

I used an regular Arduino Nano over USB. A PIR-motion sensor is wired up to +5V, GND and a digital pin. The arduino constantly monitors the sensor and relays the read value to it’s serial interface. To avoid mixing up device-numberings, I created a udev-rule for the arduino (/etc/udev/rules.d/99-arduino.rules):

SUBSYSTEMS=="usb", ATTRS{idProduct}=="7523", ATTRS{idVendor}=="1a86", SYMLINK+="arduino_motionsensor"

This tells the udev-system to always create a device called “/dev/arduino_motionsensor” in addition to the regular “/dev/ttyUSB{0-9}” link.

int pirSensor = 2;

void setup() {
 Serial.begin(9600);
 pinMode(pirSensor, INPUT);
 digitalWrite(pirSensor, HIGH);
}

void loop() {
 int pirValue = digitalRead(pirSensor);
 Serial.println(pirValue);
 delay(100);
}

On the PC I use a script that is also started via the .desktop autostart-script:

Daily news: Tagesschau in 100 Sekunden

To download the “Tagesschau in 100 Sekunden” (German TV news, distilled to the most important 100 seconds), I used the following python-script to parse their website, find the link to the current video, and download it.

I just put the script in a cronjob so it downloads the video during the night:

00 06 * * * /home/sarah/scripts/tagesschau.py > /dev/null 2>&1

Additional ideas and extensions

IoT

Including an ESP or Bluetooth to connect to other bathroom accessory (like a scale or lights) -> Like a friend of mine used to say: “I don’t give a rats arse”. Really, I couldn’t care less how my weight is behaving, and I cannot imagine any other useful Internet-of-Things application (really usefull IoT is scarse, anyway).

Non-invasive health monitoring

That’s certainly an idea that I will pursue -> using a webcam to identify the face, try to measure drowsiness or general health state. The light my bathroom is artificial only, so it’s constant and controlled, which would enable to measure the color of the skin. Plus, with the temporal amplification algorithms of MIT (http://people.csail.mit.edu/mrub/vidmag/) it could be possible to measure the pulse.

Scaring guests

That one I got from my fellow researchers when discussing useful extensions over lunch. A “bloody mary protocol” would be is pretty nice (using speech recognition, say “bloody mary” three times and then display a really scary face of a woman. Another possibility would be to use some kind of facial puppet algorithm (detect facial landmarks of the viewer, map those to another, saved face and let this face mimic the viewer).

Update (29.04.2016): The blood-mary-protocol has been implemented!

I implemented this using the CMU Sphinx speech recognition toolkit, the python (swig) wrapper and a custom dictionary.

python-swig-interface: https://github.com/cmusphinx/pocketsphinx-python

create a custom dictionary and necessary files: http://www.speech.cs.cmu.edu/tools/lmtool-new.html

I chose not to integrate this in the official GIT-repo as it would be a lot of overhead just for this small functionality. I might however dedicate an extra article on the speech recognition. If you want details, please ask!

Here is the code I used after pocketsphinx and the above wrappers have been implemented:

#!/usr/bin/python
import os
import time
from os import environ, path

from pocketsphinx.pocketsphinx import *
from sphinxbase.sphinxbase import *

MODELDIR = "/home/sarah/code/pocketsphinx-python/pocketsphinx/model"
DATADIR = "/home/sarah/code/pocketsphinx-python/pocketsphinx/test/data"

config = Decoder.default_config()
config.set_string('-hmm', path.join(MODELDIR, 'en-us/en-us'))
config.set_string('-lm', '/home/sarah/code/pocketsphinx-python/wordlist_model/8879.lm')
config.set_string('-dict', '/home/sarah/code/pocketsphinx-python/wordlist_model/8879.dic') 
config.set_string('-logfn', '/dev/null')
decoder = Decoder(config)

import pyaudio

print "---------------------------------------- Searching and selecting audio device"
pa = pyaudio.PyAudio() 
chosen_device_index = -1
for x in xrange(0,pa.get_device_count()): 
    info = pa.get_device_info_by_index(x)
    print pa.get_device_info_by_index(x)
    if info["name"] == "HD Pro Webcam C920: USB Audio (hw:1,0)":
        chosen_device_index = info["index"]
        print "Chose index ", chosen_device_index
print "-----------------------------------------"

p = pyaudio.PyAudio()
stream = p.open(format=pyaudio.paInt16, channels=1, rate=16000, input_device_index=chosen_device_index, input=True, output=False)
stream.start_stream()

in_speech_bf = False
decoder.start_utt()
while True:
    try:
        buf = stream.read(8192)
        if buf:
            decoder.process_raw(buf, False, False)
            if decoder.get_in_speech() != in_speech_bf:
                in_speech_bf = decoder.get_in_speech()
                if not in_speech_bf:
                    decoder.end_utt()
                    res = decoder.hyp().hypstr
                    print res
                    if res == "BLOODY MARY BLOODY MARY BLOODY MARY":
                        os.system("mplayer -fs -volume 100 /home/sarah/code/pocketsphinx-python/bloodymary.mp4")
                    decoder.start_utt()
        else:                                                                                                          
            break
    except:
        print "Meh."
        pass
decoder.end_utt()