Writing a command and control application with voice recognition

30 Comments »

• Arnau says:

  July 27, 2008 at 12:10

  Guai!!! Quan tingui temps ho provaré!
• Loffe says:

  July 27, 2008 at 13:52

  Wow. It’s cool to control music by voice. Unfortunately it don’t work when I play songs very load :P

  Do you know if there’s a project to make voice control easy to the user? Something to control apps with a simple gui…
• Peng’s posts for Sunday, 27 July « I’m Just an Avatar says:

  July 27, 2008 at 16:30

  [...] Gevatter: Writing a command and control application with voice command. Let’s admit it. Most of us Trek fans have long waited for the day that we can control our [...]
• RainCT says:

  July 28, 2008 at 0:32

  [@ Loffe]

  Yeah, that with the loud music is true, but I think it works quit good anyway. While I was writing the script I wondered if it would recognize anything while I have music playing, but as you see up to a certain point it does :).

  About GUI, I know of the two following ones, but I haven’t tried them yet:

  – Simon (http://simon-listens.org/index.php?id=122&L=1), which uses Julius, but it doesn’t seem to build on Hardy.

  – The panel applet gnome-voice-control (http://live.gnome.org/GnomeVoiceControl), which uses a different engine, Sphinx (which from my previous experience with it seems to require a pretty good microphone – using the cheap one I have it only recognizes like 1 word out of 50). It is in the repositories, but the version in Hardy doesn’t work (although I just spoke with the maintainer and he is working on fixing it now).
• Gunni says:

  July 29, 2008 at 9:59

  Just if any other stumbles upon the error i got.
  The .dict file was not build, i had to use “sudo mkdfa sample” to build the dict and dfa file. I dont know why, cause the directory i created was in my homefolder, belonging me and was chmod 777 -R by me.
• VoxForge says:

  August 26, 2008 at 23:14

  Hi RainCT,

  thank you for creating julius and julius-voxforge in intrepid, that will make my project for school much more simple! Will it be updated when the new audio speech-files in VoxForge are processed? (Now there are around 50 new speech submissions.)
• RainCT says:

  August 26, 2008 at 23:43

  VoxForge: Hey. I’m happy that it’s useful for you :). Drop me a mail once the new version is released and I’ll try if I can get it in, but I can’t promise anything as Feature Freeze is about to start (which means that no new features can enter Intrepid).
• VoxForge says:

  August 27, 2008 at 9:28

  Hi RainCT,

  thank you for your quick reply!
  Is speech recognition accuracy really a feature? I thought that a feature is a new package with new options. It depends, I think, on the reviewer. Maybe you can better not release it, but waiting a few months and try then to release it in intrepid-backports as it might be a bigger improvement.
  I don’t know when Ken comes back from holiday and will process the submissions, but I’ll drop you a mail when it is ready.
  On the other hand, if it will not be entered in Intrepid, is it simply possible to overwrite the acoustic model in order to improve the accuracy?
• RainCT says:

  August 27, 2008 at 14:11

  Voxforge: Well, “new upstream versions” are considered features, unless they are “bugfix only”. But as VoxForge is not an application it may well be possible to get an exception. In regards to your last question, you can of course use any acoustic model you want (just write the path to that one you want instead of to /usr/share/julius-voxforge/acoustic/, in julian.jconf).
• VoxForge says:

  August 27, 2008 at 15:56

  Thank you!
• testing says:

  November 24, 2008 at 4:14

  Hello, thanks to your work I was able to install Julius under Ubuntu. I hope that you continue your good work. Greetings
• Sarah Bolopski says:

  February 7, 2009 at 0:49

  This is really cool. It’s like in star trek almost where you tell the computer to do something (like computer, turn on holodeck) and magically the computer does it.
• Frank says:

  February 17, 2009 at 23:43

  Thanks alot for posting this, this was exactly what I was looking for for a project of mine that I had a few months ago. Thanks alot!
• Sanyynn says:

  May 13, 2009 at 21:53

  I’ve waited so long for an application like this one. :) Easy to use, and easy to place new options. I configured it to open all my developing tabs on firefox in case of the ‘develop’ word. :D
• sander says:

  May 31, 2009 at 20:44

  I’m afraid your how-to is a bit too bare-bone for me. I may be an occasional hacker, but that’s mostly not (read: never) in Python. Could you explain how you exactly get the voice commands to work on Rhymthmbox?
• RainCT says:

  May 31, 2009 at 20:52

  @sander:

  In the example I run Julius and call the Python application so that it gets Julius’s output (from stdout), which I parse at real-time line by line. If that line is of the form «sentence1: <s> [...] </s>», it contains the recognized text, which in the parse() function is compared to the known commands. If a known command is found (eg., «play»), I call rhythmbox-client (a command-line tool to control Rhythmbox) for it to do the recognized action.

  I hope this clarifies it for you; if it doesn’t, ask again :).
• RainCT says:

  June 19, 2009 at 21:47

  FYI, I’ve just uploaded julius-voxforge 0.1.1~daily20090611-0ubuntu2 which provides an updated version of the script (including support for Banshee).
• sandrar says:

  September 10, 2009 at 15:42

  Hi! I was surfing and found your blog post… nice! I love your blog. :) Cheers! Sandra. R.
• phil says:

  December 23, 2009 at 2:54

  I got the “sample” program to work fine. So then I had a go at the “controlapp” program. It says “Taking control of Rhythmbox media player.” I then try saying the commands and such but nothing changes. I even started up Rhythmbox and set it up so that if I did hit pause or play or anything like that I would know. But nothing happens. I checked the dictionary to make sure the words like play and pause were there and they are but nothing happens still.
• phil says:

  December 23, 2009 at 2:58

  I just figured out that I forgot to add the lines ” sed -i ‘s/sample\.dfa/mediaplayer\.dfa/’ *.jconf
  sed -i ‘s/sample\.dict/mediaplayer\.dict/’ *.jconf”

  I hope this helps someone
• Sujay Shitole says:

  June 2, 2010 at 5:49

  Sir,
  i am trying to develop control application with voice recognition.After executing
  julius -quiet -input mic -C julian.jconf | ./command.py
  process comes in waiting state(cursor is continuos blinking ).Means sometimes it works and shows <<> and works fine.
  But there is no surety in working.
  Please help me .
• Sujay says:

  June 2, 2010 at 6:43

  Sometimes it gives error
  Error: adin_oss: failed to read samples
• Roland says:

  June 18, 2010 at 13:02

  Hi,

  can give some explanation on how do you actually send the “word” that you get from Julius to the python language?

  Thank you
• RainCT says:

  June 18, 2010 at 14:13

  @ Roland:

  You can see that in the example included in the “julius-voxforge” Ubuntu package. For your convenience, I’ll post a copy of the current Python code here:

  #! /usr/bin/python -u
  # (Note: The -u disables buffering, as else we don't get Julius's output.)
  #
  # Command and Control Application for Julius
  #
  # How to use it:
  #  julius -quiet -input mic -C julian.jconf 2>/dev/null | ./command.py
  #
  # Copyright (C) 2008, 2009 Siegfried-Angel Gevatter Pujals 
  #
  # This program is free software: you can redistribute it and/or modify
  # it under the terms of the GNU General Public License as published by
  # the Free Software Foundation, either version 3 of the License, or
  # (at your option) any later version.
  #
  # This program is distributed in the hope that it will be useful,
  # but WITHOUT ANY WARRANTY; without even the implied warranty of
  # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  # GNU General Public License for more details.
  #
  # You should have received a copy of the GNU General Public License
  # along with this program.  If not, see .
  
  # Supported commands:
  #
  # This file is provided as an example, and should be modified to suit
  # your needs. As is, it only supports a few commands and executes them on
  # either Rhythmbox or Banshee.
  
  import sys
  import os
  
  class Rhythmbox:
  	
  	name = "Rhythmbox"
  	
  	commands = {
  			'play': 'play',
  			'pause': 'pause',
  			'next': 'next',
  			'prev': 'previous',
  			'show': 'notify',
  			'pause': 'pause',
  			'silence': 'pause',
  	}
  	
  	def parse(self, word):
  		if word in self.commands:
  			return 'rhythmbox-client --%s' % self.commands[word]
  
  class Banshee:
  	
  	name = "Banshee"
  	
  	commands = {
  			'play': 'play',
  			'pause': 'pause',
  			'stop': 'stop',
  			'next': 'next',
  			'prev': 'previous',
  			'pause': 'pause',
  			'silence': 'pause',
  	}
  	
  	def parse(self, word):
  		if word in self.commands:
  			return 'banshee --no-present --%s %% ' % self.commands[word]
  
  class CommandAndControl:
  	
  	def __init__(self, file_object):
  		
  		# Determine which media player to use
  		if os.system('ps xa | grep -v grep | grep banshee >/dev/null') == 0:
  			self.mediaplayer = Banshee()
  		elif os.system('ps xa | grep -v grep | grep rhythmbox >/dev/null') == 0:
  			self.mediaplayer = Rhythmbox()
  		elif os.system('which banshee >/dev/null') == 0:
  			self.mediaplayer = Banshee()
  			os.system('bash -c "nohup banshee >/dev/null 2>&1 < &1 & disown %%"')
  		elif os.system('which rhythmbox >/dev/null') == 0:
  			self.mediaplayer = Rhythmbox()
  		else:
  			print 'Couldn\'t find a supported media player. ' \
  				'Please install Rhythmbox or Banshee.'
  			sys.exit(1)
  		print 'Taking control of %s media player.' % self.mediaplayer.name
  		
  		startstring = 'sentence1: <s> '
  		endstring = ' </s>'
  		
  		while 1:
  			line = file_object.readline()
  			if not line:
  				break
  			if 'missing phones' in line.lower():
  				print 'Error: Missing phonemes for the used grammar file.'
  				sys.exit(1)
  			if line.startswith(startstring) and line.strip().endswith(endstring):
  				self.parse(line.strip('\n')[len(startstring):-len(endstring)])
  	
  	def parse(self, line):
  		# Parse the input
  		params = [param.lower() for param in line.split() if param]
  		if not '-q' in sys.argv and not '--quiet' in sys.argv:
  			print 'Recognized input:', ' '.join(params).capitalize()
  		
  		# Execute the command, if recognized/supported
  		command = self.mediaplayer.parse(params[1])
  		if command:
  			os.system(command)
  		elif not '-q' in sys.argv and not '--quiet' in sys.argv:
  			print 'Command not supported by %s.' % self.mediaplayer.name
  
  if __name__ == '__main__':
  	try:
  		CommandAndControl(sys.stdin)
  	except KeyboardInterrupt:
  		sys.exit(1)
  

  You execute it like this:

  julius -quiet -input mic -C julian.jconf 2>/dev/null | ./command.py

  This way Julius will run and write the recognized words (together with some other useless text) to the standard output. That text will be forwarded to the Python script which can then identify the relevant words and see whether they are recognized.

  As you can see, this is just a quick hack. Theoretically it should be possible to use libjulius to access Julius directly from Python, but I’m not sure what the state of said library is at the moment and I haven’t found time to look into it yet.
• UDS 2009 @ Eurion · RainCT’s Blog says:

  June 27, 2010 at 16:01

  [...] speech recognition engine) is already packaged since Intrepid and some time ago I wrote about how to use it to control applications (including instructions to setup some basic commands to control [...]
• Updated Voxforge packages in Ubuntu @ Eurion · RainCT’s Blog says:

  August 20, 2010 at 14:52

  [...] in some months), but especially interesting is that it supports more phonemes now (remember the old days when words like “previous” or “computer” couldn’t be [...]
• sean says:

  October 17, 2010 at 1:57

  Hi it works great, im using the command.py that comes with julius-voxforge and would like to use the script to execute other programs and scripts could you please provide a more simple generic script template to work from (i cant code in python im currently learning bash), this is hard for me to disect and use for other things other than or rhythmbox/banshee, the command.py you have in your tutorial (not the voxforge one) is more like something i am after but i cant seem to get it to work if i use it on its own, i guess its a code snippet and not the full script, but i want to be able to do something like:

  
recognized word from julius

  commands:
  word=command
  word=command
  word=command
  word=command
  etc

  if recognised word = any of the words above
then
execute recognized command


  with no fancy stuff to check whether rhythmbox or banshee are installed etc just a bare bones and very simple version that can be used to execute any command

  also is this possible to do in bash, as i would ideally like to learn one language at a time,

  Anyhelp appreciated - Thanks!
• Voice control | GIMX – Game Input MultipleXer says:

  December 16, 2011 at 10:41

  [...] Julius+Voxforge: seems to be very easy to install, but no GUI. [...]
• blob says:

  February 27, 2012 at 2:20

  Can you update this for julius 4.2.1?
• Como usar o reconhecimento de voz para executar comandos no seu desktop Linux | Alan Carvalho de Assis says:

  November 10, 2012 at 16:09

  [...] http://bloc.eurion.net/archives/2008/writing-a-command-and-control-application-with-voice-recognitio… [...]