Subject: [HOWTO] Implementing pitch-shifting and
crossfading (for filler music/other sources) for
PyKaraoke on Linux with JACK



PyKaraoke doesn't currently support pitch-shifting natively during
song playback, and this is about the only missing feature that
prevents PyKaraoke from being a complete digital replacement for the
traditional analog CD+G player.

This is the case because PyKaraoke relies on Pygame's MP3/Ogg/WAV
playback engine, which does not (currently) provide any means of
applying effects filters to the sound before it hits the sound device.
It would be possible to replace this with our own threaded MP3/Ogg/WAV
player, and use libsoundtouch to perform pitch-shifting (it is a very
high quality, faster-than-realtime pitch shifter and time
compressor/expander), but that's a significant undertaking. There are
no Python bindings for libsoundtouch, so using it would mean mucking
with SWIG, which has always given me fits.

On Linux at least, there is an alternative solution that ultimately
provides more flexibility anyway in building a sound layer that
permits all sorts of cool sound effect possibilities.

This HOWTO attempts to document how the whole thing fits together, how
to implement it on your own system, and how to use it in production at
a karaoke (or really any DJ'ed) show.

1. The Architecture.

The system we'll be building uses JACK to connect different sound
applications together and to the machine's sound card. PyKaraoke will
provide the CD+G audio source that we'll be applying pitch-shifting
effects to. XMMS will provide filler audio in this example, simply
because it has a JACK output driver and is easy to work with. It also
has a high-quality crossfader plugin that produces outstanding audio
results. JACK Rack, with Tom's Audio Processing LADSPA plugins
providing pitch-shifting. As an added bonus, I'll demonstrate how to
get MPlayer into the mix as well, to provide multimedia clips at shows
where a television or projector is available.

1.1 JACK

http://jackit.sourceforge.net/

The JACK Audio Connection Kit is a real-time virtual mixing board for
Linux (and possibly other platforms; I haven't yet investigated this)
that uses a low-latency server and protocol to connect sound
applications and devices to each other. It can control all sorts of
real hardware (sound cards, MIDI controllers, etc.) and can accept
inputs from just about anything (media players, MIDI controllers,
etc.) for both audio and control.

It can make use of a stock Linux kernel, or one that has been
configured with preempting and real-time features to ensure no pops or
gaps in audio playback even when many different tools are in active
use on the chain.

JACK includes a useful tool, qjackctl, that will be used in this HOWTO
to configure JACK and to automate patching our different tools
together.

1.2 JACK Rack

http://jack-rack.sourceforge.net/

JACK Rack is a sound processor designed for use with JACK exclusively;
it presents a configurable number of "default" channels to JACK and
can load effects modules (like "echo," "pitch shift," "reverb," etc.)
in any order. Sound flows through the modules top-to-bottom as they're
shown in the interface. Modules can be enabled or disabled (when
disabled, they do nothing to the sound; it simply flows through them
to the next module), and moved up and down in the stack.

This HOWTO uses JACK Rack for two things: first, it is used to perform
the actual pitch-shifting work when needed (audio from PyKaraoke is
passed to JACK Rack instead of straight ot the sound card). Second, it
performs crossfading work; I use different audio sources
(specifically, XMMS and MPlayer) during my shows, and smoothly
transitioning from them to CD+G music and back is important. JACK Rack
is, in this configuration, the only piece of software on the chain
that is connected to the sound card. All audio playback runs through
JACK Rack whether it is doing anything to the sound or not.

1.3 XMMS

http://www.xmms.org/

XMMS is a multimedia player for Linux. It is an older piece of
software (not actively maintained anymore; its developers are focusing
on a new 2.0 release), but assuming it'll start up and run on your
system, it's very reliable and useful for this purpose.

It supports MP3, Ogg Vorbis, WAV, FLAC, and all sorts of other audio
formats. It can play video (via MPlayer, actually), but we want manual
control over that kind of thing so we'll only be using XMMS for audio.

XMMS' plugin system means we can get automated crossfading between
songs XMMS plays *and* connect the whole thing to JACK as just another
audio source.

1.4 MPlayer

http://www.mplayerhq.hu/

MPlayer is a multimedia player for Linux. It supports audio and video
playback, tons of media formats, lots of output devices and types, and
has more features than I can adequately count here.

For our purposes, we'll simply be chaining it to JACK as an audio
source (letting it do its own thing for video, if needed) so we can
crossfade between it and the filler music XMMS provides.

1.5 Tom's Audio Processing LADSPA Plugins

http://tap-plugins.sourceforge.net/

This set of plugins, compatible with JACK Rack, includes the one we
want: pitch shifting. This is what will provide us with digital
pitch-shifting for CD+G playback. Really, it can affect all sound, but
we just want it to affect CD+G.

2. Setting Up the Parts

It's beyond the scope of this HOWTO to document the
installation/configuration procedure for the software used; if you're
using a modern Linux distribution (like Debian, Ubuntu, or Gentoo),
its package management system should already have these packages
available.

JACK is theoretically the "trickiest" part to get working, but it's
actually pretty forgiving. Once you can run qjackctl and play some
kind of sound through it, you're good to go.

You oddly enough don't need an MPlayer build with native JACK support;
you can "trick" it with ALSA and SDL, as that's what we're doing with
PyKaraoke as well. If your MPlayer has native JACK support, great; if
not, try running it with SDL audio output. If that works, you're still
golden.

XMMS needs the JACK output plugin, available at the XMMS homepage.

PyKaraoke requires no modifications for this process to work.

JACK Rack should "just work" once it's compiled and you run it.

2.1. Configuring ALSA to Talk to JACK

By default, ALSA just talks to your real sound device. We want it to
default to connecting to JACK instead, though, so it's necessary to
create a .asoundrc file. It belongs in the home directory of the
account you're using for production shows. Just edit ~/.asoundrc and
you should be fine. Its contents should be:

pcm.!default {
type plug
slave { pcm "jack" }
}

pcm.jack {
type jack
playback_ports {
0 alsa_pcm:playback_1
1 alsa_pcm:playback_2
}
capture_ports {
0 alsa_pcm:capture_1
1 alsa_pcm:capture_2
}
}

Test that this is working properly by starting JACK (run qjackctl and
start a JACK server), then watch the "Connections" bay as you run
"aplay somefile.mp3" -- you should see a new client appear, and it
will probably automatically connect to the sound card (this is JACK's
default behavior).

2.2. Setting Up the Sound Processor (JACK Rack)

Start up JACK Rack (the command is "jack-rack"). It will start with an
empty effects stack, which is what we want.

Add the first effect we want: the pitch-shifter. It's called "TAP
Pitch-Shifter".

Then, add the second effect we're after: the crossfader. It's called
"Crossfader."

Save the stack to a file you'll keep handy (you can just open this
stack later as a preset instead of having to set it up every time).
Disable both plugins for now; we want to get sound flowing first
before we start changing it.

2.3. Setting Up XMMS

Switch XMMS' output driver to the Crossfader plugin, then in the
Crossfader configuration, select the JACK output driver as the desired
output driver.

Start XMMS playing; you should immediately see it appear as a client
in the qjackctl Connections window. Again, it should automatically
connect to the sound card by default. This isn't what we want, by the
way, but it's how things start out.

2.4. Setting Up MPlayer and PyKaraoke

PyKaraoke needs no modifications because it uses SDL, which honors
.asoundrc. MPlayer needs to be told either to use JACK or SDL as its
audio output method:

# mplayer -ao sdl ...
-or-
# mplayer -ao jack ...

Fire off an MPlayer instance with a song to make sure it appears as a
client in the qjackctl Connections window.

If you're using MPlayer with SDL, you'll notice that PyKaraoke and
MPlayer appear pretty much the same within the Connections window.
That's an important detail later.

3. Putting It All Together

With all the pieces now talking to JACK, and JACK talking to the sound
card, it's time to build a Patchbay configuration to automate routing
of sound to the appropriate tools and devices. Here's how it will work
when it's all set up:

* XMMS will feed directly to the Crossfader plugin's connectors in JACK Rack.
* MPlayer and PyKaraoke will feed to JACK Rack's main connectors.
* JACK Rack will connect directly (and exclusively) to the sound card.

Open the Patchbay window in qjackctl. We'll start with the input
devices. Make sure JACK Rack is running, by the way. That makes life
much easier.

3.1. Adding Input Sockets

Select "Add" in the Input Sockets column to create a new input socket.
Name it "Sound Card." From the Client dropdown, select "alsa_pcm" (or
whatever name is assigned to your real sound device). "playback_1"
should appear in the Plug dropdown; click "Add Plug" to add it.
"playback_2" should appear there next; click "Add Plug" to add that,
too. Every Input Socket we're creating will handle stereo sound. We
need this socket to be exclusive, so check the "Exclusive" box. Hit
"OK" to add the socket.

Add the next Input Socket: name it "Processed Sound." This time,
select "jack_rack_[0-9]+" from the Client dropdown. Assuming you've
got the crossfade plugin installed in your stack in JACK Rack, if you
look at the Plug dropdown, you'll see *two* pairs of plugs, not one;
this is because the crossfader adds its own stereo channel to JACK
Rack's inputs. Add the "in_1" and "in_2" plugs to this socket, but
DON'T add the crossfader plugs. Hit "OK" to add the socket.

Add one last Input Socket: name it "Unprocessed Sound." Select
"jack_rack_[0-9]+" again from the Client dropdown, and *this* time,
add the crossfader plugs to the socket. Click "OK" to add the socket.

3.2. Adding Output Sockets

It's easier to set the output sockets up while the different media
players are running, because qjackctl automatically generates a
regular expression to match them for you when it sees them running.

JACK Rack is already running, so we'll set that output socket up first.

Click "Add" next to the Output Sockets column. Name it "Sound
Processor." Select "jack_rack_[0-9]+" from the Client dropdown, then
add both output plugs to the Socket. Save the socket with "OK."

Next, start up XMMS, start it playing (through JACK), and click "Add"
again to add a new Output Socket. Name it "Filler." There should be an
XMMS-related selection in the Client dropdown; select it, add its two
output plugs, and save the socket with "OK."

Finally, start a PyKaraoke instance (just have it play a CD+G song).
Click "Add" again to add the last Output Socket. Name it "Karaoke" (or
something similar). You should now see an "alsaP\.[0-9]+\.1" or
similar entry in the Client list; this is how SDL clients appear to
JACK. Select it, and add its two plugs.

3.3. Connect It All Together

Now comes the fun part -- teaching JACK to automatically connect
things to the right places. First, disconnect everything. Go to the
Connections panel in qjackctl and select Disconnect All. It will ask
for confirmation; give it.

The Patchbay lets us specify *automatic* connections; that is, when a
client connects, JACK will check against the active Patchbay profile
and if the client matches some part of the configuration, JACK will
automatically connect it where it needs to go.

In the Output Sockets column, click on the "Sound Processor" socket
you created. In the Input Sockets column, click the "Sound Device"
socket you created. Click "Connect" to connect them. Remember that we
set the sound device to be exclusive, so this will be the only
connection JACK will permit to the sound card. This prevents other
clients from automatically stealing the card directly.

Next, in Output Sockets, click "Filler," then in Input Sockets, click
on the "Unprocessed Sound" socket, and then click "Connect." This
connects XMMS directly to the crossfader in JACK Rack; remember that
sound flows top-down through the stack in JACK Rack, but this sound is
entering the rack *at* the crossfader. This means that no matter what
you do with the pitch-shifting plugin, your filler music won't be
pitch-shifted. In fact, any effect you apply to sounds before the
crossfade plugin won't affect the filler music. The crossfade module
can fade between this filler sound and the main inputs, where we'll be
feeding multimedia and CD+G output.

Finally, in Output Sockets, click "Karaoke," then in Input Sockets,
click "Processed Sound." Then click "Connect." This connects all
SDL-based clients to JACK Rack's main inputs, which get passed through
all the filters in the chain.

Save the Patchbay configuration, and then Activate it.

If all has gone well, you're done. Play a CD+G file with PyKaraoke and
watch the Connection panel -- it should appear and be automatically
connected. Start XMMS playing and it should be automatically connected
to the crossfader.

To play around with pitch-shifting, make sure to activate the module
in JACK Rack by clicking the "Enable" button. The crossfader needs to
be enabled, too, so do that as well. Play around with it -- you've got
a digital pitch-shifter and crossfader!

4. Conclusion

There are still ways to improve this; my next goal is to get my hands
on a MIDI controller that gives me a physical slider to use to control
the crossfader instead of using the one shown in JACK Rack's window.

It does work pretty well, though; I've used this now for three karaoke
shows (5 hours each) and have not had any performance difficulties
with it.

Feel free to contact me (at [email protected]) if you have questions about this.

I've posted this HOWTO at
http://willfe.com/index.php/Pitch-shifting_With_PyKaraoke_and_JACK_HOWTO
as well. I'll get it formatted properly here in a few minutes.

--
Looking for something to read? Visit http://willfe.com/ ... it's easy,
safe, and fun for the whole family!


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid0944&bid$1720&dat1642




Programming list archiving by: Enterprise Git Hosting