Waveform

Introduction

Of the things that WPF misses it is probably of audio-related controls. I’ve had to craft some of them and I will present you a component that renders a wave form and a little more. It features a sample-level accuracy, a theme-able interface and ‘providers’ which analyzes audio and return information of interest such as sound features.

Showcase

waveform-onsets-128
Figure 1 : provider that detects onsets and color them according their frequency band

waveform-tempo-128
Figure 2 : provider that detects beats

waveform-coloring-128
Figure 3 : provider that colors audio content like Scratch Live or rekordbox

waveform-echonest-128
Figure 4 : provider that renders feature vectors returned by an EchoNest online analysis

waveform-direct3d-128
Figure 5 :  same provider but with a Direct3D renderer and a custom shader

How it works

The rendering process components are laid out below:

AudioStream -> Waveform -> WaveformRenderer

– AudioStream reads and converts audio samples
– Waveform builds the peak data and cache
– WaveformRenderer is an abstract renderer

Finally, providers are plugged to a renderer as needed.

Status

The version in GitHub is old, I yet have to upload the latest version featuring providers.

SharpMix

sharpmix-transparent-600

Table of contents

Introduction
Features
Description
Gallery
Project status
Future
Conclusion

 

Introduction

This is an overview of the longest and most complex project I have been working on so far, a digital mixing solution for DJs written using the C# language. I will go through some aspects of the software during the development phase, the suspension of the project and the conclusions I drew from this experience.

Features

  • mixing of digital music files
  • support of turntables using a time-code
  • support of MIDI control surfaces
  • analysis and extraction of audio features
  • management of a collection of digital music files

Description

The software allows the user to mix digital files with a ‘traditional’ approach by using turntables. The system works by using a ‘time-code’ CD which is a special kind of Audio CD that contains a sequence of numeric codes along an error-correcting code for its robustness. Its role is to synchronize the playback of digital music according the user interaction with a turntable. In addition to the support of turntables, it supports the usage of MIDI-enabled DJ control surfaces for driving the performance, as well as navigating in the user interface.

Another major feature is the analysis and extraction of audio features from digital music files such as the tempo, the key chord and audio segments. These audio properties are useful to a DJ for organizing a collection of digital music files. By tagging music with keywords pertaining to the field of music composition, it allows for more creativity and the usage of the technique known as harmonic mixing.

Here you can see a video and a few screenshots of some parts and components that have been developed over time. Some of them are considered to be in a mature state, others are still in a preliminary state.

Figure 1 : a video showing the responsiveness of the system while tracking the time-code played by a CDJ-1000


Figure 2 : last version of the software using the WPF framework


Figure 3 : the presentation page of an artist, content is fetched from EchoNest


Figure 4 : an implementation of a high-quality colored bitmap font


Figure 5 : a preliminary implementation of a tag cloud


Figure 6 : the output of the time-code tracking smoothing system against burst errors


Figure 7 : first version of the software that is seen on the video

Project status

After 18 months of development I have decided to put the development of the project on hold, mainly by the lack of sufficient resources to react to market changes in a realistic time frame. If you look at the credits of the Traktor software you would see that nearly 70 people have participated in its development; obviously I can hardly compete with such workforce given the resources I have invested in and the time frame I have envisioned initially.

Between its inception and suspension there has been a lot of novelties in the field of technologies and computing. Of them, the appearance of touch-enabled devices, the advent of DJ control surfaces providing a MIDI time-code and finally, newer patterns in the software development world.

And since I was seeking for an A+ grade software, evidently I could not ignore and not adopt these newer techs in the project. The project however, is not being abandoned; it is just in a suspended state. I am currently elaborating and reviewing a strategy for the software to resurface later, with a different model, though.

Future

The software is probably going to resurface within the next year, in a slimmed-down, refined and more modern version. You can expect it to be touch-enabled, a fully-featured offline mode (previously it had to rely to online services for the analysis of music), will be free and probably provide cloud-enabled features.

Conclusion

While I will easily admit that from a commercial point of view the software has been a failure, it was however a positive experience for me where I have learned many things in regard to programming, signal processing and the design of interfaces. Even if its development time stretched to the point of it getting suspended, I still consider it as an achievement by some ways. I’ll conclude by saying that the fact this project will resurface in the future mitigates somewhat that point.

Aubio

aubio-tile-256

aubio is a C++ library for the detection of audio features such as pitch, beats, tempo and onsets. I am still working on making it available from .NET and will publish the sources along a NuGet package when it’s done.

LibKeyFinder

libkeyfinder-tile-256

LibKeyFinder is a musical key chord detection library with high accuracy. Its performance is on par with top commercial solutions available such as MixedInKey or Tonart.

I contributed the following additions to the project:

– extended the initial C++ library to support any programming language
– a first .NET assembly for using the library from C#
– a second .NET assembly that simplifies the usage of it
– NuGet packages: LibKeyFinderDotNet and LibKeyFinderDotNet.BASS

I also plan to make features in KeyFinder application available from these assemblies.

SonicApi

sonicAPI-tile-256

SonicApi is a web service that extracts audio features such as pitch, tempo and key chord. The service also provides processing services such as correction of pitch, tempo and the addition of reverberation effects. It is the web service version of zplane high quality components used by many products/companies in the field such as abletonSteinberg. Korg or Native Instruments.

I have written a library for accessing the online services from the .NET platform which uses the asynchronous version of the services. I’ve notified them about it and they decided to make it part of the official API, currently I am waiting for their review and instructions as they would like it to match the Objective C API they have developed internally.

Downloads

library source code
sample project
NuGet package

ENMFPdotNet

echonest-tile

The EchoNest is a company that provides services for the analysis and extraction of musical features, some of their clients are MTV or the BBC; it has recently been acquired by Spotify.

I developed a .NET version of the API, currently I am reviewing it and documenting it before publishing it. There is a component of the API that I have already published, ENMFPdotNet. It is the component that fingerprints audio songs for querying their identification service. A NuGet package is also available here.

Since it is not the original utility you will have to roll out your own decoding process to 32-bit 22Khz monophonic PCM as this is the format that Codegen expects. I have put a small example on how to achieve that using BASS.NET, basically you submit a file name, specify the desired sample rate, the number of channels desired and you will get your audio data converted to that format.

Also, there is a workaround explained in the README on how to use the library in a Windows Store application.

If you prefer to use the original command-line utility (codegen), it is available here.

Codegen

codegen-tile

Codegen is the command-line utility that generates the fingerprint of an audio file. I am providing below a binary of the utility that supports ASF, MP4 formats and the reading of tags.

codegen-binaries.zip

Update

I have made Codegen available from .NET through the ENMFPdotNet (or at GitHub directly) library, it does not requires the usage of the command-line utility anymore as it has been statically linked.

Note that you will have to decode the audio and resample by yourself since you must pass a 22KHz monophonic array of floats to the library. I have a put a code sample in the README that does exactly that with BASS.NET, if you need to support other formats just grab the appropriate addons for it.