sync_audio_audio_full.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Demo: Audio-audio synchronization with high-resolution features and MrMsDTW\n",
    "\n",
    "In this notebook, we'll show a full music synchronization pipeline using the SyncToolbox, including feature extraction and high-resolution synchronization. For a short example focussing on the basics only, see [`sync_audio_audio_simple.ipynb`](sync_audio_audio_simple.ipynb).\n",
    "\n",
    "We will take two recordings of the same musical piece (the third song of Franz Schubert's \"Winterreise\"), compute feature representations of both recordings, align them using multi-resolution multi-scale DTW (MrMsDTW), and show how to sonify the alignment and use it for automated transfer of annotations.\n",
    "\n",
    "The pipeline in this notebook exactly reproduces the techniques described in [1], which in turn is based on [2]. On the finest synchronization, we use the high-resolution features described in [3]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Loading some modules and defining some constants used later\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import librosa.display\n",
    "import matplotlib.pyplot as plt\n",
    "import IPython.display as ipd\n",
    "import scipy.interpolate\n",
    "from libfmp.b.b_plot import plot_signal, plot_chromagram\n",
    "from libfmp.c3.c3s2_dtw_plot import plot_matrix_with_points\n",
    "\n",
    "from synctoolbox.dtw.mrmsdtw import sync_via_mrmsdtw\n",
    "from synctoolbox.dtw.utils import compute_optimal_chroma_shift, shift_chroma_vectors, make_path_strictly_monotonic, evaluate_synchronized_positions\n",
    "from synctoolbox.feature.chroma import pitch_to_chroma, quantize_chroma, quantized_chroma_to_CENS\n",
    "from synctoolbox.feature.dlnco import pitch_onset_features_to_DLNCO\n",
    "from synctoolbox.feature.pitch import audio_to_pitch_features\n",
    "from synctoolbox.feature.pitch_onset import audio_to_pitch_onset_features\n",
    "from synctoolbox.feature.utils import estimate_tuning\n",
    "%matplotlib inline\n",
    "\n",
    "Fs = 22050\n",
    "feature_rate = 50\n",
    "step_weights = np.array([1.5, 1.5, 2.0])\n",
    "threshold_rec = 10 ** 6\n",
    "\n",
    "figsize = (9, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading two recordings of the same piece\n",
    "\n",
    "Here, we take recordings of the song \"Gefrorne Tränen\" by Franz Schubert from his song cycle \"Winterreise\" in two performances (versions). The first version is by Gerhard Hüsch and Hanns-Udo Müller from 1933. The second version is by Randall Scarlata and Jeremy Denk from 2006. In particular, the two versions are played in different keys: The second version is played one semitone higher than the first version. We will address this later.\n",
    "\n",
    "### Version 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "audio_1, _ = librosa.load('data_music/Schubert_D911-03_HU33.wav', sr=Fs)\n",
    "\n",
    "plot_signal(audio_1, Fs=Fs, ylabel='Amplitude', title='Version 1', figsize=figsize)\n",
    "ipd.display(ipd.Audio(audio_1, rate=Fs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Version 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "audio_2, _ = librosa.load('data_music/Schubert_D911-03_SC06.wav', sr=Fs)\n",
    "\n",
    "plot_signal(audio_2, Fs=Fs, ylabel='Amplitude', title='Version 2', figsize=figsize)\n",
    "ipd.display(ipd.Audio(audio_2, rate=Fs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimating tuning\n",
    "\n",
    "We use a simple comb-based algorithm to detect tuning deviations in the two audio recordings. These will be used to adjust the filterbanks for feature computation. If we do not adjust for tuning, our chroma representations may look \"smeared\", leading to bad synchronization results. We refer to <a href=\"https://www.audiolabs-erlangen.de/resources/MIR/FMP/C3/C3S1_TranspositionTuning.html\">the FMP notebook on Transposition and Tuning</a> for more information on tuning issues and the algorithm used for tuning estimation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import libfmp.c2\n",
    "# Alternative: librosa.estimate_tuning\n",
    "tuning_offset_1 = estimate_tuning(audio_1, Fs)\n",
    "tuning_offset_2 = estimate_tuning(audio_2, Fs)\n",
    "print('Estimated tuning deviation for recording 1: %d cents, for recording 2: %d cents' % (tuning_offset_1, tuning_offset_2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing quantized chroma and DLNCO features\n",
    "\n",
    "We now compute the feature representations used in the alignment procedure. Note that we include the 'tuning_offset' calculated in the previous step. In our pipeline, we use CENS features, which are similar to standard chroma but first quantized, then smoothed, downsampled and normalized. The MrMsDTW procedure just requires the quantized chromas, since smoothing, downsampling and normalization happens internally.\n",
    "\n",
    "In addition to these chroma-like features, we also use special onset-related features called DLNCO (described in [3]). These are helpful to increase synchronization accuracy, especially for music with clear onsets.\n",
    "\n",
    "Both features are computed from the audio using a multi-rate IIR filterbank. See [4] for details.\n",
    "\n",
    "In the next cell, we also display the computation steps leading to both features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def get_features_from_audio(audio, tuning_offset, visualize=True):\n",
    "    f_pitch = audio_to_pitch_features(f_audio=audio, Fs=Fs, tuning_offset=tuning_offset, feature_rate=feature_rate, verbose=visualize)\n",
    "    f_chroma = pitch_to_chroma(f_pitch=f_pitch)\n",
    "    f_chroma_quantized = quantize_chroma(f_chroma=f_chroma)\n",
    "\n",
    "    f_pitch_onset = audio_to_pitch_onset_features(f_audio=audio, Fs=Fs, tuning_offset=tuning_offset, verbose=visualize)\n",
    "    f_DLNCO = pitch_onset_features_to_DLNCO(f_peaks=f_pitch_onset, feature_rate=feature_rate, feature_sequence_length=f_chroma_quantized.shape[1], visualize=visualize)\n",
    "    return f_chroma_quantized, f_DLNCO\n",
    "\n",
    "\n",
    "f_chroma_quantized_1, f_DLNCO_1 = get_features_from_audio(audio_1, tuning_offset_1)\n",
    "f_chroma_quantized_2, f_DLNCO_2 = get_features_from_audio(audio_2, tuning_offset_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next plots illustrate the different representations of the first 30 seconds of each version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "plot_chromagram(f_chroma_quantized_1[:, :30 * feature_rate], Fs=feature_rate, title='Chroma representation for version 1', figsize=figsize)\n",
    "plt.show()\n",
    "plot_chromagram(f_DLNCO_1[:, :30 * feature_rate], Fs=feature_rate, title='DLNCO representation for version 1', figsize=figsize)\n",
    "plt.show()\n",
    "\n",
    "plot_chromagram(f_chroma_quantized_2[:, :30 * feature_rate], Fs=feature_rate, title='Chroma representation for version 2', figsize=figsize)\n",
    "plt.show()\n",
    "plot_chromagram(f_DLNCO_2[:, :30 * feature_rate], Fs=feature_rate, title='DLNCO representation for version 2', figsize=figsize)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding optimal shift of chroma vectors\n",
    "\n",
    "As mentioned above, the two versions of the same piece used in this notebook are played in different keys. This can also be seen in the chroma representations above and will lead to complete degradation of the alignment if this effect is not accounted for. The SyncToolbox provides a built-in function for finding the shift between two recordings. This is done in the following cell and the feature sequences are subsequently adjusted to account for this shift. The plots show the chroma sequences after shifting.\n",
    "\n",
    "Internally, the function just performs DTW using all possible shifts and returns the shift yielding the lowest total cost. To save computation time, we here first downsample the sequences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "f_cens_1hz_1 = quantized_chroma_to_CENS(f_chroma_quantized_1, 201, 50, feature_rate)[0]\n",
    "f_cens_1hz_2 = quantized_chroma_to_CENS(f_chroma_quantized_2, 201, 50, feature_rate)[0]\n",
    "opt_chroma_shift = compute_optimal_chroma_shift(f_cens_1hz_1, f_cens_1hz_2)\n",
    "print('Pitch shift between recording 1 and recording 2, determined by DTW:', opt_chroma_shift, 'bins')\n",
    "\n",
    "f_chroma_quantized_2 = shift_chroma_vectors(f_chroma_quantized_2, opt_chroma_shift)\n",
    "f_DLNCO_2 = shift_chroma_vectors(f_DLNCO_2, opt_chroma_shift)\n",
    "\n",
    "plot_chromagram(f_chroma_quantized_1[:, :30 * feature_rate], Fs=feature_rate, title='Version 1', figsize=figsize)\n",
    "plt.show()\n",
    "plot_chromagram(f_chroma_quantized_2[:, :30 * feature_rate], Fs=feature_rate, title='Version 2, shifted to match version 1', figsize=figsize)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performing MrMsDTW\n",
    "\n",
    "We now perform alignment using MrMsDTW. The extracted chroma sequences are used on the coarser levels of the procedure, while the DLNCO features are additionally used on the finest level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "wp = sync_via_mrmsdtw(f_chroma1=f_chroma_quantized_1, f_onset1=f_DLNCO_1, f_chroma2=f_chroma_quantized_2, f_onset2=f_DLNCO_2, input_feature_rate=feature_rate, step_weights=step_weights, threshold_rec=threshold_rec, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## For applications: Make warping path strictly monotonic\n",
    "The standard step sizes used in DTW allow for horizontal and vertical steps, which leads to warping paths that are not guaranteed to be strictly monotonous. This is usually not a problem. However, for applications such as transferring annotations, it may be better to use a strictly monotonous path and employ linear interpolation inside non-monotonous segments. See also <a href=\"https://www.audiolabs-erlangen.de/resources/MIR/FMP/C3/C3S3_MusicAppTempoCurve.html\">the FMP notebook on Tempo Curves</a> for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Length of warping path obtained from MrMsDTW:', wp.shape[1])\n",
    "wp = make_path_strictly_monotonic(wp)\n",
    "print('Length of warping path made strictly monotonic:', wp.shape[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Application 1: Sonifying warping path\n",
    "\n",
    "In order to listen to the synchronization result, version 1 will now be time-scaled (according to the computed warping path) to run synchronous to version 2. Additionally, we pitch-shift version 1 to account for the key difference mentioned earlier.\n",
    "\n",
    "For the time-scale modification, we use the libtsm [5] library. The result is sonified by putting the warped version 1 into the left channel and version 2 into the right channel of a stereo audio file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install libtsm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import libtsm\n",
    "\n",
    "pitch_shift_for_audio_1 = -opt_chroma_shift % 12\n",
    "if pitch_shift_for_audio_1 > 6:\n",
    "    pitch_shift_for_audio_1 -= 12\n",
    "audio_1_shifted = libtsm.pitch_shift(audio_1, pitch_shift_for_audio_1 * 100, order=\"tsm-res\")  \n",
    "\n",
    "# The TSM functionality of the libtsm library expects the warping path to be given in audio samples.\n",
    "# Here, we do the conversion and additionally clip values that are too large.\n",
    "time_map = wp.T / feature_rate * Fs\n",
    "time_map[time_map[:, 0] > len(audio_1), 0] = len(audio_1) - 1 \n",
    "time_map[time_map[:, 1] > len(audio_2), 1] = len(audio_2) - 1\n",
    "\n",
    "y_hpstsm = libtsm.hps_tsm(audio_1_shifted, time_map)\n",
    "stereo_sonification = np.hstack((audio_2.reshape(-1, 1), y_hpstsm))\n",
    "\n",
    "print('Original signal 1', flush=True)\n",
    "ipd.display(ipd.Audio(audio_1, rate=Fs, normalize=True))\n",
    "\n",
    "print('Original signal 2', flush=True)\n",
    "ipd.display(ipd.Audio(audio_2, rate=Fs, normalize=True))\n",
    "\n",
    "print('Synchronized versions', flush=True)\n",
    "ipd.display(ipd.Audio(stereo_sonification.T, rate=Fs, normalize=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Application 2: Transferring measure annotations\n",
    "\n",
    "The warping path obtained using MrMsDTW may also be used to facilitate other music information retrieval tasks. For example, one often has annotations (about keys, chords, instruments, ...) for a certain version of a piece and may wish to transfer these to another version of the same piece. In the following, we use the computed warping path to transfer measure positions annotated in version 1 over to version 2.\n",
    "\n",
    "In our case, we have hand-made measure annotations for version 2, as well. This allows us to evaluate the quality of the synchronization. As evaluation measures, we look at the mean average error and the percentage of correctly transferred measures (given a threshold)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "measure_annotations_1 = pd.read_csv(filepath_or_buffer='data_csv/Schubert_D911-03_HU33.csv', delimiter=';')['start']\n",
    "measure_positions_1_transferred_to_2 = scipy.interpolate.interp1d(wp[0] / feature_rate, wp[1] / feature_rate, kind='linear')(measure_annotations_1)\n",
    "measure_annotations_2 = pd.read_csv(filepath_or_buffer='data_csv/Schubert_D911-03_SC06.csv', delimiter=';')['start']\n",
    "\n",
    "mean_absolute_error, accuracy_at_tolerances = evaluate_synchronized_positions(measure_annotations_2 * 1000, measure_positions_1_transferred_to_2 * 1000)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "[1] Thomas Prätzlich, Jonathan Driedger, and Meinard Müller: Memory-Restricted Multiscale Dynamic Time Warping,\n",
    "In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP): 569–573, 2016.\n",
    "\n",
    "[2] Meinard Müller, Henning Mattes, and Frank Kurth:\n",
    "An Efficient Multiscale Approach to Audio Synchronization,\n",
    "In Proceedings of the International Conference on Music Information Retrieval (ISMIR): 192–197, 2006.\n",
    "\n",
    "[3] Sebastian Ewert, Meinard Müller, and Peter Grosche:\n",
    "High Resolution Audio Synchronization Using Chroma Onset Features,\n",
    "In Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP): 1869–1872, 2009.\n",
    "\n",
    "[4] Meinard Müller: Information Retrieval for Music and Motion, ISBN: 978-3-540-74047-6, Springer, 2007.\n",
    "\n",
    "[5] Sebastian Rosenzweig, Simon Schwär, Jonathan Driedger, and Meinard Müller: Adaptive Pitch-Shifting with Applications to Intonation Adjustment in A Cappella Recordings Proceedings of the International Conference on Digital Audio Effects (DAFx), 2021."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}