Fix synchronization
Synchronization in USDX is still very buggy. Without synchronization as in USDX 1.0.1 audio, video and lyrics will be out of synch half of the song. Especially audio and video were not very synchronous with 1.0.1 on my windows system. This was due to the fact that audio is played by bass and video by ffmpeg both using different timers.
For USDX 1.1 I tried to solve this issue by synching lyrics, audio and video. This is mostly based on the approach on an ffmpeg tutorial:
http://dranger.com/ffmpeg/
IIRC I used the lyrics time (which is generated by a timer) as a master clock. The audio clock is synched against this master clock and video is synchronized against the audio clock.
There are some problems with this solution:
There are two solutions for this problem:
1. disable synchronisation (and audio/video will be out of synch again) [not recommended but a quick fix]
2. synch lyrics and video to audio:
__* make audio the master clock
__* the lyrics clock was uses as a master in the current synch approach as I didn't know if the scoring algorithm poses problems if the lyrics time changes (and I still don't know)
__* we have to make sure that audio timestamps have a good quality:
_____* they have to be monotone (a timestamp following another is not allowed to have a lesser value than the previous one)
_____* the timestamps do not have to jitter so the scoring and the lyric display are appropriately. This means a timestamp contains the correct audio postion with only a little tolerated difference.
_____* NOTE: FFmpeg produces very bad timestamps for Ogg-Vorbis files, for example 'Dead Smiling Pirates - I 18' (already the first timestamp is wrong), MP3 timestamps are better in general. So always test synch approaches with ogg-vorbis files.
__* NOTE: The ffmpeg demo player which is part of the ffmpeg distribution disables synching if the timestamp quality is too bad. This could be done in USDX too.
3. in my opinion the synch against audio will always cause crackling so my first approach should be abandoned (sorry). This could only be done if we could dynamically increase/decrease the frequency of the audio stream to make it slightly faster or slower. But changing the resampling frequency dynamically is not possible with ffmpeg or SDL (those are currently used), it is only possible with libsamplerate (it was not used as it was too slow).
For USDX 1.1 I tried to solve this issue by synching lyrics, audio and video. This is mostly based on the approach on an ffmpeg tutorial:
http://dranger.com/ffmpeg/
IIRC I used the lyrics time (which is generated by a timer) as a master clock. The audio clock is synched against this master clock and video is synchronized against the audio clock.
There are some problems with this solution:
- this solution works good if only bass is used (without ffmpeg)
- this sometimes causes the audio to crackle as some bytes of the audio stream have to be skipped in order to keep up with the lyric timer
- the ffmpeg audio and video timestamps do not seem to be of high quality. IIRC there even were some audio timestamps with a time earlier than the previous timestamp
- on my linux system the soundcard consumes much more bytes than it should. For example in 44.1kHz mode it consumes 44150 samples instead of 44100 and so the audio stream will be played too fast and it has to be stopped for some time to be synched with the lyrics timer. This is a very nasty and noticeable problem on my system as it causes crackling etc.
There are two solutions for this problem:
1. disable synchronisation (and audio/video will be out of synch again) [not recommended but a quick fix]
2. synch lyrics and video to audio:
__* make audio the master clock
__* the lyrics clock was uses as a master in the current synch approach as I didn't know if the scoring algorithm poses problems if the lyrics time changes (and I still don't know)
__* we have to make sure that audio timestamps have a good quality:
_____* they have to be monotone (a timestamp following another is not allowed to have a lesser value than the previous one)
_____* the timestamps do not have to jitter so the scoring and the lyric display are appropriately. This means a timestamp contains the correct audio postion with only a little tolerated difference.
_____* NOTE: FFmpeg produces very bad timestamps for Ogg-Vorbis files, for example 'Dead Smiling Pirates - I 18' (already the first timestamp is wrong), MP3 timestamps are better in general. So always test synch approaches with ogg-vorbis files.
__* NOTE: The ffmpeg demo player which is part of the ffmpeg distribution disables synching if the timestamp quality is too bad. This could be done in USDX too.
3. in my opinion the synch against audio will always cause crackling so my first approach should be abandoned (sorry). This could only be done if we could dynamically increase/decrease the frequency of the audio stream to make it slightly faster or slower. But changing the resampling frequency dynamically is not possible with ffmpeg or SDL (those are currently used), it is only possible with libsamplerate (it was not used as it was too slow).
Leave a comment
Video is not synched against audio but to the lyrics timer (as is audio):
The synchronisation is done (in UScreenSing.pas) by the GetFrame() method:
fCurrentVideoPlaybackEngine.GetFrame(CurrentSong.VideoGAP + LyricsState.GetCurrentTime());
The time shown in the timeline in the sing-screen is the lyrics timer too.
In some later release of USDX it would be nicer if a video was an object and had its own timer which is synchronized via a setSynchSource() method.
The synchronisation is done (in UScreenSing.pas) by the GetFrame() method:
fCurrentVideoPlaybackEngine.GetFrame(CurrentSong.VideoGAP + LyricsState.GetCurrentTime());
The time shown in the timeline in the sing-screen is the lyrics timer too.
In some later release of USDX it would be nicer if a video was an object and had its own timer which is synchronized via a setSynchSource() method.
The synchronization can be changed by setting the SyncTo option in config.ini to one of Music/Lyrics/Off.
- "Music": lyrics are synchronized to the audio file (new default)
- "Lyrics": synchronizes audio to lyrics (old default)
- "Off": switches lyrics <-> music sync off, but video is still synchronized to the lyrics (beat-timer)
With synchronizing lyrics to music everything should work fine.
There is only one problem: If the music file returns a timestamp that is too high and a lower timestamp afterwards the lyric position will not be changed as setting the beat-timer to a position in the past will probably totally mess up the scores. So the beat-timer has to wait then. This is rather an FFmpeg problem. If this happens in some rare cases the user can still disable synchronization.
- "Music": lyrics are synchronized to the audio file (new default)
- "Lyrics": synchronizes audio to lyrics (old default)
- "Off": switches lyrics <-> music sync off, but video is still synchronized to the lyrics (beat-timer)
With synchronizing lyrics to music everything should work fine.
There is only one problem: If the music file returns a timestamp that is too high and a lower timestamp afterwards the lyric position will not be changed as setting the beat-timer to a position in the past will probably totally mess up the scores. So the beat-timer has to wait then. This is rather an FFmpeg problem. If this happens in some rare cases the user can still disable synchronization.