Synchronise plugin clocks (with JACK transport)

[riban-bw]

Is your feature request related to a problem? Please describe.
There are many elements within Zynthian that depend on timing and position, e.g. arpeggiators, sequencers, etc. It would be advantageous to be able to synchronise these elements so that they run at the same rate and start together. JAVL can use JACK transport for synchronisation including start, stop and position. @x42 has indicated this is the correct way to synchronise LV2 plugins.

Describe the solution you'd like
There should be a concept of master clock which may be derived internally or from external MIDI clock (or other sync mechanism). Zynthian master clock and transport should synchronise with JACK transport.

Describe alternatives you've considered
Until now we have tended to ignore JACK transport and just use MIDI clock.

Additional context
This article describes the concept of JACK transport. Zynthian currently uses JACK MIDI Clock for its internal clock which sends MIDI clock based on JACK clock. There should be a method of driving Zynthian (JACK?) clock from external clock or internal clock.

@jofemodo feel free to assign this back to me. (Clock and transport is currently a blocker for zynseq so I should sort this out before continuing with that.)

[x42]

JACK transport cannot be slaved to anything, so you'll need some internal clock that chases MIDI clock or another externally provided timebase.

Have a look at the MOD, they support Abelton-Link for that.

as for MIDI clock specifically, the following may come in handy for inspiration:
https://github.com/x42/jack_midi_clock/blob/master/jack_mclk_dump.c
https://github.com/Ardour/ardour/tree/master/libs/ardour/midi_clock_slave.cc

[riban-bw]

Thanks @x42. I have already implemented slaving to external clock in zynseq which seemed to work quite well so could be used. I will also look at the links provided. We need to set the clock and transport of JALV so if we can't slave JACK transport then we may need to break the link from JACK transport to JALV clocking so that we can drive it from our own master clock. I need to look how to do this but suspect it could be challenging if JALV is hard coded to use JACK transport (which doesn't seem unreasonable). This may turn into a bigger job to enhance / replace JALV (which is only really a test tool!!!).

[Edit] I think I understand better now. JACK transport can be stopped, starting or rolling. It provides position information based on frames. A timebase master may be implemented which adds more data, e.g. beat, bar, tick, etc. I think we need a module which:

• acts as timebase master
• can slave to MIDI timecode
• transforms MIDI transport control
• can be switched to use external MIDI / internal configured clock / timebase settings (external MIDI is derived using PLL)
• has an API which allows control and configuration
• allows creation of MIDI clock - probably as a dedicated JACK MIDI source

I think the use of jack_midi_clock is flawed. We don't really want to convert JACK to MIDI clock then decode MIDI clock, PLL and reimplement back into JACK time (like what zynseq currently does!!!). We should detail the timing model so that any new application use the same mechanism or we are able to adjust to suit this. We need to be aware of any other JACK clients that may wish to act as timebase master.

I will look at what is already available (doesn't look like anything does exactly what we need) and what I have already written in zynseq and come up with a solution that provides the required functionality. I will modify zynseq to work with this (which may see some performance improvement). This should give us a good foundation for clock and sync in Zynthian.

[jofemodo]

Yes, Jackd's transport framework is a nice way of having everything clock-synced, but it can't be synced (slaved) to an external source. Jackd MUST be the master clock. This is a hard-limitation for a device like zynthian.

If we could find a way of "syncing" jackd to an external clock, although it wasn't totally perfect... I found this:

http://jackctlmmc.sourceforge.net/

But it's quite old and i don't think it implements time sync, because it can't be done in a reliable way.

I think we have to take an important decision here:

• MIDI-clock way: Everything is synced by an unique MIDI clock signal, that can be internally generated or come from the outer world. Every clock consumer has to parse the MIDI-clock signal. This approach is redundant and inefficient, but it's flexible and would allow syncing to external sources.

• Jack Transport way: Every clock/transport consumer is bond to jackd transport, that acts as Master. Syncing to external clock sources is not possible. This approach is elegant and efficient, but impose a hard limit.

Or ... we could take a mixed approach, what means asumming some incoherence. This is what we currently have, more or less.
I mean:

• When internal midi clock is enabled, as it's bond to jackd transport, all consumers (midi clock & jackd transport) would work synced and Jackd is the absolute Master.

• When using an external clock (internal midi clock is disabled), we have 2 independent clocks. Consumers using midi-clock would be synced to the external source. Consumers using jackd's transport would be synced to internal jackd's clock. It could be consumers that implement both, allowing to choose. On this scenario, i think jackd transport should be forced to STOP status, being impossible to start rolling.

Anyway, we have to re-think how MIDI/Audio recorder/player currently works. Perhaps it should be detached from jackd transport.

Regards,

[jofemodo]

Regarding LV2 plugins, it seems logic to use this "LV2 host transport", what i suppose is specified by:

http://lv2plug.in/ns/ext/time

Jalv currently implements Host Transport synced to Jackd. We could implement MIDI-clock syncing on it, but it would be inefficient because a Jalv instance must executed for every plugin instance. Perhaps it's time to change (in fact, come back) to mod-host ...

Apart from LV2-plugins, currently we have several standalone engines:

• ZynAddSubFX
• FluidSynth
• LinuxSampler
• Aeolus
• setBfree
• PureData

Most of them, if not all, are irrelevant to this discussion.

Excuse i expose my ideas in a "brain-storming" way ... i would try to elaborate a more coherent scheme when i have all the pieces joined together. It's a quite complex playground what we have here!!

[riban-bw]

Brain dump is fine (and good).

Appreciate that JACK clock is only the master clock and needs to be. On top of this we can add a timebase master which effectively locks to JACK clock but provides higher level clocking info, e.g. JACK clocks 44100 frames whilst the timebase master clocks 2 beats or 48 ticks or whatever, based on its time signature and tempo. JACK had no concept of tempo. It is the timebase master which provides this information. The timebase master is a JACK client. So we have everything locked to JACK clock and transport even if we also effectively appear to lock to the timebase master's data. A MIDI clock output can be generated. (That is what hack_midi_clock does) which can be sent to anything that needs it, e.g. external MIDI devices. LV2 plugins get their timing from JALV which uses the data set by timebase master but gets it from jackd. Everything is happy... almost.

We do need to think about what modules so with timecode and transport control. We may not want the recorder to start and stop when the sequencer does and plugins like arpeggiators may want a continuous clock signal (JACK rolling).

[falkTX]

I find some information on this ticket misleading.

You can have JACK transport being slave to something else, IFF the transport master is the only one doing that.
This is how mod-host is able to use jack transport but still sync with MIDI Clock (input).

EDIT: there is jitter for midi clock yes. with midi2 supporting timestamps, hopefully the jitter will be reduced.

[x42]

JACK transport progresses at the speed of the soundcard. The sample-clock is linked to the soundcard's oscillator. and a timebase master is supposed to translate the sample position of jack-transport to music time. They're intrinsically linked.

For MIDI clock you'll have to do the opposite. Convert the incoming music time (beat) into an absolute position, and then use the clock to use the incoming clock-tick to move the transport (independently of the devices osc).

You could try to work around this, by assuming a fixed speed, but varying tempo, but otherwise it's not possible to use JACK transport for this, since JACK always moves at speed 1.0. You need a host internal position and speed representation.
JACK transport is also limited that it cannot directly start but has to cater for slow-sync clients, and state-changes only happen at cycle boundaries.

BTW, a good introduction to MIDI Clock can be found at http://midi.teragonaudio.com/tech/midispec/seq.htm

[falkTX]

MIDI runs at a fixed speed though.
So it is possible to measure the time for each tick and get a position, you say that yourself.

But yes there are limitations, the slow-sync clients are an issue.
On MOD (or any platform where you have control over all clients that are running) you can bypass this by setting the time-for-clients-to-start-transport to 0 (I forgot the exact option/setting name for this).

The biggest issue we found so far for MOD is the jitter and low-resolution of MIDI.
Some filtering can help, but still far from ideal.
On MOD's case, there is going to be something added in v1.11 so that plugins can receive the raw MIDI clock-related messages. (probably a custom LV2 Atom event-sequence type, but this is not defined yet)

[x42]

So it is possible to measure the time for each tick and get a position, you say that yourself.

Yes but this position usually does not increment at the same rate as the audio-clock, and the problem is that jack-transport cannot vari-speed.

The biggest issue we found so far for MOD is the jitter and low-resolution of MIDI.

That is easily solved: http://kokkinizita.linuxaudio.org/papers/usingdll.pdf 2nd order filter can reduce jitter by several orders of magnitude to the point where it's negligible WRT to audio-clock. (That's another reason why this should be done centrally by the host and not by individual plugins).

[jofemodo]

Thanks a lot for putting some light on this subject. There is little documentation about it and only a few braves who really understand the problem ;-)

[riban-bw]

We currently use the jack_transport as our timebase master. This is example code from jack2. I suggest we incorporate the relevant functionality into an existing Zynthian library such as zynstep then enhance it as necessary. The reason I suggest adding functionality to existing library (which conflicts with the *nix way and @jofemodo's preference of one function per module) is because each JACK client adds overhead due to context switching. The fewer JACK clients the better and if we have core functionality which is needed by Zynthian it makes sense to combine that into a core library.

As discussed above, it is possible to slave to MIDI clock by using the timebase master to set BBT parameters. There will be a degree of jitter which can be filtered as described above and/or using techniques already implemented in zynseq. There may be some drift between audio and MIDI (though I would aim to minimise this) but this may be less significant for us as we are not a DAW (and do not intend to be one). There may be some offset between audio and MIDI but I will minimise where possible. zynseq already implements a tempo map which does not work due to limitations in jack_transport. I can make this work properly.

I won't implement MIDI clock generation (yet) because we already have that available in jack_midi_clock and LV2 MIDI Clock Generator so we have ways to add that already. We may decide to add to this library later for same overhead reasons previously mentioned. jack_midi_clock will not be required for zynseq to work. zynseq will lock to JACK time, using this timebase master I intend to write.

[falkTX]

The reason I suggest adding functionality to existing library ... is because each JACK client adds overhead due to context switching.

This is only the case for external clients. If we run only internal clients, there is no context switches whatsoever. You can then on top of this slightly modify the jack2 linux futex code so that futexes for internal clients never reach the kernel. The MOD software stack (mod-host, peakmeter, ttymidi, spi2jack) all run as internal clients because of this.

[riban-bw]

I am refactoring zynseq to use Jack master timebase which will lose its ability to slave to MIDI clock. Once this is done I can look at building a Jack timebase master with feature to lock to MIDI clock using the PLL code I have just removed from zynseq. This is the right way to do this but may take a while to complete. The zynseq refactoring is almost done.

[jofemodo]

Hi @riban-bw !
I'm impatient for testing the new version of the sequencer... ;-)

[riban-bw]

Me too! Each time I refactor a bit I realise that the next bit is also required. Sequencer now uses Jack transport and clock but the example code jack-transport isn't sufficiently feature rich so I have written a new transport control with time signature and tempo change map. This revealed a need to change user interface to simplify workflow which in turn revealed more architectural changes. I need to iron out some bugs but am making progress. I have been able to work on this a lot over the past few weeks so progress is steady. Watch this space...

[jofemodo]

Ups! These are great news. Tell me if i can help you, or if you want to discuss some "metaphysical" detail ... ;-)

[riban-bw]

zynseq is now implemented as JACK timebase master handling tempo and time signature. The feature to set various tempo / time signature changes within a song are there but not yet used / fully tested. They will be used when I enable the linear song feature.

With zynseq now locked to JACK clock we should have the core of this feature requested implemented. Zynthian modules that lock to JACK clock should now run synchronously and those that use or control the transport should also work together. I have not yet reimplemented the slave to MIDI clock option but that probably should be in a separate feature request. I have also not provided MIDI clock output but that can be provided by another module LV2 MIDI Clock Generator.

With all this good news I will close the ticket and leave further development and enhancement to developer (me) or other user feature requests.