DDWFollowClock
Extension
ExtensionDescription
With DDWLeadClock, a minimalistic lead-follower approach to synchronizing multiple SuperCollider language instances over a local network. (Technical details below: DDWFollowClock: How it works.)
The local network should allow broadcast messages (IP address "255.255.255.255"). If your system has a firewall, make sure SuperCollider's ports (57120, 57110) are open for UDP traffic.
Both DDWFollowClock and DDWLeadClock inherit from TempoClock. After creating the clocks, their operation is identical to that of TempoClock. Basic TempoClock features are not documented here; consult the TempoClock help file for details.
Usage
Normal usage is:
- One lead clock, providing a global tempo for the entire network.
- Any number of follower clocks, on other machines.
For this use case, the procedure is: 1. Create the lead clock. 2. On the other machines, create one follower clock instance per machine. The followers should automatically detect sync messages coming from the lead, and synchronize beats and tempo within a second or two.
It is also possible to run multiple lead clocks, at different tempi. Each lead clock should have a different ID. (If you don't provide one, an ID will be chosen from UniqueID, but if you need multiple tempi, it is highly recommended to specify an ID as a number or Symbol.) Then, follower clocks can be created for specific IDs. (Note: This scenario is not extensively tested yet.)
Initialization
It is recommended, for the first few seconds after creating a DDWFollowClock, to avoid heavy synchronous initialization that will block the interpreter (such as loading long code blocks from other files, preparing large collections, etc.).
DDWFollowClock tries to measure the difference between the local system time and the lead's system time. This assumes the interpreter can respond instantaneously to incoming OSC messages. If a sync message arrives while the language is processing e.g. a million-item array, the local response time will be delayed and the estimate of the time difference will be incorrect.
You should allow DDWFollowClock to run without other heavy activities until it has stabilized. Either do the heavy lifting before creating the follower clock, or wait for a \ddwFollowClockSynced notification from the clock, e.g.:
(In this case, init will not happen if it can't find a lead clock.)
Class Methods
.new
Create a follower clock instance.
Arguments:
| tempo |
Retained for interface compatibility with TempoClock, but will be quickly overridden by the lead clock's tempo. |
| beats |
Retained for interface compatibility with TempoClock. |
| seconds |
Retained for interface compatibility with TempoClock. |
| queueSize |
As in TempoClock, the size of the scheduler array. |
| id |
The ID of the lead clock to which to synchronize. Optional: If you don't provide one, this clock will respond to the first sync message it receives, and use the ID in that message. |
Returns:
The new clock instance.
Instance Methods
.tempo
Get or set the current tempo. This will try to send the tempo change to the lead clock, to broadcast to other follower clocks. Sync may be slightly unstable at the moment of changing tempo, but should recover quickly.
Arguments:
| newTempo |
A float, in beats per second. |
Returns:
A number.
.id
The clock's ID.
.latency
The latency value in the sync messages coming from the lead.
Returns:
A numeric latency value.
.bias
Sets or gets a timing offset. Positive numbers shift the clock earlier. You may need to increase the lead clock's latency for a positive bias. (Network delivery times in Windows are unstable, in some cases causing sync to be offset by a seemingly fixed amount. If this happens, usually the follower clock is late.)
Returns:
A numeric timing offset.
.diff
Accounts for the difference between the lead machine's system clock and the local machine's clock. (You cannot assume SystemClock.seconds will be the same across multiple machines.) Accessible for debugging purposes only; this information is not useful for scheduling.
Returns:
A number.
.netDelay
An estimate of the time it takes for the sync messages to travel from the lead to the local machine. Accessible for debugging purposes only; this information is not useful for scheduling.
Returns:
A number.
.addr
Statistics methods (normally not used)
.clockDriftFactor
A float, representing the expected rate of drift between the local machine's system clock and the remote lead's system clock. Formally, this is the "process noise variance" in a one-dimensional Kalman filter (https://www.kalmanfilter.net/kalman1d.html#kf2) -- as variance is the average deviation from the mean squared, and we expect fractional values well below 1.0, the squared difference should be lower than the actual drift rate. Normally, you should not need to change this. If you find sync is unacceptable, you might get better results by adjusting it.
.measurementError
A float, representing the expected variance in network message transmission times. Normally, you should not need to change this. If you find sync is unacceptable, you might get better results by adjusting it.
.debug
A Boolean. If true, messages will be printed in the post window, providing statistics about clock difference and network latency measurement. Normally, you should leave it at the default (false).
.debugInstability
Either 0, or a Function returning random positive numbers to be added to network measurements. In production, do not use this! There will be enough network jitter as it is. This method is intended only for testing sync between two sclang processes on the same machine.
Examples
You can run all of this example on one machine in one sclang instance, or run the sections on different machines as indicated.
Acknowledgment
Credit is due to Scott Wilson, for one critical idea borrowed from his BeaconClock class (in the Utopia quark). A possible issue with clock sync is processor contention: integer beats tend to be busy. BeaconClock and DDWLeadClock both broadcast sync messages at randomized intervals, so that sync messages will almost certainly be sent "in between" musical events, avoiding messaging delays.
How it works
You don't need to know this in order to use the classes. But, it's interesting, and also worth explaining to somebody who might want to modify the classes in the future.
The lead clock runs as a normal TempoClock. At random intervals (random, to avoid handling sync at metrically important time points when the language is likely to be busy), the lead sends a timestamped OSC bundle containing the clock's ID, current beats, latency and tempo. Timestamps are calculated based on the system time. We assume that both the DDWLeadClock's beat counter and the system time (and therefore, the OSC timestamps) increment in a stable fashion.
Follower clocks receive these bundles, and schedule an action to match the incoming timestamp. The action updates 'this' DDWFollowClock's tempo and beat counter (see TempoClock: -beats). Sclang receives the bundle immediately; the timestamp reflects the future time when the update should happen, irrespective of network jitter (in the same way that scsynth schedules timestamped bundles for reliable timing). The timestamps are not subject to jitter, so, scheduling the clock update according to the timestamp is also not subject to jitter.
The problem: The follower's system time is probably different from the lead's. The timestamp received in OSCFunc is correct on the lead machine, but not on the follower, but the difference between the two should be constant. If we can measure the difference, then it is easy to convert the timestamp into the local SystemClock.seconds. But the follower sclang does not know when (in its own timebase) the message was sent, only when the message was received. Receipt time is subject to jitter. There is always "measurement error," which may be severe (I observed up to +130 ms inconsistency).
To address this, I use a one-dimensional Kalman filter, assuming a stable value (the difference between two clocks' times in seconds should be constant) with a large measurement error. In practice, the clocks may drift slightly, but we can assume that any drift will be relatively slow ("process noise" in Kalman filter terms). Apart from this drift, one second on one machine should be one second on the other, so, if there is no network timing jitter, SystemClock.seconds - timestamp will be constant (or extremely slowly changing).
At the same time, the follower periodically sends "ping" messages to the lead, to estimate the one-way network latency (another Kalman filter). We assume network latency is some kind of random distribution, always positive, that applies consistently to all messages. Therefore, the difference between clock times will be biased slightly more positive than it should be, by an amount roughly equal to the network delay estimated by pinging. So, the sync method subtracts the one-way network delay.
So we have 1/ stable timestamps and 2/ a relationship between remote and local time that is stabilized by Kalman filters. I have tested up to +150 ms random timing offsets and found that the follower clock performs acceptably for musical content. If network jitter can be reduced, it can perform even better.
link::ddwOSCSyncClocks/Classes/DDWFollowClock::