Implementation
This section covers the actual details of the implementation of the system in hardware.
The DOGMA tree (dog links)
Physical layer
The dog link serial transmitters and receivers are implemented in the FPGA’s fabric using normal I/O pins (not hard implemented multi-gigabit transceivers (MGTs)) to have full control over the details of the bit transfer with fixed latency and to be able to directly access the raw incoming serial signal for clock recovery without intermediate hardware (e.g. a signal splitter) in the data path. To achieve the beforementioned goal of transmitting a low-jitter clock together with trigger and control signals on one optical, serial link, we came up with a custom encoding scheme that draws inspiration from existing protocols which use duty cycle/ pulse width modulation of clock signals (for reference, older publications about clock-edge-modulation can be found here: https://ieeexplore.ieee.org/document/4295185. ).
The initial idea was to transmit an actual clock signal and only vary the duty cycle to encode data (as has been done before). However, due to the use of optical links, this requires encoding of the transmitted data to ensure that the data is dc-balanced. Tests using 8b10b encoded data revealed that the dc-level variations allowed by the 8b10b encoding still had quite an impact on the jitter of the recovered clock at least with the PLLs tested for clock recovery (a small, inexpensive and low-power PLL is needed for the use on front-end modules).
To improve on this, we came up with our own encoding scheme to meet our design goals (easy clock recovery, low rxclk jitter, ease-of-use, bandwidth).
Dog link protocol
The current dog link data encoding scheme defines a set of 9 different bit patterns (symbols) that can be transmitted in random succession. Since we want to be able to principally transmit arbitrary information (trigger numbers, 32bit/64bit timestamps etc.), we need a communication protocol based on this set of symbols (let’s call them S0-S8).
The current dog link protocol uses the symbols S0-S7 to encode 3 bits of data, while the symbol S8 is used as a control symbol (EOT/IDLE). The link decoding statemachines of the receivers are reset to the idle state whenever an EOT/IDLE symbol is detected. This ensures that the receiver cannot be permanently affected by incoming corrupt/ bogus transmissions and will always attempt to decode the succession of symbols coming in after the last received EOT/IDLE according to the protocol (stateless/ self recovering receiver).
Downlink (DCM -> Dogs)
The downlink protocol treats the first two incoming symbols as trigger identifier and the symbols thereafter as trigger payload. For technical reasons, symbols S8 and S7 are sent alternately as IDLE sequence. Therefore, symbol S7 cannot be the first symbol of a trigger transmission which means that there are only 7 options for the first symbol of a trigger transmission, yielding a total of 56 possible different trigger types when two symbols are used as identifier.
Uplink (Dogs -> DCM)
The uplinks are using the same protocol, except that only the first symbol is currently used as trigger identifier. So only 7 different trigger types are currently defined on the uplinks.
Register Access Protocol Specification
General
All modules in the dogma gbe network (= “dogs” ;-)) are listening for UDP frames on port 60678 (0xED06). Since the internal registers and the control bus of the dogma modules are 32 bits wide, the control protocol is also built on tuples of 32 bit / 4 byte words. Incoming UDP frames are checked to satisfy the minimum length (in bytes) demanded by the protocol (exact number still tbd, because protocol not fully defined, yet) and to have a length which is a multiple of 4 bytes. If one of the conditions is violated, the frame is discarded unprocessed.
Request Packet Format
Request Header
Offset |
Size |
Field Value / Description |
|---|---|---|
0x00 |
4 B |
Magic Constant 0xECC1701D |
0x04 |
4 B |
Placeholder Must be 0xFFFFFFFF |
The request header is followed by one or more command descriptors.
Command Descriptor Layout
Offset |
Size |
Field Description |
|---|---|---|
+0x00 |
4 B |
Command ID / Type Command identifier and access flags |
+0x04 |
4 B |
Register Address Start register address |
+0x08 |
4 B |
Transfer Length Number of registers to transfer (only bits [15:0] are used) |
+0x0C |
4×N B |
Write Data Present only for write commands |
For write commands, N data words follow immediately.
For read commands, no additional payload is present.
Bits |
Name |
Description |
|---|---|---|
31:16 |
command_id |
User-defined command identifier |
15:5 |
Reserved |
Must be zero |
4 |
access_write |
1 = write, 0 = read |
3:0 |
access_modifier |
Access modifier (0x0 = normal access, 0xf = delay) |
The delay access modifier is basically a dummy read command without any response (like a write command). Use register read length n to force the FSM to count from 0 to n in n cycles.
Response Packet Format
Response Header
Offset |
Size |
Field |
Value / Description |
|---|---|---|---|
0x00 |
4 B |
Magic |
Constant 0xECC1701D |
0x04 |
4 B |
Reserved |
Ignored by receiver |
0x08 |
4 B |
Timestamp |
Packet timestamp |
The response header is followed by one response entry per request command.
Response Entry Layout
Offset |
Size |
Field Description |
|---|---|---|
+0x00 |
4 B |
Command ID / Type Echoed from request |
+0x04 |
4 B |
Register Address Echoed start address |
+0x08 |
4 B |
Response Length Number of returned data words |
+0x0C |
4 B |
Timestamp / Status Packed timestamp and AXI4-Lite status |
+0x10 |
4×N B |
Read Data Present only for read commands |
Timestamp / Status Field
Bits |
Name |
Description |
|---|---|---|
31:16 |
Timestamp |
16-bit timestamp with lower 12 bits of ctime discarded |
15:3 |
Reserved |
Reserved bits for future use. |
2 |
Length error |
Set if the length of the response is too big. |
1:0 |
Return Value |
AXI4-Lite response status |
AXI4-Lite Response Status Codes
Value |
Name |
Description |
|---|---|---|
0b00 |
OKAY |
Normal access success |
0b01 |
EXOKAY |
Exclusive access success |
0b10 |
SLVERR |
Slave error |
0b11 |
DECERR |
Decode error |
The response status corresponds to the AXI4-Lite BRESP or RRESP field associated with the register access.
Ping of Death (PoD)
Dreamstream has some low level command interface to be able to reset the device via simple ping commands. The PoD resides on the RX data path of the uplink MAC and can’t be blocked by internal issues.
The POD can be sent by normal ping command (may need sudo rights in case of broadcast pings), and a special payload needs to be sent. The payload argument needs to be 32bit aligned, otherwise some implementations of ping will drop the last bytes.
A magic word ‘0xecc1701dabad1dea’ is the first part of the payload. the second word carries the command byte and (if applicable) the argument byte.
Example:
ping 10.123.0.145 -c 1 -s 8 -p ecc1701dabad1dea -W 1
This POD (sent to 10.123.0.145) will execute a warm reset (command ‘0xc3’). The option ‘-c 1’ sends only one ping packet, and ‘-W 1’ limits the waiting time for the (never arriving) answer to 1 second.
Caveat: a broadcast POD may or may not work at once. In case a module in the root area of the tree structure is affected, the first ping will reset this one, but not necessarily the following modules. If you need a reliable POD in the tree structure, you should check BlackCats DMS (DLM messenging service) implementation.
DAQ System
This section (hopefully) covers the necessary details of the implemented triggered data acquisition system.
Implemented trigger functionality
With the current dog link protocol, 56 different triggers (numbered 0-55) can be transmitted downstream (DCM -> DOGs). So far, every trigger transports 64 bits payload.