SwissFEL Event Timing System

SwissFEL is the Paul Scherrer Institut’s (PSI) new Free-Electron Laser facility.

By Babak Kalantari (PSI), Roger Biffiger (PSI), Sašo Skube (Cosylab), Tom Slejko (Cosylab), Tomaž Šuštar (Cosylab)

Introduction

The new Free-Electron Laser facility is a 740 m long accelerator with the goal of providing pulses of light between 6 and 30 fs long at a wavelength of 1 to 7 Å at 100 Hz [1, 2].

Figure 1: SwissFEL building

In order to support flexible beam rates and machine protection assistance, a reliable and adaptable triggering system is required. The triggering system must be capable of distributing events at a maximum rate of 142.8 MHz and also be able to program flexible event sequences at 100 Hz. In addition to the distribution of the events, the timing system has to distribute a unique identifier for each pulse (the Pulse ID), to support beam synchronous data acquisition [3]. This article presents solutions for the event timing system that were developed in collaboration between PSI and Cosylab.

Overview

The Event Timing System (Figure 2) distributes triggers along the machine. A trigger is distributed by sending a special event code over the timing network to the final Event Receiver (EVR). The EVR is then configured to perform the desired actions upon receipt of specific event codes, e.g. generating an electrical pulse.

Figure 2: Event Timing System layout at SwissFEL

The Timing Master (Event Master — EVM) generates an event stream (sequences of event codes) and transmits it over an optical link to the distribution stations/fanouts. There is one timing distribution station for each section of the machine (Gun, Injector, Linac1, etc.). Timing distribution stations consist of multiple fanout modules that distribute the event stream to the final clients. The two SwissFEL experimental areas each have one timing distribution station. These distribution stations have additional logic which allows users to add custom events to the incoming (original) event stream.

In addition to the event stream, the timing system also distributes a unique identifier for each pulse (Pulse ID) and the super-period [3], i.e. a signal that represents the greatest common divisor of all frequencies used in the facility. The distributed timing system is integrated into the Experimental Physics and Industrial Control System (EPICS).

Data Distribution over the Timing Network

The timing network is also used for real-time data distribution. Here, two types of data distribution mechanisms are available:

1. Distributed bus: distributes 8 binary signals across the entire timing network. SwissFEL uses this mechanism to distribute the super-period signal.

2. Distributed data: distributes arbitrary data packages that are time-multiplexed between event codes and distributed bus data (Figure 3). SwissFEL distributes per pulse data (e.g. unique Pulse ID, etc.) using this feature. Distributed data can also be sent from event receivers back to the timing master.

Timing Protocol

In each event clock cycle (142.8 MHz), two bytes are transmitted. The first byte is reserved for the Event code, while distributed bus and distributed data bytes are transmitted alternately in the second byte (Figure 3). The bytes are encoded with 8B10B encoding so in essence 20 bits are transferred in every cycle of the event clock [6].

Figure 3: How data is transmitted

Hardware

The Timing System is based on Micro Research Finland (MRF) event system products [5, 6]. SwissFEL requirements pushed MRF timing components to a new generation (300 series) that includes:

• Event clock up to 142.8 MHz
• Active delay compensation
• Masked sequencer events
• Conditional trigger outputs on the EVR, based on the presence of the events in a sequence
• Automatic identification of the timing network topology
• Improved data transfer protocol for sending arbitrary data (Distributed data), by introducing partial (segmented) data transmission
• EVR embedded inside the EVM

Figure 4: Event Timing System hardware used at SwissFEL

Figure 4 shows different hardware components that are in use at SwissFEL:

1. RF clock input: The RF clock input is used to provide the event clock (142.8 MHz) to the timing system.

2. Digital I/O: Ports can be mapped and distributed across the timing network. The EVM also uses digital inputs to trigger sequences (e.g. an AC trigger derived from the mains frequency) and handle machine interlock signals.

3. SFP optical transceivers: Timing hardware uses industry standard SFP optical transceivers thus supporting both single and multimode fibers.

4. Modular I/O: Inputs/outputs are modular and can be LVTTL, LVPECL, HFBR or NIM.

5. Recovered event clock: The EVR can generate an output using a recovered event clock that is in phase with the event master clock input

6. Triggers: The EVR can generate electrical pulses with configurable widths and delays. Pulses are triggered by received events.

7. Inputs: The EVR supports inputs that can be configured to send an event code to the timing network or propagate their input state over the Distributed bus (see: Data Distribution over the Timing Network).

8. Form factors: Different form factors support different host interfaces. At SwissFEL, VME and PCIe and embedded form factors are used.

The VME-EVM-300 is used as an Event Master as well as for Fanouts in the timing distribution stations. Event receivers are available in VME and in PCIe form factors. In addition to hardware EVRs, an open source EVR IP Core (FPGA module) “embedded EVR” is also available that allows development of custom FPGA applications.

As of August 2018, the following timing system hardware components are in operation at SwissFEL:

• 73 VME-EVM-300 boards
• 44 PCIe-EVR-300 boards
• 156 VME-EVR-300 boards
• 194 embedded EVRs

Timing Master

The Timing Master is the heart of the timing system. The Timing Master (Figure 5) consists of the EVM and an IFC1210 single board computer, both in a VME crate. The Timing Master Application that runs on the single board computer is responsible for:

• configuration and programming of the EVM,
• exposing the status and control parameters to the control system and its users.

Figure 5: Timing Master architecture

Figure 5 shows the architecture of the Timing Master:

1. The Timing Master Application is implemented as an EPICS Input Output Controller (IOC) application running on top of a Linux OS with a PREEMPT_RT patched kernel. It is responsible for:

• configuring and monitoring of the EVM
• calculating timing/event sequences
• real-time programming of the sequence buffers (switch sequences every 10 ms)

2. The Distributed bus is used to propagate the super-period signal.

3. The Timing master distributes real-time/per pulse data. (current Pulse ID, timestamp, etc.)

4. Event sequences are calculated on the fly at the machine repetition rate (100 Hz). Two sequencer RAMs are used in a ping-pong mode; while one is executing a sequence the other one is being programmed. (Figure 6)

Figure 6: Ping pong loader

Beam Rate Control

Electrons get accelerated only if the Gun Laser and RF pulse fire in sync with each other. If the Gun Laser is delayed with respect to the RF or vice versa, the electrons are not accelerated, and hence there is no beam. The adjustment of the relative delay between the Gun laser and RF is used to control the beam rate while still allowing the systems (Laser, RF) to run at constant repetition rates in order to guarantee temperature and overall stability.

Figure 7: Beam rate control

If the beam generation has to be suppressed, special events (e.g., “Laser OFF”, “RF OFF”) are distributed. If these “OFF” events are present in the current sequence, a different delay is applied when the local trigger pulse is generated (in the laser lab or at the RF station). The same approach is used to suppress the beam in the case of a Machine Protection System (MPS) alarm. This logic is initially configured through software but run directly by the hardware, in order to achieve reliability and short response times in case of the MPS alarm. For this, special masking features of the hardware components were used. On the Timing Master, the masking logic controls the distributing of the “OFF” events depending on the MPS or software inputs. On the EVR side, the masking is used to control the local delay of the trigger depending on the previously received events. An example is shown in Figure 7.

Developed Software

In order to prepare the Timing System for operation, a large set of software has been developed in collaboration between PSI and Cosylab. The software ranges from low-level device drivers to graphical user interfaces. Hardware interfaces and low-level logic are written in EPICS (C++). System properties are exposed to the network through the EPICS Channel Access protocol. User interfaces were created with CaQtDM (Display manager for EPICS used at PSI) and Python. The software that was developed or adapted for the SwissFEL Timing System includes:

1. Low-level software:

• Linux Kernel module
• mrfioc2: EVM and EVR EPICS device support (driver)
• mrfioc2_regDev : EPICS module to send and read arbitrary data across the event timing network
• epics_bsread: EPICS module to support beam synchronous data acquisition
• All EPICS modules also run on Windows

2. Triggering Applications:

• Timing Master Application
• Complex Triggering Application: allows beamline users to build their own complex control sequences which are synchronized with the machine timing. This is achieved by defining and adding custom sequencer events to the incoming (original) event stream.

3. High Level Applications and user interfaces

• Topology tool (Figure 8): this tool helps to quickly find a faulty port
• Synoptic Overviews

Figure 8: Example of a topology tool developed: Synoptic overview panel

Upcoming Challenge: Two-bunch Operation

The upcoming challenge of the SwisFEL machine is the so called “Two-bunch operation” that will eventually enable both SwissFEL beamlines (Aramis and Athos) to operate at the nominal repetition rate of 100 Hz. In the two-bunch accelerator operation mode, two electron bunches will be accelerated within the same RF pulse. To generate the two bunches, two separate lasers will be used. By controlling the relative delay of each laser as described in the “Beam Rate Control” section, the repetition rate of each bunch will be controlled separately.

References

[1] Ganter, R., Abela, R., Braun, H. H., Garvey, T., Patterson, B., Pedrozzi., M., Reiche, S., et al. (2010) SwissFEL Conceptual Design report, PSI https://www.psi.ch/swissfel/SwissFELCDREN/SwissFEL_CDR_web_small.pdf

[2] https://www.psi.ch/swissfel/ (2018, April)

[3] Kalantari, B., Biffiger.R. (2017) SwissFEL Timing System: First Operational Experience, ICALEPCS2017, Barcelona, Spain

[5] J. Pietarinen (2017) MRF Timing System Status Update, presentation from Timing Workshop ICALEPCS2017, Barcelona, Spain

[6] MRF (2017) Event System with Delay Compensation — Technical Reference

Notes: General concept and design of the timing system was done by PSI. Cosylab contributed to software design and implementation.

This article is based on the poster that was presented at the annual meeting of the Swiss Physics Society in Lausanne in August 2018.

ABOUT THE AUTHORS

Babak Kalantari is the timing expert at PSI with an academic background in communication systems and signal processing. He joined PSI at 2001 right at the start of SLS commissioning. Major projects where SLS, where he designed and implemented fill pattern feedback system, SwissFEL; where he made conception and the design of the facility timing system. In the warmer months he enjoys playing tennis and swimming and in the winter, he enjoys skiing in the Swiss Alps.

Roger Biffiger is a software developer at PSI. He has many years of experience as a software developer in industry and has been working on the SwissFEL Event Timing System for 2.5 years.

Tomaž Šuštar joined Cosylab in 2015 and shortly after left to work on-site at PSI. He has worked for SwissFEL ever since and is currently the Project Manager of the Cosylab team at SwissFEL. Two of his projects for SwissFEL include installation and commissioning of diagnostic devices, digitizer integration. In his free time, he enjoys spending time with his family, playing double bass and hiking.

Sašo Skube joined Cosylab in 2014, his first project being driver interfacing a device for post-mortem data collection. He joined the PSI on-site team in 2015 to work on timing system development. At PSI, he has worked with the timing & synchronization group, which provides pico-second precision timing across the SwissFEL machine. His other projects include medical software development for treatment control system for proton therapy. In his free time, he enjoys spending time with his family, and outdoor sports like hiking and cycling.

Tom Slejko joined Cosylab in 2009 as a software developer and then moved through the ranks of Control System Engineer, Control System Architect to Project Leader and Senior Architect for the PSI SwissFEL project. He has worked on a wide array of projects, including Fast Orbit Feedback for PAL PLS2, Remote Handling for ITER and Backoff PLC-based control system for QualySense. Tom’s professional interests include distributed real-time control systems, Linux kernel and embedded development. In his free time Tom enjoys paragliding, motorbikes, hiking, climbing and anything else that spikes his adrenaline levels :)

Don’t miss the next story!

Do you want to learn more about this topic? Please let us know in the comments below!