Embedding a Self-Managed Team at PSI-SwissFEL
Tom Slejko, our Senior Systems Architect, talks about the work of his team on the SwissFEL X-ray free-electron laser at the Paul Scherrer Institute (PSI) in Switzerland.
Tom Slejko’s team is helping the controls group of Markus Janousch, Head of Section Controls at PSI, with their control system integration project for the new SwissFEL machine. Cosylab team is, on the one side, embedded: they are full-time on-site in this period right before commissioning, while, on the other hand, they are fully self-managed, delivering what is required from a brief description and then formally accepted by PSI.
Hi Tom, before we look into the specifics of what your team is doing, can you quickly sketch the overall SwissFEL project and its current status?
Tom: Sure. SwissFEL is an X-ray laser. It’s the next large research facility at the Paul Scherrer Institute and it’s a new, cutting-edge machine in addition to the 3 existing machines which are a high-intensity proton accelerator (HIPA), the Swiss Light Source (SLS) and the superconducting proton cyclotron (COMET), used for proton therapy but I shouldn’t deviate too much.
Construction on SwissFEL began in April 2014 and right now things are being prepared to start commissioning the injector. One key feature of SwissFEL will be its extraordinary brightness, which means high intensity combined with sharp focus and narrow spectral range. X-rays produced by FELs have a brightness 12 orders of magnitude higher than those produced by synchrotron light sources.
When complete, SwissFEL will allow scientists to observe ultrafast changes in the matter because of its short pulses, around 10 femtoseconds, and short wavelengths of around 0.1 nanometers.
Wow :) Additional question here. Is there anything that sets SwissFEL apart from other FELs?
Tom: Without going into detail, in comparison with other projects that are able to produce hard X-ray pulses, e.g. the LCLS at SLAC and the European XFEL at DESY, SwissFEL will be more compact and cost-effective. The novel injector has a really low emittance, which allows for a shorter LINAC with a simpler design. Regardless, it’s altogether a very complex machine of 740 meters!
We’re guessing that the control system will have a complexity to match. Can you tell us a bit about how you fit into that project?
Tom: It’s fair to say we work in a very compatible manner with the local controls group. We assist the controls group with their day-to-day operations and development as well as support the same internal customers: diagnostics, beam dynamics, timing and synchronization, low-level RF and other groups. We provide a wide range of services: developing device support, configuring IOCs, developing high-level applications, a lot of synoptic overview screens and the list goes on.
I believe that our team has adapted very well to how things are done at PSI, both culturally as well as technically, where we have adopted existing PSI systems and development environments to allow for a seamless transition from Cosylab to PSI developers. In other words, we have adapted our standard development practices in a way that they are compatible with the ones at PSI.
Control Sheet: So it’s a wide range. Do you also have your specialty areas?
Tom: One of the bigger undertakings was definitely the development of the SwissFEL timing system. We were involved in the development of device support, assisting PSI with SAT [siteacceptance tests] for timing hardware as well as the development of the SwissFEL timing master setup. In all of these tasks, we had a very close collaboration with PSI’s timing expert Babak Kalantari.
This collaboration has been very beneficial for everyone involved, as, in addition to getting things done, everyone working on the project also got a chance to expand their knowledge.
We were able to offer our wholesome EPICS expertise and Babak shared his in-depth knowledge of accelerator timing systems.
Another area is Simon Ebner’s beam synchronous data acquisition. Here we developed a standalone serialization library for their BSDATA (beam synchronous data acquisition serialization protocol) as well as EPICS IOC modules that allow for easy addition of beam synchronous data acquisition to existing IOCs.
A third area is all things related to motion control. Here SwissFEL opted to standardize around Delta Tau’s Power PMAC, Power Brick. Cosylab has a long-standing partnership and years of experience with Delta Tau systems.
Can you tell us how you work; is there a specific process you are following?
Tom: We have weekly meetings with all group leaders where I sit in as the team’s representative. There we merely define what needs to be done and what are the priorities. Our team then takes it from there as far as our tasks are concerned and we work as autonomously as possible. But, of course, coordination with other PSI teams is often needed.
Implementation-wise, we use JIRA to track the progress of individual tasks and to log key information and decisions. For example, during the development of application ‘X’, we’d gather the requirements by talking to different people at PSI and prepare our design proposal. The internal customer then reviews it, by simply entering comments in JIRA. This allows us to have a very clean interface with the customer, but it also enables us to log data in a traceable way for review at a later stage, e.g. when the original developers are no longer on-site. The same process is followed throughout the whole development cycle: collecting all relevant information in one place, including decisions that might otherwise be omitted from the official documentation.
Once the tasks are completed, the PSI personnel can review them and accept, i.e. close them, or reopen them for rework. It might sound like an overly formal, bureaucratic system, but in practice, it causes almost zero overhead, since the majority of correspondence moved away from signed documents and emails into JIRA…
Surely people are happy when you take over bureaucratic tasks, like writing requirements :)
Tom: True! It’s a burden we lift from people’s shoulders :) Many perceive writing formal requirements as quite difficult and tedious, especially for many smaller tasks. It might even make them steer away from subcontracting, feeling that it’s just all more work. It’s my strong belief that quite the opposite is true: in the medium and long-term, this disciplined formalization saves rework big time. A control system is not the accumulation of man-hours spent writing lines of code, it’s a managed system!
An additional benefit of having the Cosylab “externals” team gathering requirements with end-users directly is that it allows the injection of ideas and best practices we have collected over the years from hundreds of other control systems assignments we have completed. That’s a massive time-saver.
Talking formal processes, are you describing a “waterfall” process here? Or do you work iteratively as well?
Tom: At times, the only way to clarify requirements is through iteration. A partial design and implementation reveals the ins and outs and allows us to define an improved set of system requirements. When working like this, we often try to rely on tools that allow for very short development cycles, for example, Python.
Cosylab is used to working with vague requirements and knows how to get to a solution in the most effective way. Here I would say the on-site presence is a major plus; it allows for instant feedback and faster turnarounds.
It is very common for us to develop a quick, proof-of-concept over a couple of days and then present this to the customer and iterate over that. Customers appreciate being presented with more than one option for what can be done, and here we can bring a lot to the table.
It sounds to me you’re using Agile in the sense of Pro-active.
Tom: And it sounds to me, you like big and fancy words! It’s simple. We work with the customer to decide about what is the next logical chunk of work that needs to be done to help the project move forward. That’s huge added-value in a complex integration project, as it is by no means trivial. We’re simply user-driven, on the lookout of what users are in need of at any point in time and ready to help.
ABOUT TOM SLEJKO
Tom Slejko joined Cosylab in 2009 as a software developer and then moved through the ranks of Control System Engineer, Control System Architect and now Project Leader and Senior Architect for the PSI SwissFEL project. Tom’s professional interests include distributed real-time control systems, Linux kernel and embedded development. In his free time Tom enjoys paragliding, motorbikes, hiking, climbing and anything else that spikes his adrenaline levels :)
Do you want to learn more about this topic? Please let us know in the comments below!
