Development of the MADOCA to EPICS Gateway
By Kazuhide Uchida (Cosylab Japan), Tilen Žagar (Cosylab), Akio Kiyomichi (Japan Synchrotron Radiation Research Institute; JASRI), Takemasa Masuda (JASRI)
In this article, we introduce our experience with MADOCA, control system framework used in SPring-8, SACLA, and other light sources in Japan. We will give the MADOCA system architecture, and the gateway between MADOCA and EPICS. Cosylab contributed to the MADOCA-to-EPICS gateway development last year. With the gateway it is possible to quickly add EPICS supported devices (e.g. Libera beam position monitors) into a MADOCA-based control system.
Introduction
SPring-8 (Super Photon Ring — 8 GeV) [1] is the highest energy third generation synchrotron light source, located in Hyōgo Prefecture, Japan. In 1997 it opened for industrial, academic, and government use. Around the same time, a new control system framework named MADOCA [2] was developed by JASRI (Japan Synchrotron Radiation Research Institute) to be used at the facility.
Message And Database Oriented Control Architecture or MADOCA also means a circle or completeness in Japanese. It runs on UNIX operating systems, and its strongest point is equipment diagnostics in combination with its database system feature that helped achieve a long and stable operation of the synchrotron [3] [4].
Over years of operation, some shortcomings of MADOCA showed up, and in 2012 development of first major upgrade of the platform started, called MADOCA II. The messaging scheme was rewritten using the ZeroMQ library [5] (adding support for Windows), and the database system was upgraded to a hybrid system of relational and NoSQL databases, greatly increasing performance.
As of 2017, four light source facilities in Japan (SPring-8, SACLA, NewSUBARU, HiSOR) are using MADOCA for their control systems.
MADOCA architecture
The MADOCA framework provides a set of software components that are required in an accelerator control system, including operator interface, message communication, data logging, alarm system etc.
Device control
To control accelerator equipment through devices such as VME boards, three software components are required in MADOCA. Minimal configuration for a single device is presented in Figure 1.
The client program is the interface to the device as used by users and services. On the bottom is the Equipment Manager (EM), where device support is implemented. Client programs and EMs for specific equipment are written by MADOCA users. In the middle is the Message Server (MS), which handles communication between them.
The accelerator equipment is controlled with a set of commands defined in the EM, which are sent and received in client programs. The EM developer decides what kind of commands to implement into the device support, and specifies them.
One client program typically communicates with more than one EM, running on the same or different computers. Each computer has its own Message Server that handles communication with other parts of the control system. A more detailed schema of such setup is displayed in Figure 2, which is based on MADOCA II. In the figure, MS and EM are noted as MS2 and EM2.
Messaging scheme
A control message sent across the MADOCA network is represented as a human readable text and consists of 4 parts: Subject (S), Verb (V), Object (O) and Complement (C):
- S is a client program name,
- V is an action for the EM,
- O is a target equipment name,
- C is an attribute such as set value and read-back value.
The message is sent from the client program to control the device. Then the message forms “S/V/O/C”. When the operation is finished by the object in the message, the return message is sent with a form “O/V/S/C”. The message always has the form of “(message publisher)/V/ (destination)/C”.
If the complement is an array instead of scalar value, the MessagePack library [6] is used to serialize/deserialize the array on both the client program and the EM.
S/V/O/C message flow is shown in Figure 3. A client program named CP1 sends a “put” command with complement 20 to Object O1 managed by EM1. After successful operation, the equipment manager EM1 returns “OK”.
Database system
The database system is one of the core components of MADOCA. The database system is independent from the MADOCA-to-EPICS gateway, but the database will be/is used in combination with the gateway. The database configuration and core functionality is based on [7] and [8].
The database manages device configuration and holds online and archived data. The database system architecture is displayed in Figure 4. Data acquisition changed significantly in MADOCA II. The acquisition process happens on computers that send data to relay servers on their own timing, which then write data into NoSQL database servers. Multiple relay servers can be used to achieve fault tolerance and load balancing.
There are three types of databases:
- Parameter database: manages static device data such as magnet positions, calibration data, device and host relations, reference value for the alarm system, etc. It is based on Sybase.
- Online database: manages live data from each equipment. GUIs, alarm system etc. access device values from this database, reducing load on devices. Fast read/write access is crucial for this database, so Redis is adopted.
- Archiving database: permanent data archiving is managed by one archiving database. Cassandra is used for this purpose as it provides high read/write performance, redundancy, and extensibility of the system.
Gateway to EPICS
EPICS (Experimental Physics and Industrial Control System) [9] is a popular Control System (CS) framework for building accelerator control systems. EPICS supports a variety of devices, including Libera beam position monitors, used in SPring-8.
To use EPICS supported devices in MADOCA, a MADOCA-to-EPICS gateway has been developed [10]. The gateway works as an MADOCA II equipment manager which uses Channel Access (CA) library [11] to communicate with EPICS. With the gateway it is possible to quickly integrate an EPICS supported device into MADOCA, reducing time and cost. Cosylab contributed to the gateway project; working on the original gateway upgrade and developing a prototype with a different underlying architecture to improve performance.
Gateway upgrade
As already mentioned, the MADOCA-to-EPICS gateway is implemented as an Equipment Manager. The gateway receives MADOCA S/V/O/C messages from client programs and access EPICS records via Channel Access. SPring-8 developed the gateway as a general purpose EM that supports caput and caget CA functions on a subset of EPICS data types.
It also has an improvised support for camonitor, where gateway subscribes to a PV for a fixed number of updates. Client programs must execute the “monitor” command periodically. Any unexpected events that can increase the load on the gateway or cause a delay on client programs can result in data loss.
All CA operations are implemented as blocking functions, waiting for a response after each request, which is also the reason why monitor is implemented as it is.
Before the upgrade, the gateway had a performance problem when the number of Channel Access operations per second increased above 50, when clients had to wait for seconds to get a reply from the gateway. The main problem was that for each operation, a Channel Access connection was freshly created and immediately destroyed, causing a bottleneck. Cosylab modified the original gateway to achieve better performance, while also adding support for additional data types. Figure 5 shows the upgraded gateway architecture.
Channel Access operations for different data types are defined as a special EM. The first time a specific PV is accessed, a CA channel is created, and connection status stored in a linked list. All sequential operations with the same PV refer to this list to find the right channel.
Performance was measured with two different methods. The first method we measured time between sending the get command from the client program, and receiving a reply from the EM, while the other method measured the execution time of Channel Access operations inside the EM. Both tests were repeated 300 times and results are shown in Figure 6.
In the original gateway, the typical CA operation time was around 20 ms (red), while after the upgrade it was consistent around 2 ms (orange). The magnitude of improvement was confirmed with the messaging time test. In some cases, the client received a reply from the gateway in more than 60 ms while averaging around 30 ms (green); the upgrade cut this time to 14 ms (blue). The performance of the gateway became more deterministic, while increased speed of CA operation improved the operations per second threshold.
Prototype of new gateway
Cosylab also contributed to the design of a new gateway and built a prototype to confirm functionality.
In the new architecture, all Channel Access operations are replaced with callbacks and subscriptions. The gateway stores PV values in local variables, and keeps track of disconnected PVs, while the CA send buffer is periodically flushed in a separate thread.
In such an architecture, multiple CA operations are sent over the network together, and the gateway returns values to the Message Server from its local memory, each increasing speed. Such designs also gave us an option for a proper monitor implementation, as PV updates can be stored in a buffer on the gateway.
When the client program asks for monitored values, it receives all the data that was acquired since the previous monitor command. This enables clients to receive updates when they are ready without missing or duplicate values. Arrays containing values is implemented as a circular buffer; in case clients don’t ask for an update, the oldest data will be overwritten first.
The architecture of the prototype gateway and monitor implementation can be seen in Figures 7 and 8.
The prototype is still one step away from the real monitor, as it doesn’t yet have the function to notify client programs when a new value is received (or when the buffer is full), something that can be implemented in the future.
References
[1] SPring-8 Home Page; http://www. spring8.or.jp/en/
[2] R. Tanaka, et al., “The first operation of control system at the Spring-8 storage ring”, Proceedings of ICALEPCS97, Beijing, China, p.1 (1997)
[3] T.Matsumoto, Y.Furukawa, M.Ishii, “NEXT-GENERATION MADOCA FOR THE SPRING-8 CONTROL FRAMEWORK”, Proc. of ICALEPCS2013, San Francisco, USA, pp944–947 (2013)
[4] SPring-8, “Development, implementation and operation of MADOCAII middleware for accelerator and beamline control”, SPring-8 Research Forum 2015
[5] ZeroMQ library; http://zeromq.org/
[6] The MessagePack library; http://msgpack.org/
[7] A.Yamashita, M.Kago, “MADOCAⅡ Data Logging System Using NoSQL Database for SPring-8”, Proc. of ICALEPCS2015, Melbourne, Australia, p.648 (2015)
[8] A.Yamashita, M.Kago, “STATUS OF MADOCAII DATA ACQUISITION AND
ABOUT THE AUTHORS
Kazuhide Uchida started working at Cosylab Japan in April 2015, first joining the iBNCT project as a control system integrator. Mostly working in EPICS, his main job was setting up the wire scanner. On the same project, he also did extensive commissioning support, which included emittance measurement, DTL detuning measurement, ion source parameter optimization, etc. His last projects include ECR ion source design, Tcl/TK and WinCC OA/C++ GUI development, and others.
Tilen Žagar is a Lead Control System Engineer in Cosylab and leader of developer team in Cosylab Japan. For the last 3 years he has been involved in development of EPICS based control system for boron neutron capture therapy machine in Japan. Currently he is working on design of EPICS V4 control systems. In his free time he likes to mountain bike, surf and spend time with his cats.
Akio Kiyomichi is a researcher at JASRI and has been working on the SPring-8 accelerator controls since 2010.He worked on the PHENIX experiment at BNL-RHIC from 1996 to 2005, first as a graduate student and later as a RIKEN postdoctoral fellow. Akio has a Ph.D. in Physics on heavy ion collisions research. As a member of JAEA and KEK, he worked on the development of the spill feedback system for the J-PARC slow extraction from 2005 to 2009.
Takemasa Masuda is an accelerator controls researcher at JASRI SPring-8. He is currently working on the control system for SPring-8. Recently, his research has focussed on the MTCA.4 platform and the White Rabbit timing system. In his free time he enjoys playing with his young son.
The article is republished from Cosylab’s Control Sheet Newsletter №32
