More elaborated but smarter and more flexible, the new ALICE shift management system based on the FENCE technology has been introduced in 2017 and tested on the field during the last weeks.
Interface from which shifts can be booked. It is also possible to see whether a specific shift block can be overwritten because the person’s institute is in overbooking. [Credit: Heron Hernique Martins Silva/CERN]
A new version of the system for managing and booking the ALICE run shifts has been released lately. Based on the FENCE software framework, it is more flexible than the previous one and allows for the definition of further rules and verifications. This might appear confusing or even annoying to some, but there are precise motivations behind this choice. Indeed, the recently developed software has been tailored to meet the needs of as many users as possible and to provide an easier and more structured organization of the shifts. The experience gained during the previous runs shaped the development of this advanced version.
In the beginning, when the LHC had started running, a very basic tool was used, called the ALICE Shift Management System (SMS). Later, in 2014, it was decided to switch to GLANCE, another technology developed by the Federal University of Rio de Janeiro (UFRJ) in collaboration with the ATLAS experiment, which had already been adopted for the ALICE collaboration membership database. The latter had proven efficient and flexible in the management of the affiliations, the roles of the collaboration members and their connected privileges, as well as the author list. This is because GLANCE allows various customizations, such as embedded directives and research filters.
The same kind of needs drove the collaboration to decide to develop the Shift Accounting Management System (SAMS) based on the GLANCE technology, in which a number of rules were introduced. For example, the system did not allow people to book two weeks of shifts in a row anymore – because the two periods have to be separated by at least a two-day break. In addition, the distribution of the time slots was changed such that a shifter would cover sequentially two days of morning, then afternoon, and finally night shifts.
A basic tool to control the shift quota booked by the institute was also included: in particular, the system used to keep track of the due and the covered quota of each shifter. But it was anyway responsibility of the team leader of each institute to calculate how much quota they still had to cover and indeed overbooking was not prevented by the software. A first improvement of the situation was accomplished by introducing the possibility of reassigning the overquota, in the sense that extra shifts could be taken away by other institutions. But still this was not sufficient to satisfy all the users. In particular, it used to happen that, as soon the booking system was opened, some institutes would hurry and book a lot of shifts, even more than needed to cover their quota. While others that reacted more slowly were left with no free spots. This approach was particularly disadvantageous for countries in different time zones than that of CERN, since the opening time of the booking could fall out of their working hours.
The situation is made rather complicated by the fact that there are actually not enough shifts to allow all of the institutes in the collaboration to complete their due quota. There are two kinds of shifts: the regular ones and the on-call ones. The latter are open only to experts of the various systems, who need to know in depth the kind of problems that can occur and to be able to debug and solve them quickly. Thus, only a restricted number of people (and, in particular, institutes) are granted permission to cover these shifts, whereas all of the other people have to share the regular ones, for which a short training is sufficient. The consequence of this was that it could happen that some institutes completed their quota, while others covered almost none of theirs.
Last year, the migration to an upgraded system was realized, since the UFRJ group responsible for the development of GLANCE had put in place the new FENCE (Front-End for GLANCE) technology. In particular, Heron Hernique Martins Silva, a UFRJ engineer who is part of the FENCE core development team, put all his effort in developing this version and ensure that the new system and related functionality were ready on time. He was helped by a team of developers at UFRJ and some ALICE collaborators for the code implementation, testing and validation.
The new system version is even more flexible than the previous one. For example, some controls and procedures that were set in the GLANCE-SAMS version could be introduced as configurable in FENCE-SAMS. An important change is that increasing thresholds for the booking were included. It means that, at first, institutes were allowed to book only up to 35% of their due quota. In this way, nobody could overbook and make impossible for the others to find a free spot. In the following weeks, the threshold was gradually increased, until it was finally removed. Still, there are not enough shifts to allow every institute to cover 100% of their quota, but at least with this system a more balanced distribution has been reached.
“Of course, this system is more elaborated than the previous one and requires the collaborators to act at various moments to book their shifts, but at the same time it meets the needs and requests of several institutes,” comments Adriana Telesca, the ALICE Resource Coordinator, who followed the development of the ALICE Membership and of the Shift Management systems since its start. “It is also smarter,” she continues, “because many features are automatized and less manual intervention is required from the System Run Coordinators and the Run Coordination. Besides, the system provides further information to the users, who do not need any more to contact an expert to know why they are not allowed to book any or some specific shift.”
Among the various new rules introduced in this upgraded system, there is the verification of the training validity. In order to perform shifts as the responsible for any of the systems in the control room, members have to follow a training course. Then, they have a time window for booking a first shift in that role, which is not the same for each system. If they do not do so, they have to repeat the training course, since it is considered that the knowledge acquired is quickly forgotten without practice. In addition, in order to keep the training valid, they have to cover at least one shift per year in that specific role. All of these constraints are known to the booking system, which will thus allow booking a shift only if the requirements are all fulfilled.
Another change is that now every shifter must follow the course to be Shift Leader In Matters Of Safety (SLIMOS). The booking system will randomly choose one of the shifters to actually cover the role of SLIMOS, nevertheless everybody in the control room will have been trained. In the past, the system had to search who among the shifters was allowed to be a SLIMOS and assigned him or her to this role. With the current SAMS, there is no need to worry about looking for somebody with the SLIMOS permission, since all of the shifters have it; furthermore, every shifter will know how to act if there is a safety issue.
“I think that the new system proved better than the previous one, even though it is more complex,” comments Kristjan Gulbrandsen, the ALICE Run Coordinator, who is following the shift booking for the 2018 run. “All of the institutes had the chance to book some shifts, so there is less disparity between them”. Nevertheless, there is some personal work that has to be done… “There are still a few shift leader spots to be filled,” Kristjan explains, “so I think I will have to contact individually some of the people who are qualified to cover this role and try to convince them to take some of those shifts.”
Run statistics on the shift booking interface. [Credit: Heron Hernique Martins Silva/CERN]