by Federico Ronchetti, Gilda Scioli and Panos Charitos. Published: 17 September 2013

ALICE Matters met the new Run Coordinator, Federico Ronchetti and asked him about the new shift model that is introduced in ALICE. One of the main novelties of the new shift model is the "rotating schedule", along with many others, like the new training and booking system as well as the role of the Training Coordinator. Read more in this interview with Federico Ronchetti and get ready for your next shift!

The shift model used so far in ALICE was based on a so-called “fixed schedule”. In this schedule a block of shifts consists of the same time slot over a a certain number of days (six in the case of ALICE). For instance, one could book six day shifts (8h to 16h), afternoon (16h to 24h) or night (24h to 8h) shifts. Since the ALICE Shift Management System works on a first-come, first-served basis, this approach allows to pick shift blocks with the “best” time slots, such as “day” or afternoon” with respect to the much harder blocks based on six nights in a row.

Federico Ronchetti, co-head together with Gilda Scioli of the ALICE Consolidation Task Force and Run Coordinator for 2015, explains: “Obviously six nights in a row strike really heavily on one's routine and it is perfectly understandable that people tend to avoid the night shift blocks. In the past we had serious problems covering night shifts, especially the role of the Shift Leader. It was an issue that the new run coordination had to address”. In fact, when large blocks of shifts remain uncovered replacements usually can be found only with the last-minute help of volunteers. This process is quite inefficient and unfortunately ends up having always the same group of experts “squeezed” to cover the leftover blocks.

Federico Ronchetti in the ALICE control room

In order to address these issues, a proposal to introduce a more rational and fairer shift schedule which could intrinsically overcome the limitations of the previous model came up already at the end of the 2012 LHC running period when Gilda Scioli was in charge of the ALICE Run Coordination. Federico, who took over from Gilda, developed her ideas to build a complete redesign of the ALICE data taking operations including a new shift schedule . He aims to implement a broader framework for personnel training and management.

A more huMAN schedule

Proposing a new shift schedule is always difficult. “People tend to get attached to the existing routine even when it proves to be inefficient” Federico says. Before changing the ALICE shift schedules Federico and the ALICE Consolidation Task Force team met and discussed with the experts of the LHC Control Centre (CCC) as well as from several other CERN experiments such as ATLAS, CMS, LHCb, and AMS.

“We had very interesting discussions with our colleagues from the other CERN teams and we learned how they envisage and implement their data taking operations. The discussions were not limited to the shift schedules but touched many other aspects of the operations, as, for instance, the differences in layout of the respective control rooms and its implications for the overall communication between the shift crews. Focusing on the shifts, we found that ALICE was imposing one of the longest night shift routines. It’s certainly true that both ATLAS and CMS have adopted fixed shift schedule; however ATLAS allows only four nights in a row in the weekly shift blocks (monday-thursday) or three nights in the week-end blocks (friday-sunday). CMS has a fully open, day-by-day, booking system with different weights for the day, afternoon or night shifts. However, collaborators cannot book more than five shifts in a row and usually no one likes to book five nights, even though it is possible. We thought none of these approaches would be suitable for ALICE. The ATLAS two-blocks model would require a modification of our basic shift block of six days into two asymmetric periods of four and three days respectively. On the other hand, the CMS model is extremely dynamic but very far from our past habits and has the drawback of no invariant shift crew travelling from one shift to another, since each day they end up with different people in the control room. In my opinion such dynamism does not favour teamwork. LHCb uses yet another approach where each collaborator gives availability fairly in advance and then web portal system perform the matching. Also this model stands quite far from our approach and did not really seem easily adaptable to our needs. On the other hand, we found inspiration from other CERN teams such as those of CCC and AMS, who do not use fixed, but rotating schedules. In fact, these teams apply a rotation of time slots within the single shift block. Another point that struck our attention was that both these teams, for different reasons, deal with very critical and challenging operations. As one can imagine the LHC shifters do perform one of the most crucial set of operations at CERN and for this reason they have preferred a shift schedule that minimizes the impact of stress and sleep disruption on the operators performance. The AMS crew is also in charge of very delicate operations: their detector is installed on the International Space Station! So after having performed a critical review the rotating schedules used by these two CERN teams, we convinced ourselves that most of our problems could be overcome by introducing an internal rotation within the ALICE six-days long block structure“, says Federico.

Scheme of the new ALICE shifts schedule.

As a matter of fact, rotating shift schedules come in different varieties. But what is the advantage of this “rotating” sequence with respect to the flat schedule? “There are many advantages from the operational point of view: we opted for breaking down the six days long block into two day shifts, two afternoon shifts and two night shifts. When one books a block, one gets always this sequence. In this way all the blocks are treated on an equal footing and the first-come first-served model gets much fairer: the number of night shifts, being only two for each person, is now evenly distributed by design. I believe this routine is much more sustainable since it dilutes the night shifts among all personnel. In addition, always starting with day shifts is a good form of refreshment for experts and should be less intimidating for novices since at daytime all the subsystems experts are around and can easily assist the shift crew. In addition, the rotation of the time slot from day to afternoon to night, gently drives the shift crew from the busy day shifts to the more lonely night ones. The approach obviously assumes that the personnel is more or less in sync with Geneva's time when the first two day shifts of the block start. However, it would be virtually impossible to tune a shift schedule over a time zone other than the local one.

The rotating schedule also ensures that all shifters are exposed to the different peculiarities of the different types of shifts. The type of rotating shift schedule adopted in ALICE is referred as MMAANN (two mornings, two afternoons and two nights) and, as said before, retains the familiar six-day block structure but interleaves it with the three different kinds of shifts. In the past, the ALICE form of the MAN schedule (as it is synthetically called) was adopted by the H1 experiment at DESY. This form of rotating shift schedule is condescended by the experts in human labour as one of the best compromises. Beyond experimental physics facilities, MAN schedules are applied in industrial environments and in general when the attention levels have to be at best, as for example for paramedical and fire-fighters crews.

On the other hand, it is known from the available literature (one can do a Google search on this topic) that personnel tend to prefer fixed schedule shifts. However, it seems that many of the reasons for this preference are more psychological than physiological. In fact, fixed schedules appear more reassuring since people feel they can more easily plan their time. Double thinking reveals that long sequences of the the “good” fixed schedule blocks such as day or afternoon ones can be also very hard and disruptive.

Federico notes that: “In terms of sleep disruption a MMAANN block should be somehow comparable to a full block of afternoon shifts. In the fixed schedule since after the first two day shifts one gets and extra 8h of rest. The same is also the case after a second pair of afternoon shifts In terms of nights, asking each collaborator to carry on two lonely shifts is quite fair and doable in my opinion.  There is no doubt that the MAN schedule is a compromise but of the kind which aims higher”.

The last run1 Shift Leader dashboard at ALICE_Experiment.

Synchronization matters

The new shift schedule will bring also another change in term of timings.  In the past ALICE was undergoing a shift crew swap at 8h, 16h and 24h. On the other hand, all other experiments (except AMS) are changing over in sync with the CCC crew, that is at 7h, 15h, and 23h.

As Federico says “Changing the ALICE crew, say at 16h, very often led to the situation where the CCC afternoon crew – who swapped shift one hour earlier, at 15h - would start to move the machine closer to physics production right when our crew is swappping which is also when the entropy in the control room peaks! In our experience this situation was quite common and to minimize the risks of mishandling a crucial step, I believe we should swap crew exactly at the same time as the CCC folks. Another beneficial side effect is on night shifts. Beginning the night shift at 23h allows exploiting some of the daytime residual energy for the shift work. In my experience, I had always trouble for the first night shift where I was forced to stay awake until midnight and then start the long night ride. I obviously could not force myself to sleep in the afternoon especially before the first shift”.

One final note is on the crediting system: with the MMAANN schedule the credits will be reset to one credit per shift, regardless of the time slot, as it was in the early days. “In the past, when we used the fixed shift schedule, Gilda raised the credit weight for the night-shifts to encourage people to provide coverage, however it did not work very well, especially for the role of the Shift Leader. I believe the reason was that six nights in a row were just too uncomfortable regardless of any rewards.  In the MAN schedule all blocks contain an equal number of shift types and hence they will yield equal credits” says Federico.

The ALICE crew during the first proton-lead collisions.

ALICE shifters; a learning experience

“Awareness and responsiveness of the personnel participating to the data taking operations are crucial factors for the ALICE final physics output. It’s a trivial consideration that less data and low quality data will impact on our ability to publish quickly and authoritatively. So our colleagues taking shifts should feel the participation to the data taking operations as a critical task to be performed with the highest awareness and motivation” says Federico.

In order to achieve these goals the individual trainings for the general ALICE subsystems will become more similar to lessons or courses probably taking place during the ALICE weeks and mini-weeks. A final verification will be required at the end of each course. In addition to the classical training sessions of the past, there will also be a dedicated course for Shift Leaders while steering and tuning the educational activity will be under the supervision of the Training Coordinator (TRC) a new role credited by the new Run Coordination.

“I think there was an unexpressed demand for more attention to the formation of highly qualified shifters. Since ALICE is a very complex experiment, one cannot demand professional grade performance without giving at least an appropriate background. So the reform of the trainings is a real change and in our view is of outmost importance. We shall of course implement more and more automatic procedures, however the day when our operations will be entirely based only on automatic procedures is still very far and until then we shall make our the best effort to produce the best shifters. I asked the former Run Coordinators to participate in this effort and I was very lucky as they both accepted this extra work-load.  Gilda will run the new Shift Leader course while Chilo who accepted to be the first ALICE Training Coordinator.

A final remark should be made concerning the training shifts. In the new scheme, one has to become eligible for a certain shift role by attending the course and passing the test. Once the educational part is cleared, three days of training will be requested and it will be mandatory for the trainee to operate on the third day under the guidance of the trainer. The Shift Leader will make sure this knowledge transfer will happen at its best. The trainee names and photos will also appear with those of the respective trainers on the new SMS large screen in the ACR.

Did you book your shift? Learn more about the new sms system

All the changes foreseen in the new shift framework will be implemented in a brand new ALICE Shift Management System (SMS) that will be based on the GLANCE platform.

Another important feature will be the interface between the new SMS and the ALICE database (which was also recently migrated under GLANCE). In fact in the past, the credit accounting for the running periods had to be done by hand by the Run Coordinator. Federico explains: “This means that using the old SMS in practice I would not be able to produce crediting reports in a regular fashion since the manual operation is extremely time consuming, as Gilda knows very well. In the new portal all this will be automated, making easier for RC to keep track of things and produce regular reports for the collaboration. We are currently finalizing a document with the full specifications for the new systems”

When do we start?

We plan to restart the operations by the second half of next year. The first phase will be the re-commissioning of the online systems. After that we will switch to the detector commissioning using cosmic rays until the restart of the LHC beam. This period will be extremely critical to establish the ALICE full operability and also to “commission” the new shift model so that when the LHC beams are back ALICE will be ready to roll.