It is our own Remote Operation Center (ROC) Dubna for the NOvA experiment, one of the 18th ROCs over the World.
The ROC-Dubna at Joint Institute for Nuclear Research (Dubna, Russia) supports the NOvA experiment located approximately 8000 km away in Fermilab (Batavia, Illinois, USA) and Ash River (Minnesota, USA). The ROC allows Russian physicists to operate the NOvA detector and monitor the NuMI neutrino beam complex. ROC-Dubna for NOvA was the first fully operational ROC outside USA.
The NOvA experiment [ http://www-nova.fnal.gov ] is one of the new concept experiment in particle physics with remote control of the detectors' operations. This idea came from LHC experiments' era to control large detectors from different scientific research centers and institutes. While each of those experiments (ATLAS, CMS, ALICE, LHCb) controls and operates just one detector from over the World, NOvA has two distant detectors as well a beam machine which the goal is to control and operate all the remote systems from one place -- Remote Operation Center (ROC). The Main Remote Operation Center ROC-West located inside Wilson Hall (FNAL).
ROC and its crew
The NOvA detectors are operation since 2013 from ROC-West at FNAL. On this stage NOvA collaboration discussed and developed an approach to run remote operation centers out of FNAL. It was first ROC-UMN at Minnesota University started to control and operate from non-Fermilab area at November, 2014. ROC-Dubna (first ROC outside USA) started to operate at October, 2015.
Currently eighteen ROCs are operating for the NOvA 24/7 days a week over the years (at least till 2020, project plan). Every ROC requires qualified specialists to support its normal functionality. Our Local ROC-Dubna experts are:
1) Oleg Samoylov, Super-liaison, Software expert
2) Alexander Antoshkin, ROC-Dubna liaison, Hardware and software expert
3) Andrey Sheshukov, Software expert
4) Nikolay Anfimov, Hardware expert
5) Chris Kullenberg, Super-Shifter
6) Nikita Balashov, Software and IT-support
7) Andrey Dolbilov, Internet and IT emergency
ROCs operation process
The NOvA experiment is operating at least till 2020 according to the project plan. All the ROCs together ensure continuous processing detectors' systems on both sites as well monitor beam status. Three run coordinators changing every two weeks communicate with Shifters, ROC contacts, Systems' experts, FNAL and Ash river crews. That allows to work together experts and shifters, to monitor beam, detector's electronics, file transfer -- to take good quality DATA.
NOvA data storage and transfer software based on Linux-nodes collect information from NOvA DAQ system. ROCs connect to control and operation nodes on both Far Detector/Near Detector sites via VNC-tunneling under secure protocols. Basic idea is that a VNC server transfer a VNC session to many VNC viewers at ROCs with control. In total there are 5 active Scientific Linux based VNC-sessions are connected directly to Near/Far Detectors' nodes at FNAL and through GateWays to another World. System includes 1 Linux node for Web-monitoring of the operated systems (Beam, ND/FD Cameras, Data transfer control, Ganglia, Nearline) and 1 Windows node for communication (NOvA electronic logbook, latest version of expert contact and Bulletin board, Slack-chat, Skype, Zoom).
ROC-Dubna at JINR has infrastructure that was developed to 8 hours (a shift period) continues work (stable internet, international land-line, kitchen, this is also public JINR area). Computing monitoring system on Zabbix controls local Linux-nodes, internet connection, server conditions and notifies JINR experts in case of troubles.
Hardware and software setup
We decided to move all PCs, storages, UPSs, monitoring based on Linux-nodes and hardware from backup Internet connections to special server room with excellent and stable cooling system. On the top floor we have only monitors with keyboards and mice.
Backup Internet connections are extremely important because we have to be online and monitor the NOvA detectors working state during all the shift without any drops. For these purposes in addition to main JINR network we have made two backup connections -- mobile (4G) and sky-fi ones.
All of these features allow ROC-Dubna to operate as stable as possible and to be ready for any hardware and software issues.
Advantages for JINR
1) ROC-Dubna amends people time while shifting for both JINR and FNAL side. We can control Detectors during day time for the Night shifts.
2) ROC-Dubna saves budget about 20k$ per year to do research on NOvA.
3) ROC-Dubna is a public place for excursions visiting by Scholars, Teachers, Students, Journalist and other JINR guests.