2.1   IT North and South and interactions¶

IT North and South meet every week on Monday to discuss current and future work of the network.

2.2   IT South¶

The IT South group is divided in 3 main areas: Infrastructure, Networking and Helpdesk.

Infrastructure and Networking provide support and development to the production environment of Rubin Observatory.

Helpdesk support is provided by a contractor, covering both sites (base and summit).

To organize work, there are several scheduled meetings covering all the aspects of the IT operations in the South.

• Daily group standup: Everyday at the beginning of the day, each team member reports in Slack the planned work for the day.
• Weekly Devops standup: Videoconference/In person meeting happening 3 days per week to discuss infrastructure related work.
• Weekly Network standup: Videoconference/In person meeting happening 3 days per week to discuss network related work.
• Weekly IT group meeting: Videoconference/In person meeting to discuss all work of the week and upcoming work.
• Monthly tickets clean: Videoconference/In person meeting to review and close Jira tickets.

2.3   IT North¶

The IT North group (aka Tucson group) has three primary roles: deliver PMO provided services, Helpdesk, Tucson on-site and particular off-site meeting support.

PMO provide services include Atlassian Suite, Docushare, Teamwork Cloud & Magic Draw, PDM/Solidworks.

Helpdesk support is for all of the users of the various web based services, and particulare hardware support of end users and particular specialized groups.

Meeting support consists of assisting meeting owners with AV and technical inside the Tucson building, general contractor and technical contact for PMO sponsored meetings or reviews, or technical advisor for meetings sponsored by other institutions.

Under a steady state, our group uses rubinobs-it-tucson email, Slack, and Jira’s comments and work log to keep informed. Our group is physically located in a single office and can meet regularlary when necessary. We also meet for one-on-ones related to training or particular questions or issues.

3.1   Base Data Center¶

The Rubin Observatory Base Datacenter is located in La Serena, Chile, in a shared location with other AURA programs. However, Rubin Observatory maintains full independence of administration and design of its cabinets. The logical design of the cabinets is based on top-of-rack switching, meaning that devices connect to a switch located at the top of each rack, facing to the back to facilitate its cabling. To avoid lengthy slacks of network cables, they are adjusted to fit the required length. All network cables are re-certified for cat6 after adjusting. The building was built on top of a seismic base isolation and features:

• Keyless access control
• 39 redundant circuits connected to two UPS
• 11 simple circuits connected to a single UPS
• Climate control system with a power of 235kw-
• Two UPS of 400kva each.

Network infrastructure deployment can be reviewed in the following links:

• La Serena Datacenter Fiber/Copper Deployment
• La Serena Building Fiber/Copper Deployment

3.2   Summit Data Center¶

Summit Data Center is located in Cerro Pachon, Chile. The racks are designed with top-of-rack switching and network cables are adjusted and re-certified to the necessary length. The Data Center has the following features:

• Keyless access
• 20 non-redundant circuits
• 1 UPS of 750kva
• Generator of 800kva, with a capacity of 1000 liters plus 32000 liters in an external pump.

Network infrastructure deployment can be reviewed in the following links:

• Summit Building Fiber/Copper Deployment
• Summit Outside of the Building Fiber/Copper Deployment

3.3   Tucson Lab(s)¶

Tucson lab(s) should resemble Chile as much as possible in terms of network and deployment.

A Kubernetes cluster is deployed as a testing stand with the same deployment done in Chile. This allows users to test tools under similar conditions of what can be found in the clusters deployed at Chile.

3.4   Tucson Data Center¶

The Tucson Data Center is maintained by NOIR Lab (formally NOAO) and is located in Tucson, AZ in the main building. RubinObs PMO services reside within two racks. PMO has a an empty rack there are racks for 3.3 Tucson Lab(s).

Power, networking, and physical security are provided by NOIR Lab.

5.1   Networking¶

To accommodate the demanding requirements of the computing and storage deployments of Rubin Observatory, IT provides a highly available and scalable network design. The network is logically separated in Campus and Control, both networks have a backplane of redundant 100GB links, and maintain the concept of TOR to connect devices.

For details of the Network design please review Network Infrastructure High-Level Design

Wireless device are controlled from a central location and connected to an identity management system to authorize users and route them to their correct network segment.

For details of the Wireless infrastructure design please review Wi-Fi Infrastructure High-Level Design

La Serena building has mostly Rubin Observatory Access Points, hence there is integration with NOIRLabs networks. Users can log in to the corresponding SSID of their program, authenticate against their backends, and use their gateways.

For details of the WiFi integration please review Wireless Integration with NOIRLabs

5.2   Desktop Support¶

IT provides desktop support to Rubin Observatory staff for all hardware and software requests. Support is provided in a timely and efficient manner. Helpdesk agents are responsible for deploying customized versions of desktop operating systems from a central repository. They are capable of installing additional software without major interruptions to the user.

Helpdesk agents are also responsible of keeping the inventory system up to date, creating documentation, and contributing to the creation of new policies.

The Helpdesk project in Jira, allows to assign tickets to IT north or south, depending on the requirement.

To review how to file tickets please visit [link to technote]

IT also provides remote support, which has been used during the Pandemic of 2020. To review details of remote work please visit [link to technote]

For more details about Helpdesk please review [link to tech doc]

5.3   Monitoring¶

IT provides monitoring metrics for the network, servers, services, and environmental variables of the Rubin Observatory datacenters. Metrics are collected by different sources such as app collectors and SNMP queries. The visualization of these metrics are managed by an open-source dashboard, which contains sub-dashboards with groups of servers. The access of the sub-dashboards is delegated to groups of users with common monitoring needs. The monitoring platform is deployed in a reliable, maintainable, and repeatable platform, to guarantee its operation in a stable state. The platform trigger notifications of outages and thresholds, that are managed by a central alerting system forwarding them to several Rubin’s communication channels.

For details of the platform, please review https://ittn-007.lsst.io/

5.4   Infrastructure¶

IT manages infrastructure as code when possible. It is operated in a continuous delivery and continuous integration pipeline. Systems are orchestrated from a central platform, allowing a rapidly scalable infrastructure and minimizing human errors by automating deployments. Rubin Observatory software, when possible, is fully integrated into this platform.

Production environment consists of virtual machines and containers running in physical clusters. Testing and development infrastructure has local resources but also leverages cloud computing resources.

The platform utilized to deploy infrastructure includes:

• Infrastructure automation - Puppet
• Orchestration - Kubernetes
• Automated user provisioning - FreeIPA

The objective is to provide highly available systems and rapidly scalable platforms.

The project in general favors open source software but uses licensed software when is the best option.

For details about the deployment platform please review https://ittn-002.lsst.io/

For details of the Puppet standards please review https://ittn-005.lsst.io/

5.5   Authentication and Authorization¶

Rubin Observatory maintains 2 types of accounts, organization accounts (typically email) and production environment accounts. The user is the same but the system managing the accounts are separated. IT provides 2FA and sudo escalation upon approval. The administration of the accounting is done in a centralized platform, servers or services register to this platform after deployment. To streamline access to resources and to promote self-service, the granting of access is delegated to a responsible or point of contact of such servers or services. IT is always capable to provide access but the approval must always be done by the responsible of the server or service.

For details of the accounting implementation please review https://ittn-010.lsst.io/

5.6   Storage¶

Rubin IT has designed a storage platform with the following features:

• Highly available
• High performance
• Scalable

Storage has been designed in several tiers to provide different level of services.

5.7   Cybersecurity¶

Rubin IT secures and protect technological resources of the Rubin Observatory, using industry standards, best practices, and cybersecurity policies. The focus of IT efforts is to keep confidentiality and integrity of the information, but also to increase the cybersecurity awareness of the users by providing training and policies.

IT enforces policies and keep systems up to date to strengthen the security of data and systems.

5.8   Backup¶

IT provides data and disaster recovery services to the production environment of Rubin Observatory. The backup platform is capable to recover historical configuration data from up to 6 months. Scientific data is not included in this backup platform.

Historical data is stored in virtual library tapes, there’s also a smaller staging area to hold backups for a few days before they are sent to historical archiving. The staging area allows the quick recovery of files due to bugs, crashings of systems, or human errors.

For details of the backup platform please review [link to technote]

5.9   Video Conference¶

Rubin IT offers video conference support in Rubin’s facilities. The system is currently independent from NoirLab platform, but will integrate as Rubin enters to operations.

At the moment, the video conferencing system is based on Blue Jeans.

5.10   Long Haul Network¶

The LHN is a dedicated network to transport science data to the USDF. For details of it please review Rubin Observatory Network Design.

Rubin IT provides support to execute tests during the construction phase of the Observatory, and also collaborates with other members of the NET-LHN team to troubleshoot problems detected in the network path to the USDF.