Challenge: As COVID spread rampantly, the restriction of movement caused a lot of challenges in managing the core vital operations for Ashok Leyland. Running around during a lockdown, talking to state officials, and getting permissions is a hassle. But what do you do when you are the only person at work and your data center temperature hits critical levels?
Story:
The first wave of the pandemic was a difficult time for all of us. With restrictions imposed on every aspect of human life, the ground situation was critical. People faced problems even getting home from work. While most companies have shifted to a remote workforce, my job isn’t one of them.
The first lockdown was a shock for all of us. But as time went on, we’ve gotten used to the situation. Working on-site, the fear of COVID is always lurking around us.
As a SPOC of the Hosur branch, I have to regularly interact with staff, colleagues, vendors, and even state authorities from both Karnataka and Tamil Nadu. My work covers various aspects of capacity management, connectivity, disaster recovery, resource/serviceability, etc. Unlike other IT infrastructure needs, mission-critical facilities, like data centers, cannot be relocated into employee’s home offices.
One of the first challenges I faced was under the Disaster Recover cycle. A UPS failure led to widespread power disruption and a standstill of several major services. As the SPOC, it is my job to arrange for vendors and provide the right support for business continuity. But we weren’t in a typical scenario.
During the peak of the first wave, lockdown restrictions at the Bangalore border made it difficult for us to obtain timely support. Our OEM could not visit the site due to the restrictions imposed by the government and local police department. It was up to me to resolve the situation quickly. I had to talk to the local authorities of both states, answer a barrage of questions, and repeatedly explain the critical nature of our business. In the end, they understood the nature of our work and let us travel to and fro between the borders. I quickly arranged for a company vehicle to help the vendor visit on-site, book a hotel room to ensure their safety, and also arrange for their travel back – all under 2 hours! It was not an easy feat. Apart from adhering to social distancing and other COVID protocols, I also had to take care of all COVID-related documentation (ESA, safety protocol) on behalf of the vendor. Fortunately, we were able to restore power soon and get our operations back online.
I always say to others – one must be ready to face any situation with a calm head. Take a step back, calmly assess the situation and arrive at a resolution. If you hurry or panic during these times, it gets difficult to arrive at a solution.
Along the way, the pandemic continued to throw curve balls our way. As we heaved a sigh of relief from one problem resolved, we were faced with another operational challenge with our DC cooling units. This time the Tamil Nadu Electricity Board (TNEB) power supply failed and we had to restore the power backup with our diesel generators. To add to the problem, the temperature sensor unit of the master cooling unit was malfunctioning, leading to overheating of the data center. Before temperatures hit critical limits, I contacted the OEM vendor to understand the resolution steps as this was a new system. Unfortunately, even our OEM was unsure if the new functionality would work. As the temperature continued to rise, I did the only thing feasible at the time to avoid the overall system failure. I shut down and disconnected the master cooling unit from other units. Once the master unit is shut down, the other cooling units operated as standalone units – bringing down the room temperature. But my work was not done yet. After again dealing with the police department on either side of the border, I traveled between our OEM vendor site in Bangalore and the office at Hosur to procure a new master cooling unit.
I am proud of how I’ve been able to think on my feet and resolve issues quickly for my company.