CALIFORNIA CITY — California City’s computers at City Hall were hit with a ransomware attack last month which shut down many functions, including billing and email.
Although city staff became aware of the problem June 2, investigation by the firm hired to fix it uncovered that the attack on the city servers actually dated to May 27, City Manager Anna Linn said during a special City Council meeting Friday.
She reported email service had been restored Friday and preparations were underway in restoring the software used across City Hall to enable other functions.
With the systems down, the regular City Council meeting scheduled for June 8 was canceled, and a special meeting was called for Friday.
“It was a very sophisticated cyberware (attack) and it was deliberate,” Linn said. “This has been a true crisis.”
It is estimated the repairs from the attack will amount to as much as $30,000 when completed, contractor Diamond IT said.
That estimated cost is based on a discount granted with a long-term contract with Diamond IT to provide information services. Without that contract, the cost will increase.
The Council approved the emergency spending Friday on a 4-0 vote, with Councilmember Jim Creighton absent, but debated whether to award a long-term contract with the firm to set up security measures and manage the city’s information technology systems.
Initially, staff believed the problem was that the system had finally failed, and that the emergency repairs would take a day or two and not cost more than about $2,000. Linn said she received a consensus from three of the five council members via text message to proceed with the emergency work. Mayor Jeanie O’Laughlin and Councilmember Karen Macedonio did not approve the expenditure at the time.
However, the damage was revealed to be a cyberattack and far more extensive than originally believed.
“We didn’t even realize how bad it was,” Linn said.
O’Laughlin and Macedonio questioned why Linn had not requested authorization from the Council for the additional expenses once the magnitude of the problem was revealed.
“I am right now,” Linn said. “I handled an emergency. What would you have done?”
O’Laughlin said she didn’t have an issue with the emergency work by Diamond IT, but was opposed to awarding a long-term contract with the firm for services, as requested by staff at the same time Friday, without a formal bidding process.
Macedonio also questioned how the city could award such a contract without seeking competing bids.
City officials have known for quite some time that the computer systems at City Hall were in desperate need of an overhaul, a task that had been pushed off due to budget constraints, according to the staff report.
However, the COVID-19 pandemic exposed the systems’ lack of capacity and capability. The recent cyberattack also exposed vulnerabilities in the systems’ security.
City Clerk Denise Hilliker said the city has had a history of “kicking the can down the road” when it came to needed upgrades to the city’s information systems, which ultimately led to the crippling cyber attack.
“Am I surprised (the attack occurred)? Absolutely not,” she said.
City staff engaged Bakersfield-based Diamond IT several months ago to audit the city’s systems in order to prepare a proposal for citywide information technology services. The company was highly recommended by the cities of Tehachapi, Wasco, Arvin, Delano and Shafter, Linn said.
Because they were already familiar with the city’s systems and their weaknesses, city staff engaged Diamond IT for the emergency repair work.
It was originally intended to include a contract for information technology services in the fiscal year 2021-22 budget, which begins on July 1.
Due to the emergency situation and to prevent any further attacks, staff requested approval for a three-year contract with Diamond IT to begin immediately.
The city at this point is still at risk of subsequent attacks without adding security measures, Diamond IT Vice President and General Manager Michael Leftwich said.
“You need to put strategy and services in place to protect you today,” he said. “This is not if. This will happen again, because your back door is wide open. They’re going to come back and at this point I can’t do anything to stop it.”
The services amount to $248,000 in the first year, which has setup costs, and includes services for City Hall, the police and fire departments and public works. The cost is $211,560 for the remaining two years.
It includes support and security training for users, searches of the dark web for compromised passwords and emails, and local and off-site backup systems.
Council balked at approving the long-term contract with Diamond IT without first undergoing a formal bidding process and seeking competing bids.
“We need to have processes in place. We need to have (request for proposals) and we need to have bids. That protects all of us,” O’Laughlin said.
Under the municipal code, a formal bidding process is not required for professional services contracts, or in emergency situations, City Attorney Baron Bettenhausen said.
Former Councilmember Ron Smith agreed that the contract should be bid to seek competing proposals and protect taxpayers’ money.
Given staffing constraints and the ongoing budget development, it will be at least mid-July before a request for proposals could be prepared, Linn said.
Bettenhausen cautioned the Council that personal information is at risk without proper security measures in place, leading way to privacy issues.
Along these lines, Councilmember Kelly Kulikoff said the city was opening itself up to potentially costly litigation and liability if it did not act to fix what are now known security risks.
“(The contract) is a low cost compared to the potential damage we’re going to create in the future by not addressing this today,” he said.
The Council ultimately voted 3-1 to approve the proposed three-year contract with Diamond IT, with Mayor Pro Tem Nick Lessenevitch dissenting.