Kaseya, an IT solutions developer for MSPs and enterprise clients, announced that it had become the victim of a cyberattack on July 2, over the American Independence Day weekend.
It appears that attackers have carried out a supply chain ransomware attack by leveraging a vulnerability in Kaseya’s VSA software against multiple managed service providers (MSP) — and their customers.
Also: Kaseya issues patch for on-premise customers, SaaS rollout underway
According to Kaseya CEO Fred Voccola, less than 0.1% of the company’s customers were embroiled in the breach — but as their clientele includes MSPs, this means that smaller businesses have also been caught up in the incident.
Present estimates suggest that 800 to 1500 small to medium-sized companies may have experienced a ransomware compromise through their MSP.
The attack is reminiscent of the SolarWinds security fiasco, in which attackers managed to compromise the vendor’s software to push a malicious update to thousands of customers. However, we are yet to find out just how widespread Kaseya’s ransomware incident will prove to be.
Here is everything we know so far. ZDNet will update this primer as we learn more.
What is Kaseya?
Kaseya’s international headquarters is in Dublin, Ireland, and the company has a US headquarters in Miami, Florida. The vendor maintains a presence in 10 countries.
Kaseya provides IT solutions including VSA, a unified remote-monitoring and management tool for handling networks and endpoints. In addition, the company provides compliance systems, service desks, and a professional services automation platform.
The firm’s software is designed with enterprises and managed service providers (MSPs) in mind, and Kaseya says that over 40,000 organizations worldwide use at least one Kaseya software solution. As a provider of technology to MSPs, which serve other companies, Kaseya is central to a wider software supply chain.
What happened?
On July 2 at 2:00 PM EDT, as previously reported by ZDNet, Kaseya CEO Fred Voccola announced “a potential attack against the VSA that has been limited to a small number of on-premise customers.”
At the same time, out of an abundance of caution, Voccola urged clients to immediately shut down their VSA servers.
“It’s critical that you do this immediately because one of the first things the attacker does is shut off administrative access to the VSA,” the executive said.
Customers were notified of the breach via email, phone, and online notices.
As Kaseya’s Incident Response team investigated, the vendor also decided to proactively shut down its SaaS servers and pull its data centers offline.
By July 4, the company had revised its thoughts on the severity of the incident, calling itself the “victim of a sophisticated cyberattack.”
Cyber forensics experts from FireEye’s Mandiant team, alongside other security companies, have been pulled in to assist.
“Our security, support, R&D, communications, and customer teams continue to work around the clock in all geographies to resolve the issue and restore our customers to service,” Kaseya said, adding that more time is needed before its data centers are brought back online.
Once the SaaS servers are operational, Kaseya will publish a schedule for distributing a security patch to on-prem clients.
In a July 5 update, Kaseya said that a fix has been developed and would first be deployed to SaaS environments, once testing and validation checks are complete.
“We are developing the new patch for on-premises clients in parallel with the SaaS Data Center restoration,” the company said. “We are deploying in SaaS first as we control every aspect of that environment. Once that has begun, we will publish the schedule for distributing the patch for on-premises customers.”
The ransomware attack, explained
The FBI described the incident succinctly: a “supply chain ransomware attack leveraging a vulnerability in Kaseya VSA software against multiple MSPs and their customers.”
Huntress (1,2) has tracked 30 MSPs involved in the breach and believes with “high confidence” that the attack was triggered via an authentication bypass vulnerability in the Kaseya VSA web interface.
According to the cybersecurity firm, this allowed the attackers to circumvent authentication controls, gain an authenticated session, upload a malicious payload, and execute commands via SQL injection, achieving code execution in the process.
Kyle Hanslovan, CEO and co-founder of Huntress, told attendees of a webinar discussing the technical aspects of the attack on July 6 that the threat actors responsible were “crazy efficient.”
“There is no proof that the threat actors had any idea of how many businesses they targeted through VSA,” Hanslovan commented, adding that the incident seemed to be shaped more due to a “race against time.”
“Some of the functionality of a VSA Server is the deployment of software and automation of IT tasks,” Sophos noted. “As such, it has a high level of trust on customer devices. By infiltrating the VSA Server, any attached client will perform whatever task the VSA Server requests without question. This is likely one of the reasons why Kaseya was targeted.”
The vendor has also provided an in-depth technical analysis of the attack.
Security expert Kevin Beaumont said that ransomware was pushed via an automated, fake, and malicious software update using Kaseya VSA dubbed “Kaseya VSA Agent Hot-fix”.
“This fake update is then deployed across the estate — including on MSP client customers’ systems — as it [is] a fake management agent update,” Beaumont commented. “This management agent update is actually REvil ransomware. To be clear, this means organizations that are not Kaseya’s customers were still encrypted.”
With a tip from RiskIQ, Huntress is also investigating an AWS IP address that may have been used as a launch point for the attack.
On July 5, Kaseya released an overview of the attack, which began on July 2 with reports of ransomware deployment on endpoints.
“In light of these reports, the executive team convened and made the decision to take two steps to try to prevent the spread of any malware: we sent notifications to on-premises customers to shut off their VSA servers and we shut down our VSA SaaS infrastructure,” the company says.
According to the firm, zero-day vulnerabilities were exploited by the attackers to trigger a bypass authentication and for code execution, allowing them to infect endpoints with ransomware. However, Kaseya emphasizes that there is no evidence of the VSA codebase being “maliciously modified”.
Wietse Boonstra, a Dutch Institute for Vulnerability Disclosure (DIVD) researcher, previously identified a number of vulnerabilities, tracked as CVE-2021-30116, which were used in the ransomware attacks. They were reported under a Coordinated Vulnerability Disclosure pact.
“Once Kaseya was aware of our reported vulnerabilities, we have been in constant contact and cooperation with them. When items in our report were unclear, they asked the right questions,” DIVD says. “Also, partial patches were shared with us to validate their effectiveness. During the entire process, Kaseya has shown that they were willing to put in the maximum effort and initiative into this case both to get this issue fixed and their customers patched. ”
[embedded content]
Who has been impacted?
Over the weekend, Kaseya said that SaaS customers were “never at risk” and current estimates suggest that fewer than 40 on-prem clients worldwide have been affected.
However, it should be noted that while a small number of Kaseya clients may have been directly infected, as MSPs, SMB customers further down the chain relying on these services could be impacted in their turn.
According to reports, 800 Coop supermarket chain stores in Sweden had to temporarily close as they were unable to open their cash registers.
Huntress said in a Reddit explainer that an estimated 1,000 companies have had servers and workstations encrypted. The vendor added that it is reasonable to suggest “thousands of small businesses” may have been impacted.
“This is one of the farthest-reaching criminal ransomware attacks that Sophos has ever seen,” commented Ross McKerchar, Sophos VP. “At this time, our evidence shows that more than 70 managed service providers were impacted, resulting in more than 350 further impacted organizations. We expect the full scope of victim organizations to be higher than what’s being reported by any individual security company.”
On July 5, Kaseya revised previous estimates to “fewer than 60” customers, adding that “we understand the total impact thus far has been to fewer than 1,500 downstream businesses.”
Now, on July 6, the estimate is between 50 direct customers, and between 800 and 1,500 businesses down the chain.
When it comes to SaaS environments, Kaseya says, “We have not found evidence that any of our SaaS customers were compromised.”
In a press release dated July 6, Kaseya has insisted that “while impacting approximately 50 of Kaseya’s customers, this attack was never a threat nor had any impact to critical infrastructure.”
The number of vulnerable Kaseya servers online, visible, and open to attackers dropped by 96% from roughly 1,500 on July 2 to 60 on July 8, according to Palo Alto Networks.
Kaseya CEO Fred Voccola said that the attack, “for the very small number of people who have been breached, it totally sucks.”
“We are two days after this event,” Voccola commented. “We have about 150 people that have probably slept a grand total of four hours in the last two days, literally, and that’ll continue until everything is as perfect as can be.”
Less than 0.1% of the company’s customers experienced a breach.
“Unfortunately, this happened, and it happens,” the executive added. “Doesn’t make it okay. It just means it’s the way the world we live in is today.”
What is ransomware?
Ransomware is a type of malware that specializes in the encryption of files and drives.
In what has become one of the most severe and serious security problems modern businesses now face, ransomware is used by threat actors worldwide to hijack systems and disrupt operations.
Once a victim’s system or network has been encrypted, cyber criminals will place a ransom note on the system, demanding payment in return for a decryption key (which may, or may not, work).
Today’s ransomware operators may be part of Ransomware-as-a-Service (RaaS), when they ‘subscribe’ to access and use a particular type of ransomware. Another emerging trend is double extortion, in which a victim will have their information stolen during a ransomware raid.
If they refuse to pay up, they may then face the prospect of their data being sold or published online.
Common and well-known ransomware families include REvil, Locky, WannaCry, Gandcrab, Cerber, NotPetya, Maze, and Darkside.
Read on: What is ransomware? Everything you need to know about one of the biggest menaces on the web
See also:
Who is responsible?
The cyberattack has been attributed to the REvil/Sodinikibi ransomware group, which has claimed responsibility on its Dark Web leak site, “Happy Blog.”
In an update over the weekend, the operators, believed to have ties to Russia, claimed that more than “a million” systems have been infected.
REvil has offered a decryption key, allegedly universal and, therefore, able to unlock all encrypted systems, for the ‘bargain’ price of $70 million in the bitcoin (BTC) cryptocurrency.
REvil has been previously linked to ransomware attacks against companies, including JBS, Travelex, and Acer.
What are the ransomware payment terms?
The ransomware note claims that files are “encrypted, and currently unavailable.” A file extension .csruj has reportedly been used. Operators are demanding payment in return for a decryption key and one ‘freebie’ file decryption is also on the table to prove the decryption key works.
The operators add (spelling unchanged):
“Its just a business. We absolutely do not care about you and your deals, except getting benefits. If we do not do our work and liabilities – nobody will not cooperate with us. Its not in our interests. If you will not cooperate with our service –for us, its does not matter. But you will lose your time and data, cause just we have the private key. In practice – time is much more valuable than money.”
Sophos malware analyst Mark Loman shared a screenshot on Twitter of a ransomware note planted on an infected endpoint demanding $44,999.
John Hammond, senior security researcher at Huntress, told ZDNet that the company has already seen ransom demands of up to $5 million.
Kevin Beaumont says that, unfortunately, he has observed victims “sadly negotiating” with the ransomware’s operators.
Fabian Wosar, CTO of Emsisoft, has also explained in a Twitter thread why using a key obtained by a single organization paying up is unlikely to be a viable path for unlocking all victims.
“REvil absolutely has the capability of decrypting only a single victim without these purchased decryption tools being applicable for other victims hit by the same campaign public key,” the security expert noted.
CNBC reports that the universal ransom demand has been reduced to $50 million in private conversations. However, as of July 7, the public demand for $70 million on the threat group’s leak site remains unchanged.
What are the reactions so far?
At the time of the breach, Kaseya notified law enforcement and cybersecurity agencies, including the Federal Bureau of Investigation (FBI) and US Cybersecurity and Infrastructure Security Agency (CISA).
The FBI and CISA have released a joint statement on the security incident and are urging customers to run a tool provided by Kaseya to determine the risk of exploit, and to both enable and enforce multi-factor authentication (MFA) on enterprise accounts, wherever possible.
Kaseya has been holding meetings with the FBI and CISA “to discuss systems and network hardening requirements prior to service restoration for both SaaS and on-premises customers.”
The White House is asking organizations to inform the Internet Crime Complaint Center (IC3) if they suspect they have been compromised.
On Saturday, US President Biden said he has directed federal intelligence agencies to investigate.
“Targeting [an] MSP platform (that is managing many customers at once) was very well thought and planned,” Amit Bareket, CEO of Perimeter 81, told ZDNet. “What’s unique is that hackers are becoming more strategic and targeting platforms that will filtrate down to many companies with one shot. RMMs [remote monitoring and management] are basically keys to many many companies, which amount to the kingdom for bad actors.”
The White House has attempted to strengthen its stance on cybercrime in light of this attack, warning Russian President Vladimir Putin that unless he deals with the problem in his own backyard, “we will take action or reserve the right to take action on our own.”
Are there any recovery plans?
As of July 4, Kaseya says the company has now moved on from a root cause analysis of the attack to recovery and patch plans, consisting of:
- Communication of our phased recovery plan with SaaS first followed by on-premises customers.
- Kaseya will be publishing a summary of the attack and what we have done to mitigate it.
- Some lightly-used legacy VSA functionality will be removed as part of this release out of an abundance of caution. A specific list of the functionality and its impact on VSA capabilities will be outlined in the release notes.
- There will be new security measures implemented including enhanced security monitoring of our SaaS servers by FireEye and enablement of enhanced WAF capabilities.
- We have successfully completed an external Vulnerability Scan, checked our SaaS Databases for Indicators of Compromise, and have had external security experts review our code to ensure a successful service restart.
Data centers starting with the EU will be restored, followed by the UK, APAC, and then North American systems.
By late evening on July 5, Kaseya said a patch has been developed and it is the firm’s intention to bring back VSA with “staged functionality” to hasten the process. The company explained:
- The first release will prevent access to functionality used by a very small fraction of our user base, including:
- Classic Ticketing
- Classic Remote Control (not LiveConnect).
- User Portal
Kaseya has now published an updated timeline for its restoration efforts, starting with the relaunch of SaaS servers, now set for July 6, 4:00 PM EDT and 7:00 PM EDT. Configuration changes to improve security will follow, including an on-premise patch, expected to land in 24 hours, or less, from the time SaaS servers come back online.
“We are focused on shrinking this time frame to the minimal possible — but if there are any issues found during the spin-up of SaaS, we want to fix them before bringing our on-premises customers up,” the firm says.
Additional security improvements include the creation of 24/7 SOCs for VSA, as well as a complimentary CDN with a web application firewall (WAF) for every VSA.
Update July 7: The timeline has not been met. Kaseya said that “an issue was discovered that has blocked the release” of the VSA SaaS rollout.
“We apologize for the delay and R&D and operations are continuing to work around the clock to resolve this issue and restore service,” Kaseya commented.
In a service update, the vendor said it has been unable to resolve the problem.
“The R&D and operations teams worked through the night and will continue to work until we have unblocked the release,” Kaseya added.
July 7, 12 pm EDT:
Kaseya hopes to resolve the SaaS systems rollout no later than the evening of Thursday, July 8. A playbook is currently being written up, due to be published today, which will provide guidelines for impacted businesses to deploy the upcoming on-prem VSA patch.
Current recovery status
As of July 8, Kaseya has published two run books, “VSA SaaS Startup Guide,” and “On Premises VSA Startup Readiness Guide,” to assist clients in preparing for a return to service and patch deployment.
Recovery, however, is taking longer than initially expected.
“We are in the process of resetting the timelines for VSA SaaS and VSA On-Premises deployment,” the company says. “We apologize for the delay and changes to the plans as we work through this fluid situation.”
In a second video message recorded by the firm’s CEO, Voccola said:
“The fact we had to take down VSA is very disappointing to me, it’s very disappointing to me personally. I feel like I’ve let this community down. I let my company down, our company let you down. [..] This is not BS, this is the reality.”
The new release time for VSA is Sunday, in the afternoon, Eastern Time, in order to also harden the software and bolster its security ahead of deployment.
July 12: Kaseya has now released a patch and is working with on-prem customers to deploy the security fix. Now, 100% of all SaaS customers are live, according to the company.
“Our support teams continue to work with VSA on-premises customers who have requested assistance with the patch,” Kaseya added.
What can customers do?
Kaseya has released a tool, including Indicators of Compromise (IoC), which can be downloaded via Box. There are two PowerShell scripts for use: one on a VSA server, and the other has been designed for endpoint scanning.
The self-assessment scripts should be used in offline mode. They were updated on July 5 to also scan for data encryption and REvil’s ransom note.
However, the scripts are only for potential exploit risk detection and are not security fixes. Kaseya will release patches as quickly as it can, but in the meantime, customers simply have to wait until Sunday.
Kaseya intends to bring customers back online on July 11, at 4 PM EDT.
“All on-premises VSA Servers should continue to remain offline until further instructions from Kaseya about when it is safe to restore operations,” the firm said. “A patch will be required to be installed prior to restarting the VSA.”
Cado Security has provided a GitHub repository for responders, including malware samples, IoCs, and Yara Rules.
Truesec CSIRT has also released a script on GitHub to identify and mitigate damage on infected systems.
Kaseya has also warned that scammers are trying to take advantage of the situation.
“Spammers are using the news about the Kaseya Incident to send out fake email notifications that appear to be Kaseya updates. These are phishing emails that may contain malicious links and/or attachments.
Do not click on any links or download any attachments claiming to be a Kaseya advisory.”
Are REvil still active?
After Biden made his stance clear to Putin on ransomware gangs, the REvil ransomware group’s leak site was seized and taken down by law enforcement.
The takedown included REvil’s payment site, public domain, helpdesk chat platform, and the negotiation portal.
While the intention was to secure some form of control over the group, it should be noted that ransomware operators often close down sites, rebrand, and regroup.
A side effect of the takedown is that the removal of negotiation and the possibility of purchasing a decryption key have left victims with unrecoverable systems. One victim who paid up for a decryption key — which ended up not working — is now out of pocket and unable to secure assistance from the cybercriminals.
A decryption key?
On July 22, Kaseya said that the company has managed to secure a decryption key. Obtained by a “third-party,” the decryption key has been tested successfully in victim environments — and the suggestion is that the decryption key may be universal.
The company is working with Emsisoft to reach customers still suffering due to locked systems and in need of a decryption key.
“We remain committed to ensuring the highest levels of safety for our customers and will continue to update here as more details become available,” Kaseya said. “Customers who have been impacted by the ransomware will be contacted by Kaseya representatives.”
