022 - Solar Winds

Intro

 

It's said that during the Trojan War, to end a 10 year long siege Odysseus convinced the Greeks to construct a giant wooden horse. After the construction though, the Greeks sailed away leaving the horse behind as an offering to Athena as recompense for the annihilation of her temple at Troy. Sinon, tells the trojans about the offering, and goads the trojans a bit by telling them the horse was built to be too big to fit into their walls. Only one priest, named Laocoon, and a handful of royalty question the offering but nevertheless the Trojans pulled the horse into their own city as a prize for their victory in the conflict.

 

What the Trojans didn't realize, however, was that inside that horse was a company of the Achaeans best soldiers, led by Odysseus himself. And that night, the Greeks would sneak out of the horse to open the gates for the remainder of the army which had sailed back, using the nightfall to their advantage. This would result in the destruction of Troy, and the end of the war.

 

This image of the trojan horse is something that's transcended the myths and history books and made it's way into many a lexicon. In fact, it's something we use pretty frequently in cyber security. Ever hear  of a Trojan virus? Well, same concept? Instead of endlessly attacking attacking from the front, what if you disguised yourself as something legitimate? What if you were something that was expected to be seen, and in fact you went out and bought yourself?

 

I'm John Kordis, and today, we're going back to the end of 2020, because I've got a story that would make Odysseus proud, even if it was the bane of the industry for a while. So come with me because today I'm going to tell you What the Shell happened in the Solar Winds hack, and why this was such a big deal. 

 

 

MUSIC

 

On a side note before we get started. If Odyseus and his crew made the first trojan horse and Laocoon and the others tried to warn the king but he went with it anyways. Does that make Laocoon and his crew the first anti-virus to be bypassed?

 

 

December 8 - FireEye

 

Okay, so this might be a little counter intuitive, but before we can talk about what the Solar Winds hack even is, I need to take you to another hack that was discovered in early December of 2020.

 

On Tuesday December 8th 2020, a company named FireEye announced that it had been breached and some of their own propietary tools had been taken by what they suspected to be a nation state level attack. FireEye itself being a cybersecurity had many different things that could be valuable from information on customers to tools in use in by the firm.

 

They weren't small time cyber either. These are seasoned veterans in the field, and people that really know what they're doing. You've probably actually heard me talk about them because many times FireEye is one of the first companies that the government will look toward to help in investigating breaches. At the time of their hack they were worth about 3.5 billion dollars. That's to say that this isn't some start up or small operation, this is a massive target to go after.

 

The tools that seemed to have been stolen included their own internal suite for red teaming. That's a tool set that's used for when they're testing the security of other companies that contract them to try and break in or replicate an attack against their firm. This immediately reminded a lot of people of the shadow brokers leak and how the Lazarus group immediately weaponized some of what came out of the publishing of the NSA based tools and exploits, so needless to say there was a bit of a hustle to assess the risk.

 

Now this suite wasn't quoted as having any zero-day or unknown vulnerabilities in use, but it was a rather powerful platform that made performing the testing more streamlined and easy to do. This was one of their secret sauces and now it was in the hands of a nation state? They were not pleased. And so, they began the process to investigate to see how exactly this could have happened. 

 

Well, to figure out how this happened FireEye would need to comb through logs and impacted systems to start to trace where things may have been exfiltrated to and how. It was a daunting ask, and would take many hours and analysts to begin to assess.

 

By the 13th of December here is what they had for the public. First they're tracking whoever did this using the name "UNC2452". Second, after an initial compromise they become very patient and use rather advanced techniques. This is done in order to avoid detection and evade any kind of tool that might try to restrict their access. We've talked about this before, a thorough and effective attack doesn't need to be quick, and often times it isn't. A true attacker can and will wait, gathering everything they can before risking any kind of alerting of their presence.

 

Next, FireEye told everyone that they are going to share common ways they can detect the use of their own platform to help defenders figure out if a malicious actor is using the FireEye red team platform.

 

And perhaps most prescient of all, they determined that initial vector point stemmed not from a phishing email or from a propietary system they owned. They determined that this hack? It came from a tool they'd bought, called Orion, made by a company named Solar Winds.

 

 

Let's use that to pivot and get to know Solar Winds a little bit. Solarwinds is an IT company that was started in 1999 by a former Walmart Executive and his brother. In the years since it's foundation it would move from Tulsa Oklahoma to Austin Texas on the physical front and into IT service management on the product side of it.

 

 

Solar Winds would grow to the point where it and its 1800 or so employees were acquired for 4.5 billion dollars in a private deal in 2015, before going back public in 2018. Another company that wasn't exactly small beans.  As of December 2020, it had 300,000 customers inclduing almost every single fortune 500 company. But, not every one of those would be impacted because what it was found out was that one tool was compromised, the tool used by FireEye and 33,000 other companies. A tool called Orion.

 

Orion was a platform that was simple enough. Essentially it offered a view into all the different vendors and systems in use for IT process management under one roof. It was a way to monitor at a high level and avoid needing to move from one place to enough to aggregate data.

 

What Fireeye determined happened was that Solar Winds was breached in a way that made it so the could apply malicious code to legitimate updates that were going out. Specifically, there was one key element that was called the Sunburst backdoor which would make use of a legitimately signed and validated component of the update to send code that communicated with the HTTP protocol back to attacker servers.

 

The code had a bit of a dormant period too. It wouldn't be applied and then immediately start performing attacks, it would wait about two weeks.

 

And think about this from any kind of investigative mindset. It doesn't need even need to be cyber related. If you notice a problem, any kind of problem, that wasn't there before the first question is often "Well what's different?". You would look for recent changes to the environment. So by waiting two weeks,  you're increasing how far back someone needs to go and the amount of potential changes that need to be sifted through.

 

After that dormancy period, the Sunburst plugin would execute jobs that could do anything from transfer files, do reconnaisance like system profiling, turn specific services on and off, or even run executable files. These are things that add up to the ability to do some serious damage to anything integrated with the platform.

 

I briefly mentioned it earlier but this threat group, UNC2452 was found to be using really advanced methods to disguise their traffic. According to FireEye the network traffic that would identify this was hidden within a legitimate protocol called the Orion Improvement Program, something that if anyone had a snippet of network traffic to look at might just appear as background noise and nothing to really bat an eye at.

 

And like I said, this was all placed in legitimately created updates. The updates themselves ran from March 2020 to May 2020.

 

And if you're at home sitting there thinking "Well it's okay, as long as they upgraded afterwards and the system would be fine". Sure, yeah the system would be fine and the update wouldn't have the problem anymore. But how often do you think a major threat actor is going to just use one way to keep access? UN2452 had several months where if you upgraded you could have been compromised and the attackers would have time to move in and establish other footholds in any system they could get their hands on. Not only that, but we're talking about this breach that was discovered at the start of December. I'm sure each and every one of these 33,000 companies that used the total have a great patch management program that would do testing and updates on a regular cadence right? Right?

 

If you couldn't pick up on the sarcasm there, so many places wouldn't do that. People lag frequently behind on updates whether it's because they aren't a priority or they don't have the manpower to do that and the other work. There's a lot of reasons to let your system just sit on whatever it's on, especially if it's working. After all, why risk downtime? That mentality is a big doorway for hackers or situations like this where those updates would probably stay online long after a clean version was rolled out to the public. 

 

And so we 're back to December 8th. After sifting through over 50,000 lines of code and analysis, the FireEye team has found out how their tool Kit was stolen. Next up? Well, I think they needed to talk with Solar Winds and make the public aware that this is happening.

 

December 13 - expanded out

 

And that's when on December 13, the @SolarWinds twitter account posted a tweet saying quote "SolarWinds asks all customers to upgrade immediately to Orion Platform Version 2020.2.1 HF 1 to address a security vulnerability." They also go on to link their own security advisory. If you're a little confused about that software version, it can be broken down by saying 2020 is the year the update occurred, it was major version 2, minor version 1, and HF 1 is hotfix 1. That kind of "hotfix" update is usually something small like removing or disabling the impacted component if it's not critical, or applying a workaround that might not fix the root problem but would stop the attack.

 

I don't have the exact day of advisory because those are updated as they go to provide information. But this is what they had to say regarding the attack as a part of the current iteration:

 

 This attack was a very sophisticated supply chain attack, which refers to a disruption in a standard process resulting in a compromised result with a goal of being able to attack subsequent users of the software. In this case, it appears that the code was intended to be used in a targeted way as its exploitation requires manual intervention. We’ve been advised that the nature of this attack indicates that it may have been conducted by an outside nation state, but SolarWinds has not verified the identity of the attacker

 

While SUNSPOT is the means by which the attackers injected the SUNBURST backdoor during the build process of the Orion Platform, TEARDROP and RAINDROP are reportedly malware loaders that could be deployed as secondary tools using the SUNBURST backdoor. SUNSPOT, TEARDROP, and RAINDROP are NOT new vulnerabilities within our products as some reports in the media have indicated, but instead, they are elements of the SUNBURST attack chain.

 

That last bit? That's exactly what I was talking about when I said they wouldn't stop at that initial access. TEARDROP and RAINDROP was the inevitable progression after the initial exploit to further compromise the victims and maintain a persistent access to the compromised network.

 

 

Solar Winds wasn't the only group issuing advisories on the 13th either.

 

The Cybersecurity and Infrastructure Security Agency, or CISA, issued their own alert regarding this. They echoed pretty much the same as the above, except they also included a more firm hand on how impacted government agencies need to respond. That included immediately disconnecting the Orion product from their network, blocking any and all traffic to devices with Orion software installed, and removing potentially compromised accounts completely.

 

 

Additionally, they asked any agencies to report by NOON on the following day EST if they saw any of a the indicators of compromise they'd listed.

 

This was an all hands on deck effort from every public agency because the government used solar winds extensively.

 

I remember this being a big moment in the field. The CISA response wasn't something we see very often, and honestly it was handled pretty alright all things considered. A swift call to action, with deadlines, those can sometimes be tough but this was warranted.

 

December 14 - Financial Impact

 

Those both came pretty late on Sunday that week, to let's hop to bed and wake up on Monday the 14th. This is the first day the markets are open since the public was notified and Solar Winds had some work to do.

 

Remember how I told you that SolarWinds went public again after the private deal? Well it was now time to let the share holders that "hey, something funky is going on and you need to be informed or legally we're screwed".

 

There's no form specifically for "someone poisoning the secret sauce" so SolarWinds settled for the next best thing which is an SEC form 8-K.

 

A form 8-K is something the companies must file when majors events that impact the firm occur that the shareholders should be made aware of.

 

It's a pretty easy form to find online, but it's not a fun read so I'll spare you the details. Some of the reasons you might file for a form 8-K are Bankruptcy, debt incurrence, a change to the fundamental business itself, corporate restructuring, ammendments to by laws, election results and more. There's a big umbrella that a lot of stuff can fall under here but anytime it's filed, it's usually for something important.

 

So on the 14th, the shareholders would find out directly that this was something big.

 

So for SolarWinds at this point it was time to keep on damage control. And that would lead into the next day, the 15th.

 

December 15 - Major victims

 

Now that this had been made public and people were clued in, it was partially on the industry to respond and get their stuff together. SolarWinds at this point estimated 18,000 companies downloaded impacted updates, that's 18,000 possible victims that need to do a full response to this, and however many more that will just verify.

 

The main stream press started reporting on it as well. After all this was a nation-state level attack and now details were starting to come forward that pointed the finger at Russia. The servers that were being communicated to, and the tactics involved pointed at APT 29 specifically. This isn't very surprising as we're already familiar with how Russia can do a chunk of damage with a hacking group.

 

This though, this wasn't a spam email. It remains to this day, one of the most well crafted cyber attacks known to have occurred.

 

It was mentioned before, but this is what's called a supply chain attack. If your victim represents a link on the chain, and you don't want to do damage, you could theoretically aim slightly higher up the chain. And then, when you do so, you might see that others are clipped to that link as well and it can ripple back down to those other areas.

 

Well, we've been talking about that big link which is Orion, but now let's list some of the links that were impacted downstream.

 

We know government agencies are a big one, after all CISA named them specifically as a victim. So on that list we've got the US Commerce and Treasury department, the Department of Homeland Security, the National Institute of Health, and the State department. Talk about big arms of the government being hit here. Want to do some damage to the financial arm of the government? Maybe snoop on the covid numbers and response in the Institute of Health? Hell, want to see what's going on in the executive branch?

 

The attack surface was being unveiled minute by minute and growing far and wide. Not only that but at this point they had only just found out that the malicious updates went as far back as March of 2020. So now we've got a bigger scope of impact on SolarWinds.

 

December 17 - 20 - More victims announced

 

Over the coming days more government victims would be announced, like the Energy Department, the National Nuclear Security Administration, and many of those fortune 500 companies as well.

 

December 20 - Trump v China and Russia?

 

The reason I'm building up all this stress on the governmental impact is because of how easily it crept into every facet of the government infrastructure. And yeah CISA responded well but you might notice that there's one government figure I haven't yet commented about because he was silent all the way up to December 19th.

 

After over a week of investigations concluding that this was in all likelihood as cyber attack performed by Russia, with much evidence to support that, President Donald Trump would take to twitter to address the hack publically for the first time. He said quote:

 

"The Cyber Hack is far greater in the Fake News Media than in actuality," Trump wrote. "I have been fully briefed and everything is well under control. Russia, Russia, Russia is the priority chant when anything happens because Lamestream is, for mostly financial reasons, petrified of...discussing the possibility that it may be China (it may!). There could also have been a hit on our ridiculous voting machines during the election, which is now obvious that I won big, making it an even more corrupted embarrassment for the USA."

 

There's a lot to unpack there even before it's completely derailed to his own personal opinion on an entirely separate topic. And to play it delicately I'm not going to say anything opinionated about that tweet. Only this fact that I want you to do with what you will.

 

A week of investigation by multiple private companies, CISA, and hundreds of experts in the field has occurred up to this point. Servers that have been previously tied to Russia were in use, tactics that have been noted with use of malware developed by Russian actors were in play. And still after being briefed, with CISA (an organization that was build under his own administration) presenting APT 29 and Russia being involved, he is downplaying the impact and casting doubt as to who could have done it.

 

This was a blow to the traction of getting this handled on time because to cast doubt from such a high level meant that some people would want to reassess what's going on. Not only that this was one of those moments where an entire industry basically looked back at Mr Trump, and industry where I've known people on every possible side of the political spectrum, and said "What the hell are you talking about?".

 

Luckily for the us, what largely ended up happening was that the industry may have stutter stepped for a moment, but Secretary of State Pompeo came back and affirmed that this was pretty clearly Russia. The industry kept updating and would move along with trying to corral the problems that were popping up from the spread of the malware.

 

New years eve - Microsoft Impact

 

Perhaps one of the biggest impact events would be made public on New Years Eve, the night before 2021. This is when Microsoft revealed that after they were impacted Russian actors made out with some of the source code for their software.

 

Microsoft affirmed that they couldn't have made any changes to code that could have propogated the same way that happened with Orion, but they were able to read it in plain text. Now, some of you might think "Oh good that's great Microsoft isn't as impacted as they could have been". But with source code like this it presented a unique opportunity.

 

After source code is compiled it becomes a lot harder to actually figure out. There are ways to obfuscate the code, encrypt it, and any number of possible methods to ensure that you don't see behind the curtain. Part of that is so that people don't just steal and remake your software, and part of it is because coders aren't perfect and not every tool is perfect.

 

By looking at the source code you get a glimpse at all the possible imperfections that could be in play. All the different places that might be worth looking at if you want to try and break the application or get it to do something that you want.

 

There could be vulnerable or old libraries of code that are in use, or maybe they just didn't do a great job at perfectly checking each piece of their own homebrewed stuff.

 

Coders aren't perfect and that's fine, but when stuff like this gets leaked, it shines a light on directly where those imperfections can lie and can have big ripples.

 

Think about it more like this. With each month that passes, as Microsoft releases more and more updates to address security vulnerabilities that might be under active attack. How many of those might have come about because an enemy researcher got their hands on this code and found the blemish.

 

January 5th - A day before things got weird

Joint Statement

 

So we're into the first week of 2021 now. I know for most of us, that week and the sixth in particular were of particular impact. But just before that, on the fifth, A joint statement was released by the FBI, CISA, and the NSA. Here are some particularly interesting snippets:

 

 

On behalf of President Trump, the National Security Council staff has stood up a task force construct known as the Cyber Unified Coordination Group (UCG), composed of the FBI, CISA, and ODNI with support from NSA, to coordinate the investigation and remediation of this significant cyber incident involving federal government networks. The UCG is still working to understand the scope of the incident but has the following updates on its investigative and mitigation efforts.

 

This work indicates that an Advanced Persistent Threat (APT) actor, likely Russian in origin, is responsible for most or all of the recently discovered, ongoing cyber compromises of both government and non-governmental networks. At this time, we believe this was, and continues to be, an intelligence gathering effort. We are taking all necessary steps to understand the full scope of this campaign and respond accordingly.

 

 

So now, we're at least presenting a unified front on the response and acknowledgement. But one thing that was of concern is that there were seemingly no political responses from the white house on this. No sanctions, no acknowledgement of this being a legitimate act of cyber war, nothing beyond saying "yeah it was probably them".  It was a frustrating point for many in the field.

 

 

The next few weeks were marked in a bit of turmoil as the country recovered from the January sixth insurrection events.

 

On the Solar Winds front things weren't necessarily winding down but they had reached an apex I'd say. Mitigating controls and identifiers were developed, plans should have been implemented to detect and resolve, and things were starting to return to a status quo.

 

CISA would expand their report to include more specific details on how this attack might have gone undetected for so long. For example, we talked about how they disguised their traffic under a legitimate protocol. The malware would try to look for indicators that it might be in a sandbox environmnet. That's an environment that forensic investigators might use to research and observe how a malware behaves. If it hit any of a predetermined number of built in checks that said "yeah, I'm in a sandbox" it would stop functioning.

 

Once in the network, If possible they might try to add legitimate tokens for access that wouldn't trigger alerts when used, then use those to move laterally and have it look like legitimate traffic as well.

 

February 19 - Biden Admin response

 

As the next President took office, the Biden administration was quickly left with the pieces that still needed to be picked up. Namely, was the government going to take any official action against Russia for this? Pressure did start to mount. After all, this was a pretty easy win for them to start the term with. There was legitimate evidence and no response yet, so almost any action at all would be a step up from what we had.

 

In mid February that response started to take real shape as Biden started to come out decrying his predecessors neglect of the issue. Jake Sullivan, the national security advisor to Biden at the time, said that the president would begin to look at responses after a more thorough congressional investigation of the incident took place. Here's what he said specifically about the threat

 

Biden Says Hack of U.S. Shows Trump Failed at Cyber Security

 

Feb 23 -26  Congressional Hearings

 

 

The following week, the hearing began. The first Congressional hearing had SolarWinds, FireEye, CrowdStrike and Microsoft testifying to the Intelligence Committee of the Senate. It was an interesting start when it was found that Amazon did not attend.

 

The reason they were expected to attend is that amazon servers were among the attack infrastructure used here and I suspect they wanted to know how amazon didn't know about this and why they may or may not have acted on it.

 

One thing I like about these hearings was that in order to get the most clear and transparent assessment of the situation there were amnesty and protections put in place to a certain extent that would allow a level of liability protection to those that came forward.

 

And this is something that is really needed I think, that is to say a level of transparency and open communication about cyber incidents.

 

Currently today there are guidelines for when a breach should be reported and what kind of information should be disclosed. Currently, for example, the EU General Data Protection Regulation, or GDPR as you may know it,  requires breaches to be reported within 72 hours of it becoming known. That's a fine clock to assess and report and almost assuredly won't have a full picture of the incident but it's a great start.

 

In the US, we do have breach notifications as well, but there's always more that can be done. I think that gross cyber negligence should be allowed similar protections to whistleblowers that report financial crimes under the Sarbanes-Oxley Act, or the False Claims Act that protects the whistle blowers coming forward with reports of government fraud. It needs to be codified that cyber whistleblowers should receive some level of protection so that we can potentially hit this risk much earlier in the process.

That's not to say that this could have been hit as hard early on, but there is a trend where often times what happens is a security researcher might report on a vulnerability forcing the company to come out and address it, even if internally it may have been known.

 

 

Speaking of early on. There is one pice I'm realizing we still never got to. We know how far this went, we know how FireEye got hacked. But, how did they get into SolarWinds?

 

I want to pause here for a second and say there still isn't a 100% clear answer on that. It doesn't seem like it was  phishing, but it was many do believe that it involved the use of open source data that was readily available for a period of time.

 

You see, much like many coders a lot of people at SolarWinds used github to publish pieces of code. Well in 2018 an intern accidentally published one that was accessible and contained a password in clear text. That password was "solarwinds123". That password was something that could have been used in one of their file upload platforms and some speculate that's how they got some of the malicious updates into the rotation.

 

Solarwinds denies that this is the reason, and suggests some level of password spray or brute force attempt could also have been the ticket in. But this is indicitive of a bigger issue, namely how was that password allowed to be in use.

 

Most companies have a policy that prevents the use of certain passwords. Passwords that are based on company name, the region or sports teams tangent to the physical locations of the company, seasons, and years are all things that should be banned when creating a password.

 

After all, these are all things that can be pretty easily turned out by entering a small amount of information into a password generating tool and letting that thing do its work.

 

The hearings would continue and ultimately in April President Biden would enact fiscal sanctions against Russia for their involvement in this and other crimes, as well as expelling a certain number Russian diplomats that were representatives of the intelligence services.. Here is a snippet from that sanction saying that quote:

 

Today the United States is formally naming the Russian Foreign Intelligence Service (SVR), also known as APT 29, Cozy Bear, and The Dukes, as the perpetrator of the broad-scope cyber espionage campaign that exploited the SolarWinds Orion platform and other information technology infrastructures. The U.S. Intelligence Community has high confidence in its assessment of attribution to the SVR.

 

The SVR’s compromise of the SolarWinds software supply chain gave it the ability to spy on or potentially disrupt more than 16,000 computer systems worldwide. The scope of this compromise is a national security and public safety concern. Moreover, it places an undue burden on the mostly private sector victims who must bear the unusually high cost of mitigating this incident.

 

Today, the National Security Agency, the Cybersecurity & Infrastructure Security Agency, and the Federal Bureau of Investigation are jointly issuing a cybersecurity advisory, “Russian SVR Targets U.S. and Allied Networks,” that provides specific details on software vulnerabilities that the SVR uses to gain access to victim devices and networks. The advisory also provides specific steps that network defenders can take to identify and defend against the SVR’s malicious cyber activity.

 

Additionally, the SVR’s compromise of SolarWinds and other companies highlights the risks posed by Russia’s efforts to target companies worldwide through supply chain exploitation. Those efforts should serve as a warning about the risks of using information and communications technology and services (ICTS) supplied by companies that operate or store user data in Russia or rely on software development or remote technical support by personnel in Russia.  The U.S. government is evaluating whether to take action under Executive Order 13873 to better protect our ICTS supply chain from further exploitation by Russia.

 

We've known Russia to be a big bad in the cyber world before. We've known the damage they can cause. But between election issues and this coming so close together, I hope this demonstrates how big a hold they can really get. This uprooted many lives for weeks on end, and while it's mostly over now, damage was done, and shots were fired. This extended far enough that even if you weren't directly impacted, you were still reaching out to all the companies that might have been working with you and asking how impacted they were, just in case they had any data on you that may have been taken.

 

Ask any professional that was working in the field during that time and I'm sure they'll have some kind of story about it or say that they're just thankful they weren't a part of it.

 

I'm John Kordis, thanks for listening to me explain what the shell happened with the solarwinds hack.

 

Just a couple quick things before we close out the shell. As a reminder, we've got a store now if you want a shirt, sticker, or patch. We've got some fun little things that I think you might like, it's at store.whattheshellpod.com. If you want to participate a bit in some of the discussions we have, you can also just go to whattheshellpod.com and click the discord logo. That'll take you to our channel where I try to be pretty active with anyone that's there.

 

Lastly, if you want to see more behind the scenes kind of stuff or show tangential stuff, you can follow me on instagram and twitter at shell_pod.

 

Thanks, and I'll see you all again in two weeks!