Tom: Hi, Camille, how are you doing today?
[00:48] – Hi Tom. I’m doing well today. I’ll tell you, you know, the, the winter is passing along quickly. I cannot wait for spring to hit us.
[00:58] Camille: It’ll come. (laughs)
[01:01] Tom: (laughs) Slowly, but surely it will, it will make its way here anyway. So what’s on your mind today?
[01:06] Camille: Well, to be honest with you, I have seen a bunch of headlines around SolarWinds in the news, but I’m not really sure.
[01:15] Tom: Yeah, boy, there have been a lot of stories about it. And I think you’re right in that, uh, first a lot has been written about it, but it tends to be sort of bits and pieces of the story. Um, and I think it would be, it would be good to pull what we know all together. And do that with, you know, an external perspective.
[01:28] Camille: I would really be interested in hearing that somebody walked through it kind of what may have happened and how to think about it and what types of things to [00:01:00] be warned about in the future.
[01:49] Tom: Yeah. Based, obviously, on what we know today and what’s public because not everything’s public, but yeah, let’s, let’s do that. I think that’s a great podcast today.
[01:58] Camille: Yeah. Sounds interesting.
[02:03] Tom: Our guest today is Dr. Eric Cole who’s the CEO and founder at Secure Anchor Consulting, helping businesses improve their cyber security. He started his career as a hacker with CIA and then moved to the defensive side of cyber security. He has successfully started and sold two companies. Today, he works with a variety of clients from Fortune 500 to top international banks. He develops and teaches his own cyber security courses. And as an author of six books and has a new one being released in May of this year. Welcome Eric.
[02:40] Eric: A pleasure to be here. Thank you, Tom.
[02:42] Tom: So today we’ve got such a great time in, in store for this podcast, but I want to start first with maybe just a little bit about your experiences of some of the larger attacks that we’ve probably all heard of. Um, and maybe some of the themes that you see across these various attacks.
[00:03:01] Eric: When you’re looking at any of the large-scale data breaches over the last five years, anytime you’re seeing more than 50 million records compromised, it’s pretty much the same exact playbook. There is a server visible from the Internet that the organization isn’t aware of. It’s missing a patch. It contains critical data. And that data is not properly encrypted or protected.
Now it goes a little deeper than that because that’s what happened. But I sit there with these large organizations that have all this money, hundreds of people on security. I’m like, “there’s no possible way they have no asset inventory, they have no patching, they have no data control.” So [something else has to be there. And what I discovered is what I nicknamed the rule of 90%. If you look at these organizations, they know about 90% of their assets. 90% of their assets are patched. And 90% of their data is protected. And it could even be as high as 95 or 98%.
But the problem is if you have a thousand servers, and only 90%, you know about or 90% are patched, that’s still a hundred servers that are vulnerable. So that’s the real big problem is companies don’t have a hundred percent asset inventory and therefore they don’t know what’s out there and they can’t patch it, protect it or secure their data.
[00:04:26] Tom: And, and so is it just awareness that you typically, as you’re, as you’re, uh, you know, talking to these companies, you just raise the awareness in these levels, or do you have a specific, you know, other set of key lessons that you try to impart on your clients?
[00:04:43] Eric: To me, the big thing, especially when it comes to asset inventory, patching configuration management is really, automation is key. There’s technology out there that can easily do this. There’s technology that can tell you when there’s new assets plugged into the, your network, when patches are missing. The problem is we always want to go back to awareness and people and humans, and we just have to face it. Humans, get tired. Humans go on vacation. Humans win the lottery. Humans quit, right?
Anything that’s based on a human is eventually going to fail. But computers are systematic and can be programmed. So what we try to do is put an automated processes and systems so now what happens is we plug in public IP address range that companies have registered. We then scan and find all viable assets. And then we’ve, rescanned multiple times a day. And any time there’s a new asset that appears that wasn’t previously there, we send a text or an email to the appropriate parties. So now they only have to react when there’s a problem. They don’t have to monitor a 24 seven.
[00:05:49] Tom: Yeah. That’s interesting. So, uh, you know, today we want to talk about SolarWinds. You know, there’s been so many. Conversations on email and online about SolarWinds and what actually happened. And so we thought that it would be great to talk to you about this and to get your perspective.
So of course, you know, there it goes without saying that we don’t know everything, right? We just know sort of what’s been written and in a lot of cases, I think you may need to, uh, you know, fill in some of the blanks, but with your background, we thought that this would be interesting for our listeners who obviously have probably run across SolarWinds, but doesn’t really understand like what’s going on, what, what actually happened to the best of our knowledge.
And so I, you know, can you start first with just the basic background on SolarWinds? And what we know at least at the highest level. And then we’ll go into the specific sort of, um, uh, vulnerabilities.
[00:06:53] Eric: Okay. So at a high level, there’s really two components to the SolarWinds attack. The first one is the attack against SolarWinds to modify their source code for their Orion product. And then the second piece, is the distribution of a malicious update to all of their clients that then created a back door.
So let me just give a little more detail on those. So if we’re looking at the source code, this is actually a pretty–and I hate to give credit to the attackers–but a pretty brilliant attack because in the past, if I wanted to target Company X, I break into company X. I want to break into Company Y, I target them individually. But in this case, they went in and said, “okay, we want to break in to all these companies, how do we go after it?”
Now, some of this is a little speculation because all the details haven’t been given out, but I will tell you how I would have done this attack when I was on the offensive side. I would have put together a list of the companies and government entities that I wanted to break into. I would then start looking at what is the common denominator? What is the components? Are they all running routers from a certain vendor? Are they all running a certain operating system? Are they all going in and using a certain piece of software?
Now, my guess is that they had multiple hits because if you look at the companies and the government agencies, they have commonalities on hardware vendors with routers and switches. They have commonalities with operating systems and they have commonalities with security software.
You would then go in and say, “okay, which of these entities can I go in and gain access to?” So at that point you would start harvesting users at all those companies, you would start sending out various forms of malicious email, phishing attacks and things like that and you would see which one you’re going to get into. And my guess is they were able to get access to computers at SolarWinds first. So that became their target factor.
Now whether they broke into other vendors, right, is yet to be seen. Remember most organizations don’t detect attacks for two to three years. And it’s funny, cause when the SolarWinds first came out, they were like, “yeah, we think we will compromised three months.” And then a week later “It was nine months” and then we, oh, so I think they’re up to 18 or 20 months. And my guess is when all is said and done, it’ll be, uh, probably 25 to 30 months they were compromised.
So then they got access to one of those computers. They use that computer to set up what we call a pivot point. They did lateral movement into the network and ultimately found the source code computers. Then from there, they were able to upload malicious code into that source code.
Now, now just pausing there for a second. There were multiple points of failure because first of all, source code shouldn’t be directly accessible from Internet facing systems. That’s where if you’re doing it correctly, you should really have traditional air gaps. Next, you should be doing integrity, checks and validation before you send out updates and you should be doing checking and testing in-house.
So to me, there’s a lot of flags that go up saying, “Okay, some of our standard processes weren’t followed.” They then push that update out to all of the clients. And this is the second portion. And then all of those systems got infected, installed malware and then set up outbound command and control channels to communicate with the adversary.
Once again, don’t know all the details, but another big failure comes up. If you have servers or software from a third party vendor, you need to follow the rules of that needs to be isolated on a separate segment and going through a firewall. If that was being done, the impact would be much smaller and you should always be watching outbound traffic to look for the anomalies. So it’s one of those with what we know, there was a lot in my mind, failures and security of how things should have been set up and should have been performed.
[00:11:00] Camille: Hey, Eric, what, uh, how big of an operation in your opinion, having done some of this work yourself in the past, would it take to pull off this kind of a hack? I mean, are we talking like one person or are we talking a gigantic operation with a lot of hardware?
[00:11:17] Eric: Unfortunately, to pull it off, wouldn’t require a big team. Uh, usually it’s three to four people and the reason is they each have specialties. So you would have somebody who specializes in email exploits. So, so that would be one person that would get you in to the network.
Then you would need a second person that specializes in the lateral movement–cause they’re are different skill sets. So that would be Person 2. And then Person 3 would be the one that really understands and knows source code, so they can embed that malicious code into that software. And then the fourth person would be the one that really specializes in the outbound command and control channels.
So, uh, four people, you might have two people in each of those, six to eight, but it’s not a huge, huge requirement. You don’t need 300 people to pull this off. But what we’ve seen so far, it’s not a very advanced attack. It was pretty straightforward methods they use to break in and compromise the systems.
[00:12:18] Tom: So I guess up until this point, what you described is an attacker that is looking for commonalities across their target customers that they’re looking to exploit’ they find, in this case, a common tool, I guess is how we would classify SolarWinds or application that runs across lots of these target companies. And so, and they were able to exploit SolarWinds by gaining access to their source code. So they broke in, they got access to the source code. They changed the source code. And then now that they’ve changed the source code, they’re able to extract data. Uh, from the SolarWinds clients.
Do we know what they did to the source code? Like what were they going after?
[00:13:10] Eric: What, what they were going after on the source code is the ability to take control of the client computers that ran the SolarWind software. So essentially what they wanted to do is have a command and control piece of code that once it was installed on the system would then be able to take control, make outbound connections and give somebody access to those networks.
And there’s many ways of doing it, but the easiest way is with a patch update. Because these vendors are always pushing out patches, it’s not uncommon to get anywhere from five to 15 patches a month for any piece of software. So they just went in and they took their malicious code and they made it look like a patch update, so when it ran on the system, instead of updating the functionality of SolarWinds, it essentially created a backdoor to exploit the network.
[00:14:05] Tom: Okay. So now you have a, a client, we’ll just say Company ABC. And Company ABC is a customer of SolarWinds. So Company ABC now has a client platform that has this malicious code on it. And that allows now the attacker’s access to that device. Do we know what they were going after in the now these client devices of Company ABC?
[00:14:39] Eric: The only thing we could really tell is that they wanted to gain access to a large number of systems because, if you look at the entities that were impacted, they’re across the board. You had FireEye come out, then you have some other vendors. You have government entities, you have other folks out there. So I think what it comes down to is they had a target list for specific reasons. Like a lot of people speculate that they were going after FireEye because FireEye acquired Mandiant. And if you remember, many years ago, Mandiant had put out the APT one report, which really sort of shined a lot of light on a lot of these foreign adversaries.
So some people think that, Hey, they were going after them for revenge. Some of the government entities were for information, data, or intellectual property. So I don’t think, it was one thing like they weren’t going after all the banks for money, they weren’t going after all R&D for IP, I believe they had a long list and they had specific reasons and goals for each of those. Because the malicious code that got distributed with the solar wind software, it didn’t specifically gather data, exfiltrate data or delete data. What it did is created access paths for the adversary. So all we know is that the adversary wanted to gain access to this list of networks.
[00:16:00] Tom: Okay. And, and did this exploit extend beyond SolarWinds customers? So if, if somebody didn’t have SolarWinds in their environment, is there still a risk to a company in some other way?
[00:16:15] Eric: Of this specific attack, no. And it’s even more specific. You had to be running a specific version of SolarWinds. So if you look at the total percent of SolarWinds customers, it was only a certain percent of them that was running a certain version that contain these malicious DLL. So it wasn’t all SolarWinds customers only certain ones, at least for this specific attack. If you weren’t a SolarWind customer, then you would not have been impacted.
[00:16:43] Camille: So how did they, how was it discovered that there had been an attack? Was it by an ABC company or by SolarWinds itself?
[00:16:54] Eric: I believe–and once again, this is where the information gets a little fuzzy–but, but I believe what happened was SolarWinds did come out and initially saying, and once again, this is my understanding, this isn’t a guaranteed factual, but it’s my understanding of what happened is they came out and said (what typically happens) “We were breached, but there wasn’t really any impact. We don’t think they did anything that we don’t think anything happened there.”
So, so they caught an initial attack. But I don’t think they realize the impact. And then from what I saw, it was several commercial and government agencies that started having outbound proxies and started going, “Hmm. This traffic doesn’t make sense.”
And they started overloading the proxies. And that, that’s the interesting thing with not only SolarWinds, but most of these other attacks that we’ve seen over the last three years. It’s typically the IT department that catches it. It’s not the security department. The security department didn’t go in and say, “Oh, we saw the attack because there’s so many alerts and false positives.” It was somebody in IT that said “my proxy server isn’t working correctly. The memory is at 100%. What’s going on?” And then when they went in, they said, “why did the traffic going outbound through my proxy increased by 300% in the last 48 hours.” And then they went in and said, “wait, this system looks like it’s compromised.”
So it’s sort of interesting that it’s usually performance issues of IT equipment that catches most of these attack factors.
[00:18:22] Tom: So Eric, if I, if I recall though, you, you said that you thought that this exploit was probably something that’s been in place for, you know, 25 plus months. So why, why would all of a sudden the traffic spike up like what you were just describing? Why, why wouldn’t it well, if it does spike up like that, why wouldn’t it have been detected 25 months ago? Or was there something that changed that made him spike up here towards the end?
[00:18:52] Eric: It comes down and it’s the way that we catch almost [00:18:30] all of our attackers. They get greedy. So, so two components at play. When these systems first got compromised in many of these cases, they’re what we call sleeper cells. They have access, but they’re not doing anything with them yet. So the code is just sitting there dormant, but there’s not a lot of activity. So that’s sort of explanation one.
The second, which we see in most cases is these attackers are fairly smart. So they’re going to go in when they activate one of these command-and-control bots, they’re going to go in and start taking a little data and they’re going to take one or two records. And then they’re going to get greedy and go, “Hmm, two records a day is good. Four would be better.” So then they take four, then a, so they slowly start ramping it up cause they don’t think anybody’s watching.
And then at some point they make the false conclusion. “Oh no, one’s going to catch us. And we’ve been doing it for two years. No, no one’s ever—“ And then they start cranking it up and they inadvertently go in and overload the computer systems. Cause these attackers know how to bypass the security equipment, but they don’t know the thresholds of the IT equipment.
So it’s never–I shouldn’t say never–it’s hardly ever the security equipment that catches it cause they’re clever enough to fly under the radar. But they don’t understand the hardware and they tend to overclock it cause performance issues and that’s how they inadvertently get caught.
[00:20:19] Tom: That’s interesting. So, um, now I’ve, I’ve read also there’s a Microsoft, um, aspect or element here. I wonder if you can try to draw the connection and how, how are they related in, in the SolarWinds, uh, exploit.
[00:20:35] Eric: My understanding with that, and this is where it really starts to get interesting is that Microsoft had connections and or was a customer of SolarWinds, but then what happened there is when they got access to Microsoft systems, instead of just making an outbound connection, they realized where they were and then they started going after their code and their software. So now you have sorta this there’s ripple effect where we get one software vendor. We get here, we get the next software vendor. And we sort of continue down the path in that manner.
[00:21:10] Camille: Are there ever, um, you were saying before, it’s not, it’s not entirely clear what the hackers were going after in this case. And I’m just wondering, is, is there kind of a model out there where hackers are just proactively seeing what they can get access to and then having that access and maybe taking a couple documents so that they can prove that they have the access, but then they go get a customer for the access? As opposed to coming in with a plan in advance and then thereby having a specific target?
[00:21:46] Eric: Exactly the goal in this case, as far as we can tell was a broad goal, which is gain access. Gain access to as many networks as possible, but you’re right. It’s not like it’s a targeted attack. Where if I go after a bank, I specifically know I’m going after a bank. I want to steal money. I want to steal records and there’s a specific game plan.
And a lot of these cases it’s often “let’s get access, and maintain the access to see what we can do so we can use it at a later point of time.” And you’re right. Sometimes it’s the cell access. Sometimes it’s to ransom it back to the company, sometimes it’s to sell to a third party, but the name of the game now is whoever has access wins the game. And that definitely looks like what they were after with the SolarWinds attack.
[00:22:31] Tom: So the typical listener here for this podcast, then I guess the takeaway for them is obviously if they’re a customer of SolarWinds, then there’s an area of concern for sure. But even if they’re not a SolarWinds customer, there could be secondary ways they could be compromised.
[00:23:13] Eric: That is correct. But what I tell all of my customers is assume that you were compromised. So if you were a SolarWinds customer and you had an infected version and you were notified that you were compromised, what action would you have taken? You would have gone in and you would have redesigned your network. You would have temporarily removed the SolarWinds, put them on a separate segment and you would have done better outbound monitoring. That’s what you need to do.
The best bet is assume that you were compromised and use this as a lesson learned. It will happen to you. You have software vendors, you have components. SolarWinds was not the first and they won’t be the last. So you need to go in and assume that you were compromised. Be proactive.
And then whatever you would have done to respond to an actual compromise, those are the things you need to put in place today.
And in a lot of cases, you can talk to the vendor and the vendor can go in and give you all the assurances. But ultimately if it’s your network that gets compromised, you’re the one that’s going to have the liability. So you really want to design out now before the compromise, assuming that it will happen at some point in the future.
[00:24:23] Tom: Yeah, so if, if you let’s say your, the CIO at, at Company ABC right now, and you’re going to follow your own advice that you just said. What are–at least at the highest level–what are those maybe four or five steps that you would take today now, knowing what you know?
[00:24:45] Eric: The first one would be to take all third party servers that contain vendor software and put them on a separate isolated segment. Don’t plug them directly into my private network, put them through a firewall or a filtering device to limit and control the access.
Second, is test and verify the current software that you have today to make sure it’s stable. Put monitoring software sniffers to make sure there’s no extraneous activity or connections.
Third, make sure you’re very careful and deliberate about updates. A lot of vendor software updates are functionality that you don’t need an add complexity. So have a strict rule that you’re only going to update after verification and validation.
And then the last one is put proxying in place for all outbound connections. So anything leaving your organization, you’re going to put proxy in place and turn on automatic anomaly detection. So if all of a sudden the number of connections, type of connections, or volume of connections changes by a certain percent, you’re going to get an immediate automated alert.
[00:25:56] Camille: I guess I would have kind of a parallel or a complimentary question, which is if you’re a supplier, how can you give them some sort of assurance that you’re well protected?
[00:26:15] Eric: Uh, it would really be to two big areas. The first one is processes for software development and deployment. So if I was a software developer, I would go in and immediately document my environment. I would show that we have air gap set up where we’re controlling and managing it as a trade secret, our source code–cause that’s what it is. We have an isolated and separate. We do verification and validation and we do robust testing before we roll it out to the vendor.
Then I would give all my customers a referenced architecture and how they should deploy my software within their environment. I would never recommend my client directly plug my server or software onto their network. I would give them an architecture to say, “here’s how you should set it up. Here’s the firewalls.” And then I would also provide them a list of all authorized connections so they can go in and properly firewall it. And then if they see anything outside of those connections, they instantly know it’s an attack.
And then finally, I would offer for them as a service for us to be able to do monitoring of their network for my component, that if we see anomalous connections or anything unusual that we can get an alert so we can respond quickly.
[00:27:06] Tom: So Eric, this has been really interesting. I, I certainly appreciate the fact that you’ve walked us through what we know so far and filled in the gaps. I know that it can be uncomfortable sometimes. Um, I guess out of curiosity, how much longer do you think this, this particular SolarWinds attack is going to be playing out? Do you think we’re kind of towards the end of where, where, what we’re likely to learn or [00:27:30] would you expect that we’re going to continue to, to have new revelations coming out?
[00:27:28] Eric: Typically with these types of attacks from when it publicly comes out. So if we say the real public was maybe two months ago, three months ago, then it’s usually about six to nine months before we find out all the details. So I would say probably, uh, August September, we’re going to know all the details and all the information.
But my prediction is just because of a [00:28:00] lot of the poor practices that we put in place because of the pandemic in 2020, that there’s probably going to be one or two other SolarWinds attacks that people are going to forget about SolarWinds and move on to the next one.
And I’ll tell ya, uh, if I was working at SolarWinds, I would be praying for another breach, right? Cause it’s all about who’s getting the attention, but my guess is somebody else is going to get the limelight before we get all the details.
[00:28:52] Tom: Well, before we let you go, we like to have a fun segment in our podcast, uh, where our guests and Camille and I also, where we try to share a fun fact or something that you think that our listeners would find interesting. It could be a movie, a book you read, and I wonder if you have something like that that you want to share with us?
[00:29:12] Eric: Uh, sure. So my, my, my fun fact is prior to 1954, for a hundred years, nobody ran, um, ran the mile in under four minutes. It was actually, people believed that you would die if you ran the mile in under four minutes. And a hundred years, nobody, zero people did it. And then in 1954, a gentleman by the name of Roger Bannister broke the four-minute mile.
Now, what I love about this is it’s the power of limiting beliefs. If he was an anomaly, if it really couldn’t be done, we would have saw him break it and then nobody would have done it for the next 75 years. But after Roger Bannister did it within six months, another person broke the four-minute mile. Within a year, 10 people did.
Today, 70 years later, over 1,400 people have ran a mile in over, sorry, in under four minutes, which to me just shows the power of a limiting belief. People believe they couldn’t do it, and nobody tried. And then as soon as somebody broke that four-minute mile and gave people permission to do it, everyone else did so. So to me, my fun fact is what are the four-minute miles in your life that are limiting you or holding you back?
[00:30:27] Tom: That’s great. And it was a great story. Yeah, it is. It is incredible that they, they, they thought your heart would explode. Exactly. That’s pretty important. Pretty interesting. All right. So Camille, what, uh, what do you have for us today?
[00:30:41] Camille: Unfortunately, I guess mine is not so uplifting, but, um, I did find this to be very interesting when I learned that one of the early signs of Alzheimer’s or dementia is that people can no longer read analog clocks. And where this kind of, um, came up for me is, you know, my kids are learning to tell time and they have a lot of trouble with the analog clock, uh, because it just doesn’t make sense that the hour hand is the shortest and it’s difficult for them to kind of comprehend each one of these ticking seconds is passing the other ones, but only makes up a small increment, even though it goes the fastest.
So I just thought it was an interesting, kind of a parallel that. People struggle with that mentally as, as they age.
[00:31:31] Tom: Wow, you guys both have deep, deep, uh, uh, fun facts. My mine is the opposite end of the spectrum of things that actually matter. Um, mine is the video game Donkey Kong. Uh, I came across this, I thought it was fascinating. So like, when you think about the name, what, what Donkey Kong, what is there so Donkey in Donkey Kong. So w what is that?
Well, it turns out that the maker of Donkey Kong obviously was not from this country, um, or an English speaking country. Um, and they mistakenly thought that Donkey meant stupid. And so what they were going for was a name that meant “stupid ape.” You know, they kind of missed the mark, but we all, as a, as a society were given the name Donkey Kong as a result. So that’s where Donkey Kong came from.
All right, well, with that deep, deep insight, we will bring this podcast to an end. So, Eric, thank you so much for coming in and sharing your insights on SolarWinds. And I found it fascinating. I think our listeners will really love it.
[00:32:48] Eric: My pleasure. Thank you for having me.
