Tag: cyber

Log4Shell Defense

Log4Shell exploitation and hunting on VMware Horizon (CVE-2021-44228)

TLDR

Go and run this on the connection servers:

https://github.com/mr-r3b00t/CVE-2021-44228

It’s crude so also look for the modified timestamps, recent unexpected blast service restarts and if you have process logging go and check for suspicious child processes over the period. Once you have checked, run a backup, then if they aren’t patched, patch the servers! (i know patching isn’t as simple as just patch!)

Introduction

In Decemebr a critical vulnerability (created by a feature request) in Log4J was discovered (named Log4Shell), unveiling the reality that an enormous amount of products may be vulnrable to a relativley simple remote code execution vulnerability (which includs a huge range of internet facing systems, such as vmware horizon).

I’m going to be vaugue here on purpose, mainly because I’m not omnipotent and the scale of the challenge here is significnatly large that it’s subject to change. The constant phrase with log4shell is “dynamic and evolving”. To be blunt, the intel we are getting is changing very rapdily from both a threat and vulnerability perspective.

The Log4J scenario to some is a non event, but when we look at this at scale and when we look at certainly technology stacks it has really serious poential for negative impact. Public facing services such as:

  • VMware Horizon
  • VMware Vcenter (don’t ask why people put this online but it seems lots them do!)
  • VMware Worksapce One
  • Mobile Iron
  • Unifi
  • Citrix XenMobile
  • Fidelis commandpost

For a list of currently known affected products please see:

https://github.com/cisagov/log4j-affected-db/blob/develop/SOFTWARE-LIST.md

Vmware Horizon World View

As you can see there are potentially one or two horizon services exposed! (let alone vcenters)

Timeline

Eraly December ~9th Decemeber 2021 the vulnerability was publically disclosed

12/12/2021 – Vmware publishes KB to partially address the vulnerability (workaround) on vmware horizon (https://kb.vmware.com/s/article/87081) – this has been updated all through December

12/12/2021 – VMware publishes advisory https://www.vmware.com/security/advisories/VMSA-2021-0028.html

13/12/2021 – UK NCSC Advisory https://www.ncsc.gov.uk/news/apache-log4j-vulnerability

16/12/2021 – VMWare Horizon “Fixed” Builds released https://kb.vmware.com/s/article/87073

17/12/2021 – VMware Horizon releases new builds for some version of Horizon https://kb.vmware.com/s/article/87073

23/12/2021 – Exploitation of VMware Horizon discovered in the wild (across geos from the CTI we have)

24/12/2021 Active in the wild exploitaiton of vmware horizon

25/12/2021 – Active in the wild exploitation of vmware horizon

03/01/2022 – Microsoft Update https://www.microsoft.com/security/blog/2021/12/11/guidance-for-preventing-detecting-and-hunting-for-cve-2021-44228-log4j-2-exploitation/

05/01/2022 – mRr3b00t publishes initial backdoor detection script in Github (https://github.com/mr-r3b00t/CVE-2021-44228)

05/01/2022 – NHSD Publishes https://digital.nhs.uk/cyber-alerts/2022/cc-4002

07/01/222 – PwnDefend Post with detection examples

14/01/2022 – Increased detection activity noted (https://twitter.com/TheDFIRReport/status/1482078434327244805)

Detections

The Log4J Payloads into the web services aren’t so easy to detect. They will basically look like standard traffic and without full packet captures and TLS inspection you almost certainly (based on research in the lab) see the malicious payloads.

You will in the logs however see error events, so there will be a ton of logs in the UAGs and Connection server logs that occur when a failed attempt to exploit log4j occurs.

In short (but subject to loads of configuration and environmental variance) we have found:

In the standard vmware logs you will largely not see exploitation. You will likely see failed exploitaiton attempts. There are some logs which show a connection but the metadata is limited. E.g. it will hav source IP, time and path however depending upon your load balancer configurations you may just see the UAG connect to the Connection server and access a path used in normal operations.

Process Logging

Process logging in sysmon will show excution of both log4j when the java child processes are spawned. It however possible that a malicious class load could run in memory and use native win32 APIs and NOT spawn a child process (we haven’t tested that yet).

Backdoor

Backdoors can be detected by looking for file modifications here: (default path) C:\Program Files\VMware\VMware View\Server\appblastgateway\lib

The script I knocked up is crude but will detect the activity seen recently in the wild.

https://github.com/mr-r3b00t/CVE-2021-44228

you can also use a PowerShell one liner:

$path=gwmi win32_service|?{$_.Name -like "*VMBlastSG*"}|%{$_.PathName -replace "nssm.exe","lib\absg-worker.js"};$path = $path -replace'"',''  ;Get-Content $path|Select-String "req.headers\[\'data\'\]"

you can also look at the modification stamps:

$path=gwmi win32_service|?{$_.Name -like "*VMBlastSG*"}|%{$_.PathName -replace "nssm.exe","lib\"};$path = $path -replace'"',''  ;dir $path

In our testing we have found the stamps on all files should be the same, a file with a different date has likely been modified in a suspicious manner.

Microsoft Defender for Endpoint

These queries can be narrowed down and you should filter these onto your specific Horizon infrastrcuture, so they are examples for guidance, you will likely need to do some tweaks and mods:

Log4J (TCP 443) child process creations (check for benign normal child processes)

Look for evil using powershell etc.

DeviceProcessEvents 
| where DeviceName has_any("horizon-con-001") //connection server name
| where InitiatingProcessParentFileName ==  @"ws_TomcatService.exe"
| order by Timestamp desc

Check for backdoors being created by powershell for file modification events:

DeviceFileEvents
| where FileName has_any("absg")
| where FolderPath has_any("appblastgateway")
| where ActionType == "FileModified"
| where InitiatingProcessCommandLine has_any("powershell")
| order by Timestamp desc 

Check network connections from ws_TomcatService.exe

DeviceNetworkEvents 
| where DeviceName has_any("horizon-con-001")
| where InitiatingProcessCommandLine == @"""ws_TomcatService.exe"" -SCMStartup TomcatService"
| where RemoteIP != @"127.0.0.1"
| where ActionType == @"ConnectionSuccess"

Please note that in our lab testing we do not see all the connections in MDE. So this data is deemed to be incomplete:

Backdoor Usage (TCP 8443)

The backdoor seen is in absg-worker.js (but remember the log4j rce here could be used in many many ways:

You can see in this instance the modified date looking quite out of place against the files peers.

DeviceProcessEvents 
| where DeviceName has_any("horizon-con-001") //connection server name
| where InitiatingProcessParentFileName == @"node.exe"
| order by Timestamp desc

In our limited testing we can see backdoor usage whch spawns child processes from node.exe

We should also note both in the lab and in the wild we have seen the following:

A very simmilar message is logged on the connection server when a failed backdoor attempt is made. This can be found in:

C:\ProgramData\VMware\VDM\logs\Blast Secure Gateway\absg.log

Service Restart Events

If a backdoor is installed to the BLAST service then you will likely see the service restart at an unexpected time:

Get-EventLog -LogName "System" -Source "Service Control Manager" -EntryType "Information" -Message "*VMWARE*running*"

that will list all the service restarts in the SYSTEM log on the Connection Server/s or we can just grab the blast service:

Get-EventLog -LogName "System" -Source "Service Control Manager" -EntryType "Information" -Message "*Horizon View Blast Secure Gateway*running*"

Useful logs

  • If WAF is inline WAF logs would be useful
  • SYSMON (process launches, dns events)
  • EDR Process Logging (process launches, file writes)
  • Load Balancer HTTP Traffic Logs
  • Connection Server debug logs and blast logs
  • Firewall (ingress and egress traffic logs)
  • UAG blast logs
  • DNS Logs (however we are seeing threat actors use IP addresses for the LDAP call backs)

Vmware Horizon Log Levels

In the lab we observed the following:

  • By default the UAG log level was set to INFO
  • Debug logging is set on the connection server (assumed based on filename)

With INFO logging on the UAG we weren’t able to determine if a malicious payload had been sent. In the ESMANAGER log successful connections were not logged.

We could see the connection on the backend connection server however we could not determine this was a malicous payload in the default logging configuration.

in DEBUG mode we could see the connections and PALOADS on the UAG.

Exploitation Entry Points

There are at least two pages on the HTML Access services that are vulnerable when ther server are unpatched:

  • /broker/xml
  • /portal/info.jsp

Please note we’ve had mixed results with /brokes/xml which may be build version specific.

Threat Intel

Micosoft has reported DEV-0401 using Log4Shell in relation to ransomware activity:

Advisory

This post will likely be updated, it’s not a step by step of how to find all the evil but it hopefully will help identify malcious activity seen in the real world. We will update this if new intel comes in.

Also if you want to wite nicer detections feel free 🙂 these are just examples and are by no means the only ways to do this!

Exploitation Tutorial

When people have had more time to patch and the landscape looks better we will blog how to exploit this and talk about why the currently known backdoor has some limitations due to the service architecture and how the backdoor has been created. It’s important to share exploitation knowledge but there are things to consider, if you are sharing exploitation without detection rules / tools this creates a risk to people. VMware products and services are leveraged by organisations worldwide and the log4j RCE on hoirzon let alone the backdoor can be leveraged for high impact actions by threat actors.

Thanks

Thanks to everyone in the community and industry who has and is helping, thanks to all of those who have stayed up late, missed familty events and generally been super helpful either indirectly or directly. There are lots of people involved in this world who everyday work to keep people safe! (also to my friends and family who have put up with my not being round much!).

Defense

I’m the CEO, why should I care about Cyber…

Introduction

First and foremost, I’m going to start by saying if I include any cliché quotes it’s probably in an ironic context or used to show how they aren’t practically useful. Why are we here? Well, based on the title, it’s because you are either a CEO/MD or you are in a leadership position and want to learn a little more about cyber security.

I’m sure you have read the news, I’m sure you have seen vendor adverts explaining something like:

  • Zero Trust
  • The Security Skills Gap
  • How phishing can be solved through security awareness training (pro tip: it can’t)

And I’m sure someone on your LinkedIn feed you have seen people exclaim all kinds of crazy things like:

  • TLS Weaknesses Lead to Ransomware
  • Security is Simple (it, I’m afraid, is not)
  • Managed Security Service Providers ensure security

Read more “I’m the CEO, why should I care about Cyber Security?”

Leadership

The Security Challenges of 2021

The gaps between strategic security improvement and keeping the wolves out, today!

The Cyber Realities in 2021

Most organisations today honestly don’t have great cyber security postures. Cyber security has improved since the 80’s and 90s’s but still common gaps can be found in the same old areas.

So, whilst security possibilities and technical capabilities for defence have greatly improved, this hasn’t really translated into the level of change we would like to see on the ground inside organisations.

I’m writing this post after giving a talk today about the challenges I see in cyber security across different organisations but also after watching a talk by Dave Kennedy which from my perspective emulates my experiences and largely my views. Read more “The Security Challenges of 2021”

Defense

Phishing your own people – path to eroding trust…

Introduction

“Security education and awareness darling, it’s all the rage! It’s simply to hot right now.” Ok stop, let’s take a minute to get some context. It’s the year 2021, organisations are taking a battering round the globe from cyber criminals who are deploying ransomware, extortion, and fraud via a range of methods but one you can’t not have heard of is phishing.

In this post today, I’m going to look at realities of initial access, phishing and some questions I think people should be asking themselves about the idea of phishing their own userbase. I try and look at this from multiple perspectives because I think it’s a complex subject. Let’s start with initial access methods!

Common Patterns of Access

If we look at the world of technology and cyber security, you will see logs of references to frameworks and language that is enough to send even the committed to sleep! However, let’s abstract from our TTPs, our MITRE ATT&CK frameworks and our “threat actors” and let’s talk in normal English. Read more “Phishing your own people – path to eroding trust or a useful tool?”

Guides

If I was looking for entry level jobs in…

Sales darling, it’s all about sales. It’s a harsh but true part of the world where you need to be able to sell. I’m not talking about business to business or hunters, farmers etc. I’m talking about being able to sell to someone that you are the right person to help them and their organisation.

Now this isn’t easy in the middle or tail end of a career let alone when you are starting off. But let’s for a minute role play and look at what I would do if I was new to the cyber world and was looking for a role?

This isn’t meant as a guide, it’s off the back of a convo I’ve just had with someone struggling in the job-hunting space. So, it’s a rough brain dump from me. The key thing I would say is: Read more “If I was looking for entry level jobs in cyber security – what would I do?”

Defense

HID Attacks using OMG Cables

Human Interface Devices is the science way of saying (in this case) keyboard! Now that doesn’t sound amazing but then we look at the details. What we are talking about here is a wireless remote controlled programable keyboard emulator disguised as a USB cable or a cable between a real USB keyboard (must be detachable). This provides attack opportunities to both key log and hijack inputs to PC devices covertly and remotely (within WIFI range). Just imagine what you could do with one of these.

Read more “HID Attacks using OMG Cables”

Defense

Password Auditing with L0phtcrack 7 – A quick intro

If you know me that one of the first things, I recommend organisations do is conduct password audits against active directory on a regular basis. There are a ton of ways to do this and depending upon size of directory and budget you will likely want to do this with more than a CPU however the process remains the same. So, with the news that a new release of L0phtcrack (open source) is online let’s take a look at how we can deploy and start cracking those hashes! This isn’t an end to end guide to cracking with l0phtcrack – but it does show the install process and provide considerations for your cracking adventures. Remember, only do this where you have authorisation. Read more “Password Auditing with L0phtcrack 7 – A quick intro”

Guides

Reporting an email as phishing in Office 365 with…

Did you ever just ignore or delete a phishing email? I mean that’s great in one sense that you won’t have any negative impact. But if the email did get past the mail security filters, you can report it using the “Mark as phishing” option.

What if as well you wanted to not only enable users to report but also pass the intelligence onto the NCSC Suspicious Email Reporting Service (SERS)? How cool would that be! Well, have no fear people, we are going to show you how easy this stuff is to deploy and configure. Read more “Reporting an email as phishing in Office 365 with NCSC SERS”

Leadership

The problem with gatekeeping in the cyber security industry

Stark Realities

Imagine having an industry where you can’t be in it without already being an expert in all fields, imagine having to be able to command policy and drive strategy but not having anyone having ever helped you learn how to do this, imagine that if you did all the activities involved with secure service and yet people say you aren’t part of the industry because your job title doesn’t have the word “security” in it and imagine if that you are told you aren’t part of the cyber security industry because you also have to worry about budgets, sales, marketing, new business initiatives, IT services and well anything else!

What would happen if we had this as our cyber security industry principles… well that’s simple?

Read more “The problem with gatekeeping in the cyber security industry”

Defense

Cloud Security – 26 Foundational Security Practises and Capabilities…

That is quite the catchy title don’t you agree? Ok so that needs some work and when we think about cloud security, we need to realise that Computing as a Service isn’t a silver bullet.

One Cloud to Rule them all and in the darkness bind them

Ok so the cloud was promised as the saviour of IT and Cyber security but the promise vs the reality. Well, let’s be frank, they don’t really match up. But have no fear – secure cloud design is here (omg cringe)! Ok now we have that out of my system let’s look at some basic cloud security considerations to make when thinking about cloud services.

Checklist

Ok so the world doesn’t work with a checklist however, if you are like me you will want to use lists and aides to jog the little grey cells into action. Let’s think about cloud services and security: Read more “Cloud Security – 26 Foundational Security Practises and Capabilities Checklist”