Category: Vulnerabilities

Log4Shell Defense

Log4Shell exploitation and hunting on VMware Horizon (CVE-2021-44228)

TLDR

Go and run this on the connection servers:

https://github.com/mr-r3b00t/CVE-2021-44228

It’s crude so also look for the modified timestamps, recent unexpected blast service restarts and if you have process logging go and check for suspicious child processes over the period. Once you have checked, run a backup, then if they aren’t patched, patch the servers! (i know patching isn’t as simple as just patch!)

Introduction

In Decemebr a critical vulnerability (created by a feature request) in Log4J was discovered (named Log4Shell), unveiling the reality that an enormous amount of products may be vulnrable to a relativley simple remote code execution vulnerability (which includs a huge range of internet facing systems, such as vmware horizon).

I’m going to be vaugue here on purpose, mainly because I’m not omnipotent and the scale of the challenge here is significnatly large that it’s subject to change. The constant phrase with log4shell is “dynamic and evolving”. To be blunt, the intel we are getting is changing very rapdily from both a threat and vulnerability perspective.

The Log4J scenario to some is a non event, but when we look at this at scale and when we look at certainly technology stacks it has really serious poential for negative impact. Public facing services such as:

  • VMware Horizon
  • VMware Vcenter (don’t ask why people put this online but it seems lots them do!)
  • VMware Worksapce One
  • Mobile Iron
  • Unifi
  • Citrix XenMobile
  • Fidelis commandpost

For a list of currently known affected products please see:

https://github.com/cisagov/log4j-affected-db/blob/develop/SOFTWARE-LIST.md

Vmware Horizon World View

As you can see there are potentially one or two horizon services exposed! (let alone vcenters)

Timeline

Eraly December ~9th Decemeber 2021 the vulnerability was publically disclosed

12/12/2021 – Vmware publishes KB to partially address the vulnerability (workaround) on vmware horizon (https://kb.vmware.com/s/article/87081) – this has been updated all through December

12/12/2021 – VMware publishes advisory https://www.vmware.com/security/advisories/VMSA-2021-0028.html

13/12/2021 – UK NCSC Advisory https://www.ncsc.gov.uk/news/apache-log4j-vulnerability

16/12/2021 – VMWare Horizon “Fixed” Builds released https://kb.vmware.com/s/article/87073

17/12/2021 – VMware Horizon releases new builds for some version of Horizon https://kb.vmware.com/s/article/87073

23/12/2021 – Exploitation of VMware Horizon discovered in the wild (across geos from the CTI we have)

24/12/2021 Active in the wild exploitaiton of vmware horizon

25/12/2021 – Active in the wild exploitation of vmware horizon

03/01/2022 – Microsoft Update https://www.microsoft.com/security/blog/2021/12/11/guidance-for-preventing-detecting-and-hunting-for-cve-2021-44228-log4j-2-exploitation/

05/01/2022 – mRr3b00t publishes initial backdoor detection script in Github (https://github.com/mr-r3b00t/CVE-2021-44228)

05/01/2022 – NHSD Publishes https://digital.nhs.uk/cyber-alerts/2022/cc-4002

07/01/222 – PwnDefend Post with detection examples

14/01/2022 – Increased detection activity noted (https://twitter.com/TheDFIRReport/status/1482078434327244805)

Detections

The Log4J Payloads into the web services aren’t so easy to detect. They will basically look like standard traffic and without full packet captures and TLS inspection you almost certainly (based on research in the lab) see the malicious payloads.

You will in the logs however see error events, so there will be a ton of logs in the UAGs and Connection server logs that occur when a failed attempt to exploit log4j occurs.

In short (but subject to loads of configuration and environmental variance) we have found:

In the standard vmware logs you will largely not see exploitation. You will likely see failed exploitaiton attempts. There are some logs which show a connection but the metadata is limited. E.g. it will hav source IP, time and path however depending upon your load balancer configurations you may just see the UAG connect to the Connection server and access a path used in normal operations.

Process Logging

Process logging in sysmon will show excution of both log4j when the java child processes are spawned. It however possible that a malicious class load could run in memory and use native win32 APIs and NOT spawn a child process (we haven’t tested that yet).

Backdoor

Backdoors can be detected by looking for file modifications here: (default path) C:\Program Files\VMware\VMware View\Server\appblastgateway\lib

The script I knocked up is crude but will detect the activity seen recently in the wild.

https://github.com/mr-r3b00t/CVE-2021-44228

you can also use a PowerShell one liner:

$path=gwmi win32_service|?{$_.Name -like "*VMBlastSG*"}|%{$_.PathName -replace "nssm.exe","lib\absg-worker.js"};$path = $path -replace'"',''  ;Get-Content $path|Select-String "req.headers\[\'data\'\]"

you can also look at the modification stamps:

$path=gwmi win32_service|?{$_.Name -like "*VMBlastSG*"}|%{$_.PathName -replace "nssm.exe","lib\"};$path = $path -replace'"',''  ;dir $path

In our testing we have found the stamps on all files should be the same, a file with a different date has likely been modified in a suspicious manner.

Microsoft Defender for Endpoint

These queries can be narrowed down and you should filter these onto your specific Horizon infrastrcuture, so they are examples for guidance, you will likely need to do some tweaks and mods:

Log4J (TCP 443) child process creations (check for benign normal child processes)

Look for evil using powershell etc.

DeviceProcessEvents 
| where DeviceName has_any("horizon-con-001") //connection server name
| where InitiatingProcessParentFileName ==  @"ws_TomcatService.exe"
| order by Timestamp desc

Check for backdoors being created by powershell for file modification events:

DeviceFileEvents
| where FileName has_any("absg")
| where FolderPath has_any("appblastgateway")
| where ActionType == "FileModified"
| where InitiatingProcessCommandLine has_any("powershell")
| order by Timestamp desc 

Check network connections from ws_TomcatService.exe

DeviceNetworkEvents 
| where DeviceName has_any("horizon-con-001")
| where InitiatingProcessCommandLine == @"""ws_TomcatService.exe"" -SCMStartup TomcatService"
| where RemoteIP != @"127.0.0.1"
| where ActionType == @"ConnectionSuccess"

Please note that in our lab testing we do not see all the connections in MDE. So this data is deemed to be incomplete:

Backdoor Usage (TCP 8443)

The backdoor seen is in absg-worker.js (but remember the log4j rce here could be used in many many ways:

You can see in this instance the modified date looking quite out of place against the files peers.

DeviceProcessEvents 
| where DeviceName has_any("horizon-con-001") //connection server name
| where InitiatingProcessParentFileName == @"node.exe"
| order by Timestamp desc

In our limited testing we can see backdoor usage whch spawns child processes from node.exe

We should also note both in the lab and in the wild we have seen the following:

A very simmilar message is logged on the connection server when a failed backdoor attempt is made. This can be found in:

C:\ProgramData\VMware\VDM\logs\Blast Secure Gateway\absg.log

Service Restart Events

If a backdoor is installed to the BLAST service then you will likely see the service restart at an unexpected time:

Get-EventLog -LogName "System" -Source "Service Control Manager" -EntryType "Information" -Message "*VMWARE*running*"

that will list all the service restarts in the SYSTEM log on the Connection Server/s or we can just grab the blast service:

Get-EventLog -LogName "System" -Source "Service Control Manager" -EntryType "Information" -Message "*Horizon View Blast Secure Gateway*running*"

Useful logs

  • If WAF is inline WAF logs would be useful
  • SYSMON (process launches, dns events)
  • EDR Process Logging (process launches, file writes)
  • Load Balancer HTTP Traffic Logs
  • Connection Server debug logs and blast logs
  • Firewall (ingress and egress traffic logs)
  • UAG blast logs
  • DNS Logs (however we are seeing threat actors use IP addresses for the LDAP call backs)

Vmware Horizon Log Levels

In the lab we observed the following:

  • By default the UAG log level was set to INFO
  • Debug logging is set on the connection server (assumed based on filename)

With INFO logging on the UAG we weren’t able to determine if a malicious payload had been sent. In the ESMANAGER log successful connections were not logged.

We could see the connection on the backend connection server however we could not determine this was a malicous payload in the default logging configuration.

in DEBUG mode we could see the connections and PALOADS on the UAG.

Exploitation Entry Points

There are at least two pages on the HTML Access services that are vulnerable when ther server are unpatched:

  • /broker/xml
  • /portal/info.jsp

Please note we’ve had mixed results with /brokes/xml which may be build version specific.

Threat Intel

Micosoft has reported DEV-0401 using Log4Shell in relation to ransomware activity:

Advisory

This post will likely be updated, it’s not a step by step of how to find all the evil but it hopefully will help identify malcious activity seen in the real world. We will update this if new intel comes in.

Also if you want to wite nicer detections feel free 🙂 these are just examples and are by no means the only ways to do this!

Exploitation Tutorial

When people have had more time to patch and the landscape looks better we will blog how to exploit this and talk about why the currently known backdoor has some limitations due to the service architecture and how the backdoor has been created. It’s important to share exploitation knowledge but there are things to consider, if you are sharing exploitation without detection rules / tools this creates a risk to people. VMware products and services are leveraged by organisations worldwide and the log4j RCE on hoirzon let alone the backdoor can be leveraged for high impact actions by threat actors.

Thanks

Thanks to everyone in the community and industry who has and is helping, thanks to all of those who have stayed up late, missed familty events and generally been super helpful either indirectly or directly. There are lots of people involved in this world who everyday work to keep people safe! (also to my friends and family who have put up with my not being round much!).

Defense

CVE-2021-22005 – vCenter RCE

Introduction

Protecting admin interfaces is a really good idea, network segmentation however is one thing that many organisations struggle with. Most networks are what we call flat. They may be carved up into VLANs but generally speaking, in a lot of networks if you are “inside” then you have full access across the TCP/IP space.

Now here we are talking about the internal attack surface, so a threat actor would need network routable access which should not be the case for things like vcenter interfaces from the internet, however it appears that’s not really exactly how the world works.

Let’s look in Shodan! Read more “CVE-2021-22005 – vCenter RCE”

Defense

Exchange Emergency Mitigation Service with new CU Update

This is a big thing in the Exchange world from my POV! I believe this is dropping sometime today (28th September!)

History

Exchange OWA, ECP etc. are exposed online not only for mailbox access, calendar sharing but also are a requirement for hybrid mode sync capabilities.

Vulnerabilities in Exchange this year (ProxyLogon/ProxyShell) have shown how problematic an attack surface this is. The good news is, Microsoft have created a feature for Exchange to help mitigate these attacks in the future via the EM Service!

Read more “Exchange Emergency Mitigation Service with new CU Update”
Defense

Vulnerability Management – Actually doing it!

Vulnerability Management, Assessments and Vulnerability scanning is sometimes treated a with distain in the Offensive security community, I personally don’t understand that. Vulnerability management is key to inputting into security strategy, architecture, and operations. It’s coupled heavily to many other processes such as:

  • Asset Management
  • Risk Management
  • Patch Management
  • Change & Release Management
  • Security Testing
  • Security Monitoring

Before we start deploying let’s think about some areas for consideration when performing vulnerability scans:

  • Scope
    • Asset/Hosts
      • IP Ranges
      • Hostnames
    • Connectivity
      • VPNs
      • LAN/WAN
    • Device Types and Configuration
      • Domain
      • Workgroup
      • Appliance
      • ICS
      • Printers
      • Network Equipment
    • Unauthenticated View
    • Authenticated View
      • Auth Types
      • Protocols
    • Scheduling
    • Authority to execute
  • Impact
    • Performance
    • Availability
    • Confidentiality
  • Objectives and Outcomes
  • Reporting
    • Information Flow
    • Report Storage and Confidentiality

Read more “Vulnerability Management – Actually doing it!”

A picture containing text, electronics, monitor, indoor Description automatically generated Defense

Razer Privilege Escalation Vulnerability

“And I looked and behold a pale horse: and his name that sat on him was Death, and Hell followed with him.”

Firstly, Kudos to @j0nh4t for finding this!

I woke up this morning to see twitter fun with a LPE discovered in the Razer driver installation. Basically, when you plug a Razer mouse into a Windows machine, it will download (via windows update) and execute a process as system which has user interaction. This interface includes an install path selector, with this a right click + SHIFT (LULZ) on whitespace will allow you to launch a command prompt/PowerShell window (as SYSTEM).

A black shoe on a wood surface

Description automatically generated with medium confidence

A picture containing text, electronics, monitor, indoor

Description automatically generated Read more “Razer Privilege Escalation Vulnerability”

Defense

Why are ransomware attacks so devastating? – Part 1

Introduction

“Ransomware is a major issue!”

Hang on maybe we need to re-phrase that:

“Weak security postures are a major issue!”

or perhaps.. why not both!

I’ve been working with digital technology management for over 20 years, I started out when I was a kid (literally) fixing people’s PCs in their offices, removing malware, improving configurations, writing batch file menus, and playing games. As time has gone on technology has shrunk and continually become more and more of our everyday lives.

Back in 2003 I responded internally to MSBlaster, an SMB worm that had a devastating effect for the time, by today’s standards it was child play, however I remember saying “it’s a good job it didn’t delete everything whilst it was here.” (Or something very similar. Post NACHI/Blaster my friends and I were talking about how worse it could get. Fast forward in time and it’s much worse. Yet when I look at networks, they don’t look very different to how they did back in the 2000s.

Despite a multi-billion-dollar cyber security industry, it seems daily that organisations are succumbing to “cyber-attacks” which commonly include ransomware. Why are they successful and why are they so impactful? Well, let’s take a look! Read more “Why are ransomware attacks so devastating? – Part 1”

Defense

Dumping Credentails with MIMIKATZ and Passing the Hash (PTH)

I kid you not, I forget the commands, so I thought, hey let’s write a small blog post on credential dumping and pass the hash.

To achieve this we need: Debug privileges on a single machine or we need access to a disk that does not have full disk encryption. We also need the password to be re-used.

Mimikatz

Ok for this demo I’m going to run with the out of the box release for Mimikatz on a domain joined windows PC with Defender disabled.

To gain system we launch mimikatz from an admin shell and run:

privilege::debug

token::elevate

Now we are SYSTEM we access a range of high privilege level areas. Read more “Dumping Credentails with MIMIKATZ and Passing the Hash (PTH)”

PetitPotam Guides

From Zero to DA using ‘PetitPotam’

Introduction

Whilst I was on ‘holiday’ (seriously even when on holiday I almost always must do some work!) a few Windows vulnerabilities were published. Great work by Gilles Lionel, Benjamin Delpy and many many others!

Lab Setup

  • A Domain Controller
  • A Separate ADCS Install with Web Enrolment or two DCs one with ADCS installed.
  • A windows Client Device (non-domain joined)
  • An attacker device (I used Kali)

You do not need any domain credentials to conduct this exploit chain, so from a network adjacent unauthenticated position you can get DA with the right circumstances (default configuration). Read more “From Zero to DA using ‘PetitPotam’”

Defense

Ransomware Realities

Everything is much worse now, or is it? 

”The world is burning, the world is burning but then if you look around, it always has been…” 

Computer systems and security go together much like chalk and cheese! Probably sounds a bit odd but miniaturization, consumerization and mobility have put more technology out in the world than we can really comprehend, yet technology security is still dramatically overlooked by most organizations. 

The insane pace of change, the drive for faster, better, cheaper and the reality that it probably isn’t a stretch to say most people (and organizations) do not really understand what ‘secure’ or ‘hardened’ looks like.

Read more “Ransomware Realities”
Defense

Ransomware Defence: Part 2a – Persistence, Privilege Escalation and…

Recap

In Part 1 (Initial Access Defence and Checklist) we looked at ways of hardening your attack surface to defend against initial access. When it comes to ransomware there is a range of elements and variables in the kill chain that need to be successful for the outcomes to be achieved by the criminals. Here we are going to move further into the kill chain to look at further defences. Remember you need to have an “Assume Breach” mindset if you are going to be able to defend against ransomware, that being said, there is a hell of a lot of things you can do for 0 to low investment costs that provide a great ROI. Now some of this is going to be repeated guidance from part 1, that’s ok repetition is good (make sure you are covered from multiple perspectives). Ok let us get to it! Read more “Ransomware Defence: Part 2a – Persistence, Privilege Escalation and Lateral Movement”