Threat Hunting – Pursue Your Adversaries

By Brett Williams, Sales Engineering Manager – Asia Pacific & Japan, Carbon Black

Instead of sitting back passively and waiting for cyber attackers to set off alarms, organisations should be pursuing them like a cheetah hunting for its next meal. We know the attackers are out there – they are perpetually trying to break in, and many are succeeding.

The challenge is to start hunting them to find the shreds of evidence they invariably leave behind. First an organisation needs to build a hunting team. Team members should be knowledgeable about the internals of the operating systems (OS) found on their endpoints. The OS will usually be Microsoft Windows, but also Apple Mac OS and perhaps Linux. Threat hunters need to know how these operating systems work at a detailed level, including the following:

  • OS process tree structure
  • Files used by the OS
  • Registry used by the OS (Windows only)

Expertise at this level of detail is important because malware operates within these domains and makes subtle changes to the OS. Threat hunters need to understand what to look for and what ‘normal’ looks like at the business application and humanactivity level — it’s not just about packets on the network and processes in the OS, so anomalies will be more apparent. Those anomalies are the primary sign that malware is lurking on endpoints.

Making the time to threat hunt

It might be necessary to carve out time from the work schedules of existing staff for threat hunting. Depending on an organisation’s size, the time spent threat hunting may vary. In part, it depends a lot on security posture and risk tolerance.

Start with two to four man/hours a week dedicated to hunting. When the results emerge, adjust as needed. It is important to see early results from hunts, to show a return on the time investment.

The chosen threat hunters need to have passion! They must think like predators and have a hunger to hunt adversaries. After that important characteristic comes other trained skills including:

  • Operating system internals: This skill is critical. Threat hunters need to understand the rules and practices of process management and the file system operation and network communication in each operating system in use.
  • Endpoint application behaviour: It’s important to understand how any locally used applications function on the organisation’s endpoints.
  • Threat hunting tools: The team needs to understand thoroughly how to use the tools at their disposal, to maximise their effectiveness.
  • Incident response procedures: They need to know what steps to take when they discover signs of intrusion, then preserve that evidence for potential future legal proceedings.

Put the necessary processes in place

Threat hunting needs to be a structured, longterm effort. There must be a vision for what threat hunting is about and how it works with other IT and IT security processes. This means learning several things, including:

  • Endpoint baselines: The need to hone continuously threat hunters’ knowledge of what constitutes ‘normal’ in the endpoints, so anomalies can be recognised faster. The local context that humans have makes all the difference in detection.
  • Improving hunting tools, practices, and skills: Hunts must become more effective over time, and threat hunters must learn quickly from the seasoned warriors on their team.
  • Improving response: Finding prey requires response that includes containment and remediation. Mainly, this means doing these things more accurately and faster.
  • Improving skills: Threat hunters need to improve their skills and knowledge, not just from threat hunting itself, but from continuing education on ethical hacking, system and network internals, and incident response.

Put the necessary tools in place

Threat hunting is a manmachine activity — it cannot be done with just people or just machines. Without threat hunting tools, there’s no hunt.

Endpoints are today’s battleground where intrusions into enterprises begin. Endpoints are the attackers’ crown jewels, and they’re used to make a landing into an environment. While the data that attackers seek lives on servers, access to servers starts with endpoints.

Endpoint visibility is the ability to capture, in detail, the activities going on inside of every endpoint. If an organisation allows Bring Your Own Device (BYOD), it should achieve this visibility on those machines, too.

Include information about every process, including its parents and children, as well as every file that’s created, read, written and removed, plus network activity. This information needs to be accessible across the entire organisation, so threat hunters can quickly understand what anomalous activity is going on at any place and time.

Another important aspect of endpoint visibility is known as retrospection, which is the ability to hunt back in time. For example, mine the data for suspicious activity that took place not just yesterday, but last week, last month or even earlier.

In addition to endpoint visibility, having access to network event data is essential. Sometimes the first sign of intrusion is in the command and control (C&C) network traffic from a bot that has already compromised an endpoint. Intrusion prevention systems (IPS), web filtering, firewall logs, packet capture and netflow tools are good sources for obtaining this data. Threat hunters must be able to reference one or more of these tools from time to time, to better understand what’s going on in the network.

Threat intelligence feeds inform threat hunters of the new tools and techniques that attackers are using against other organisations, as well as the domains and IP ranges they may be using. Threat intel feeds are often high volume and delivered in structured formats such as Structured Threat Information Expression (STIX) and OpenIOC and Cyber Observable Expression [CybOX]. All these are designed to be fed into an organisation’s security information and event management (SIEM) system, endpoint detection and response tools (EDR) or other threat management platform.

Remember that threat hunting is a manmachine activity. In many respects, there is a high volume of information on threats and activities in your environment. To capitalise on this information, the threat team needs to understand what tools they are using and where there might be opportunities to integrate them.

A prime example is the fusion of endpoint data, SIEM data and threat intel feeds. By themselves, they’re useful, but when fused together they become invaluable. APIs should also be used where available, so that threat hunters can consume this data and get it into their other systems.

Know the environment

Successful threat hunters need to know as much about their environment as possible, so they can better sense what’s normal and what’s abnormal. As their hunts progress, they begin to have an intimate familiarity with their environment.

Threat hunters spend much of their time observing and becoming more familiar with normal routing events in their environments. However, they also need to be familiar with the organisation’s architecture: networks, systems, tools and applications. It is key they understand this independently of their threat hunting, because anything they might observe in the environment may or may not be normal. What they find and consider normal may include things that aren’t allowed.

Threat hunters need to know what their goals are. Depending on the attackers and their objectives, this could be information like customer or employee data, or it could be critical assets such as public facing web servers. They need to know all these high value targets (HVTs) – and they need to understand how attackers might go about attacking them.

Hunters also need to know how attackers are likely to try to break into their environments. This is part gut feel and part knowing the environment:

  • Architecture: Attackers will seek out the weak spots in an organisation’s architecture and data flows. Assisting them discover whatever valuable data they’re seeking and how to extract it unnoticed.
  • Security posture: Attackers will target an organisation’s weak spots. They discover these through simple techniques like port scanning to find unpatched and vulnerable systems. Therefore, threat hunters need to know where those weak spots are.
  • People: An organisation’s security culture is a great indicator of its vulnerability. Attackers will be able to gauge how easy it is to lure employees into clever social engineering, phishing or spear phishing campaigns, whether they’re purely online or on-site.
  • Threat intel: Understanding how attackers are targeting other organisations gives threat hunters a better idea of how attacks might target their own organisation. While they will be creative and unpredictable at times, attackers are creatures of habit, apt to use tools and techniques that have worked for them in the past. Just as organisations tend to protect themselves in similar ways, attackers are likely to attack in similar ways.

Threat hunters need to know their environment inside and out: How does everything work, where are the gaps and weak spots, and where are the risks? They need to think like attackers, so they can better anticipate threats and stop attacks early.

Finally, Threat hunting is becoming a part of information security table stakes: the essential tools and practices required by all organizations. Threat hunting will soon be a part of the due care for information protection expected by customers, regulators, and the legal system.

To learn more about threat hunting, download the guide: “Threat Hunting for Dummies.”

Leave a Reply