Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 

Windows Event Forwarding (WEF)

 

WEF Basics

Using Windows Event Forwarding (WEF) with WinCollect works well for companies that have a strong Windows IT staff.

With WEF, all of the filtering is handled at the subscription level. Every company is different, and some SOC analysts might not be comfortable making global changes to all workstations in a particular subscription. For more granular filtering options at the computer level, subscriptions might not be the best solution.

Group similar devices in your subscriptions. For example, don't place Domain Controllers in the same subscription as workstations.

WEF basics

Windows Event Forwarding (WEF) is a powerful log forwarding solution that is integrated in current versions of Microsoft Windows.

  • WEF allows event logs to be sent, either via a push or pull mechanism, to one or more centralized Windows Event Collector (WEC) servers.

  • WEF is agent-free, and relies on native components integrated into the operating system. WEF is supported for both workstation and server builds of Windows.

  • WEF supports mutual authentication and encryption through Kerberos (in a domain), or it can be extended through the usage of TLS (additional authentication or for non-domain-joined machines).

  • WEF uses an XML-based language to control which event IDs are submitted, suppress noisy events, batch events together, and configure submission frequency. Subscription XML supports a subset of XPath, which simplifies the process of writing expressions to select the events you’re interested in.

Mechanics

WEF can be configured as either a source or a collector-based model. However, this topic focuses on a source-initiated model, where each device forwards logs to a centralized collector. This allows mobile devices, such as laptops, to connect back to the network and forward logs on their own schedule.

A WEF connection requires these basic components:

  • Group Policy Objects (GPOs) to control security auditing and event logging.

  • One or more servers with a configured Windows Event Log Collector service (often referred to as the “WEF Server” or “WEF Collector”).

  • Functional Kerberos for all endpoints (domain) or a valid TLS certificate (non-domain) for the Event Log Collector servers.

  • Windows Remote Management (WinRM) enabled on all workstations and servers that forward events.

  • Firewall rules that permit WinRM connectivity between the devices.

  • GPOs to specify the URL of the WEF subscription manager(s).

  • One or more event log subscriptions. A subscription is a collection of events based on Event IDs or other criteria to tell the endpoints which event logs to forward.

After a workstation first receives the appropriate GPOs, the workstation performs the following actions:

  • Configures security auditing and starts writing to the local event log.

  • Connects to the subscription manager(s) using WinRM, and authenticates using Kerberos or TLS. In both cases, transport-layer encryption is applied.

  • Registers itself in the registry of the Event Log Collector and downloads a list of all relevant WEF Subscriptions.

  • Periodically sends events to the Event Log Collector(s) as defined in the subscription files. Additionally, the workstation connects on a periodic heartbeat.

Limitations

While WEF is a valuable tool, it does have limitations. These limitations should be considered when evaluating a WEF deployment for your organization:

  • Load balancing can be difficult. When using Kerberos, it is very difficult⌂ to effectively load balance the forwarded events between multiple nodes. While events can be forwarded to multiple WEF servers, traditional methods of load balancing the traffic do not work, because the Service Principle Name (SPN) cannot be duplicated.

  • Active diagnosis and troubleshooting is limited. When WEF fails, it is often difficult to diagnose why. There are limited tools for troubleshooting client issues or validating the health of a given node. For more information, see the Troubleshooting.

  • WEF has a steep learning curve. Out of the box, WEF requires multiple components of server and client infrastructure, GPOs, subscriptions, and functional Kerberos. For many organizations, it can be daunting to ensure that all of the prerequisite components are functional, making a WEF deployment difficult.

Troubleshooting

It can be frustrating to stand up logging infrastructure, only to discover that it’s not sending any of the logs you expected it to. Although sometimes unintuitive, there are a few key tools you can use to gain deeper insight into where a breakdown exists.

Start by reviewing the necessary components described in the Mechanics section. Ensure the all required components exist in your environment and are configured correctly. If you’re in the testing phase, consider setting the Subscription Manager refresh interval to a small value, such as 60 seconds. This ensures that logs are offloaded from the clients in a timely basis and reduces the amount of time you must wait for logs to arrive. If you need to force push logs to the Subscription manager, running gpupdate /force from the client also forces a check-in.

Additionally, information about errors or misconfiguration can be found in the Microsoft-Windows- Eventlog-ForwardingPlugin Event Log Channel on each of your clients. This event log is helpful for determining when ACLs are incorrectly configured on event logs, subscriptions are invalid, or logging channels are missing from a host.

On a subscription manager, the Event Viewer tool can help you gain insight into the status of each subscription. Click the Subscriptions option, select a Subscription, and then click Runtime status.

WEF Performance

Microsoft recommends that no WEF collector exceeds 10,000 active source computers, sending more than 10,000 EPS. The registry also supports a mamimum of 100,000 lifetime events, at which point using the Event Viewer to load subscriptions is no longer possible.

Microsoft also recommends that WEF should be used only in a small to medium sized organization. SCOM is recommended for enterprise level event collection service.

WEC server limitations

Three factors limit the scalability of WEC servers. The general rule for a stable WEC server on commodity hardware is “10k x 10k” – meaning, no more than 10,000 concurrently active WEF Clients per WEC server, and no more than 10,000 EPS average event volume.

Disk I/O

The WEC server does not process or validate the received event, but instead buffers the received event and logs it to a local event log file (EVTX file). The speed of logging to the EVTX file is limited by the disk write speed. Isolating the EVTX file to its own array or using high speed disks can increase the number of events per second that a single WEC server can receive.

Network connections

While a WEF source does not maintain a permanent, persistent connection to the WEC server, it does not immediately disconnect after sending its events. This means that the number of WEF sources that can simultaneously connect to the WEC server is limited to the open TCP ports available on the WEC server.

Registry size

For each unique device that connects to a WEF subscription, a registry key (corresponding to the FQDN of the WEF Client) is created to store bookmark and source heartbeat information. If you do not periodically remove inactive clients, this set of registry keys can grow to an unmanageable size over time.

  • When a subscription has more than 1,000 WEF sources connect to it over its operational lifetime (lifetime WEF sources), Event Viewer can become unresponsive for a few minutes when selecting the Subscriptions node, but should function normally afterwards.

  • At more than 50,000 lifetime WEF sources, Event Viewer is no longer an option. You must use wecutil.exe (included with Windows) to configure and manage subscriptions.

  • At more than 100,000 lifetime WEF sources, the registry will be unreadable, and it is likely you must rebuild the WEC server.

Hardware

Hardware benchmarking and performance breakdowns with WEC and EPS are unavailable at this time. Microsoft notes that if you dedicate your EVTX file to its own array you will have much better write performance.