Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 

WinCollect Deployment Discussion

 

There are several topics to take into consideration when deploying an Windows Event Collection solution.

After you decide to gather Windows Event Logs from your endpoints, there are many scenarios that you need to research and discuss. These are discussions to have with your Windows IT group and with the JSA group (if seperate). Work together to answer the following questions:

  • Which endpoints do you need to collect data from?

  • Do you need Windows Event Logs only, or do you need IIS/DHCP/Exchange logs?

  • How many Event Processors/Collectors do you need?

    • How many events per second (EPS) are you licensed for?

    • How many EPS are the Event Processors/Collectors rated for?

  • How many EPS will the endpoints generate?

  • Do certain endpoints require more configuration changes than others?

Example: Collecting from point-of-sale (POS) devices are typically low EPS and rarely require updates. A Domain Controller may require changes to which events are filtered due to high amounts of noisy events.

Terminology

WinCollect Installation Options

You can install WinCollect agents in an environment that is managed by JSA or as a stand-alone agent.

Managed

The WinCollect agent is managed by JSA. Code updates and configuration changes are provided by the JSA console to the agent installed on the Windows endpoint. This option requires TCP communication over port 8413 between the Windows endpoint and JSA. Customers manage what data the agent will collect by adding log sources in the JSA Console.

The agent also requires access to port 514 UDP or TCP to send the syslog data to JSA. In smaller deployments that don't exceed the managed limitations, customers typically choose managed installation to maintain control of WinCollect code and configuration changes.

Current JSA managed limitations

If you want to manage your WinCollect agents and associated log sources using JSA, the recommended limit is 500 agents per console/managed host. For example, to install WinCollect on 1,200 endpoints in managed mode, divide the endpoints between the Console and Event Collectors/Processors.

  • 500 endpoints - Console

  • 300 endpoints - Event Processor/Collector 1

  • 300 endpoints - Event Processor/Collector 2

Stand-alone

In a stand-alone installation, the WinCollect agent is not managed by JSA. The only communication the agent has with JSA is via TCP/UDP over port 514. To upgrade these agents, you need to reinstall the agent or use the patch installer to update the code. Currently the patch installer is a separate installation provided by JSA that includes code updates and the WinCollect Configuration Console.

To make configuration changes, you need to either install the WinCollect Configuration Console GUI tool or make changes directly to the agents' configuration. For large deployments, customers typically chose stand-alone installations, so they can control the installation and configuration using BigFix or Microsoft System Center Configuration Manager.

Note

The WinCollect Configuration Console GUI requires .NET 3.5.

Destinations

The Destination where the syslog is sent. In most cases, this is a JSA Console or Event Collector. WinCollect log sources break destinations down into two groups: Internal and External destinations.

Internal destination - The JSA UI references a host table to see which IP/Hosts are considered Internal or known to the JSA Console. By default, the UI will list all known hosts (Console, Event Processor/Collector), each with an option for UDP/TCP. You can also add a destination to the WinCollect Destination GUI (Click the WinCollect icon on the Admin tab). The hostname/IP is cross-referenced with the host table, and the custom destination is added to either the Internal or External destination list.

External destination - An External destination is a hostname/IP that is not considered a known host by JSA. This can be used to send events outside of the JSA network.

Note

You can select one internal destination and one or more external destinations per agent.

XPath Query

XPath queries are structured XML expressions that you use to retrieve customized events from the Windows event logs.

Use XPath queries to collect events from the Applications and Services event logs. Xpath queries can also be used to filter events that you wish to not collect. For examples and directions about how to generate an XPath query Windows Event Logs.

Event filtering - Need more info

  • NSA Event filter

  • Noise suppression filter

Event Filtering - Need More Info

  • NSA Event filter

  • Noise suppression filter

Collection Methods

Local collection

The WinCollect agent is installed on the endpoint in either managed or stand-alone configuration and collects Windows event logs from the local endpoint. You can use this collection method on Windows hosts that are busy or have limited resources, such as domain controllers. Domain controllers typically have a heavier event per second (EPS) rate than member servers.

You can also use local collection when you don't want to worry about managing credentials and adding/subtracting endpoints as they come online. You must install agents on the endpoints as they are added to their network either manually or via a BigFix or Microsoft System Center Configuration Manager (SCCM) solution. The agent can also be included with a master image so that an agent is up and running when a new endpoint is deployed.

Note

When WinCollect agents collect events from the local host, the event collection service uses the local system account credentials to collect and forward events.

Remote Collection

The WinCollect agent is installed on the endpoint in either managed or stand-alone configuration and collects Windows event logs from the local endpoint and one or more remote endpoints. For remote collection, you must provide login credentials for a user with remote event log access. WinCollect agents that remotely poll other Windows endpoints systems require access to following remote ports:

Table 1: Ports Used for Remote Collection

Port

Protocol

Usage

136

TCP

Microsoft Endpoint Mapper

137

UDP

NetBIOS name service

138

UDP

NetBIOS datagram service

139

TCP

NetBIOS session service

445

TCP

Microsoft Directory Services for file transfers that use Windows share

49152-65535

TCP

Default dynamic port range for TCP/IP

Note

Use Windows Server to perform remote polling whenever you are polling a large number of remote machines.

Note

The MSEVEN protocol uses port 445. You can use the NETBIOS ports (137 - 139) for host name resolution. When the WinCollect agent polls a remote event log by using MSEVEN6, the initial communication with the remote machine occurs on port 135 (dynamic port mapper), which assigns the connection to a dynamic port. The default port range for dynamic ports is between port 49152 and port 65535. To allow traffic on these dynamic ports, enable and allow the two following inbound rules on the Windows server that is being polled:

  • Remote Event Log Management (RPC)

  • Remote Event Log Management (RPC-EPMAP)

The MSEVEN6 protocol exposes RPC methods for reading events in both live and backup event logs on remote computers. This protocol was originally made available for Windows Vista and replaces the MSEVEN protocol.

Windows Event Forwarding (WEF)

The WinCollect agent can leverage the built-in Microsoft function Windows Event Forwarding (WEF). WEF reads any operational (security) oHardware benchmarking and performance breakdowns with WEC and EPS are unavailable at this time. Microsoft notes that if you dedicate your EVTX file to its own array you will have much better write performance.r administrative (sysmon) event log on a device in your organization and forwards the events you choose to a Windows Event Collector (WEC) server. You can install the WinCollect agent on the WEC Server and collect from the forwarded event log. Before sending these forwarded events to JSA, the agent packages them in such a manner so they will appear as they are coming directly from each the endpoints. JSA automatically creates log sources for each endpoint that is sending logs to the Windows Event Collector (WEC) server.

Highlights of WEF:

  • Events can be pushed or pulled from the WEC Server

  • Can be configured via GPO

  • Uses Windows Remote Management (Kerberos) to prevent man in the middle

  • Recommended to target certain event logs and event ids (use Xpath)

  • Events are collected to one central event log file (EVTX file) that WinCollect can poll

Fore more information on Windows Event Forwarding, see these sources:

What are the WEC server’s limitations?

There are three factors that limit the scalability of WEC servers. The general rule for a stable WEC server on commodity hardware is 10k x 10k, meaning, no more than 10,000 concurrently active WEF Clients per WEC server and no more than 10,000 events/second average event volume.

Disk I/O

The WEC server does not process or validate the received event, but rather buffers the received event and then logs it to a local event log file (EVTX file). The speed of logging to the EVTX file is limited by the disk write speed. Isolating the EVTX file to its own array or using high speed disks can increase the number of events per second that a single WEC server can receive.

Network Connections

While a WEF source does not maintain a permanent, persistent connection to the WEC server, it does not immediately disconnect after sending events. This means that the number of WEF sources that can simultaneously connect to the WEC server is limited to the open TCP ports available on the WEC server.

Registry size

For each unique device that connects to a WEF subscription, a registry key (corresponding to the FQDN of the WEF Client) is created to store bookmark and source heartbeat information. If this is not pruned to remove inactive clients, this set of registry keys can grow to an unmanageable size over time.

  • When a subscription has more than 1,000 WEF sources connected to it over its operational lifetime, also known as lifetime WEF sources, Event Viewer can become unresponsive for a few minutes when selecting the Subscriptions node in the left-navigation, but will function normally afterwards

  • At more than 50,000 lifetime WEF sources, Event Viewer is no longer an option. You must use wecutil.exe (included with Windows) to configure and manage subscriptions.

  • At more than 100,000 lifetime WEF sources, the registry becomes unreadable, and it is likely you must rebuild the WEC server.

Note

Minimum recommendations for a WEC servier is 4-8 vCPU and 16+ GB RAM. It is also recommended to collect "rendered" events, which is more memory intensive than vCPU.

Note

For Windows XP with SP2, Windows Server 2003 with SP1, Windows Server 2003 with SP2, or Windows Server 2003 R2, WS-Management 1.1 is not installed by default, which is a minimum required for subscriptions to work. You must install this or a later version for event forwarding to work on these systems

What are the WEC server’s limitations?

There are three factors that limit the scalability of WEC servers. The general rule for a stable WEC server on commodity hardware is 10k x 10k, meaning, no more than 10,000 concurrently active WEF Clients per WEC server and no more than 10,000 events/second average event volume.

Disk I/O

The WEC server does not process or validate the received event, but rather buffers the received event and then logs it to a local event log file (EVTX file). The speed of logging to the EVTX file is limited by the disk write speed. Isolating the EVTX file to its own array or using high speed disks can increase the number of events per second that a single WEC server can receive.

Network Connections

While a WEF source does not maintain a permanent, persistent connection to the WEC server, it does not immediately disconnect after sending events. This means that the number of WEF sources that can simultaneously connect to the WEC server is limited to the open TCP ports available on the WEC server.

Registry size

For each unique device that connects to a WEF subscription, a registry key (corresponding to the FQDN of the WEF Client) is created to store bookmark and source heartbeat information. If this is not pruned to remove inactive clients, this set of registry keys can grow to an unmanageable size over time.

  • When a subscription has more than 1,000 WEF sources connected to it over its operational lifetime, also known as lifetime WEF sources, Event Viewer can become unresponsive for a few minutes when selecting the Subscriptions node in the left-navigation, but will function normally afterwards.

    At more than 50,000 lifetime WEF sources, Event Viewer is no longer an option. You must use wecutil.exe (included with Windows) to configure and manage subscriptions.

  • At more than 100,000 lifetime WEF sources, the registry becomes unreadable, and it is likely you must rebuild the WEC server.

Note

Minimum recommendations for a WEC servier is 4-8 vCPU and 16+ GB RAM. It is also recommended to collect "rendered" events, which is more memory intensive than vCPU.

Note

For Windows XP with SP2, Windows Server 2003 with SP1, Windows Server 2003 with SP2, or Windows Server 2003 R2, WS-Management 1.1 is not installed by default, which is a minimum required for subscriptions to work. You must install this or a later version for event forwarding to work on these systems.

Events Per Second (EPS)

Collecting event logs from the Windows endpoints could result in a major increase to the overall incoming Events Per Second to JSA.

Calculate EPS

Events per second (EPS) is the number of log or system events that are generated by a device every second.

Agent EPS rates

Local collection - Local collection EPS

Remote collection - 10,000 EPS total across 500 remote endpoints

Remote polling channels

WinCollect can use either the MSEVEN or MSEVEN6 protocol to query for events. Each time the WinCollect agent polls a remote Windows host, the agent creates a channel to complete the query for every event log type being collected. The number of remote hosts polled can impact performance when too many channels are opened simultaneously. Too many queries per second can cause remote procedure call (RPC) errors when WinCollect attempts to remotely poll a Windows host. The WinCollect agent must open a channel to read the events from the remote event viewer based on the log source configuration set by the administrator.

Note

Do not exceed 30 channel queries per second with a WinCollect agent.

A channel is created for each path in the remote event viewer when a query is created. For example, to poll the Application, System, and Security log, the WinCollect agent must open three channels to the remote host. Each log type that is specified within the log source or within an XPath query creates a channel when an endpoint is polled, as a path needs to be created to that location.

Administrators who experience collection issues with remote polling can ensure that the agent is creating 30 queries per second or less to prevent RPC errors when remote polling. Use the following formula to determine the number of queries made per second from the WinCollect agent:

(Endpoints x Channels) ÷ Polling interval(s) = Queries per second

For example, a WinCollect agent is collecting Application, System, and Security events from 300 endpoints every 10 seconds. In this example, the number of queries per second exceeds the recommended limit of 30 queries per second. When the query limit is exceeded, it can generate a Syslog LEEF message from the WinCollect agent: RPC server is unavailable.

(380 Endpoints x 3 Channels) ÷ 10 Polling interval(s) = Queries per second

Planning

Work with your Windows IT group and the JSA group to answer the following questions to plan your WinCollect deployment.

  • How many endpoints do we need to collect data from?

    • Which Windows operating systems do these endpoints use?

  • Identify the high value servers (i.e. Domain Controllers / Web Servers etc.).

    • Are you allowed to install an Agent on these servers?

    • Do you want to collect additional logs on these servers (i.e. IIS logs / DHCP logs)?

    • What type of EPS do these high value servers generate? These servers will generate the highest EPS rates.

Note

WinCollect is not supported on versions of Windows that have been moved to End Of Life by Microsoft. After software is beyond the Extended Support End Date the product might still function as expected, however, Juniper will not make code or vulnerability fixes to resolve WinCollect issues for older operating systems

What data do we need to collect?

Which Event logs do you need to collect? Apart from the standard Windows logs (i.e. Application / System / Security), do you need data from applications and services logs such as Powershell or Sysmon? Application and services logs are collected by providing an Xpath to the WinCollect Agent.

  • In addition to event logs, identify any of the following logs you may want to collect:

    • IIS

    • IAS

    • ISA

    • DHCP

    • DNS Debug

    • Exchange

    • NetApp

    • Juniper SBR

    • File Forwarder (generic log file forwarder)

Note

All of these logs can be collected by a local Agent or a Remote agent.

Where are these endpoints located?

  • Are these computers on a domain or are they off of the network?

    Note

    Configuring a WEF setup is much easier when the computers are on a domain.

  • Line of Sight - Which Console/Event Collector/Event Processor do the endpoints have visibility to?

How many events per second (EPS) will my endpoints generate?

After you have a list of computers you want to collect from, you need to identify the volume of EPS these endpoints might generate. It is also important to estimate the Peak EPS for the endpoints. Your Event Collector might handle 40k EPS, but when all of your employees login at 8 am, is this EPS going to spike to 80k EPS? And if so, for how long? Will your JSA appliances handle these spikes, or do you need to spread out the load across 1 to many ECs?

One option is to throttle the agents to a certain EPS, so the Agent only sends out a certain amount of events, regardless of what it collects. In this case, the Agent buffers the extra events to disk until the EPS rate decreases. This allows you to know for certain the total EPS that might be sent to your EC at any given time. If you select this option, you need to understand EPS rates different endpoints will generate. For example, you wouldn't want to throttle a Domain Controller to 2 EPS, as this server may send at a rate of 5-10 EPS. Then the Agent would always be behind.

Table 2: Typical EPS rates for endpoints

Endpoint type

Average EPS

Peak EPS

Employee Endpoints Desktops & Laptops

0.005

0.05

Windows Domain Server

5 - 10

350

Web Servers (IIS, Apache)

5 - 10

350

Windows DNS Server

0.5

5

Database Server

0.5

10

Note

Estimates include security event logs only.

Potential Deployment Strategies

After you know which endpoints you want to collect from, you need to plan how to collect this data. Deployment strategies vary from customer to customer. The following use cases and options are designed to help guide your deployment.

USE Case: Large Retail Point of Sale (POS) Collection

Customer needs to collect Windows Security Event logs from all Point of Sale devices at each of their retail stores.

Architecture

Device count - 4,500 POS devices

Retail stores - 500 (~ 9 POS per store)

JSA deployment - 1 Console, 2 Event Collectors

Visibility

No visibility of POS devices outside retail store

Option 1 - POS visibility to local retail store Domain Controller

Proposed solution

See section Windows Event Forwarding - Local WEF Server in Retail Point of Sale (POS) Deployment Strategies for more information.

Advantages

POS devices can be brought up and down and will start sending events to WEF Collector once they receive GPO Update. No need to manage stores POS devices once GPO is configured.

Disadvantages

  • IT Group must monitor 500 WEF Collection servers to make sure WEF is configured and working.

  • GPO must be configured for each retail store. Must ensure each POS is pointing to the correct WEF Collector.

  • WEF must be configured on a Domain Controller.

Option 1a - POS visibility outside of retail store

Proposed solution

See section Windows Event Forwarding - Remote WEF Server in Retail Point of Sale (POS) Deployment Strategies for more information.

Advantages

  • POS devices can be brought up and down and will start sending events to WEF Collector once they receive GPO Update. No need to manage stores POS devices once GPO is configured.

  • 1 WEF Collector can be configured to handle traffic from all retail stores. Collector can handle 10k endpoints. Customer must validate EPS rates are not higher than the supported 10k EPS.

  • Gather logs from 1 server to troubleshoot any issues.

Disadvantages

POS devices need visibility outside of the retail store.

Option 2 - WinCollect Agent installed on each POS device

This includes the option to make WinCollect part of the "Golden Master Image." This allows the customer to spin up new registers at any time and have WinCollect configured and ready to send events. This option requires the Agent to be installed on the POS device to have visibility into a JSA appliance onsite at the retail store, or to have visibility to a JSA Event Processor/Collector outside of the retail location.

Proposed solution

See section WinCollect Agent per POS in Retail Point of Sale (POS) Deployment Strategies for more information.

Advantages

  • After the initial work of creating a WinCollect image is complete, bringing POS devices on and off line would be part of standard process.

  • No management necessary to install agents on each POS device.

Disadvantages

  • Gathering logs from POS to troubleshoot issues might take time.

  • Updating WinCollect agent to new versions requires new testing and updates to master image.

Option 3 - WinCollect Agent installed on store local domain controller or server

If this server has visibility to JSA, it could be installed in Managed Mode. The agents could then be configured to remote poll each of the POS devices. With 500 retail stores, the customer could install each Agent in Managed mode, divided between the Console and Event Collectors/Processors. Management of the solution would be quite difficult and require a lot of maintenance as new POS are brought on and off line.

Proposed solution

See section WinCollect Agent Remote Poll POS in Retail Point of Sale (POS) Deployment Strategies for more information.

Advantages

  • Management of Agents from the JSA Console.

  • Easier to make changes to what events are collected and what data is filtered.

Disadvantages

  • Customer would be responsible for adding/subtracting POS devices that Agent is remote polling. This would increase during high retail volume periods.

USE Case: Large Endpoint Deployment

Customer needs to collect Windows Security Event logs and Sysmon event logs from all of the endpoints in their company along with specific logs on their Application, Web, and Mail servers.

Architecture - Device count

  • 15,000 endpoints (workstations and laptops)

  • 50 servers (Domain controllers, web, mail, database)

Architecture - JSA deployment

1 Console, 10 Event Collectors

Option 1 - Windows Event Forwarding - Several event collectors

Proposed solution

See section Windows Event Forwarding Full Deployment in Large Endpoint Deployment Strategies for more information.

Advantages

  • Devices can be brought up and down and will start sending events to WEF Collector after they receive GPO Update.

  • Managing minimal WinCollect Agents.

  • Events can be filtered in the WEF Subscription, reducing a lot of the Event Log noise before the data is sent to JSA.

Disadvantages

  • IT Group must monitor WEF servers to ensure they do not exceed EPS thresholds.

  • GPO must be configured and maintained by IT Group.

    - Addition / Subtraction of Event IDs is handled by the subscription.

Option 2 - Mix of WEF and Managed WinCollect

Proposed solution

See section Windows Event Forwarding Managed Deployment in Large Endpoint Deployment Strategies for more information.

Advantages

  • POS devices can be brought up and down and will start sending events to WEF Collector once they receive GPO Update. No need to manage stores POS devices once GPO is configured.

  • Managing minimal WinCollect Agents.

  • Events can be filtered in the WEF Subscription, reducing a lot of the Event Log noise before the data is sent to JSA.

  • High EPS servers are handled by WinCollect Agent, reducing the load on the WEF Collector.

  • Configuration and code updates can be deployed from JSA Console.

Disadvantages

  • GPO must be configured and maintained by IT Group. (For endpoints, this should be minimal).

    - Addition / Subtraction of Event IDs is handled by the subscription.

Option 3 - Stand-alone deployment

Proposed solution

See section Stand-alone deployment in Large Endpoint Deployment Strategies for more information.

Advantages

  • Installation of Agents can be controlled by Big Fix or Microsoft SCCM tool.

  • EPS rates should be of no concern with local collection on each endpoint.

  • Upgrade issues can be resolved by reinstalling the Agent.

Disadvantages

  • Must rely on Big Fix / SCCM groups to manage installations.

  • Changes to WinCollect configurations are typically slow.

  • Gathering logs from endpoints to troubleshoot issues might take time.

  • Updating WinCollect agents to new versions requires new testing and updates to master image.

  • No control of Agents in JSA.

Option 4 - Remote-poll deployment

Maximum of 500 Remote Polls per Agent = 30 Agents collecting from 500 endpoints each to collect 15,000 Endpoints.

To avoid reaching the maximum, start with 500 remote polls per Agent, depending on the strength and reliability of the network. If the Agent has quick response times from each of the remote polled servers, you can remote poll more endpoints per Server. Even in a remote poll deployment, run local collection on the high value servers.

  • JSA Managed

  • 30 Agents remote polling 15,000 endpoints

  • 50 Agents local collection

Proposed solution

See section Remote-poll deployment in Large Endpoint Deployment Strategies for more information.

Advantages

  • Fewer agents to Manage.

  • Endpoints can be added/subtracted using JSA bulk Management.

Disadvantages

  • If Servers are turned off or are non-responsive, Agent continues to poll.

  • Must apply correct permissions and firewall rules to all endpoints so they are reachable from Agent server.