Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Navigation
Guide That Contains This Content
[+] Expand All
[-] Collapse All

    Understanding the Security Director Log Collector Deployment Modes

    You can set up log collector(s) in both the VM and JA2500 appliance. You can configure log collector as an All-in-One node for small-scale deployments. For easy scaling, begin with a single Log Receiver node and Log Storage node, and incrementally add Log Storage nodes as your needs expand. You can add a maximum of one Log Receiver node and three Log Storage nodes.

    For a VM environment, a single OVA image is used to deploy the All-in-One, Log Receiver, and Log Storage nodes. The image presents a configuration script after you log in. At deployment, you must select the memory and CPU configuration values, as is appropriate for the role of the VM.

    For JA2500 deployments, a single ISO image is used to install the All-in-One, Log Receiver, and Log Storage nodes. The image presents a configuration script after you log in.

    Requirements

    To use the Junos Space Security Director Logging and Reporting module, your system must meet the following prerequisites:

    • A single Security Director image installs Security Director, Log Director, and Security Director Logging and Reporting applications.
    • You must deploy the Log Collector for receiving and viewing logs.
    • The Junos Space Network Management Platform VM must be deployed on the ESX server.
    • The Platform must be configured with Ethernet Interface eth0 and management IP addresses. Note that the platform can also run on a JA2500 appliance.
    • The Junos Space Network Management Platform must be up and running, and you must be able to log in to the Junos Space Network Management Platform user interface.
    • The following ports are required for functioning of log collector and these ports must be open between the space server and the Log Collector:
      • Port 8004 (TCP)—Used for communication between the space and the node agent.
      • Port 8003 (TCP)—Used for log data queries.
      • Port 9200 (TCP)—Used for configuring the Log storage.
      • Port 9300 (TCP)—Used for communicating across elasticsearch cluster.
      • Port 4567 (TCP)—Used for communication between the log receiver and log storage.
      • Port 514 (TCP)—Used for syslog receiving.
      • Port 514 (UDP)—Used for syslog receiving.
      • Port 22 (TCP)—Used for ssh connectivity.
    • The following ports are not required for functioning of Log Collector, but these are used by other peripheral services:
      • Port 5671 (TCP)
      • Port 32803 (TCP)
      • Port 32769 (UDP)

    Specifications for Deploying a Log Collector Virtual Machine on an ESX Server

    You can use Table 1 and Table 2 to decide if you require a single Log Collector or multiple Log Collectors.

    Table 1 and Table 2 provide the VM configuration recommended for the log collection to work effectively. It lists the required specifications for deploying a Log Collector VM on an ESX server for various sustained events-per-second (eps) rates. The eps rates shown in table were achieved in a testing environment. Your results might differ, depending on your configuration and network environment.

    Table 1: Using Solid State Drives (SSD)

    Setup

    Log Receiver Node

    Log Storage Node

    Total Nodes

    Number of Nodes

    CPU

    Memory

    Number of Nodes

    CPU

    Memory

    4K eps

    1

    4

    16 GB

    -

    -

    -

    1

    10K eps

    1

    8

    32 GB

    1

    8

    64 GB

    2

    20K eps

    1

    8

    32 GB

    2

    8

    64 GB

    3

    Table 2: Using Non-Solid State Drives (SSD)

    Setup

    Log Receiver Node

    Log Storage Node

    Total Nodes

    Number of Nodes

    CPU

    Memory

    Number of Nodes

    CPU

    Memory

    3K eps

    1

    4

    16 GB

    -

    -

    -

    1

    10K eps

    1

    8

    32 GB

    2

    8

    64 GB

    3

    20K eps

    1

    8

    32 GB

    3

    8

    64 GB

    4

    Note: VMs with 64GB memory gives better stability for the log collection.

    Log Collector Deployment Nodes

    Table 3 shows different node types in which the Log Collector can be deployed.

    Table 3: Log Collector Deployment Nodes

    Node Type

    Description

    All-in-One Node (Combined deployment)

    • Both log receiver and log storage nodes run on the same VM or JA2500 appliance.
    • Supports eps of up to 3,000 with spinning disks and 4,000 with SSD drives.
    • Suitable for demos and small-scale deployments.

    Log Receiver Node (Distributed deployment)

    This node receives system logs from SRX devices. SRX devices must be configured with the log receiver node IP to send system logs. Upon configuration this node parses and forwards logs to log storage node. You must provide the IP address of the log storage node while configuring this node.

    Log Storage Node (Distributed deployment)

    This node analyzes, indexes, and stores the system logs. It receives the system logs from log receiver node.

    Integrated

    It is installed on Space node itself and it works as both the Log Receiver node and Log Storage node, similar to an All-in-one node.

    Note: Using vSphere Client version 5.5 and previous versions, you cannot edit the settings of virtual machines of version 10 or later versions. For more details, see VMware Knowledge Base.

    Note: You can only configure the IP address of all Log Collector nodes by using the configuration script. If an IP address is configured manually, the Log Collector node cannot be added to Security Director.

    Storage Requirements

    The total storage required for retaining X number of days at a given eps rate is:

    eps * 0.155 * X = (in GB)

    For example, the storage requirement for 7 days at 500 eps is 500 * 0.155 * 7 = 542 GB, with a +20% margin. The storage space is allocated and equally distributed to the log storage nodes.

    Note: The logs get rolled over under the following scenarios:

    • Time-based rollover—Logs that are older than 45 days are automatically rolled over, even if the disk space is available.
    • Disk size-based rollover—Older logs get rolled over when the disk size reaches 400GB.

    Deploying Log Collector as an All-in-One Node

    An All-in-One node acts both as a Log Receiver and Log Storage node. For a VM environment, a single OVA image is used to deploy the All-in-One, Log Receiver, and Log Storage nodes. The image presents a configuration script after you log in and you must select All-in-One to configure the node. For JA2500 deployments, a single ISO image is used to install the All-in-One, Log Receiver, and Log Storage nodes. During setup, you can configure the node as an All-in-One node.

    Figure 1 shows the deployment example using an All-in-One node.

    Figure 1: Using an All-in-One Node

    Using an All-in-One Node

    Deploying Multiple Log Collectors

    If you have a scenario where you require more log reception capacity or eps, you can add multiple logging nodes. Multiple logging nodes provide higher rates of logging and better query performance. You can add a maximum of one Log Receiver node and three Log Storage nodes.

    For a VM environment, a single OVA image is used to deploy a Log Receiver node and a Log Storage node. The image presents a configuration script after you log in. During setup, you can configure the node as either a Log Receiver or Log Storage. At deployment, the user must select the memory and CPU configuration values, as is appropriate for the role of the VM or appliance.

    For JA2500 deployments, a single ISO image is used to install the Log Receiver and Log Storage nodes. During setup, you can configure the node as either a Log Receiver or a Log Storage node.

    The use of multiple logging nodes provide the following benefits:

    • Improves performance
    • Provides high-volume log storage on a virtual device
    • Provides scalability for log collection and management

    Figure 2 shows the deployment example using multiple node for up to 10K eps.

    Figure 2: Using Multiple Node for up to 10K eps

    Using Multiple Node for up to
10K eps

    Figure 3 shows the deployment example using multiple node for greater than 10K eps.

    Figure 3: Using Multiple Node for Greater than 10K eps

    Using Multiple Node for Greater
than 10K eps

    Deploying Log Collector VM on an ESX Server

    To deploy the Log Collector on an ESX Server:

    1. Download the latest Log Collector open virtual appliance (OVA) image from here.
    2. Using vSphere or vCenter, deploy the Log Collector OVA image onto the ESX server.
    3. Edit the CPU and memory as per the system requirement for the required eps.

      Note: Log Collector VM contains a Virtual Appliance Management Infrastructure (VAMI) agent. The agent allows the VM to use the configuration from the ESX server based on the server configuration.

    4. Power on the Log Collector VM.

      A configuration script lets you choose the node type and configure the network settings.

    5. Use the default credentials to log in to Log Collector. The username is root and password is juniper123.
    6. Change the root password of the VM.
    7. Select one of the following node types:
      • Enter 1 to deploy Log Collector as All-in-One node.
      • Enter 2 to deploy Log Collector as a Log Receiver node.
      • Enter 3 to deploy Log Collector as Log Storage node.
    8. Configure your network settings using the same wizard.

    Deploying Log Collector VM on a KVM Server

    Starting with Security Director Release 15.2R2, you can deploy Log Collector on a kernel-based virtual machine (KVM) server installed on CentOS Release 6.5.

    Prerequisites

    The prerequisites to deploy a Log Collector on a KVM server are as follows:

    • Knowledge about configuring and installing a KVM server.
    • The KVM server and supported packages must be installed on a machine running CentOS with the required kernels and packages. For information about installing a KVM server and supported packages on a machine running CentOS, see http://wiki.centos.org/HowTos/KVM.
    • The Virtual Machine Manager (VMM) client must be installed on your local system.
    • The Bridge Interface must be configured according to your environment and you must have at least two static IP addresses that are unused.

    Note: You can deploy the Log Collector virtual appliance on a KVM sever by using other virtual machine clients other than VMM. However, Juniper Networks does not provide support for installing the Junos Space virtual appliance using clients other than VMM.

    To deploy Log Collector VM on a KVM server:

    1. Download the Log Collector KVM image from here on the KVM host and extract the tgz file, which contains the system.qcow2 and data.qcow2 files.
    2. Launch the VMM client by typing virt-manager from your terminal or you can search from the System Tools menu.
    3. Select File > New Virtual Machine to install a new virtual machine.

      The new VM dialog box appears and displays Steps 4 to 8.

    4. Select Import existing disk image and click Next.
    5. Click Browse and then select the system.qcow2 file.
    6. Select Linux as the operating system and the versions as Red Hat Enterprise Linux 6.6 or later.
    7. Click Forward.
    8. Set CPU settings as 4 , and then select or enter the memory (RAM) value as 16384 MB (minimum).
    9. Click Forward.
    10. Edit the Name field, select or set up the network for each bridge or interface configured, and select the Customize Configuration Before Install option.
    11. Click Finish.
    12. Select the Storage option from the left navigation on the Add New Virtual Hardware window, and then click Add Hardware.
    13. On the Storage Window:
      • Select Select managed or other existing storage and choose the data.qcow2 file.
      • Select the storage format as qcow2 under Advanced Options.
      • Click Finish.
    14. Click Begin Installation to start the Log Collector VM.
    15. After the installation, you can configure the IP, name server, and time zone.

    Installing Log Collector on the JA2500 Appliance Using a USB Flash Drive

    To install the Log Collector on the JA2500 appliance using a USB flash drive, you must create a bootable USB flash drive, install the Log Collector node using the USB flash drive, and add the Log Collector node to Security Director.

    Create a Bootable USB Flash Drive

    Before creating a bootable USB flash drive, download and install Rufus software on your system.

    1. Plug the USB storage device into the USB port of a laptop or PC.
    2. Download the Log collector ISO image from the download site.
    3. Create a bootable USB flash drive by performing the following steps on Microsoft Windows:
      1. Open Rufus software installed on your computer.

        The Rufus window opens.

      2. From the Device list, select the USB storage device.
      3. In the Format Options section, select the ISO image downloaded in Step 2. Click the open or browse icon next to the Create a bootable disk using option to select the ISO image.
      4. Click Start.

        A progress bar on the Rufus page indicates the status of the bootable USB flash drive creation. A success message is displayed once the process completes successfully.

      5. Click Exit to exit the window.
      6. Eject the USB storage device and unplug it from the computer.

        If you are using a computer with Linux as the operating system, do the following:

        Note: While you can use any of the available tools, it is recommended to use the dd command in Linux to create a bootable USB drive.

        1. Open a shell prompt.
        2. Use the cd command to go to the directory containing the software image file.
        3. Type [user@host ~]$ dd if=Log-collector-version.spinnumber.img of=/dev/usb-drive command to copy the image file to the USB drive and press Enter.

          Log-Collector-version.spin-number.img is the name of the downloaded Junos Space image file, and /dev/usb-drive is the name of the device drive to which your USB drive is mapped. The image file is copied to the USB drive and you are taken to the command prompt.

        4. Eject the USB drive and unplug it from the computer.

    Install Log Collector Using USB Flash Drive

    1. Plug the USB storage device into the USB port of the JA2500 appliance.
    2. To access the JA2500 appliance boot menu, perform the following steps:
      1. Power on the JA2500 appliance.
      2. While the JA2500 appliance powers ON, press the key mapped to send the DEL character in the terminal emulation utility.

        Note: Typically, the Backspace key is mapped to send the DEL character.

      3. The boot menu appears after a minute or shortly after.
    3. Ensure that the USB boot is at the top of the appliance boot-priority order.

      If USB KEY: CBM USB 2.0 - (USB 2.0) is not at the top of the list, perform the following steps:

      1. Use the down arrow to select USB KEY:CBM USB 2.0- (USB 2.0), and use the + key to move the entry to the top of the list.
      2. Press the F4 key to save your changes and exit the BIOS setup.
    4. After Verifying the BIOS setting, power off the JA2500 appliance.
    5. Power ON the appliance again. The boot menu displays the following options:
      1. Install Log Collector on Juniper JA2500 Hardware.
      2. Boot from local drive.
    6. Select Install Log Collector on Juniper JA2500 Hardware.
    7. Power OFF the appliance once the installation is completed.
    8. Restart the appliance and select Boot from local drive.
    9. Use the default credentials to log in to the JA2500 appliance; username is root and password is juniper123.
    10. Change the default root password when prompted.
    11. After logging in, select the desired node type.
    12. Configure the IP address and gateway.
    13. Configure settings for the DNS name server and the NTP server.

    Add Log Collector Node to Security Director.

    Installing Integrated Log Collector on a JA2500 Appliance or Junos Space Virtual Appliance

    The prerequisites to install the integrated log collector on a JA2500 appliance or virtual machine (VM) are as follows:

    • Install the Junos Space Network Management Platform Release 16.1R2 image on a JA2500 appliance or VM from the download site.
    • Install the Junos Space Security Director Release 16.2R1 image on a JA2500 appliance or VM from the download site.
    • Integrated Log Collector uses the 9200, 514, and 4567 ports.
    • The Junos Space Network Management Platform must be configured with Ethernet Interface eth0 and management IP addresses.
    • OpenNMS must be disabled on Junos Space Network Management Platform.
    • Ethernet Interface eth0 on the Junos Space platform must be connected to the network to receive logs.
    • /var should have a minimum of 500GB HDD for the integrated Log Collector installation to complete.

    Note: Security Director Logging and Reporting is not supported on JA1500 appliance.

    Specifications

    Table 4 shows the specifications for installing the integrated Log Collector on JA2500 appliance.

    Table 4: Specifications for Installing an Integrated Log Collector on a JA2500

    Component

    Specification

    Memory

    8 GB

    Log Collector uses 8 GB of memory of the available 32-GB system RAM.

    Disk space

    500 GB

    This is used from the existing JA2500 appliance disk space.

    CPU

    Single core

    Note: These specifications are used internally by the integrated Log Collector on a JA2500 appliance.

    Table 5 shows the specifications for installing the integrated Log Collector on Junos Space Virtual Appliance.

    Table 5: Specifications for Installing an Integrated Log Collector on a VM

    Component

    Specification

    Memory

    8 GB

    If the Integrated Log Collector is running on the Junos Space VM, we recommend adding 8 GB of RAM to maintain the Junos Space performance. It uses 8 GB of system RAM from the total system RAM.

    Disk space

    500 GB

    Minimum 500 GB is required. You can add any amount of disk space.

    CPU

    2 CPUs of 3.20 GHz

    Note: These specifications are used internally by the integrated Log Collector running on the Junos Space Virtual Appliance.

    To install the integrated Log Collector on a JA2500 appliance or Virtual Appliance:

    1. Download the integrated Log Collector image from the download site.
    2. Copy the integrated Log Collector script to a JA2500 appliance or Virtual Appliance.
    3. Connect to the CLI of a JA2500 appliance or virtual appliance with admin privileges.
    4. Navigate to the location where you have copied the integrated Log Collector script.
    5. Change the file permission using the following command: chmod+x Integrated-Log-Collector-16.2.R1.xxx.sh.
    6. Install the integrated Log Collector script using the following command: sh Integrated-Log-Collector-16.2.R1.xxx.sh.
      • The installation stops if the following error message is displayed while installing the integrated Log Collector on VM. You must expand the Network Management Platform disk size to proceed with the installation.

        [root@space-005056b40fef ~]# sh Integrated-Log-Collector-16.2.R1.xxx.sh ERROR: Insufficient HDD size, Please upgrade the VM HDD size to minimum 500 GB to install Log Collector

        To expand the hard disk size for Space VM:

        1. Add a hard disk with a 500-GB capacity on the Junos Space VM using the vSphere client.
        2. Connect to the console of Junos Space through SSH.
        3. Select Expand VM Drive Size.
        4. Enter the admin password and expand /var with 500 GB.
        5. Once /var is expanded, you are prompted for any further HDD expansion. select No, the system reboots.

          Note: Junos Space Network Management platform must be active and functioning. You must be able to log into the Junos Space Network Management Platform and Security Director user interfaces before attempting to run the integrated Log Collector setup script again.

        6. After the disk size is expanded and Junos Space Network Management Platform and Security Director user interfaces are accessible, run the sh Integrated-Log-Collector-16.2.R1.xxx.sh command.
      • The installation stops if the following error message is displayed while installing the integrated log collector on the JA2500 appliance or VM. You must disable OpenNMS by following the steps mentioned in the error message to proceed with the installation.

        [root@space-005056b41440 ~]# sh Integrated-Log-Collector-16.2.R1.157.sh ERROR: Opennms is running... Please try to disable opennms as described below or in document and retry Log Collector installation... STEPS: Login to Network Management Platform --> Administration --> Applications Right Click on Network Management Platform --> Manage Services -> Select Network Monitoring and click Stop Service Status should turn to Disabled

        After OpenNMS is disabled, run the sh Integrated-Log-Collector-16.2.R1.xxx.sh command.

      When integrated Log Collector installation is complete on the JA2500 appliance or VM, the following message is displayed:

      Shutting down system logger: [ OK ]

      Starting jingest ... jingest started.

      {"log-collector-node": {"id":376,"ip-address":"x.x.x.x","priority":0,"node-type":

      "INTEGRATED","cpu-usage":0,"memory-usage":0, "fabric-id":0,"display-name": "Integrated","timestamp":0}}

      Once the installation is complete, a logging node is automatically added in Administration > Logging Management > Logging Nodes.

    Adding Log Collector to Security Director

    Once Log Collector is configured, you can add it to Security Director.

    To add Log Collector to Security Director, see Adding Logging Nodes.

    To learn more about increasing the disk size of your VM when log files are too large, see Expanding the Size of the VM Disk for Log Collector.

    To learn more about enabling vMotion and Fault tolerance logging, see Enabling vMotion and Fault tolerance logging.

    To learn more about VMWare chassis cluster and fault tolerance, see vSphere Availability PDF Document.

    To learn more about configuring vMotion, see Creating a VMkernel port and enabling vMotion on an ESXi/ESX host and Set Up a Cluster for vMotion.

    Modified: 2017-03-21