ELK

-

 1. What is ELK?

  • Definition: A powerful suite of open-source tools used to collect, search, analyze, and visualize data in real time.
  • Official Name: Today, it is officially called the Elastic Stack.
  • Core Purpose: Centralized monitoring, log management, security analytics, and search operations.
🏗️ 2. Core Architecture (The 4 Components)
🔹 E – Elasticsearch
  • Role: The Heart (Database & Engine)
  • What it does: Stores and indexes all incoming data.
  • Key Feature: Extremely fast search results because it uses inverted indexing.
  • Format: Stores data as structured JSON text documents.
🔹 L – Logstash
  • Role: The Data Processor (Pipeline)
  • What it does: Gathers data from various sources, cleans/filters it, and pushes it to Elasticsearch.
  • Pipeline Mechanism: Uses an Input → Filter → Output workflow.
🔹 K – Kibana
  • Role: The Face (User Interface)
  • What it does: Connects to Elasticsearch to build live charts, graphs, maps, and dashboards.
  • Common Use: Running search queries and monitoring system health visually.
🔹 B – Beats (The Add-on)
  • Role: The Messengers (Lightweight Shippers)
  • What it does: Low-resource software agents installed on individual servers to send logs straight to Logstash or Elasticsearch.
  • Examples: Filebeat (for text logs), Metricbeat (for CPU/RAM metrics).
⚙️ 3. Deployment Modes: Agent-Based vs. Agentless
Feature🏢 Agent-Based Setup🌐 Agentless Setup
How it worksInstall a tool (like Filebeat) directly on the target machine.Target machine sends data over the network via protocols.
Data SafetyHigh (Buffers data locally if the network goes down).Medium (Relies entirely on network stability).
Resource ImpactVery low resource usage on target machines.Higher network load on the central ELK server.
Best ForCloud instances, virtual machines, and custom apps.Routers, firewalls, switches, and locked devices.
📡 4. SNMP-Based Network Monitoring in ELK
  • Architecture: Always Agentless for network devices.
  • Mechanism 1 (Polling): ELK uses Metricbeat to actively request data from routers every few seconds.
  • Mechanism 2 (Traps): Network devices instantly push alert packets to Logstash on port 162 during a failure.
  • Critical Component (MIB Files): Dictionary files provided by vendors (e.g., Cisco). They translate numeric string codes (OIDs) into human-readable data (e.g., CPU_Usage).
🎯 5. Top Use Cases
  1. Log Management: Finding app errors and system bugs instantly across thousands of servers.
  2. Infrastructure Monitoring: Watching live system health metrics like CPU, RAM, and disk space.
  3. Application Performance (APM): Tracking website page loading times and API response delays.
  4. Security Operations (SIEM): Detecting network cyber attacks and tracking suspicious logins.
  5. Business Analytics: Monitoring real-time sales numbers, checkout counts, and user behavior.

Comments

Post a Comment

Popular posts from this blog

How to enable the syslog monitoring-Zabbix

-
Steps need to be performed:  On the Syslog server side:  1. Install the rsyslog sudo yum install rsyslog 2.  Enable rsyslog service: Start and enable the rsyslog service if it’s not already running: sudo systemctl start rsyslog sudo systemctl enable rsyslog 3.  Configure rsyslog to accept remote logs: Open the rsyslog configuration file: vi /etc/rsyslog.conf Uncomment or add the following lines to enable UDP or TCP log reception: # Provides UDP syslog reception module(load="imudp") input(type="imudp" port="514") # Provides TCP syslog reception module(load="imtcp") input(type="imtcp" port="514") 4.  Restart the rsyslog service: After saving the configuration file, restart rsyslog to apply the changes: sudo systemctl restart rsyslog 5. Ensure the firewall is configured to allow the TCP/UDP port 514 or disable the firewall service. 6. Ensure that SELinux is disable if not disable it Check the status of SELinux sestatus Disa...
Read more

Zabbix installation: Distribution setup

-
  On Zabbix core server: Step1:   Install Zabbix repository rpm -Uvh https://repo.zabbix.com/zabbix/6.0/rhel/8/x86_64/zabbix-release-latest-6.0.el8.noarch.rpm dnf clean all Step2: Install Zabbix server, frontend, agent dnf install zabbix-server-mysql zabbix-web-mysql zabbix-apache-conf zabbix-selinux-policy zabbix-agent Step3:   Configure the database for Zabbix server Edit file /etc/zabbix/zabbix_server.conf DBHost= DBserver_ip DBName= DBUser= DBPassword= DBPort=3306 Step4:  Start Zabbix server and agent processes systemctl restart zabbix-server zabbix-agent httpd php-fpm systemctl enable zabbix-server zabbix-agent httpd php-fpm On Zabbix Database Server: Step1:   Install Zabbix repository rpm -Uvh https://repo.zabbix.com/zabbix/6.0/rhel/8/x86_64/zabbix-release-latest-6.0.el8.noarch.rpm dnf clean all Step2: Install the sql script, zabbix agent  dnf install zabbix-sql-scripts zabbix-selinux-policy zabbix-agent Step3: Install the mysql server yum install my...
Read more

SNMP OID: Full Insight

-
 SNMP uses OIDs to uniquely identify pieces of data (like CPU usage, interface status, etc.). Standard OIDs (defined by IETF) are common across all devices , e.g.: 1.3.6.1.2.1.1.1.0 → sysDescr (system description) 1.3.6.1.2.1.2.2.1.10 → ifInOctets (interface input bytes) Vendor-specific OIDs: For example: Vendor Enterprise OID Base Example Cisco 1.3.6.1.4.1.9 1.3.6.1.4.1.9.1.516 (Cisco 2960 model) Juniper 1.3.6.1.4.1.2636 Juniper-specific OIDs HP 1.3.6.1.4.1.11 HP printer/router OIDs Fortinet 1.3.6.1.4.1.12356 FortiGate device OIDs Part 1: Standard SNMP OID – .1.3.6.1.2.1.1.1.0 This is used for system description , and it works on all SNMP-supported devices (routers, switches, servers, etc.). Let’s break this path: 🔹 .1 → iso International Organization for Standardization This is the root of the SNMP OID tree. Every OID begins from here. 🔹 .3 → org Organization This level is used for organizations that are registered under ISO. 🔹 .6 → dod Depart...
Read more