mini-siem-elk

Install Logstash on Ubuntu Server VM

This guide explains how to install and configure Logstash 8.17.4 on an Ubuntu VM, and how to forward data both from manual input and structured CSV files to Elasticsearch, and visualize it in Kibana.

📥 1. Download and Install Logstash

Download and extract Logstash into your VM:

curl -O https://artifacts.elastic.co/downloads/logstash/logstash-8.17.4-linux-x86_64.tar.gz
tar -xzf logstash-8.17.4-linux-x86_64.tar.gz
cd logstash-8.17.4

🧪 2. Test Logstash with STDIN Output

Before connecting to Elasticsearch, test if Logstash is functioning:

bin/logstash -e 'input { stdin { } } output { stdout {} }'

Once you see:

The stdin plugin is now waiting for input:

Type:

Hello World

🔗 3. Connect Logstash to Elasticsearch

Make sure Elasticsearch and Kibana are running.

Create a config file logstash.conf:

input {
  stdin { }
}

filter {
  mutate {
    add_field => { "message_type" => "test_hello_world" }
  }
}

output {
  elasticsearch {
    hosts => ["https://localhost:9200"]
    user => "elastic"
    password => "DPT9mObPqDLNXmk*=dXS"
    ssl_enabled => true
    ssl_verification_mode => "full"
    ssl_certificate_authorities => ["/home/ubuntu/elasticsearch-8.17.4/config/certs/http_ca.crt"]
    index => "logstash-helloworld"
  }

  stdout { codec => rubydebug }
}

🔒 Note: Update the user, password, and ssl_certificate_authorities path according to your environment.

Run it:

bin/logstash -f logstash.conf

Type:

Hello World

🔍 View Data in Kibana

1. Go to Stack Management > Data Views
2. Create a new Data View with index pattern: logstash-helloworld

1. Go to Discover, and select your new data view

📊 4. Send CSV Data to Elasticsearch

🗃️ Create Sample CSV File

Create a CSV file random_college_student.csv with sample student data:

Name,Roll Number,Email,Phone,Department,Semester,CGPA
Mrs. Brianna Shelton,730172,johnsontimothy@montgomery-hunt.com,273.738.4140x711,Electrical,8,8.87
Dennis Maldonado,886997,hessalexandra@gmail.com,509-113-9327,Civil,2,5.58
Walter Strickland,420281,jacqueline87@johnson-padilla.com,(091)603-5907,Computer Science,5,7.19
Stephen Ruiz,826570,julie27@hotmail.com,908.605.7188,Civil,5,7.12
Sandra Young,203512,davisleslie@walls-mullen.biz,001-554-385-3575x19038,Mechanical,4,8.52
Jennifer Cruz,315047,elopez@yahoo.com,+1-058-075-7988x1719,Civil,4,7.48
Mary Moore,899339,adrianaarnold@kennedy.com,(674)528-0106,Computer Science,8,8.22
Barbara Pena,701614,kayla30@sutton-cross.net,001-469-223-4426x42051,Computer Science,6,8.07
Bryan Braun,181973,bdoyle@welch.biz,(091)693-3733x65327,Electronics,1,7.38
Roger Anderson,739750,sandersjulie@gmail.com,(173)794-6139x11000,Mechanical,1,5.83
Brooke Clark,627517,kennedytravis@nichols.com,548-579-7402x53496,Electrical,7,5.91
Kenneth Shaw,503197,yjackson@hotmail.com,327.495.6020x8048,Computer Science,7,9.8
Christopher Pacheco,577018,nicole30@yahoo.com,001-969-830-6045x5108,Electronics,2,5.59
Elizabeth Brown,650283,victorgarner@sutton.org,776.918.9885,Electronics,4,7.71
Randy Allen,789359,sonyaesparza@gmail.com,(453)641-6529x9940,Mechanical,3,9.63
Christine Ford,211493,vsmith@gmail.com,001-837-387-6267x77071,Electronics,3,9.31
Andrew Yang,839129,perryjohn@dickerson-davis.com,422.587.1743x0038,Civil,2,6.66
Eric Young,655777,john16@yahoo.com,(868)397-3535,Civil,3,7.36
James Johnson,335272,powellshawn@hodges-rose.com,2708600279,Civil,6,8.73
Lindsey Townsend,758240,tinaknight@williams.com,581-831-4979,Electrical,5,7.6
Heather Luna,269300,benjamin68@warren.biz,+1-772-624-2611x644,Civil,2,6.75
Michael Robbins,581886,dscott@gmail.com,+1-458-590-8905x591,Civil,1,8.0
Ernest Larson,277233,webbwilliam@patrick-phillips.net,072-489-0794,Computer Science,8,6.88
Austin Frazier,577797,margaretkennedy@hotmail.com,515-631-0958x36356,Electronics,1,5.75
Ronald Klein,513958,michaelroberts@schultz.com,+1-355-392-6534x031,Electronics,2,9.62
Cole Barnes,563615,morrisonmelanie@daniel.com,(079)037-6073,Electrical,5,7.42
Ashley Thomas,403435,kgomez@hotmail.com,7724112938,Civil,8,6.27
Frederick Gillespie,450784,cmccormick@harrison.biz,(075)261-2749x27499,Electrical,7,7.79
Michael Ward,783284,taylorcynthia@matthews.info,(783)228-4260x4333,Mechanical,6,9.52
Jared Brown,654717,jessica44@jackson-nichols.info,(567)197-5230x65376,Electrical,8,5.77

⚙️ Update Logstash Config to Parse CSV

Update logstash.conf:

input {
  file {
    path => "/home/ubuntu/logstash-8.17.4/random_college_student.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  csv {
    separator => ","
    skip_header => "true"
    columns => ["Name", "RollNumber", "Email", "Phone", "Department", "Semester", "CGPA"]
  }

  mutate {
    convert => {
      "Semester" => "integer"
      "CGPA" => "float"
    }
  }
}

output {
  elasticsearch {
    hosts => ["https://localhost:9200"]
    user => "elastic"
    password => "DPT9mObPqDLNXmk*=dXS"
    ssl_enabled => true
    ssl_verification_mode => "full"
    ssl_certificate_authorities => ["/home/ubuntu/elasticsearch-8.17.4/config/certs/http_ca.crt"]
    index => "college-students"
  }

  stdout { codec => rubydebug }
}

▶️ Run Logstash with CSV Configuration

bin/logstash -f logstash.conf

Logstash will process the CSV and send it to Elasticsearch.

📈 View CSV Data in Kibana

1. Go to Stack Management > Data Views
2. Create a Data View with index pattern: college-students
3. Go to Discover to explore the ingested student data
4. Apply filters (e.g., CGPA <= 6) as needed

📁 Screenshots

All screenshots referenced above should be placed in:

/images/03-install-logstash

This site is open source. Improve this page.