Linux: Elastic and Kibana Install

This guide walks you through setting up a single node Elasticsearch cluster and Kibana instance. It should take around 2 hours to complete the Elastic and Kibana install.

If you are setting up a production environment, you will want to set up an Elasticsearch cluster. Please use the Elasticsearch website for more information on how to do this.

The Elastic token used to configure Kibana is only valid for 30 minutes. Once you have extracted Elastic you have 30 minutes to Install and Configure Kibana. 
It is not a long process but please ensure you have that time to complete these steps.

---

Installing Elasticsearch

Ubuntu

1. Downloading Elastic 8.19.7 & Kibana 8.19.7. To download Elasticsearch and Kibana using a Debian package:

   cd /data
   wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo yum add -
   wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.19.7-amd64.deb
   wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.19.7-amd64.deb.sha512
   wget https://artifacts.elastic.co/downloads/kibana/kibana-8.19.7-amd64.deb
   wget https://artifacts.elastic.co/downloads/kibana/kibana-8.19.7-amd64.deb.sha512

2. To navigate to where your Elasticsearch files are stored:

cd/data

1. To compare the SHA of the downloaded Debian package with the published checksum:

shasum -a 512 -c elasticsearch-8.19.7-amd64.deb.sha512

1. To install Elasticsearch:

sudo dpkg -i elasticsearch-8.19.7-amd64.deb

1. To confirm that Elasticsearch is installed:

sudo systemctl status elasticsearch.service

Red Hat

Importing the Elasticsearch PGP Key

Download and install the public signing key:

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Downloading and installing the RPM manually

The RPM for Elasticsearch v8.19.7 can be downloaded from the website and installed as follows, this will autoconfigure default security settings unless a pre-existing elasticsearch.yml config file is present in /etc/elasticsearch:

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.19.7-x86_64.rpm
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.19.7-x86_64.rpm.sha512
shasum -a 512 -c elasticsearch-8.19.7-x86_64.rpm.sha512
sudo rpm --install elasticsearch-8.19.7-x86_64.rpm

Configuring Elasticsearch

1

Edit the Elasticsearch.yml file with the editor of your choice (nano, vim, emacs etc.):

sudo nano /etc/elasticsearch/elasticsearch.yml

2

A text editor is displayed in the terminal. To provide custom values for settings (remove the # before a line to ensure the setting is used).

Set a name for the Elastic cluster: The name must be unique, descriptive of the server, and distinctive to other clusters to prevent auto joining.

cluster.name: im-elasticcluster

Set the Elastic node name: The node name must be unique to the environment to prevent clusters auto joining.

node.name: esnode1

Set the network host: This binds Elastic to all IP addresses assigned to this machine, so that remote access is possible.

network.host: 0.0.0.0

Set the discovery.seed_hosts: These addresses can be given as hostnames or IP addresses. Hosts specified as hostnames are resolved to IP addresses during each round of discovery.

discovery.seed_hosts: [“127.0.0.1”] 

Set the cluster.initial_master_nodes: Specifying the host name determines the set of master-eligible nodes whose votes are counted in the very first election. You must explicitly list the master-eligible nodes whose votes will be counted in the very first election.

cluster.initial_master_nodes: [“esnode1”]

Set the Elasticsearch data path. Elasticsearch stores the node data across all provided paths but keeps each shard’s data on the same path.

To use a custom path for data:

sudo mkdir /data/elasticsearchdata

To exit the editor, press Ctrl+x, then enter Y when prompted. Press the Enter key to save your changes.

3

To update permissions of all Elasticsearch directories:

sudo chown elasticsearch:elasticsearch -R /usr/share/elasticsearch      
/var/log/elasticsearch /var/lib/elasticsearch /etc/sysconfig/elasticsearch
/etc/elasticsearch /data/elasticsearchdata

4

Open the /etc/sysconfig/elasticsearch file:

sudo nano /etc/sysconfig/elasticsearch

Add the following two entries at the bottom of the file:

ES_USER=elasticsearch
ES_GROUP=elasticsearch

Press Ctrl+x to exit the editor. When prompted enter Y, then press Enter to save your changes.

You may need to run:

sudo systemctl daemon-reload

5

Manually starting up elasticsearch can cause some complications with ownership of files (logs / data etc.), so I would suggest running via systemd from the start unless unavoidable for debugging purposes. The process described further on in the guide can be used to set the elastic account password and generate an enrolment token for kibana.

Running Elasticsearch with systemd

To configure Elasticsearch to start automatically when the system boots up:

This guide assumes that the RHEL version is using systemd.

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

Elasticsearch can be started and stopped as follows:

sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service

To confirm that Elasticsearch was started successfully, check the log files located in /var/log/elasticsearch/.

By default the Elasticsearch service doesn’t log information in the systemd journal. To enable journalctl logging, the --quiet option must be removed from the ExecStart command line in the elasticsearch.service file.

When systemd logging is enabled, the logging information are available using the journalctl commands:

To tail the journal:

sudo journalctl -f

To list journal entries for the Elasticsearch service:

sudo journalctl --unit elasticsearch

To list journal entries for the Elasticsearch service starting from a given time:

sudo journalctl --unit elasticsearch --since  "2016-10-30 18:17:16"

Password and Token Administration

By default, Elasticsearch will autogenerate an elastic user password and Kibana enrollment token at startup. If you didn’t make a note of these or prefer to use your own password, reset the elastic password with this command:

/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic -i

If the Kibana enrollment token has expired (after 30 minutes) or you don’t have it to hand, you can generate a new one with the following command:

/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana

Checking that Elasticsearch is running

To test that your Elasticsearch node is running by sending an HTTP request to port 9200 on localhost, you will need to know the password for the elastic account to try this. If you don’t have this to hand it is not a problem as we will be resetting it later:

curl -X GET https://localhost:9200 -k -u elastic

Which should return something akin to the following:

{
"name" : "Cp8oag6",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "AT69_T_DTp-1qgIJlatQqA",
"version" : {
"number" : "7.9.3",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "f27399d",
"build_date" : "2016-03-30T09:51:41.449Z",
"build_snapshot" : false,
"lucene_version" : "8.6.2",
"minimum_wire_compatibility_version" : "1.2.3",
"minimum_index_compatibility_version" : "1.2.3"
  },
 "tagline" : "You Know, for Search"
}

Installing Kibana

If installing on multiple servers, this should be done on the same server as the apache web server, to allow for outside communication, not on the elasticsearch server, unless you have an ability to reach :5601 on there from your browser. It is preferable to proxy requests via apache.

Ubuntu

1. Change directory to /data:

cd /data

1. Check checksum:

shasum -a 512kibana-8.19.7-amd64.rpm.sha512

1. Unpackage and install Kibana:

sudo dpkg -i kibana-8.19.7-amd64.rpm

Red Hat

1. Download and install the public signing key:

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

1. Download and install the RPM manually. The RPM for Kibana v7.9.3 can be downloaded from the website and installed:

wget https://artifacts.elastic.co/downloads/kibana/kibana-8.19.7-x86_64.rpm
shasum -a 512 kibana-8.19.7-x86_64.rpm 
sudo rpm --install kibana-8.19.7-x86_64.rpm

Configuring Kibana

1

Change the directory to the /etc/kibana location:

sudo cd /etc/kibana

2

Open kibana.yml in a text editor:

sudo nano kibana.yml

3

Apply the following config:

server.port: 5601

server.host: "0.0.0.0"

elasticsearch.hosts: ["https://localhost:9200"]

elasticsearch.ssl.verificationMode: none

Press Ctrl+x to exit the editor. When prompted enter Y, then press Enter to save your changes.

4

Apply the Kibana enrollment token, see previous section for generating this if required

/usr/share/kibana/bin/kibana-setup --enrollment-token <enrollment-token>

5

Enable the Kibana service:

sudo systemctl enable kibana.service

6

Ensure Elasticsearch service is running, then start the Kibana service to check for any issues:

sudo systemctl start kibana.service

Check the logs for any further issues.

7

Once the service has started, open a browser and navigate to http://[insert-ip-address-or-host-name]:5601 to log in to Kibana.

8

Open Kibana and apply the Elasticsearch license: click Stack Management > License Management > Update License.

9

Upload your Elasticsearch license.

If the kibana.yml file is installed in a location that cannot be accessed by the service, copy the file from the etc/kibana folder to the usr/share/kibana folder.

Configuring JAVA_HOME environment variable

If the AIE components are on separate server to Elasticsearch, or the client has a preferred JDK, you should be able to install via the appropriate package manager. If not, you can use the bundled one present in Elasticsearch, though note that this will not receive any updates until you update Elasticsearch itself.

There have also been instances where the below configuration does not work properly with the forked processes tika generates, causing the tika service to fail. In that scenario, some additional configuration may be required.

To set the permanent environment variable for JAVA_HOME, do the following:

1

Change directory to /etc:

cd/etc

2

Open the profile file in nano:

sudo nano profile

3

Add the following values to the bottom of the file:

JAVA_HOME=/usr/share/elasticsearch/jdk
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
export JAVA_HOME
export PATH

4

Press Ctrl+x to exit nano.

5

Enter Y to save changes, N to discard, then click Enter.

6

Change directory to the elasticsearch jdk folder and run the following two commands to ensure the folders are accessible:

sudo chmod +x bin/
sudo chmod +x lib/

7

Restart your server.

8

Once logged back in, check that JAVA_HOME has been set correctly and is running the correct version:

echo $JAVA_HOME

This will return the path that you specified above.

java –version

Enabling firewalls & allowing HTTP/HTTPS

Ubuntu

1. Check the current status of the firewall

sudo ufw status verbose

1. Turn the firewall on:

sudo ufw enable

When prompted to proceed, enter Y, then click Enter.

1. Verify the firewall is now active:

sudo ufw status verbose

1. Open firewall ports for HTTP, HTTPS, SSH, Elastic and Kibana:

Note: Only open ports 9200/tcp and 5601/tcp to access Elastic and Kibana remotely. If you are not installing AIE on the same server, you can skip http / https

sudo ufw allow http
sudo ufw allow https
sudo ufw allow 9200/tcp
sudo ufw allow 5601/tcp
sudo ufw allow ssh

If these firewall ports aren’t opened you won’t be able to connect through ssh after reboot.

1. To verify updates:

sudo ufw status verbose

Red Hat

1. Check the current status of the firewall:

 systemctl status firewalld 

If listed as inactive, turn it on. Whether firewall is present at all by default varies between cloud providers. It can be installed via dnf if required.

1. Turn the firewall on:

sudo service firewalld start

OR 

sudo systemctl start firewalld 

When prompted enter Y, then press Enter to save your changes.

1. Verify the firewall is now active:

sudo systemctl status firewalld

1. Open the firewall ports for http, https, Elastic and Kibana:

Note: If you are not installing AIE on this same server, http (80) / https (443) will not be required

Add as a service

sudo firewall-cmd --zone=public --permanent --add-service=http 
sudo firewall-cmd --zone=public --permanent --add-service=https 
sudo firewall-cmd --zone=public --permanent --add-service=ssh 
sudo firewall-cmd --zone=public --permanent --add-service=elasticsearch 
sudo firewall-cmd --zone=public --permanent --add-service=kibana 

Alternatively add specific ports. This will work if the OS doesn’t have the service definitions above. Note 9300 will also be required if more than one elastic node is present.

sudo firewall-cmd --zone=public --permanent --add-port 80/tcp 
sudo firewall-cmd --zone=public --permanent --add-port 443/tcp 
sudo firewall-cmd --zone=public --permanent --add-port 22/tcp 
sudo firewall-cmd --zone=public --permanent --add-port 9200/tcp 
sudo firewall-cmd --zone=public --permanent --add-port 5601/tcp 

Important: If these firewall ports aren’t opened you won’t be able to connect through ssh after reboot.

1. Reload the firewall service:

firewall-cmd –-reload

1. Reload the firewall service:

 firewall-cmd –-list-all 

Setting up Elastic Transport Layer Security (TLS)

1

Generate the certificates from /usr/share/elasticsearch/bin

./elasticsearch-certutil ca

When prompted, leave the file name blank then press Enter.

Enter your secure CA password.

./elasticsearch-certutil cert –ca elastic-stack-ca.p12

Enter your secure CA password as the CA password.

When prompted, leave the file name blank then press Enter.

Enter your secure cert password as the password.

2

Create a certs folder in /etc/elasticsearch:

 sudo mkdir /etc/elasticsearch/certs 

3

Copy the certificates to the new certs directory:

sudo cp elastic-stack-ca.p12 /etc/elasticsearch/certs/ 

sudo cp elastic-certificates.p12 /etc/elasticsearch/certs/

4

Edit the elasticsearch.yml file with the editor of your choice:

sudo nano /etc/elasticsearch/elasticsearch.yml 

Change the autoconfigured section at the bottom to the following:

xpack.security.http.ssl: 
    enabled: true 
    keystore.path: certs/elastic-certificates.p12 
    truststore.path: certs/elastic-certificates.p12 

xpack.security.transport.ssl: 
    enabled: true 
    verification_mode: certificate 
    keystore.path: certs/elastic-certificates.p12 
    truststore.path: certs/elastic-certificates.p12 

When prompted enter Y, then press Enter to save your changes.

5

Update the certificate credentials in the Elasticsearch keystore:

 cd /usr/share/elasticsearch/bin 

If security autoconfiguration has run, there will be some existing entries present in the keystore file we will need to remove.

./elasticsearch-keystore remove xpack.security.transport.ssl.keystore.secure_password 
./elasticsearch-keystore remove xpack.security.transport.ssl.truststore.secure_password 
./elasticsearch-keystore remove xpack.security.transport.transport.keystore.secure_password 

You may encounter an error around changing the owner of the keystore file, this is because it is owned by the elastic user but you are trying to update it as (probably if you’re using sudo) root, to work around this, temporarily change the ownership of the file:

sudo chown root:root /etc/elasticsearch/elasticsearch.keystore 

Enter the cert password as the password 1/2:

 ./elasticsearch-keystore add 
 xpack.security.transport.ssl.keystore.secure_password 

./elasticsearch-keystore add 
xpack.security.transport.ssl.truststore.secure_password

./elasticsearch-keystore add 
xpack.security.http.ssl.keystore.secure_password

./elasticsearch-keystore add 
xpack.security.http.ssl.truststore.secure_password

You can use ‘list’ and ‘show’ with a specific key to validate the current state of the keystore file

If you had to change the owner of the keystore earlier, change it back:

sudo chown elasticsearch:elasticsearch 
/etc/elasticsearch/elasticsearch.keystore

6

Ensure permissions are updated for the certs in the new directory:

sudo chown elasticsearch:elasticsearch -R /etc/elasticsearch/certs

7

Start the Elasticsearch service.

sudo systemctl daemon-reload 
sudo systemctl start elasticsearch.service  

8

Elastic has now been secured and the self-signed certificate is applied. Run a curl command to check that the output is correct:

curl --insecure https://localhost:9200 -u elastic

You are prompted to enter the elastic user password.

You can also check the output in a browser: navigate to https://[insert-ip-address-or-host-name]:9200. You are prompted to enter a secure password.

9

Update the kibana.yml file to include https:

cd /etc/kibana 
sudo nano kibana.yml 
elasticsearch.hosts: ["https://localhost:9200"] 
elasticsearch.ssl.verificationMode: none

Press Ctrl+x to exit the editor. When prompted enter Y, then press Enter to save your changes.

10

Restart Kibana:

sudo systemctl daemon-reload 
sudo systemctl restart kibana.service