At Indatify we use PostgreSQL for most of the time when dealing with data. We tried to use Cassandra, but for now it is too much of a constraint in such dynamic data environment. We put it on shelf. However, for textual data we choose Elasticsearch, because we know it and it provides full text search out of the box. Later, we will come back to Cassandra, but with more specified use case, as it requires precise data model to be predefined by query and not by structure.

So, to install Elasticsearch (if running without sudo, then run from root):

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
sudo apt-get install apt-transport-https
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt-get update && sudo apt-get install elasticsearch

Now, for local development environment (only) we will disable xpack:

sudo vi /etc/elasticsearch/elasticsearch.yml

Configure the following:

networkhost 0.0.0.0
xpack.security.enabled false

And then:

sudo systemctl enable elasticsearch && sudo systemctl start elasticsearch

To verify if it works fine:

curl -XGET "localhost:9200"

Now, to install Kibana:

sudo apt install kibana

And then, configure it:

sudo vi /etc/kibana/kibana.yml

With the following:

server.port 5601
server.host "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]

Finally, start Kibana:

sudo systemctl enable kibana && sudo systemctl start kibana

We are good to go. Remember to allocate at least 8GB of memory, 4GB is too little. More you have memory, more Elastic will put into it and run more quickly, instead of loading data directly from drives.