Skip to main content

Elasticsearch install and configuration on Ubuntu

If you plan to use elastic integrated with Django then it would be better to use an old version of Elastic or replace Haystack with Django-Elasticsearch-DSL (Not tested) See old version install at bottom of this post. 

New version Elasticsearch ( attention no haystack at this time ) 

The Elasticsearch components are not available in Ubuntu’s default package repositories. They can, however, be installed with APT after adding Elastic’s package source list.

curl -fsSL | sudo apt-key add -

E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation

apt-get install gnupg

curl -fsSL | sudo apt-key add -

Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).


echo "deb stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

apt update

apt install elasticsearch

vim /etc/elasticsearch/elasticsearch.yml

vim /etc/security/limits.conf

elasticsearch - nofile 65535

elasticsearch - memlock unlimited


Set ES_HEAP_SIZE to 50% of available RAM, but no more than 31g new versions do it automatically.




vim /etc/elasticsearch/elasticsearch.yml

bootstrap.mlockall: true ( bootstrap.memory_lock )

Backup Elasticsearch's,or%20indices%20in%20the%20cluster.

Upgrade python lib if you get errors or timeouts.

curl -XGET '

curl '

Elasticsearch cluster 'master_not_discovered_exception'

The root cause of the master not discovered exception is the nodes are not able to ping each other on port 9300. And this needs to be both ways. i.e node1 should be able to ping node2 on 9300 and vice versa.

Note : Elasticsearch reserves port 9300-9400 for cluster communication and port 9200-9300 for accessing the elasticsearch APIs.

cluster.initial_master_nodes: ["vindazofrDb"]

ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(

This error can be fixed with setting in django.

'default': {

#'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',

'ENGINE': 'elasticstack.backends.ConfigurableElasticSearchEngine',

'URL': '',

'TIMEOUT': 10000,

'INDEX_NAME': 'jobs',

'EXCLUDED_INDEXES': ['icrm.search_indexes.CVIndex', 'job.search_indexes.ArchiveIndex', 'spontaneousmail.search_indexes.SpontaneousProfileIndex'],


curl ''


"cluster_name" : "vindazofr",

"status" : "red",

"timed_out" : false,

"number_of_nodes" : 1,

"number_of_data_nodes" : 0,

"active_primary_shards" : 0,

"active_shards" : 0,

"relocating_shards" : 0,

"initializing_shards" : 0,

"unassigned_shards" : 2,

"delayed_unassigned_shards" : 0,

"number_of_pending_tasks" : 0,

"number_of_in_flight_fetch" : 0,

"task_max_waiting_in_queue_millis" : 0,

"active_shards_percent_as_number" : 0.0


Problem with shards when index is created.

Show unassigned_shards

curl -XGET | grep UNASSIGNED | awk {'print $1'}

curl -H'Content-Type: application/json' -XPUT '' -d '{"index.routing.allocation.disable_allocation": false}'

or check the breaking changes documentation for removed settings

If you get this error

Content-Type header [application/x-www-form-urlencoded] is not supported

This error is due to strict content-type checking introduced in ElasticSearch 6.0, as explained in

-H'Content-Type: application/json'

curl -XGET

First use cluster health API to get the current health of the cluster, where RED means one or more primary shards missing and Yellow means one or more replica shards are missing.

After this use the cluster allocation explain API to know why a particular shard is missing and elasticsearch is not able to allocate it on data-node.

Once you get the exact root cause, try to address the issue, which often requires, changing few cluster settings(mentioned in @wilfred answer earlier) But in some cases, if it's replica shards, and you have another copy of same shard(ie another replica) available, you can reduce the replica count using update replica setting and later on again increase it, if you need it.

Apart from above, if your cluster allocation API mentions it doesn't have a valid data node to allocate a shard, then you need to add a new data node, or change the shard allocation awareness settings.

curl -X GET ""

{"index":"jobs","shard":0,"primary":true,"current_state":"unassigned","unassigned_info":{"reason":"CLUSTER_RECOVERED","at":"2021-03-01T08:31:52.432Z","last_allocation_status":"no"},"can_allocate":"no","allocate_explanation":"cannot allocate because allocation is not permitted to any of the nodes"}

To make your node yellow you have only to add “data” in the role.

vim /etc/elasticsearch/elasticsearch.yml

node.roles: [master, data]

But if you want green status.

And as you have a single node cluster, so you will not have another node where your replicas can be assigned.


Add more nodes to your cluster, so that replicas can be assigned on other nodes. (preferred way)

More information about Node, Cluster and roles

Configuration examples

Configuration example If you are running a single node of Elasticsearch, then you have a cluster of one node. vindazofr vindazofrReplica

node.roles: [master, data, data_content, data_hot, data_warm, data_cold, ingest, ml, remote_cluster_client, transform] /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

http.port: 9200

discovery.seed_hosts: [""]

cluster.initial_master_nodes: ["vindazofrDb"]

cluster.routing.allocation.enable: all

Configuration cluster with 2 nodes vindazofr vindazofrReplica

node.roles: [master, data, data_content, data_hot, data_warm, data_cold, ingest, ml, remote_cluster_client, transform] /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

http.port: 9200

discovery.seed_hosts: ["", ""]

cluster.initial_master_nodes: ["vindazofrDb", "vindazofrReplica"]

cluster.routing.allocation.enable: all

Django related errors

Parsing error in django haystack backend.

Elasticsearch using 'payed:(True)': RequestError(400, u'parsing_exception'

Elasticsearch already deprecated filtered query. Use bool instead.

{ "query": { "bool": { "filter": {} } } }

Big parts of it could also be used by haystack to abstract away some of the differences between elastic versions (filtered query vs {"bool": {"filter": []}} for example) and make the overall haystack code a bit simpler. It even provides hooks for custom (de)serialization and other goodies. I would also be happy to add more if it would make haystack easier to port.

Elasticsearch 1.x and 2.x. Elasticsearch 5.x is not supported yet, if you would like to help, please see #1383.

Old version for vindazo ( with Django haystack ) 

If you plan to use elastic integrated with Django then it would be better to use an old version of Elastic or replace Haystack with Django-Elasticsearch-DSL (Not tested)

Downgrade you would like to

apt-get remove elasticsearch

(Reading database ... 58301 files and directories currently installed.)

Removing elasticsearch (7.11.1) ...

Stopping elasticsearch service... OK

Deleting log directory... OK

Download package from site.


sudo dpkg -i elasticsearch-2.3.5.deb

For this version you have to install Java..

sudo apt install openjdk-8-jdk

Don’t forget to add in old configuration hosts to ping via discovery module ["", ""]

And only one should be installed as master i think.

curl '' 

Old version config Example vindazo nodeWeb ["", ""]
cluster.initial_master_nodes: ["nodeWeb"]
node.roles: [master, data] # only one master
Don't forget firewall configuration
vim /etc/ufw/user.rules 

### tuple ### allow any 9200 any in
-A ufw-user-input -p tcp --dport 9200 -s 88.99.98.
11 -j ACCEPT
-A ufw-user-input -p udp --dport 9200 -s 88.99.98.
11 -j ACCEPT

### tuple ### allow any 9300 any 88.99.98.
11 in
-A ufw-user-input -p tcp --dport 9300 -s 88.99.98.
11 -j ACCEPT
-A ufw-user-input -p udp --dport 9300 -s 88.99.98.
11 -j ACCEPT

List of common used command

curl ''
curl ''
curl -XGET  


Popular posts from this blog

Pgpool PgBouncer Postgresql streaming replication, load balancing and administration

The term scalability refers to the ability of a software system to grow as the business that uses it grows. PostgreSQL provides some features to help you build scalable solutions, but strictly speaking, PostgreSQL itself is not scalable. It can effectively use the following resources from one computer. Now, we will show you some configurations that may be useful for your use case. However, this can be problematic when distributing the database solution to multiple computers, because the standard PostgreSQL server can only run on a single computer. In this article, we will study different extension schemes and their implementation in PostgreSQL. Replication can be used in many expansion scenarios. Its main purpose is to create and maintain a backup database when the system fails. This is especially true for physical replication. However, replication can also be used to improve the performance of PostgreSQL-based solutions. Sometimes third-party tools can be used to implement complex exp

Tekstverwerking python Text processing python SpaCy, TensorFlow, NLTK, Allen-NLP, Stanford-NLP

 Dit post maakt gebruik van spaCy, een populaire Python-bibliotheek die de taalgegevens en algoritmen bevat die je nodig hebt om teksten in natuurlijke taal te verwerken. Zoals u in dit post zult leren, is spaCy gemakkelijk te gebruiken omdat het containerobjecten biedt die elementen van natuurlijke taalteksten vertegenwoordigen, zoals zinnen en woorden. Deze objecten hebben op hun beurt attributen die taalkenmerken vertegenwoordigen, zoals delen van spraak. Op het moment van schrijven bood spaCy voorgetrainde modellen aan voor Engels, Duits, Grieks, Spaans, Frans, Italiaans, Litouws, Noors BokmÃ¥l, Nederlands, Portugees en meerdere talen gecombineerd. Bovendien biedt spaCy ingebouwde visualizers die u programmatisch kunt aanroepen om een grafische weergave van de syntactische structuur van een zin of benoemde entiteiten in een document te genereren.   De spaCy-bibliotheek ondersteunt ook native geavanceerde NLP-functies die andere populaire NLP-bibliotheken voor Python niet hebben. Spa

Google Closure

   Closure Library De Closure-bibliotheek is een JavaScript-bibliotheek, vergelijkbaar met andere moderne producten zoals jQuery, Angular, Vue.js, Dojo en MooTools. De coderingsstijl en het gebruik van opmerkingen in de Closure-bibliotheek zijn op maat gemaakt voor Closure Compiler. In vergelijking met andere JavaScript-bibliotheken is het de belangrijkste onderscheidende factor van Closure Compiler. Een eenvoudig compressie-experiment ontdekte dat wanneer Closure Compiler wordt gebruikt in plaats van YUI Compressor, de Closure Lib-code met 85% kan worden verminderd, wat een enorme impact kan hebben op de codecompressiecapaciteit van de compiler.    De implementatie van de  closure bibliotheek richt zich ook op leesbaarheid en prestaties. Wees zuinig bij het maken van objecten, maar wees genereus bij het benoemen en opnemen van objecten. Het heeft ook een prachtig gebeurtenissysteem, ondersteuning voor klassen en overerving en verschillende UI-componenten, waaronder bijvoorbeeld een ri