If you plan to use elastic integrated with Django then it would be better to use an old version of Elastic or replace Haystack with Django-Elasticsearch-DSL (Not tested) See old version install at bottom of this post.
New version Elasticsearch ( attention no haystack support at this time, outdated and not used anymore )
The Elasticsearch components are not available in Ubuntu’s default package repositories. They can, however, be installed with APT after adding Elastic’s package source list.
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation
apt-get install gnupg
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
OK
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
apt update
apt install elasticsearch
vim /etc/elasticsearch/elasticsearch.yml
vim /etc/security/limits.conf
elasticsearch - nofile 65535
elasticsearch - memlock unlimited
/etc/default/elasticsearch
Set ES_HEAP_SIZE to 50% of available RAM, but no more than 31g new versions do it automatically.
ES_HEAP_SIZE=30GB
MAX_OPEN_FILES=65535
MAX_LOCKED_MEMORY=unlimited
vim /etc/elasticsearch/elasticsearch.yml
bootstrap.mlockall: true ( bootstrap.memory_lock )
https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html
Backup Elasticsearch
https://www.elastic.co/guide/en/elasticsearch/reference/current/backup-cluster-data.html#:~:text=To%20back%20up%20your%20cluster's,or%20indices%20in%20the%20cluster.
Upgrade python lib if you get errors or timeouts.
curl -XGET 'http://159.69.65.2:9200'
curl 'http://159.69.65.2:9200/_cluster/health?pretty=true'
Elasticsearch cluster 'master_not_discovered_exception'
The root cause of the master not discovered exception is the nodes are not able to ping each other on port 9300. And this needs to be both ways. i.e node1 should be able to ping node2 on 9300 and vice versa.
Note : Elasticsearch reserves port 9300-9400 for cluster communication and port 9200-9300 for accessing the elasticsearch APIs.
cluster.initial_master_nodes: ["vindazofrDb"]
ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(
This error can be fixed with setting in django.
'default': {
#'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
'ENGINE': 'elasticstack.backends.ConfigurableElasticSearchEngine',
'URL': 'http://127.0.0.1:9200/',
'TIMEOUT': 10000,
'INDEX_NAME': 'jobs',
'EXCLUDED_INDEXES': ['icrm.search_indexes.CVIndex', 'job.search_indexes.ArchiveIndex', 'spontaneousmail.search_indexes.SpontaneousProfileIndex'],
},
curl 'http://159.69.65.2:9200/_cluster/health?pretty=true'
{
"cluster_name" : "vindazofr",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 0,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 2,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 0.0
}
Problem with shards when index is created.
Show unassigned_shards
curl -XGET http://159.69.65.213:9200/_cat/shards | grep UNASSIGNED | awk {'print $1'}
curl -H'Content-Type: application/json' -XPUT '159.69.65.2:9200/_settings' -d '{"index.routing.allocation.disable_allocation": false}'
or check the breaking changes documentation for removed settings
If you get this error
Content-Type header [application/x-www-form-urlencoded] is not supported
This error is due to strict content-type checking introduced in ElasticSearch 6.0, as explained in https://www.elastic.co/blog/strict-content-type-checking-for-elasticsearch-rest-requests
-H'Content-Type: application/json'
curl -XGET http://159.69.65.2:9200/_cat/shards
First use cluster health API to get the current health of the cluster, where RED means one or more primary shards missing and Yellow means one or more replica shards are missing.
After this use the cluster allocation explain API to know why a particular shard is missing and elasticsearch is not able to allocate it on data-node.
Once you get the exact root cause, try to address the issue, which often requires, changing few cluster settings(mentioned in @wilfred answer earlier) But in some cases, if it's replica shards, and you have another copy of same shard(ie another replica) available, you can reduce the replica count using update replica setting and later on again increase it, if you need it.
Apart from above, if your cluster allocation API mentions it doesn't have a valid data node to allocate a shard, then you need to add a new data node, or change the shard allocation awareness settings.
curl -X GET "159.69.65.2:9200/_cluster/allocation/explain?pretty=true"
{"index":"jobs","shard":0,"primary":true,"current_state":"unassigned","unassigned_info":{"reason":"CLUSTER_RECOVERED","at":"2021-03-01T08:31:52.432Z","last_allocation_status":"no"},"can_allocate":"no","allocate_explanation":"cannot allocate because allocation is not permitted to any of the nodes"}
To make your node yellow you have only to add “data” in the role.
vim /etc/elasticsearch/elasticsearch.yml
node.roles: [master, data]
But if you want green status.
And as you have a single node cluster, so you will not have another node where your replicas can be assigned.
Solutions
Add more nodes to your cluster, so that replicas can be assigned on other nodes. (preferred way)
More information about Node, Cluster and roles
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html
Configuration examples
Configuration example If you are running a single node of Elasticsearch, then you have a cluster of one node.
cluster.name: vindazofr
node.name: vindazofrReplica
node.roles: [master, data, data_content, data_hot, data_warm, data_cold, ingest, ml, remote_cluster_client, transform]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 159.69.65.2
http.port: 9200
discovery.seed_hosts: ["159.69.65.2"]
cluster.initial_master_nodes: ["vindazofrDb"]
cluster.routing.allocation.enable: all
Configuration cluster with 2 nodes
cluster.name: vindazofr
node.name: vindazofrReplica
node.roles: [master, data, data_content, data_hot, data_warm, data_cold, ingest, ml, remote_cluster_client, transform]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 144.76.157.29
http.port: 9200
discovery.seed_hosts: ["159.69.65.213", "144.76.157.29"]
cluster.initial_master_nodes: ["vindazofrDb", "vindazofrReplica"]
cluster.routing.allocation.enable: all
Django related errors
Parsing error in django haystack backend.
Elasticsearch using 'payed:(True)': RequestError(400, u'parsing_exception'
Elasticsearch already deprecated filtered query. Use bool instead.
{ "query": { "bool": { "filter": {} } } }
Big parts of it could also be used by haystack to abstract away some of the differences between elastic versions (filtered query vs {"bool": {"filter": []}} for example) and make the overall haystack code a bit simpler. It even provides hooks for custom (de)serialization and other goodies. I would also be happy to add more if it would make haystack easier to port.
Elasticsearch 1.x and 2.x. Elasticsearch 5.x is not supported yet, if you would like to help, please see #1383.
Elasticsearch install old version for vindazo ( with Django haystack )
It is very important to run the same version of Elasticsearch on all nodes in cluster.
For django haystack "number" : "2.3.5",
If you plan to use elastic integrated with Django then it would be better to use an old version of Elastic or replace Haystack with Django-Elasticsearch-DSL (Not tested)
https://medium.com/analytics-vidhya/integrating-elasticsearch-7-to-django-project-c3812de78246
Downgrade you would like to
apt-get remove elasticsearch
(Reading database ... 58301 files and directories currently installed.)
Removing elasticsearch (7.11.1) ...
Stopping elasticsearch service... OK
Deleting log directory... OK
Download package from site.
wget
https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/deb/elasticsearch/2.3.5/elasticsearch-2.3.5.deb
sudo dpkg -i elasticsearch-2.3.5.deb
For this version you have to install Java..
sudo apt install openjdk-8-jdk
vim /etc/elasticsearch/elasticsearch.yml
Don’t forget to add in old configuration hosts to ping via discovery module
discovery.zen.ping.unicast.hosts: ["159.69.65.2", "144.76.157.4"]
And only one should be installed as master i think.
See more information.
https://www.elastic.co/guide/en/elasticsearch/reference/2.4/modules-discovery-zen.html
curl 'http://144.76.157.4:9200/_cat/indices?pretty=true'
Old version config Example
vim /etc/elasticsearch/elasticsearch.yml
cluster.name: vindazofr
node.name: vindazofrReplica
node.roles: [data, data_content, data_hot, data_warm, data_cold, ingest, ml, remote_cluster_client, transform]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 144.76.157.29
http.port: 9200
discovery.seed_hosts: ["159.69.65.213", "144.76.157.29"]
discovery.zen.ping.unicast.hosts: ["159.69.65.213", "144.76.157.29"]
cluster.routing.allocation.enable: all
Don't forget firewall configuration
vim /etc/ufw/user.rules
### tuple ### allow any 9200 0.0.0.0/0 any 88.99.98.11 in
-A ufw-user-input -p tcp --dport 9200 -s 88.99.98.11 -j ACCEPT
-A ufw-user-input -p udp --dport 9200 -s 88.99.98.11 -j ACCEPT
### tuple ### allow any 9300 0.0.0.0/0 any 88.99.98.11 in
-A ufw-user-input -p tcp --dport 9300 -s 88.99.98.11 -j ACCEPT
-A ufw-user-input -p udp --dport 9300 -s 88.99.98.11 -j ACCEPT
-A ufw-user-input -p tcp --dport 9200 -s 88.99.98.11 -j ACCEPT
-A ufw-user-input -p udp --dport 9200 -s 88.99.98.11 -j ACCEPT
### tuple ### allow any 9300 0.0.0.0/0 any 88.99.98.11 in
-A ufw-user-input -p tcp --dport 9300 -s 88.99.98.11 -j ACCEPT
-A ufw-user-input -p udp --dport 9300 -s 88.99.98.11 -j ACCEPT
List of common used command
curl 'http://138.201.8.28:9200/_cat/indices?pretty=true'
curl 'http://138.201.8.28:9200/_cluster/health?pretty=true'
curl -XGET http://159.69.65.2:9200/_cat/shards
Comments
Post a Comment