That doesn't matter if you even have 128 GB Ram and 16 processors with fast SSD or scsi hard disk. You both run into problems in this configuration. In our experience per site, the following configurations work better.
Here in this post I will go through the installation and configuration process of a number of databases and Elasticsearch services so that they can work together in a Django project. I'll cryptically run through video key points and see the rest in text documentation on our site and search google if anything isn't clear.
Install and configuration postgres
Buy server AX61-NVMe
AMD Ryzen™ 9 3900
Simultaneous Multithreading
Check OS version in Ubuntu Linux
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.10
Release: 20.10
Codename: groovy
Always try to install the same version so that you have the same version everywhere. That is important for the success and speed of your configuration.
Also see what support is offered for which version. Version for LTS is usually better but sometimes it is limited. With software we use like Postgres and Elastic will be ok. So choose 22 LTS.
The price of such a server is of course dependent on the provider but approx.
Install server and create user and database.
apt-get install postgresql
sudo -u postgres createuser -D -P vindazo_nl
sudo -u postgres createdb -O vindazo_nl vindazo_nl
Sometimes you need to install a pooling for connections. But that is not always necessary per cluster / site you only need 1 setting of pool.
apt-get install pgpool2
Now you can add data via sql file.. Copy data from the backup host to your download directory.
/home/download
For example via scp or ftp command..
scp vindazo_nl.sql.gz root@23.88.74.XXX:/home/download
gzip -d vindazo_nl.sql.gz
Set up data
psql -Uusername -Wpassword -hlocalhost -p 5432 -f backup.sql
Now you need to configure the server and I like this website. https://www.pgconfig.org/ You can enter all hardware parameters and you will get the correct configuration. Then you just need to copy everything in the config file and comment options that are changed.
Now you update django setting for connection
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'vindazo',
'USER': 'vindazo',
'PASSWORD': ‘xxxx’,
'HOST': '23.88.74.XXX,
'PORT': '5432',
'AUTOCOMMIT': True,
You may get an error like this
FATAL: no pg_hba.conf entry for host
To fix this issue you have to add a Client host for connections to the database..
vim /etc/postgresql/14/main/pg_hba.conf
hostnossl vindazo vindazo 148.251.89.XXX/0 trust
service postgresql restart
So when it starts and you see the admin panel.
from django.contrib.auth.models import User
users = User.objects.filter(is_superuser=True)
user.set_password('pwd')
Sometimes you have problems with certain processes that crash or stop working for some reason. You can find and delete these processes with this command:
ps -ax |grep <process name>
Kill -9 <process id>
It often happens that on an overloaded server backup takes several days and until backup is done and ready to transfer data, it is not recent in general. Therefore you can write certain views to save data twice. Or command to sync data from two databases. This way you backup with full data and then sync data with a separate command. You can do it via SQL or via Django objects with python commands. Editing videos where a lot of data comes in is better extended with an extra function so that new records are always saved.
su postgres
psql -U postgres -p 5432
DROP DATABASE vindazo_nl;
Install Pgpool on the server with database
Install service from package manager
apt-get install pgpool2
vim /etc/pgpool2/pgpool.conf
Update configuration file for external connection.
Server is in default configuration ‘localhost’ or something like this.
So you
========
conn = _connect(dsn, connection_factory=connection_factory, async=async)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "23.88.74.XXX" and accepting
TCP/IP connections on port 5433?
listen_addresses = '*'
service pgpool2 restart
Then you will get error messages like if your configuration is wrong.
Not working
======
connection = Database.connect(**conn_params)
File "/home/admin/deployment/nl_vindazo/venv/local/lib/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect
conn = _connect(dsn, connection_factory=connection_factory, async=async)
django.db.utils.OperationalError: ERROR: failed to authenticate with backend using SCRAM
DETAIL: valid password not found
This is authentication problem
# Use pool_hba.conf for client authentication
pool_passwd = 'pool_passwd'
If you want to allow non-local connections, you need to add more
"host" records. In that case you will also need to make pgpool listen
on a non-local interface via the listen_addresses configuration parameter.
For example config
host all all 48.251.89.72/24 trust
vim /etc/pgpool2/pgpool.conf
Very important hostname as IP and listen_addresses = '*'
backend_hostname0 = '23.88.74.XXX'
backend_port0 = 5432
backend_weight0 = 1
backend_data_directory0 = '/var/lib/pgsql/data'
backend_flag0 = 'DISALLOW_TO_FAILOVER'
backend_application_name0 = 'master'
UFW firewall configuration example
Edit configuration and restart service. Make sure only the right IPs and ports are open for the right services.
vim /etc/ufw/user.rules
*filter
:ufw-user-input - [0:0]
:ufw-user-output - [0:0]
:ufw-user-forward - [0:0]
:ufw-before-logging-input - [0:0]
:ufw-before-logging-output - [0:0]
:ufw-before-logging-forward - [0:0]
:ufw-user-logging-input - [0:0]
:ufw-user-logging-output - [0:0]
:ufw-user-logging-forward - [0:0]
:ufw-after-logging-input - [0:0]
:ufw-after-logging-output - [0:0]
:ufw-after-logging-forward - [0:0]
:ufw-logging-deny - [0:0]
:ufw-logging-allow - [0:0]
:ufw-user-limit - [0:0]
:ufw-user-limit-accept - [0:0]
### RULES ###
### tuple ### allow any 9200 0.0.0.0/0 any 88.99.98.XXX in
-A ufw-user-input -p tcp --dport 9200 -s 88.99.98.89 -j ACCEPT
-A ufw-user-input -p udp --dport 9200 -s 88.99.98.XXX -j ACCEPT
### tuple ### allow any 9300 0.0.0.0/0 any 88.99.98.XXX in
-A ufw-user-input -p tcp --dport 9300 -s 88.99.98.XXX -j ACCEPT
-A ufw-user-input -p udp --dport 9300 -s 88.99.98.XXX -j ACCEPT
### tuple ### allow any 5433 0.0.0.0/0 any 88.99.98.XXX in
-A ufw-user-input -p tcp --dport 5433 -s 88.99.98.89 -j ACCEPT
-A ufw-user-input -p udp --dport 5433 -s 88.99.98.89 -j ACCEPT
### tuple ### allow any 5433 0.0.0.0/0 any 138.201.8.XXX in
-A ufw-user-input -p tcp --dport 5433 -s 138.201.8.118 -j ACCEPT
-A ufw-user-input -p udp --dport 5433 -s 138.201.8.118 -j ACCEPT
### tuple ### allow any 22 0.0.0.0/0 any 88.99.98.XXX in
-A ufw-user-input -p tcp --dport 22 -s 88.99.98.89 -j ACCEPT
-A ufw-user-input -p udp --dport 22 -s 88.99.98.89 -j ACCEPT
### END RULES ###
### LOGGING ###
-A ufw-after-logging-input -j LOG --log-prefix "[UFW BLOCK] " -m limit --limit 3/min --limit-burst 10
-A ufw-after-logging-forward -j LOG --log-prefix "[UFW BLOCK] " -m limit --limit 3/min --limit-burst 10
-I ufw-logging-deny -m conntrack --ctstate INVALID -j RETURN -m limit --limit 3/min --limit-burst 10
-A ufw-logging-deny -j LOG --log-prefix "[UFW BLOCK] " -m limit --limit 3/min --limit-burst 10
-A ufw-logging-allow -j LOG --log-prefix "[UFW ALLOW] " -m limit --limit 3/min --limit-burst 10
### END LOGGING ###
### RATE LIMITING ###
-A ufw-user-limit -m limit --limit 3/minute -j LOG --log-prefix "[UFW LIMIT BLOCK] "
-A ufw-user-limit -j REJECT
-A ufw-user-limit-accept -j ACCEPT
### END RATE LIMITING ###
COMMIT
Install Elasticsearch in replication
Mostly simple installation but has its own specific issues with versions and haystack modules of Django.
Formerly popular Django application which creates an abstraction like data ORM for database but for Elasticsearch and has abstraction for lookup, edit and delete objects from Elasticsearch.
https://www.webdeveloper.today/2021/03/elasticsearch-install-and-configuration.html
In many cases projects developed more than 5 years ago use configurations such as .
It is very important to run the same version of Elasticsearch on all nodes in the cluster.
"number" : "2.3.5",
============== Data node
cluster.name: vindazonl
node.name: "dbElastic"
node.roles: [data, data_content, data_hot, data_warm, data_cold, ingest, ml, remote_cluster_client, transform]
bootstrap.mlockall: true
network.host: 23.88.74.232
discovery.seed_hosts: ["148.251.89.72", "23.88.74.232"]
discovery.zen.ping.unicast.hosts: ["148.251.89.72", "23.88.74.232"]
cluster.routing.allocation.enable: all
============== Master
cluster.name: vindazonl
node.name: nodeWeb
node.roles: [master, data, data_content, data_hot, data_warm, data_cold, ingest, ml, remote_cluster_client, transform]
bootstrap.mlockall: true
network.host: 148.251.89.72
http.port: 9200
cluster.initial_master_nodes: ["148.251.89.72"]
discovery.seed_hosts: ["148.251.89.72", "23.88.74.232"]
discovery.zen.ping.unicast.hosts: ["148.251.89.72", "23.88.74.232"]
cluster.routing.allocation.enable: all
Here is important information
node.roles: [data, data_content, data_hot, data_warm, data_cold, ingest, ml, remote_cluster_client, transform] could be master or data only
curl 'http://23.88.74.XXX:9200/_cluster/health?pretty=true'
curl 'http://148.251.89.XXX:9200/_cluster/health?pretty=true'
curl 'http://159.69.65.213:9200/_cluster/health?pretty=true'
You can test your server and see status via http request..
{
"cluster_name" : "vindazonl",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
{
"cluster_name" : "vindazofr",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 25,
"active_shards" : 50,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
You could find more information on elastic site
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-bootstrap-cluster.html
Summary of this document
Try this
shutdown all, setup this two line on config of all node
cluster.initial_master_nodes: ["node-0"]
discovery.seed_hosts:["node-0", "node-1", "node-2", "node-3"]
Now start.
if this does not work you might have to remove your path.data dir
rm -r /var/lib/elasticsearch/*
Troubleshooting
Sometimes you get status with master_not_discovered_exception.
{
"error" : {
"root_cause" : [ {
"type" : "master_not_discovered_exception",
"reason" : null
} ],
"type" : "master_not_discovered_exception",
"reason" : null
},
"status" : 503
}
In many cases it is firewall configuration. Try to disable the firewall and test it again.
Then next problem NullPointerException by transport
[2022-05-30 10:39:45,897][WARN ][transport.netty ] [dbElastic] exception caught on transport layer [[id: 0x8bcd65c7, /23.88.74.232:39912 => /148.251.89.72:9300]], closing connection
java.lang.NullPointerException
This error you can get when different versions of Elasticsearch are running.. For example
curl 'http://148.251.89.72:9200/?pretty=true'
{
"status" : 200,
"name" : "nodeWeb",
"cluster_name" : "vindazonl",
"version" : {"
"number" : "1.7.3",
{
"name" : "dbElastic",
"cluster_name" : "vindazonl",
"version" : {
"number" : "2.3.5",
This is due to a protocol incompatibility between 1.x and 2.x. The easiest way to reproduce this:
Start a 1.x node, bind it to localhost
Start 2.x nodes with default values
2.x node tries to ping 1.x node. NPEs with the exact same stack trace are shown in the log. Why is this happening? The 1.x node throws an exception because it doesn't understand the ping from 2.x and tries to send this exception back to the 2.x node. 1.x relies on Java serialization to serialize exceptions. 2.x doesn't, using our own exception serialization instead. 2.x found an exception on the remote node, but tried to deserialize it using our own serialization instead of the Java serialization used by 1.x. This doesn't cause any exceptions to be read from the line (if you use an assert, an assert is thrown) and returns null. Then dereference this null pointer,
dpkg: unrecoverable fatal error, aborting:
unknown user 'elasticsearch' in statoverride file
dpkg-statoverride --remove /var/log/elasticsearch
dpkg-statoverride --remove /var/lib/elasticsearch
Synchronize data between two database instances
Sometimes when you move from a big database backup to an intensive server that has many registrations then users are not sycron while a backup is being done, maybe 2000 users are registered, stat changed etc..
So you need a synchronisation between two instances, new and old. You can achieve it by creating commands that will select and sync necessary objects with datus between two databases.
https://docs.djangoproject.com/en/4.0/topics/db/multi-db/
Configuration for database connection is very simple.
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'vindazo_nl',
'USER': 'vindazo_nl',
'PASSWORD': 'vindazo_nl',
'HOST': '23.88.74.xxxx',
'PORT': '5433',
'AUTOCOMMIT': True,
},
'old_data': {
'NAME': 'vindazo_nl',
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'USER': 'vindazo_nl',
'PASSWORD': 'vindazo_nl',
'HOST': 'localhost',
'PORT': '5433',
'AUTOCOMMIT': True,
}
}
Command could be something like this.
from django.core.management.base import BaseCommand
from spontaneousmail.models import SpontaneousProfile
import sys, traceback
class Command(BaseCommand):
def handle(self, *args, **options):
"""
Select all profiles created from 19 to today and save it in nieuwe database
"""
profiles = SpontaneousProfile.objects.using('old_data').filter(created__gt='2022-05-18')
import pdb;pdb.set_trace()
print(profiles.count())
for profile in profiles:
try:
profile.save(using='default')
except:
traceback.print_exc(file=sys.stdout)
Use backup and recovery blog post to install backups on the server.
https://www.webdeveloper.today/2021/04/backup-recovery-strategy.html
Comments
Post a Comment