Django project add extra host with Postgressql database and Elasticsearch

If you run all services of a django project on 1 server that is Web, DB, Elastic and Mail server. That could be that your architecture will cause problems. Because all services compete with each other for resources and will overload each other. Especially if it's not about a small site but already a working site hosting let's say thousands of users per day.

That doesn't matter if you even have 128 GB Ram and 16 processors with fast SSD or scsi hard disk. You both run into problems in this configuration. In our experience per site, the following configurations work better.

Here in this post I will go through the installation and configuration process of a number of databases and Elasticsearch services so that they can work together in a Django project. I'll cryptically run through video key points and see the rest in text documentation on our site and search google if anything isn't clear.

Install and configuration postgres 

Buy server AX61-NVMe

AMD Ryzen™ 9 3900

Simultaneous Multithreading 




2 x 1.92 TB NVMe SSD Datacenter Edition


1 GBit/s-Port

Guaranteed bandwidth:

1 GBit/s

Check OS version in Ubuntu Linux

lsb_release -a

No LSB modules are available.

Distributor ID:    Ubuntu

Description:    Ubuntu 20.10

Release:    20.10

Codename:    groovy

Always try to install the same version so that you have the same version everywhere. That is important for the success and speed of your configuration.

Also see what support is offered for which version. Version for LTS is usually better but sometimes it is limited. With software we use like Postgres and Elastic will be ok. So choose 22 LTS.

The price of such a server is of course dependent on the provider but approx.

Install server and create user and database. 

apt-get install postgresql

sudo -u postgres createuser -D -P vindazo_nl 

sudo -u postgres createdb -O vindazo_nl vindazo_nl


Sometimes you need to install a pooling for connections. But that is not always necessary per cluster / site you only need 1 setting of pool.

apt-get install pgpool2

Now you can add data via sql file.. Copy data from the backup host to your download directory. 


For example via scp or ftp command.. 

scp vindazo_nl.sql.gz root@23.88.74.XXX:/home/download

gzip -d vindazo_nl.sql.gz 


Set up data

psql -Uusername -Wpassword -hlocalhost -p 5432 -f backup.sql


Now you need to configure the server and I like  this website. You can enter all hardware parameters and you will get the correct configuration. Then you just need to copy everything in the config file and comment options that are changed. 

Now you update django setting for connection

'default': {

        'ENGINE': 'django.db.backends.postgresql_psycopg2',

        'NAME': 'vindazo',

        'USER': 'vindazo',

        'PASSWORD': ‘xxxx’,

        'HOST': '23.88.74.XXX,

        'PORT': '5432',

        'AUTOCOMMIT': True,

You may get an error like this 

FATAL:  no pg_hba.conf entry for host

To fix this issue you have to add a Client host for connections to the database.. 

vim /etc/postgresql/14/main/pg_hba.conf

hostnossl    vindazo  vindazo     148.251.89.XXX/0        trust

service postgresql restart

So when it starts and you see the admin panel. 

from django.contrib.auth.models import User

users = User.objects.filter(is_superuser=True) 


Sometimes you have problems with certain processes that crash or stop working for some reason. You can find and delete these processes with this command: 

ps -ax |grep <process name>

Kill -9 <process id>

It often happens that on an overloaded server backup takes several days and until backup is done and ready to transfer data, it is not recent in general.  Therefore you can write certain views to save data twice. Or command to sync data from two databases. This way you backup with full data and then sync data with a separate command. You can do it via SQL or via Django objects with python commands.  Editing videos where a lot of data comes in is better extended with an extra function so that new records are always saved. 

su postgres

psql -U postgres -p 5432

DROP DATABASE vindazo_nl; 

Install Pgpool on the server with database

Install service from package manager 

apt-get install pgpool2

vim /etc/pgpool2/pgpool.conf

Update configuration file for external connection.  

Server is in default configuration ‘localhost’ or something like this. 

So you 


conn = _connect(dsn, connection_factory=connection_factory, async=async)

django.db.utils.OperationalError: could not connect to server: Connection refused

        Is the server running on host "23.88.74.XXX" and accepting

        TCP/IP connections on port 5433?

listen_addresses = '*'

service pgpool2 restart

Then you will  get error messages like if your configuration is wrong. 

Not working 


connection = Database.connect(**conn_params)

  File "/home/admin/deployment/nl_vindazo/venv/local/lib/python2.7/site-packages/psycopg2/", line 164, in connect

    conn = _connect(dsn, connection_factory=connection_factory, async=async)

django.db.utils.OperationalError: ERROR:  failed to authenticate with backend using SCRAM

DETAIL:  valid password not found

This is authentication problem 

 # Use pool_hba.conf for client authentication

pool_passwd = 'pool_passwd'

If you want to allow non-local connections, you need to add more

"host" records. In that case you will also need to make pgpool listen

 on a non-local interface via the listen_addresses configuration parameter.

For example config 

host    all         all       trust 

vim /etc/pgpool2/pgpool.conf

Very important hostname as IP and listen_addresses = '*' 

backend_hostname0 = '23.88.74.XXX'

backend_port0 = 5432

backend_weight0 = 1

backend_data_directory0 = '/var/lib/pgsql/data'

backend_flag0 = 'DISALLOW_TO_FAILOVER'

backend_application_name0 = 'master'

UFW firewall configuration example 

Edit configuration and restart service. Make sure only the right IPs and ports are open for the right services. 

vim /etc/ufw/user.rules


:ufw-user-input - [0:0]

:ufw-user-output - [0:0]

:ufw-user-forward - [0:0]

:ufw-before-logging-input - [0:0]

:ufw-before-logging-output - [0:0]

:ufw-before-logging-forward - [0:0]

:ufw-user-logging-input - [0:0]

:ufw-user-logging-output - [0:0]

:ufw-user-logging-forward - [0:0]

:ufw-after-logging-input - [0:0]

:ufw-after-logging-output - [0:0]

:ufw-after-logging-forward - [0:0]

:ufw-logging-deny - [0:0]

:ufw-logging-allow - [0:0]

:ufw-user-limit - [0:0]

:ufw-user-limit-accept - [0:0]

### RULES ###

### tuple ### allow any 9200 any 88.99.98.XXX in

-A ufw-user-input -p tcp --dport 9200 -s -j ACCEPT

-A ufw-user-input -p udp --dport 9200 -s 88.99.98.XXX -j ACCEPT

### tuple ### allow any 9300 any 88.99.98.XXX in

-A ufw-user-input -p tcp --dport 9300 -s 88.99.98.XXX  -j ACCEPT

-A ufw-user-input -p udp --dport 9300 -s 88.99.98.XXX -j ACCEPT

### tuple ### allow any 5433 any 88.99.98.XXX in

-A ufw-user-input -p tcp --dport 5433 -s -j ACCEPT

-A ufw-user-input -p udp --dport 5433 -s -j ACCEPT

### tuple ### allow any 5433 any 138.201.8.XXX in

-A ufw-user-input -p tcp --dport 5433 -s -j ACCEPT

-A ufw-user-input -p udp --dport 5433 -s -j ACCEPT

### tuple ### allow any 22 any 88.99.98.XXX  in

-A ufw-user-input -p tcp --dport 22 -s -j ACCEPT

-A ufw-user-input -p udp --dport 22 -s -j ACCEPT

### END RULES ###

### LOGGING ###

-A ufw-after-logging-input -j LOG --log-prefix "[UFW BLOCK] " -m limit --limit 3/min --limit-burst 10

-A ufw-after-logging-forward -j LOG --log-prefix "[UFW BLOCK] " -m limit --limit 3/min --limit-burst 10

-I ufw-logging-deny -m conntrack --ctstate INVALID -j RETURN -m limit --limit 3/min --limit-burst 10

-A ufw-logging-deny -j LOG --log-prefix "[UFW BLOCK] " -m limit --limit 3/min --limit-burst 10

-A ufw-logging-allow -j LOG --log-prefix "[UFW ALLOW] " -m limit --limit 3/min --limit-burst 10



-A ufw-user-limit -m limit --limit 3/minute -j LOG --log-prefix "[UFW LIMIT BLOCK] "

-A ufw-user-limit -j REJECT

-A ufw-user-limit-accept -j ACCEPT



Install Elasticsearch in replication 

Mostly simple installation but has its own specific issues with versions and haystack modules of Django.

Formerly popular Django application which creates an abstraction like data ORM for database but for Elasticsearch and has abstraction for lookup, edit and delete objects from Elasticsearch. 

In many cases projects developed more than 5 years ago use configurations such as .

It is very important to run the same version of Elasticsearch on all nodes in the cluster. 

"number" : "2.3.5",

============== Data node vindazonl "dbElastic"

node.roles: [data, data_content, data_hot, data_warm, data_cold, ingest, ml, remote_cluster_client, transform]

bootstrap.mlockall: true

discovery.seed_hosts: ["", ""] ["", ""]

cluster.routing.allocation.enable: all

============== Master vindazonl nodeWeb

node.roles: [master, data, data_content, data_hot, data_warm, data_cold, ingest, ml, remote_cluster_client, transform]

bootstrap.mlockall: true

http.port: 9200

cluster.initial_master_nodes: [""]

discovery.seed_hosts: ["", ""] ["", ""]

cluster.routing.allocation.enable: all

Here is important information

node.roles: [data, data_content, data_hot, data_warm, data_cold, ingest, ml, remote_cluster_client, transform]   could be master or data only

curl 'http://23.88.74.XXX:9200/_cluster/health?pretty=true'

curl 'http://148.251.89.XXX:9200/_cluster/health?pretty=true'

curl ''

You can test your server and see status via http request..


  "cluster_name" : "vindazonl",

  "status" : "green",

  "timed_out" : false,

  "number_of_nodes" : 1,

  "number_of_data_nodes" : 1,

  "active_primary_shards" : 0,

  "active_shards" : 0,

  "relocating_shards" : 0,

  "initializing_shards" : 0,

  "unassigned_shards" : 0,

  "delayed_unassigned_shards" : 0,

  "number_of_pending_tasks" : 0,

  "number_of_in_flight_fetch" : 0,

  "task_max_waiting_in_queue_millis" : 0,

  "active_shards_percent_as_number" : 100.0



  "cluster_name" : "vindazofr",

  "status" : "green",

  "timed_out" : false,

  "number_of_nodes" : 2,

  "number_of_data_nodes" : 2,

  "active_primary_shards" : 25,

  "active_shards" : 50,

  "relocating_shards" : 0,

  "initializing_shards" : 0,

  "unassigned_shards" : 0,

  "delayed_unassigned_shards" : 0,

  "number_of_pending_tasks" : 0,

  "number_of_in_flight_fetch" : 0,

  "task_max_waiting_in_queue_millis" : 0,

  "active_shards_percent_as_number" : 100.0


You could find more information on elastic site 

Summary of this document 


Try this

shutdown all, setup this two line on config of all node

cluster.initial_master_nodes: ["node-0"]

discovery.seed_hosts:["node-0", "node-1", "node-2", "node-3"]

Now start.

if this does not work you might have to remove your dir

 rm  -r /var/lib/elasticsearch/*


Sometimes you get status with master_not_discovered_exception. 


  "error" : {

    "root_cause" : [ {

      "type" : "master_not_discovered_exception",

      "reason" : null

    } ],

    "type" : "master_not_discovered_exception",

    "reason" : null


  "status" : 503


In many cases it is firewall configuration. Try to disable the firewall and test it again. 

Then next problem NullPointerException by transport

[2022-05-30 10:39:45,897][WARN ][transport.netty          ] [dbElastic] exception caught on transport layer [[id: 0x8bcd65c7, / => /]], closing connection


This error you can get when different versions of Elasticsearch are running.. For example

curl ''


  "status" : 200,

  "name" : "nodeWeb",

  "cluster_name" : "vindazonl",

  "version" : {"

    "number" : "1.7.3",


  "name" : "dbElastic",

  "cluster_name" : "vindazonl",

  "version" : {

    "number" : "2.3.5",

This is due to a protocol incompatibility between 1.x and 2.x. The easiest way to reproduce this:

Start a 1.x node, bind it to localhost

Start 2.x nodes with default values

2.x node tries to ping 1.x node. NPEs with the exact same stack trace are shown in the log. Why is this happening? The 1.x node throws an exception because it doesn't understand the ping from 2.x and tries to send this exception back to the 2.x node. 1.x relies on Java serialization to serialize exceptions. 2.x doesn't, using our own exception serialization instead. 2.x found an exception on the remote node, but tried to deserialize it using our own serialization instead of the Java serialization used by 1.x. This doesn't cause any exceptions to be read from the line (if you use an assert, an assert is thrown) and returns null. Then dereference this null pointer,

dpkg: unrecoverable fatal error, aborting:

 unknown user 'elasticsearch' in statoverride file

dpkg-statoverride --remove /var/log/elasticsearch 

dpkg-statoverride --remove /var/lib/elasticsearch

Synchronize data between two database instances

Sometimes when you move from a big database backup to an intensive server that has many registrations then users are not sycron while a backup is being done, maybe 2000 users are registered, stat changed etc..

So you need a synchronisation between two instances, new and old. You can achieve it by creating commands that will select and sync necessary objects with datus between two databases.

Configuration for database connection is very simple. 


    'default': {

        'ENGINE': 'django.db.backends.postgresql_psycopg2',

        'NAME': 'vindazo_nl',

        'USER': 'vindazo_nl',

        'PASSWORD': 'vindazo_nl',

        'HOST': '23.88.74.xxxx',

        'PORT': '5433',

        'AUTOCOMMIT': True,


    'old_data': {

        'NAME': 'vindazo_nl',

        'ENGINE': 'django.db.backends.postgresql_psycopg2',

        'USER': 'vindazo_nl',

        'PASSWORD': 'vindazo_nl',

        'HOST': 'localhost',

        'PORT': '5433',

        'AUTOCOMMIT': True,



Command could be something like this. 

from import BaseCommand

from spontaneousmail.models import SpontaneousProfile

import sys, traceback

class Command(BaseCommand):

    def handle(self, *args, **options):


        Select all profiles created from 19 to today and save it in nieuwe database


        profiles = SpontaneousProfile.objects.using('old_data').filter(created__gt='2022-05-18')

        import pdb;pdb.set_trace()


        for profile in profiles:





Use backup and recovery blog post to install backups on the server.