Postgres into a distributed database Citus Django Ubuntu

Citus is an open source extension of Postgres that can distribute data and queries among multiple nodes in a cluster. Because Citus is an extension (not a branch) of Postgres, when you use Citus, you are also using Postgres. You can take advantage of the latest Postgres features, tools, and ecosystem.




In apt-get I could not add repositories because it is configured by default only have support for only LTS versions of operating systems.

I will try manually downloading and installing the package.

https://repos.citusdata.com/community/  


This repository is all moved… 


I found packages on this repository. 

https://packagecloud.io/citusdata/community 

wget --content-disposition https://packagecloud.io/citusdata/community/packages/ubuntu/focal/postgresql-12-citus-10.0_10.0.2.citus-1_amd64.deb/download.deb

 sudo dpkg -i postgresql-12-citus-10.0_10.0.2.citus-1_amd64.deb


Package libpq5 is not installed.
postgresql-12-citus-10.0 depends on postgresql-12; however
Package postgresql-12 is not installed.
dpkg: error processing package postgresql-12-citus-10.0 (--install):
dependency problems - leaving unconfigured
Errors were encountered while processing:

postgresql-12-citus-10.0
 

Depends


libc6 (>= 2.17), libcurl4 (>= 7.16.2), liblz4-1 (>= 0.0~r130), libpq5 (>= 9.2~beta3), libssl1.1 (>= 1.1.0), libzstd1 (>= 1.3.2), postgresql-12

 

apt --fix-broken install




sudo dpkg -i postgresql-12-citus-10.0_10.0.2.citus-1_amd64.deb


(Reading database ... 78223 files and directories currently installed.)

Preparing to unpack postgresql-12-citus-10.0_10.0.2.citus-1_amd64.deb ...

Unpacking postgresql-12-citus-10.0 (10.0.2.citus-1) over (10.0.2.citus-1) ...

Setting up postgresql-12-citus-10.0 (10.0.2.citus-1) ...

Processing triggers for postgresql-common (225.pgdg20.10+1) ...

Building PostgreSQL dictionaries from installed myspell/hunspell packages...

Removing obsolete dictionary files:




Ok, now we will install django with Google's F1 paper is a good example, showing a multi-tenant database expanded in this way. This article discusses the technical challenges associated with expanding the Google AdWords platform; it is centered on a multi-tenant database. The F1 paper also focused on how to best model data to support many tenants/customers in a distributed database.

http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41344.pdf 


Probably, the known example of this model is

http://www.developerforce.com/media/ForcedotcomBookLibrary/Force.com_Multitenancy_WP_101508.pdf 



Citus configuration 


sudo su - postgres

mkdir citus

export PATH=$PATH:/usr/lib/postgresql/12/bin/

initdb -D citus

pg_ctl -D citus -o "-p 9700" -l citus_logfile start

psql -p 9700 -c "CREATE EXTENSION citus;"

psql -p 9700 -c "select citus_version();"


You can connect directly to postgres via 


postrges psql -p 5432

psql (12.6 (Ubuntu 12.6-1.pgdg20.10+1))

Type "help" for help.



Check first python version and install django 


 python3

Python 3.8.6 (default, Jan 27 2021, 15:42:20) [GCC 10.2.0] on linux

sudo apt install python3-pip
pip3 --version
pip 20.1.1 from /usr/lib/python3/dist-packages/pip (python 3.8)
pip3 install Django
python3


Python 3.8.6 (default, Jan 27 2021, 15:42:20)
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.


>>> import django
>>> print(django.get_version())
3.1.7

 

Ok, You can now install apps and add databases for django applications. 




sudo -u postgres createuser -D -A -P vacatures

Enter password for new role:

Enter it again:

root@Ubuntu-2010-groovy-64-minimal /home/sites/vacatures # sudo -u postgres createdb -O vacatures vacatures




Install connection pooling for Postgres.. 


apt install pgpool2 


Default configuration is fine if you want to use postgres installation from Ubunut 

Or config it for citus.. 


vim /etc/pgpool2/pgpool.conf 


backend_port0 = 9700 


Try to connect 


psql -Uvacatures.today -Wvacatures.today -hlocalhost -p 5433

Password:

psql (12.6 (Ubuntu 12.6-1.pgdg20.10+1))

Type "help" for help.


vacatures.today=>




For more information about table distribution or references in citus cluster see documentation http://docs.citusdata.com/en/v10.0/


Furthermore I will focus only on Django applications.

Create Django application


django-admin startproject vacatures


Postgresql configuration 


DATABASES = {

    'default': {

       'ENGINE': 'django.db.backends.postgresql_psycopg2',

       'NAME': 'vacatures',

       'USER': 'vacature',

       'PASSWORD': 'vacatures',

       'HOST': 'localhost',

       'PORT': '5433', # pgpool port not postgres or citus… 

       'AUTOCOMMIT': True,

    }

}




sudo apt install python3-dev libpq-dev

pip install psycopg2

python3 manage.py migrate





Furthermore, I will move old django applications into a new setup and fix masses of minor bugs related to Django changes. So this one is no longer related to this post, I can close it.








Comments