Skip to main content

Firebase Realtime Database / Cloud Firestore Connection Python Django

It is always interesting to combine firebase with another server such as elasticsearch or crawler or mailing server on a separate server.



Then of course a connection is needed, we use python in this example.

sudo pip install firebase-admin



import firebase_admin

from firebase_admin import credentials


cred = credentials.Certificate("path/to/serviceAccountKey.json")

firebase_admin.initialize_app(cred)



https://firebase.google.com/docs/admin/setup#python 



Init and Retrieve Data from 

https://firebase.google.com/docs/database/admin/start 

https://firebase.google.com/docs/database/admin/retrieve-data




def handle(self, *args, **options):

        cred = credentials.Certificate("/")

        firebase_admin.initialize_app(cred, {

                'databaseURL': 'https://database.firebaseio.com'

        })

        jobs = Job.objects.all()[:3]

        ref = db.reference("/jobs")

        import pdb;pdb.set_trace()

        for job in jobs:

            json = self.create_json(job)

            ref.push(json)



    def create_json(self, job):

        json = {

        "id": job.source_unique,

        "company_name": self.clean_json_value(job.company_name),

        "description": self.clean_json_value(job.description),

        "url": self.clean_json_value(job.url),

        "title": self.clean_json_value(job.title),

        "city": self.clean_json_value(job.city),

        "zip_code": self.clean_json_value(job.zip_code)

        }

        return json



snapshot = ref.order_by_child('id').equal_to().get()




If you get an error something like this.


firebase_admin.exceptions.InvalidArgumentError: Index not defined, add ".indexOn": "id", for path "/jobs", to the rules

https://firebase.google.com/docs/database/security/indexing-data 


Add index in your database rules 




for job in jobs:

            json = self.create_json(job)

            snapshot = ref.order_by_child('id').equal_to(job.source_unique).get()

            if not snapshot:

                ref.push(json)

 



Example read data via Javascript


var firebaseConfig = { .... }

 

firebase.initializeApp(firebaseConfig);
 var db = firebase.database();


    var ref = db.ref("jobs");
    ref.orderByChild("id").equalTo(job_id).on("value", function(snapshot) {
        obj = snapshot.val();
        window.location = obj[Object.keys(obj)[0]]["url"];
        //console.log(snapshot.val());
    });


    
    var ref2 = db.ref("jobs/-MWEL0phaE5glY_PjsPa/url");
    ref2.on('value', (snapshot) => {
        data = snapshot.val();
        console.log(data);
    });



Cloud Firestore 


 If database is efficient with object handling and frequent updates then Firestore is effective for complicated queries. Here is very good document about difference.

https://firebase.google.com/docs/database/rtdb-vs-firestore

Connection and authentication is the same as by Realtime Database but there are three layers to our connection.

Database > Collection > Document


from firebase_admin import credentials, firestore


db = firestore.client()  # this connects to our Firestore database

collection = db.collection(‘jobs’)  # opens 'places' collection

doc = collection.document(‘python-developer’)  # specifies the 'rome' document








Each layer has its own set of methods that enable us to perform different operations at the database, collection, or document level.


We use the get method to retrieve data. Let's use this method to get pythondeloper documentation:


doc = collection.document(‘python-developer’)

res = doc.get().to_dict()

print(res)



When collecting, we can also perform a .get() operation to return an array of all documents contained in it. If we have two documents, it will look like this:






To select advanced queries or quick test you can use this tool.






Add documents from the django query set. 


jobs = Job.objects.filter(for_index=False)


        for job in jobs:

            json = self.create_json(job)

            if not collection.document(job.source_unique).get().exists:

                res = collection.document(job.source_unique).set(json)

                print(res)

Delete by key. 


for job in offline_jobs:

            print("Delete:" + str(job.source_unique))

            collection.document(job.source_unique).delete()


Perform simple and compound queries in Cloud Firestore

https://firebase.google.com/docs/firestore/query-data/queries

python 

        firebase_admin.initialize_app(cred)
        db = firestore.client()
        jobs = db.collection("jobs")
        jobs.where(u"category", u"==", category)
        jobs = self.convert_to_dict(jobs.get())


Very important use unicode in WHERE and stream if it collection filter

 

 

Get started with Cloud Firestore Security Rules

https://firebase.google.com/docs/firestore/security/get-started


Use Shell to test your Rules 




Simple Example Rules


rules_version = '2';

service cloud.firestore {

  match /databases/{database}/documents {

    match /clicks/{document=**} {

      allow write: if true;

      allow read: if false;

    }

    match /jobs/{document=**} {

      allow write: if false;

      allow read: if true;

    }

    match /{document=**} {

      allow read, write: if false;

    }

    

  }

}



Why use Cloudflare with Firebase?


  1. Security & Protection

  2. Cost Savings & Prevention

  3. Speed


Have you heard of the infamous Firebase cost increase by 7000%? If you are using the Blaze plan, you will be charged as needed. If you do not monitor costs carefully, you may receive unwelcome bills at the end of the month.


If you save timestamps you can get all the information from logs. Like IP, User agent etc..




Comments

Popular posts from this blog

Tekstverwerking python Text processing python SpaCy, TensorFlow, NLTK, Allen-NLP, Stanford-NLP

 Dit post maakt gebruik van spaCy, een populaire Python-bibliotheek die de taalgegevens en algoritmen bevat die je nodig hebt om teksten in natuurlijke taal te verwerken. Zoals u in dit post zult leren, is spaCy gemakkelijk te gebruiken omdat het containerobjecten biedt die elementen van natuurlijke taalteksten vertegenwoordigen, zoals zinnen en woorden. Deze objecten hebben op hun beurt attributen die taalkenmerken vertegenwoordigen, zoals delen van spraak. Op het moment van schrijven bood spaCy voorgetrainde modellen aan voor Engels, Duits, Grieks, Spaans, Frans, Italiaans, Litouws, Noors BokmÃ¥l, Nederlands, Portugees en meerdere talen gecombineerd. Bovendien biedt spaCy ingebouwde visualizers die u programmatisch kunt aanroepen om een grafische weergave van de syntactische structuur van een zin of benoemde entiteiten in een document te genereren.   De spaCy-bibliotheek ondersteunt ook native geavanceerde NLP-functies die andere populaire NLP-bibliotheken voor Python niet hebben. Spa

Google Closure

   Closure Library De Closure-bibliotheek is een JavaScript-bibliotheek, vergelijkbaar met andere moderne producten zoals jQuery, Angular, Vue.js, Dojo en MooTools. De coderingsstijl en het gebruik van opmerkingen in de Closure-bibliotheek zijn op maat gemaakt voor Closure Compiler. In vergelijking met andere JavaScript-bibliotheken is het de belangrijkste onderscheidende factor van Closure Compiler. Een eenvoudig compressie-experiment ontdekte dat wanneer Closure Compiler wordt gebruikt in plaats van YUI Compressor, de Closure Lib-code met 85% kan worden verminderd, wat een enorme impact kan hebben op de codecompressiecapaciteit van de compiler.    De implementatie van de  closure bibliotheek richt zich ook op leesbaarheid en prestaties. Wees zuinig bij het maken van objecten, maar wees genereus bij het benoemen en opnemen van objecten. Het heeft ook een prachtig gebeurtenissysteem, ondersteuning voor klassen en overerving en verschillende UI-componenten, waaronder bijvoorbeeld een ri

Elasticsearch install and configuration on Ubuntu

If you plan to use elastic integrated with Django then it would be better to use an old version of Elastic or replace Haystack with Django-Elasticsearch-DSL (Not tested) See old version install at bottom of this post.  New version Elasticsearch ( attention no haystack at this time )  The Elasticsearch components are not available in Ubuntu’s default package repositories. They can, however, be installed with APT after adding Elastic’s package source list. curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation apt-get install gnupg curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). OK echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list apt update apt