Skip to main content

Unsubscribe automation cyclus from Sendgrid, MailChimp Spam, Bounce, and blocked messages


It is often necessary in current projects that your mailbox is as empty as possible and people who ask to unsubscribe are helped. Normal method of unsubscribing is just via a form on your website, but usually people work in different ways. They throw message in spam folder and then loopback notification comes to you via Mail provider via Abuse report or via log file at SendGrid or MailChimp. Or people simply forward or reply to a message with a question to delete all data, that can become a very method of managing a mailbox where many automatic notifications are sent. We'll cover standard ways we use to keep feedback loop cleaner in this post.

This document is based on internal documentation and has been made a bit generic so that it can be used in various other cases. Original document can be found in documents cloud with name:

Unsubscribe cyclus from Sendgrid, MailChimp Spam and bounce

Checklijst

  • Default unsubscribe Form
  • Mailbox map
  • Blacklisted global
  • Bounce via sending providers 
  • Spam reports via loopback sending providers

First method Exporting Gmail emails


First source of requests that is the mailbox itself where people forward and ask for their data to be deleted. Sometimes they don't want to click on a link or search or expect a complicated process but that's the fact quite a few unsubscribes end up in mailbox and in order not to manually process such requests one can automate Gmail in various ways.

Export data per label and then read this local mbox file read all messages and unsubscribe senders.

Exporting data might work even better than giving access to your account with script. Because then we can edit archive even better locally.

You can manage all your exports from this page. Create find where they are now etc..

https://takeout.google.com/



This method is good if you want to initiate it manually and it is more secure.

The other real time downloading of messages is not so good I think because you have to make your account less secure, there is no possibility to allow access only by IP, for example. When you open IMAP it is open to everyone and all that is left is your password.
Export is done almost immediately without delay as first mentioned hours or days.. That took 1 minute and I already had my files.. Imported emails via Drive with 1 label..
Now I have a zipped mbox file that I should now be able to read and write out all messages in this file so that they don't exist anymore.


With this library it seems to be possible so we will test first and see how effective it is. For now this solution I think is the best option.. So how do I go about that just add all people in global unsubscribe list so they can leave all other services or work by list. People who have retired or complain can be completely removed from the system, others who have found work can still stay in system but put on inactive so that we ask back in 1 year if they are available.



Code will look something like this depending on what exactly is needed.
An example of script that can be used and further extended to a Django command or a similar script with a correct functional purpose.



# -*- coding: utf-8 -*-
#!/usr/bin/python
#
# Copyright (C) 2021 Web App Development
#


"""


"""

__author__ = 'sergej[A]dergatsjev[D]be (Sergej Dergatsjev)'

from django.db import models, transaction, IntegrityError
from django.core.management.base import BaseCommand
from django.template.defaultfilters import slugify

from crawler import utils
import re
from time import sleep
import sys, traceback
from newsletter.models import Subscriber
import mailbox


class Command(BaseCommand):
def handle(self, *args, **options):
"""
Entery point
"""
#import pdb;pdb.set_trace()
for message in mailbox.mbox("data/Unsubscribe.mbox"):
subject = message['subject']
from_email = message['From']
print("-----------------")
print(from_email)
if from_email.find('MAILER-DAEMON@secure-mail.be') != -1:
from_email = self.parse_from_email(message.as_string(), "for <([\w\-][\w\-\.]+@[\w\-][\w\-\.]+[a-zA-Z]{1,4})>")
print(from_email)
self.unsubscribe(from_email, "Undelivered")
elif from_email.find('<') != 1:
from_email = self.parse_from_email(from_email, "<([\w\-][\w\-\.]+@[\w\-][\w\-\.]+[a-zA-Z]{1,4})>")
self.unsubscribe(from_email, "Unsubscribe Mailbox")
else:
self.unsubscribe(from_email, "Unsubscribe Mailbox")
print(subject)
#email = utils.normalization_email(cols[0])

def parse_from_email(self, text, pattern):
pattern = re.compile(pattern)
result = re.findall(pattern, text)
for item in result:
if "vindazo.be" in item:
result.remove(item)
if len(result) > 0:
return result[0]
else:
return None


def unsubscribe(self, from_email, status):
if from_email:
subscriber = None
print(from_email)
try:
subscriber = Subscriber.objects.get(email=from_email, has_unsubscribe=False)
except:
try:
subscriber = Subscriber.objects.get(email=from_email)
print("All ready unsubscribed")
return None
except:
print("Email not found")

if subscriber:
subscriber.has_unsubscribe = True
subscriber.status = status
subscriber.save()
return from_email
 
Second example.. More complex case with different regular expressions to find "from mail" 
 
# -*- coding: utf-8 -*-
#!/usr/bin/python
#
# Copyright (C) 2020 Web App Development EOOD
#


"""

Unsubscribe via profile

"""

import imaplib
import re
import requests
from datetime import datetime, date
import sys, traceback, os

from time import sleep

from spontaneousmail import utils
from django.core.management.base import BaseCommand
from spontaneousmail.models import SpontaneousProfile
import mailbox

class Command(BaseCommand):
verbose = True
timeout = 5
count = 0
mailbox = "data/Unsubscribe.mbox"

def handle(self, *args, **options):
"""
Unsubscribe via mailbox
"""
contacts = self.read_contacts()
count = 1
for contact in contacts:
print(count)
print(contact['email'])
count = count + 1
utils.unsubscribe(contact['email'], "Unsubscribe_mailbox")

def parse_from_email(self, text, pattern):
pattern = re.compile(pattern)
result = re.findall(pattern, text)
for item in result:
if ("vacaturemails.com" in item) or ("vacatures.today" in item):
result.remove(item)
if len(result) > 0:
return result[0]
else:
return None

def read_contacts(self):
"""
"""
#import pdb;pdb.set_trace()
contacts = []
count = 0
for message in mailbox.mbox(self.mailbox):
count = count + 1
subject = message['subject']
from_email = message['From']
print(count)
print(from_email)
if from_email.find('mailer-daemon') != -1:
# for <info@vacaturemails.com>
#import pdb;pdb.set_trace()
from_email = self.parse_from_email(message.as_string(), "for <([\w\-][\w\-\.]+@[\w\-][\w\-\.]+[a-zA-Z]{1,4})>")
if not from_email:
# Final-Recipient: rfc822; dd.ff@europe.com
from_email = self.parse_from_email(message.as_string(), "Final-Recipient: rfc822; ([\w\-][\w\-\.]+@[\w\-][\w\-\.]+[a-zA-Z]{1,4})")
elif from_email.find('<') != -1:
from_email = self.parse_from_email(from_email, "<([\w\-][\w\-\.]+@[\w\-][\w\-\.]+[a-zA-Z]{1,4})>")
if not from_email:
#import pdb;pdb.set_trace()
from_email = self.parse_from_email(message.as_string(), "To:.*([\w\-][\w\-\.]+@[\w\-][\w\-\.]+[a-zA-Z]{1,4})")
#import pdb;pdb.set_trace()
#print(subject)
if from_email:
contacts.append({"email":from_email})
else:
#import pdb;pdb.set_trace()
f = open("/tmp/emails/" + str(count), "w")
try:
f.write(message.as_string())
except:
traceback.print_exc(file=sys.stdout)
f.close()
return contacts
 

Unsubscribe Real Time via IMAP Gmail

Then the next method is about connection via IMAP and direct access to the correct label and processing all messages under a correct label such as Unsubscribe for example. People who manage mailbox move all mails under this label and script comes once a day to check if there is something to process and move in folder with all other ready messages in archive.

I don't recommend this way because it exposes your account to atteres from outside. But you can do it if you want. We've done it once for less important accounts too.


You will have an issues with login via IMAP.


b'[AUTHENTICATIONFAILED] Invalid credentials (Failure)'


python3 manage.py unsubscribe_imap



https://accounts.google.com/b/0/DisplayUnlockCaptcha

Occasionally Gmail will stop working and you can see a Postie error message that says something like “[ALERT] Please log in via your web browser”. This means you must log into the account Postie is trying to access via https://gmail.com where you will be asked to confirm a login from wherever your server is located.

If this doesn’t work try http://www.google.com/accounts/DisplayUnlockCaptcha

Turn off 2-Step Verification https://support.google.com/accounts/answer/1064203?hl=en If you are still having difficulty you may need to change the Google “Less secure apps” setting. Be sure you are only signed into the gmail account that postie will use then go to https://www.google.com/settings/security/lesssecureapps and make sure it is set to “Turn off”


While the wording on this page makes it sound dangerous, what it really means is “allow traditional authentication” which uses very high levels of encryption.

Finally check these pages for other options:

https://support.google.com/accounts/answer/6009563

https://support.google.com/mail/answer/14257


Code voorbeeld:

def login():

    im = imaplib.IMAP4_SSL(INBOX_IMAP, 993)

    im.login(INBOX_USER, INBOX_PASS)

    return im


def unsubscribe(mailbox, im):

    typ, data = im.search(None, 'ALL')

    for num in data[0].split():

        typ, msg = im.fetch(num, '(RFC822)')

        unsubscribe_message(msg[0][1].decode("utf-8"), num, mailbox)


im = login()

unsubscribe("Unsubscribe", im)


 

 

# -*- coding: utf-8 -*-
#!/usr/bin/python
#
# Copyright (C) 2020 Web App Development EOOD
# Online Solutions Group BVBA


"""

Unsubscribe via profile

"""

import imaplib
import re
import requests
from datetime import datetime, date
import sys, traceback, os

from time import sleep

from spontaneousmail import utils
from django.core.management.base import BaseCommand
from spontaneousmail.models import SpontaneousProfile

INBOX_IMAP = 'imap.gmail.com'
INBOX_USER = ''
INBOX_PASS = ''


class Command(BaseCommand):
def handle(self, *args, **options):
"""
Unsubscribe via imap
"""
im = login()
unsubscribe("Unsubscribe", im)

def login():
im = imaplib.IMAP4_SSL(INBOX_IMAP, 993)
im.login(INBOX_USER, INBOX_PASS)
return im


def is_own(item):
OWN = ["vacaturemails.com", "vacatures.today"]
for own_email in OWN:
if item.find(own_email) != -1:
return True
return False


def get_to(message):
#import pdb;pdb.set_trace()
pattern = re.compile("From: [^\r]*\r\nTo: ([^\r]*)\r\nDate: ")
result = re.findall(pattern, message)
for item in result:
if not is_own(item):
return item
pattern = re.compile("\r\nTo:([^\r]*)\r\n")
result = re.findall(pattern, message)
for item in result:
if not is_own(item):
return item


def get_from(message):
pattern = re.compile("From: ([^\r]*)\r\n")
result = re.findall(pattern, message)
for item in result:
if not is_own(item):
return item



def clean(email):
if email.find("<") != -1:
pattern = re.compile("<([^>]*)>")
return re.findall(pattern, email)[0]
return email.strip()


def unsubscribe_remove(mailbox, im):
# Use onther map to move emails first
# Or if it realy should be deleted
print(im.select(mailbox))
typ, data = im.search(None, 'ALL')
for num in data[0].split():
typ, msg = im.fetch(num, '(RFC822)')
unsubscribe_message(msg[0][1], num, mailbox)
im.store(num, '+FLAGS', '\\Deleted')
im.expunge()

def remove_emails(mailbox, im):
# Like priviews but not used in this script only for exampole saved
print(im.select(mailbox))
typ, data = im.search(None, 'ALL')
import pdb;pdb.set_trace()
count = 0
max_count = 100
total = get_count(im)
while total > 0:
total = get_count(im)
count = count + 1
print(str(count))
if count == max_count:
im.expunge()
count = 1
try:
im.store(count, '+FLAGS', '\\Deleted')
except:
traceback.print_exc(file=sys.stdout)
im.expunge()
im.expunge()




def unsubscribe(mailbox, im):
# Usage like
#for mailbox in FOR_UNSUBSCRIBE:
# unsubscribe(mailbox, im)
print(im.select(mailbox))
import pdb;pdb.set_trace()
typ, data = im.search(None, 'ALL')
for num in data[0].split():
typ, msg = im.fetch(num, '(RFC822)')
unsubscribe_message(msg[0][1].decode("utf-8"), num, mailbox)
#im.expunge() deleted items from selected mailbox

def unsubscribe_message(message, num, label):
email = get_to(message)
if email == None:
email = get_from(message)
if email:
email = clean(email)
print(email)
# Here unsubscribe logic from database or via API url
#print(response)
utils.unsubscribe(email, "Unsubscribe_imap")

def get_count(im):
try:
typ, data = im.search(None, 'ALL')
except:
traceback.print_exc(file=sys.stdout)
im = login()
typ, data = im.search(None, 'ALL')
return len(data[0].split())

SendGrid MailChip unsubscribe automation



SendGrid and MailChip have the option to export their logs so you can export blocked and/or bounced messages from these services and unsubscribe automatically. They use CSV which is text format divided with koma.


Note they do not have the same format and are not in the same order so check the export order carefully.

Code example:

def handle(self, *args, **options):

        """

        """

        files = [

            'data/bounce/suppression_bounces.csv'

        ]

        for file in files:

            print(file)

            f = open(file, "r")

            items = f.readlines()

            self.unsubsribe_emails(items)


    def unsubsribe_emails(self, items):

        import pdb;pdb.set_trace()


        for item in items:

            time.sleep(0.1)

            item = item.split(',')[2]

            try:

                term = item.strip().lower()

                print(term)

                utils.unsubscribe(term, "Bounce")

            except:

                traceback.print_exc(file=sys.stdout)


More examples see in intern commands of mailing application in CRM. 
 
For example: 

mailing/management/commands/importsendgridspam.py
mailing/management/commands/unsubscribe_suppression.py
mailing/management/commands/importsendgridbounce.py
mailing/management/commands/collect_suppression.py
mailing/management/commands/copyemailsfromstatus.py

If you start to unsubscribe automatically then look very carefully at status and message because some messages mean nothing but blocked and that is per domain or even message level and bounce mailbox is still OK but sender is not OK and he is blocked and this bounce actually belongs in category blocked en not bounce.. Bounce is when mailbox is closed or overquite. So look good, such message from from skynet.be provider   is about your blacklisted IP, Domain or message by skynet provider..
 

550 #5.1.0 Address rejected.

When you filter this messages and unsubscribe it from list, unsubscribe it not on global level but only per domain and message. 
 
To read messages in mail log from big logfile about 50 GB.. For example, if you have a configuration that contains all baskets or even years, it could be that it is not readable with VIM.  Then you can cut and read the last 100K from this file.
 
du -h /var/log/mail.log
50G    /var/log/mail.log
 
tail -n 100000  /var/log/mail.log > /tmp/log
 
 

Unsubscribe via Firebase form


We have several ways to accomplish this task. We can use the REST API and lib which was made 5 years ago.

https://firebase.google.com/docs/database/rest/start 


https://github.com/thisbejim/Pyrebase 


Or even older like


https://github.com/ozgur/python-firebase 



So it's not recommended that we write new applications and use Realtime Database for my purposes, I've found Firestore Database works much better, and I'm trying to limit or eliminate this old use altogether in the future.

Next option is also an export of data. If for example you only need to perform this unsubscribe operation 1 time per month then you can export data and unsubscribe.



But we will automate immediately in a normal way because I think this application will be used for a long time and proper automation is needed from the start.



Pyrebase was written for python 3 and will not work correctly with python 2.


The first step is the configuration.


import pyrebase


config = {

  "apiKey": "AIzaSyAfGNahILALZrp25B_-l97-dfdfdf",

  "authDomain": "vacatures.firebaseapp.com",

  "databaseURL": "https://vacatures.firebaseio.com",

  "storageBucket": "vacatures.appspot.com"

  "serviceAccount": "path/to/serviceAccountCredentials.json"

}


firebase = pyrebase.initialize_app(config)


In other words you can find all config information in Web or App configs.

https://firebase.google.com/docs/projects/api-keys 


var firebaseConfig = {



apiKey: "AIzaSyAfGNahILALZrp25B_-l97-dfdf",


authDomain: "vacature.firebaseapp.com",


databaseURL: "https://vacature.firebaseio.com",


projectId: "vacature",


storageBucket: "vacatures.appspot.com",


messagingSenderId: "4455454",


appId: "1:479857347200:web:343434",


measurementId: "G-DFDF4545"


};








But if we need full Admin access we can still
"serviceAccount": "path/to/serviceAccountCredentials.json"

In this way we can read data that is restricted with rights for other users.

With this wrapper you can address a number of services for example auth service.

https://github.com/thisbejim/Pyrebase#authentication 



For example, it is possible to create or delete a user automatically.

So if your unsubscribe logic includes removing users then you can do it automatically via this wrapper reference.
At the moment we will only remove people who have unsubscribed from mailing and leave users unchanged.  
 
Example
 
class Command(BaseCommand):
    """
    Unsubscriebe emails from firebase database
    """
    def handle(self, *args, **options):
        """
        """
        firebase = pyrebase.initialize_app(config)
        db = firebase.database()
        contacts = db.child("unsubscribe").get()
        #import pdb;pdb.set_trace()
        for contact in contacts.each():
            self.unsubsribe_emails(contact.val())

    def unsubsribe_emails(self, item):
        #import pdb;pdb.set_trace()
        try:
            term = item['email'].strip().lower()
            print(term)
            utils.unsubscribe(term, "Unsubscribe form")
        except:
            traceback.print_exc(file=sys.stdout)
 
 



Comments

Popular posts from this blog

Pgpool PgBouncer Postgresql streaming replication, load balancing and administration

The term scalability refers to the ability of a software system to grow as the business that uses it grows. PostgreSQL provides some features to help you build scalable solutions, but strictly speaking, PostgreSQL itself is not scalable. It can effectively use the following resources from one computer. Now, we will show you some configurations that may be useful for your use case. However, this can be problematic when distributing the database solution to multiple computers, because the standard PostgreSQL server can only run on a single computer. In this article, we will study different extension schemes and their implementation in PostgreSQL. Replication can be used in many expansion scenarios. Its main purpose is to create and maintain a backup database when the system fails. This is especially true for physical replication. However, replication can also be used to improve the performance of PostgreSQL-based solutions. Sometimes third-party tools can be used to implement complex exp

Tekstverwerking python Text processing python SpaCy, TensorFlow, NLTK, Allen-NLP, Stanford-NLP

 Dit post maakt gebruik van spaCy, een populaire Python-bibliotheek die de taalgegevens en algoritmen bevat die je nodig hebt om teksten in natuurlijke taal te verwerken. Zoals u in dit post zult leren, is spaCy gemakkelijk te gebruiken omdat het containerobjecten biedt die elementen van natuurlijke taalteksten vertegenwoordigen, zoals zinnen en woorden. Deze objecten hebben op hun beurt attributen die taalkenmerken vertegenwoordigen, zoals delen van spraak. Op het moment van schrijven bood spaCy voorgetrainde modellen aan voor Engels, Duits, Grieks, Spaans, Frans, Italiaans, Litouws, Noors BokmÃ¥l, Nederlands, Portugees en meerdere talen gecombineerd. Bovendien biedt spaCy ingebouwde visualizers die u programmatisch kunt aanroepen om een grafische weergave van de syntactische structuur van een zin of benoemde entiteiten in een document te genereren.   De spaCy-bibliotheek ondersteunt ook native geavanceerde NLP-functies die andere populaire NLP-bibliotheken voor Python niet hebben. Spa

Elasticsearch install and configuration on Ubuntu

If you plan to use elastic integrated with Django then it would be better to use an old version of Elastic or replace Haystack with Django-Elasticsearch-DSL (Not tested) See old version install at bottom of this post.  New version Elasticsearch ( attention no haystack at this time )  The Elasticsearch components are not available in Ubuntu’s default package repositories. They can, however, be installed with APT after adding Elastic’s package source list. curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation apt-get install gnupg curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). OK echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list apt update apt