Skip to main content

Posts

Showing posts from May, 2021

Perform commands with Python over ssh

I need to run some commands on a remote computer. Manually, I will log in with ssh and run the command. How would you automate this in Python? You  need for example to log in to the remote computer with a (known) password, so I can't just use cmd = ssh user @ remotehost, and then use the paramiko module for authentication and automation on the server side.  Very simple example: ssh = paramiko.SSHClient() ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) ssh.connect(server, username=username, password=password) ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command(cmd_to_execute) If you are using ssh keys, do: k = paramiko.RSAKey.from_private_key_file(keyfilename) k = paramiko.DSSKey.from_private_key_file(keyfilename) ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) ssh.connect(hostname=host, username=user, pkey=k) Or  commands.getstatusoutput("ssh machine 1 'your script'") www.paramiko.org Paramiko is a Python (2.7, 3.4+) implementation of the SSHv

Can't Use apt i.e. /boot is 100% full

  Note: Only in cases where cleanup cannot be done with apt due to 100% full/boot Get a list of kernel images and determine which will be removed. This command will display the installed kernels, except for the currently running kernel dpkg --list 'linux-image*'|awk '{ if ($1=="ii") print $2}'|grep -v `uname -r` linux-image-4.15.0-46-generic linux-image-4.15.0-50-generic linux-image-4.15.0-51-generic linux-image-4.15.0-52-generic linux-image-4.15.0-54-generic Linux-image-4.15.0-55-generic First, get the list of currently running Linux kernels and run: v="$(uname -r | awk -F '-virtual' '{ print $1}')" 4.15.0-47-generic Use brace expansion to keep your boot clear, you can write a command to delete all files in /boot that are not related to the kernel. Remember to exclude the current and two latest kernel images. In the example above, it is rm -rf /boot/*-4.15.0-{46,50,51,52,54,55}-* Or move these files in another directory. mv /boot/*-4

Sendgrid simple integration on localhost python TinyDB sendgridclient

  Configure mail to send email from a local host without any platform. Sometimes, you need a quick application to test an idea, but don't want to invest in a separate server with DKIM, SPF and other tools installed, but only need to test a business case for 2-3 months. The perfect tool for sendgrid that you can control via your laptop. There is no need to pay for the server and django, and there is no need to use a completely naive integration without a code solution. This API integration is at least 3 times cheaper than any other solution on the market, and it can be used for professional purposes. Note this integration only works for small lists of about 200k-300k maximum. Otherwise database is too slow and updates are slower than postgres or other databases.   Register your account on the Sendgrid website. Then, at this point in the API configuration wizard are out of date docs. Ignore it!  We use the code found on github. See the example below.   This is an old version in docs

Install WebDriver Chrome or Firefox for Selenium for automated testing web application

Installation for automated testing  sudo apt-get update apt-get install -y unzip openjdk-8-jre-headless xvfb libxi6 libgconf-2-4   apt-get install firefox Pip install selenium Sometimes you can get some exceptions  Selenium “Unable to find a matching set of capabilities” despite driver being in /usr/local/bin Or selenium.common.exceptions.WebDriverException: Message: Unable to find a matching set of capabilities Updating Firefox and Selenium solved it for me. I don't pretend to have an explanation for the root cause however. Updated Firefox 48 → 53 Updated to Selenium 3.4.1 https://pypi.org/project/selenium/ https://github.com/mozilla/geckodriver/releases   Expected browser binary location, but unable to find binary in default location, no 'moz:firefoxOptions.binary' capability provided, and no binary flag set on the command line Firefox browser has installed?  apt-get install firefox  Script example for chrome WebDriver automated installation on Ubuntu  #!/usr/bin/env bash

Problem with postgres Index and database size on dedicated server

  Often it happens in the morning that you get up and then you see that your site is about 2-3 hours offline. If you see some message like.. OperationalError at file_name ERROR: all backend nodes are down, pgpool requires at least one valid node HINT: repair the backend nodes and restart pgpool You have to check df -h and see of space on hardrive is full. Or you can use Ncdu This is a very good command to find hard drive problems. First check that you are /var/logs. Sometimes this is caused by you forgetting the debugger in verbose mode. And you are running a hard drive full of logs. But in this case, we will check for more difficulties. I will write some design decisions to delete some unused indexes, but these indexes are automatically for very large database tables with about 25 million records and more. SELECT pg_size_pretty( pg_database_size(“db_name”) ); 894GB Show tables \dt Shows supper full output sorted by table and index size WITH RECURSIVE pg_inherit(inhrelid, inhparent) A

Elasticsarch Python combinated with Haystack delete objects by id

    I cannot delete objects from the index in real time by object ID. In this case, the Haystack django application cannot work properly. I need a custom script to move records out of the current table to archive and delete objects from the elastic  in real time.  Old version of the application, python 2 and old Elastic make it more difficult. See Example in Vindazo  vim job/management/commands/archiveringbykeyword.py  When you move a record to the archive and then delete them from the elastic index, the errors in haystack can happen. This fix will provide new opportunities for adding and deleting jobs after indexing and cleaning.  In old versions of Haystack it doesn't work as expected. And if It is not working properly, you need to run update_index or rebuild_index command every day. Example for low level python Elastic object removing without Haystack from elasticsearch import Elasticsearch es = Elasticsearch() res = es.get(index="jobs", id=result.id) es.delete(index=&