run “pip install” behind proxy

Even when you trust the MITM proxy devices’ self-signed certificates, pip can still fail.

# cat ~/.config/pip/pip.conf
cert = /etc/pki/ca-trust/extracted/openssl/

Even with telling the server names to be explicitly trusted.

# pip install --trusted-host --trusted-host google-auth
Collecting google-auth
  HTTP error 403 while getting (from
  Could not install requirement google-auth from because of error 403 Client Error: Forbidden for url:
Could not install requirement google-auth from because of HTTP error 403 Client Error: Forbidden for url: for URL (from


I had to use a squid proxy to make it happen.

# export https_proxy=http://10.123.456.5:3128
# pip install --trusted-host --trusted-host google-auth
Collecting google-auth
  Downloading (73kB)
    100% |████████████████████████████████| 81kB 3.0MB/s
Collecting pyasn1-modules>=0.2.1 (from google-auth)
  Downloading (66kB)
    100% |████████████████████████████████| 71kB 5.4MB/s
Collecting cachetools>=2.0.0 (from google-auth)
Requirement already satisfied (use --upgrade to upgrade): six>=1.9.0 in /usr/lib/python2.7/site-packages (from google-auth)
Collecting rsa>=3.1.4 (from google-auth)
Collecting pyasn1<0.5.0,>=0.4.1 (from pyasn1-modules>=0.2.1->google-auth)
  Downloading (73kB)
    100% |████████████████████████████████| 81kB 37.5MB/s
Installing collected packages: pyasn1, pyasn1-modules, cachetools, rsa, google-auth
  Found existing installation: pyasn1 0.1.9
    Uninstalling pyasn1-0.1.9:
      Successfully uninstalled pyasn1-0.1.9
  Found existing installation: pyasn1-modules 0.0.8
    Uninstalling pyasn1-modules-0.0.8:
      Successfully uninstalled pyasn1-modules-0.0.8
Successfully installed cachetools-3.1.0 google-auth-1.6.3 pyasn1-0.4.5 pyasn1-modules-0.2.4 rsa-4.0

Ansible find first accessible proxy and use it

If you need to find the first available http proxy and use it for a process, you can use a python snippet to discover it and use it.



  - name: learn which proxy to use
    script: {{ http_proxies | join( " " ) }}
    changed_when: false
    register: open_ports

  - set_fact:
      http_proxy: "{{ open_ports.stdout_lines[0] }}"
    - 'open_ports.stdout | length > 0'
    - 'open_ports.stdout | length = 0'

  - name: use http_proxy environment variable
    script: -i {{ inputvar }}
      http_proxy: "http://{{ http_proxy | default(omit) }}"

The sole output is the first hostname and port available.

# Filename:
# Location: /etc/ansible/roles/install_sccm/files/
# Author: bgstack15
# Startdate: 2018-10-02 10:13
# Title: Script that Gets the First Open Port
# Purpose: Return to standard output the first valid host and port to use as a http proxy
# Project: projects derived from ansible role certreq
# History:
# Usage:
#    in ansible
# Reference:
#    string split
# Improve:
# Documentation:

import socket, sys
from contextlib import closing


def check_socket(host, port):
   with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as sock:
      if sock.connect_ex((host,port)) == 0:
         print host + ":" + str(port)
         return True
   return False

x = 0
for myarg in sys.argv:

   # show version
   if myarg == "--version" or myarg == "-V":
      print(sys.argv[0]+" "+GFOP_VERSION)

   x = x + 1
   # skip the script $0 name itself
   if x > 1:
      # split on the colon
      host, port = myarg.split(":",2)
      # short-circuit upon first successful one
      if check_socket(host,int(port)):

Set word-read permissions on python libs

In an environment where the default umask is 0077 or similar, the pip utility for installing python libraries can set up new files that cannot be read by all users.

I dropped this script into my ~/bin dir so I can enforce other-read permissions easily on all the python libs. I wrote this script after hours and hours of troubleshooting python libs just to find out it’s an old-school permissions issue.

# File: /usr/local/bin/set-readable-python-libs
for word in /usr/{local/,}lib{,64}/python* ;
find ${word} ! -perm -o+rX -exec chmod g+rX,o+rX {} + 2>/dev/null


Python get Linux-compatible password hash

This snippet gets you a sha-512 ($6) password hash suitable for putting in /etc/shadow.

# Reference:
# python 2
import crypt, getpass, sys;
if len(sys.argv) >= 2: 
 thisraw=getpass.getpass(prompt='New password: ')

Debug the values passed to a function in python

Tested on python 2.

import inspect

def caller_args():
   frame = inspect.currentframe()
   outer_frames = inspect.getouterframes(frame)
   caller_frame = outer_frames[1][0]
   return inspect.getargvalues(caller_frame)

def updateval(infile,regex,result,verbose=False,apply=False,debug=0,stanza="",stanzaregex="",atbeginning=False):
   print caller_args()

It’s that simple!




Pretty print json in python

For python2

I wanted to show what variables are in use in a function, and I wanted to see it in a nicer format than a really long, single line.

import inspect, json
def function():
print json.dumps(locals(),indent=3,separators=(',',': '))


To view what parameters were passed in to a function, add these.

def caller_args():
   frame = inspect.currentframe()
   outer_frames = inspect.getouterframes(frame)
   caller_frame = outer_frames[1][0]
   return inspect.getargvalues(caller_frame)

def function():
print caller_args()


  2. compact encoding


python readlink function


def readlinkf(_inpath):
   if os.path.islink(infile):
      return os.path.join(os.path.dirname(_inpath),os.readlink(_inpath))
      return _inpath



In GNU, there’s a fantastic utility named readlink. The main way I use it is to get the full and true path of a file or symlink.

For example:

$ ls foo .config/qux
.config/qux  foo
$ readlink -f foo

In python, I wanted to achieve the same thing. it is possible with the os.path.islink and os.readlink functions. Here is my quick function for a readlink -f, since I didn’t find a specific implementation that outputs the full path.




Online reference


Search expressions

  1. python detect if file is symlink
  2. python readlink

Python script to update value in file


I do initial system configuration, for images and so on. All the time I have to update arbitrary values in config files. It’s all very repetitive. Sometimes though, instead of being able to use a whole templated config file, I need to modify the one that is in place.

Here today for you is my second-generation solution (the first was a crude shell script). It’s also a part of my bgscripts package available on gitlab.

#!/usr/bin/env python3
# File: /usr/bgscripts/
# Author:
# Startdate: 2016-10-11 15:59
# Title: Python Script that Updates/Adds Value
# Purpose: Allows idempotent and programmatic modifications to config files
# Package: bgscripts
# History:
#    2016-07-27 wrote shell script 
#    2016-09-14 added the shell script version to bgscripts package
#    2016-10-12 added flags
# Usage:
# /etc/rc.conf "^ntpd_enable=.*" 'ntpd_enable="YES"' --apply
# Reference:
#    /usr/bgscripts/
#    re.sub from
#    shutil.copy2
#    keepalive (python script) from keepalive-1.0-5
# Improve:
#    idea: use argparse "nargs" optional input file to use stdin piping/redirection!

import re, shutil, os, argparse

# Parse parameters
parser = argparse.ArgumentParser(description="Idempotent value updater for a file",epilog="If searchstring is not found, deststring will be appended to infile")
parser.add_argument("-v","--verbose",help="displays output",      action="store_true",default=False)
parser.add_argument("-a","--apply",  help="perform substitution", action="store_true",default=False)
parser.add_argument("-L","--all",    help="replace all instances",action="store_true",default=False)
parser.add_argument("infile", help="file to use")
parser.add_argument("searchstring", help="regex string to search")
parser.add_argument("deststring", help="literal string that should be there")
parser.add_argument("-V","--version", action="version", version="%(prog)s " + updatevalversion)
args = parser.parse_args()

# Configure variables after parameters
verbose = args.verbose
doapply = args.apply
doall = args.all
infile = args.infile
searchstring = args.searchstring
destinationstring = args.deststring

wasfixed = False
outfile = infile + ".updateval-new"

# Make file if it does not exist
if not os.path.isfile(infile): open(infile, "w").close()

# If line exists, replace it
shutil.copy2(infile,outfile) # initialize duplicate file with same perms
with open(outfile, "w") as outf:
   for line in open(infile, "r"):
      p = re.compile(searchstring)
      if p.match(line) and ( not wasfixed or doall ):
         outline = re.sub(searchstring,destinationstring, line).rstrip('\n')
         wasfixed = True
         outline = line.rstrip('\n')
      if verbose: print(outline)
      #print(outline,file = outf)
      outf.write(outline + '\n')

# Append line if it has not been fixed yet
if not wasfixed:
   with open(outfile, "a") as outf:
      if verbose: print(destinationstring)
      outf.write(destinationstring + '\n')

# replace old file with new file
if doapply:

# Clean up outfile just in case
except Exception as e:

I’ll explain some of the code, just in case the comments don’t do a good job.
Lines 37-42 Whenever i use argparse, I always pull the information out of the argparse object into local variables. If I rename the object or parameter names, I don’t want to have to go update my code everywhere any of those variables are mentioned.

Line 45: This is an arbitrarily named, temporary export file. I wasn’t sure I could do an open() with read-write, so I just duplicate the file and then replace the original (if doapply=true).

Line 48: Basically, touch the file to make sure it exists.

Lines 51-62: This right here (specifically lines 55-56) is the reason I rewrote this in python. This was much harder in shell. Observe around line 61 how the print command was commented out in favor of the outf.write() object method call. The reason I had to give up the print command here was because when I was packaging my bgscripts set for rpm, it was compiling the python with the python2 interpreter, and it was easier to adjust my python code than which python interpreter rpmbuild uses. So a google search got me the output command. I don’t care how I get the task done; I want the task done.

My requirements included appending the destinationstring if the searchstring did not appear in the code. So lines 65-68 handle that.

Lines 70-78 are needed only because of the writing to the other file. If I could write to the original file, I wouldn’t need to do file manipulation.