run “pip install” behind proxy

Even when you trust the MITM proxy devices’ self-signed certificates, pip can still fail.

# cat ~/.config/pip/pip.conf
[global]
cert = /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt

Even with telling the server names to be explicitly trusted.

# pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org google-auth
Collecting google-auth
  HTTP error 403 while getting https://files.pythonhosted.org/packages/c5/9b/ed0516cc1f7609fb0217e3057ff4f0f9f3e3ce79a369c6af4a6c5ca25664/google_auth-1.6.3-py2.py3-none-any.whl#sha256=20705f6803fd2c4d1cc2dcb0df09d4dfcb9a7d51fd59e94a3a28231fd93119ed (from https://pypi.org/simple/google-auth/)
  Could not install requirement google-auth from https://files.pythonhosted.org/packages/c5/9b/ed0516cc1f7609fb0217e3057ff4f0f9f3e3ce79a369c6af4a6c5ca25664/google_auth-1.6.3-py2.py3-none-any.whl#sha256=20705f6803fd2c4d1cc2dcb0df09d4dfcb9a7d51fd59e94a3a28231fd93119ed because of error 403 Client Error: Forbidden for url: https://files.pythonhosted.org/packages/c5/9b/ed0516cc1f7609fb0217e3057ff4f0f9f3e3ce79a369c6af4a6c5ca25664/google_auth-1.6.3-py2.py3-none-any.whl
Could not install requirement google-auth from https://files.pythonhosted.org/packages/c5/9b/ed0516cc1f7609fb0217e3057ff4f0f9f3e3ce79a369c6af4a6c5ca25664/google_auth-1.6.3-py2.py3-none-any.whl#sha256=20705f6803fd2c4d1cc2dcb0df09d4dfcb9a7d51fd59e94a3a28231fd93119ed because of HTTP error 403 Client Error: Forbidden for url: https://files.pythonhosted.org/packages/c5/9b/ed0516cc1f7609fb0217e3057ff4f0f9f3e3ce79a369c6af4a6c5ca25664/google_auth-1.6.3-py2.py3-none-any.whl for URL https://files.pythonhosted.org/packages/c5/9b/ed0516cc1f7609fb0217e3057ff4f0f9f3e3ce79a369c6af4a6c5ca25664/google_auth-1.6.3-py2.py3-none-any.whl#sha256=20705f6803fd2c4d1cc2dcb0df09d4dfcb9a7d51fd59e94a3a28231fd93119ed (from https://pypi.org/simple/google-auth/)

Solution

I had to use a squid proxy to make it happen.

# export https_proxy=http://10.123.456.5:3128
# pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org google-auth
Collecting google-auth
  Downloading https://files.pythonhosted.org/packages/c5/9b/ed0516cc1f7609fb0217e3057ff4f0f9f3e3ce79a369c6af4a6c5ca25664/google_auth-1.6.3-py2.py3-none-any.whl (73kB)
    100% |████████████████████████████████| 81kB 3.0MB/s
Collecting pyasn1-modules>=0.2.1 (from google-auth)
  Downloading https://files.pythonhosted.org/packages/da/98/8ddd9fa4d84065926832bcf2255a2b69f1d03330aa4d1c49cc7317ac888e/pyasn1_modules-0.2.4-py2.py3-none-any.whl (66kB)
    100% |████████████████████████████████| 71kB 5.4MB/s
Collecting cachetools>=2.0.0 (from google-auth)
  Downloading https://files.pythonhosted.org/packages/39/2b/d87fc2369242bd743883232c463f28205902b8579cb68dcf5b11eee1652f/cachetools-3.1.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): six>=1.9.0 in /usr/lib/python2.7/site-packages (from google-auth)
Collecting rsa>=3.1.4 (from google-auth)
  Downloading https://files.pythonhosted.org/packages/02/e5/38518af393f7c214357079ce67a317307936896e961e35450b70fad2a9cf/rsa-4.0-py2.py3-none-any.whl
Collecting pyasn1<0.5.0,>=0.4.1 (from pyasn1-modules>=0.2.1->google-auth)
  Downloading https://files.pythonhosted.org/packages/7b/7c/c9386b82a25115cccf1903441bba3cbadcfae7b678a20167347fa8ded34c/pyasn1-0.4.5-py2.py3-none-any.whl (73kB)
    100% |████████████████████████████████| 81kB 37.5MB/s
Installing collected packages: pyasn1, pyasn1-modules, cachetools, rsa, google-auth
  Found existing installation: pyasn1 0.1.9
    Uninstalling pyasn1-0.1.9:
      Successfully uninstalled pyasn1-0.1.9
  Found existing installation: pyasn1-modules 0.0.8
    Uninstalling pyasn1-modules-0.0.8:
      Successfully uninstalled pyasn1-modules-0.0.8
Successfully installed cachetools-3.1.0 google-auth-1.6.3 pyasn1-0.4.5 pyasn1-modules-0.2.4 rsa-4.0
Advertisements

Ansible find first accessible proxy and use it

If you need to find the first available http proxy and use it for a process, you can use a python snippet to discover it and use it.

https://gitlab.com/bgstack15/former-gists/tree/master/get_first_open_port.py

vars:
  http_proxies:
  - 192.168.1.5:3128
  - proxy5.internal.example.com:3128

tasks:

  - name: learn which proxy to use
    script: get_first_open_port.py {{ http_proxies | join( " " ) }}
    changed_when: false
    register: open_ports

  - set_fact:
      http_proxy: "{{ open_ports.stdout_lines[0] }}"
    when:
    - 'open_ports.stdout | length > 0'
    failed_when:
    - 'open_ports.stdout | length = 0'

  - name: use http_proxy environment variable
    script: script_needing_internet.sh -i {{ inputvar }}
    environment:
      http_proxy: "http://{{ http_proxy | default(omit) }}"

The sole output is the first hostname and port available.

#!/usr/bin/python2
# Filename: get_first_open_port.py
# Location: /etc/ansible/roles/install_sccm/files/
# Author: bgstack15
# Startdate: 2018-10-02 10:13
# Title: Script that Gets the First Open Port
# Purpose: Return to standard output the first valid host and port to use as a http proxy
# Project: projects derived from ansible role certreq
# History:
# Usage:
#    in ansible
# Reference:
#    https://stackoverflow.com/questions/19196105/python-how-to-check-if-a-network-port-is-open-on-linux
#    string split https://stackoverflow.com/questions/6670290/split-string-into-different-variables-instead-of-array-in-python
# Improve:
# Documentation:

import socket, sys
from contextlib import closing

GFOP_VERSION="2018-10-02a"

def check_socket(host, port):
   with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as sock:
      sock.settimeout(2)
      if sock.connect_ex((host,port)) == 0:
         print host + ":" + str(port)
         return True
   return False


x = 0
for myarg in sys.argv:

   # show version
   if myarg == "--version" or myarg == "-V":
      print(sys.argv[0]+" "+GFOP_VERSION)
      sys.exit(0)

   x = x + 1
   # skip the script $0 name itself
   if x > 1:
      # split on the colon
      host, port = myarg.split(":",2)
      # short-circuit upon first successful one
      if check_socket(host,int(port)):
         sys.exit(0)

Set word-read permissions on python libs

In an environment where the default umask is 0077 or similar, the pip utility for installing python libraries can set up new files that cannot be read by all users.

I dropped this script into my ~/bin dir so I can enforce other-read permissions easily on all the python libs. I wrote this script after hours and hours of troubleshooting python libs just to find out it’s an old-school permissions issue.


#!/bin/sh
# File: /usr/local/bin/set-readable-python-libs
worldreadpythonlibs_version="2019-05-31a"
for word in /usr/{local/,}lib{,64}/python* ;
do
find ${word} ! -perm -o+rX -exec chmod g+rX,o+rX {} + 2>/dev/null
done

 

Python get Linux-compatible password hash

This snippet gets you a sha-512 ($6) password hash suitable for putting in /etc/shadow.

# Reference: https://www.shellhacks.com/linux-generate-password-hash/
# python 2
import crypt, getpass, sys;
if len(sys.argv) >= 2: 
 thisraw=str(sys.argv[1]);
else:
 thisraw=getpass.getpass(prompt='New password: ')
 #sys.exit(1)
print(crypt.crypt(thisraw,crypt.mksalt(crypt.METHOD_SHA512)))

Debug the values passed to a function in python

Tested on python 2.

import inspect

def caller_args():
   frame = inspect.currentframe()
   outer_frames = inspect.getouterframes(frame)
   caller_frame = outer_frames[1][0]
   return inspect.getargvalues(caller_frame)

def updateval(infile,regex,result,verbose=False,apply=False,debug=0,stanza="",stanzaregex="",atbeginning=False):
   print caller_args()

It’s that simple!

Reference

Weblinks

  1. https://stackoverflow.com/questions/29935276/inspect-getargvalues-throws-exception-attributeerror-tuple-object-has-no-a/29935277#29935277

Pretty print json in python

For python2

I wanted to show what variables are in use in a function, and I wanted to see it in a nicer format than a really long, single line.

import inspect, json
def function():
print json.dumps(locals(),indent=3,separators=(',',': '))

Bonus

To view what parameters were passed in to a function, add these.

def caller_args():
   frame = inspect.currentframe()
   outer_frames = inspect.getouterframes(frame)
   caller_frame = outer_frames[1][0]
   return inspect.getargvalues(caller_frame)

def function():
print caller_args()

References

  1. https://stackoverflow.com/questions/29935276/inspect-getargvalues-throws-exception-attributeerror-tuple-object-has-no-a#29935277
  2. compact encoding https://docs.python.org/2/library/json.html

					

python readlink function

tl;dr

def readlinkf(_inpath):
   if os.path.islink(infile):
      return os.path.join(os.path.dirname(_inpath),os.readlink(_inpath))
   else:
      return _inpath

infile=readlinkf(infile)

Explanation

In GNU, there’s a fantastic utility named readlink. The main way I use it is to get the full and true path of a file or symlink.

For example:

$ ls foo .config/qux
.config/qux  foo
$ readlink -f foo
/home/bgstack15/.config/qux

In python, I wanted to achieve the same thing. it is possible with the os.path.islink and os.readlink functions. Here is my quick function for a readlink -f, since I didn’t find a specific implementation that outputs the full path.

Reference

Weblinks

  1. https://stackoverflow.com/questions/11068419/check-if-file-is-symlink-in-python

Online reference

  1. https://docs.python.org/3/library/os.path.html#os.path.islink
  2. https://docs.python.org/3.5/library/os.html#os.readlink

Search expressions

  1. python detect if file is symlink
  2. python readlink

Python script to update value in file

Overview

I do initial system configuration, for images and so on. All the time I have to update arbitrary values in config files. It’s all very repetitive. Sometimes though, instead of being able to use a whole templated config file, I need to modify the one that is in place.

Here today for you is my second-generation solution (the first was a crude shell script). It’s also a part of my bgscripts package available on gitlab.

#!/usr/bin/env python3
# File: /usr/bgscripts/updateval.py
# Author: bgstack15@gmail.com
# Startdate: 2016-10-11 15:59
# Title: Python Script that Updates/Adds Value
# Purpose: Allows idempotent and programmatic modifications to config files
# Package: bgscripts
# History:
#    2016-07-27 wrote shell script 
#    2016-09-14 added the shell script version to bgscripts package
#    2016-10-12 added flags
# Usage:
#   updateval.py /etc/rc.conf "^ntpd_enable=.*" 'ntpd_enable="YES"' --apply
# Reference:
#    /usr/bgscripts/updateval.sh
#    re.sub from http://stackoverflow.com/questions/5658369/how-to-input-a-regex-in-string-replace-in-python/5658377#5658377
#    shutil.copy2 http://pythoncentral.io/how-to-copy-a-file-in-python-with-shutil/
#    keepalive (python script) from keepalive-1.0-5
# Improve:
#    idea: use argparse "nargs" optional input file to use stdin piping/redirection!

import re, shutil, os, argparse
updatevalversion="2016-10-12c"

# Parse parameters
parser = argparse.ArgumentParser(description="Idempotent value updater for a file",epilog="If searchstring is not found, deststring will be appended to infile")
parser.add_argument("-v","--verbose",help="displays output",      action="store_true",default=False)
parser.add_argument("-a","--apply",  help="perform substitution", action="store_true",default=False)
parser.add_argument("-L","--all",    help="replace all instances",action="store_true",default=False)
parser.add_argument("infile", help="file to use")
parser.add_argument("searchstring", help="regex string to search")
parser.add_argument("deststring", help="literal string that should be there")
parser.add_argument("-V","--version", action="version", version="%(prog)s " + updatevalversion)
args = parser.parse_args()

# Configure variables after parameters
verbose = args.verbose
doapply = args.apply
doall = args.all
infile = args.infile
searchstring = args.searchstring
destinationstring = args.deststring

wasfixed = False
outfile = infile + ".updateval-new"

# Make file if it does not exist
if not os.path.isfile(infile): open(infile, "w").close()

# If line exists, replace it
shutil.copy2(infile,outfile) # initialize duplicate file with same perms
with open(outfile, "w") as outf:
   for line in open(infile, "r"):
      p = re.compile(searchstring)
      if p.match(line) and ( not wasfixed or doall ):
         outline = re.sub(searchstring,destinationstring, line).rstrip('\n')
         wasfixed = True
      else:
         outline = line.rstrip('\n')
      if verbose: print(outline)
      #print(outline,file = outf)
      outf.write(outline + '\n')

# Append line if it has not been fixed yet
if not wasfixed:
   with open(outfile, "a") as outf:
      if verbose: print(destinationstring)
      outf.write(destinationstring + '\n')

# replace old file with new file
if doapply:
   shutil.move(outfile,infile)

# Clean up outfile just in case
try:
   os.remove(outfile)
except Exception as e:
   pass

I’ll explain some of the code, just in case the comments don’t do a good job.
Lines 37-42 Whenever i use argparse, I always pull the information out of the argparse object into local variables. If I rename the object or parameter names, I don’t want to have to go update my code everywhere any of those variables are mentioned.

Line 45: This is an arbitrarily named, temporary export file. I wasn’t sure I could do an open() with read-write, so I just duplicate the file and then replace the original (if doapply=true).

Line 48: Basically, touch the file to make sure it exists.

Lines 51-62: This right here (specifically lines 55-56) is the reason I rewrote this in python. This was much harder in shell. Observe around line 61 how the print command was commented out in favor of the outf.write() object method call. The reason I had to give up the print command here was because when I was packaging my bgscripts set for rpm, it was compiling the python with the python2 interpreter, and it was easier to adjust my python code than which python interpreter rpmbuild uses. So a google search got me the output command. I don’t care how I get the task done; I want the task done.

My requirements included appending the destinationstring if the searchstring did not appear in the code. So lines 65-68 handle that.

Lines 70-78 are needed only because of the writing to the other file. If I could write to the original file, I wouldn’t need to do file manipulation.