New user in freeipa has plain bash shell instead of reading .bashrc

So you have a new user in freeipa, and he can successfully log in to a freeipa client. And you know for certain you executed ipa-client-install with the –mkhomedir option. But when you open a terminal as the new user, it shows you the boring bash prompt ‘bash-4.1$’ or whatever version.

You checked the /etc/skel, and it has a valid .bashrc file, and when you dot source your own ~/.bashrc, it then loads the prompt you expect.

Here’s your issue: do a getent passwd username. Look at the login shell of the user. It’s going to be the default /bin/sh. Just change it in ipa to be /bin/bash! An sss_cache -E command was not enough; you have to log out and then back in to have it take effect. It’s probably because the terminal emulator is being called from a process that was started before the account was changed.

Linux use tee with color

Output in color on the console

The console normally can display color. The most obvious example is when you do a standard ls command. By default on most systems, ls includes a –color=auto flag that changes filenames of certain types to different colors.

Some applications display colorized output like systemctl status nfsd.service.

Using tee

Tee is a nifty cli utility that duplicates the input to both the standard out as well as to a file. You can use it like so:
ls -l --color=always | tee ~/ls.out
Some tools will disable colorized output if it is going to a pipe, but ls can be forced to provide color with the –color=always flag. For applications that don’t have such an option, you can usually prepend the command unbuffer.
unbuffer ansible-playbook playbook.yml | tee ansible.log
When you examine the file, however, you will observe that the console color commands were saved. That’s perfectly fine if you cat the file, or less -R, but normally you want a text file to just have the text. Sysadmins are quite used to dealing with logs that are all the standard color of the console.

The solution: ctee!

Using some fantastic resources on the Internet which I list at the bottom of this post, I assembled a tool that will tee the input, and send color to the stdout (usually the console) and send regular text to the file. I call my creation ctee.

Code walkthrough

So this script was generated using newscript which copies the template from the same bgscripts package. The template sources framework, which is my big flagship library of functions for shell scripts.

Lines 39-54 are the parseFlag function, which parses the command line passed parameters. I implemented this before I ever learned about Python’s argparse library.
The only flag worthy to note is –append. All it does is set the variable to 1.

The meat of this script starts at line 85 and goes to the end, which is shown below.

# this whole operation calculates where the stdout of this whole pipe goes
ttyresolved=0
_count=0
_pid=$$
_fd1="$( readlink -f /proc/${_pid}/fd/1 )"
while test ${ttyresolved} -lt 10;
do
ttyresolved=$(( ttyresolved + 1 ))
echo "before ${ttyresolved}, _pid=${_pid}, _fd1=${_fd1}" > ${devtty}
case "${_fd1}" in
*pipe:* )
newpid=$( find /proc -type l -name '0' 2>/dev/null | xargs ls -l 2>/dev/null | grep -F "$( basename ${_fd1} )" | grep -viE "\/${_pid}\/"; )
newpid=$( echo "${newpid}" | head -n1 | grep -oiE "\/proc\/[0-9]*\/" | grep -o "[0-9]*" )
_pid=${newpid}
_fd1="$( readlink -f /proc/${_pid}/fd/1 )"
;;
*dev*|*/* )
thisttyout="${_fd1}"
ttyresolved=10
;;
esac
done
echo "thisttyout ${thisttyout}" > ${devtty}
# MAIN LOOP
case "${append}" in
1)
tee ${thisttyout} | sed -u -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g" >> "${outfile1}"
;;
*) tee ${thisttyout} | sed -u -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g" > "${outfile1}"
;;
esac

The idea behind ctee is that it uses tee, but inverts the outputs. Ctee uses the console tty as the file to output to, and then standard out continues on to a sed command that strips out the console color escape sequences which is then redirected to the file specified to ctee as a parameter.

So one of the big tricks is to derive which console you are executing this command from. The while loop from lines 90-106 use a /proc filesystem trick to find the process on the other end of a pipe. This loop travels from this process’s standard out to the standard in of another process, and then takes that standard out, and tracks that down until it includes the phrase “dev” or “/” in its name, indicating either a tty or a file (technically the dev is redundant, but it makes the line easier to understand).

The second main trick of ctee is that it has a sed command strip out the console color control characters. I found a nice way to remove that, and you can see those commands on lines 113 and 115. One line is for appending to a file, and the other is for overwriting.

Summary

You can use ctee.sh like so:
unbuffer ansible-playbook playbook.yml | ctee.sh /var/log/ansible/playbook.$( date "+%Y-%m-%d-%H%M%S" ).log
Which will get your console the colorized output that helps for easy interpretation of the output, with the normal logging for later analysis.

Moving the application installation directory for bgscripts

Overview

My bgscripts package in version 1.1-27 and before was installed in /usr/bgscripts. The more I thought about it, and the more I read the spec for FHS 3.0 (filesystem hierarchy standard) (PDF), the more I realized it should actually be in /usr/share/bgscripts.

Moving the files in the build directory was easy.

Wrapper script

But despite my symlinks in /usr/bin, which were easy to update, I have many, many hard-coded pathname calls of my scripts everywhere. So to mitigate the problem, I wrote a fancy wrapper script for regular scripts, including bup, send, bgscripts.bashrc, rdp, and treesize.

I needed to send myself a message with the various information about what script called the old location. I picked send.sh because that’s what it’s for– sending emails. I slapped in a bunch of diagnostic information and wrote a call to the new location.

#!/bin/bash
# Wrapper at legacy location for bgscripts. Sends alert and then continues the process.
# DETERMINE LOCATION OF FRAMEWORK
while read flocation; do if test -x ${flocation} && test "$( ${flocation} --fcheck )" -ge 20161212; then frameworkscript="${flocation}"; break; fi; done <<EOFLOCATIONS
./framework.sh
${scriptdir}/framework.sh
/usr/share/bgscripts/framework.sh
/home/bgirton/rpmbuild/SOURCES/bgscripts-1.1-28/usr/share/bgscripts/framework.sh
EOFLOCATIONS
test -z "${frameworkscript}" && echo "$0: framework not found. Aborted." 1>&2 && exit 4
. ${frameworkscript} || echo "$0: framework did not run properly. Aborted." 1>&2
### CUSTOM WRAPPER BEGINS HERE
pwd="$( pwd )"
psout="$( ps -ef )"
parentprocess="$( ps -ef | grep -iE -- "${thisppid}" | grep -viE -- "grep.*-iE.*--" )"
/usr/share/bgscripts/send.sh -hs "bgscripts alert ${server}: ${scriptfile}" bgstack15@gmail.com <<EOF
----- Description -----
This alert means that script "${scriptfile}" was called, and you need to fix it because /usr/bgscripts/* has been deprecated and will be removed in a future version.
----- Debug info-----
pwd: ${pwd}
Scriptdir: ${scriptdir}
Scriptfile: ${scriptfile}
All parameters: $@
Server: ${server}
Now: ${now}
Thisos: ${thisos}
Thisflavor: ${thisflavor}
Thisppid: ${thisppid}
Parent process: ${parentprocess}
----- Process list -----
${psout}
EOF
### CALL NEW LOCATION
/usr/share/bgscripts/${scriptfile} "${@}"

So if something of mine calls /usr/bgscripts/bgscripts.bashrc, this wrapper script will throw an email alert, and then redirect the script call so workflows are not interrupted!

The very first production use of the wrapper script was when I logged onto my test Ubuntu box to conduct a deep-dive search (more on that in the next heading) of scripts that call framework.sh. I got an email right away, saying I had dot-sourced the bashrc script. Looking at the diagnostic info, I was reminded that I tended to hardcode into my ~/.bashrc the call to /usr/bgscripts/bgscripts.bashrc.

I have now updated that to point to /usr/bin/bp, so in the future it will point to wherever I want it to. However, I seriously doubt the location will change now. My package is now FHS 3.0 compliant, and I don’t want to break that.

Deep dive search

So, I wanted to conduct a deep-dive search of all shell scripts (defined by me as everything ending in .sh, because I almost always use the extension) on all systems that refer to framework.sh, which is going to be the big hangup. So I whipped up this oneliner.
sudo updatedb; locate .sh | grep -iE "\.sh$" | grep -viE "\/usr\/src|\/usr\/share|\/usr\/lib|\/home\/.*\/rpmbuild/|\/home\/work\/.*clean" | xargs grep -iE "framework\.sh"

Now, an reading of my code shows that many of my scripts do a lookup for framework.sh in various locations, including ./framework.sh, ~/framework.sh, /usr/local/bin/framework.sh, and so on. You can observe this pattern in my wrapper script in this post.

So the serach will catch those, but I will recognize those blocks of code because they are just the filename on a separate line (in a here-doc) and as long as I see the /usr/share/bgscripts/framework.sh final resting place of framework, I’ll be satisfied.

References

Weblinks

  1. Filesystem hierarchy standard 3.0 http://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html
  2. FHS 3.0 in pdf http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.pdf
  3. Here document https://en.wikipedia.org/wiki/Here_document

Shell one-liners for manipulating symlinks

Showing symlink targets

I was trying to clean up a directory filled with files and symlinks. Many of the symlinks’ targets did not exist anymore (old package versions). I wrote a few one-liners to help me examine and clean up the directory.

Here’s the first one-liner.

find . -type l -print0 | sed -e 's/\.\///g;' | xargs -0 -n1 | while read line; do canon=$( readlink -f "${line}" 2>/dev/null ); test -e "${canon}" && echo "${line} is $( /bin/ls -d --color=always "${canon}" )" || echo "INVALID ${line}"; done

In expanded form:
find . -type l -print0 | \
sed -e 's/\.\///g;' | \
xargs -0 -n1 | \
while read line; do \
canon=$( readlink -f "${line}" 2>/dev/null );
test -e "${canon}" && echo "${line} is $( /bin/ls -d --color=always "${canon}" )" || echo "INVALID ${line}";
done

The find command lists symlinks in the current directory, with special null characters as the separators. This will allow correct operation with spaces in filenames. You don’t normally encounter this in GNU/Linux, but you never know!

The sed removes the dot slash ./ from the beginning of filenames. I just don’t like seeing that. If I had selected find * earlier, I wouldn’t need the sed command, but I would miss any files whose names begin with dot (e.g., .gitignore) and I would still find file . which I would have to exclude somehow anyway (well, not here because I’m searching for symlinks, but I personally never try to run find * no matter what).

xargs -0 -n1 now removes the null character separators, and feeds each individual entry one at a time to the next command.

While: So for each entry, read the symlink. If the target is a file, display it in whatever color ls would show it (without entering directories), because colors are pretty! Otherwise, print the line with “INVALID” in front of it.

Cleaning up the invalid symlinks

find . -type l -print0 | sed -e 's/\.\///g;' | xargs -0 -n1 | while read line; do canon=$( readlink -f "${line}" 2>/dev/null ); test -e "${canon}" && true || { echo /bin/rm "./${line}"; }; done
Replace the echo commands with true for the affirmative and the echo /bin/rm command.
To actually effect the change, remove the echo in front of /bin/rm.