Prepend output with time to generate each line

To show how long it takes before showing each new line of output, use this neat command.

long_command | ts -i "%.s"
$ ./configure --prefix=/tools | ts -i %.s
0.082337 checking for a BSD-compatible install... /tools/bin/install -c
0.002841 checking whether build environment is sane... yes
0.008164 checking for a thread-safe mkdir -p... /tools/bin/mkdir -p
0.000040 checking for gawk... gawk
0.005892 checking whether make sets $(MAKE)... yes

References

Weblinks

  1. St├ęphane Chazelas at https://unix.stackexchange.com/questions/391210/prepending-each-line-with-how-long-it-took-to-generate-it/391222#391222
Advertisements

Watch for new processes and list them only once

tl;dr

while true; do ps -ef | grep -iE "processname" | grep -viE "grep \-"; done | awk '!x[$10]++'

Discussion

Top shows everything currently running, and updates every so often (default is 2 seconds). But it has so much information for all the processes it does not show you the details of a single entry.
ps -ef or ps auxe only shows you the processes at that instant. You can run that in a loop piped to grep, but then it continues to show you the same things again and again.

Watch for new processes and list them only once

You can use this snippet to show you new entries for the process you’re looking for.

while true; do ps -ef | grep -iE "processname" | grep -viE "grep \-"; done | awk '!x[$10]++'

What this does is a while loop of all the processes. It updates constantly, because of the while true. The second grep command just prevents it from finding the first grep statement. Piping the whole thing to the special awk statement that removes duplicate lines makes it show only unique ones. And the awk $10 is the tenth column, which for me was the process parameter which is what I wanted to show the uniqueness of.

An improvement upon while true loop

The story

So normally when I want to see output for something, I’ll run a while true loop.

while true; do zabbix_get -s rhevtester.example.com -k 'task.converter_cpu'; done

That doesn’t always stop even when I mash ^C (CTRL+C).

The solution

So I offer my improvement. The way to stop the loop is obvious, and also I will explain it.

while test ! -f /tmp/foo; do zabbix_get -s rhevtester.example.com -k 'task.converter_cpu'; done

To stop the loop, in another shell just:

touch /tmp/foo

Edit terminal title from the command line

tl;dr

export PROMPT_COMMAND='echo -ne "\033]0;NEW TEXT HERE\007"'

Edit terminal title from command line

To modify the window title directly, you just need to use this:

echo -ne "\033]0;NEW TEXT HERE\007"

But in a normal bash environment, your PROMPT_COMMAND will be executed before each display of the prompt, so to affect your interactive shell, you will need that export PROMPT_COMMAND.

References

Weblinks

  1. https://askubuntu.com/questions/22413/how-to-change-gnome-terminal-title

Quickly bounce nic over ssh without losing ssh session

The story

If you make changes to a network card settings and need to restart it to take effect, you might find this useful.
Ssh is resilient enough to usually keep your session if you take the card down and up fast enough. Obviously you want to make sure the change you make will not prevent the card from being enabled correctly. I was making changes to my dns settings for the card, and I wanted them to take effect immediately.

The snippet

echo "ifdown eth0; sleep 2; ifup eth0;" > ~/foo.sh; chmod u+x ~/foo.sh; ~/foo.sh

Remove only certain duplicate lines with awk

Basic solution

http://www.unix.com/shell-programming-and-scripting/153131-remove-duplicate-lines-using-awk.html demonstrates and explains how to use awk to remove duplicate lines in a stream without having to sort them. This statement is really useful.
awk '!x[$0]++'

The fancy solution

But if you need certain duplicated lines preserved, such as the COMMIT statements in the output of iptables-save, you can use this one-liner:
iptables-save | awk '!asdf[$0]++; /COMMIT|Completed on|Generated by/;' | uniq
The second awk rule prints again any line that matches “COMMIT” or “Completed on” or “Generated by,” which appear multiple times in the iptables-save output. I was programmatically adding rules and one host in particular was just adding new ones despite the identical rule already existing. So I had to remove the duplicates and save the output, but keep all the duplicate “COMMIT” statements. I also wanted to keep all the comments as well.

Moving the application installation directory for bgscripts

Overview

My bgscripts package in version 1.1-27 and before was installed in /usr/bgscripts. The more I thought about it, and the more I read the spec for FHS 3.0 (filesystem hierarchy standard) (PDF), the more I realized it should actually be in /usr/share/bgscripts.

Moving the files in the build directory was easy.

Wrapper script

But despite my symlinks in /usr/bin, which were easy to update, I have many, many hard-coded pathname calls of my scripts everywhere. So to mitigate the problem, I wrote a fancy wrapper script for regular scripts, including bup, send, bgscripts.bashrc, rdp, and treesize.

I needed to send myself a message with the various information about what script called the old location. I picked send.sh because that’s what it’s for– sending emails. I slapped in a bunch of diagnostic information and wrote a call to the new location.

#!/bin/bash
# Wrapper at legacy location for bgscripts. Sends alert and then continues the process.
# DETERMINE LOCATION OF FRAMEWORK
while read flocation; do if test -x ${flocation} && test "$( ${flocation} --fcheck )" -ge 20161212; then frameworkscript="${flocation}"; break; fi; done <<EOFLOCATIONS
./framework.sh
${scriptdir}/framework.sh
/usr/share/bgscripts/framework.sh
/home/bgirton/rpmbuild/SOURCES/bgscripts-1.1-28/usr/share/bgscripts/framework.sh
EOFLOCATIONS
test -z "${frameworkscript}" && echo "$0: framework not found. Aborted." 1>&2 && exit 4
. ${frameworkscript} || echo "$0: framework did not run properly. Aborted." 1>&2
### CUSTOM WRAPPER BEGINS HERE
pwd="$( pwd )"
psout="$( ps -ef )"
parentprocess="$( ps -ef | grep -iE -- "${thisppid}" | grep -viE -- "grep.*-iE.*--" )"
/usr/share/bgscripts/send.sh -hs "bgscripts alert ${server}: ${scriptfile}" bgstack15@gmail.com <<EOF
----- Description -----
This alert means that script "${scriptfile}" was called, and you need to fix it because /usr/bgscripts/* has been deprecated and will be removed in a future version.
----- Debug info-----
pwd: ${pwd}
Scriptdir: ${scriptdir}
Scriptfile: ${scriptfile}
All parameters: $@
Server: ${server}
Now: ${now}
Thisos: ${thisos}
Thisflavor: ${thisflavor}
Thisppid: ${thisppid}
Parent process: ${parentprocess}
----- Process list -----
${psout}
EOF
### CALL NEW LOCATION
/usr/share/bgscripts/${scriptfile} "${@}"

So if something of mine calls /usr/bgscripts/bgscripts.bashrc, this wrapper script will throw an email alert, and then redirect the script call so workflows are not interrupted!

The very first production use of the wrapper script was when I logged onto my test Ubuntu box to conduct a deep-dive search (more on that in the next heading) of scripts that call framework.sh. I got an email right away, saying I had dot-sourced the bashrc script. Looking at the diagnostic info, I was reminded that I tended to hardcode into my ~/.bashrc the call to /usr/bgscripts/bgscripts.bashrc.

I have now updated that to point to /usr/bin/bp, so in the future it will point to wherever I want it to. However, I seriously doubt the location will change now. My package is now FHS 3.0 compliant, and I don’t want to break that.

Deep dive search

So, I wanted to conduct a deep-dive search of all shell scripts (defined by me as everything ending in .sh, because I almost always use the extension) on all systems that refer to framework.sh, which is going to be the big hangup. So I whipped up this oneliner.
sudo updatedb; locate .sh | grep -iE "\.sh$" | grep -viE "\/usr\/src|\/usr\/share|\/usr\/lib|\/home\/.*\/rpmbuild/|\/home\/work\/.*clean" | xargs grep -iE "framework\.sh"

Now, an reading of my code shows that many of my scripts do a lookup for framework.sh in various locations, including ./framework.sh, ~/framework.sh, /usr/local/bin/framework.sh, and so on. You can observe this pattern in my wrapper script in this post.

So the serach will catch those, but I will recognize those blocks of code because they are just the filename on a separate line (in a here-doc) and as long as I see the /usr/share/bgscripts/framework.sh final resting place of framework, I’ll be satisfied.

References

Weblinks

  1. Filesystem hierarchy standard 3.0 http://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html
  2. FHS 3.0 in pdf http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.pdf
  3. Here document https://en.wikipedia.org/wiki/Here_document

Bash one liner: remove extra slashes from file path

If you have a generated path, you might end up with extra slashes, such as

/mnt/foo//bar

In a typical Linux file system such as ext2, ext3, or ext4, /mnt/foo//bar is the same file as /mnt/foo/bar. The slashes two in a row are the same as a /./, and a single dot is the same directory. So /mnt/. is directory /mnt/ — that’s just how they made the filesystem.
The notable exception is at the beginning of a path, where //servername would actually be a reference to the network host servername. I leave it as an exercise to the reader to know when he is parsing network paths and when he is not.

You can clean duped slashes easily by doing this:

file="$( file | sed -e 's!\/\+!\/!g;' )"