Fedora 27 ssh and default kerberos config

On my new Fedora 27 system which I joined to my FreeIPA domain, I encountered an error I hadn’t seen before.
In the past I could just say “ssh remotehost” and it would connect me with GSSAPI auth using my kerberos key– no password or ssh key needed! It was wonderful. However, I ran into this issue, as seen with ssh -v remotehost

debug1: Unspecified GSS failure.  Minor code may provide more information
Server host/remotehost@IPA.EXAMPLE.COM not found in Kerberos database

But I know for a fact it’s in the kerberos database!
I duckducked (new verb) the error message and found the culprit.
In file /etc/krb5.conf, this variable should be set to this value:

[libdefaults]
  dns_canonicalize_hostname = true

The default is true according to man krb5.conf. but for whatever reason, whether by joining the domain, or some default of some package in Fedora 27, it was set to false.

For the followers of my bgscripts package, just use this command:

sudo updateval -a /etc/krb5.conf -s '[libdefaults]' '^(\s*dns_canonicalize_hostname\s*=\s*).*' '  dns_canonicalize_hostname = true'

References

Weblinks

  1. https://superuser.com/questions/1166094/ssh-single-sign-on-with-kerberos/1166101#1166101
Advertisements

X forwarding for virt-manager to Windows

Story time! When I was working on my virtual environment, I rebooted my main desktop. So I was stuck using my Windows desktop for a minute, and I wanted to work on my virtual machines.

I decided to do some X forwarding, which for virt-manager on CentOS 7 needs some special steps.

On the server

The first thing is to install virt-manager. You also will need a piece of software named xauth, and some special fonts.

yum install -y virt-manager xauth dejavu-\*fonts

Reference: https://superuser.com/questions/119792/how-to-use-x11-forwarding-with-putty/119908#119908

On the client

On the Windows client, you should install an X server. I picked Xming. It also needs it fonts installed.

Run Xming.

Connect to server with PuTTY. You will need to configure PuTTY to allow X forwarding, and to use the right X server.

Screenshot of Putty configuration screen showing X11 forwarding options
Telling PuTTY to allow X11 forwarding to localhost:0

Notes

Unfortunately, my keyboard input to the virtual machine does not work when I have it configured with a spice display. But it works when I use a vnc display. See https://bugzilla.redhat.com/show_bug.cgi?id=1236412 for a closed bug report that has similar symptoms to this problem.

virt-manager settings of a virtual machine, showing where to change display to VNC server type.
Setting on virtual machine of VNC server or spice server.

References

Weblinks

  1. http://straightrunning.com/XmingNotes/#head-131
  2. https://superuser.com/questions/119792/how-to-use-x11-forwarding-with-putty/119908#119908
  3. https://sourceforge.net/projects/xming/files/Xming/
  4. https://sourceforge.net/projects/xming/files/Xming-fonts/
  5. https://robert.penz.name/354/how-to-fix-the-font-for-virt-manager-via-x-forwarding/
  6. https://bugzilla.redhat.com/show_bug.cgi?id=1236412
  7. PuTTY https://www.chiark.greenend.org.uk/~sgtatham/putty/

List outbound ssh sessions

tl;dr

sudo netstat -Watp | grep 'ESTABLISHED.*ssh' | awk '{print $5}' | sed 's/:ssh//;' | sort | uniq | while read line; do ps -ef | grep -o "ssh\s.*${line}"; done | sort | uniq | sed -r -e 's/ssh //g;' -e 's/-l (\w*) /\1@/;'

Backstory

During other work, it came up that I was interested in seeing what outbound ssh sessions I was using. Now I don’t even know why it came up, because I was just writing a shell script to programmatically adjust my xfce settings using its xfconf-query API.

Walking through the command

sudo netstat -Watp | grep 'ESTABLISHED.*ssh' | awk '{print $5}' | sed 's/:ssh//;' | sort | uniq | while read line; do ps -ef | grep -o "ssh\s.*${line}"; done | sort | uniq | sed -r -e 's/ssh //g;' -e 's/-l (\w*) /\1@/;'

This whole statement lists the established ssh connections and then finds the running processes for those and tries to identify the usernames for them.
Step by step:
Everything before the while collects the list of established ssh connections.
sudo netstat -Watp | grep ‘ESTABLISHED.*ssh’ gets the list of ssh connections, and awk | sed | sort | uniq just gets the information we want from each row and removes duplicates.
The while read line; do :; done loop iterates over the list. So for each line in the list, search all running processes for that name on the same line as the expression ‘ssh.’
sort | uniq removes duplicates (apparently qemu+kvm in virt-manager uses a lot of separate ssh processes).
sed -r -e ‘s/ssh //g;’ -e ‘s/-l (\w*) /\1@/;’ trims extra characters and also converts compatible outputs into “username@hostname.”

Improvements to be made

This snippet as is only works if the ssh command issued matches exactly the description of the output of netstat. If dns reverse zones are not configured correctly, so that the netstat shows an IP address but the ssh command was a hostname, this snippet will not find it. I need to improve that, which will probably require a fancier script and not just a oneliner.

References

Weblinks

  1. https://serverfault.com/questions/431034/getting-list-of-opened-ssh-connections-by-name

sshd_config match negate address

tl;dr

Match Address *,!192.168.1.0/24

Negating address in match statement in sshd_config

I was locking down my ssh server configuration on a host, so that it will not accept password auth from outside a certain IP address range.
I had to learn how to get the Match Address directive to work with a negation. To make it work, you need to insert a wildcard before you then state the exclusion.

Match Address *,!192.168.1.0/24

And then I added the directives for this matched IP address range.

   AuthenticationMethods publickey
   PubkeyAuthentication yes
   PasswordAuthentication no
   X11Forwarding no

References

Weblinks

  1. https://serverfault.com/questions/408284/how-can-the-address-condition-in-a-match-conditional-block-in-sshd-config-be-neg

Man pages

  1. sshd_config
  2. ssh_config

Do task until it succeeds

A story

I was working on my vm and needed to reboot it. In order to ssh back into the machine, I would have to wait for it to come back online and start up ssh.

Instead of manually polling myself, I whipped up this little one-liner:

while ! ssh centos7-01a; do true; done

So it failed silently at first, and then started showing ssh_exchange_identification: Connection closed by remote host.
Then when OpenSSH was finally ready for me, my kerberos authentication proceeded normally and I was in.
Upon closing my session, the while loop concluded and returned me to my shell.

I came up with this little snippet on a whim, and it actually helped me out and was not obtrusive and did not fail in any way.

Quickly bounce nic over ssh without losing ssh session

The story

If you make changes to a network card settings and need to restart it to take effect, you might find this useful.
Ssh is resilient enough to usually keep your session if you take the card down and up fast enough. Obviously you want to make sure the change you make will not prevent the card from being enabled correctly. I was making changes to my dns settings for the card, and I wanted them to take effect immediately.

The snippet

echo "ifdown eth0; sleep 2; ifup eth0;" > ~/foo.sh; chmod u+x ~/foo.sh; ~/foo.sh

Ssh into NATted VM via AutoSSH

Overview

Suppose you have a virtual machine running on a natted network connection. It can get out just fine to the Internet.
Now, with the network address translation (NAT), you can’t send traffic in to that network. So you can’t just ssh username@vm.example.com and get in. You are going to need something a little fancier to accomplish that. This document explains how to do that.

Sample environment

For this document, these example values will be used. The vm is centos.vm.example.com and the system on the main network is desktop.example.com. The desktop will have port 5000 be forwarded to the vm’s port 22, which is the standard ssh port.

Set up autossh

You need ssh connectivity between the vm and the physical host. Make sure you have an automatic log in from the vm to the physical host. A good way to do that is by having an ssh key. If you need to generate one on the vm and copy it to the physical machine, use these commands.
# on vm
ssh-keygen # follow the prompts

The next step is to copy it to the physical machine.
# still on vm
ssh-copy-id username@desktop.example.com

Check that you can log in automatically with
ssh username@desktop.example.com
Once that is done, install autossh and make a systemd service file and enable and start it.
yum -y install autossh
tf=/usr/lib/systemd/system/autossh-ssh.service
touch "${tf}"; chmod 0644 "${tf}"
cat <<EOF > "${tf}"
[Unit]
Description=AutoSSH tunnel service for ssh
After=network.target
[Service]
Environment="AUTOSSH_GATETIME=0"
ExecStart=/usr/bin/autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -N -R 5000:localhost:22 username@desktop.example.com -p 22
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable autossh-ssh.service
systemctl start autossh-ssh.service

Using the reverse tunnel

To connect to centos.vm.example.com, you just need to do this:
ssh centosusername@localhost -p 5000

Bonus: nickname the connection and copy ssh key

What I did was set up my ~/.ssh/config file with the following snippet, so I can just use the nickname “centosvm.”

# in ~/.ssh/config, mode 0600
Host centosvm centosvm.vm.example.com
 User centosuser
 Hostname localhost
 Port 5000

I then copied my ssh id to that, so I could connect without a password.
ssh-copy-id centosvm

References

  1. https://www.everythingcli.org/ssh-tunnelling-for-fun-and-profit-autossh/
  2. http://www.harding.motd.ca/autossh/index.html
  3. http://surniaulula.com/2012/12/10/autossh-startup-script-for-multiple-tunnels/

Ansible use ssh tunnel for http proxy

Overview

If you need to use a web proxy on an ansible node to get to the Internet, you can use the ansible role I hacked together. I had a need for using an ad-hoc ssh tunnel to a proxy server.

So your node will need an ssh key for automatic authentication to the proxy server over ssh. The role just executes a “ssh -N -p 22 username@proxy.example.com -L 1234/localhost/1234” and sets up the environment hash which includes http_proxy and https_proxy.

Check out the role at my github page.

Code walkthrough

My roles always have incredibly basic main.ymls because I always use a subdirectory architecture.
---
- hosts: all
tasks:
- include: tasks/main.yml
handlers:
- handlers/main.yml
vars_files:
- vars/main.yml

Now the tasks/main.yml file is where half of the work occurs.
---
# File: /etc/ansible/roles/use-proxy/tasks/main.yml
- name: clear any ssh tunnel
shell: ps -ef | grep -iE -- "ss[h].*{{local_proxy_port}}" | awk '{print $2}' | xargs kill -9
ignore_errors: yes
- name: start ssh tunnel
shell: nohup ssh -N -p {{proxy_server_ssh_port}} {{proxy_server}} -L {{local_proxy_port}}/localhost/{{proxy_port}} &
notify: stop ssh tunnel

It’s pretty clear what each task is doing. The start ssh tunnel opens up an ssh connection. If it is successful, it notifies the stop ssh tunnel handler which will execute at the end of the play.

- name: stop ssh tunnel
shell: ps -ef | grep -iE -- "ss[h].*{{local_proxy_port}}" | awk '{print $2}' | xargs kill -9

The handler searches for and kills the ssh connection that is using the specified port.

The vars/main.yml file is important because you must customize it to your environment.
---
proxy_port: 3128
local_proxy_port: "{{proxy_port}}"
proxy_server: tunnel@demo.example.com
proxy_server_ssh_port: 22
proxy_env:
http_proxy: "http://localhost:{{local_proxy_port}}"
https_proxy: "http://localhost:{{local_proxy_port}}"

How you use this role in a playbook is at the beginning of the list of roles. You also need to define the environment as seen below.
---
- name: Playbook that uses an ssh tunnel for http_proxy
hosts: test
remote_user: root
environment: "{{ proxy_env | default(omit) }}"
roles:
- { role: use-proxy, when: usehttpproxy is defined and usehttpproxy|bool == true }
- example

You can run this playbook with the use-proxy role and environment as is against any hosts that need the http proxy. To minimize traffic back and forth, just comment out the use-proxy role, and run the play against the regular hosts.

References

Weblinks

  1. https://ansiblemaster.wordpress.com/2016/04/29/run-ansible-tasks-to-a-remote-server-using-a-ssh-tunnel/
  2. https://www.engadget.com/2006/03/21/how-to-ssh-tunnels-for-secure-network-access/

Ssh use password this time

Problem

If you normally use passwordless authentication for your ssh sessions, you know how helpful it is. It saves you from having to type your password in all the time. You might have ssh keys set up (ssh-keygen) or kerberos.

In either case, you found this post because you want to use ssh with a password this time. You need to force ssh to use the password, just this once, without having to make all sorts of complicated requirements.

Solution

In either case, you found this post because you want to use ssh with a password this time. You need to force the password. Here’s how to do that:

function sshp { ssh -o PreferredAuthentications=password,keyboard-interactive -o PubkeyAuthentication=no -o GSSAPIAuthentication=no "$@"; }

The above snippet comes from my latest and greatest bgscripts.bashrc in my bgscripts package.

What the function does is execute ssh with a few extra options, that are very straightforward. It specifically lists the preferred methods for authentication, while disabling public key (which is what most people use) and kerberos auth (GSSAPI).

References

Weblinks