Kerberos notes and sssd Internal credentials cache error

If sssd gives you errors about unable to connect, it’s probably the host password (keytab) is out of date with what AD has. You have to reset the host account in AD, or even delete the computer account and rejoin the domain.

kdestroy -A
kinit domainadmin
msktutil -f -s host
msktutil -u -s host
kinit -k "$( hostname -s | tr '[[:lower:]]' '[[:upper:]]' )\$@MSAD.EXAMPLE.COM"
klist -kt

The kvno value in the output of klist -kt should match the attribute “msDS-KeyVersionNumber” of the server object in AD.

Error can include:

(Thu Aug  9 15:28:57 2018) [[sssd[krb5_child[3177]]]] [create_ccache] (0x0020): 1009: [-1765328188][Internal credentials cache error]
(Thu Aug  9 15:28:57 2018) [[sssd[krb5_child[3177]]]] [map_krb5_error] (0x0020): 1657: [-1765328188][Internal credentials cache error]
(Thu Aug  9 15:29:22 2018) [[sssd[krb5_child[3333]]]] [privileged_krb5_setup] (0x0080): Cannot open the PAC responder socket
(Thu Aug  9 15:29:22 2018) [[sssd[krb5_child[3333]]]] [sss_send_pac] (0x0040): sss_pac_make_request failed [-1][2].
(Thu Aug  9 15:29:22 2018) [[sssd[krb5_child[3333]]]] [validate_tgt] (0x0040): sss_send_pac failed, group membership for user with principal [bgstack15\@MSAD.EXAMPLE.COM@MSAD.EXAMPLE.COM] might not be correct.
Advertisements

RHEL7 and sssd: getent works but users cannot authenticate with passwords

In my situation, we used realm to install and configure sssd. Getent shows the domain users, and the groups are in /etc/ssh/sshd_config and /etc/sssd/sssd.conf. So why can my users not authenticate to the server?!

I get this error:

May 01 13:53:43 lhostname sshd[22167]: pam_sss(sshd:auth): authentication success; logname= uid=0 euid=0 tty=ssh ruser= rhost=l982487.prod1.example.com user=bgstack15
May 01 13:53:44 lhostname sshd[22167]: Failed password for bgstack15 from 10.155.16.240 port 34340 ssh2

What eventually solved the problem was fixing /etc/pam.d/system-auth-ac and also /etc/pam.d/password-auth-ac. Either file by itself did not seem to help. It looks like authconfig may have messed up or was not used or something. I don’t bother using it myself because I just modify pam myself. Yes, this makes me a terrible admin.

/etc/pam.d/system-auth-ac

#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      pam_env.so
auth        required      pam_faildelay.so delay=2000000
auth        required      pam_faillock.so preauth silent deny=4 unlock_time=1200
auth        sufficient    pam_fprintd.so
auth        [default=1 ignore=ignore success=ok] pam_succeed_if.so uid >= 1000 quiet
auth        [default=1 ignore=ignore success=ok] pam_localuser.so
auth        sufficient    pam_unix.so nullok try_first_pass
auth        requisite     pam_succeed_if.so uid >= 1000 quiet_success
auth        sufficient    pam_sss.so forward_pass
auth        required      pam_faillock.so authfail deny=4 unlock_time=1200
auth        required      pam_deny.so

account     required      pam_faillock.so
account     required      pam_unix.so
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 1000 quiet
account     [default=bad success=ok user_unknown=ignore] pam_sss.so
account     required      pam_permit.so

password    requisite     pam_pwquality.so minlen=16 dcredit=-1 ucredit=-1 ocredit=-1 lcredit=-1 try_first_pass retry=3
password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok remember=5
password    sufficient    pam_sss.so use_authtok
password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
-session     optional      pam_systemd.so
session     optional      pam_oddjob_mkhomedir.so umask=0077
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so
session     optional      pam_sss.so

/etc/pam.d/password-auth-ac

#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      pam_env.so
auth        required      pam_faildelay.so delay=2000000
auth        required      pam_faillock.so preauth silent deny=4 unlock_time=1200
auth        [default=1 ignore=ignore success=ok] pam_succeed_if.so uid >= 1000 quiet
auth        [default=1 ignore=ignore success=ok] pam_localuser.so
auth        sufficient    pam_unix.so nullok try_first_pass
auth        requisite     pam_succeed_if.so uid >= 1000 quiet_success
auth        sufficient    pam_sss.so forward_pass
auth        required      pam_deny.so

account     required      pam_faillock.so
account     required      pam_unix.so
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 1000 quiet
account     [default=bad success=ok user_unknown=ignore] pam_sss.so
account     required      pam_permit.so

password    requisite     pam_pwquality.so minlen=16 dcredit=-1 ucredit=-1 ocredit=-1 lcredit=-1 try_first_pass retry=3
password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok
password    sufficient    pam_sss.so use_authtok

auth        required      pam_faillock.so authfail deny=4 unlock_time=1200

password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
-session     optional      pam_systemd.so
session     optional      pam_oddjob_mkhomedir.so umask=0077
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so
session     optional      pam_sss.so

getent passwd -s sss LOCALUSER shows local user

tl;dr

I want to easily and quickly tell if a user is local or domain (don’t care which domain).

Environment

  • freeipa-client-4.6.1-3.fc27.x86_64
  • sssd-1.16.0-4.fc27.x86_64

Full story

I am writing a userinfo.sh script that will show if a user is local, sssd, can ssh, and is permitted by sssd.

Currently I am doing the check for if the user is from the domain with the getent passwd -s sss $USERNAME command. But I ran into an issue where checking the sssd database returns a local user!

# getent passwd -s sss 'bgstack15-local'
bgstack15-local:x:1000:1000:bgstack15-local:/home/bgstack15-local:/bin/bash

Checking the contents of the database (cache) for sss shows sssd apparently caches all sorts of information about the local user.

# sudo su root -c 'strings /var/lib/sss/db/* | grep bgstack15-local' | sort | uniq
name=bgstack15-local@implicit_files,cn=groups,cn=ih
name=bgstack15-local@implicit_files,cn=groups,cn=implicit_files,cn=sysdb
name=bgstack15-local@implicit_files,cn=users,cn=implicit_files,cn=sysdb
[...output truncated]

I tried clearing the sssd cache overall, and just for the user. Neither made a difference.

# sss_cache -U
# sss_cache -u bgstack15-local

The user does show up as a local user, and I promise it is only a local user!

getent passwd -s files 'bgstack15-local'
bgstack15-local:x:1000:1000:bgstack15-local:/home/bgstack15-local:/bin/bash

The man pages for getent(1) and getpwent(3) don’t help me understand what could be going on. sssd(8) shows me that sssd can cache local users, which actually goes against what I want! The nss section of sssd.conf(5) doesn’t help, but maybe I didn’t take enough time to read it. I’m a little stuck.

My sssd.conf

[domain/ipa.example.com]
id_provider = ipa
ipa_server = _srv_, dns1.ipa.example.com
ipa_domain = ipa.example.com
ipa_hostname = fc27c-01a.ipa.example.com
auth_provider = ipa
chpass_provider = ipa
access_provider = ipa
cache_credentials = True
ldap_tls_cacert = /etc/ipa/ca.crt
krb5_store_password_if_offline = True
[sssd]
services = nss, pam, ssh, sudo
domains = ipa.example.com
[nss]
homedir_substring = /home
[pam]
[sudo]
[autofs]
[ssh]
[pac]
[ifp]
[secrets]
[session_recording]

Last resort

I can try doing my checks against ${USERNAME}@${DOMAIN} when doing the -s sss check, but that means I then have to iterate over all domains in sssd.conf and that would slow the process down.


Answer

The option that controls this behavior is buried in sssd.conf(5) on CentOS 7 and Fedora, but not in the online man page.

sssd.conf

[sssd]
enable_files_domain = false

Reference 3 shows that sssd makes a “fast cache for local users.”

From man sssd.conf(5) on my Fedora system:

   enable_files_domain (boolean)
       When this option is enabled, SSSD prepends an implicit domain with
       “id_provider=files” before any explicitly configured domains.

       Default: true

Disabling this behavior lets me make a simple check to see if it is a local user or domain user.

References

  1. ddg: sssd disable caching local users
  2. https://bugzilla.redhat.com/show_bug.cgi?id=1357418
  3. https://fedoraproject.org/wiki/Changes/SSSDCacheForLocalUsers
  4. Fedora 27 sssd.conf(5)

sssd disable known_hosts hashing

If you use Fedora GNU/Linux, and you want bash autocompletion for the hosts in your FreeIPA domain when using the OpenSSH client, you have to disable knownhosts hashing in the global knownhosts file.

To find the global knownhosts file, check /etc/ssh/ssh_config

# grep -i knownhosts /etc/ssh/ssh_config
ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h
GlobalKnownHostsFile /var/lib/sss/pubconf/known_hosts

The file displayed here is the default on Fedora: /var/lib/sss/pubconf/known_hosts.

Now, to disable the hashing inside that file, you have to configure sssd.

# cat /etc/sssd/conf.d/50_ssh_hash_known_hosts.conf 
[ssh]
ssh_hash_known_hosts = false

I don’t know at what version the dot-dee directory for sssd.conf was introduced specifically (other then when it is compiled with libini>=1.3.0), but mine had it as of Fedora 27 and sssd-1.16.0. But with this entry in the config, restart sssd and maybe delete the previous /var/lib/sss/pubconf/known_hosts file which will be re-generated as it needs it.

Reference

Weblinks

https://jhrozek.fedorapeople.org/sssd/1.14.1/man/sssd.conf.5.html

Custom userinfo.sh

History

I needed to query certain information about a user on a Linux system. Specifically this output:

user: bgstack15
getent: YES
getent_type: sss
can_ssh: YES
can_sss: YES

I wanted to know if a user is defined (getent), and if so, in which database (local or in Active Directory). Also, is the user in the AllowUsers list of the sshd_config, or a member of a group in the AllowGroups list. And then the same question for the sssd config file.

The script

Freeipa client uninstall and reinstall

If you are changing ipa domains on a client, you first uninstall the client.

ipa-client-install --uninstall

Then you install in the new domain. (The lack of options here indicates it will search dns, so make sure your _kerberos entries are correct!)

ipa-client-install --mkhomedir --force-ntpd --enable-dns-updates

If you have problems with user accounts on the client for the new domain, it’s possible you need to manually clear out the sss cache to remove traces of the old domain.

rm -rf /var/lib/sss/db/*
systemctl restart sssd.service

References

Weblinks

  1. https://serverfault.com/questions/582854/how-to-reset-keytab-for-freeipa-server-and-client#583319