CF Web>Infrastructure>CSCFActiveDirectory>ADAddKrb5NFS (2014-04-02, ClaytonTucker)

-- Main.ctucker - 23 Nov 2010

NFS Shar"> How to Mount Kerberized NFS Shares as a Member of the CSCF Active Directory

Document UNDER DEVELOPMENT and intended for use in CSCF - University of Waterloo

For the purposes of this document, the NetApp server providing NFS shares is called fs02.student.cs.uwaterloo.ca and the Ubuntu host which will be the client of this NFS server is called ubuntutest01.cs.uwaterloo.ca. Both systems will be made members of the CS-TEACHING domain (STUDENT.CS.UWATERLOO.CA Kerberos realm) within the CSCF Active Directory forest.

How to Mount Kerberized NFS Shares as a Member of the CSCF Active Directory

Configuring The NetApp Server

Configuration of the NetApp server for kerberized NFS service follows consistently from sections 5.2 and 5.3 of Network Appliances Inc.'s white paper, 'Unified Windows and UNIX Authentication Using Microsoft Active Directory Kerberos' by Ellie Berriman, 2006.

To start, the following NetApp server files must have the correct information. These files are normally accessed through a mount point on an administrative UNIX system such as cscf.cs:/autofs/NetApp/fs02.student.cs. From there the path follows the mount point, through the NetApp volume root to the etc directory. In the case of files under /etc on fs02.student.cs, they would be found on cscf.cs at /autofs/NetApp/fs02.student.cs/etc/.

/etc/resolv.conf: nameserver 129.97.152.10 nameserver 129.97.152.100
/etc/nsswitch.conf: hosts: files dns nis passwd: files ldap netgroup: files nis group: files ldap shadow: files ldap
/etc/exports: / -sec=sys,rw=backup-0.cs:cscf.cs:student.cs,root=backup-0.cs:cscf.cs:student.cs /vol/vol0/home -sec=sys:krb5:krb5i:krb5p,rw,root=cscf.cs:student.cs:backup.cs /vol/sandbox -sec=krb5:krb5i:krb5p,rw,root=student.cs:cscf.cs:129.97.170.228

The /etc/exports defines the shares to be provided by the NFS server. For the purposes of this exercise we will be attempting to mount personal user disk space from the sandbox share. This share's security settings are strictly kerberos (no sys option as on the home share). It is important to ensure that sys security is not available on a share that must use kerberos because NFS client applications will default to sys if a kerberized connection cannot be done - we definitely do not want that to occur. Although the shares are defined they do not become available (or update) until the exportfs -a command is run on the NetApp server - see below.

As a sidebar, once configuration of the NFS server and client systems is complete and successfully using kerberos, an administrator can expand the permissions on a share like sandbox such that legacy systems with no TCP based kerberos client (our AD only uses TCP for kerberos operations) may still use the share using the old sys security mechanism. Chris Kranz's blog A NetApp Technical Diary outlines, with several examples, how various permission strategies can be accomplished in a NetApp exports file. Below is example of a modified sandbox share where the hosts cpu18.student.cs, student.cs and cscf.cs may obtain read/write access to the share through sys whereas all other hosts must use kerberos.
/vol/sandbox -sec=sys,rw=cpu18.student.cs:student.cs:cscf.cs,sec=krb5:krb5i:krb5p,rw,root=student.cs:cscf.cs:129.97.170.228

Within the sandbox share, a user's home directories are located in a fashion consistent with their personal unixHomeDirectory attribute value. That is /vol/sandbox/u#/long_username. Unfortunately, the pam_mount.so module for mounting user disk space to be used on the Ubuntu host or workstation, is too simple minded to expand the information in a user's unixHomeDirectory attribute to locate the disk space under /vol/sandbox. A simple /vol/sandbox/u/username format has to be used to find personal disk space. As a result, all user disk space will need to be symbolically linked to a corresponding /vol/sandbox/u/username path.

The next step in the configuration of the NetApp server requires logging in directly to fs02.student.cs.uwaterloo.ca and using the NetApp command interface. Using the options command, we configure the DNS client for the server.

options dns.cache.enable on
options dns.domainname student.cs.uwaterloo.ca
options dns.enable on

When you set the value for dns.domainname you may be asked to update the corresponding value in /etc/rc, this should be done to ensure the domain name is retained after any server reboot.

Now configure the CIFS service on the NetApp. It does not matter whether you intend to use CIFS, this procedure binds the NetApp server to the Active Directory domain. This will result in a computer object for the NetApp server being created within the Active Directory domain. You will be prompted for an account name and password with sufficient privileges to add a computer to the domain.

cifs setup Windows Domain Name: student.cs.uwaterloo.ca Filer AD Name: fs02

Next configure the NTP client for time synchronization services. Kerberos authentication is very sensitive t time differentials between KDC and client systems.

options timed.enable on
options timed.proto ntp
options timed.servers ntp.student.cs.uwaterloo.ca

If you wish users to be able to mount personal disk space directly during login then the following option must be disabled.

options nfs.mount_rootonly off

Also configure the NetApp's LDAP client as this does not happen as part of joining the NetApp to the Active Directory through the CIFS setup process above. The settings listed below are consistent with those used to add an Ubuntu system into one of our Active Directory domains (see: ADAddUbuntu). There is no need to set the ldap.passwd option in CSCF's current LDAP server setup so it is not listed.

options ldap.ADdomain student.cs.uwaterloo.ca
options ldap.base dc=student,dc=cs,dc=uwaterloo,dc=ca
options ldap.base.group dc=student,dc=cs,dc=uwaterloo,dc=ca
options ldap.base.netgroup dc=student,dc=cs,dc=uwaterloo,dc=ca
options ldap.base.passwd dc=student,dc=cs,dc=uwaterloo,dc=ca
options ldap.enable on
options ldap.minimum_bind_level anonymous
options ldap.nssmap.attribute.gecos name
options ldap.nssmap.attribute.gidNumber gidNumber
options ldap.nssmap.attribute.groupname displayName
options ldap.nssmap.attribute.homeDirectory homeDirectory
options ldap.nssmap.attribute.loginShell loginShell
options ldap.nssmap.attribute.memberNisNetgroup memberNisNetgroup
options ldap.nssmap.attribute.memberUid memberUid
options ldap.nssmap.attribute.netgroupname cn
options ldap.nssmap.attribute.nisNetgroupTriple nisNetgroupTriple
options ldap.nssmap.attribute.uid sAMAccountName
options ldap.nssmap.attribute.uidNumber uidNumber
options ldap.nssmap.attribute.userPassword userPassword
options ldap.nssmap.objectClass.nisNetgroup nisNetgroup
options ldap.nssmap.objectClass.posixAccount user
options ldap.nssmap.objectClass.posixGroup group
options ldap.port 389
options ldap.servers ldap.student.cs.uwaterloo.ca
options ldap.servers.preferred ldap.student.cs.uwaterloo.ca
options ldap.ssl.enable off
options ldap.timeout 20
options ldap.usermap.attribute.unixaccount unixaccount
options ldap.usermap.attribute.windowsaccount windowsaccount
options ldap.usermap.base
options ldap.usermap.enable off

Now enable Kerberized NFS.

options nfs.kerberos.enable on

nfs setup Enable Kerberos for NFS? y The filer supports these types of Kerberos Key Distribution Centers (KDCs): 1 - UNIX KDC 2 - Microsoft Active Directory KDC Enter the type of your KDC (1-2): 2 Kerberos now enabled for NFS. NFS setup complete.

With this process complete the FS02 computer object in the Active Directory should have the following attributes defined with Kerberos principal names. It is important that these principal names be associated only with this computer object. If any of these names have become assigned elsewhere with the domain then the NetApp server may not be able to bind with the Active Directory and kerberos authentication will fail.

userPrincipalName: nfs@fs02.student.cs.uwaterloo.ca

servicePrincipalName: HOST/fs02
                      HOST/fs02.student.cs.uwaterloo.ca
                      nfs/fs02
                      nfs/fs02.student.cs.uwaterloo.ca

If, at a later time, an administrator chooses to add an alternate hostname and IP address to the NetApp file server (due to firewall issues say) then the alternate hostname (fs02-alt.student.cs.uwaterloo.ca for example) must be added to the servicePrincipalName attribute for the NetApp server in a corresponding fashion.

servicePrincipalName: HOST/fs02
                      HOST/fs02.student.cs.uwaterloo.ca
                      HOST/fs02-alt
                      HOST/fs02-alt.student.cs.uwaterloo.ca
                      nfs/fs02
                      nfs/fs02.student.cs.uwaterloo.ca
                      nfs/fs02-alt
                      nfs/fs02-alt.student.cs.uwaterloo.ca

The last step in this process is to create or update the shares for export on the NFS server. This command must be run each time the share definitions in /etc/exports are modified.

exportfs -a (running exportfs -r will also work)

Configuring The Ubuntu Host

The first thing to be cognizant of when preparing an Ubuntu system for kerberized NFS mounting from a NetApp server is that it must have two (2) service principal names (SPN) associated with it in the Active Directory domain (Kerberos realm) where the Ubuntu host or workstation is to authenticate. And according to section 6.2.4.1 of the above mentioned Network Appliances white paper, this definitely requires two (2) user objects to represent the single Ubuntu host or workstation within the Active Directory domain in question. In the case of this exercise these principal names would be as follows.

host/ubuntutest01.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA used for logins and network services such as ssh
- This one is regularly created when configuring an Ubuntu client to authenticate from an AD domain as outlined in ADAddUbuntu and ADAddSolaris10
- In accordance to those documents, the corresponding user object name in CSCF's Active Directory domain (CS-TEACHING for this example) for this principal would be ubuntutest01_host
root/ubuntutest01.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA used for mounting NFS mounts
- In a similar vein, the corresponding user object name in CSCF's Active Directory domain for this principal would be ubuntutest01_root

As with the principal names associated with the NetApp server, the host and root SPNs for the Ubuntu host must not be mapped to any other objects than those listed above. Otherwise binding to the Active Directory for authentication may not work.

Standard Ubuntu Active Directory Configuration

Join the Ubuntu Linux host or workstation (ubuntutest01.cs.uwaterloo.ca in this case) to an Active Directory domain as outlined in the ADAddUbuntu wiki page. This system will be a client of the above configured NetApp server from where user disk space shall be mounted. For the purpose of this example the Ubuntu system will join (authenticate from) our CS-TEACHING domain (STUDENT.CS.UWATERLOO.CA Kerberos realm).

NOTE: CSCF uses a scripting mechanism on cscf.cs to speed and stream line the joining process but it is just as valid to perform the join to CS-TEACHING manually.

During the join process the ubuntutest01_host user object is created in the CS-TEACHING domain for the Ubuntu system. As well, the service principal name (SPN) host/ubuntutest01.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA is mapped to the userPrincipalName attribute of the new user object.
userPrincipalName: host/ubuntutest01.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA
NOTE: For a user object representing a computer it is the userPrincipalName attribute that carries the host SPN and not the servicePrincipalName attribute. The same will be true for the root user object and SPN created in the next step.

Create the `root` User Object and SPN Within the Active Directory

Within the CS-TEACHING domain create the root user object and SPN for the Ubuntu host or workstation in a manner similar to that outlined in creating a host SPN and user object in ADAddSolaris10. The difference being that a user object called ubuntutest01_root in CS-TEACHING will be created and the SPN root/ubuntutest01.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA will be written to its userPrincipalName attribute. You do not require the same password as used to create the ubuntutest01_host user object, but it does make this process easier.

For this new user object, note the value of the msDS-KeyVersionNumber attribute. This is equivalent to the KVNO number which will be required for the Ubuntu keytab file later. Every time a user object's password is set or reset this number increments. Therefore each time a keytab file is updated it is important to know the value of this attribute. See Notes On KVNO Numbers and msDS-KeyVersionNumber Attributes for more details.

Check The Ubuntu System's Kerberos Client Version

At a command prompt of the Ubuntu system enter the following command and check the krb5-user package version number.
aptitude show krb5-user

If the version for krb5-user is 1.8 or greater, then chances are the DES type encryption may be disabled for the Ubuntu kerberos client. To be noted below, DES-CBC-CRC is required for NetApp NFS servers when securing shares with kerberos. And even if you have legitimately created a keytab line for the root instance using DES-CBC-CRC encryption, it still won't work. And the local rpc.gssd daemon will generate an error message in the /var/log/syslog file similar to the one listed below when the mount is attempted (during user login).

Apr  8 12:12:47 nettop1 rpc.gssd[791]: rpcsec_gss: gss_init_sec_context: (major) Unspecified GSS failure.  Minor code may provide more information - (minor) No supported encryption types (config file error?)

A recommended workaround for this problem (aside from trying to recompile the kerberos client with DES enabled) is to add the following line to the [libdefaults] section of the Ubuntu system's /etc/krb5.conf file.
allow_weak_crypto = 1

Update The `/etc/krb5.keytab` File On The Ubuntu System

Because we have already joined the Ubuntu host to an AD domain, there should exist a valid /etc/krb5.keytab on this system. Instead of transferring the AD generated keytab file to the Ubuntu system, it is easier to simply add a new line to the Ubuntu systems' keytab file for the root SPN using the its local ktutil utility. This process is similar to the manner used to correct bad keytab files in the Troubleshooting kinit Errors section of ADAddSolaris10

Make a backup copy of the existing /etc/krb5.keytab file on the Ubuntu system as a precaution.
Take note of the value of the msDS-KeyVersionNumber attribute for the ubuntutest01_root user object for the KVNO number.
At an Ubuntu command prompt, enter the ktutil utility.
- Read in the local keytab file:
  rkt /etc/krb5.keytab
- Add a new line for the root SPN:
  addent -password -p root/ubuntutest01.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA -k KVNO_number -e encryption_mechanism
  - The encryption_mechanism used is the same one used for the host SPN entry in your keytab file. Most often it is either rc4-hmac or DES-CBC-MD5. If you are unsure as to which should be used, it does no harm to create an additional keytab line for a separate encryption mechanism.
  - You will be prompted for a password at this point, enter the same complex password used to create the ubuntutest01_root account object in CS-TEACHING
- Add a second line for the root SPN specifically for DES-CBC-CRC encryption:
  addent -password -p root/ubuntutest01.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA -k KVNO_number -e DES-CBC-CRC
  - Use the same complex password here.
- Write the updated keytab file:
  wkt /etc/krb5.keytab
- Quit addent: q
Initialize the Kerberos client for both principal names.
- kinit -k host/ubuntu.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA
- kinit -k root/ubuntu.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA
Create two cron entries to re-run initialization on a daily basis.
Create an /etc/init.d so that kinit will also occur at boot time.

NOTE: The above steps for updating the /etc/krb5.keytab are valid, however it is highly recommended that one create a script that will manage this process due to the greater complexity of the keytab file.

Install and Configure `pam_mount` and Other Additional Software

Install the libpam-mount package onto the local Ubuntu host.
- On the local Ubuntu host run, sudo apt-get install libpam-mount.
- This will install the pam_mount.so module required for the local PAM stack as well as the configuration file /etc/security/pam_mount.conf.xml which will be used to define the mounts to be created during user login.
- Within pam_mount.conf.xml add the following lines in the Volume-related section
  <volume fstype="nfs" server="fs02.student.cs.uwaterloo.ca" options="sec=krb5,tcp,nodiratime,rsize=32768,wsize=32768" path="/vol/sandbox/u0/%(GROUP)" mountpoint="~" />
  <volume fstype="nfs" server="fs02.student.cs.uwaterloo.ca" options="sec=krb5,tcp,nodiratime,rsize=32768,wsize=32768" path="/vol/sandbox/u1/%(GROUP)" mountpoint="~" />
  <volume fstype="nfs" server="fs02.student.cs.uwaterloo.ca" options="sec=krb5,tcp,nodiratime,rsize=32768,wsize=32768" path="/vol/sandbox/u2/%(GROUP)" mountpoint="~" />
  <volume fstype="nfs" server="fs02.student.cs.uwaterloo.ca" options="sec=krb5,tcp,nodiratime,rsize=32768,wsize=32768" path="/vol/sandbox/u3/%(GROUP)" mountpoint="~" />
  <volume fstype="nfs" server="fs02.student.cs.uwaterloo.ca" options="sec=krb5,tcp,nodiratime,rsize=32768,wsize=32768" path="/vol/sandbox/u4/%(GROUP)" mountpoint="~" />
  <volume fstype="nfs" server="fs02.student.cs.uwaterloo.ca" options="sec=krb5,tcp,nodiratime,rsize=32768,wsize=32768" path="/vol/sandbox/u5/%(GROUP)" mountpoint="~" />
  <volume fstype="nfs" server="fs02.student.cs.uwaterloo.ca" options="sec=krb5,tcp,nodiratime,rsize=32768,wsize=32768" path="/vol/sandbox/u6/%(GROUP)" mountpoint="~" />
  <volume fstype="nfs" server="fs02.student.cs.uwaterloo.ca" options="sec=krb5,tcp,nodiratime,rsize=32768,wsize=32768" path="/vol/sandbox/u7/%(GROUP)" mountpoint="~" />
  <volume fstype="nfs" server="fs02.student.cs.uwaterloo.ca" options="sec=krb5,tcp,nodiratime,rsize=32768,wsize=32768" path="/vol/sandbox/u8/%(GROUP)" mountpoint="~" />
  <volume fstype="nfs" server="fs02.student.cs.uwaterloo.ca" options="sec=krb5,tcp,nodiratime,rsize=32768,wsize=32768" path="/vol/sandbox/u9/%(GROUP)" mountpoint="~" />
  <msg-authpw>Please Enter Your Password:</msg-authpw>
  <logout wait="750000" kill="yes" />
- For the above NFS volume mounts, only one is expected to be successful during user login and the others will fail with an error. However, in the CSCF environment, user home disk space is scattered over several partitions of the NFS server and all partitions need to be checked for each user. This is a consequence of the '~' symbol (home directory path) not expanding in the path option as it does in the mountpoint option.
- For the above NFS paths, the GROUP variable is used to represent the username portion of the unixHomeDirectory path. This is a consequence of CSCF's unique environment: home paths depend on the long form username string (full surname, longer than eight characters) which is not used as the actual login username - a maximum of eight characters and represented by the USER variable in the pam_mount.conf.xml file. The long form username is however the name of a user's default group in the CSCF environment. Thus we use the GROUP variable as part of the path option.
- The logout wait option is required by some X Windows GUIs, such as gdm. During logout, some GUI services attempt to unmount user disk space too soon causing the "device is busy" error found in /var/log/auth.log - the user disk space fails to unmount. The above command string delays the umount process for three quarters of a second to allow all files and directories used by X Windows within the user disk space to be closed. If X Windows is not complete after that period then all session X Windows processes are killed.
Install the nfs-common as well.
- On the local Ubuntu host run, sudo apt-get install nfs-common.
- This will create the configuration file /etc/default/nfs-common.
- Within the newly created /etc/default/nfs-common file, set NEED_GSSD=yes and NEED_IDMAPD=yes

The Ubuntu host will require a reboot at this point.

Update The PAM Stack to Mount Personal NFS Disk Space At Logon

Update the PAM stack.
- /etc/pam.d/common-auth needs to be slightly modified from the configuration laid out in ADAddUbuntu in order to accommodate the automatic mounting of user disk space from the NetApp. In particular, the pam_mount.so module handles the initial user password entry and the pam_krb5.so is instructed to use the first password pass obtained from pam_mount.so. For certain login services, such as gdm, pam_mount.so will not use a password or ticket obtained by the pam_krb5.so module. Though this anomaly does not affect authentication, the session stack will hang during the gdm login process if pam_mount.so does not obtain the user password during the auth stack:
```
auth  required     pam_mount.so
auth  sufficient   pam_krb5.so use_first_pass forwardable ignore_root
auth  required     pam_unix.so use_first_pass nullok_secure
```
- /etc/pam.d/common-session needs to be slightly modified from the configuration laid out in ADAddUbuntu in order to accommodate the automatic mounting of user disk space from the NetApp:
```
session optional   pam_mount.so
session sufficient pam_krb5.so
session required   pam_unix.so
session optional   pam_foreground.so
```
- In this case the pam_mount.so is added to the top of the PAM stack as an option the other lines staying the same.
- The files /etc/pam.d/common-account and /etc/pam.d/common-password do not change from how they are set in ADAddUbuntu#Configuration_Of_The_Ubuntu_PAM.

Some Known Glitches

pam_mount.so will attempt to mount disk space for root user even though that is a local account on the Ubuntu host with not disk space on the NetApp server.
This mechanism will fail if a kerberos ticket cache called krb5_machine_realm_name is destroyed. This cache is not directly associated with the Ubuntu kerberos client and is not affected if a kdestroy command is issued. However, like any file it can be deleted by the root user. A system reboot will do the same thing since all kerberos tickets are cached in /tmp by default.
To fix this situation the rpc.gssd daemon must be restarted.
service gssd restart
A system reboot will have the same result.
The krb5_machine_realm_name cache will reappear after the first login with a user mounting kerberized NFS disk space.
We should come up with a mechanism to auto renew this cache on a daily basis.

Contents Of Common Kerberos Cache Files

Below is the contents of /tmp/krb5cc_0, created as a result of the kinit command when the kerberos client is initialized. The default principal is always the last one to be initialized by the system.

root@cseng10:/tmp#  klist -e -c /tmp/krb5cc_0
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: root/cseng10.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA

Valid starting     Expires            Service principal
12/01/10 10:57:44  12/01/10 20:57:44  krbtgt/STUDENT.CS.UWATERLOO.CA@STUDENT.CS.UWATERLOO.CA
        renew until 12/02/10 10:57:44, Etype (skey, tkt): ArcFour with HMAC/md5, ArcFour with HMAC/md5

Below is the contents of /tmp/krb5cc_machine_STUDENT.CS.UWATERLOO.CA which is created by the rpc.gssd daemon as part of authenticating mounts from the NetApp NFS server fs02.student.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA and potentially others. Note that the encryption type for addressing the CS-TEACHING domain is rc4-hmac (though for some systems it is DES-CBC-MD5) whereas the encryption type for addressing the NFS server is DES-CBC-CRC. Hence the requirement of maintaining at least two entries for the root principal in the Ubuntu host's krb5.keytab file - each representing a different but required mode of encryption.

root@cseng10:/tmp# klist -e -c /tmp/krb5cc_machine_STUDENT.CS.UWATERLOO.CA
Ticket cache: FILE:/tmp/krb5cc_machine_STUDENT.CS.UWATERLOO.CA
Default principal: root/cseng10.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA

Valid starting     Expires            Service principal
12/01/10 10:58:51  12/01/10 21:01:50  krbtgt/STUDENT.CS.UWATERLOO.CA@STUDENT.CS.UWATERLOO.CA
        renew until 12/02/10 10:58:51, Etype (skey, tkt): ArcFour with HMAC/md5, ArcFour with HMAC/md5 
12/01/10 10:58:53  12/01/10 20:58:53  nfs/fs02.student.cs.uwaterloo.ca@STUDENT.CS.UWATERLOO.CA
        renew until 12/02/10 10:58:51, Etype (skey, tkt): DES cbc mode with CRC-32, ArcFour with HMAC/md5

Topic revision: r28 - 2014-04-02 - ClaytonTucker

Information in this area is meant for use by CSCF staff and is not official documentation, but anybody who is interested is welcome to use it if they find it useful.