"mount error(126): Required key not available" with CIFS & Kerberos - cifs

Question

My application needs to securely mount an Isilon share using CIFS and Kerberos. My mount attempt returns: Required key not available:
mount -t cifs //fileserver.example.com/client123/files
/mnt/client123/files -o username=acoder,password=XXXXXX,sec=krb5
Response:
mount error(126): Required key not available
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
Here are corresponding entries from /var/log/messages
Sep 16 16:33:49 clientbox kernel: CIFS VFS: Send error in SessSetup = -126
Sep 16 16:33:49 clientbox kernel: CIFS VFS: cifs_mount failed w/return code = -126
Background & Config
I added a keytab using:
/usr/bin/ktutil
addent -password -p acoder#EXAMPLE.COM -k 1 -e rc4-hmac
addent -password -p acoder#EXAMPLE.COM -k 1 -e aes256-cts
wkt /etc/krb5.keytab
Checked with klist -kte:
[acoder#clientbox]# klist -kte
Keytab name: FILE:/etc/krb5.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
1 09/16/15 16:24:32 acoder#EXAMPLE.COM (arcfour-hmac)
1 09/16/15 16:25:46 acoder#EXAMPLE.COM (aes256-cts-hmac-sha1-96)
Here's request-key.conf:
#OP TYPE DESCRIPTION CALLOUT INFO PROGRAM ARG1 ARG2 ARG3 ...
#====== ======= =============== =============== ===============================
create user debug:* negate /bin/keyctl negate %k 30 %S
create user debug:loop:* * |/bin/cat
create user debug:* * /usr/share/keyutils/request-key-debug.sh %k %d %c %S
negate * * * /bin/keyctl negate %k 30 %S
create cifs.spnego * * /usr/sbin/cifs.upcall %k
create dns_resolver * * /usr/sbin/cifs.upcall %k
Ticket cache:
# klist | grep "Ticket cache:"
Ticket cache: FILE:/tmp/krb5cc_0
What could be causing the "Required key not available" error?
EDIT:
I enabled debugging in CIFS, and attempted to mount the share again. Here's that output:
fs/cifs/cifsfs.c: Devname: //fileserver.example.com/client123/files flags: 0
fs/cifs/connect.c: prefix path /files
fs/cifs/connect.c: Username: acoder
fs/cifs/connect.c: file mode: 0x1ed dir mode: 0x1ed
fs/cifs/connect.c: CIFS VFS: in cifs_mount as Xid: 8 with uid: 0
fs/cifs/connect.c: UNC: \\fileserver.example.com/client123/files ip: 1.2.3.4
fs/cifs/connect.c: Socket created
fs/cifs/connect.c: sndbuf 19800 rcvbuf 87380 rcvtimeo 0x1b58
fs/cifs/connect.c: CIFS VFS: in cifs_get_smb_ses as Xid: 9 with uid: 0
fs/cifs/connect.c: Demultiplex PID: 22937
fs/cifs/connect.c: Existing smb sess not found
fs/cifs/cifssmb.c: secFlags 0x9
fs/cifs/cifssmb.c: Kerberos only mechanism, enable extended security
fs/cifs/transport.c: For smb_command 114
fs/cifs/transport.c: Sending smb: smb_len=78
fs/cifs/connect.c: RFC1002 header 0xbc
fs/cifs/transport.c: cifs_sync_mid_result: cmd=114 mid=1 state=4
fs/cifs/cifssmb.c: Dialect: 2
fs/cifs/asn1.c: OID len = 7 oid = 0x1 0x2 0x348 0x1bb92
fs/cifs/asn1.c: OID len = 6 oid = 0x1 0x3 0x5 0x1
fs/cifs/asn1.c: OID len = 7 oid = 0x1 0x2 0x348 0xbb92
fs/cifs/asn1.c: OID len = 10 oid = 0x1 0x3 0x6 0x1
fs/cifs/asn1.c: Need to call asn1_octets_decode() function for not_defined_in_RFC4178#please_ignore
fs/cifs/cifssmb.c: negprot rc 0
fs/cifs/connect.c: Security Mode: 0x3 Capabilities: 0x8000e2fc TimeAdjust: 0
fs/cifs/sess.c: sess setup type 4
fs/cifs/cifs_spnego.c: key description = ver=0x2;host=fileserver.example.com;ip4=1.2.3.4;sec=krb5;uid=0x0;creduid=0x0;user=acoder;pid=0xXXXXX
fs/cifs/sess.c: ssetup freeing small buf ffff8804359b02701
CIFS VFS: Send error in SessSetup = -126
fs/cifs/connect.c: CIFS VFS: leaving cifs_get_smb_ses (xid = 9) rc = -126
fs/cifs/connect.c: CIFS VFS: leaving cifs_mount (xid = 8) rc = -126
CIFS VFS: cifs_mount failed w/return code = -126

Answer 1

"Required key not available" means that cifs.upcall — run by the kernel in response to the mount request — was not able to get a Kerberos ticket for the CIFS server and from that generate the key needed for authenticating to the server (it would go in the kernel keyring of the client thread). cifs.upcall logs to daemon.debug; check those messages first. Usually that’s /var/log/daemon, but you may need to adjust your syslog configuration to include debug-level messages. On my system these look like so:
Aug 19 20:00:26 client.example.com cifs.upcall: [daemon.debug] key description: cifs.spnego;0;0;3f000000;ver=0x2;host=server.example.com;ip4=10.12.0.6;sec=krb5;uid=0x0;creduid=0x2cec;user=res;pid=0x1997
Aug 19 20:00:26 client.example.com cifs.upcall: [daemon.debug] ver=2
Aug 19 20:00:26 client.example.com cifs.upcall: [daemon.debug] host=server.example.com
Aug 19 20:00:26 client.example.com cifs.upcall: [daemon.debug] ip=10.12.0.6
Aug 19 20:00:26 client.example.com cifs.upcall: [daemon.debug] sec=1
Aug 19 20:00:26 client.example.com cifs.upcall: [daemon.debug] uid=0
Aug 19 20:00:26 client.example.com cifs.upcall: [daemon.debug] creduid=11500
Aug 19 20:00:26 client.example.com cifs.upcall: [daemon.debug] user=res
Aug 19 20:00:26 client.example.com cifs.upcall: [daemon.debug] pid=6551
Aug 19 20:00:26 client.example.com cifs.upcall: [daemon.debug] find_krb5_cc: considering /tmp/krb5cc_5601
Aug 19 20:00:26 client.example.com cifs.upcall: [daemon.debug] find_krb5_cc: /tmp/krb5cc_5601 is owned by 5601, not 11500
Aug 19 20:00:26 client.example.com cifs.upcall: [daemon.debug] find_krb5_cc: considering /tmp/krb5cc_5702
...
Ordinarily you’d use a mount command like this:
$ sudo mount -t cifs -o user=acoder,cruid=acoder,sec=krb5 ...
The cruid parameter tells cifs.upcall on behalf of which account this mount is occurring. It will look for Kerberos credential caches (“ccaches”) owned by this account (/tmp/krb5cc_*) first, to see if that account is logged in and has current credentials (e.g. if it’s a person and they’ve done kinit); you can see this in action in the log above where it is “considering” various ccaches. If that fails, it tries to kinit with a keytab. Earlier versions just use the system default keytab, which means the client principal’s keys must go there (usually /etc/krb5.keytab). Later versions have a -K flag you can use to deploy per-user keytabs for this, obviously better on a multi-user system. Note that you don’t need the password in the mount command; the keytab provides that information.
A separate thing to check, is that the Kerberos configuration on the client allows getting a CIFS ticket for the server to succeed at all. E.g.:
$ kinit acoder#EXAMPLE.COM
... type your password
$ klist
... see your TGT
$ kvno cifs/fileserver.example.com#EXAMPLE.COM
$ klist
... see CIFS ticket
Anyway there are many variables; start with the cifs.upcall debug log and let’s go from there.
(Note that the first answer is confused and wrong; you should ignore it. There is no need to join the client host to the realm, and its host principal is irrelevant here.)

Answer 2

Assuming that you've posted the full content from your krb5.keytab, it seems to be missing the host's key. In order to get a successful authentication on behalf of a user, your server needs both a user and a service ticket. The easiest way would be to join the server to the domain through sssd/samba (which would fill up your keytab with , and then add the user to the same keytab.
Anyway, there are many ways to do that, but you must ensure that your keytab (or keytabs) have both keys, so that it can get both tickets.