Cephfs Clustered Share on Proxmox server

Assume that the AD server already exists and we create a clustered member

Install packages

apt install ctdb samba winbind krb5-config krb5-user libnss-winbind

Mount CephFS

First make sure that CephFS is mounted on the host.

ln -s /etc/pve/priv/ceph.client.admin.keyring /etc/ceph/ceph.client.admin.keyring

One-time mount of cephfs:

 mount -t ceph -o name=admin,noatime,nofail :/ /mnt/cephfs

Persistent mount of cephfs using systemd mounts:

vi /etc/systemd/system/mnt-cephfs.mount

[Unit]
Description=Mount CephFS
After=ceph-fuse.target pve-cluster.service
Wants=ceph-fuse.target pve-cluster.service

[Mount]
What=:/
Where=/mnt/cephfs
Type=ceph
Options=name=admin,noatime,nofail,_netdev

systemctl daemon-reload
systemctl enable --now mnt-cephfs.mount

Configure CTDB

This is minimal configuration file

root@pvetest2:~# cat /etc/ctdb/ctdb.conf
[logging]
        location = file:/var/log/ctdb.log

[cluster]
        recovery lock = /mnt/cephfs/ctdb/lock/ctdb.lck

Create cluster configuration

Note that mgmt in the public_addresses is the interface name (you might need to use ens18 or something like that) Here I use 10.255.202.50 as the cluster IP address, and 10.255.202.138 and 10.255.202.93 are my two nodes

cd /etc/ctdb
echo 10.255.202.50/24 mgmt >public_addresses
echo 10.255.202.138 >nodes
echo 10.255.202.93 >>nodes
mkdir -p /var/lib/ctdb/{volatile,persistent,state}
mkdir /var/run/ctdb
systemctl enable --now ctdb

You can check the status

root@pvetest2:/etc/ctdb# ctdb status
Number of nodes:2
pnn:0 10.255.202.138   UNHEALTHY (THIS NODE)
pnn:1 10.255.202.93    DISCONNECTED|UNHEALTHY|INACTIVE
Generation:1608962994
Size:1
hash:0 lmaster:0
Recovery mode:NORMAL (0)
Recovery master:0

Samba member setup

We will use a clustered configuration that will be shared between

Edit `/etc/samba/smb.conf`

[global]
      clustering = yes
      ctdb:registry.tdb = yes
      include = registry

The rest of the settings would be imported from a single configuration file and shared between the cluster members

Import configuration into registry

Edit /root/smb.conf on the first node (the first three lines need to be set to your names). idmap lines also need to be adjusted Here MIFE is my Netbios domain, AD.MIFE.CA is the Kerberos realm matching my AD domain ad.mife.ca. pve is the computer account that will be created in AD and shared between the samba nodes

[global]
   workgroup = MIFE
   realm = AD.MIFE.CA
   netbios name = pve

   security = ADS

   winbind refresh tickets = Yes
   vfs objects = acl_xattr
   map acl inherit = Yes
   store dos attributes = Yes

# server string is the equivalent of the NT Description field
   server string = %h server (Samba, Ubuntu)

#   dedicated keytab file = /etc/krb5.keytab
#  kerberos method = secrets and keytab

   winbind enum users = yes
   winbind enum groups = yes
   winbind use default domain = yes

   load printers = no
   printing = bsd
   printcap name = /dev/null
   disable spoolss = yes

   idmap config * : range = 3000-7999
   idmap config * : backend = tdb
   idmap config MIFE:unix_nss_info = no
   idmap config MIFE:backend = ad
#   idmap config MIFE:backend = rid
   idmap config MIFE:schema_mode = rfc2307
   idmap config MIFE:range = 10000-19999

   template shell = /bin/bash
   template homedir = /home/%U


[cephfs]
vfs objects = ceph
path = /
kernel share modes = no
#valid users = mfedotov root
writable = yes

[Users]
vfs objects = ceph
path = /Users
kernel share modes = no
writable = yes

Import that configuration into the shared Samba registry

net conf import /root/smb.conf

Check that the parameters imported:

testparm

Edit `/etc/nsswitch.conf` (add winbind as the last entry in passwd and group lines)

# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd:         files systemd winbind
group:          files systemd winbind
shadow:         files
gshadow:        files

hosts:          files dns
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

netgroup:       nis

Join the node to domain

net ads join -U Administrator

systemctl restart winbind
systemctl restart smbd
wbinfo --ping-dc

Configuration of additional nodes

Refer to section above to mount CephFS
Install packages

apt install ctdb samba winbind krb5-config krb5-user libnss-winbind

Edit /etc/ctdb/ctdb.conf

[logging]
        location = file:/var/log/ctdb.log

[cluster]
        recovery lock = /mnt/cephfs/ctdb/lock/ctdb.lck

cd /etc/ctdb
scp node1:/etc/ctdb/{ctdb.conf,public_addresses,nodes} /etc/ctdb
mkdir -p /var/lib/ctdb/{volatile,persistent,state}
mkdir /var/run/ctdb
systemctl enable --now ctdb

Edit /etc/samba/smb.conf

[global]
      clustering = yes
      ctdb:registry.tdb = yes
      include = registry

Modify /etc/nsswitch.conf
Make sure that the clustered parameters are inherited from clustered config

testparm

Restart services

systemctl restart winbind
systemctl restart smbd
wbinfo --ping-dc

Table of Contents