Compiling SAMBA 4 file-sharing server on Mac OS-X High Sierra

The pain of getting SMB file sharing working on Mac.

Photo by Javi Hoffens on Unsplash

A little history

There’s quite a bit of history to running SMB file sharing on OSX. My understanding of the history is that SAMBA was originally MIT or BSD licensed — and Apple used it. Then SAMBA went GPL licensed and Apple had to develop their own SMB code. Oh dear, oh dear!

After trying Apple’s Server product on a Mac mini, and failing, I ended up using Homebrew to compile SAMBA.

This ran absolutely wonderfully, using SMB v1, all the while we had WinXP. The Mac mini, coupled with TimeMachine, proved a great choice.

Eventually, though Microsoft made it harder to enable SMB v1, then impossible. So, it was time to upgrade…

First attempt

After a week or so of fishing around, it became evident that SAMBA was not going to be a simple MacPorts or Homebrew experience.

I ended up installing VirtualBox, with Ubuntu Server doing the SAMBA part. By matching up the UIDs within the Ubuntu machine with the Mac mini’s UIDs of the data to be served it all worked pretty well. The shared area was living in the OSX domain (so that it was TimeMachined), and a VirtualBox shared folder provided a helpful gateway for the Ubuntu server to access and deliver the files.

But Ubuntu crashed about every week or two… and then the VM drive would be mounted in read-only mode… so ‘fsck’ to the rescue (mostly). Until ‘fsck’ didn’t work because the file system had been scrubbed so badly.

Ugh.

TimeMachine managed to save the day — I restored the VM disk and got Ubuntu working again. But it was time to ditch it. I do not have time to baby sit an unreliable server.

Compiling SAMBA

This link provided a really good starting point:

Compiling and Running Samba 4 Server on macOS

I strongly recommend you use that page as a reference as I’m going to highlight the additional steps.

Getting SAMBA to work on High Sierra was not very straightforward. Here are my observations to getting it to work in 2019.

Prepping Xcode

First, download Xcode from the App Store.

Then, install the command line bits:

xcode-select --install

xcode-select -switch /Applications/Xcode.app

Maybe there’s an easier way, but this got things working for me :-)

Downloading and Configuring SAMBA

This first part is the same as Will’s page…

Clone SAMBA:

git clone https://git.samba.org/samba.git

Change into the samba directory:

cd samba

Check out the latest version that works with this process:

git checkout samba-4.8.9

I just couldn’t get later versions building on OSX. Dependencies have changed, and it just got too messy.

Configure SAMBA to be installed at /opt/samba.

./configure \ 
       --prefix=/opt/samba \
       --without-ad-dc \
       --without-acl-support

(Be prepared for a long wait!)

As mentioned on Will Haley’s article, download the nss.diff patch ( this tiny diff/patch file as nss.diff )and apply it, running this command from the samba directory:

git apply ~/path/to/nss.diff

Additional Tweaks

I found that SAMBA would build and run from this point, but ended up with constant authentication errors when trying to connect to the SAMBA server. Nothing worked as it should.

After many unsuccessful attempts, I worked through the verbose debug logs and finally worked out that the error failed to get the unix group list was the key to the problem.

A little Googling (actually much more than a little!) came to https://trac.macports.org/ticket/35568 and the patch listed at https://attachments.samba.org/attachment.cgi?id=7727

The patch is different for SAMBA 4.8.9, but the key element of max_grp = 128 being the same.

The file that needs editing is in samba/source3/lib/system_smbd.c

Look for the function get groups_unix_user (line 204).

Make the following change:

#if defined(DARWINOS)
 int max_grp = 128;
#else
 int max_grp = MIN(128, groups_max());
#endif

Compiling SAMBA

Now that the patches have been applied you can build:

make && sudo make install

Configuring

I’d suggest you take a look at Will Haley’s article for a starting point.

Also ensure you setup the OSX users — because they have to exist — and use the smbpasswd tool to create the appropriate entries for SAMBA.

Debugging

Before you launch into creating a LaunchDaemon, do use the debug command:

sudo /opt/samba/sbin/smbd \
          --log-stdout \
          --debuglevel=10 \
          --foreground \
          --configfile=/opt/samba/etc/smb.conf

To kill the SAMBA process, use Activity Monitor, look for the smbd processes — and find the one with the lowest PID number. Then force-quit that process and you’ll get the command prompt back. (Or you could just close the terminal session).

LaunchDaemon

Once it’s all working nicely you can create a plist file in /Libray/LaunchDaemons to auto-start SAMBA on boot.

I just called mine homebrew.samba.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>KeepAlive</key>
    <true/>
    <key>Label</key>
    <string>homebrew.samba</string>
    <key>ProgramArguments</key>
    <array>
      <string>/opt/samba/sbin/smbd</string>
      <string>--configfile=/opt/samba/etc/smb.conf</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>WorkingDirectory</key>
    <string>/usr/local/var</string>
    <key>StandardErrorPath</key>
    <string>/usr/local/var/log/samba.log</string>
    <key>StandardOutPath</key>
    <string>/usr/local/var/log/samba.log</string>
  </dict>
</plist>

You’ll need to sudo chown root:wheel homebrew.samba.plist and then install:

sudo launchctl load /Library/LaunchDaemons/homebrew.samba.plist
sudo launchctl start homebrew.samba

Increasing File Limits on the server

Even though this initially worked, I found that some apps were failing to open files. An hour or two of researching uncovered file limits in Mavericks and beyond.

ulimit -a

Shows only 256 open files!

Following this link: https://blog.dekstroza.io/ulimit-shenanigans-on-osx-el-capitan/

I disabled csr (using Recovery mode Terminal), added the limits.maxfiles.plist and limits.maxproc.plist files. Finally, ulimit -a shows 524288 max files, and the apps now seem to function correctly.

Conclusion

I really wish someone else had fried their brain on my behalf… I apologise if I’ve missed out any steps. This was completed over a course of many, many weeks (interleaving with other ‘real’ work).

Now we have a very fast Mac mini SAMBA server that’s running entirely from a USB 3 Samsung T5 1TB storage that’s serving an office of Windows 10 and Mac machines.

Let me know if I’ve slipped up anywhere, or if this process has helped you out.

2019–06–20

Please let me know the comments about your successes and failures… for me this has been running wonderfully well for some weeks. Maybe someone could bring these tweaks to Homebrew?