Karl’s Work Mac Setup

(This post is in beta)

So, what all do I install, and why?

  • The OpenSSH shipped in El Capitan is partially broken (GSSAPI Key Exchange doesn’t work).  So, we build our own, including GSSAPI authentication and GSSAPI Key Exchange.
  • MacPorts’ OpenSSH depends on MIT Kerberos, so we install that.
  • MacPorts’ remctl also depends on MIT Kerberos.
  • OpenSSL is needed because macOS doesn’t give you OpenSSL headers.
  • I also do a lot of Perl development.
  • Finally, Proxychains is useful for tunneling remctl through SSH.

This assumes that you’re running macOS 10.9 or later, and that you have admin access on your system.

NOTE: Some of these commands have you running them as root, via sudoOnly use sudo for these specific commands!  For example, you need sudo when you are installing stuff, but you do not need sudo for day-to-day things (like using Xcode, or using the software that you install).

  1. If you don’t have one, get an Apple account.
  2. Install Xcode from the App store.
    Although it’s possible to download the installer package, it’s easier to just install it from the App Store.  The only reason I’d install the package directly is if I was using some sort of system-management platform to push out software.
  3. Install the Command Line Tools that match your version of Xcode and macOS.  The download will be named “Command Line Tools (OS X 10.???) for Xcode ???”.  Make sure you install the correct command-line tools for your version of Mac OS X and your version of Xcode.
    Unfortunately, there is no App Store entry for this.
  4. Open a Terminal window, and view/accept the Xcode license.
    xcodebuild -license
  5. Download and run the MacPorts installer, for 10.11 El Capitan, 10.10 Yosemite, or 10.9 Mavericks.
  6. In a Terminal window, update the package lists and build ported sofware:
    sudo port selfupdate
    sudo port install kerberos5 remctl openssl proxychains-ng openssh +kerberos5 +gsskex -xauth
  7. The MacPorts installer updates your PATH, but for some reason it doesn’t update your MANPATH.  Add the following line to your .bash_profile:
    export MANPATH="/opt/local/share/man:$MANPATH"
  8. If you use Eclipse, which ships its own Git implementation, you might want to ensure that it uses the OpenSSH you’ve installed, so add this line to your .bash_profile:
    export GIT_SSH=/opt/local/bin/ssh
  9. Copy and customize one of the SSH configs from my Stanford web space.  Even though the MacPorts version of OpenSSH is used, it’s not acting as the system ssh daemon, so you need to match that with the SSH config you download.  In other words, if you’re running Mac OS X El Capitan, that’s the file you should download.  Name the file config, and put it into the .ssh folder in your home directory.
  10. Open the config file you just downloaded, and make some changes:
    1. Look for the line “Host *.stanford.edu“, and change that to cover your own company’s domain, or to cover the systems that you use.
    2. Look for the ProxyCommand line below “Host *.stanford.edu“, and change it to use your own group’s bastion host.  If you don’t have a group bastion host, and you want all connections to go direct, then comment out the ProxyCommand line.
    3. Somewhere above the “Host *.stanford.edu” line, insert a new section, named “Host YOUR_BASTION_HOST_HERE“.  In that section, add the configuration line “ProxyCommand none“.  This will tell SSH not to use the proxy command when connecting to your bastion host.  If you don’t do this, then every SSH connection, including connections to the bastion host, will cause a loop.  Also add the line “DynamicForward localhost:1080“, to open a local SOCKS proxy on the bastion host.  This is useful if you need to proxy other traffic (like HTTP traffic) through your bastion host, or if you will be using proxychains.
  11. Pull in BASH customizations:
    1. Copy all of my BASH files to your home directory, and then add a dot to the start of each filename.
    2. Make a symlink from .bash_stanford to either .bash_stanford_mit (if using MacPorts’ kerberos5 package) or .bash_stanford_heimdal (if using macOS’ built-in Kerberos).
    3. Open .bash_stanford with a text editor, and change the “BASTION=” line to use the name of your group’s bastion host.  Note that this is just the hostname.  If you are using a different domain name, then you’ll need to go through the entire shell script, making changes wherever you see “stanford.edu”.  If you bastion host uses something other than port 44, change the “bastion” alias at the bottom of the script.  You should also change (or remove) the command that is run on your bastion host automatically.
    4. Finally, update your .bash_profile file to run the .bashrc:
      . ~/.bashrc
    5. Close and re-open all Terminal windows, so that they pick up the changes made to your profile scripts.  To test out your bastion host connection, run the command bastion.
  12. To set up proxychains, create a file at ~/.proxychains/proxychains.conf, with the following content:
    tcp_read_time_out 15000
    tcp_connect_time_out 8000
    socks5 1080
  13. If you do a lot of Perl work, consider installing Perlbrew.  But, if you do, be sure to read my warning!  Also, consider if you want to enable Perlbrew in your .bash_profile before or after you set the PATH for MacPorts.  MacPorts can also install Perl, and Perl packages, so if you aren’t careful you’ll get weird clashing between your Perl and MacPorts’ Perl.
  14. If you do anything with PGP/GPG, including signing, encrypting, or authenticating, then install the GPG Suite.  In my case, I’m doing package signing (Debian packages) using a hardware key (on a Yubikey 4), so I need the functionality that the GPG Suite provides.

That’s it!  Most of the software (like Kerberos and OpenSSH) is documented elsewhere.  Here are my specific usage notes for other stuff.

To switch which Kerberos principal you’re using, use the aliases pag (to switch to regular principal), rootpag (root principal), adminpag (admin principal), or sunetpag (sunet principal).  The principal type appears in the command prompt, or the message (nc) to indicate that you don’t have a principal right now.

Unfortunately, I’ve noticed a tendancy for the OS to switch prinicpals in the background unexpectedly.  Send a noop command (that is, just press enter at shell prompt) to see if your active principal has changed.

My bash profile also provides a number of aliases to help with regular SSH stuff, including:

  • sshs HOSTNAME” will run “ssh -l YOUR_USERNAME HOSTNAME.stanford.edu
  • sshr HOSTNAME” will run “ssh -l root HOSTNAME.stanford.edu“.  Make sure you’re using your root principal before you use this alias!

Using proxychains is as simple as prefixing your command with proxychains.  For example:

proxychains remctl my-server command subcommand arg1

That routes all network connections through the SOCKS5 proxy at localhost port 1080, which should be set up by the DynamicForward line in your SSH configuration.  However, that also means proxychains will only work when you have a SSH connection to the bastion host!

Using an OpenPGP card/Yubikey with SSH authentication? Don’t forget about sshcontrol!

Here’s yet another thing that hit me at work today, and getting the answer involved annoying searching & testing, so here it is for you!

I use a Debian jessie workstation, and my SSH key lives on a Yubikey.  We don’t really use pubkey-based authentication in my area (we use Kerberos), so the first time I needed to do a pubkey-authenticated SSH connection was to clone one of my Github repositories.  When doing the clone, I got the mysterious message “Agent admitted failure to sign”.  GitHub talks about this error, but their article only talks about locally-stored keys (like id_rsa keys).  My private key lives on the Yubikey, so their article didn’t really work for me.

Doing some investigation, I could see that my SSH client was trying to authenticate, and it did see my Yubikey, but still the authentication was failing.  If you have this problem, there are two important things to check:

  1. Your GPG Agent must be running with SSH support.
  2. Your authentication key’s keygrip needs to be loaded in the .gnupg/sshcontrol file.

My problem was actually #2, but #1 is important to mention!

SSH Support in GPG Agent

The GPG Agent is able to act as an SSH Agent, speaking the ssh-agent protocol, as long as it has been told to do so.  In order for this to work, you need to specify –enable-ssh-support in the gpg-agent command line, or add “enable-ssh-support” in ~/.gnupg/gpg-agent.conf.

To confirm this, check your gpg-agent, and check for the appropriate environment variables:

blargh:~> env | grep SSH
blargh:~> ps xaw | grep gpg-agent
7631 ?        Ss     0:00 gpg-agent -c --daemon --enable-ssh-support
9425 pts/6    S+     0:00 grep gpg-agent

SSH_AGENT_PID contains the process ID of the ssh-agent, which is actually gpg-agent.  Also, SSH_AUTH_SOCK points to a special socket that gpg-agent creates, meant for connections speaking the ssh-agent protocol.

If SSH_AGENT_PID is pointing to a different process, you may have ssh-agent running somewhere else.  You should find out what is running ssh-agent, and stop it.  gpg-agent is able to manage locally-stored keys (like id_rsa), so I don’t think there’s a need to run ssh-agent, as long as you are running gpg-agent with SSH support.

If SSH_AGENT_PID isn’t specified at all, or you just added “enable-ssh-support” to ~/.gnupg/gpg-agent.conf, then you’ll need to restart gpg-agent.  In many cases, it’s easier to log out and log back in, so that everything picks up the new gpg-agent.

The sshcontrol File

This is what affected me.

The file at path ~/.gnupg/sshcontrol contains a list of “keygrips” for SSH that gpg-agent will use for ssh-agent authentication.  Even if gpg-agent is configured and can see your key, it will not work unless it’s listed in the sshcontrol file!  For local keys, running ssh-add will automatically add them to the sshcontrol file, but that doesn’t work for keys that live on an OpenPGP card.

I had gpg-agent running with SSH support, but gpg-agent does not automatically add keys that are already on an OpenPGP card, so it’s up to you.  What you need to do is…

  1. Extract the keygrip.
  2. Add the keygrip to the sshcontrol file.
  3. Restart gpg-agent.

Extracking the keygrip is pretty easy to do, using this string of commands:

blargh:~> echo 'scd learn --force' | gpg-connect-agent | grep KEYPAIRINFO
S KEYPAIRINFO 6E6D1675B29724F4D4A2250EDE5A0634B4E06884 OPENPGP.1

Each line of output contains the keygrip for one of the three keys on your OpenPGP card: The encryption key, the signing key, and the authentication key (which is the one we want).  The third line, for “OPENPGP.3”, contains the keygrip for the authentication key.

To update the sshcontrol file, simply add the keygrip to the file, on its own line.  So, in my case I would add the following line to ~/.gnupg/sshcontrol:


After you update the sshcontrol file, you need to restart gpg-agent.  Again, the safest way of doing that is to log out and log back in, so that everything picks up the new gpg-agent environment variables.

Identifying spammers in your shared web service (featuring Postfix, auditd, and iptables)

Over the last week, we’ve been having a problem with spam in our shared web service: Something was sending out lots of low-quality, easily-blockable spam, and the bouncebacks were filling up the Postfix queues in our outgoing email cluster.  The way we tracked down the spammer was interesting, so I’m writing it up here in case it’s of interest to anyone else!

Continue reading

Developing on El Capitan? Need OpenSSL? Install MacPorts!

I’m right now experiencing the joys of setting up a Perl development environment on Mac OS X 10.11 (El Capitan).  I’ve already talked about the weird linker warnings that appear when building Perl, and I’ve just come across another fun roadblock: A lack of OpenSSL header files.  This is not unintentional, and there is a solution.

Continue reading

Weird object file warnings building Perl on OS X 10.11 El Capitan

I just recently got a Mac laptop with Mac OS X 10.11 (El Capitan) installed, and one of the things I do in a new system is install a local Perl environment using perlbrew.  It allows me to install and upgrade modules without worrying that I am getting in the way of the system environment.  Problem is, when I built my first perlbrew environment on El Capitan, I saw some weird stuff happening in the build.

Continue reading

My First iOS Security Bug

About an hour ago, I opened bug #18403015 on Radar, Apple’s bug reporting web site.

This is not the first time I’ve reported a bug to Apple:  I recently opened a different bug (#18347673) because iOS has problems handling identify certificates, and I’ve opened other bugs as well in the past, for other stuff.  What makes this bug different is that I’m reporting it as a security bug.

Continue reading