How to Fix a Broken Pipe

Quick tale of fixing an ssh connection from a macbook to a remote linux server.

Photo by Zulmaury Saavedra on Unsplash

I started to have problems connecting remotely via ssh to some of my servers. I didn’t bother to investigate except for checking my VPN connection settings because I can connect fine when I’m at the office. I found out I was actually using the wrong VPN server, so I made some changes there, but that didn’t fix the issue.

Yesterday I needed to connect to continue doing my work, and didn’t want to make a trip to the office just because of this problem. So I decided it was time to fix this problem.

After trying to connect with ssh username@xx.xxx.xxx.xx, I was getting this response:

packet_write_wait: Connection to xx.xxx.xxx.xx port 22: Broken pipe

After a search online, I saw a lot of entries such as this one talking about changing my .ssh/config for a single user, or /etc/ssh/ssh_config for all users on that system. They proposed adding this entry to that file:

Host *
   ServerAliveInterval 120

This tells the ssh connection to send a null request to the server every 120 seconds requesting a response. This is useful because some servers are configured to drop inactive ssh sessions. If 120 didn’t work, try decreasing to 60 or 30. Many users reported being able to fix their connection using this approach.

I tried it, but didn’t fix it. And I knew it had to be something else because I could connect from an ubuntu laptop, and I could connect a few months before from my macbook, but not anymore. There were some search results that even talked about changing the corresponding setting on the server by adding ClientAliveInterval to the ssh server configuration.

So I tried a new online search making sure I specified the literal “Broken pipe” message above. And I found a new result that offered a different solution; use the IPQoS option with a value of “throughput” like this:

ssh -o IPQoS=throughput username@xx.xxx.xxx.xx

Turns out that the source of the problem might have to do with how vmware handles network connections (my server is a vm that runs on vmware). But it also started to happen only after an upgrade to macOS Mojave (10.14.1, according to the last web result I linked above).

To avoid having to specify the -o IPQoS flag and option every time I connect, or every time I copy files for that matter, using scp, I put this option in my ~/.ssh/config file. So this is how my full file looks like now:

Host *
   AddKeysToAgent yes
   UseKeychain yes
   IdentityFile ~/.ssh/id_rsa
   IPQoS=throughput

And now I can connect without issues again! Hope this helps you if you run into this issue.

Other Resources

The most detailed explanation I found on this issue is here.

For more info on what the option IPQoS means and its possible values, do man ssh_config on a terminal. man is the command you can run on any unix/linux/macOS terminal to read the “manual” on a specific OS command.