SSH Tunnel

Linux
GNOME Terminal, Konsole, Terminology, lx-terminal, guake (GNOME), Yakuake (KDE), ~~finalterm~~, alacritty, kitty, etc.
macOS
iTerm2 or macOS built-in Terminal.app
Windows
Cygwin (mintty), Windows Terminal, PuTTY, SuperPutty (puttycm successor), MobaXterm (good GUI for SSH TCP Forwarding)

SSH Dynamic Application Level Port Forwarding via SOCKS5

TL;DR

The process

Establish SSH connection to target host (sshd)
Dynamic application-level port forwarding by allocating a socket to listen to port on the local side, optionally bound to bind_address
When a connection is made to this port, the connection is forwarded over the secure tunnel, and the application protocol is then used to determine where to connect to from the remote host
by default the local port is bound in accordance with the GatewayPorts setting

ssh -g -qfnNT -D 1080 <SSH_SERVER>

ssh -g -qfnNT -D *:1080 <SSH_SERVER>

# local use only

ssh -g -qfnNT -D localhost:1080 <SSH_SERVER>

Options explained:

-q Quiet mode.
-T Disable pseudo-terminal allocation.
-f Requests ssh to go to background just before command execution. This is useful if ssh is going to ask for passwords or passphrases, but the user wants it in the background.
-n Redirects stdin from /dev/null (actually, prevents reading from stdin). This MUST be used when ssh is running in the background.
-N Do not execute remote command
-D specifies a local "dynamic" application level port forwarding (DynamicForward in ssh_config)
-g Allows remote hosts to connect to local forwarded ports, share the SOCKS5 proxy across the network (GatewayPorts in ssh_config)

In addition on a slow connection you can gain performance by enabling compression with the -C option. In addition, putty-tools package provides plink which is very useful, here is a simple sample which shares the tunnel with all other hosts in the LAN.

NOTE: SOCKS5 operates in layer 5 (session) of the OSI model. Above Transport layer, below presentation.

NOTE: If you feel the options are too hard to remember, just use a simple version:

ssh -g -D 1080 <SSH_SERVER>

Alternative - plink (Provided by putty on Arch, Red Hat based distros, putty-tools on Debian based)

Linux / macOS

plink -C -A -N user@ssh_host -D 0.0.0.0:port

Windows

plink.exe -C -A -N user@ssh_host -D 0.0.0.0:port

Options explained:

- -C
  Enable SSH Compression
- -A
  Enable agent forwarding
- -N
  Don't start a remove command or shell at all
- -D
  Dynamic port forwarding, same as -D for ssh

It is the same as ticking - Tunnels - Local ports accept connections from other hosts. Others in the same LAN will be able to share the ssh tunnel as well.

Note: -D 0.0.0.0 means to bind all interfaces (most likely eth{0..n}) so that other hosts in the same network will be able to use the tunnel.

How to browse via the secure tunnel (SOCKS5 proxy)?

Firefox Proxy Selector

~~Multiproxy Swith~~ ~~add-on~~

Note: set network.proxy.socks_remote_dns to true, use remote socks5 proxy (SSH Host) to lookup DNS instead of local DNS. This is normally used to avoid GFW DNS pollution/spoofing in China mainland. If NOT in PRC, it protects your privacy;-)

IMPORTANT NOTE: For DNS to be encrypted as well, consider running dnscrypt-proxy locally, or use DNS over TLS / DNS over HTTPS (e.g. Cloudflare's 1.1.1.1).

Chrome

Option 1: Proxy SwitchyOmega

Option 2: pass the parameter when launching chrome, create a launcher or shortcut

Linux in Terminal

google-chrome --proxy-server=socks5://localhost:port

macOS (doesn't work as expected...)

/Application/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --proxy-server=socks5://host:port

Windows in Command Line Prompt

%userprofile%\AppData\Local\Google\Chrome\Application\chrome.exe --proxy-server=socks5://localhost:port

Otherwise, use Proxy Switchy! extension to switch proxy settings.

Recommended: Extension: Proxy Switchy!

Use SSH Port Forwarding (Tunnel) To Access Server Behind Firewall / NAT Gateway

sshd <============================ reverse direction

GatewayPorts=yes ssh -R 22222:*:22 user@EC2

| |

SSH Client (Laptop) <=== Internet ===> EC2 Instance <=== Firewall ===> Linux Server (behind NAT)

jump server

Listen on *:22222

1. On the middle EC2 instance (Linux), set GatewayPorts yes in sshd_config /etc/ssh/sshd_config, restart sshd (systemctl restart sshd.service).
  RTFM: man sshd_config
2. Establish SSH connection from Linux Server (behind firewall) to the EC2 instance
  On the Linux Server behind firewall (reverse direction)
  # * indicates that the port should be available from all interfaces
  ssh -R 22222:*:22 user@<EC2 INSTANCE>
  # want to save step 3? bind to loopback only
  ssh -R 22222:localhost:22 user@<EC2 INSTANCE>
3. Block access to port 22222 on public facing network interfaces (elastic IP, this can be done by using security group, too, but at a different layer - Linux bridge + iptables in hypervisor dom0) by using iptables, e.g. iptables -A INPUT -p tcp -i <interface> --dport 22222 -j DROP
  NOTE: if the same trick has been used multiple times, occupying port 12222, 22222, etc
  iptables -A INPUT -p tcp -i <interface> -m multiport --dport 12222,22222 -j DROP
4. On the laptop, establish SSH connection to the EC2 instance (optionally acts as SOCKS server using -D which does dynamic application level port forwarding) ssh -D 1080 user@EC2_INSTANCE
  NOTE: applications can use localhost:1080 as SOCKS5 proxy, communication between laptop and the EC2 instance is encrypted and secure.
5. Then in the already established SSH session (on the EC2 instance), open a tmux session and connect to port 22222 bind to the loopback address: ssh -p 22222 <user on host behind NAT>@localhost
6. If public key or host-based authentication is configured, you should be now into the Linux Server through the middle man ;-D

NOTE: In this case the SSH client (laptop) CANNOT directly access the Linux server (SSH port 22) behind firewall (no access to edge router or gateway to configure port forwarding). However, the Linux server can ssh to the EC2 instance (via NAT gateway) in the reverse direction. That's why we use the -R option.

IMPORTANT: This works fine as a PoC or temporary workaround but NOT suitable as highly-available, secure, reliable and scalable long term solution. Overlay network solutions like nebula or tailscale and VPN (IPsec using strongSwan, or WireGuard) are better solutions. See below.

There are new open source scalable global overlay network implementation available to solve mesh network (network mesh?) use cases, like Nebula (good!) and others like: Zerotier, Tailscale (WireGuard powered).

NOTE: Tinc works well in some cases.

OpenSSH Tips & Tricks

Some tips:

- Customise ~/.ssh/config to suite your needs (be careful with storm - manage ssh like a boss, it helps when scripting or searching hosts but has a outstanding bug converting keywords to lowercase [1]

- Use ed25519 key over RSA

- OpenSSH 8.1 added support for FIDO/U2F (use your YubiKey or equivalent)

- ~/.ssh/authorized_keys can do a lot more than just adding trusted keys' public bit, e.g. it can restrict source IPs by using from="pattern-list" like from="192.168.55.0/24,172.16.55.0/24,*.terry.im" nice, isn't it ;-) RTFM for more.

- leverage ssh-copy-id

- ssh -vvv | ssh -G (troubleshooting from client side)

- /usr/sbin/sshd -p 2222 -f /path/to/sshd_config -D -ddd (troubleshooting sshd server side)

- Be careful with `UsePAM no`, make sure the user account has a usable password (P), instead of L (locked password).

- use AllowUsers / DenyUsers vs DenyGroups vs AllowGroups , mind the order

- know how to use ssh-add / ssh-keygen / ssh-agent / ssh-keyscan

- audit SSH config (ssh-audit / lynis), version control ssh_config / sshd_config properly if possible

- openssh + tmux magic (share troubleshooting session or pair programming)

Personal favourite tips/tricks:

- ssh -D (used to use this dynamic port forwarding, open a local Socks5 proxy to punch hole in firewall, encrypt traffic, it worked for a while against the infamous GFW, only a little while though)

- ssh -L | -R TCP forwarding

- ssh -X | -Y X11 forwarding (run X11 apps remotely and display it on X Server locally)

- More personal SSH tricks put together over the years, surprise to find that my person OpenSSH notes are 150+ pages in Google Docs, sorry can't put all in a comment... [2]

- RTFM works, OpenSSH is worth the time ;-)

[1]: https://github.com/emre/storm/issues/157

HN post: https://news.ycombinator.com/item?id=23038264

Default IPQoS changed since OpenSSH 7.8p1

IMPORTANT: OpenSSH 7.8p1 introduced a change to default IPQoS which may break the SSH connection for people who runes rolling Linux distribution (e.g. Arch, Gentoo, etc.) or on macOS using homebrew installed openssh (why wouldn't they?).

Cause:

Client and server use different IPQoS for interactive traffic and bulk.

Example:

- Client 8.1 using IPQoS af21 cs1
- Server 7.6p1 using IPQoS lowdelay throughput

Symptom: client is able to connect to the server successfully but the terminal will stop responding within 5~10s. On the server side, it resets the connection without detailed error information (hard to troubleshoot and identify root cause).

ssh(1)/sshd(8): the default IPQoS used by ssh/sshd has changed. They will now use DSCP AF21 for interactive traffic and CS1 for bulk.For a detailed rationale, please see the commit message: https://cvsweb.openbsd.org/src/usr.bin/ssh/readconf.c#rev1.284

Solution

add into your ~/.ssh/config

Host *

KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256

ChallengeResponseAuthentication no

PubkeyAuthentication yes

HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa

Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr

Macs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com

ServerAliveInterval 30

VisualhostKey yes

HashKnownHosts no