util/linuxfw: add additional nftable detection logic

We were previously using the netlink API to see if there are chains/rules that
already exist. This works fine in environments where there is either full
nftable support or no support at all. However, we have identified certain
environments which have partial nftable support and the only feasible way of
detecting such an environment is to try to create some of the chains that we
need.

This adds a check to create a dummy postrouting chain which is immediately
deleted. The goal of the check is to ensure we are able to use nftables and
that it won't error out later. This check is only done in the path where we
detected that the system has no preexisting nftable rules.

Updates #5621
Updates #8555
Updates #8762

Signed-off-by: Maisem Ali <maisem@tailscale.com>

util/linuxfw/nftables.go

@@ -105,6 +105,29 @@ func DebugNetfilter(logf logger.Logf) error {
 
 // detectNetfilter returns the number of nftables rules present in the system.
 func detectNetfilter() (int, error) {
+	// Frist try creating a dummy postrouting chain. Emperically, we have
+	// noticed that on some devices there is partial nftables support and the
+	// kernel rejects some chains that are valid on other devices. This is a
+	// workaround to detect that case.
+	//
+	// This specifically allows us to run in on GKE nodes using COS images which
+	// have partial nftables support (as of 2023-10-18). When we try to create a
+	// dummy postrouting chain, we get an error like:
+	//  add chain: conn.Receive: netlink receive: no such file or directory
+	nft, err := newNfTablesRunner(logger.Discard)
+	if err != nil {
+		return 0, FWModeNotSupportedError{
+			Mode: FirewallModeNfTables,
+			Err:  fmt.Errorf("cannot create nftables runner: %w", err),
+		}
+	}
+	if err := nft.createDummyPostroutingChains(); err != nil {
+		return 0, FWModeNotSupportedError{
+			Mode: FirewallModeNfTables,
+			Err:  err,
+		}
+	}
+
 	conn, err := nftables.New()
 	if err != nil {
 		return 0, FWModeNotSupportedError{
@@ -129,6 +152,7 @@ func detectNetfilter() (int, error) {
 		}
 		validRules += len(rules)
 	}
+
 	return validRules, nil
 }