Only do the type assertion to *net.UDPAddr when addr is non-nil. This prevents a bunch of log spam during tests.