Go to file
Joe Tsai 7a77a2edf1
logtail: optimize JSON processing (#11671)
Changes made:

* Avoid "encoding/json" for JSON processing, and instead use
"github.com/go-json-experiment/json/jsontext".
Use jsontext.Value.IsValid for validation, which is much faster.
Use jsontext.AppendQuote instead of our own JSON escaping.

* In drainPending, use a different maxLen depending on lowMem.
In lowMem mode, it is better to perform multiple uploads
than it is to construct a large body that OOMs the process.

* In drainPending, if an error is encountered draining,
construct an error message in the logtail JSON format
rather than something that is invalid JSON.

* In appendTextOrJSONLocked, use jsontext.Decoder to check
whether the input is a valid JSON object. This is faster than
the previous approach of unmarshaling into map[string]any and
then re-marshaling that data structure.
This is especially beneficial for network flow logging,
which produces relatively large JSON objects.

* In appendTextOrJSONLocked, enforce maxSize on the input.
If too large, then we may end up in a situation where the logs
can never be uploaded because it exceeds the maximum body size
that the Tailscale logs service accepts.

* Use "tailscale.com/util/truncate" to properly truncate a string
on valid UTF-8 boundaries.

* In general, remove unnecessary spaces in JSON output.

Performance:

    name       old time/op    new time/op    delta
    WriteText     776ns ± 2%     596ns ± 1%   -23.24%  (p=0.000 n=10+10)
    WriteJSON     110µs ± 0%       9µs ± 0%   -91.77%  (p=0.000 n=8+8)

    name       old alloc/op   new alloc/op   delta
    WriteText      448B ± 0%        0B       -100.00%  (p=0.000 n=10+10)
    WriteJSON    37.9kB ± 0%     0.0kB ± 0%   -99.87%  (p=0.000 n=10+10)

    name       old allocs/op  new allocs/op  delta
    WriteText      1.00 ± 0%      0.00       -100.00%  (p=0.000 n=10+10)
    WriteJSON     1.08k ± 0%     0.00k ± 0%   -99.91%  (p=0.000 n=10+10)

For text payloads, this is 1.30x faster.
For JSON payloads, this is 12.2x faster.

Updates #cleanup
Updates tailscale/corp#18514

Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2024-04-12 12:05:36 -07:00
.bencher bencher: add config to suppress failures on benchmark regressions. 2021-10-01 16:16:02 -07:00
.github licenses: add gliderlabs/ssh license 2024-04-11 16:22:23 -07:00
appc appc: optimize dns response observation for large route tables 2024-02-07 14:11:41 -08:00
atomicfile various: add golangci-lint, fix issues (#7905) 2023-04-17 18:38:24 -04:00
chirp all: update copyright and license headers 2023-01-27 15:36:29 -08:00
client ipn/{ipnlocal,localapi}: add API to toggle use of exit node 2024-04-07 16:01:00 -07:00
clientupdate clientupdate, net/dns: fix some "tailsacle" typos 2024-04-03 21:08:25 -07:00
cmd logtail: optimize JSON processing (#11671) 2024-04-12 12:05:36 -07:00
control control/controlclient: delete old naclbox code, require ts2021 Noise 2024-04-03 09:17:27 -07:00
derp derp/derphttp: don't block in LocalAddr method 2024-04-08 10:57:05 -07:00
disco disco: correct noun for nacl box type in disco docs 2023-12-14 16:41:53 -08:00
docs docs/policy: update ADMX and ADML files with new Windows 1.62 syspolicies 2024-04-08 15:21:27 -05:00
doctor doctor/ethtool, ipn/ipnlocal: add ethtool bugreport check 2024-02-15 10:17:05 -05:00
drive drive: move normalizeShareName into pkg drive and make func public (#11638) 2024-04-05 11:43:13 -07:00
envknob envknob: ensure f is not nil before using it 2024-03-15 12:46:41 -04:00
health health: warn about reverse path filtering and exit nodes 2024-02-27 00:43:01 +00:00
hostinfo hostinfo: cache device model to speed up init 2024-04-02 09:09:43 -07:00
internal/tooldeps various: add golangci-lint, fix issues (#7905) 2023-04-17 18:38:24 -04:00
ipn ipn/{ipnlocal,localapi}: add API to toggle use of exit node 2024-04-07 16:01:00 -07:00
jsondb all: update copyright and license headers 2023-01-27 15:36:29 -08:00
k8s-operator Revert "cmd/{k8s-nameserver,k8s-operator},k8s-operator: add a kube nameserver, make operator deploy it (#11017)" (#11669) 2024-04-08 21:31:52 +01:00
kube cmd/k8s-operator,ipn/store/kubestore: patch secrets instead of updating 2023-08-29 13:24:05 -07:00
licenses licenses: update license notices 2024-04-12 10:47:24 -07:00
log all: use zstdframe where sensible (#11491) 2024-03-21 12:20:38 -07:00
logpolicy all: use zstdframe where sensible (#11491) 2024-03-21 12:20:38 -07:00
logtail logtail: optimize JSON processing (#11671) 2024-04-12 12:05:36 -07:00
metrics metrics: fix outdated docs on MultiLabelMap 2024-04-04 20:53:47 -07:00
net net/dns: unconditionally write NRPT rules to local settings 2024-04-12 11:56:26 -06:00
packages/deb go.mod: upgrade nfpm to v2 (#8786) 2023-08-03 13:00:45 -07:00
paths safesocket, paths: add Plan 9 support 2023-09-07 08:48:21 -07:00
portlist all: cleanup unused code, part 2 (#10670) 2023-12-21 17:40:03 -08:00
posture ci: run 'go vet' in golangci-lint; fix errors in tests 2023-12-07 15:08:28 -05:00
prober prober: export probe class and metrics from bandwidth prober 2024-04-08 12:02:58 +01:00
proxymap wgengine, proxymap: split out port mapping from Engine to new type 2023-09-17 20:06:43 +01:00
release version/mkversion: enforce synology versions within int32 range 2024-03-08 12:47:59 -05:00
safesocket version,cli,safesocket: detect non-sandboxed macOS GUI (#11369) 2024-03-14 14:28:06 -07:00
safeweb safeweb: move http.Serve for HTTP redirects into lib (#11592) 2024-04-02 12:04:24 -07:00
scripts scripts/installer.sh: add rpm GPG key import (#11686) 2024-04-10 16:58:35 -07:00
smallzstd all: update copyright and license headers 2023-01-27 15:36:29 -08:00
ssh/tailssh all: use reflect.TypeFor now available in Go 1.22 (#11078) 2024-02-08 17:34:22 -08:00
syncs syncs: add Swap method 2024-03-19 18:39:44 -04:00
tailcfg tailscale: switch tailfs to drive syntax for api and logs (#11625) 2024-04-04 13:07:58 -07:00
taildrop all: use new AppendEncode methods available in Go 1.22 (#11079) 2024-02-08 17:55:03 -08:00
tempfork net/tstun: use gaissmai/bart instead of tempfork/device 2024-03-25 12:31:14 -05:00
tka all: use new AppendEncode methods available in Go 1.22 (#11079) 2024-02-08 17:55:03 -08:00
tool tool/gocross: add android autoflags (#11465) 2024-03-19 16:08:20 -07:00
tsconst all: update copyright and license headers 2023-01-27 15:36:29 -08:00
tsd tailscale: update tailfs functions and vars to use drive naming (#11597) 2024-04-03 10:09:58 -07:00
tsnet tailscale: update tailfs functions and vars to use drive naming (#11597) 2024-04-03 10:09:58 -07:00
tstest control/controlclient: delete old naclbox code, require ts2021 Noise 2024-04-03 09:17:27 -07:00
tstime tstime/rate: implement Value.{Marshal,Unmarshal}JSON (#8481) 2024-01-16 13:48:34 -08:00
tsweb tsweb: switch to fastuuid for request ID generation 2024-04-09 14:05:20 -07:00
types ipn/ipnlocal, types/netmap: replace hasCapability with set lookup on NetworkMap 2024-03-22 15:30:46 -07:00
util util/fastuuid: add a more efficient uuid generator 2024-04-09 14:05:20 -07:00
version net/tsdial: partially fix "tailscale nc" (UserDial) on macOS 2024-04-07 16:04:32 -07:00
wf go.mod, all: move away from inet.af domain seized by Taliban 2024-02-13 19:21:09 -08:00
wgengine wgengine/magicsock: allow disco communication without known endpoints 2024-04-11 09:29:49 -07:00
words words: add a list of things you should yahoo! 2024-02-07 14:47:20 -08:00
.gitattributes .: add .gitattributes entry to use Go hunk-header driver 2021-12-03 17:56:02 -08:00
.gitignore client/web: always use new web client; remove old client 2023-08-28 11:11:16 -07:00
.golangci.yml ci: run 'go vet' in golangci-lint; fix errors in tests 2023-12-07 15:08:28 -05:00
ALPINE.txt ALPINE.txt,Dockerfile{.base},build_docker.sh: bump alpine (#10543) 2023-12-11 07:03:18 +00:00
api.md api.md: add missing backtick to GET searchpaths doc (#11459) 2024-03-19 11:31:03 -06:00
AUTHORS
build_dist.sh client/web: switch to using prebuilt web client assets 2023-09-19 10:09:54 -07:00
build_docker.sh Revert "cmd/{k8s-nameserver,k8s-operator},k8s-operator: add a kube nameserver, make operator deploy it (#11017)" (#11669) 2024-04-08 21:31:52 +01:00
CODE_OF_CONDUCT.md
CODEOWNERS CODEOWNERS: add the start of an owners file 2023-08-16 15:57:29 -07:00
Dockerfile Dockerfile: use Go 1.22 2024-02-07 18:10:15 -08:00
Dockerfile.base ALPINE.txt,Dockerfile{.base},build_docker.sh: bump alpine (#10543) 2023-12-11 07:03:18 +00:00
flake.lock flake.nix: build tailscale with go 1.22 2024-02-11 20:43:40 -08:00
flake.nix go.mod.sri: update SRI hash for go.mod changes 2024-04-06 15:12:24 -07:00
go.mod go.mod: bump github.com/gaissmai/bart 2024-03-26 16:06:52 -07:00
go.mod.sri go.mod.sri: update SRI hash for go.mod changes 2024-04-06 15:12:24 -07:00
go.sum go.mod: bump github.com/gaissmai/bart 2024-03-26 16:06:52 -07:00
go.toolchain.branch go.toolchain.rev: bump to Go 1.22.0 (#11055) 2024-02-07 14:57:57 -07:00
go.toolchain.rev go.toolchain.rev: bump to Go 1.22.2 2024-04-03 11:11:07 -07:00
gomod_test.go go.mod: add test that replace directives aren't added in oss 2023-09-29 12:31:52 -07:00
header.txt cmd/k8s-operator: operator can create subnetrouter (#9505) 2023-12-14 13:51:59 +00:00
LICENSE all: update tools that manage copyright headers 2023-01-27 15:36:29 -08:00
Makefile Revert "cmd/{k8s-nameserver,k8s-operator},k8s-operator: add a kube nameserver, make operator deploy it (#11017)" (#11669) 2024-04-08 21:31:52 +01:00
PATENTS
pull-toolchain.sh pull-toolchain.sh: don't run update-flake.sh 2023-02-09 15:04:01 -08:00
README.md go.mod, README.md: use Go 1.22 2024-02-07 18:10:15 -08:00
SECURITY.md
shell.nix go.mod.sri: update SRI hash for go.mod changes 2024-04-06 15:12:24 -07:00
staticcheck.conf all: cleanup unused code, part 2 (#10670) 2023-12-21 17:40:03 -08:00
update-flake.sh Code Improvements (#11311) 2024-03-08 15:24:36 -08:00
version_test.go go.mod,wgengine/netstack: bump gvisor 2024-01-19 18:23:53 -08:00
version-embed.go version: return correct Meta.MajorMinorPatch in non-dev builds 2023-02-10 13:00:44 -08:00
VERSION.txt VERSION.txt: this is v1.65.0 2024-04-11 14:20:42 -04:00

Tailscale

https://tailscale.com

Private WireGuard® networks made easy

Overview

This repository contains the majority of Tailscale's open source code. Notably, it includes the tailscaled daemon and the tailscale CLI tool. The tailscaled daemon runs on Linux, Windows, macOS, and to varying degrees on FreeBSD and OpenBSD. The Tailscale iOS and Android apps use this repo's code, but this repo doesn't contain the mobile GUI code.

Other Tailscale repos of note:

For background on which parts of Tailscale are open source and why, see https://tailscale.com/opensource/.

Using

We serve packages for a variety of distros and platforms at https://pkgs.tailscale.com.

Other clients

The macOS, iOS, and Windows clients use the code in this repository but additionally include small GUI wrappers. The GUI wrappers on non-open source platforms are themselves not open source.

Building

We always require the latest Go release, currently Go 1.22. (While we build releases with our Go fork, its use is not required.)

go install tailscale.com/cmd/tailscale{,d}

If you're packaging Tailscale for distribution, use build_dist.sh instead, to burn commit IDs and version info into the binaries:

./build_dist.sh tailscale.com/cmd/tailscale
./build_dist.sh tailscale.com/cmd/tailscaled

If your distro has conventions that preclude the use of build_dist.sh, please do the equivalent of what it does in your distro's way, so that bug reports contain useful version information.

Bugs

Please file any issues about this code or the hosted service on the issue tracker.

Contributing

PRs welcome! But please file bugs. Commit messages should reference bugs.

We require Developer Certificate of Origin Signed-off-by lines in commits.

See git log for our commit message style. It's basically the same as Go's style.

About Us

Tailscale is primarily developed by the people at https://github.com/orgs/tailscale/people. For other contributors, see:

WireGuard is a registered trademark of Jason A. Donenfeld.