Docker Engine 23.0 release notes

Note

From Docker Engine version 23.0.0, Buildx is distributed in a separate package: docker-buildx-plugin. In earlier versions, Buildx was included in the docker-ce-cli package. When you upgrade to this version of Docker Engine, make sure you update all packages. For example, on Ubuntu:

$ sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Refer to the Docker Engine installation instructions for your operating system for more details on upgrading Docker Engine.

This page describes the latest changes, additions, known issues, and fixes for Docker Engine version 23.0.

For more information about:

Starting with the 23.0.0 release, Docker Engine moves away from using CalVer versioning, and starts using the SemVer versioning format. Changing the version format is a stepping-stone towards Go module compatibility, but the repository doesn’t yet use Go modules, and still requires using a “+incompatible” version. Work continues towards Go module compatibility in a future release.

23.0.1

2023-02-09

For a full list of pull requests and changes in this release, refer to the relevant GitHub milestones:

Bug fixes and enhancements

  • Fix containers not starting if the kernel has AppArmor enabled, but apparmor_parser is not available. moby/moby#44942
  • Fix BuildKit-enabled builds with inline caching causing the daemon to crash. moby/moby#44944
  • Fix BuildKit improperly loading cached layers created by previous versions. moby/moby#44959
  • Fix an issue where ipvlan networks created prior to upgrading would prevent the daemon from starting. moby/moby#44937
  • Fix the overlay2 storage driver failing early in metacopy testing when initialized on an unsupported backing filesystem. moby/moby#44922
  • Fix exec exit events being misinterpreted as container exits under some runtimes, such as Kata Containers. moby/moby#44892
  • Improve the error message returned by the CLI when recieving a truncated JSON response caused by the API hanging up mid-request. docker/cli#4004
  • Fix an incorrect CLI exit code when attempting to execute a directory with a runc compiled using Go 1.20. docker/cli#4004
  • Fix mishandling the size argument to --device-write-bps as a path. docker/cli#4004

Packaging

23.0.0

2023-02-01

For a full list of pull requests and changes in this release, refer to the relevant GitHub milestones:

New

  • Set Buildx and BuildKit as the default builder on Linux. moby/moby#43992
    • Alias docker build to docker buildx build. docker/cli#3314
    • The legacy builder can still be used by explicitly setting DOCKER_BUILDKIT=0.
    • There are differences in how BuildKit and the legacy builder handle multi-stage builds. For more information, see Multi-stage builds.
  • Add support for pulling zstd compressed layers. moby/moby#41759, moby/moby#42862
  • Add support for alternate OCI runtimes on Linux, compatible with the containerd runtime v2 API. moby/moby#43887, moby/moby#43993
  • Add support for the containerd runhcs shim on Windows (off by default). moby/moby#42089
  • Add dockerd --validate to check the daemon JSON config and exit. moby/moby#42393
  • Add the ability to configure the daemon’s HTTP proxy via flags or JSON config. moby/moby#42835
  • Add support for RFC 3021 point-to-point networks (IPv4 /31s) and single hosts (IPv4 /32s). For networks with two or fewer addresses, IPAM won’t reserve a network and broadcast address. moby/moby#42626
  • Add support for setting ipvlan_flag and using the l3s ipvlan_mode in the ipvlan network driver. moby/moby#42542
  • Add support for displaying the value of the metacopy option for the overlay2 storage driver. moby/moby#43557
  • Add support for describing Windows devices using the syntax IDType://ID. moby/moby#43368
  • Add RootlessKit, slirp4netns, and VPNKit version reporting. moby/moby#42330
  • Add experimental support for SwarmKit cluster volumes (CSI). moby/moby#41982
  • Add support for SwarmKit jobs in docker stack deploy. docker/cli#2907
  • Add the docker stack config command to output the merged and interpolated config files as utilized by stack deploy. docker/cli#3544
  • Add a new docker context show command that prints the name of the current context. docker/cli#3567
  • Add the --format=json shorthand variant of --format="{{ json . }}" to all commands supporting the --format flag. docker/cli#2936
  • Add a --quiet option to docker create and docker run commands to suppress output when pulling an image. docker/cli#3377
  • Add a --force option to docker network rm subcommand. Causes CLI to return a 0 exit code even if the network doesn’t exist. Has no effect on the server-side procedure for removing a network. docker/cli#3547
  • Add a --signal option to docker stop and docker restart. docker/cli#3614
  • Add a -v/--version flag to docker-proxy. moby/moby#44703
  • Plugins are now discovered in well-known user-level paths when the daemon is running in rootless mode. moby/moby#44778
  • The daemon now handles common alternate JSON encodings in the JSON configuration file gracefully, and reports useful errors. moby/moby#44777, moby/moby#44832
    • UTF-8 with a byte order mark is accepted.
    • UTF-16 with a byte order mark is accepted.
    • Invalid UTF-8 is reported early and with a comprehensible error message.
  • Allow use of STOPSIGNAL via docker commit. moby/moby#43369
  • Add a new option to the awslogs log driver to allow skipping log stream creation in CloudWatch. moby/moby#42132
  • Add a new option to the awslogs log driver to specify the log format that’s sent to CloudWatch. moby/moby#42838
  • Add a new option to the fluentd log driver to set the reconnection interval. moby/moby#43100
  • Add new options-setters to the Go API client: WithTLSClientConfigFromEnv(), WithHostFromEnv(), and WithVersionFromEnv(). moby/moby#42224
  • Add generation of shell command completion through a docker completion subcommand. docker/cli#3429
  • API: Add a Swarm header to GET /_ping and HEAD /_ping, allowing single-request detection of Swarm support. moby/moby#42064
  • API: Add a signal parameter to POST /containers/{id}/stop and POST /containers/{id}/restart to set the signal used. moby/moby#43206
  • API: Add a CreateMountPoint parameter to POST /containers/create. moby/moby#43484
  • API: Add a shared-size parameter to GET /images/json to enable shared-size computation of images. moby/moby#42531
  • API: Add a type parameter to GET /system/df, to control what object types to are considered when computing disk usage. moby/moby#42559
  • systemd: Use a systemd-managed containerd instead of a daemon-managed containerd. moby/moby#42373
  • systemd: Start docker.service after time-set.target. moby/moby#43107

Removed

Deprecated

  • Require Windows Server RS5 / LTSC 2019 (build 17763) as the minimum to run the daemon. moby/moby#43254
  • Deprecate BuilderSize on API version >= 1.42. moby/moby#42608
  • Deprecate BuildCache.Parent in favor of the newly introduced BuildCache.Parents on API version >= 1.42. moby/moby#43908
  • Deprecate pkg/urlutil, moving the implementation to builder/remotecontext/urlutil. moby/moby#43477

Upgrades

Security

  • Change permissions on container hostconfig.json files to 0600 (was 0644). moby/moby#41620
  • Fix --seccomp-profile not accepting unconfined and renamed the default seccomp profile to builtin. moby/moby#42481
  • Always build with seccomp support, and remove the seccomp build tag. moby/moby#42501
  • Add seccomp support on riscv64. moby/moby#43553
  • Add support for setting flags passed to seccomp(2) in seccomp profiles. moby/moby#42648
  • Refactor seccomp types to reuse runtime-spec, and add support for ErrnoRet. moby/moby#42005
  • Add support for DefaultErrnoRet in seccomp profiles. moby/moby#42604
  • Add an explicit DefaultErrnoRet field to the default seccomp profile, with no behavior change. moby/moby#42649
  • Block socket with AF_VSOCK in the default seccomp profile. moby/moby#44563
  • Re-enable process_vm_readv and process_vm_writev in the default seccomp profile. moby/moby#42083
  • Add syscalls related to PKU to the default seccomp profile. moby/moby#43812
  • Allow clock_settime64 with CAP_SYS_TIME. moby/moby#43775
  • Allow bpf with CAP_BPF and perf_event_open with CAP_PERFMON. moby/moby#43988
  • Explicitly set the clone3 syscall to return ENOSYS in the default seccomp profile, in order to ensure glibc will correctly fallback to using clone. moby/moby#42681

Bug fixes and enhancements

  • Promote overlay2 to be the default storage driver (btrfs and zfs are now opt-in). moby/moby#42661
  • Add a loading spinner to the docker cp command. docker/cli#2708
  • Deprecate the ElectAuthServer function, and made it return the default registry without calling the GET /info API endpoint. docker/cli#2819
  • Progress bars are no longer reversed when rolling back Swarm services. docker/cli#2940
  • Use net.JoinHostPort() to fix formatting with IPv6 addresses. docker/cli#2972
  • CLI error messages are now printed to stderr. docker/cli#3044
  • Improve performance of docker info if a custom --format is used that only uses local information. With this change, the CLI only uses the daemon API if it detects that information from the daemon is needed. docker/cli#3179
  • Remove the default value from the --stop-signal flag, as it may not reflect the actual default used by the daemon. docker/cli#3245
  • Add Compose schema 3.10 to docker stack; allow omitting the version field (resulting in latest). docker/cli#3257
  • Compose version 3 is now equivalent to 3.x (latest) in docker stack. docker/cli#3445
  • Fix <Ctrl-c> hanging on Windows to exit after running a container in non-interactive mode. docker/cli#3302
  • Add relative source paths to the run command in the -v/--volume and -m/--mount flags. docker/cli#3469
  • docker exec -t now sets the console size for the executed process immediately when it’s created. docker/cli#3627
  • Update the pretty-print format of docker info to provide more details on installed plugins. docker/cli#3645
  • Print warning messages for the docker context list and docker context use commands when the context is overridden by the environment. docker/cli#3668
  • Add a custom aliases annotation that can be used to print all available aliases for a command. docker/cli#3694
  • The CLI no longer creates or updates the CLI configuration file when running docker context use and selecting the current context. docker/cli#3721
  • Non-existing contexts are now ignored when running docker context rm --force. docker/cli#3791
  • Add the ability to override integers to 0 in Compose files. docker/cli#3812
  • SIGINT (<Ctrl-c>) now passes through to running containers instead of causing the CLI to exit. docker/cli#3849
  • Improve docker port CONTAINER UX by sorting ports before printing. docker/cli#3892
  • API: GET /containers/{id}/logs and POST /containers/{id}/attach now report which raw-stream format is in use using the Content-type response header on API version >= 1.42. moby/moby#39812
  • Set default sandbox size for Windows layers to 127GB, and ensure that the --storage-opts flag applies to all storage on Windows. moby/moby#41636
  • Remove the plugin section from the containerd configuration file (/var/run/docker/containerd/containerd.toml). moby/moby#41675
  • Reject null manifests during tar import. moby/moby#41842
  • Add shim config for custom runtimes for plugins. moby/moby#41854
  • Container health checks now resume when the daemon is restarted. moby/moby#41935
  • Quota is no longer disabled on cleanup of the btrfs driver. moby/moby#42273
  • Host devices that are accessible can now be mounted in --privileged rootless containers. moby/moby#42638
  • Fix incorrect handling of **/foo recursive wildcard directory patterns in .dockerignore. moby/moby#42676
  • Extend docker import --platform to allow marking an imported image as a foreign architecture. moby/moby#43103
  • Validation of CPU real-time options is now performed when the daemon starts instead of performing validations for each individual container, allowing startup to fail early. moby/moby#43131
  • Freeze the namesgenerator package against new additions. Users will have to be satisfied with the existing 25359 adjective-name combinations. moby/moby#43210
  • API: containers/{id}/attach/ws only to streams according by stdin, stdout and stderr parameters on API version >= 1.42. moby/moby#43322
  • Fix UDP traffic in containers not working after the container is restarted under sustained traffic. moby/moby#43409
  • Add support for pulling images with custom amd64 micro-architecture feature levels as supported by the latest versions of Go, GCC, LLVM, and other compiler tools. moby/moby#43434
  • Improve validation of invalid JSON requests in the API. moby/moby#43463
  • Mitigate the impact of slow exec starts on health checks. Check timeout now only applies to the duration that the health check command is running. The time it takes to start the command no longer counts against the timeout. moby/moby#43480
  • Console tty size is set immediately on creation. moby/moby#43593, moby/moby#43622
  • Fix overlay2 mounts not being cleaned up after failed container starts, or daemon shutdown. moby/moby#43659
  • Match manifest list resolution with containerd. moby/moby#43675
  • Skip use of firewalld for networking when the daemon is running in rootless mode. moby/moby#43813
  • Custom NAT networks are now re-created after daemon restart if missing on Windows. moby/moby#43858
  • Fix terminating the container health-check process when it times out. moby/moby#43994
  • Fix live-restore with restart policies and volume refs. moby/moby#44237
  • API: Only anonymous volumes now pruned by default on API version >= v1.42. Pass the filter all=true to prune named volumes in addition to anonymous. moby/moby#44259
  • API: Support concurrent calls on the GET /system/df endpoint. moby/moby#42715
  • Improve the reliability of the daemon dumping the stack and exits with code 2 when sent a SIGQUIT. moby/moby#44831
  • Improve the reliability of docker logs -f on Windows, and prevent newlines from being dropped in the local log driver. moby/moby#43294
  • Fix a rare deadlock in the daemon caused by buffering of container logs. moby/moby#44856
  • Improve error handling in misc filesystem operations so that the daemon can start on a overlayfs backing filesystem. moby/moby#44834
  • Fix an issue where --ipc=host wasn’t handled correctly when the daemon is running in rootless mode. moby/moby#44863
  • Fix a long-standing set of issues where stale conntrack entries caused incorrect routing of UDP traffic for containers. moby/moby#44752
  • Fix half-registered containers being listed in the API, as well as a nil pointer de-reference and panic caused by using a partially registered container in API calls. moby/moby#44633
  • Fix a failure to create the DOCKER-USER ip6tables chain. moby/moby#44845
  • Fix a failure to clean up iptables rules when the ip6tables command isn’t available. moby/moby#44727
  • Fix an issue where some iptables NAT rules weren’t cleaned up after enabling the userland proxy. moby/moby#44811
  • Fix a potentially leaked process in rare situations where cleaning up a failed attempt to start a container was mishandled. moby/moby#44400
  • Fix the CreatedAt time of a volume reflecting initialization and not creation. moby/moby#44725
  • Fix an issue where the CLI incorrectly reported an incompatible server instead of an unreachable server in some commands. docker/cli#3901, docker/cli#3904
  • Fix broken completion of volumes in Zsh. docker/cli#2998
  • Improve output of docker context when an invalid context is present. docker/cli#3847
  • Remove ANSI decoration of CLI help annotations when the output isn’t a TTY, and added a newline for readability. docker/cli#3973
  • Add docker container remove as an alias for docker container rm. docker/cli#3986

Known issues

apparmor_parser (tracking issue)

Some Debian users have reported issues with containers failing to start after upgrading to the 23.0 branch. The error message indicates that the issue is due to a missing apparmor_parser binary:

Error response from daemon: AppArmor enabled on system but the docker-default profile could not be loaded: running `apparmor_parser apparmor_parser --version` failed with output:
error: exec: "apparmor_parser": executable file not found in $PATH
Error: failed to start containers: somecontainer

The workaround to this issue is to install the apparmor package manually:

apt-get install apparmor

BuildKit inline cache (tracking issue)

Attempting to build an image with BuildKit’s inline cache feature (e.g. docker build --build-arg BUILDKIT_INLINE_CACHE=1 ., docker buildx build --cache-to type=inline .) will result in the daemon unexpectedly exiting:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x147ff00]

goroutine 693 [running]:
github.com/docker/docker/vendor/github.com/moby/buildkit/cache.computeBlobChain.func4.1({0x245cca8, 0x4001394960})
        /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/blobs.go:206 +0xc90
github.com/docker/docker/vendor/github.com/moby/buildkit/util/flightcontrol.(*call).run(0x40013c2240)
        /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/flightcontrol/flightcontrol.go:121 +0x64
sync.(*Once).doSlow(0x0?, 0x4001328240?)
        /usr/local/go/src/sync/once.go:74 +0x100
sync.(*Once).Do(0x4001328240?, 0x0?)
        /usr/local/go/src/sync/once.go:65 +0x24
created by github.com/docker/docker/vendor/github.com/moby/buildkit/util/flightcontrol.(*call).wait

The daemon will restart if configured to do so (e.g. via systemd) after such a crash. The only available mitigation in this release is to avoid performing builds with the inline cache feature enabled.

BuildKit with warm cache (tracking issue)

If an image was built with BuildKit on a previous version of the daemon, and is built with a 23.0 daemon, previously cached layers will not be restored correctly. The image may appear to build correctly if no lines are changed in the Dockerfile; however, if partial cache invalidation occurs due to changing some lines in the Dockerfile, the still valid and previously cached layers will not be loaded correctly.

This most often presents as files that should be present in the image not being present in a RUN stage, or any other stage that references files, after changing some lines in the Dockerfile:

[+] Building 0.4s (6/6) FINISHED
 => [internal] load build definition from Dockerfile
 => => transferring dockerfile: 102B
 => [internal] load .dockerignore
 => => transferring context: 2B
 => [internal] load metadata for docker.io/library/node:18-alpine
 => [base 1/2] FROM docker.io/library/node:18-alpine@sha256:bc329c7332cffc30c2d4801e38df03cbfa8dcbae2a7a52a449db104794f168a3
 => CACHED [base 2/2] WORKDIR /app
 => ERROR [stage-1 1/1] RUN uname -a
------
 > [stage-1 1/1] RUN uname -a:
#0 0.138 runc run failed: unable to start container process: exec: "/bin/sh": stat /bin/sh: no such file or directory
------
Dockerfile:5
--------------------
   3 |
   4 |     FROM base
   5 | >>> RUN uname -a
   6 |
--------------------
ERROR: failed to solve: process "/bin/sh -c uname -a" did not complete successfully: exit code: 1

To mitigate this, the previous build cache must be discarded. docker builder prune -a will completely empty the build cache, and allow the affected builds to proceed again by removing the mishandled cache layers.

ipvlan networks (tracking issue)

When upgrading to the 23.0 branch, the existence of any ipvlan networks will prevent the daemon from starting:

panic: interface conversion: interface {} is nil, not string

goroutine 1 [running]:
github.com/docker/docker/libnetwork/drivers/ipvlan.(*configuration).UnmarshalJSON(0x40011533b0, {0x400069c2d0, 0xef, 0xef})
        /go/src/github.com/docker/docker/libnetwork/drivers/ipvlan/ipvlan_store.go:196 +0x414
encoding/json.(*decodeState).object(0x4001153440, {0x5597157640?, 0x40011533b0?, 0x559524115c?})
        /usr/local/go/src/encoding/json/decode.go:613 +0x650
encoding/json.(*decodeState).value(0x4001153440, {0x5597157640?, 0x40011533b0?, 0x559524005c?})
        /usr/local/go/src/encoding/json/decode.go:374 +0x40
encoding/json.(*decodeState).unmarshal(0x4001153440, {0x5597157640?, 0x40011533b0?})
        /usr/local/go/src/encoding/json/decode.go:181 +0x204
encoding/json.Unmarshal({0x400069c2d0, 0xef, 0xef}, {0x5597157640, 0x40011533b0})
        /usr/local/go/src/encoding/json/decode.go:108 +0xf4
github.com/docker/docker/libnetwork/drivers/ipvlan.(*configuration).SetValue(0x4000d18050?, {0x400069c2d0?, 0x23?, 0x23?})
        /go/src/github.com/docker/docker/libnetwork/drivers/ipvlan/ipvlan_store.go:230 +0x38

To mitigate this, affected users can downgrade and remove the network, then upgrade again. Alternatively, the entire network store can be removed, and networks can be recreated after the upgrade. The network store is located at /var/lib/docker/network/files/local-kv.db. If the daemon is using an alternate --data-root, substitute /var/lib/docker for the alternate path.

Kata Containers (tracking issue)

The 23.0 branch brings support for alternate containerd shims, such as io.containerd.runsc.v1 (gVisor) and io.containerd.kata.v2 (Kata Containers).

When using the Kata Containers runtime, exiting an exec session stops the running container, and hangs the connected CLI if a TTY was opened. There is no mitigation at this time beyond avoiding execing into containers running on the Kata runtime.

The root cause of this issue is a long-standing bug in Moby. This will be resolved in a future release. Be advised that support for alternate OCI runtimes is a new feature and that similar issues may be discovered as more users start exercising this functionality.