Replacing Docker Desktop with minikube on Mac

On 2021-08-31 Docker announced that Docker Desktop would no longer be free for larger organizations. So we need a convenient replacement for Docker Desktop with minimal disruption — such as minikube, Hyperkit, or podman.

Copyright notice: This is an original article by xwi88, licensed under CC BY-NC 4.0. Commercial use is prohibited; please cite the source when reposting. Follow at https://github.com/xwi88

Terminology

• Docker Inc a US-based company that built open-source and closed-source software making it easier to develop, test, and run applications in containers.
• Docker Engine the core technology behind Docker. Open-source software that runs as a daemon on Linux and makes running containers on the Linux kernel possible. It owns the container lifecycle and the isolation of physical resources (compute, memory, storage) a container can access. It can run on physical or virtual machines, but only on a Linux kernel — i.e. on any Linux-flavored OS. This is important: the Docker engine runs only on Linux.
• Docker CLI CLI that developers usually use to interact with the docker engine. Includes the docker and docker-compose commands. Again, open source.
• Docker Desktop since Docker Engine runs only on Linux, developers on Windows and macOS can only run the engine after starting a Linux VM. That’s where Docker Desktop comes in. Docker Desktop is proprietary software that lets Windows/macOS developers use containers seamlessly in dev without managing VMs and all the details that entails (networking, virtualization, Linux knowledge, …). Docker Desktop is designed for use during software development; it has no place in production-like containers, where only Docker Engine is involved.

Requirements

• At least 2 CPUs
• At least 2 GB of free memory
• At least 20 GB of free disk space
• A network connection
• A container or VM manager, e.g. Docker, Hyperkit, Hyper-V, KVM, Parallels, podman, VirtualBox, or VMware Fusion/Workstation

Installation

If you’re sure you won’t run images in docker desktop anymore, remove all docker / docker desktop related config.

1
2
3
4
5

brew install docker
brew install docker-compose # optional, as needed

brew install hyperkit
brew install minikube

1
2

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64
sudo install minikube-darwin-amd64 /usr/local/bin/minikube

Verify the install

Run which minikube

1
2

brew unlink minikube
brew link minikube

Drivers

minikube needs a Linux VM to start on mac/windows; that’s the driver we configure. With --driver=docker you need the matching Docker Desktop installed to reuse its linux vm. Otherwise pick another driver. Supported drivers:

Linux

• Docker - container-based (preferred)
• KVM2 - VM-based (preferred)
• VirtualBox - VM
• None - bare-metal
• Podman - container (experimental)
• SSH - remote ssh

macOS

• Docker - VM + Container (preferred)
• Hyperkit - VM
• VirtualBox - VM
• Parallels - VM
• VMware Fusion - VM
• SSH - remote ssh

Windows

• Hyper-V - VM (preferred)
• Docker - VM + Container (preferred)
• VirtualBox - VM
• VMware Workstation - VM
• SSH - remote ssh

1
2
3
4
5
6
7
8

# need: install Docker 18.09 or higher
#       amd64 or arm64 system.

# Start a cluster using the docker driver:
minikube start --driver=docker

# To make docker the default driver:
minikube config set driver docker

1
2
3
4
5
6
7

# Requirements
#    Docker 20.10 or higher, see https://rootlesscontaine.rs/getting-started/docker/
#    Cgroup v2 delegation, see https://rootlesscontaine.rs/getting-started/common/cgroup2/

dockerd-rootless-setuptool.sh install -f
docker context use rootless
minikube start --driver=docker --container-runtime=containerd

More driver config & usage: minikube drivers

Config commands

minikube config SUBCOMMAND [flags] [options]

• minikube config help
• minikube config defaults PROPERTY_NAME [flags] list displays all valid default settings for PROPERTY_NAME
• minikube config view [flags] Display values currently set in the minikube config file.
• minikube config set PROPERTY_NAME PROPERTY_VALUE [flags] Sets an individual value in a minikube config file
• minikube config get PROPERTY_NAME [flags] Returns the value of PROPERTY_NAME from the minikube config file
• minikube config unset PROPERTY_NAME [flags] unsets PROPERTY_NAME from the minikube config file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

config modifies minikube config files using subcommands like "minikube config set driver kvm2"
Configurable fields:

 * driver
 * vm-driver
 * container-runtime
 * feature-gates
 * v
 * cpus
 * disk-size
 * host-only-cidr
 * memory
 * log_dir
 * kubernetes-version
 * iso-url
 * WantUpdateNotification
 * WantBetaUpdateNotification
 * ReminderWaitPeriodInHours
 * WantNoneDriverWarning
 * WantVirtualBoxDriverWarning
 * profile
 * bootstrapper
 * insecure-registry
 * hyperv-virtual-switch
 * disable-driver-mounts
 * cache
 * EmbedCerts
 * native-ssh

Available Commands:
  defaults    Lists all valid default values for PROPERTY_NAME
  get         Gets the value of PROPERTY_NAME from the minikube config file
  set         Sets an individual value in a minikube config file
  unset       unsets an individual value in a minikube config file
  view        Display values currently set in the minikube config file

Usage:
  minikube config SUBCOMMAND [flags] [options]

Use "minikube <command> --help" for more information about a given command.
Use "minikube options" for a list of global command-line options (applies to all commands).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

minikube config set driver hyperkit
minikube config set cpus 2
minikube config set memory 2000mb
minikube config set disk-size 20gb

# insecure-registry didn't take effect in testing; pass it at start time!
minikube config set insecure-registry https://docker.mirrors.ustc.edu.cn,https://reg-mirror.qiniu.com,https://mirror.ccs.tencentyun.com
# minikube config set kubernetes-version <>

# The name of the minikube VM being used. This can be set to allow having multiple instances of minikube independently. (default "minikube")
minikube config set profile

Quick start

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

minikube start # minikube start --container-runtime=docker
# Tell Docker CLI to talk to minikube's VM
eval $(minikube docker-env)
# Save IP to a hostname
# echo "`minikube ip` docker.local" | sudo tee -a /etc/hosts > /dev/null
# control panel, must start minikube with cluster model, without flag --no-kubernetes
minikube dashboard
# stop kubernetes related
minikube pause
# test
docker run hello-world

Advanced start

commands start

Start command

• After some config changes you need minikube delete then restart

Possible issues

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

😄  minikube v1.24.0 on Darwin 12.2.1
✨  Using the docker driver based on existing profile

⛔  Docker Desktop only has 1986MiB available, you may encounter application deployment failures.
💡  Suggestion:

    1. Click on "Docker for Desktop" menu icon
    2. Click "Preferences"
    3. Click "Resources"
    4. Increase "Memory" slider bar to 2.25 GB or higher
    5. Click "Apply & Restart"
📘  Documentation: https://docs.docker.com/docker-for-mac/#resources

❗  You cannot change the memory size for an existing minikube cluster. Please first delete the cluster.
👍  Starting minikube without Kubernetes minikube in cluster minikube

1
2
3

🔥  Deleting "minikube" in docker ...
🔥  Removing ~/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.

Post-start environment check

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

  minikube v1.24.0 on Darwin 10.15.7
    ▪ MINIKUBE_ACTIVE_DOCKERD=minikube
✨  Using the hyperkit driver based on user configuration
👍  Starting minikube without Kubernetes minikube in cluster minikube
🔥  Creating hyperkit VM (CPUs=2, Memory=2000MB, Disk=20480MB) ...
🏄  Done! minikube is ready without Kubernetes!
╭───────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                       │
│                       💡  Things to try without Kubernetes ...                        │
│                                                                                       │
│    - "minikube ssh" to SSH into minikube's node.                                      │
│    - "minikube docker-env" to point your docker-cli to the docker inside minikube.    │
│    - "minikube image" to build images without docker.                                 │
│                                                                                       │
╰───────────────────────────────────────────────────────────────────────────────────────╯

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

Kernel Version: 4.19.202
 Operating System: Buildroot 2021.02.4
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 1.894GiB
 Name: minikube
 ID: STNM:GBUS:PMHK:ASTU:BZHZ:WIEY:L6F3:YOMH:M432:S5XK:PAT5:IU2D
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
  provider=hyperkit
 Experimental: false
 Insecure Registries:
  reg-mirror.qiniu.com
  docker.mirrors.ustc.edu.cn
  mirror.ccs.tencentyun.com
  10.96.0.0/12
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

Kernel Version: 5.10.76-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 3
Total Memory: 4.083GiB
Name: docker-desktop

HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 Registry: https://index.docker.io/v1/
 Labels:
  Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://registry-1.docker.io/
  https://hub-mirror.c.163.com/
  https://mirror.baidubce.com/
  https://registry.cn-hangzhou.aliyuncs.com/
  https://docker.mirrors.ustc.edu.cn/
  https://mirror.ccs.tencentyun.com/
  https://registry.docker-cn.com/
  https://reg-mirror.qiniu.com/
  https://dockerhub.azk8s.cn/

Change image registries

Don’t make this change inside the minikube environment. Pass it at start: minikube start --image-mirror-country='cn' etc.; see minikube start or Advanced start

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

{
     "registry-mirrors": [
        "https://registry-1.docker.io",
        "https://hub-mirror.c.163.com",
        "https://mirror.baidubce.com",
        "https://registry.cn-hangzhou.aliyuncs.com",
        "http://f1361db2.m.daocloud.io",
        "https://docker.mirrors.ustc.edu.cn",
        "https://mirror.ccs.tencentyun.com",
        "https://registry.docker-cn.com",
        "https://reg-mirror.qiniu.com",
        "https://dockerhub.azk8s.cn"
  ]
}

Dashboard

minikube dashboard

Directory mounting

This will start the mount daemon and automatically mount files into minikube.

minikube mount <local directory>:<host directory>

Mapping/mount chain: host volumes<->docker desktop VM volumes<->docker container volumes

Driver default mount paths

Some hypervisors, have built-in host folder sharing. Driver mounts are reliable with good performance, but the paths are not predictable across operating systems or hypervisors:

These mounts can be disabled by passing –disable-driver-mounts to minikube start.

Running local apps

If you use directory mapping you must enable mounting, and mind the mount path — otherwise files or configs won’t be found.

Files or configs not found

Solution

Map/mount your working directory on the host:

• minikube mount $HOME/workspace:$HOME/workspace
• Mind the mapping path inside the minikube vm: the mount path must match the real path
• Watch directory permissions — mounting under your own user dir usually has no permission issues
• You can mount multiple directories

Start on boot

--mount --mount-string=$HOME/workspace:$HOME/workspace may have issues; mounting via minikube mount after start is recommended

Mount only the current project directory

• Pro: fewer mount dirs, less space
• Con: switching projects means re-mounting — tedious

Prefer mounting the whole working directory (mind the size) — no extra steps when switching projects

1
2
3
4
5
6
7
8

# --mount --mount-string=$HOME/workspace:$HOME/workspace

#minikube delete
minikube start --no-kubernetes --driver=hyperkit --cpus=2 --memory=2gb --disk-size=20gb \
--image-mirror-country=cn \
--registry-mirror=https://docker.mirrors.ustc.edu.cn,https://reg-mirror.qiniu.com,https://mirror.ccs.tencentyun.com
eval $(minikube docker-env)
minikube mount ${PWD}:${PWD}

Mount the whole working directory

1
2
3
4
5
6
7
8
9

# issues here {a lone --mount will mount: /Users:/minikube-host/; --mount-string mounts only the given dir, no --mount needed then — recommended}
# --mount --mount-string=$HOME/workspace:$HOME/workspace

#minikube delete
minikube start --no-kubernetes --driver=hyperkit --cpus=2 --memory=2gb --disk-size=20gb \
--image-mirror-country=cn \
--registry-mirror=https://docker.mirrors.ustc.edu.cn,https://reg-mirror.qiniu.com,https://mirror.ccs.tencentyun.com
eval $(minikube docker-env)
minikube mount ${HOME}/workspace:${HOME}/workspace

Mount the whole working directory and run in the background

1
2
3
4
5
6
7
8

# --mount --mount-string=$HOME/workspace:$HOME/workspace

#minikube delete
minikube start --no-kubernetes --driver=hyperkit --cpus=2 --memory=2gb --disk-size=40gb \
--image-mirror-country=cn \
--registry-mirror=https://docker.mirrors.ustc.edu.cn,https://reg-mirror.qiniu.com,https://mirror.ccs.tencentyun.com
eval $(minikube docker-env)
nohup minikube mount ${HOME}/workspace:${HOME}/workspace &

Caveats

• After minikube starts, run eval $(minikube docker-env)
• Mount directories after minikube starts — see Running local apps
• Pass --driver at start; if it’s docker, make sure the docker daemon is running
• If the mount flag is wrong, run minikube mount --kill and remove the dir from the minikube vm
• minikube delete — you don't need to run it every time

More

• Using minikube as Docker Desktop Replacement
• How to interact with registries
• minikube-image-build
• containerd

References

• run-docker-without-docker-desktop-on-macos
• minikube docs