Unexpected JSON Output

[darkn3rd]

I tried to use the latest released version with docker-machine, and I am getting this error. I am not deeply familiar with KVM or libvirt, so I am not sure how this works. I have libvirt working fine with vagrant and mini-kube (using their embedded kvm2 docker-machine driver) without any problems. I am not sure how I could debug further.

$ docker-machine -D create -d kvm myengine1
Docker Machine Version:  0.16.2, build bd45ab13
Found binary path at /usr/local/bin/docker-machine-driver-kvm
Launching plugin server for driver kvm
Plugin server listening at address 127.0.0.1:41571
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(flag-lookup) Calling .GetMachineName
(flag-lookup) Calling .DriverName
(flag-lookup) Calling .GetCreateFlags
Found binary path at /usr/local/bin/docker-machine-driver-kvm
Launching plugin server for driver kvm
Plugin server listening at address 127.0.0.1:45025
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(myengine1) Calling .GetMachineName
(myengine1) Calling .DriverName
(myengine1) Calling .GetCreateFlags
(myengine1) Calling .SetConfigFromFlags
(myengine1) DBG | SetConfigFromFlags called
Reading certificate data from /home/joaquin/.docker/machine/certs/ca.pem
Decoding PEM data...
Parsing certificate...
Reading certificate data from /home/joaquin/.docker/machine/certs/cert.pem
Decoding PEM data...
Parsing certificate...
Running pre-create checks...
(myengine1) Calling .PreCreateCheck
(myengine1) DBG | About to check libvirt version
(myengine1) DBG | Validating private network
(myengine1) DBG | Validating network default
(myengine1) Calling .GetConfigRaw
Creating machine...
(myengine1) Calling .Create
(myengine1) DBG | local Boot2Docker ISO version:  v19.03.12
(myengine1) Copying /home/joaquin/.docker/machine/cache/boot2docker.iso to /home/joaquin/.docker/machine/machines/myengine1/boot2docker.iso...
(myengine1) Creating SSH key...
(myengine1) DBG | Verifying executable bit set on /home/joaquin/.docker/machine/machines/myengine1
(myengine1) DBG | Setting executable bit set on /home/joaquin/.docker/machine/machines/myengine1
(myengine1) DBG | Verifying executable bit set on /home/joaquin/.docker/machine/machines
(myengine1) DBG | Verifying executable bit set on /home/joaquin/.docker/machine
(myengine1) DBG | Verifying executable bit set on /home/joaquin/.docker
(myengine1) DBG | Verifying executable bit set on /home/joaquin
(myengine1) DBG | Verifying executable bit set on /home
(myengine1) DBG | Creating VM data disk...
(myengine1) DBG | Creating 20000 MB hard disk image...
(myengine1) DBG | Defining VM...
(myengine1) DBG | Starting VM myengine1
(myengine1) DBG | GetIP called for myengine1
(myengine1) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(myengine1) Failed to decode dnsmasq lease status: unexpected end of JSON input
(myengine1) DBG | Unable to locate IP address for MAC 52:54:00:d2:f1:a0
(myengine1) DBG | Waiting for the VM to come up... 0
(myengine1) DBG | GetIP called for myengine1
(myengine1) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(myengine1) DBG | Unable to locate IP address for MAC 52:54:00:d2:f1:a0
(myengine1) DBG | Waiting for the VM to come up... 1
(myengine1) Failed to decode dnsmasq lease status: unexpected end of JSON input
(myengine1) DBG | GetIP called for myengine1
(myengine1) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(myengine1) Failed to decode dnsmasq lease status: unexpected end of JSON input
(myengine1) DBG | Unable to locate IP address for MAC 52:54:00:d2:f1:a0
(myengine1) DBG | Waiting for the VM to come up... 2
(myengine1) DBG | GetIP called for myengine1
(myengine1) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(myengine1) Failed to decode dnsmasq lease status: unexpected end of JSON input
(myengine1) DBG | Unable to locate IP address for MAC 52:54:00:d2:f1:a0
(myengine1) DBG | Waiting for the VM to come up... 3
(myengine1) DBG | GetIP called for myengine1
(myengine1) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(myengine1) Failed to decode dnsmasq lease status: unexpected end of JSON input
(myengine1) DBG | Unable to locate IP address for MAC 52:54:00:d2:f1:a0
(myengine1) DBG | Waiting for the VM to come up... 4
(myengine1) DBG | GetIP called for myengine1
(myengine1) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(myengine1) Failed to decode dnsmasq lease status: unexpected end of JSON input
(myengine1) DBG | Unable to locate IP address for MAC 52:54:00:d2:f1:a0
(myengine1) DBG | Waiting for the VM to come up... 5
(myengine1) DBG | GetIP called for myengine1
(myengine1) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(myengine1) Failed to decode dnsmasq lease status: unexpected end of JSON input
(myengine1) DBG | Unable to locate IP address for MAC 52:54:00:d2:f1:a0
(myengine1) DBG | Waiting for the VM to come up... 6
(myengine1) DBG | GetIP called for myengine1
(myengine1) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(myengine1) Failed to decode dnsmasq lease status: unexpected end of JSON input
(myengine1) DBG | Unable to locate IP address for MAC 52:54:00:d2:f1:a0
(myengine1) DBG | Waiting for the VM to come up... 7
(myengine1) DBG | GetIP called for myengine1
(myengine1) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(myengine1) Failed to decode dnsmasq lease status: unexpected end of JSON input
(myengine1) DBG | Unable to locate IP address for MAC 52:54:00:d2:f1:a0
(myengine1) DBG | Waiting for the VM to come up... 8
(myengine1) DBG | GetIP called for myengine1
(myengine1) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(myengine1) Failed to decode dnsmasq lease status: unexpected end of JSON input
(myengine1) DBG | Unable to locate IP address for MAC 52:54:00:d2:f1:a0
(myengine1) DBG | Waiting for the VM to come up... 9

There's no such file /var/lib/libvirt/dnsmasq/docker-machines.leases.

$ ls -la /var/lib/libvirt/dnsmasq
total 24
drwxr-xr-x 2 root root 4096 Oct 11 03:46 .
drwxr-xr-x 7 root root 4096 Oct 11 02:00 ..
-rw-r--r-- 1 root root    0 Oct 11 02:01 default.addnhosts
-rw------- 1 root root  619 Oct 11 02:01 default.conf
-rw-r--r-- 1 root root    0 Oct 11 02:01 default.hostsfile
-rw-r--r-- 1 root root    0 Oct 11 03:25 docker-machines.addnhosts
-rw------- 1 root root  688 Oct 11 03:25 docker-machines.conf
-rw-r--r-- 1 root root    0 Oct 11 03:25 docker-machines.hostsfile
-rw-r--r-- 1 root root   85 Oct 11 03:46 virbr0.macs
-rw-r--r-- 1 root root    0 Oct 11 01:02 virbr0.status
-rw-r--r-- 1 root root   85 Oct 11 03:46 virbr1.macs
-rw-r--r-- 1 root root    0 Oct 11 03:25 virbr1.status

[afbjorklund]

This is related to libvirt/libvirt@0f87054 since libvirt doesn't use a dnsmasq file for this anymore (--leasefile-ro).

As you say, the error format is misleading. It should probably have been "file not found", not that it would have helped...

[afbjorklund]

Also seen in #60 and #72

minikube uses virtio now

[afbjorklund]

Actually it was not only using the correct code (status instead of leases), it also needs to use a different network card.
The current "rtl8139" is not supported with the kernel config, it doesn't have to use virtio but can at least use "e1000".

model type rtl8139 (fails)

model type e1000 (works)

It is possible to backport code from minikube to this driver, but since it is not supported anymore there will be no releases.
With enough interest it would be possible to make an version at https://github.com/machine-drivers/docker-machine-kvm

diff --git a/kvm.go b/kvm.go
index 1e6c891..c73de61 100644
--- a/kvm.go
+++ b/kvm.go
@@ -665,11 +667,22 @@ func (d *Driver) GetIP() (string, error) {
        if err != nil {
                return "", err
        }
-       /*
-        * TODO - Figure out what version of libvirt changed behavior and
-        *        be smarter about selecting which algorithm to use
-        */
-       ip, err := d.getIPByMACFromLeaseFile(mac)
+       conn, err := d.getConn()
+       if err != nil {
+               return "", err
+       }
+       defer conn.Close()
+
+       var ip string
+       libVersion, err := conn.GetLibVersion()
+       if err != nil {
+               return "", err
+       }
+
+       // Earlier versions of libvirt use a lease file instead of a status file
+       if libVersion < 1002006 {
+               ip, err = d.getIPByMACFromLeaseFile(mac)
+       }
        if ip == "" {
                ip, err = d.getIPByMacFromSettings(mac)
        }

[darkn3rd]

As newbie to all of this, some questions:

• These are virtual NICs from kvm/libvirt? If so, what determines use of one NIC or another?
• For the lease file mechanism, is there any docs from libvirt on that? or just in the code? I want to learn more how it works.
• No releases: meaning there will be no more tag versions? Or meaning this is in archival mode, no more commits for master? It would be nice to add to master or docs...

[afbjorklund]

Those were the virtual nics, the"rtl8139" is the default and I changed it to "e1000" in a local build:

--- a/kvm.go
+++ b/kvm.go
@@ -60,9 +60,11 @@ const (
     </graphics>
     <interface type='network'>
       <source network='{{.Network}}'/>
+      <model type='e1000'/>
     </interface>
     <interface type='network'>
       <source network='{{.PrivateNetwork}}'/>
+      <model type='e1000'/>
     </interface>
   </devices>
 </domain>`

The lease file used to carry the IPs allocated, but it is now in the status file (in dnsmasq) instead.

No releases: meaning there will be no more tag versions? Or meaning this is in archival mode, no more commits for master? It would be nice to add to master or docs...

See #67

Only the VirtualBox driver was officially supported by Docker Inc., and now Machine is deprecated too.

So far there has been no volunteers to take over the development and release, in the "new" organization.

As the name implies, it was originally only for drivers:

https://github.com/machine-drivers/discussion/issues

[afbjorklund]

@darkn3rd : You can find a prebuilt binary here: https://github.com/afbjorklund/docker-machine-kvm

[afbjorklund]

The missing kernel configuration for "rtl8139cp" was reported here: boot2docker/boot2docker#1407

[jernejr]

@darkn3rd : You can find a prebuilt binary here: https://github.com/afbjorklund/docker-machine-kvm

That solved this exact issue for me