java.io.IOException: Unable to create directory /var/cache/unifi-video/exports

[jbenamy]

I believe this is related to #161. I think I have a permissions issue? No files are being written to /var/cache/unifi-video on the host. I created /var/cache/unifi-video and /var/cache/unifi-video/exports manually but didn't seem to help. It seems that recordings are being stored intermittently as a result.

Server error log:

1592679670.938 2020-06-20 15:01:10.938/EDT: ERROR  [uv.recording.svc] [RecordingService] Unable to do emergency shutoff check, shutting off recordings: Mount point for /var/cache/unifi-video/hls not found in RecordingService-EmergencyShutOffCheck
	at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243) [commons-daemon-1.0.15.jar:1.0.15]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_252]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_252]
	at com.ubnt.airvision.Main.start(Unknown Source) [airvision.jar:?]
	at com.ubnt.airvision.Main.o00000(Unknown Source) [airvision.jar:?]
	at com.ubnt.airvision.service.String.Ó00000(Unknown Source) [airvision.jar:?]
	at com.ubnt.airvision.service.String.Ò00000(Unknown Source) [airvision.jar:?]
	at com.ubnt.airvision.service.recording.RecordingService.Ó00000(Unknown Source) [airvision.jar:?]
	at org.apache.commons.io.FileUtils.forceMkdir(FileUtils.java:2491) ~[commons-io-2.6.jar:2.6]
java.io.IOException: Unable to create directory /var/cache/unifi-video/hls
1592679660.901 2020-06-20 15:01:00.901/EDT: ERROR  [uv.recording.svc] [RecordingService] Unable to create temporary recording diriectory in main
	at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243) [commons-daemon-1.0.15.jar:1.0.15]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_252]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_252]
	at com.ubnt.airvision.Main.start(Unknown Source) [airvision.jar:?]
	at com.ubnt.airvision.Main.o00000(Unknown Source) [airvision.jar:?]
	at com.ubnt.airvision.service.String.Ó00000(Unknown Source) [airvision.jar:?]
	at com.ubnt.airvision.service.String.Ò00000(Unknown Source) [airvision.jar:?]
	at com.ubnt.airvision.service.server.super.Ó00000(Unknown Source) [airvision.jar:?]
	at org.apache.commons.io.FileUtils.forceMkdir(FileUtils.java:2491) ~[commons-io-2.6.jar:2.6]
java.io.IOException: Unable to create directory /var/cache/unifi-video/exports
1592679660.330 2020-06-20 15:01:00.330/EDT: ERROR  [uv.server.svc] Unable to create recording exports diriectory in main

This is my docker run command:

docker run \
        --name unifi-video \
        --cap-add DAC_READ_SEARCH \
        -p 10001:10001 \
        -p 1935:1935 \
        -p 6666:6666 \
        -p 7080:7080 \
        -p 7442:7442 \
        -p 7443:7443 \
        -p 7444:7444 \
        -p 7445:7445 \
        -p 7446:7446 \
        -p 7447:7447 \
        -v /share/unifi-video-data:/var/lib/unifi-video \
        -v /share/Recordings:/var/lib/unifi-video/videos \
        --tmpfs /var/cache/unifi-video \
        -e TZ=America/New_York \
        -e PUID=99 \
        -e PGID=100 \
        -e CREATE_TMPFS=no \
        -e DEBUG=1 \
        --net qnet-static-eth0-79e6cc \
        --ip 192.168.1.191 \
        pducharme/unifi-video-controller

When I inspect the container, I see this:

"Tmpfs": {
                "/var/cache/unifi-video": ""
            },

Directory permissions on host:

[~] # ls -la /var/cache/unifi-video/
total 0
drwxr-xr-x 3 admin administrators  60 2020-06-20 15:00 ./
drwxr-xr-x 6 admin administrators 180 2020-06-20 14:53 ../
drwxr-xr-x 2 admin administrators  40 2020-06-20 15:00 exports/

[thoschworks]

I believe this is related to #161. I think I have a permissions issue? No files are being written to /var/cache/unifi-video on the host. I created /var/cache/unifi-video and /var/cache/unifi-video/exports manually but didn't seem to help. It seems that recordings are being stored intermittently as a result.

There should not any file be written to /var/cache/unifi-video on the host, because...

docker run \
        --name unifi-video \
...
        -v /share/unifi-video-data:/var/lib/unifi-video \
        -v /share/Recordings:/var/lib/unifi-video/videos \
...

...you only write /var/lib/unifi-video and /var/lib/unifi-video/videos to bind mounts on the host.

The folder /var/cache/unifi-video is only accessible in the container. The tmpfs mount is a special sort of mount, which persists only in the memory of the host (see here).

Take a look near the end of this error message:

1592679670.938 2020-06-20 15:01:10.938/EDT: ERROR  [uv.recording.svc] [RecordingService] Unable to do emergency shutoff check, shutting off recordings: Mount point for /var/cache/unifi-video/hls not found in RecordingService-EmergencyShutOffCheck

The folder /var/cache/unifi-video/hls is missing. (You can check this out by bashing into the conatiner: docker exec -it unifi-video /bin/bash)

I think it looks like the infamous Issue #178.

When does the error occur? Just after the start or after some time? Or after a reboot?

[jbenamy]

I believe this is related to #161. I think I have a permissions issue? No files are being written to /var/cache/unifi-video on the host. I created /var/cache/unifi-video and /var/cache/unifi-video/exports manually but didn't seem to help. It seems that recordings are being stored intermittently as a result.

There should not any file be written to /var/cache/unifi-video on the host, because...

docker run \
        --name unifi-video \
...
        -v /share/unifi-video-data:/var/lib/unifi-video \
        -v /share/Recordings:/var/lib/unifi-video/videos \
...

...you only write /var/lib/unifi-video and /var/lib/unifi-video/videos to bind mounts on the host.

The folder /var/cache/unifi-video is only accessible in the container. The tmpfs mount is a special sort of mount, which persists only in the memory of the host (see here).

Take a look near the end of this error message:

1592679670.938 2020-06-20 15:01:10.938/EDT: ERROR  [uv.recording.svc] [RecordingService] Unable to do emergency shutoff check, shutting off recordings: Mount point for /var/cache/unifi-video/hls not found in RecordingService-EmergencyShutOffCheck

The folder /var/cache/unifi-video/hls is missing. (You can check this out by bashing into the conatiner: docker exec -it unifi-video /bin/bash)

I think it looks like the infamous Issue #178.

When does the error occur? Just after the start or after some time? Or after a reboot?

Thank you for the clarification on the /var/cache/unifi-video directory. Makes sense. I do see /var/cache/unifi-video/hls in the container:

root@22bb6b73856d:~# ls -la /var/cache/unifi-video/
total 4
drwxrwxrwt 4 root        root     80 Jun 20 16:14 .
drwxr-xr-x 1 root        root   4096 Jun 20 16:13 ..
drwx------ 2 unifi-video users    40 Jun 20 16:14 exports
drwxrwxr-x 2 unifi-video users 56160 Jun 20 21:13 hls

The issue occurs at container start and then clears up and then comes back. So recordings save intermittently it seems. The container is now getting Caused by: java.io.IOException: No space left on device errors and won't start at all. The mount points have 4 TB of space available. Came back up after deleting the logs in the volume. Wonder if this is a related issue or not. I reviewed #178 and am now running with --tmpfs /var/cache/unifi-video:rw,noexec,nosuid,size=65536k,mode=777,uid=99 and will see if the issue persists.

Appreciate the good info. Haven't spent much time with Docker, but trying to learn.

[jbenamy]

Ran out of disk space for logs again... what the heck. Only 2.3MB in the logs folder. Caused by: java.io.IOException: No space left

[thoschworks]

Ran out of disk space for logs again... what the heck. Only 2.3MB in the logs folder. Caused by: java.io.IOException: No space left

This reminds me of a similar error I stumbled upon at the beginning: 🤔

The image is setup assuming that the recording path is /var/lib/unifi-video/videos. This may have been correct in the past, but a new installation of the UniFi Video software stores the videos in /usr/lib/unifi-video/data/videos (see picture).

I think changing the path in the web ui to /var/lib/unifi-video/videos should solve the problem.

I'll file an issue.

[jbenamy]

Ran out of disk space for logs again... what the heck. Only 2.3MB in the logs folder. Caused by: java.io.IOException: No space left

This reminds me of a similar error I stumbled upon at the beginning: 🤔

The image is setup assuming that the recording path is /var/lib/unifi-video/videos. This may have been correct in the past, but a new installation of the UniFi Video software stores the videos in /usr/lib/unifi-video/data/videos (see picture).

I think changing the path in the web ui to /var/lib/unifi-video/videos should solve the problem.

I'll file an issue.

I've just switched the recording path in the UI to /var/lib/unifi-video/videos and cleared the logs directory out again to get the container to boot (it crashed again). I will keep an eye on it and see if I run into either of these issues again. Appreciate the quick responsiveness. Glad to be using a project that is well supported.

[thoschworks]

In this cases you should throw the container away:

docker stop unifi-video
docker rm unifi-video
docker run …

Or quoting @fryfrog:

Docker containers are meant to be disposable, […]

If something gets brown or f***s up, don't tinker around in the container trying to fix something: Just stop and remove the container. Everything important should be persisted in the mounted volumes. Fix the problem from the outside (command options, volumes or fix the dockerfile and make you own build) and run a new container.

In this case the folder with the logs was the symptom but not the problem. The problems which caused the No-space-left were the video files in /usr/lib/unifi-video/data/videos. As long as you don't delete them (or just dispose the container) the problem will reoccur over and over again.