NVIDIA_VISIBLE_DEVICES in my docker images no longer respected

[sl077]

I'm running a server with two A6000 GPUs, a jupyterhub with dockerspawner and a list of docker images, some of them are supposed to have GPU support and some are not.
To achieve this, I configured the images with

ENV NVIDIA_VISIBLE_DEVICES=... \
    NVIDIA_DRIVER_CAPABILITIES=... \
    ...

in their Dockerfiles, setting NVIDIA_VISIBLE_DEVICES to "all" where the GPUs should be available and to "" where not and it used to work great that way.

Until I did some server updates yesterday, incl. installing version 1.17.4 of the nvidia-container-toolkit (old version was 1.16.1). Now I have no GPU support at all in any of those images and my container environments show NVIDIA_VISIBLE_DEVICES="void". It seems that my settings are actively being overwritten by the newer toolkit version, but I don't understand the reason for this. I couldn't find any indication for it in the changelogs either.
As a workaround I now configure my spawner with

 self.extra_host_config.update(
   {
        'device_requests': [
            docker.types.DeviceRequest(
                driver = 'nvidia',
                count = 2,
                capabilities = [['gpu']],
            ),
        ]
    }
)

This is pretty much the same as before, but I had to add "count = 2" to get my GPUs working, albeit in all containers and no longer selective.

What am I missing, how can I restore the old behavior?