VCF 9.1 VKS and Supervisor Content Libraries with Offline Depot

This is a quick post to explain what to do when you have an offline repo and want to enable the supervisor cluster and deploy a VKS cluster or two. Unfortunately, the vks-download-tool with 9.1 does not currently download the artefacts for the Supervisor OVA or Kubernetes Releases. If your depot is connected and online, then this isn’t an issue; however, for offline depot users, you will run into problems. There is a planned fix for the 9.1.1 release.

Navigation

• vSphere Supervisor Artifacts
• Kubernetes Releases Artifacts
• Assign vSphere Supervisor Content Library
• Conclusion

vSphere Supervisor Artifacts

The default supervisor OVF and vSpherelet artefacts are available using the public link below. This should be added as a content library by default before configuring your supervisor; however, as you’ll notice by browsing the directory, there is no OVF for 9.1. Engineering is working on it, and in the next 9.1.1 release, the vcf-download-tool will include the ability to download the artefacts and include them in the offline depot.

https://wp-content.broadcom.com/supervisor/v1/latest/lib.json

Until then, we have to head over to the Broadcom download page > My Downloads> VMware Cloud Foundation > 9.1.0.0 > Primary Downloads > VMware vSphere Supervisor and grab the relevant files. Here is a direct URL.

Once you have these files, we can upload and extract them to the offline depot in the correct path using the following commands.

## copy zip to offline depot
scp VMware-vSphere-Supervisor-9.1.0.0100-25417926.zip depot01.aclab.uk:/home/vcf

## login to the offline depot and create the required directory structure
ssh depot01.aclab.uk
sudo su
mkdir SUPERVISOR
unzip VMware-vSphere-Supervisor-9.1.0.0100-25417926.zip -d SUPERVISOR
mv SUPERVISOR /var/www/html/PROD/COMP/

## fix ownership and permissions
chown apache:apache -R /var/www/html/
find /var/www/html -type d -exec chmod 0500 {} \;
find /var/www/html -type f -exec chmod 0400 {} \;
chmod 755 /var/www/
chmod 755 /var/www/html/

At this point, this documentation suggests how to create a content library for vSphere Supervisor releases in an air-gapped environment; however, the article incorrectly references 9.0.x, not 9.1. Since we have already added the Supervisor OVF to the offline depot, this will also be available via the VCF Fleet Manager lifecycle manager URL (see below), which in my environment is here. Please refer to this article to build out your URL.

https://fleetlcm.aclab.uk/depot-service/content-gateway/PROD/COMP/SUPERVISOR/lib.json

To put the fleet LCM in context, SSH to the VMSP Kubernetes cluster and run the following commands.

## reveal whats inside the VMSP cluster.
export KUBECONFIG=/etc/kubernetes/super-admin.conf
alias k=kubectl
k get httproute -n vcf-fleet-depot
k get gateway -n vmsp-platform
k get all -l app.kubernetes.io/instance=depot-service -n vcf-fleet-depot

You’ll notice that Envoy proxy is configured with the Gateway API, and there are HTTPRoutes exposed. The HTTPRoute redirects to the depot-service, which has an endpoint slice of the depot-service deployment. There is a 1TB PVC mounted to the file-server container, mounted as /data inside the depot-service pod. It’s this microservice which is used when performing LCM management operations from VCF Operations or SDDC Manager.

As we now have the binaries in place on our offline depot, we can use the VCF Fleet Manager lifecycle manager URL to create a content library, for example. I’m going to use plain and straightforward names.

Click Next and add the Fleet LCM URL.

Click Next and enable the default OVF security policy.

Click Next and select an appropriate datastore to store the CL.

Click next and finish.

The library will happily sync up.

And shows OVA and Spherelet contents, stored locally.

Kubernetes Releases Artifacts

For VKS, you can follow William Lams’ article, which contains a PowerShell script to download the entire content library by parsing the lib and items.json files. These can then be copied onto a web server and used for the content library. The alternative is to follow this article and manually download the required VKR releases; for example, v1.34.2 shipped with 9.1, so you would only need to download from this version through to the latest.

I wanted something written in Bash so we can run it directly on the offline depot without having to install anything. Before running, ensure you have at least 500GB of free disk space; there are many old VKR releases, from v1.16.8 through to v1.35.5. When using the below mirror script, all are downloaded, not just the latest.

## connect to the offline depot, download a script
## prepare prepare a directory already shared by HTTPS
sudo su
curl https://raw.githubusercontent.com/amayacitta/vcf9-lab/refs/heads/main/depot/vkr-mirror.sh -o vkr-mirror.sh
chmod +x vkr-mirror.sh
mkdir /var/www/html/PROD/COMP/VKR && cd /var/www/html/PROD/COMP/VKR

## grab all content using bash script
/home/vcf/vkr-mirror.sh

## confirm content
tree /var/www/html/PROD/COMP/VKR
du -hd 0 /var/www/html/PROD/COMP/VKR

Once this is done, on deployment an automatically created content library will appear using the following URL.

https://fleetlcm.aclab.uk/depot-service/content-gateway/PROD/COMP/VKR/lib.json

Assign vSphere Supervisor Content Library

Before deploying the supervisor, ensure that you assign the supervisor-releases content library.

Select the supervisor-releases content library, which now contains the 9.1 OVF and Spherelet artefacts.

Note, failing to mirror the VKR content into the offline depot will cause repeated errors because the installation process cannot find the 9.1 VKR artefacts. Here is an example of the errors you will see; these will only appear with an offline depot.

If you’re fully connected and online, there is no issue, as the 9.1 Supervisor Artefacts are automatically downloaded.

Conclusion

I hope this unpacks some of the issues with offline depots and VCF 9.1. The above workarounds have worked for my customers.

Rest assured, in 9.1.1, these issues will go away, as the download tool will have a new option to grab all of the VKS/Supervisor artefacts and automatically download them to the offline depot, without the need for the above fixes.