I would like to be able to use a Google Cloud Composer cluster to launch kubernetes pods from its DAGs onto a separate GKE Autopilot cluster instead of onto the GKE cluster of Cloud Composer.
I have created a GKE autopilot cluster with "control plane global access" set to disabled and only allowing certain authorised networks to connect to the control plane. (based on the recommended security best practices in the documentation)
My pods all fail to launch with the following error message:
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='3X.XXX.XXX.XX6', port=443): Max retries exceeded with url: /api/v1/namespaces/sink/pods?labelSelector=dag_id%3Dtest_dag%2Cexecution_date%3D2021-03-17T212059.4745700000-f0b251c80%2Ctask_id%3Dtest_sync (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f0f3f6a4e80>: Failed to establish a new connection: [Errno 110] Connection timed out',)
I am using the GKEStartPodOperator which previously was able to start pods on a GKE cluster that was self managed (not autopilot) and which did not have "control plane global access" disabled.
Is there any documentation about how to setup Composer to be able to connect to a GKE autopilot cluster that is not exposing global access to the control plane and launch pods?