0

I would like to be able to use a Google Cloud Composer cluster to launch kubernetes pods from its DAGs onto a separate GKE Autopilot cluster instead of onto the GKE cluster of Cloud Composer.

I have created a GKE autopilot cluster with "control plane global access" set to disabled and only allowing certain authorised networks to connect to the control plane. (based on the recommended security best practices in the documentation)

My pods all fail to launch with the following error message:

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='3X.XXX.XXX.XX6', port=443): Max retries exceeded with url: /api/v1/namespaces/sink/pods?labelSelector=dag_id%3Dtest_dag%2Cexecution_date%3D2021-03-17T212059.4745700000-f0b251c80%2Ctask_id%3Dtest_sync (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f0f3f6a4e80>: Failed to establish a new connection: [Errno 110] Connection timed out',)

I am using the GKEStartPodOperator which previously was able to start pods on a GKE cluster that was self managed (not autopilot) and which did not have "control plane global access" disabled.

Is there any documentation about how to setup Composer to be able to connect to a GKE autopilot cluster that is not exposing global access to the control plane and launch pods?

0

Even with GKE Autopilot, you can use the same set of operators that was originally written for use with normal GKE clusters, such as GKEStartPodOperator. Since the error you are seeing is a timeout to the Kubernetes control plane, it is most likely that your authorized networks setting does not include the addresses used by your Cloud Composer environment's workers.

If you are using a standard Composer environment (non-private IP), you will need to ensure that GCP ranges are included within your authorized networks (because your environment's nodes are assigned ephemeral, public addresses).

If you are using a private IP environment, then you can use private connectivity to reach the Kubernetes control plane, or alternatively, you can configure a Cloud NAT to allow your environment to reach network resources using a static IP address. In the latter case, the IP address of the NAT would need to be included within your authorized networks settings.

2
  • Thank you for the feedback. I suspected that it was the authorized networks settings that needed to include an IP range from Google. Is there somewhere that I can find these IP ranges? (are they called something specific?) or should I simply include the IP ranges of the worker nodes in the Composer Cluster? – Twist Mar 18 at 9:03
  • The closest I got was this link from the GCP documentation gstatic.com/ipranges/cloud.json. I assume the IP addresses of the GKE workers are internal addresses and thus not helpful when a standard Composer environment is used. – Twist Mar 18 at 9:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.