We will also create Spot-backed managed node groups and taint them so Kubernetes schedules only Airflow workers on them.Īlthough this post uses managed node groups to create and manage data plane, you can also use Fargate or self-managed nodes to run Airflow core components and workers. This post uses node labels and nodeSelector to schedule Airflow UI and scheduler on On-Demand nodes. You can also use the node group to apply Kubernetes labels to nodes and update them at any time. A managed node group configures an Auto Scaling group on your behalf with Spot best practices applied as explained in the Amazon EKS documentation.Ī managed node group automatically tags the nodes it creates for auto-discovery by the Kubernetes cluster autoscaler. However, with managed nodes, you don’t need to install any additional tools to handle Spot interruptions. Both managed node groups and self-managed nodes can create Spot instances.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |