US 12,346,737 B2
Method and system for managing hybrid spark cluster for efficient spark job execution
Dong-Wook Lee, Seoul (KR); Gyu-Hong Choi, Seoul (KR); Gi-Beom Kil, Seoul (KR); and Seung-Tae Chun, Seoul (KR)
Assigned to DATASTREAMS CORP., Seoul (KR)
Filed by DataStreams Corp., Seoul (KR)
Filed on Sep. 2, 2022, as Appl. No. 17/929,413.
Claims priority of application No. 10-2022-0099002 (KR), filed on Aug. 9, 2022.
Prior Publication US 2024/0061718 A1, Feb. 22, 2024
Int. Cl. G06F 9/50 (2006.01); G06F 9/455 (2018.01)
CPC G06F 9/505 (2013.01) [G06F 9/45541 (2013.01)] 4 Claims
OG exemplary drawing
 
1. A cluster management method which is performed by a cluster management system, the cluster management method comprising:
in a hybrid Spark cluster environment comprising a bare-metal-based cluster and a Kubernetes-based cluster,
selecting a cluster manager comprising the bare-metal-based cluster or the Kubernetes-based cluster before a Spark client performs a Spark driver program; and
performing a Spark job by using the selected cluster manager,
wherein selecting the cluster manager comprises:
converting, into a time feature weight, a resource feature weight, and an operation feature weight, an importance measure as importance numerical values related to a time feature, a resource feature, and an operation feature are designated by a user,
calculating importance of the time feature for the bare-metal-based cluster or the Kubernetes-based cluster by multiplying a value obtained by weighted-summing an idle resource amount and a resource consumption rate by the time feature weight according to Equation 1 and Equation 2,

OG Complex Work Unit Math
wherein Corer means the number of available CPU cores that remain in the cluster, Memr means the amount of available memory that remains in the cluster, Mema means the amount of memory allocated to the cluster, a calibration factor (CF) means a core conversion index, and αt means an idle resource amount and a resource consumption rate weight,
calculating importance of the resource feature for the bare-metal-based cluster or the Kubernetes-based cluster by multiplying a calculated value of a degree of parallel processing (Vt) by the resource feature weight when an identical resource is allocated to an identical job by assigning a weight to each of parallel processing in a process level and parallel processing in a thread level according to Equation 3 or Equation 4,

OG Complex Work Unit Math
wherein Coreexec means the number of CPU cores allocated to an executor, Coretask means the number of CPU cores a allocated a task, OFr means resource overhead when a container is used, and αr means a process/thread parallel processing weight,
calculating importance of the operation feature for the bare-metal-based cluster or the Kubernetes-based cluster by multiplying a value obtained by weighted-summing overhead according to the preparation of container-based driving and a container-based driving gain (container gain) by the operation feature weight according to Equation 5 or Equation 6,

OG Complex Work Unit Math
wherein OFK8s means overhead for Spark application container driving in K8s, OFBM means overhead for Spark application driving in the bare-metal-based cluster, a container gain means 1-OFK8s in case of K8s and 0 in case of BM, and αo means a container operation weight.