Data Processing - Cloud Computing

Create cluster

create-cluster ${CLUSTER_NAME} ${ENV} ${NUM_CLUSTERS} ${UP_TIME(HOURS)} ${MACHINE_TYPE}
create-cluster -n cluster-hephia -e dev -w 2 -u 6 -c 4

Submit job

submit-job ${CLUSTER_NAME} ${CLASS} ${ARGS}
submit-job -n cluster-hephia -e dev -w local -c com.hephia.product.packagedsolutions.numerical_hard_som_with_hardclustering

Monitoring a job

You can list the jobs with this dashboad

You can click on a job to monitor it. You’ll have some graphics about memory, CPU utilization, etc….

In a second section, you can have the logs. That’s the section you have to check when a job failed.

Add clusters to firewall

When a new cluster is created, its machines has to whitelisted in the hephia network. Qo we have to add their IP addresses to the a firewall rule called allow-hephia.

update-rule ${FIREWALL} ${MACHINE/CLUSTER NAME} ${ENV}
update-rule -r allow-hephia -c cluster-hephia -e staging

Delete cluster

A cluster should be deleted when it’s not used. If it has been created without an auto-deletion rule, then it has to be delete with this command.

delete-cluster ${CLUSTER_NAME} ${ENV}
delete-cluster -n cluster-hephia -e staging