Getting started

Setup your local env

Make sure to some prerequisites:

Python3.9.2
Docker
GCP Cloud SDK
Bazelisk
Docker Compose
After installing Docker and docker compose, follow this: To run docker without having to call sudo follow this link. Open the terminal and type this command:

sudo chmod 666 /var/run/docker.sock
sudo gpasswd -a $USER docker
newgrp docker

After pulling the repo, you have

Step 1: Add credential files

Copy the content of the dev from this Drive to local_dev_template/credentials/dev, then:

Add the file storage-sa.json to client_service/jupyter/, scala_engine_service/ and visualization_service/data/storage.json
Add the file dataproc-sa.json to scala_engine_service/dataproc/.
Add the file pip.conf to client_service/jupyter/ and scala_engine_service/
Add the files id_rsa* to client_service/jupyter/
Open the file secrets.env, copy the variables there and paste them into local_dev_template/build.env

Step 2: Setting variables

Initialize the setup to copy the env variables, the scripts, etc…

bash setup.sh init

Several scripts will be installed $HOME/.local/bin. The credentials folders will be added in $HOME/.local/credentials and the env variable $HOME/.local/env.

Step 3: Specify you repo location

bash setup.sh repo /path/to/HephIA-product

You’re now ready to do some operations!

Day-to-day use cases

Add new scripts to your bin
Update the env files
Install HephIA python packages
Edit and test scala classes
Build and run HephIA images locally
Use the deployed Scala Engine Artifact on Zeppelin
Work with a cluster Dataproc
Run the deployed versions of the product

Add new scripts to your bin

To add new scripts to your bin, you have to follow those steps:

#copy your script to local_dev_template utils
cp /path/to/my-script.sh local_dev_template/utils
cd local_dev_template
bash setup.sh scripts

Update the env file

To add new scripts to your bin, you have to follow those steps:

#edit your file: add, change or remove a variable in build.env or run.env 
#then run this command to update 
bash setup.sh env

Install HephIA python packages

Follow this documentation

Edit and test scala classes

Step 1: Create the Scala Main

Step 2: Build your code and create a jar

Set a custom version for the jar in local_dev_template/build.env

# Add your custom versions there
#the version of the product
export VERSION="2.1-SNAPSHOT"
#the versions of jars to load in the zeppelin
export PRODUCT_VERSION="2.1-SNAPSHOT"
export CORE_VERSION="2.1-SNAPSHOT"
export SPARKMODULE_VERSION="2.1-SNAPSHOT"

Create the jar

#Use sbt or Maven to build and generate your jar
mvn-build -m ${MODULE} -c ${PRODUCT_VERSION} -s ${CORE_VERSION} -p ${SPARKMODULE_VERSION}

If changes has been made in core, then use

mvn-build -m core -c '2.1-SNAPSHOT' -s "2.1-SNAPSHOT" -p '2.1-SNAPSHOT'

This previous command will build, core, then sparkmodule and finally product.

If changes has been made in sparkmodule, then use

mvn-build -m sparkmodule -s "2.1-SNAPSHOT" -p '2.1-SNAPSHOT'

This previous command will build sparkmodule then product.

If changes has been made in product, then use

mvn-build -m product -p '2.1-SNAPSHOT'

This previous command will only build product.

N.B: The previous steps will push the jar to GCS.

Push the jar(optional)

You don’t need to apply this step if the previous one has been done properly Step 2: Build the jar and push it to GCS

push-jar /path/to/the/jar

Step 3: [Launch the HephIA services ](Launch HephIA services)

Build and run HephIA images locally

You’re now ready to build and run the services!

launch-hephia client viz workflow scala data-steward mongo mongo_seed zeppelin