Product documentation

List of services

Client Service.
Workflow Manager Service.
HephIA Service.
Visualization Service.
Data Steward Service.

Description of the client Service

This service is used by the client in order to:

Load in the data that he wants to be processed.
Send them to the HephIA service in order to process them.
The processing results are then sent to the Data Steward.
The Steward returns an ID of the model that has been stored in the Data Steward .
It then sends this ID in an HTTP POST method to the Visualization Service in order to visualize the results.

Description of the Workflow Manager Service

This service can be considered as a bridge between the client service and the other services: HephIA service, visualization service and the MongoDB service.

It is responsible for transfering the different requests made by the client service to the other services:

Sending the data ready to be processed and provided by the client to the HephIA service, and send back the results to the client service.

Description of the HephIA Service

This service is responsible for executing HephIA processings on data sent by the client via the workflow manager. It uses a jar that is located in /jar/obfuscatedProduct-assembly-0.42-SNAPSHOT.jar. For the moment, it contains two main classes: com.hephia.integration.clustering.algorithms.demo1.Mtslbm and com.hephia.integration.clustering.algorithms.demo1.WriteSoftClusteringAndAnomalyScores.

The Mtslbm main class reads the data from the /data/input/5x3Clustering.tsv file, and stores the result in the /data/output/MTSLBM1.json file. The results are then sent to the client service via the workflow manager.
The WriteSoftClusteringAndAnomalyScores main class uses implemented data generators to execute the soft clustering and anomaly score processings, so it does not need input data sent by the client service. It stores the processing results on these two files: /data/output/softClustering.json and /data/output/AnomalyScore.json. The results are then sent to the client service via the workflow manager.

Description of the Visualization Service

This service is responsible for receiving an ID of a representation, that is sent by the client. It then makes an HTTP POST request to the Data Steward Service and gets the representation associated with the received ID. Then, it runs a bokeh server that will visualize them. Finally, it sends back a link which allows the client to acess the visualization.

Description of the Data Steward Service

This service is responsible for storing the models (results of processings) in a MongoDB database. The HephIA Service sends the results of precessing to this Service, and this last, returns an ID that represents the stored model.

Run client, workflow, HephIA and visualization services

In order to get the processing results on your local machine, you can share a volume with the client service. Here are the steps required to do that:

Create a directory where you want to put the project. And sub directories input and db into it.

On a terminal, go to the directory you just created and execute the following lines:

git clone git@gitlab.com:hephia/core/product/client-service.git
git clone git@gitlab.com:hephia/core/product/hephia-service.git
git clone git@gitlab.com:hephia/core/product/visualization-service.git
git clone git@gitlab.com:hephia/core/product/workflow-manager.git
git clone git@gitlab.com:hephia/core/product/data-steward.git
git clone git@gitlab.com:hephia/core/product/api-tools.git

Go to the the client service project you just pulled and edit the configuration file located in client-service/conf/shared-volume-conf.sh.
Enter the path to the folder which is going to be the shared volume between your machine and the client service.
- You can put the data that you want to process in this shared volume on your machine, and it will be accessible from the client docker container side. You can also access the processing results in this shared volume on your machine side.

Create a .sh file (e.g. run.sh), and copy in the following bash script:

 docker network create my-bridge
 docker stop $(docker ps -q)
 docker rm $(docker ps -a -q)
 docker rmi workflow-manager
 docker rmi hephia-service
 docker rmi client-service
 docker rmi visualization-service
 docker rmi data-steward
 cp -r $1/api-tools client-service/
 rm -rf client-service/api-tools/.git
 cp -r $1/api-tools visualization-service/
 rm -rf visualization-service/api-tools/.git
 cp -r $1/api-tools data-steward/
 rm -rf data-steward/api-tools/.git
 cp -r $1/api-tools hephia-service/
 rm -rf hephia-service/api-tools/.git
 cp -r $1/api-tools workflow-manager/
 rm -rf workflow-manager/api-tools/.git
 source $1/client-service/conf/shared-volume-conf.sh
 cd $1/workflow-manager
 docker build -t workflow-manager .
 docker run -d --name workflow -e "DOCKER_NAME=${workflow_manager}" --network my-bridge workflow-manager
 cd $1/hephia-service
 docker build -t hephia-service .
 docker run -d --name hephia -e "DOCKER_NAME=${hephia_service}" --network my-bridge hephia-service
 cd $1/visualization-service
 docker build -t visualization-service .
 docker run -d --name viz -e "DOCKER_NAME=${visualization_service}" --network my-bridge -p 5006:5006 visualization-service
 cd $1/data-steward
 docker build -t data-steward .
 kill -9 $(lsof -t -i :27017)
 docker run -d --name data-steward -e "DOCKER_NAME=${data_steward}" --network my-bridge -v $shared_volume_path/db:/data/db data-steward
 cd $1/client-service/
 docker build -t client-service .
 docker run --name client -v $shared_volume_path:/shared:z -e SHARED_VOLUME=$shared_volume_path -e "DOCKER_NAME=${client_service}" --network my-bridge -p 8888:8888 client-service --port=8888

Make the .sh file you just created executable:
```
chmod +x run.sh
```
Run the shell script using this command:
```
./run.sh /absolute/path/to/the/folder/containing/the/six/git/repositories
```
PS: If the script is put in the same directory as the six git repositories, then you can simply execute this command:
```
./run.sh $(pwd)
```
To access the client notebook, copy and paste one of the URLs displayed on the terminal.