Get started
Product
Docs

Docs Menu

Docs

Getting started

HephIA Product

Materials to include

General Knowledge

Overview

Traversable models
Clustering models
- CategoricalClustering models
- HardClustering models
Dimension reduction models
Outlier detection models
Preprocessing models
- Gradient Ascent ???
Evaluation models

Ipnyb notebooks

Common Section Templates

It gathers writting samples that are common in many documentation files with the list of linked algorithms.
- Be sure that when you made an upgrade on a referenced sample from this file, it is update on every other linked ones.

For algorithms working on any type of data.
Linked documentations
Not updated yet

Concepts

Distance

Distance between a point and a cluster

Closest cluster point
- Complexity
Farest cluster point
- Complexity
Prototype cluster point
- Complexity
- Prototype
Miscellaneous

Predictor

Documentation template

labels/tags
Principle
Input
- parameters
Ouput format
Associated visualization
Business case

Features

Clustering

Algorithms

Agglomerative Hierarchical Clustering (AHC)

Labels / Tags
Principle
Scalability
Input
Parameters
Ouput
Predictor
Associated visualization
Practical strategies
Recommended association
Business case
Usage
Recommended association

Canopy

Labels / Tags
Principle
Scalability
Input
Parameters
- Requirements
Output
Predictor
Associated visualization
Recommended Associations
Business case
Usage

Divisive Hierarchical Clustering (AHC)

Labels / Tags
Principle
- How divide a cluster.
- How consider a cluster as terminal.
Scalability
Input
Parameters
Ouput
Predictor
Associated visualization
Practical strategies
Recommended association
Business case
Usage
Recommended association

DSBCAN

Labels / Tags
Principle
Scalability
Input
Parameters
Output
Predictor
Associated visualization
Practical strategies
Recommended association
Business case
Usage

Epsilon Proximity

Labels / Tags
Principle
Scalability
Input
Parameters
Output
Predictor
Associated visualization
Practical strategies
Recommended association
Business case
Usage

Gaussian Mixture Models

Labels / Tags
Principle
Scalability
Pro
Cons
Input
Parameters
Ouput
Predictor
Associated visualization
Practical strategies
Business case
Usage

Included K-Means

Labels / Tags
Principle
Scalability
Input
Parameters
Ouput
Predictor
Associated visualization
Practical strategies
- Find the best K value
- Choose the right stopping criteria
Recommended association
Business case
Usage

Jenks Natural Breaks

labels/tags
Principle
scalability
Input
- parameters
Ouput format
Associated visualization
Business case
Usage

K-Modes

Labels / Tags
Principle
Scalability
Input
Parameters
Ouput
Predictor
Associated visualization
Practical strategies
- Find the best $K$ value
- Choose the right stopping criteria
Business case
Usage

K-Prototypes To Implement

Labels / Tags
Principle
Scalability
Input
Parameters
Output
Predictor
Associated visualization
Practical strategies
Recommended Associations
Business case
Usage

Local Area

labels/tags
Principle
Scalability
Input
- General idea
- Available version
Hyperparameters
Ouput format
Associated visualization
Business case
Usage

Mean Shift

labels/tags
Principle
Scalability
Input
Parameters
Ouput format
Associated visualization
Business case
Usage

Patchwork

Labels / Tags
Principle
Scalability
Input
Parameters
Ouput

Associated visualization
Practical strategies
Business case
Usage

Self Organized Map (SOM)

Labels / Tags
Principle
Scalability
Input
Parameters
Output
Predictor
Associated visualization
Practical strategies
Recommended Associations
Business case
Usage

TEMPLATE Clustering Documentation

Labels / Tags
Principle
Scalability
Input
Parameters
Output
Predictor
Associated visualization
Practical strategies
Recommended association
Business case
Usage

TODO Algorithms

DCDPM

labels/tags
Principle
scalability
Input
- parameters
- Requirements
Ouput format
Associated visualization
Business case
Usage

Divisive Hierarchical Clustering

labels/tags
Principle
scalability
Input
- parameters
- Requirements
Ouput format
Associated visualization
Business case
Usage

K-Medoids

labels/tags
Principle
Scalability
Input
Parameters
Ouput format
Associated visualization
Business case
Usage

Latent Block Model

labels/tags
Principle
scalability
Input
- parameters
- Requirements
Ouput format
Associated visualization
Business case
Usage

Indices

Externals

Accuracy TODO

Labels / Tags
Principle
Scalability
Input
Parameters
- TODO COPIER COLLER de Ball Hall
Ouput format
Associated visualization
Business case
Usage

ARAND TODO

Labels / Tags
Principle
Scalability
Input
Parameters
- TODO COPIER COLLER de Ball Hall
Ouput format
Associated visualization
Business case
Usage

Czekanowski-Dice TODO

Labels / Tags
Principle
Scalability
Input
Parameters
- TODO COPIER COLLER de Ball Hall
Ouput format
Associated visualization
Business case
Usage

F-measure TODO

Labels / Tags
Principle
Scalability
Input
Parameters
- TODO COPIER COLLER de Ball Hall
Ouput format
Associated visualization
Business case
Usage

Fowlkes-Mallows TODO

Labels / Tags
Principle
Scalability
Input
Parameters
- TODO COPIER COLLER de Ball Hall
Ouput format
Associated visualization
Business case
Usage

Jaccard TODO

Labels / Tags
Principle
Scalability
Input
Parameters
- TODO COPIER COLLER de Ball Hall
Ouput format
Associated visualization
Business case
Usage

Kulcztnski TODO

Labels / Tags
Principle
Scalability
Input
Parameters
- TODO COPIER COLLER de Ball Hall
Ouput format
Associated visualization
Business case
Usage

MCC TODO

Labels / Tags
Principle
Scalability
Input
Parameters
- TODO COPIER COLLER de Ball Hall
Ouput format
Associated visualization
Business case
Usage

McNemar TODO

Labels / Tags
Principle
Scalability
Input
Parameters
- TODO COPIER COLLER de Ball Hall
Ouput format
Associated visualization
Business case
Usage

MI TODO

Labels / Tags
Principle
Scalability
Input
Parameters
- TODO COPIER COLLER de Ball Hall
Ouput format
Associated visualization
Business case
Usage

NMI TODO

Labels / Tags
Principle
Scalability
Input
Parameters
- TODO COPIER COLLER de Ball Hall
Ouput format
Associated visualization
Business case
Usage

Purity TODO

Labels / Tags
Principle
Scalability
Input
Parameters
- TODO COPIER COLLER de Ball Hall
Ouput format
Associated visualization
Business case
Usage

RAND TODO

Labels / Tags
Principle
Scalability
Input
Parameters
- TODO COPIER COLLER de Ball Hall
Ouput format
Associated visualization
Business case
Usage

Recall TODO

Labels / Tags
Principle
Scalability
Input
Parameters
- TODO COPIER COLLER de Ball Hall
Ouput format
Associated visualization
Business case
Usage

Rogers Tanimoto TODO

Labels / Tags
Principle
Scalability
Input
Parameters
- TODO COPIER COLLER de Ball Hall
Ouput format
Associated visualization
Business case
Usage

Russel Rao TODO

Labels / Tags
Principle
Scalability
Input
Parameters
- TODO COPIER COLLER de Ball Hall
Ouput format
Associated visualization
Business case
Usage

Sokal Sneath 1 & 2 TODO

Labels / Tags
Principle
Scalability
Input
Parameters
- TODO COPIER COLLER de Ball Hall
Ouput format
Associated visualization
Business case
Usage

Internals

Ball Hall

Labels / Tags
Principle
Scalability
Input
Parameters
Ouput
Associated visualization
Business case
Usage

Davies Bouldin

Labels / Tags
Principle
Scalability
Input
Parameters
- TODO COPIER COLLER de Ball Hall
Ouput format
Associated visualization
Business case
Usage

Silhouette

Labels / Tags
Principle
Scalability
Input
Parameters
- 1 : metric : a dissimilarity measure the given input type.
Ouput format
Associated visualization
Practical strategy
Business case
Usage

Outlier Detection

LBM Outlier Predictor

Labels / Tag
Prediction
Input
Output
- Interpretation

Complete pipelines

MTS LBM SOM

Principle

SOM

Principle

Predictor

$K$NN Predictor

Labels / Tags
Principle
Scalability
Input
- Single query
- Multiple queries, i.e collection of data observations
Parameters
- 1 : Instance[R], i.e. a collection of values of any handled type symbolized by R
- 2 : $K$, the integer value of how many neirest neighbors have to be returned
Output
- Single query
- Multiple queries, i.e. a collection of queries
Associate visualizations
Practical strategies
Recommended associations
Business case
Usage

Closest Prototype predictor

Labels / Tags
Principle
Scalability
Input
- Single query
- Multiple queries, i.e collection of data observations
Parameters
- 1 : Set of prototypes
- 2 : A dissimilarity measure on R and P
Output
Associated visualizations
Similar to
Practical strategies
Recommended associations
Business case
Use case

Gaussian Mixture Model Predictor TODO

Labels / Tags
Principle
Scalability
Input
- Single query
- Multiple queries, i.e collection of data observations
Parameters
- 1 : $K$MeansModel
Output
- Single query
- Multiple queries, i.e. a collection of queries of numerical vectors
Associate visualizations
Practical strategies
Recommended associations
Business case
Usage

K Means Predictor

Labels / Tags
Principle
Scalability
Input
- Single query
- Multiple queries, i.e collection of data observations
Parameters
- 1 : $K$MeansModel
Output
- Single query
- Multiple queries, i.e. a collection of queries of numerical vectors
Associate visualizations
Practical strategies
Recommended associations
Business case
Usage

KNN Categorical Clustering Predictor

Labels / Tags
Principle
Scalability
Input
- Single query
- Multiple queries, i.e collection of data observations
Parameters
Output
- Single query
- Multiple queries, i.e. a collection of queries
Associated visualizations
Practical strategies
Recommended association
Business case
Usage

KNN HardClustering Predictor

Labels / Tags
Principle
- Reminders
Scalability
Input
- Single query
- Multiple queries, i.e collection of data observations
Parameters
Output
- Single query
- Multiple queries, i.e. a collection of queries
Associated visualizations
Practical strategies
Recommended association
Business case
Usage

Latent Block Model Predictor TODO

Labels / Tags
Principle
Scalability
Input
- Single query
- Multiple queries, i.e collection of data observations
Parameters
- 1 : $K$MeansModel
Output
- Single query
- Multiple queries, i.e. a collection of queries of numerical vectors
Associate visualizations
Practical strategies
Recommended associations
Business case
Usage

Mixed Prob SOM Predictor TODO

Labels / Tags
Principle
Scalability
Input
- Single query
- Multiple queries, i.e collection of data observations
Parameters
- 1 : KMeansModel
Output
- Single query
- Multiple queries, i.e. a collection of queries of numerical vectors
Associate visualizations
Practical strategies
Recommended associations
Business case
Usage

Patch Work Predictor

Labels / Tags
Principle
Scalability
Input
- Single query
- Multiple queries, i.e collection of numerical vectors.
Parameters
- 1 : PatchWorkModel
Output
- Single query
- Multiple queries, i.e. a collection of queries of numerical vectors
Associate visualizations
Practical strategies
Recommended associations
Business case
Usage

PCA Predictor TODO

Labels / Tags
Principle
Scalability
Input
- Single query
- Multiple queries, i.e collection of data observations
Parameters
- 1 : KMeansModel
Output
- Single query
- Multiple queries, i.e. a collection of queries of numerical vectors
Associate visualizations
Practical strategies
Recommended associations
Business case
Usage

SOM Predictor TODO

Labels / Tags
Principle
Scalability
Input
- Single query
- Multiple queries, i.e collection of data observations
Parameters
- 1 : KMeansModel
Output
- Single query
- Multiple queries, i.e. a collection of queries of numerical vectors
Associate visualizations
Practical strategies
Recommended associations
Business case
Usage

TEMPLATE Predictor Documentation

Labels / Tags
Principle
Scalability
Input
Parameters
Output
Associated visualization
Similar to
Practical strategies
Recommended association
Business case
Usage

Preprocessing

Gradient Ascent

Labels / Tags
Principle
Scalability
Input
Parameters
- 1 : kernel, defines how values are shifted.
Ouput
Associated visualization
Practical strategies
Business case
Usage

Vectorization

Fourier transform in Time series context

Principle
- Monovariate time series
- Multivariate time series

Product documentation

Chapters

Algorithms

Implemented algorithm combinations

MTS LBM SOM

Predictor

KNN Hard Clustering Predictor

KNN Soft Clustering Predictor

K-NN Predictor

Unsupervised Learning

Clustering

Hard Clustering

K-Means

Tags
Principle
Scalability
Input
Parameters
Ouput
K-Means default predictor.
Associated visualization
Practical strategies
- Find the best K value
- Choose the right stopping criteria
Recommended association
Business case
Usage

K-Modes

K-Prototypes

Soft Clustering

Gaussian Mixture Model

Vectorizer

Binarizer

Develop in scala engine

Global and Traversable Models and Algorithms Patterns

Model
- Memory
- Model
Algorithm

Traversable pattern

Root of the pattern
At InstanceMemory level
- Apply on a concrete collection

Installation

Pipelines

Toy Examples

Advanced examples

Fake advanced examples

Minimal examples

Another K-Means example

Core concepts

Clustering

Cluster id

Hard clustering

Tag : HardClustering
Scala Engine

Soft clustering

Tag : Softlustering

Quality Indices

Externals

Internals

Ball Hall

Davies Bouldin

Silhouette

Algorithms

Unsupervised learning algorithms
- Clustering
Dimensions reduction algorithms
Outliers detection
Set of available algorithms

clustering

Hard clustering
- ClusterId, cf. the definition
Soft clustering
- Categorical clustering

Dimensions reduction

PCA …

Distance

Dissimilarity measure
Similarity measure

Distribution

Evaluation

Evaluator

Extension

EntityExtension
RepExtension

Id

Instance

InstanceMemory

Instance descendants

Markdown tricks

Model

Model Algorithm Paradigm

What’s link an Algorithm and its output Model

Outlier detection

Pipe

Pipeline

Predictor

Quality indices

Scalability

Computational complexity
Space complexity

Traversable model

TraversableMemory

SSOT Home

Utility classes

Payload

Descriptions

CAH

Hyper-parameters
- Payload JSON template example :

Clustering internal indices

Hyper-parameters
- Payload JSON template example :

Denormalization

Hyper-parameters
- Payload JSON template example :

Load csv numerical processing/domain data

Hyper-parameters

MTS SOM Hardclustering projected

Normalize data

Hyper-parameters
- Payload JSON template example :

Normalize data per column

Hyper parameters
- Payload JSON template example :

Numerical Hard SOM with Hardclustering

Hyper-parameters

PCA

Hyper-parameters
- Payload JSON template example :

SOM general dashboard

Hyper-parameters

Visualisations

SOM general dashboard

JSON template

Workflow Manager to Scala Engine

Load csv numerical processing/domain data

JSON template

Numerical Hard SOM with Hardclustering

JSON template

Client to Workflow Manager

Load csv numerical processing/domain data

JSON template

MTS SOM Hardclustering projected

Hyper-parameters
JSON template

Numerical Hard SOM with Hardclustering

JSON template

Referential

ProcessingKeywords
VisualisationKeywords

User classes

datalocation template

Parquet data storage
CSV in GCS data storage
Mongo data storage

Column Strategies

Strategy parameters

Outcomes

Load csv outcome

SOM with Hardclustering outcome

Dataset template

Jobs

Mongo Context template

Representations and observations

AHC

Clustering Internal Indices

Predicted Outputs

Anomaly score

Representation
Observation

Clustering

Features clustering

Representation
Observation

Hard clustering

Representation
- Observation

Regular Input Data

Mixed data

Representation
Observation

Multivariate time series

Numerical vector

Representation
Observation

Unsupervised Models

Clustering

Numerical Hard SOM model

Representation
Observation

Denormalization

Domain/Processing data

Domain data

Representation
Observation

Processing data

Representation
Observation

Normalization

Principal Component Analysis

Glossary

SSOT README

Improving remarks
- Future Hyperparameters handling
- Unsupervised models possible improvements

How to fill Jupiter Doc

Predictor
Clustering Algorithms
Quality indices
- Internals
- Externals
Dimension reduction
Vectorizator

Mandatory Reading

Install Jupyter Book
Glossary
Conventions
- snake_case usage
- EVERY file of production HephIA documentation MUST use snake case convention
- EVERY folder of production HephIA documentation MUST use CamelCase convention
- Syntactic convention for Jupyter Book
Repository organization
User documentation.
- How to build a Jupyter Books in this repository.
- How to clean a previous built book.
Jupyter Book additional documentation.
Developer documentation, i.e. HephIA core library.
MANDATORY READING.
Annexes

Required Improvement

Glossary
- We call value the data itself. // Or each ROW
Other improvements
Questions

Usefull References

Infrastructure homepage

Data Processing - Cloud Computing

Database Usage

HephIA python packages

IP Whitelisting

- Motivation
- How to do that?

Local Dev Template

Package and image Building

- Motivation
- How to do that?

Product documentation

List of services
Run client, workflow, HephIA and visualization services

Purpose

Develop in scala engine

Global and traversable models and algorithms patterns
Traversable pattern

© 2023