ML-Scaler

A comprehensive declarative ML deployment management system for Kubernetes-based machine learning model for real time scaling and orchestration. ML-Scaler provides automated deployment, scaling, real time inference and lifecycle management for ML models on Kubernetes Service such as AKS, GKE, EKS etc levearging KEDA.

Features

• Automated ML Model Deployment - Deploy ML models to Kubernetes with automated configuration
• Dynamic Scaling - Auto-scale model deployments based on demand
• Deployment Management - List, update, and delete model deployments via CLI
• Database Persistence - Track deployment specs, compute instances, storage, and event sources
• Azure DevOps Integration - Trigger CI/CD pipelines for model deployments
• CLI Interface - Typer-based command-line tool for operations

Prerequisites

• Python 3.10 or higher
• Docker and Docker Compose
• PostgreSQL database
• Azure Kubernetes Service (AKS) cluster
• Azure DevOps account (for pipeline integration)
• Service Principal credentials such as Azure Service Principal

Quick Start Guide

Installation

Using Docker Compose (Recommended)

Using a Docker Compose file along with a local environment file and configuration YAML values will bring the CLI wrapper up and running.

# Clone the repository
git clone <repository-url>
cd <directory>

# Start services
docker-compose up -d

Local Installation

# Install dependencies
pip install -e .

# Set up environment variables (see Configuration)
cp env.example .env
# Edit .env with your configuration

Configuration

Security Setup

IMPORTANT: Never commit credentials to version control. Use environment variables for all sensitive data.

1. Create a .env file from the example:

   cp .env.example .env

2. Configure required environment variables in .env:

   • Database: DATABASE_URL
   • Azure DevOps: AZURE_DEVOPS_PAT
   • Azure Service Principal: TENANT_ID, CLIENT_ID, CLIENT_SECRET
   • Azure Databricks Configurations: DATABRICKS_HOST, DATABRICKS_TOKEN

3. Update app/config.yaml with non-sensitive configuration:

   • Azure DevOps organization, project, pipeline, and branch name
   • AKS api_server & namespace
   • You can also handle the logging_level, db_config non-sensitive informations accordingly.

NOTE:----> You can set the configurations either in ".env" file or "app/config.yaml" accordingly based on sensitivity.

4. For Production: Use Azure Key Vault or similar secret management service instead of .env files

CLI Interface

CLI commands offered by mlscaler:

mlscaler --help

[ABOVE COMMAND WILL GUIDE YOU HOW TO USE DEFINED CLI COMMANDS]

initialize

Initialize environment, check DB health, and ensure required tables are present.

add-resource

Add configuration for required instance resources (storage, compute, event source, deployment spec) to DB.

create-deployment --deployment-name <name>

Trigger deployment for a specific deployment name. Checks AKS for existing deployment first. Use --overwrite-deployment flag if deployment already exists. It has provision to check the pipeline status as well.

update-deployment --deployment-name <name>

Update configuration key-values for a deployment in DB. On user confirmation, triggers pipeline with updated configurations. On trigger, it has provision to check the pipeline status as well.

view-deployment --deployment-name <name>

View DB configurations and AKS deployment details for a specific deployment.

list-deployments

List all deployments available in AKS under the configured namespace.

check-deployment-status --deployment-name <name>

Check deployment status in AKS for a specific deployment.

delete-deployment --deployment-name <name>

Delete deployment from AKS and DB configuration on user confirmation.

Architecture

• SQLAlchemy - ORM for database operations
• PostgreSQL - Relational database for persistence
• Typer - Modern CLI framework
• Kubernetes Client - AKS cluster interaction
• Azure DevOps API - CI/CD pipeline orchestration

Code Quality

This project follows Python best practices and PEP 8 guidelines. Future versions may include automated testing.

Contributing

Contributions are welcome! Please ensure:

• Code follows project style guidelines
• Documentation is updated as needed

License

This project is licensed under the MIT License - see the LICENSE file for details.

Support

For issues, questions, or contributions, please open an issue in the repository.