Cost Effective Generic Machine Learning Operation: A Case Study

• Published in: 2023 International Conference on Data Science and Network Security (ICDSNS) pp. 1 - 6
• Main Authors: Jain, Samridhi, Kumar, Puneet
• Format: Conference Proceeding
• Language: English
• Published: IEEE 28.07.2023
• Subjects: cloud platform; Costs; Data science; Industries; Low Budget Architecture; Machine learning; ML Engineering; ML pipeline; MLOps; Network security; Organizations; Pipelines
• DOI: 10.1109/ICDSNS58469.2023.10245408

In this research, we have proposed a mechanism to implement a typical Mops pipeline for small scale organization who cannot afford the operational expenditures to bring the pipeline at Cloudera, Horton works platform or cloud premises like AWS, GCP or Azure. This paper gives a very detailed understanding of operationalization of a typical ML pipelines to adhere all the elements and artifacts without even using any Docker, Kubernetes or even any API generating platforms like Flask or FastAPI. Using the combination of a simple Python/R along with SQL and Shell scripts we can manage the entire workflow at on premises with a very low-cost approach. From some angle this mechanism would not be comparable with the architectures like market ready MLOps platforms like Azure Devops, MLflow, Kubeflow, Apache Airflow, Databricks with Data factory or Sagemaker Studio workflow but from conceptual point of view, suffice almost 90% of the requirements with efficient manner. We have also done a latest review related to MLOps in recent past and also listed out the several research gaps that can be solved in future research.