MLOps is a crucial discipline that oversees the process of bringing machine learning (ML) models into production. While it’s common to focus on training and deploying a single model, in reality, organizations often work with multiple models and complex workflows. To handle this, it’s essential to have the right infrastructure in place to track, train, deploy, and monitor these models at scale. This is where MLOps tooling comes into play.
MLOps tooling helps simplify and automate the development and maintenance of ML models. Amazon SageMaker Pipelines, a feature of Amazon SageMaker, is a purpose-built workflow orchestration service that enables you to automate end-to-end ML workflows at scale. It provides a centralized platform to manage tasks like data preparation, model training, tuning, and validation. With SageMaker Pipelines, you can streamline workflow management, accelerate experimentation, and easily retrain models.
One exciting feature of SageMaker Pipelines is Selective Execution. This feature allows you to selectively run specific portions of your ML workflow, saving significant time and compute resources. Instead of rerunning the entire pipeline, you can choose to run only the steps that are relevant to your current needs. This feature also allows you to modify the runtime parameters associated with the selected steps.
Selective Execution relies on a reference run of the pipeline that has already completed and can be marked as Successful, Failed, or Stopped. The outputs of non-selected steps are reused from the reference run. You can choose any number of steps to run, as long as they form a contiguous portion of the pipeline.
There are various use cases where Selective Execution can be beneficial. For example, you can focus on specific steps like training and modify the training parameters to improve the model. This saves time and reduces costs as you only run the necessary steps. Additionally, you can selectively run a subset of steps to evaluate model performance against a test dataset without running the entire pipeline.
To use Selective Execution, you need to set up the required components in your SageMaker environment, such as the SageMaker Python SDK and access to SageMaker Studio (optional). You can then import the SelectiveExecutionConfig class and retrieve the reference pipeline ARN, pipeline steps, and runtime parameters. From there, you can start executing the pipeline subset using the SelectiveExecutionConfig and the desired parameters.
Overall, Selective Execution in SageMaker Pipelines provides flexibility and efficiency in managing and executing ML workflows. It allows you to save time and resources by running only the necessary steps and modifying parameters as needed.