Amazon SageMaker AI Deployments by Example

Focusing on the four inference options in Amazon SageMaker AI.

The series, starting with Amazon SageMaker AI (JupyterLab) by Example (Part 1), mostly focuses on the early tasks of the machine learning (ML) workflow and only lightly covers the later deployment task. Here we dive into the deployment task in more detail.

Prerequisites

If you wish to follow-along, you need to have an Amazon SageMaker AI model; we will start by dissecting it and recreating it manually. One way of getting one is by following the steps in the last article in the aforementioned series; Amazon SageMaker AI (JupyterLab) by Example (Part 4).

We also need a Amazon S3 bucket; here we will also be following the guidance in Deploy models for real-time inference and use the recommended bucket naming convention (with the recommended CORS configuration).

We recommend that you use the bucket naming convention sagemaker-{Region}-{accountID}. This bucket is used to store the artifacts that you upload.

Amazon SageMaker AI Model

If one creates a model through Amazon SageMaker AI Studio, one will find models under the Amazon SageMaker AI Studio Models menu and Deployable models tab.

One will also find models under the Amazon SageMaker AI > Inference > Models menu.

So what precisely are they? From, Deploy models for real-time inference, the closest to a definition that we have is.

Before you deploy a SageMaker AI model, locate and make note of the following:

- The AWS Region where your Amazon S3 bucket is located
- The Amazon S3 URI path where the model artifacts are stored
- The IAM role for SageMaker AI
- The Docker Amazon ECR URI registry path for the custom image that contains the inference code, or the framework and version of a built-in Docker image that is supported and by AWS

For completeness, the Registered models tab (not covered here) is related to Amazon SageMaker AI Model Registry as described in Model Registration Deployment with Model Registry.

Catalog models by creating SageMaker Model Registry Model (Package) Groups that contain different versions of a model. You can create a Model Group that tracks all of the models that you train to solve a particular problem. You can then register each model you train and the Model Registry adds it to the Model Group as a new model version.

Recreating the Amazon SageMaker AI Model

Before we recreate the model, we need to understand a model’s key components.

AWS Region

From Deploy models for real-time inference we have to be consistent in which AWS region we use.

The Amazon S3 bucket where the model artifacts are stored must be in the same AWS Region as the model that you are creating.

Model Artifact Amazon S3 URI

As part of the setup of the model training, we specify the Amazon S3 bucket and folder to store the model artifact (the computed parameters).

Once the training is complete we can get the Amazon S3 URI of the model artifact.

The AWS IAM Role

The JupyterLab examples of working with Amazon SageMaker AI use the JupyterLab space’s execution role for the model.

Because this role has excessive permissions, we follow the document Deploy models for real-time inference to create a new role; a SageMaker Execution AWS service role with just the AmazonSageMakerFullAccess policy.

Among other things, this policy gives read / write access to any Amazon S3 bucket named as follows; thus gives access to the model artifact in the bucket sagemaker-{Region}-{accountID} used in this example

Docker Amazon ECR URI

Finally, we need the container URI which we can output.

Recreating

Using these components, we navigate to the Amazon SageMaker AI Studio Models menu and Deployable models tab and press the Create button; here we start by giving the model a unique name.

We select Bring your own container and supply the Docker Amazon ECR URI from above.

Next, we supply the model artifact Amazon S3 URI from above.

We then supply the AWS IAM role from above; as an ARN.

Finally, we press the Create deployable model button and observe that we now have a new model.

Inference Options in Amazon SageMaker AI

Now that we have an Amazon SageMaker AI Model, we consider the four inference options outlined in the document Inference options in Amazon SageMaker AI. The remainder of the document, walks through implementing them.

Real-Time Inference

Here we will be following the Create a deployable model section of the document Deploy models for real-time inference.

Amazon SageMaker AI Studio

We navigate to the Amazon SageMaker AI Studio Models menu and Deployable models tab. Selecting our newly created model, we press the Deploy button.

Here we can create a custom endpoint name, e.g. linear-learner-realtime. For now we choose the least cost option, i.e., one instance of the ml.c6i.xlarge instance type.

Here the choice of a CPU instance was justified as the the section, EC2 instance recommendation for the linear learner algorithm, of the Linear Learner Algorithm document has.

The linear learner algorithm supports both CPU and GPU instances for training and inference. For GPU, the linear learner algorithm supports P2, P3, G4dn, and G5 GPU families.Serverless Inference

For reference the ml.c6i.xlarge instance type has 4 CPU and 8 GiB of memory.

Also notice that there is only an option for Real-time inference here (interesting).

Scrolling down and expanding the model we selected, we observe (but do not change) the default settings for the model.

note: What is not clear is the fact that the underlying instance has 4 CPU but here we are only allocating 1 CPU to the model. The documentation is not overly clear on this topic.

We press the Deploy button; it takes some time (5 minutes or so) for both the endpoint and the model to come In service.

We can validate the operation of the endpoint by navigating to the Test inference tab.

Here we can supply the features as a CSV; in this case a single feature.

The equivalent request in JSON format.

{
  "instances": [
    {
      "features": [ 13.0 ]
    }
  ]
}

We delete the endpoint from the Amazon SageMaker AI Studio > Deployments > Endpoints menu.

Amazon SageMaker AI (Console)

When we created the endpoint in Amazon SageMaker AI Studio, the endpoint could also be observed from the Amazon SageMaker AI (console) > Inference > Endpoints menu; much of the functionality here, however, is disabled indicating that it should be managed in Amazon SageMaker AI Studio.

Here we re-create the endpoint using Amazon SageMaker AI (console); starting with the Endpoint Configurations menu and pressing the Create endpoint configuration button. We name as we see fit.

We next add the model to the endpoint by selecting Create production variant.

note: Interestingly here the instance type is already selected with no option to change. Also, there is no option to assign CPU and memory to model itself as we saw earlier.

At this point, our only option is to press the Create endpoint configuration button; here we see the result of our work.

Next using the Endpoints menu we press the Create endpoint button. We name as we see fit and select the endpoint configuration we just created and pressing the Select endpoint configuration button.

At this point, the only option is to press the Create endpoint button. After a few minutes, we see the result of our work.

Interestingly, the endpoint is visible in Amazon SageMaker AI Studio; so we can validate the operation of the endpoint by navigating to the Test inference tab.

Python SDK

Creating endpoints in Amazon SageMaker AI Studio and Amazon SageMaker AI (console) had some oddities and were inconsistent with each other; hoping the Python SDK provide a better experience.

Roughly following Amazon’s example, we can write the code in a Amazon SageMaker AI Studio JupyterLab space to both create and deploy the model in much the same way we we did in the Amazon SageMaker AI Studio section above. Found it useful to use Amazon Q to help write the code.

note: However, like the other approaches to creating endpoints, the experience here was suboptimal, e.g. the example code had to be tweaked to work.

Serverless Inference

Here we will be following the document Serverless endpoint creation.

Amazon SageMaker AI (Console)

As we observed earlier, Amazon SageMaker AI Studio can only used to deploy real-time inference endpoints.

Here we create the endpoint using Amazon SageMaker AI (console); starting with the Endpoint Configurations menu and pressing the Create endpoint configuration button. The options are similar to what we did before, except we select a Serverless type of endpoint.

We also here edit the Max concurrency to be only 1 for our experiement.

Next using the Endpoints menu we press the Create endpoint button; using the same options as before.

As before, the endpoint is visible in Amazon SageMaker AI Studio; so we can validate the operation of the endpoint by navigating to the Test inference tab. The difference in behavior is that calling the endpoint the first time it is used (or after some time period) takes several seconds to response (this is what is referred to as the cold start).

The effects of the cold start can be mitigated by enabling provisioned concurrency as we saw when creating the endpoint configuration.

Python SDK

Roughly following Amazon’s example, we can write the code in a Amazon SageMaker AI Studio JupyterLab space to both create and deploy the model in much the same way we we did in the Amazon SageMaker AI section above.

Interestingly, this code uses the boto3 package instead of the sagemaker package.

Asynchronous Inference

Here we will be following the document How to create an Asynchronous Inference Endpoint.

As we observed earlier, Amazon SageMaker AI can only used to deploy real-time and serverless inference endpoints (not asynchronous inference endpoints); and we observed that Amazon SageMaker AI Studio only deploys real-time inference endpoints.

Looking at this Python SDK example, it is not much different than the serverless inference example above; the main differences are:

• Using an Amazon S3 bucket for input and output
• Validating the operation of the endpoint is done using the Python SDK

Got lazy and did not implement this one separately here.

Batch Transform

The document Batch transform for inference with Amazon SageMaker AI explains the use of batch transforms. Struggled to find a good end to walkthrough of how to implement this using the Python SDK; the gist, however, it that it does not involve creating an endpoint, rather it involves taking the model and using it as a transformer to input data from an Amazon S3 bucket to make batch predictions.