Cleartext storage of HMAC signing key in Amazon SageMaker Python SDK ModelBuilder/Serve path

Summary

Amazon SageMaker Python SDK is an open-source library for training and deploying machine learning models on Amazon SageMaker. An issue exists where, under certain circumstances, the ModelBuilder/Serve component stores an HMAC signing key in cleartext as a container environment variable, which is returned in plaintext by SageMaker describe APIs.

Impact

When using ModelBuilder to build and deploy models with affected model servers (TorchServe, Multi-Model Server, TensorFlow Serving, SMD, or Triton), the SDK generates an HMAC secret key for model artifact integrity verification and stores it as the SAGEMAKER_SERVE_SECRET_KEY environment variable in the SageMaker model container configuration. This environment variable is returned in plaintext by the DescribeModel, DescribeEndpointConfig, and DescribeModelPackage APIs. A remote authenticated actor with permissions to call these describe APIs and S3 write access to the model artifact path could extract the key, forge valid integrity signatures for specially crafted model artifacts, and achieve code execution in inference containers with the SageMaker execution role's IAM permissions.

Impacted versions: >= v2.199.0 AND <= v2.257.1, >= v3.0.0 AND <= v3.7.1

Patches

This issue has been addressed in Amazon SageMaker Python SDK v2.257.2 and v3.8.0. AWS recommend upgrading to the latest version and rebuilding any models previously created with ModelBuilder using the updated SDK. Models created with affected versions may still have the HMAC key stored in their container environment variables until they are rebuilt with the patched SDK. Ensure any forked or derivative code is patched to incorporate the new fixes.

Workarounds

If upgrading is not immediately possible, users can manually remove the SAGEMAKER_SERVE_SECRET_KEY environment variable from existing SageMaker models by recreating the model without this variable in the container environment configuration.

References

If there any questions or comments about this advisory, contact AWS Security via the vulnerability reporting page or directly via email to aws-security@amazon.com. Please do not create a public GitHub issue.