DSSTNE - Amazon's library for building Deep Learning models
An open-source library for training and deploying deep learning recommendation models with sparse data at scale using multi-GPU support.
What is DSSTNE - Amazon's library for building Deep Learning models?
Amazon DSSTNE is an open-source deep learning library for training and deploying recommendation models with sparse inputs and outputs. It solves the problem of scaling large neural networks for production recommendation systems by using model-parallelism across multiple GPUs and optimizing for sparse datasets common in e-commerce and personalized recommendations.
Machine learning engineers and data scientists building large-scale recommendation systems, particularly those dealing with sparse data like e-commerce product recommendations, who need production-ready, GPU-accelerated model training and deployment.
Developers choose DSSTNE for its specialized optimization for sparse data, multi-GPU scaling capabilities enabling larger models than single-GPU alternatives, and production-focused design prioritizing speed and scalability over experimental flexibility.
Overview
Deep Scalable Sparse Tensor Network Engine (DSSTNE) is an Amazon developed library for building Deep Learning (DL) machine learning (ML) models
Use Cases
Best For
- Training deep learning recommendation models with sparse input features
- Scaling neural network training across multiple GPUs for large models
- Deploying production recommendation systems at e-commerce scale
- Handling sparse datasets common in personalized recommendations
- Optimizing GPU memory usage for large weight matrices
- Building model-parallel neural networks for recommendation workloads
Not Ideal For
- Research projects requiring flexible model architectures beyond sparse, fully connected networks
- Applications with dense data, such as image recognition or natural language processing
- Teams without access to multi-GPU infrastructure for model-parallel scaling
- Developers needing extensive community support or pre-built integrations with modern ML frameworks
Pros & Cons
Pros
Training and prediction scale out across multiple GPUs using model-parallelism, enabling near-linear scaling as shown in benchmarks for the MovieLens dataset.
Custom GPU kernels perform fast sparse computations without filling zeroes, optimized for recommendation datasets common in e-commerce.
Emphasizes speed and scale for real-world deployment, proven at Amazon's scale for personalized product recommendations.
Model-parallel scaling allows networks larger than a single GPU's memory capacity, handling weight matrices that exceed GPU limits.
Cons
Focused solely on sparse, fully connected layers for recommendations, lacking support for other architectures like CNNs or RNNs, as admitted in the philosophy emphasizing production over experimental flexibility.
Requires specific GPU infrastructure and detailed setup steps, with documentation pointing to multi-step guides that can be challenging for quick adoption.
The project has seen few recent updates, potentially lacking new features and active community support compared to mainstream frameworks like TensorFlow or PyTorch.
Fewer tutorials, pre-trained models, and integrations, making it harder to find resources or extend beyond core use cases mentioned in the README.
Frequently Asked Questions
Related Projects
Tensorflow - Open source software library for numerical computation using data flow graphsAn Open Source Machine Learning Framework for Everyone
PyTorch - Tensors and Dynamic neural networks in Python with strong GPU accelerationTensors and Dynamic neural networks in Python with strong GPU acceleration
kerasDeep Learning for humans
streamlitStreamlit — A faster way to build and share data apps.
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.