Prem AI introduces Prem-1B, an open-source Small Language Model built for Retrieval-Augmented Generation (RAG) tasks. Based on a decoder-only transformer architecture, it supports up to 8192 tokens. The model is available on Hugging Face under Apache 2.0.

Prem’s "SLM Journey Unveiled" details training a 1B parameter Small Language Model with 8K context length. It covers dataset challenges, Distributed Data Parallelism (DDP) with Ray, and optimization techniques for data partitioning and gradient synchronization.