Within the field of artificial intelligence, “small language models” have gained significant traction lately due to their ability to operate locally on a device rather than requiring cloud-based data center-grade computers. On Wednesday, Apple unveiled OpenELM, a collection of minuscule AI language models that are available as open source and small enough to run on a smartphone. For now, they’re primarily proof-of-concept research models, but they might serve as the foundation for Apple’s on-device AI products in the future.
Apple’s new AI models, collectively named OpenELM for “Open-source Efficient Language Models,” are currently available on the Hugging Face under an Apple Sample Code License. Since there are some restrictions in the license, it may not fit the commonly accepted definition of “open source,” but the source code for OpenELM is available.
A similar goal is pursued by Microsoft’s Phi-3 models, which we discussed on Tuesday. These models are small, locally executable AI models that can comprehend and process language to a reasonable degree. Although Apple’s OpenELM models range in size from 270 million to 3 billion parameters across eight different models, Phi-3-mini has 3.8 billion parameters.
By contrast, OpenAI’s GPT-3 from 2020 shipped with 175 billion parameters, and Meta’s largest model to date, the Llama 3 family, has 70 billion parameters (a 400 billion version is on the way). Although parameter count is a useful indicator of the complexity and capability of AI models, recent work has concentrated on making smaller AI language models just as capable as larger ones were a few years ago.
Eight OpenELM models are available in two flavors: four that are “pretrained,” or essentially a next-token version of the model in its raw form, and four that are “instructional-tuned,” or optimized for instruction following, which is more suitable for creating chatbots and AI assistants:
The maximum context window in OpenELM is 2048 tokens. The models were trained using datasets that are publicly available, including RefinedWeb, a subset of RedPajama, a version of PILE that has had duplications removed, and a subset of Dolma v1.6, which contains, according to Apple, roughly 1.8 trillion tokens of data. AI language models process data using tokens, which are broken representations of the data.
According to Apple, part of its OpenELM approach is a “layer-wise scaling strategy” that distributes parameters among layers more effectively, supposedly saving computational resources and enhancing the model’s performance even with fewer tokens used for training. This approach has allowed OpenELM to achieve 2.36 percent accuracy gain over Allen AI’s OLMo 1B (another small language model) with half as many pre-training tokens needed, according to Apple’s published white paper.
In addition, Apple made the code for CoreNet, the library it used to train OpenELM, publicly available. Notably, this code includes reproducible training recipes that make it possible to duplicate the weights, or neural network files—something that has not been seen in a major tech company before. Transparency, according to Apple, is a major objective for the organization: “The reproducibility and transparency of large language models are crucial for advancing open research, ensuring the trustworthiness of results, and enabling investigations into data and model biases, as well as potential risks.”
By releasing the source code, model weights, and training materials, Apple says it aims to “empower and enrich the open research community.” However, it also cautions that since the models were trained on publicly sourced datasets, “there exists the possibility of these models producing outputs that are biased, or objectionable in response to user prompts.”
Though the company may hire Google or OpenAI to handle more complex, off-device AI processing to give Siri a much-needed boost, Apple has not yet integrated this new wave of AI language model capabilities into its consumer devices. It is anticipated that the upcoming iOS 18 update—which is expected to be revealed in June at WWDC—will include new AI features that use on-device processing to ensure user privacy.