Mumbai, March 24: Apple’s latest innovation is making waves in the AI domain, albeit with some limitations. The iPhone 17 Pro has successfully run a staggering 400-billion parameter language model locally, a feat that seems almost unbelievable for a smartphone.
This demonstration originates from an open-source initiative known as Flash-MoE, shared by developer @anemll. Typically, models of this magnitude require over 200GB of memory to load, making it seemingly impossible for a device equipped with just 12GB of RAM.
However, the process employed here is quite innovative. Instead of loading the entire model into memory, the system retrieves segments from storage as needed. It also utilizes a Mixture of Experts configuration, activating only a fraction of the model at any given time. This unique approach is what enables it to function.
The main challenge lies in speed. The model operates at approximately 0.6 tokens per second, resulting in a wait of several seconds for a single word. This sluggishness can make even straightforward prompts feel frustratingly slow. Additionally, battery drain is a likely concern, which is expected given the intensive workload.
Despite these drawbacks, this development is noteworthy. While it may not be practical for everyday use at the moment, it indicates a potential future direction. The ability to run such a large model entirely on-device, without cloud dependency, was not even a topic of discussion until recently.
Currently, there remains a significant gap between what is technically feasible and what is practically usable. Smaller models continue to be the more sensible option. However, experiments like this offer a glimpse into the capabilities future smartphones may eventually possess.
My name is Ganpat Singh Choughan. I am an experienced content writer with 7 years of expertise in the field. Currently, I contribute to Daily Kiran, creating engaging and informative content across a variety of categories including technology, health, travel, education, and automobiles. My goal is to deliver accurate, insightful, and captivating information through my words to help readers stay informed and empowered.
Leave a Comment