In an effort to improve developer experiences, Fastly has launched the Fastly AI Accelerator, which aims to lower costs and improve performance in apps that use large language models (LLMs). The enormous volume of identical prompts that well-known AI systems process presents issues that the new technique is intended to solve.
“AI technologies in general, and large language models in particular, are rapidly changing the technology sector and the way millions of people around the world—developers included—work on a daily basis,” said RedMonk Principal Analyst Stephen O’Grady. Although the largest models receive a lot of attention, he pointed out that developers and businesses are starting to take medium and smaller models into account more frequently because to their affordability, quicker training cycles, and compatibility with a wider range of hardware profiles.
Semantic caching is used by Fastly’s new AI Accelerator to lessen the number of API calls required to retrieve identical data, eliminating latency and associated expenses. By using a customized API gateway, this method significantly improves performance by utilising Fastly’s Edge Cloud Platform and its cutting-edge caching technology. The AI Accelerator will accommodate more models in the future, and it currently supports ChatGPT.
“We’re always listening to developers at Fastly to understand what they’re excited about and what their biggest pain points are,” said Anil Dash, vice president of developer experience at the company. With Fastly AI Accelerator, developers can concentrate on what makes their apps and websites distinctive and what keeps their customers satisfied, as the program streamlines and expedites the experience of their favorite LLMs.
With the help of Fastly’s high-performance edge platform, the semantic caching feature of the Fastly AI Accelerator delivers pre-canned answers to frequently asked or comparable queries. Processes are streamlined as a result of not having to ask the AI supplier for the same information many times.
It usually only takes one line of code modification for developers to update their applications to use a new API endpoint in order to incorporate the Fastly AI Accelerator. Beyond standard caching strategies, the solution is focused on comprehending the context of incoming requests and providing comparable results when queries are similar.
Apart from introducing the AI Accelerator, Fastly is also expanding its free account tier to make the platform more accessible to developers. With the help of several features including large memory and storage allocations, access to Fastly’s Content Delivery Network, and multiple security technologies like TLS and ongoing DDoS mitigation, developers can quickly build up new websites, create apps, or launch services.
Fastly hopes to enable developers to create online experiences that are more effective, safe, and interesting through these initiatives. By focusing on utilizing its edge cloud platform, the company hopes to increase performance and control costs by addressing some of the common issues experienced by developers who use huge language models.
“Whether it’s to lower costs, to shorten training cycles, or to run on more limited hardware profiles, they’re an increasingly important option,” said Stephen O’Grady, summarizing the continuing trend and underlining the growing significance of medium and smaller models. This emphasizes how crucial products like Quickly AI Accelerator are in the rapidly changing fields of artificial intelligence and software development.