Session

Create cloud native agents and extensions for LLMs

Vivian Hu - Second State

Slides

There is an enormous interest from the ecosystem to build LLM applications. But it is not yet clear the best practices to build those applications in cloud native environments. We will be presenting lessons learnt from real world use cases.

–

LLMs like ChatGPT are analogous to brains without memory, senses, or hands. APIs and plugins allow developers to add these key functionalities to LLMs and enable LLM applications. The popularity of OpenAI plugins and function templates has shown that there is a great demand for such services.

However, current LLM plugins are often built upon complex cloud native infrastructure designed for fully fledged services. The infrastructure is too heavyweight for plugins/serverless applications that mostly act as the lightweight links between LLMs and external services. Wasm has emerged as a compelling lightweight, portable, and secure runtime for this type of functions.

In this talk, we will discuss how the WasmEdge community (a CNCF sandbox project) has leveraged and built up the Wasm container infrastructure for LLM plugins. We will cover the major use cases and adoptions such as R&D management, DevRel, marketing automation, and training/learning. Then we will dive into the technology stack, as well as the Rust and JavaScript SDKs the community has created, that enables these LLM use cases.