Package com.embabel.agent.spi.streaming

Interface StreamingLlmOperations

  • All Implemented Interfaces:
    
public interface StreamingLlmOperations

    Streaming extension of LlmOperations for real-time LLM response processing.

    This SPI interface provides reactive streaming capabilities that support the API layer StreamingPromptRunner interfaces, enabling:

    • Real-time processing of LLM responses as they arrive

    • Streaming lists of objects from JSONL responses

    • Mixed content streams with both objects and LLM reasoning (thinking)

    • Progressive agent progress monitoring

    All streaming methods return Project Reactor Flux streams for integration with Spring WebFlux and other reactive frameworks.

    • Constructor Detail

    • Method Detail

      • generateStream

         Flux<String> generateStream(String prompt, LlmInteraction interaction, AgentProcess agentProcess, Action action)

        Generate streaming text in the context of an AgentProcess. Returns a Flux that emits text chunks as they arrive from the LLM.

        Parameters:
        prompt - Prompt to generate text from
        interaction - Llm options and tool callbacks to use, plus unique identifier
        agentProcess - Agent process we are running within
        action - Action we are running within if we are running within an action
        Returns:

        Flux of text chunks as they arrive from the LLM

      • generateStream

         abstract Flux<String> generateStream(List<Message> messages, LlmInteraction interaction, AgentProcess agentProcess, Action action)

        Generate streaming text from messages in the context of an AgentProcess. Returns a Flux that emits text chunks as they arrive from the LLM.

        Parameters:
        messages - messages in the conversation so far
        interaction - Llm options and tool callbacks to use, plus unique identifier
        agentProcess - Agent process we are running within
        action - Action we are running within if we are running within an action
        Returns:

        Flux of text chunks as they arrive from the LLM

      • createObjectStream

         abstract <O extends Any> Flux<O> createObjectStream(List<Message> messages, LlmInteraction interaction, Class<O> outputClass, AgentProcess agentProcess, Action action)

        Create a streaming list of objects from JSONL response in the context of an AgentProcess. Each line in the LLM response should be a valid JSON object matching the output class. Objects are emitted to the Flux as they are parsed from individual lines.

        Supports the API layer createObjectStream() method.

        Parameters:
        messages - messages in the conversation so far
        interaction - Llm options and tool callbacks to use, plus unique identifier
        outputClass - Class of the output objects
        agentProcess - Agent process we are running within
        action - Action we are running within if we are running within an action
        Returns:

        Flux of typed objects as they are parsed from the response

      • createObjectStreamIfPossible

         abstract <O extends Any> Flux<Result<O>> createObjectStreamIfPossible(List<Message> messages, LlmInteraction interaction, Class<O> outputClass, AgentProcess agentProcess, Action action)

        Try to create a streaming list of objects in the context of an AgentProcess. Return a Flux that may error if the LLM does not have enough information to create objects. Streaming equivalent of createObjectIfPossible().

        Parameters:
        messages - messages
        interaction - Llm options and tool callbacks to use, plus unique identifier
        outputClass - Class of the output objects
        agentProcess - Agent process we are running within
        action - Action we are running within if we are running within an action
        Returns:

        Flux of Result<O> objects, where each Result indicates success/failure for that object

      • createObjectStreamWithThinking

         abstract <O extends Any> Flux<StreamingEvent<O>> createObjectStreamWithThinking(List<Message> messages, LlmInteraction interaction, Class<O> outputClass, AgentProcess agentProcess, Action action)

        Create a streaming list of objects with LLM thinking content from mixed JSONL response. Supports both JSON object lines and //THINKING: lines in the LLM response. Returns StreamingEvent objects that can contain either typed objects or thinking content.

        This enables real-time visibility into LLM reasoning process alongside structured results. Supports the API layer createObjectStreamWithThinking() method.

        Parameters:
        messages - messages in the conversation so far
        interaction - Llm options and tool callbacks to use, plus unique identifier
        outputClass - Class of the output objects
        agentProcess - Agent process we are running within
        action - Action we are running within if we are running within an action
        Returns:

        Flux of StreamingEvent objects containing either objects or thinking content

      • doTransformStream

         abstract Flux<String> doTransformStream(List<Message> messages, LlmInteraction interaction, LlmRequestEvent<String> llmRequestEvent, AgentProcess agentProcess, Action action)

        Low level streaming transform with optional platform context. Streams text chunks as they arrive from the LLM. When agentProcess is provided, tools are resolved from ToolGroups and decorated.

        Parameters:
        messages - messages in the conversation so far
        interaction - The LLM call options
        llmRequestEvent - Event already published for this request if one has been
        agentProcess - Optional agent process for tool resolution and decoration
        action - Optional action context for tool decoration
        Returns:

        Flux of text chunks as they arrive from the LLM

      • doTransformObjectStream

         abstract <O extends Any> Flux<O> doTransformObjectStream(List<Message> messages, LlmInteraction interaction, Class<O> outputClass, LlmRequestEvent<O> llmRequestEvent, AgentProcess agentProcess, Action action)

        Low level object streaming transform with optional platform context. Streams typed objects as they are parsed from JSONL response. When agentProcess is provided, tools are resolved from ToolGroups and decorated.

        Parameters:
        messages - messages in the conversation so far
        interaction - The LLM call options
        outputClass - Class of the output objects
        llmRequestEvent - Event already published for this request if one has been
        agentProcess - Optional agent process for tool resolution and decoration
        action - Optional action context for tool decoration
        Returns:

        Flux of typed objects as they are parsed from the response

      • doTransformObjectStreamWithThinking

         abstract <O extends Any> Flux<StreamingEvent<O>> doTransformObjectStreamWithThinking(List<Message> messages, LlmInteraction interaction, Class<O> outputClass, LlmRequestEvent<O> llmRequestEvent, AgentProcess agentProcess, Action action)

        Low level mixed content streaming transform with optional platform context. Streams both typed objects and thinking content from mixed JSONL response. When agentProcess is provided, tools are resolved from ToolGroups and decorated.

        Parameters:
        messages - messages in the conversation so far
        interaction - The LLM call options
        outputClass - Class of the output objects
        llmRequestEvent - Event already published for this request if one has been
        agentProcess - Optional agent process for tool resolution and decoration
        action - Optional action context for tool decoration
        Returns:

        Flux of StreamingEvent objects containing either objects or thinking content