Package com.embabel.agent.spi.streaming

Interface StreamingLlmOperations

  • All Implemented Interfaces:
    
public interface StreamingLlmOperations

    Streaming extension of LlmOperations for real-time LLM response processing.

    This SPI interface provides reactive streaming capabilities that support the API layer StreamingPromptRunner interfaces, enabling:

    • Real-time processing of LLM responses as they arrive

    • Streaming lists of objects from JSONL responses

    • Mixed content streams with both objects and LLM reasoning (thinking)

    • Progressive agent progress monitoring

    All streaming methods return Project Reactor Flux streams for integration with Spring WebFlux and other reactive frameworks.

    • Constructor Detail

    • Method Detail

      • generateStream

         Flux<String> generateStream(String prompt, LlmInteraction interaction, AgentProcess agentProcess, Action action)

        Generate streaming text in the context of an AgentProcess. Returns a Flux that emits text chunks as they arrive from the LLM.

        Parameters:
        prompt - Prompt to generate text from
        interaction - Llm options and tool callbacks to use, plus unique identifier
        agentProcess - Agent process we are running within
        action - Action we are running within if we are running within an action
        Returns:

        Flux of text chunks as they arrive from the LLM

      • generateStream

         abstract Flux<String> generateStream(List<Message> messages, LlmInteraction interaction, AgentProcess agentProcess, Action action)

        Generate streaming text from messages in the context of an AgentProcess. Returns a Flux that emits text chunks as they arrive from the LLM.

        Parameters:
        messages - messages in the conversation so far
        interaction - Llm options and tool callbacks to use, plus unique identifier
        agentProcess - Agent process we are running within
        action - Action we are running within if we are running within an action
        Returns:

        Flux of text chunks as they arrive from the LLM

      • createObjectStream

         abstract <O extends Any> Flux<O> createObjectStream(List<Message> messages, LlmInteraction interaction, Class<O> outputClass, AgentProcess agentProcess, Action action)

        Create a streaming list of objects from JSONL response in the context of an AgentProcess. Each line in the LLM response should be a valid JSON object matching the output class. Objects are emitted to the Flux as they are parsed from individual lines.

        Supports the API layer createObjectStream() method.

        Parameters:
        messages - messages in the conversation so far
        interaction - Llm options and tool callbacks to use, plus unique identifier
        outputClass - Class of the output objects
        agentProcess - Agent process we are running within
        action - Action we are running within if we are running within an action
        Returns:

        Flux of typed objects as they are parsed from the response

      • createObjectStreamIfPossible

         abstract <O extends Any> Flux<Result<O>> createObjectStreamIfPossible(List<Message> messages, LlmInteraction interaction, Class<O> outputClass, AgentProcess agentProcess, Action action)

        Try to create a streaming list of objects in the context of an AgentProcess. Return a Flux that may error if the LLM does not have enough information to create objects. Streaming equivalent of createObjectIfPossible().

        Parameters:
        messages - messages
        interaction - Llm options and tool callbacks to use, plus unique identifier
        outputClass - Class of the output objects
        agentProcess - Agent process we are running within
        action - Action we are running within if we are running within an action
        Returns:

        Flux of Result<O> objects, where each Result indicates success/failure for that object

      • createObjectStreamWithThinking

         abstract <O extends Any> Flux<StreamingEvent<O>> createObjectStreamWithThinking(List<Message> messages, LlmInteraction interaction, Class<O> outputClass, AgentProcess agentProcess, Action action)

        Create a streaming list of objects with LLM thinking content from mixed JSONL response. Supports both JSON object lines and //THINKING: lines in the LLM response. Returns StreamingEvent objects that can contain either typed objects or thinking content.

        This enables real-time visibility into LLM reasoning process alongside structured results. Supports the API layer createObjectStreamWithThinking() method.

        Parameters:
        messages - messages in the conversation so far
        interaction - Llm options and tool callbacks to use, plus unique identifier
        outputClass - Class of the output objects
        agentProcess - Agent process we are running within
        action - Action we are running within if we are running within an action
        Returns:

        Flux of StreamingEvent objects containing either objects or thinking content

      • doTransformStream

         abstract Flux<String> doTransformStream(List<Message> messages, LlmInteraction interaction, LlmRequestEvent<String> llmRequestEvent)

        Low level streaming transform, not necessarily aware of platform. Streams text chunks as they arrive from the LLM without platform mediation.

        Parameters:
        messages - messages in the conversation so far
        interaction - The LLM call options
        llmRequestEvent - Event already published for this request if one has been
        Returns:

        Flux of text chunks as they arrive from the LLM

      • doTransformObjectStream

         abstract <O extends Any> Flux<O> doTransformObjectStream(List<Message> messages, LlmInteraction interaction, Class<O> outputClass, LlmRequestEvent<O> llmRequestEvent)

        Low level object streaming transform, not necessarily aware of platform. Streams typed objects as they are parsed from JSONL response.

        Parameters:
        messages - messages in the conversation so far
        interaction - The LLM call options
        outputClass - Class of the output objects
        llmRequestEvent - Event already published for this request if one has been
        Returns:

        Flux of typed objects as they are parsed from the response

      • doTransformObjectStreamWithThinking

         abstract <O extends Any> Flux<StreamingEvent<O>> doTransformObjectStreamWithThinking(List<Message> messages, LlmInteraction interaction, Class<O> outputClass, LlmRequestEvent<O> llmRequestEvent)

        Low level mixed content streaming transform, not necessarily aware of platform. Streams both typed objects and thinking content from mixed JSONL response.

        Parameters:
        messages - messages in the conversation so far
        interaction - The LLM call options
        outputClass - Class of the output objects
        llmRequestEvent - Event already published for this request if one has been
        Returns:

        Flux of StreamingEvent objects containing either objects or thinking content