Add support for Deferred Functions

[heyitsaamir]

Deferred Function Calling

(Found out later that Pydantic AI also has a feature like this - https://ai.pydantic.dev/deferred-tools/)

Motivation

Normally, when a function call happens in an AI system, it looks like this:

sequenceDiagram
    participant CP as Chat Prompt
    participant LLM
    participant Fn as Function
    CP ->> LLM: Messages, system prompt<br/>function metadatas
    LLM ->> CP: Call Fn1 with arguments X
    CP ->> Fn: execute(X)
    Fn ->> CP: result (string)
    CP ->> LLM: Messages + Fn Result

Loading

This works for many use-cases. In this example, we're assuming F1 is some function that can be called and resolved in a reasonable amount of time (maybe a few seconds).

However, there are some scenarios where the function cannot be easily (or quickly) be executed. Examples include:

• Requires human input or approval - Humans are slow. You might have gone to get coffee and your approval is sticking around forever. Here's an example from Claude Code:

• The function needs to execute in Business Hours, or next month, or next year!
• The result of the function call can take time to execute. Consider having to send an order to a system that can take a day to process.

Proposal

In this PR, I'm proposing the ability for functions to return a DeferredResult in addition to a string. The idea is that if the ChatPrompt sees "DeferredResult", it saves that state, and stops executing.
Then, when the system is ready (aka gets an external signal) to continue, then it "resumes" and continues the ChatPrompt. The updated sequence diagram for a Deferrable Function looks like so:

sequenceDiagram
    participant Teams
    participant CP as Chat Prompt
    participant LLM
    participant Fn as Deferrable Function
    Teams ->> CP: send("buy some stock") (via Message handler)
    CP ->> LLM: Messages, system prompt<br/>function metadatas
    LLM ->> CP: Call Fn1 with arguments X
    CP ->> Fn: execute(X)
    Fn ->> Teams: Ask for approval
    Fn ->> CP: DeferredResult(state)
    CP ->> Teams: No response
    break When signal is received
        Teams ->> CP: "yes" go ahead<br/>resume(input)
        CP ->> Fn: Can this resume using input?
        Fn ->> CP: true
        CP ->> Fn: Resume(input)
        Fn ->> CP: Result after resuming (string)<br/>"the user approved the buying of the stock"
    end
    CP ->> LLM: Messages + Fn Result

Loading

Implementation

• We add a new type of Message that can be stored in Memory (which can store Message which is a union of different types of messages). This DeferredMessage basically stores the deferred state that the executable function can optionally return.
• A function can return DeferredResult now (on top of string)
• Added a "resume" function in ChatPrompt that can take in "Any" right now (Ideally I want to make this "Activity", but not totally sure). This resume function checks with memory if a what the deferred function is, then asks the deferred function if it can handle this incoming activity, and if it can, it converts this DeferredMessage with a FunctionMessage which is a normal function call result.

The sample currently shows an approval message that the LLM can invoke. It demonstrates two main scenarios:

1. In the first scenario, the LLM has access to get_approval and buy_stock. And then we prompt the LLM to call approve before buy-stock. This showcases the raw calling of a Deferrable function, and how it can be resumed when the user replies. I use a normal message here to showcase this, but honestly, using an adaptive card would drive the point in better.
2. In the second scenario, I wrap a normal function call (buy_stock) which is a regular function, into a deferred function. After doing this, when buy_stock is called, the LLM actually defers the calling of the actual buy stock function by first asking for an approval. This way you can't really fool the LLM by clever prompting.

I'm planning on moving some of the examples in the Sample to the main library as utilities.