-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Add provider model and token limit overrides to ProviderConfig #966
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
f993631
09c17f7
c434f4d
56af92f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1528,6 +1528,43 @@ public class ProviderConfig | |
| /// </summary> | ||
| [JsonPropertyName("headers")] | ||
| public IDictionary<string, string>? Headers { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Well-known model ID used to look up agent configuration (tools, prompts, | ||
| /// reasoning behavior) and default token limits from the capability catalog. | ||
| /// Useful for fine-tuned models that should inherit the configuration of a | ||
| /// known base model. | ||
| /// Defaults to the session's configured model (see <see cref="SessionConfig.Model"/>) | ||
| /// when not explicitly set. | ||
| /// </summary> | ||
| [JsonPropertyName("modelId")] | ||
| public string? ModelId { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Model identifier sent to the provider API for inference. | ||
| /// Use this when the name your provider knows (e.g. an Azure deployment name | ||
| /// or a custom fine-tune name) differs from the well-known model ID used for | ||
| /// configuration lookup. | ||
| /// Defaults to the session's configured model (see <see cref="SessionConfig.Model"/>) | ||
| /// when not explicitly set. | ||
| /// </summary> | ||
| [JsonPropertyName("wireModel")] | ||
| public string? WireModel { get; set; } | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Out of the three: what's the meaning behind ProviderConfig.ModelId having an "Id" suffix and the other two not?
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I figured the "ID" more strongly implied that the value was identifying a well-known model kind. We could also consider |
||
|
|
||
| /// <summary> | ||
| /// Maximum number of tokens allowed in the prompt for a single LLM API request. | ||
| /// Used by the runtime to trigger conversation compaction before sending a request | ||
| /// when the prompt (system message, history, tool definitions, user message) exceeds this limit. | ||
| /// </summary> | ||
| [JsonPropertyName("maxPromptTokens")] | ||
| public int? MaxPromptTokens { get; set; } | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should this be called MaxInputTokens instead of MaxPromptTokens? Does this include cached tokens? Same question as above... is this about one request or across a sequence of calls?
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Per-request, but not sent to the API. The runtime uses this internally to decide when to truncate or compact conversation history before each LLM call. "Prompt" here means everything sent to the model in one request: system message, full conversation history up to that point, tool definitions, and the new user message. Cached tokens are counted toward the limit. The name matches the upstream CAPI
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd prefer MaxInputTokens. That's the more modern terminology, right? And it maps to what we show in the CLI UI?
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We could change this; it would just require changing the runtime representation as well (including the
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Changing the sdk name wouldn't, right? Only if we also wanted to change the wire name?
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh, if we just changed the public API but configured it to serialize with the |
||
|
|
||
| /// <summary> | ||
| /// Maximum number of tokens the model can generate in a single response. | ||
| /// When hit, the model stops generating and returns a truncated response. | ||
| /// </summary> | ||
| [JsonPropertyName("maxOutputTokens")] | ||
| public int? MaxOutputTokens { get; set; } | ||
| } | ||
|
MackinnonBuck marked this conversation as resolved.
|
||
|
|
||
| /// <summary> | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So there's:
SessionConfig.Model
ProviderConfig.ModelId
ProviderConfig.WireModel
Help me understand the relationship? If I specify SessionConfig.Model, it's used as the default for both options on ProviderConfig, and those options on provider config then represent the two different groupings in which a model would be used, such that I can override one of them? Is there any situation where I would specify the same model ID for both ModelId and WireModel?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If only
ProviderConfig.ModelIdis specified, then that controls multiple things:If the model provider recognizes a model name that doesn't match the model ID known by the runtime, then
ProviderConfig.WireModelcan specify that.SessionConfig.Modelacts a default in case neither option is specified.It has the same effect as just specifying
ModelId, so it's not really necessary.