Skip to content

Change auto-model default batch size#3036

Merged
charisk merged 1 commit intomainfrom
charisk/change-default-batch-size
Oct 31, 2023
Merged

Change auto-model default batch size#3036
charisk merged 1 commit intomainfrom
charisk/change-default-batch-size

Conversation

@charisk
Copy link
Copy Markdown
Contributor

@charisk charisk commented Oct 31, 2023

Lowering the batch size to allow higher number of requests required for new functionality.

Basically due to sources we now need double the number of LLM requests, and because we havent increased parallelism we need to half the number of candidates.

Checklist

N/A:

  • CHANGELOG.md has been updated to incorporate all user visible changes made by this pull request.
  • Issues have been created for any UI or other user-facing changes made by this pull request.
  • [Maintainers only] If this pull request makes user-facing changes that require documentation changes, open a corresponding docs pull request in the github/codeql repo and add the ready-for-doc-review label there.

@charisk charisk requested a review from starcke October 31, 2023 14:29
@charisk charisk requested a review from a team as a code owner October 31, 2023 14:29
Copy link
Copy Markdown
Contributor

@starcke starcke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@charisk charisk enabled auto-merge (squash) October 31, 2023 14:38
@charisk charisk merged commit 2988ace into main Oct 31, 2023
@charisk charisk deleted the charisk/change-default-batch-size branch October 31, 2023 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants