Skip to content

Add model evaluation run to store and view#3385

Merged
charisk merged 3 commits intomainfrom
charisk/model-evalution-run
Feb 22, 2024
Merged

Add model evaluation run to store and view#3385
charisk merged 3 commits intomainfrom
charisk/model-evalution-run

Conversation

@charisk
Copy link
Copy Markdown
Contributor

@charisk charisk commented Feb 21, 2024

This PR introduces the ModelEvalutionRun domain entity and wires up model evaluation run information to the modeling store and the model editor view. Some notes:

  • The ModelEvaluationRun interface is pretty minimal at the moment. I expect it to grow while we build more features (e.g. it might need to contain model packs).
  • The model evaluation run information is not yet persisted - it's only in memory (in the ModelingStore).
  • A new ModelEvaluator class has been added to contain logic around model evaluation in order to keep changes in model-editor-view minimal. The implementation is pretty minimal for now and will be fleshed out in the near future.

Checklist

N/A:

  • CHANGELOG.md has been updated to incorporate all user visible changes made by this pull request.
  • Issues have been created for any UI or other user-facing changes made by this pull request.
  • [Maintainers only] If this pull request makes user-facing changes that require documentation changes, open a corresponding docs pull request in the github/codeql repo and add the ready-for-doc-review label there.

@charisk charisk requested review from a team as code owners February 21, 2024 12:33
variantAnalysisId: number | undefined;
}

type ModelEvaluationStatus =
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering are we duplicating too much data from the variant analysis status?

Whenever the variant analysis run changes status from in-progress to some other state, we have to see that event and then update the state here. I worry it could lead to bugs if the two states get out of sync.

An alternative could be essentially ModelEvaluationStatus = "preparing" | "delegateToVariantAnalysisStatus" (but with better names).

Is this something you considered? Apologies I haven't been following the earlier PRs or issues closely so I likely missed any discussion there.

Copy link
Copy Markdown
Contributor Author

@charisk charisk Feb 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a valid point. I did consider it, but because VariantAnalysisStatus is an enum it's quite awkward - you can't just extend it. And since the variant analysis statuses don't really change (since they come from the back-end), it felt safe enough to re-define them. If we want to go down that route, I would suggest we change VariantAnalysisStatus to use strings rather than enums.

The way the type is created right now also comes with more flexibility about what states we want to expose for model evaluation runs.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What you've said makes perfect sense, but it wasn't quite what I was suggesting. Apologies I didn't make myself clear enough.

It being an enum does indeed make it hard to extend, and I don't have any objection to redefining the type for passing around transient values internally. My point was rather should we be storing that type anywhere for any length of time (even in memory, let alone to disk).

What I was trying to describe wasn't to "extend" the VariantAnalysisStatus type, but rather that one of the values in ModelEvaluationStatus means "I don't know. Go check the variant analysis status instead". By doing this it means we don't have to worry about keeping the value up to date whenever the variant analysis status changes, and instead we go calculate it when needed.

Therefore the logic for getting the model evaluation status would look like:

getModelEvalationStatus(): InternalModelEvaluationStatus {
  if (modelEvaluationRun.status === "preparing") {
    // It's preparing still and the variant analysis hasn't been triggered yet
    return "preparing";
  } else {
    // Ask the variant analysis manger for the status and process / return that
    const variantAnalysis = variantAnalysisManager.getVariantAnalysis(modelEvaluationRun.variantAnalysisId);
    if (variantAnalysis.status === "in_progress") {
      return "in_progress";
    } else if (variantAnalysis.status === "canceled") {
      return "canceled";
    } else if (...) {
      etc...
    }
  }
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I see! That's interesting - it's not something I considered. Do you mind if we discuss this separately (we have an issue to track work around domain entities)? Seems like something that would be good to discuss with @shati-patel when she's back, and ideally I wouldn't like it to block this PR.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, we can discuss on that issue. Although this PR is introducing domain entities itself, the code is still very easy to change at this stage. I'm happy to discuss on that issue instead.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've made some updates following our discussion.

const run: ModelEvaluationRunState = {
isPreparing: event.evaluationRun.isPreparing,

// TODO: Get variant analysis from id.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will do this in the next PR. It needs the VariantAnalysisManager wired in and it would make the PR bigger than it is so I left it out for now.

Copy link
Copy Markdown
Contributor

@shati-patel shati-patel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good ✨ Just a few minor comments!

Comment thread extensions/ql-vscode/src/model-editor/shared/model-evaluation-run-state.ts Outdated
Comment thread extensions/ql-vscode/src/model-editor/model-evaluator.ts
Comment thread extensions/ql-vscode/src/model-editor/model-evaluator.ts Outdated
Comment thread extensions/ql-vscode/src/view/model-editor/ModelEditor.tsx Outdated
Comment thread extensions/ql-vscode/src/view/model-editor/ModelEditor.tsx Outdated
Copy link
Copy Markdown
Contributor

@shati-patel shati-patel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good stuff 💪🏽 😀

@charisk charisk merged commit 948c1e2 into main Feb 22, 2024
@charisk charisk deleted the charisk/model-evalution-run branch February 22, 2024 16:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants