MELvan Frequently Asked Questions

What does MELvan stand for?

MELvan is a wordplay on Monitoring, Evaluation, and Learning.

What is MELvan?

MELvan is an AI tool that helps program leaders capture evaluation-relevant moments as they happen — a short note, a photo, a quick observation during a meeting or in the field. MELvan then classifies each entry against a program's Results-Based Management (RBM) structure called logic model (output, outcome, ultimate goal), assigns a confidence score, and produces a short rationale.

Read the product guide in our Github (open access) repo for a high-level summary of the system and roles.

Why was MELvan made?

Unintended and emergent program outcomes are crucial for accountability and learning, but they're easily lost without the right tools and processes. Program leaders often notice an evaluation-relevant moment as it happens — an unintended result mentioned in a meeting, an aside in an informal conversation — but don't have time to reflect on it or document it on the spot. The moment passes, and usually so does the memory of the emergent or unintended outcome by the time an evaluator asks about it.

How is MELvan integrated into everyday work of leaders?

A leader captures a short note or photo during a meeting or daily work. MELvan classifies the entry and logs it with a confidence score and rationale. At an agreed upon cadence (e.g.: quarterly), program leadership and an external evaluation consultant review the log together, confirm or correct MELvan's classification, and feed accepted entries into the program's monitoring, learning, and evaluation work.

How does MELvan change the way evaluation data gets collected?

Traditional data collection is essential but periodic. MELvan adds a complementary layer: continuous, leader-driven capture that surfaces emergent or unintended outcomes as they unfold, rather than waiting for the next scheduled evaluation exercise. It's designed to lower the effort required to capture and classify evaluation-relevant information in daily leadership workflows, while deferring deeper interpretation to dedicated review sessions.

Who developed MELvan?

MELvan is a co-designed concept and product developed collaboratively by the Precision Child and Youth Mental Health (PCYMH) Collaboratory at CHEO Research Institute — Christina Honeywell and Kathleen Pajer — and Vitus Consulting — Roxana Salehi. AI product architect for the open-source version of MELvan is Martin Mills.

What is the methodology behind MELvan?

MELvan's knowledge base combines open-source evaluation literature with program-specific information. MELvan uses a combination of deterministic (rules-based) and non-deterministic (AI) approaches to classify reported statements against the logic model. It provides a confidence score for each classification, as well as a rationale. Classification, mapping, and their rationales are fully automated processes; the review session is where an evaluation team makes the interpretive call as to their quality. During the review, MELvan's outputs are assessed by the reviewers and accepted, modified, or rejected, and that feedback becomes an input to MELvan's learning mechanism to improve its classification model, meaning that over time it should become progressively more accurate.

Who does what in MELvan?

A Reporter submits an entry — text, a voice recording, or a photo — and can add extra context up until it's reviewed.

MELvan summarizes the entry, classifies it as an Activity, Output, or Outcome, and proposes one or more mappings to the program's Logic Model, each with its own confidence score and rationale.

A Reviewer accepts a correct mapping, corrects it, or makes a correction right in the tool itself — nothing becomes Evidence until a Reviewer has approved at least one mapping.

An Administrator manages the program's Reporter and Reviewer accounts and its Logic Model (e.g. add a new version).

An Owner creates Administrators and can suspend accounts. Owners have read-only visibility into the workspaces (e.g. logic model or other source documents).

How is MELvan evaluated as a tool?

MELvan is assessed against five quality measures:

First-review approval: whether the Reviewer's first action accepts the Statement and MELvan's associated decisions without a substantive correction. This is the primary outcome MELvan is intended to improve.

Correct level: whether MELvan correctly distinguishes Activity, Output, or Outcome.

Correct Outcome direction: whether an Outcome is correctly identified as Positive or Negative.

Correct mapping: whether MELvan identifies the correct existing Logic Model element or elements, or correctly determines that there is no suitable existing match. An undisputed mapping or no-match decision that the reviewer accepted is considered correct.

No hallucination/toxic content: whether the summary, rationale, criteria, terminology, tone, and other language are accurate, supported, appropriate, and free from invented details or overstated claims.

What kind of values are behind MELvan?

Traceability: every log entry carries a rationale, so the reasoning behind a classification or mapping is visible, not a black box.

Open science: the source code and supporting resources are publicly available

Resource-conscious design: constrained AI requests, transient voice processing, image normalization, temporary-file cleanup, bounded logging, and configurable infrastructure capacity all limit unnecessary computation and data retention. Environmental impact, however, isn't currently measured as we are still thinking through the best way to do this.

Is MELvan suitable for my program?

Most programs will benefit from being able to quickly capture program results on the go. However, for MELvan to be useful for your specific program, and for it to provide accurate classification tailored to your needs, you need to give it your specific program logic model. Therefore, you need a program logic model as a pre-condition to use MELvan.

If your program logic model isn't built yet: that's your starting point. If you need help with that, contact *Roxana Salehi (roxana@vitus.ca).*

If you already have a logic model: contact Roxana Salehi (*roxana@vitus.ca)* to request access to the demo version in order to assess MELvan's suitability for your purposes. Access is typically given for 2-4 weeks.

Newly created administrator accounts can configure the logic model for their workspace — they have the choice to either clone the 'demo' logic model, a generic logic model for an imaginary mental health program for adults, or they can create their own logic model which must comply with the required structure.

How can I get a version of MELvan for my program?

If you want to use MELvan on an ongoing basis, then we recommend you discuss its use with your organization and ensure it complies with any existing responsible AI frameworks in your organization.

If your technology team is able to support installation of cloud-based applications for Azure (currently, Azure is the only supported Cloud provider), you are invited to fork our open access GitHub to install your own version on your own Azure instance.

Otherwise, contact Vitus Consulting if (a) you require professional services to install MELvan on your own infrastructure, or (b) if your organization has determined that it can use a third-party (i.e. Vitus's own) version of MELvan. Consulting and/or subscription fees will apply.

Are there any free open access resources about MELvan?

Yes.

The open-access case study: Salehi, R., Honeywell, C., & Pajer, K. (2026). MELvan: An AI-Assisted Approach to Capturing Program Impact. Zenodo. https://doi.org/10.5281/zenodo.18902209.

MELvan's underlying evaluation methodology, RECAP: https://doi.org/10.5281/zenodo.18456041.

Demo — http://melvan.vitus.global

GitHub Repository to fork (copy) the source code: