AgentsComparatorWorker

from AID.selector.interface import AgentsComparatorWorker

Step 15: Performance Benchmarking (Objective Evaluation). Evaluates all agent predictions against a set of rigorous performance metrics. It computes Scoring accuracy, Lift charts, and Calibration curves to determine which agent provides the most reliable predictive power on unseen data.

Parameters

inputs : AgentsComparatorInput

Custom evaluation configuration. If None, resolves from ‘test.ml’ state.

**kwargs : dict

Additional parameters passed to the Comparator worker.

Inputs full schema can be obtained by ctx.workers.agents_comparator.spec with main params:

`inputs == None`	→	pipeline data context
`data != None`	→	custom data for agents_comparator assessment

Returns

AgentsComparatorWorker

The populated instance of AgentsComparatorWorker.

Populates attributes into the worker context:

`model`	→	ctx.workers.agents_comparator
`inputs`	→	ctx.workers.agents_comparator.spec
`result`	→	ctx.workers.agents_comparator.result
`state`	→	ctx.workers.agents_comparator.state
`report`	→	ctx.workers.agents_comparator.report
`stats`	→	ctx.workers.agents_comparator.stats
`fitter`	→	ctx.workers.agents_comparator.fitter

Registers data into the datasets context:

stats → ctx.datasets.stats.agents.comparison

Examples

Defining the Project Context (Control Plane) & Project Scope:

import aidflow as aid

ctx = aid.ProjectContext()

ctx.genesis(
    project      = 'test',
    input_data   = 'test.csv',
    target       = 'target',
    yy_mm        = 'decision_date',
    yy_mm_format = '%Y-%m-%d'
)
ctx.params()

Worker → Model → Fitter

Accessing worker’s inputs schema → parameters & types (shown in the display output):

ctx.workers.agents_comparator.spec

Accessing model’s fitter options and schema → parameters & types (shown in the display output):

ctx.workers.agents_comparator.fitters

Executing workers with defaults → auto provided by the context:

ctx.flow.process('data')         # 1. process workers 0-3 → data load - data stability
ctx.flow.process('encoding')     # 2. process workers 4-9 → train and apply encoding to train and test samples
ctx.flow.process('training')     # 3. process workers 10-13 → agents trainining on encoded train sample
ctx.workers.agents_comparator()  # 4. agents comparison on test sample

Once worker has been executed, thus its main model has been built, we have access to all populated attributes.

worker   = ctx.workers.agents_comparator
inputs   = worker.spec.inputs
result   = worker.result
state    = worker.state
features = worker.features
params   = worker.params
stats    = worker.stats
fitter   = worker.fitter  # jump to chosen internal fitter
...

View to model report of executed worker:

ctx.workers.agents_comparator.report.show()

Model → Fitter

Once worker has been executed, we can obtain internal fitter state and its parametrization (shown in the display output):

ctx.workers.agents_comparator.fitter

The fitted fitter provides all populated attributes.

fitter    = ctx.workers.agents_comparator.fitter
params_   = fitter.params
stats_    = fitter.stats

Direct choice and parametrization of internal fitter:

fx  = ctx.workers.agents_comparator.fitters[0]
prm = dict(
    alpha = 0.05,
    proba_bins = 15
)
agents_comparator = ctx.workers.agents_comparator(fx_choice = fx, fx_params = prm)

In the experimenting phase, when various parametrization settings might be under testing (note that internal fitter API (fx_config) can vary with respect to chosen fitter):

cfg = dict(
    target = 'target2',
    preprocess = True
)
agents_comparator = ctx.workers.agents_comparator(fx_config = cfg)

Attributes

All worker attributes return None prior its execution.

Attribute	Type	Description
`inputs`	`ModelInput`	Registered populated inputs containing execution timestamps and feature definitions.
`result`	`ModelResult`	Execution output payload containing model predictions and fit results.
`fitter`	`Fitter`	Underlying fitted estimator instance.
`state`	`ModelState`	Consolidated model state (fitter, choice, params, features, report, stats).
`report`	`DashboardInterface`	Interactive report interface. Call `report.show()` to render the visualization dashboard.
`stats`	`FitStats`	Performance metrics and training metadata gathered during fitting.
`features`	`TransResCols`	Feature schema dataclass required for downstream inference.
`params`	`tuple[FitParams, Parameters, ...]`	Tuple of fit settings, execution parameters, optionally transformer parameters.
`choice`	`RegGet.Choice`	Active configuration parameters including internal fitter choice (`fx_choice`).
`model_signature`	`str`	Unique hash identifier for the fitted model instance.

Execution Interface

Data Interface

Encoding Interface

Training Interface

Selection Interface

Inference Interface

Parameters

Returns

Examples

Worker → Model → Fitter

Model → Fitter

Attributes

​Parameters

​Returns

​Examples

​Worker → Model → Fitter

​Model → Fitter

​Attributes

Parameters

Returns

Examples

Worker → Model → Fitter

Model → Fitter

Attributes