Class: GgufInsightsConfigurationResolver
Accessors
ggufInsights
get ggufInsights(): GgufInsights
Returns
Defined in
gguf/insights/GgufInsightsConfigurationResolver.ts:21
Methods
resolveAndScoreConfig()
resolveAndScoreConfig(options: {
targetGpuLayers: number | "max";
targetContextSize: number;
embeddingContext: false;
flashAttention: false;
}, hardwareOverrides: {
getVramState: Promise<{
total: number;
free: number;
}>;
getRamState: Promise<{
total: number;
free: number;
}>;
llamaVramPaddingSize: number;
llamaGpu: false | "cuda" | "vulkan" | "metal";
llamaSupportsGpuOffloading: boolean;
}): Promise<{
compatibilityScore: number;
bonusScore: number;
totalScore: number;
resolvedValues: {
gpuLayers: number;
contextSize: number;
modelRamUsage: number;
contextRamUsage: number;
totalRamUsage: number;
modelVramUsage: number;
contextVramUsage: number;
totalVramUsage: number;
};
}>
Resolve the best configuration for loading a model and creating a context using the current hardware.
Specifying a targetGpuLayers
and/or targetContextSize
will ensure the resolved configuration matches those values, but note it can lower the compatibility score if the hardware doesn't support it.
Overriding hardware values it possible by configuring hardwareOverrides
.
Parameters
Parameter | Type | Description |
---|---|---|
options | object | |
options.targetGpuLayers ? | number | "max" | - |
options.targetContextSize ? | number | - |
options.embeddingContext ? | boolean | - |
options.flashAttention ? | boolean | - |
hardwareOverrides | object | |
hardwareOverrides.getVramState ? | - | |
hardwareOverrides.getRamState ? | - | |
hardwareOverrides.llamaVramPaddingSize ? | number | - |
hardwareOverrides.llamaGpu ? | false | "cuda" | "vulkan" | "metal" | - |
hardwareOverrides.llamaSupportsGpuOffloading ? | boolean | - |
Returns
Promise
<{ compatibilityScore
: number
; bonusScore
: number
; totalScore
: number
; resolvedValues
: { gpuLayers
: number
; contextSize
: number
; modelRamUsage
: number
; contextRamUsage
: number
; totalRamUsage
: number
; modelVramUsage
: number
; contextVramUsage
: number
; totalVramUsage
: number
; }; }>
compatibilityScore
compatibilityScore: number;
A number between 0
(inclusive) and 1
(inclusive) representing the compatibility score.
bonusScore
bonusScore: number;
A number starting at 0
with no upper limit representing the bonus score. For each multiplier of the specified contextSize
that the resolved context size is larger by, 1 bonus point is given.
totalScore
totalScore: number;
The total score, which is the sum of the compatibility and bonus scores.
resolvedValues
resolvedValues: {
gpuLayers: number;
contextSize: number;
modelRamUsage: number;
contextRamUsage: number;
totalRamUsage: number;
modelVramUsage: number;
contextVramUsage: number;
totalVramUsage: number;
};
The resolved values used to calculate the scores.
resolvedValues.gpuLayers
gpuLayers: number;
resolvedValues.contextSize
contextSize: number;
resolvedValues.modelRamUsage
modelRamUsage: number;
resolvedValues.contextRamUsage
contextRamUsage: number;
resolvedValues.totalRamUsage
totalRamUsage: number;
resolvedValues.modelVramUsage
modelVramUsage: number;
resolvedValues.contextVramUsage
contextVramUsage: number;
resolvedValues.totalVramUsage
totalVramUsage: number;
Defined in
gguf/insights/GgufInsightsConfigurationResolver.ts:35
scoreModelConfigurationCompatibility()
scoreModelConfigurationCompatibility(__namedParameters: {
contextSize: number;
embeddingContext: false;
flashAttention: false;
maximumFittedContextSizeMultiplier: 100;
}, __namedParameters: {
getVramState: Promise<{
total: number;
free: number;
}>;
getRamState: Promise<{
total: number;
free: number;
}>;
llamaVramPaddingSize: number;
llamaGpu: false | "cuda" | "vulkan" | "metal";
llamaSupportsGpuOffloading: boolean;
}): Promise<{
compatibilityScore: number;
bonusScore: number;
totalScore: number;
resolvedValues: {
gpuLayers: number;
contextSize: number;
modelRamUsage: number;
contextRamUsage: number;
totalRamUsage: number;
modelVramUsage: number;
contextVramUsage: number;
totalVramUsage: number;
};
}>
Score the compatibility of the model configuration with the current GPU and VRAM state. Assumes a model is loaded with the default "auto"
configurations. Scored based on the following criteria:
- The number of GPU layers that can be offloaded to the GPU (only if there's a GPU. If there's no GPU then by how small the model is)
- Whether all layers can be offloaded to the GPU (gives additional points)
- Whether the resolved context size is at least as large as the specified
contextSize
If the resolved context size is larger than the specified context size, for each multiplier of the specified contextSize
that the resolved context size is larger by, 1 bonus point is given in the bonusScore
.
maximumFittedContextSizeMultiplier
is used to improve the proportionality of the bonus score between models. Set this to any value higher than <max compared model context size> / contextSize
. Defaults to 100
.
contextSize
defaults to 4096
(if the model train context size is lower than this, the model train context size is used instead).
Parameters
Parameter | Type |
---|---|
__namedParameters | object |
__namedParameters.contextSize ? | number |
__namedParameters.embeddingContext ? | boolean |
__namedParameters.flashAttention ? | boolean |
__namedParameters.maximumFittedContextSizeMultiplier ? | number |
__namedParameters | object |
__namedParameters.getVramState ? | |
__namedParameters.getRamState ? | |
__namedParameters.llamaVramPaddingSize ? | number |
__namedParameters.llamaGpu ? | false | "cuda" | "vulkan" | "metal" |
__namedParameters.llamaSupportsGpuOffloading ? | boolean |
Returns
Promise
<{ compatibilityScore
: number
; bonusScore
: number
; totalScore
: number
; resolvedValues
: { gpuLayers
: number
; contextSize
: number
; modelRamUsage
: number
; contextRamUsage
: number
; totalRamUsage
: number
; modelVramUsage
: number
; contextVramUsage
: number
; totalVramUsage
: number
; }; }>
compatibilityScore
compatibilityScore: number;
A number between 0
(inclusive) and 1
(inclusive) representing the compatibility score.
bonusScore
bonusScore: number;
A number starting at 0
with no upper limit representing the bonus score. For each multiplier of the specified contextSize
that the resolved context size is larger by, 1 bonus point is given.
totalScore
totalScore: number;
The total score, which is the sum of the compatibility and bonus scores.
resolvedValues
resolvedValues: {
gpuLayers: number;
contextSize: number;
modelRamUsage: number;
contextRamUsage: number;
totalRamUsage: number;
modelVramUsage: number;
contextVramUsage: number;
totalVramUsage: number;
};
The resolved values used to calculate the scores.
resolvedValues.gpuLayers
gpuLayers: number;
resolvedValues.contextSize
contextSize: number;
resolvedValues.modelRamUsage
modelRamUsage: number;
resolvedValues.contextRamUsage
contextRamUsage: number;
resolvedValues.totalRamUsage
totalRamUsage: number;
resolvedValues.modelVramUsage
modelVramUsage: number;
resolvedValues.contextVramUsage
contextVramUsage: number;
resolvedValues.totalVramUsage
totalVramUsage: number;
Defined in
gguf/insights/GgufInsightsConfigurationResolver.ts:153
resolveModelGpuLayers()
resolveModelGpuLayers(gpuLayers?: number | "auto" | "max" | {
min: number;
max: number;
fitContext: {
contextSize: number;
embeddingContext: boolean;
};
}, __namedParameters?: {
ignoreMemorySafetyChecks: false;
getVramState: Promise<{
total: number;
free: number;
}>;
llamaVramPaddingSize: number;
llamaGpu: false | "cuda" | "vulkan" | "metal";
llamaSupportsGpuOffloading: boolean;
defaultContextFlashAttention: false;
}): Promise<number>
Parameters
Parameter | Type |
---|---|
gpuLayers ? | number | "auto" | "max" | { min : number ; max : number ; fitContext : { contextSize : number ; embeddingContext : boolean ; }; } |
__namedParameters ? | object |
__namedParameters.ignoreMemorySafetyChecks ? | boolean |
__namedParameters.getVramState ? | |
__namedParameters.llamaVramPaddingSize ? | number |
__namedParameters.llamaGpu ? | false | "cuda" | "vulkan" | "metal" |
__namedParameters.llamaSupportsGpuOffloading ? | boolean |
__namedParameters.defaultContextFlashAttention ? | boolean |
Returns
Promise
<number
>
Defined in
gguf/insights/GgufInsightsConfigurationResolver.ts:315
resolveContextContextSize()
resolveContextContextSize(contextSize: undefined | number | "auto" | {
min: number;
max: number;
}, __namedParameters: {
modelGpuLayers: number;
modelTrainContextSize: number;
flashAttention: false;
batchSize: number;
sequences: number;
getVramState: Promise<{
total: number;
free: number;
}>;
llamaGpu: false | "cuda" | "vulkan" | "metal";
ignoreMemorySafetyChecks: false;
isEmbeddingContext: false;
}): Promise<number>
Parameters
Parameter | Type |
---|---|
contextSize | undefined | number | "auto" | { min : number ; max : number ; } |
__namedParameters | object |
__namedParameters.modelGpuLayers | number |
__namedParameters.modelTrainContextSize | number |
__namedParameters.flashAttention ? | boolean |
__namedParameters.batchSize ? | number |
__namedParameters.sequences ? | number |
__namedParameters.getVramState ? | |
__namedParameters.llamaGpu ? | false | "cuda" | "vulkan" | "metal" |
__namedParameters.ignoreMemorySafetyChecks ? | boolean |
__namedParameters.isEmbeddingContext ? | boolean |
Returns
Promise
<number
>