mac_ai remote_ramalama_run_model

Runs a model with ramalama, on a remote host

Parameters

base_work_dir

• The base directory where to store things

path

• The path to the llama-server binary

port

• The port number on which llama-cpp should listen

name

• The name of the model to run

env

• The env values to set before running ramalama

ngl

• Number of layers to store in VRAM

• default value: 99

device

• Name of the device to pass to the container

unload

• If True, unloads (stops serving) this model