Skip to content

[Feature] Lambda model runner #331

Description

@achad4

While LLMs will not be hosted directly on lambdas, allowing customers to proxy their eval requests through Lambda invocations would expand the breadth of use cases of FMEval, without needing to maintain a vast set of model runner classes. Use cases include:

  1. Use cases where customers host LLMs hosted on bare metal (EC2, etc).
  2. Use cases where LLMs that are hosted on Sagemaker or Bedrock, but invoked through a web service that performs preprocessing / post-processing.

The Lambda model runner can function identically to the Bedrock model runner, replacing the Bedrock runtime call with a lambda invocation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions