Embedding for sentence

[Mukish45]

@abetlen I would like to thank you for the contribution you have made. I am very curious to know how the embedding is calculate when we pass sentence as an input.
embed_op = llm.create_embedding("I like play sports during rain")
Will it calculate for each word and average it or something else?

[chanlilong]

Would like to know what the embedding represents as well 😄

[ringohoffman]

@abetlen What is the significance of the single embedding returned from llm.create_embedding("Hello world!")? Since it is composed of 4 tokens, I was expecting 4 embeddings to be returned, but len(llm.create_embedding("Hello world!")["data"]) == 1... and the data changes when you provide either "Hello world! or "Hello world"... Is create_embedding() working as expected?

I am hoping to have access to the embeddings of the prompt at inference so that I can manipulate them. Can you advise on this?

[ringohoffman]

Related:

ggml-org/llama.cpp#3643
ggml-org/llama.cpp#3625
ggml-org/llama.cpp#3243
ggml-org/llama.cpp#2180
ggml-org/llama.cpp#1013

[KXXH]

I also want to know the answer to this question. The document does not seem to provide much information on how embedding works, so I think it is necessary to supplement these explanations.