Show HN: PromptTools – open-source tools for evaluating LLMs and vector DBs 37 by krawfy | 5 comments on Hacker News. Hey HN! We’re Kevin and Steve. We’re building PromptTools ( https://ift.tt/WSbC0fZ ): open-source, self-hostable tools for experimenting with, testing, and evaluating LLMs, vector databases, and prompts. Evaluating prompts, LLMs, and vector databases is a painful, time-consuming but necessary part of the product engineering process. Our tools allow engineers to do this in a lot less time. By “evaluating” we mean checking the quality of a model's response for a given use case, which is a combination of testing and benchmarking. As examples: - For generated JSON, SQL, or Python, you can check that the output is actually JSON, SQL, or executable Python. - For generated emails, you can use another model to assess the quality of the generated email given some requirements, like whether or not the email is written professionally. - For a question-answering chatbot, you