Open Source Python Semantic Search Tools

SentenceTransformers is a Python framework for state-of-the-art sentence, text and image embeddings. The initial work is described in our paper Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. You can use this framework to compute sentence / text embeddings for more than 100 languages. These embeddings can then be compared e.g. with cosine-similarity to find sentences with a similar meaning. This can be useful for semantic textual similar, semantic search, or paraphrase mining. The framework is based on PyTorch and Transformers and offers a large collection of pre-trained models tuned for various tasks. Further, it is easy to fine-tune your own models. Our models are evaluated extensively and achieve state-of-the-art performance on various tasks. Further, the code is tuned to provide the highest possible speed.

Use Vim As IDE is a comprehensive configuration repository (by YangYangWithGnu) that guides you how to turn Vim into a full-fledged Integrated Development Environment (IDE). The project isn’t just a single plugin; it’s more like a curated set of plugins, configuration tips, and workflow suggestions to enable syntax highlighting, smart code completion, project navigation, semantic search, file-switching, build-integration, undo-history, templating and more—particularly geared toward C/C++ development, but with many ideas applicable more broadly. The documentation is long and detailed, walking users from the fundamentals of Vim configuration (.vimrc, plugin management) through higher-order capabilities like semantic navigation and project toolchain integration. The philosophy: Vim already offers “what you need when you need it; what you want when you want it” and this repo shows how to tap that potential.

txtai executes machine-learning workflows to transform data and build AI-powered semantic search applications. Traditional search systems use keywords to find data. Semantic search applications have an understanding of natural language and identify results that have the same meaning, not necessarily the same keywords. Backed by state-of-the-art machine learning models, data is transformed into vector representations for search (also known as embeddings). Innovation is happening at a rapid pace, models can understand concepts in documents, audio, images and more. Machine-learning pipelines to run extractive question-answering, zero-shot labeling, transcription, translation, summarization and text extraction. Cloud-native architecture that scales out with container orchestration systems (e.g. Kubernetes). Applications range from similarity search to complex NLP-driven data extractions to generate structured databases. The following applications are powered by txtai.