Instill VDP (Versatile Data Pipeline)
? Instill VDP (Versatile Data Pipeline) is an open-source tool to seamlessly integrate AI to process unstructured data in the modern data stack
Infrastructure
🔮 Instill Core
✨ Make your unstructured data AI-ready FAST & build unique versatile AI-first applications EASILY. ✨
Quick start
There are a few ways to start exploring:
- Instill Cloud: Explore our public cloud powered by Instill Core with ready-to-use APIs, automations, AI Assistants and more. See the quickstart on how to get started.
- Self-hosting Instill Core: Follow the installation steps below or documentation for more details to build versatile AI applications locally.
What is Instill Core?
Instill Core is your end-to-end AI platform for data, pipeline and model orchestration.
Instill Core simplifies infrastructure hassle and encompasses these core features:
- DATA - Instill Artifact: Transforms unstructured data (documents, images, audio, video) into AI-ready formats.
- PIPELINE - Instill VDP: Quickly build any versatile AI-first APIs or automated workflows.
- MODEL - Instill Model: MLOps/LLMOps platform for model serving and monitoring.
What can you build?
- 📖 Parsing PDF Files to Markdown: Cookbook
- 🧱 Generating Structured Outputs from LLMs: Cookbook & Tutorial
- 🕸️ Web scraping & Google Search with Structured Insights: Tutorial & YouTube
- 🌱 Instance segmentation on microscopic plant stomata images: Cookbook
See Examples for more!
Installation
Prerequisites
| Operating System | Requirements and Instructions |
|---|---|
| macOS or Linux | Instill Core works natively |
| Windows | • Use Windows Subsystem for Linux (WSL2) • Install latest yq from GitHub Repository• Install latest Docker Desktop and enable WSL2 integration (tutorial) • (Optional) Install cuda-toolkit on WSL2 (NVIDIA tutorial) |
| All Systems | • Docker Engine v25 or later • Docker Compose v2 or later • Install latest stable Docker and Docker Compose |
Steps
Use stable release version
Execute the following commands to pull pre-built images with all the dependencies to launch:
$ git clone -b v0.48.0-beta https://github.com/instill-ai/instill-core.git && cd instill-core
# Launch all services
$ make all
[!NOTE] We have restructured our project repositories. If you need to access 🔮 Instill Core projects up to version
v0.13.0-beta, please refer to the instill-ai/deprecated-core repository.
Use the latest version for local development
Execute the following commands to build images with all the dependencies to launch:
$ git clone https://github.com/instill-ai/instill-core.git && cd instill-core
# Launch all services
$ make latest PROFILE=all
[!IMPORTANT] Code in the main branch tracks under-development progress towards the next release and may not work as expected. If you are looking for a stable alpha version, please use latest release.
🚀 That's it! Once all the services are up with health status, the UI is ready to go at http://localhost:3000. Please find the default login credentials in the documentation.
To shut down all running services:
$ make down
Explore the documentation to discover all available deployment options.
Deployment
You can either self-host 🔮 Instill Core or choose our fully managed ☁️ Instill Cloud to get access to all the latest features. Visit the Deployment Overview for more details.
Client Access
- 📺 Instill Console
- ⌨️ Instill CLI
- 📦 Instill SDK:
- Python SDK
- TypeScript SDK
- Stay tuned, as more SDKs are on the way!
Documentation
Please visit our official documentation for more.
Additional resources:
Contributing
We welcome contributions from our community! Checkout the methods below:
-
Cookbooks: Help us create helpful pipelines and guides for the community. Visit our Cookbook repository to get started.
-
Issues: Contribute to improvements by raising tickets using templates here or discuss in existing ones you think you can help with.
Community Standards
We are committed to maintaining a respectful and welcoming atmosphere for all contributors. Before contributing, please read:
Support
Get help by joining our Discord community where you can post any questions on our #ask-for-help channel.
Contributors ✨
Thank you to all these wonderful people (emoji key):
This project follows the all-contributors specification. Contributions of any kind welcome!
License
See the LICENSE file for licensing information.