单目3D初始代码
This commit is contained in:
zhao.zhu
395
docs/en/datasets/explorer/api.md
Executable file
395
docs/en/datasets/explorer/api.md
Executable file
@@ -0,0 +1,395 @@
|
||||
---
|
||||
comments: true
|
||||
description: Explore the Ultralytics Explorer API for dataset exploration with SQL queries, vector similarity search, and semantic search. Learn installation and usage tips.
|
||||
keywords: Ultralytics, Explorer API, dataset exploration, SQL queries, similarity search, semantic search, Python API, embeddings, data analysis
|
||||
---
|
||||
|
||||
# Ultralytics Explorer API
|
||||
|
||||
!!! warning "Community Note ⚠️"
|
||||
|
||||
As of **`ultralytics>=8.3.10`**, Ultralytics Explorer support is deprecated. Similar (and expanded) dataset exploration features are available in [Ultralytics Platform](https://platform.ultralytics.com/).
|
||||
|
||||
## Introduction
|
||||
|
||||
<a href="https://colab.research.google.com/github/ultralytics/ultralytics/blob/main/docs/en/datasets/explorer/explorer.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a>
|
||||
The Explorer API is a Python API for exploring your datasets. It supports filtering and searching your dataset using SQL queries, vector similarity search, and semantic search.
|
||||
|
||||
<p align="center">
|
||||
<br>
|
||||
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/3VryynorQeo?start=279"
|
||||
title="YouTube video player" frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
<br>
|
||||
<strong>Watch:</strong> Ultralytics Explorer API Overview
|
||||
</p>
|
||||
|
||||
## Installation
|
||||
|
||||
Explorer depends on external libraries for some of its functionality. These are automatically installed when you use Explorer. To manually install these dependencies, use the following command:
|
||||
|
||||
```bash
|
||||
pip install ultralytics[explorer]
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
from ultralytics import Explorer
|
||||
|
||||
# Create an Explorer object
|
||||
explorer = Explorer(data="coco128.yaml", model="yolo26n.pt")
|
||||
|
||||
# Create embeddings for your dataset
|
||||
explorer.create_embeddings_table()
|
||||
|
||||
# Search for similar images to a given image/images
|
||||
df = explorer.get_similar(img="path/to/image.jpg")
|
||||
|
||||
# Or search for similar images to a given index/indices
|
||||
df = explorer.get_similar(idx=0)
|
||||
```
|
||||
|
||||
!!! note
|
||||
|
||||
[Embeddings](https://www.ultralytics.com/glossary/embeddings) table for a given dataset and model pair is only created once and reused. These use [LanceDB](https://lancedb.github.io/lancedb/) under the hood, which scales on-disk, so you can create and reuse embeddings for large datasets like COCO without running out of memory.
|
||||
|
||||
In case you want to force update the embeddings table, you can pass `force=True` to `create_embeddings_table` method.
|
||||
|
||||
You can directly access the LanceDB table object to perform advanced analysis. Learn more about it in the [Working with Embeddings Table section](#4-working-with-embeddings-table)
|
||||
|
||||
## 1. Similarity Search
|
||||
|
||||
Similarity search is a technique for finding similar images to a given image. It is based on the idea that similar images will have similar embeddings. Once the embeddings table is built, you can get run semantic search in any of the following ways:
|
||||
|
||||
- On a given index or list of indices in the dataset: `exp.get_similar(idx=[1,10], limit=10)`
|
||||
- On any image or list of images not in the dataset: `exp.get_similar(img=["path/to/img1", "path/to/img2"], limit=10)`
|
||||
|
||||
In case of multiple inputs, the aggregate of their embeddings is used.
|
||||
|
||||
You get a pandas DataFrame with the `limit` number of most similar data points to the input, along with their distance in the embedding space. You can use this dataset to perform further filtering.
|
||||
|
||||
!!! example "Semantic Search"
|
||||
|
||||
=== "Using Images"
|
||||
|
||||
```python
|
||||
from ultralytics import Explorer
|
||||
|
||||
# create an Explorer object
|
||||
exp = Explorer(data="coco128.yaml", model="yolo26n.pt")
|
||||
exp.create_embeddings_table()
|
||||
|
||||
similar = exp.get_similar(img="https://ultralytics.com/images/bus.jpg", limit=10)
|
||||
print(similar.head())
|
||||
|
||||
# Search using multiple indices
|
||||
similar = exp.get_similar(
|
||||
img=["https://ultralytics.com/images/bus.jpg", "https://ultralytics.com/images/bus.jpg"],
|
||||
limit=10,
|
||||
)
|
||||
print(similar.head())
|
||||
```
|
||||
|
||||
=== "Using Dataset Indices"
|
||||
|
||||
```python
|
||||
from ultralytics import Explorer
|
||||
|
||||
# create an Explorer object
|
||||
exp = Explorer(data="coco128.yaml", model="yolo26n.pt")
|
||||
exp.create_embeddings_table()
|
||||
|
||||
similar = exp.get_similar(idx=1, limit=10)
|
||||
print(similar.head())
|
||||
|
||||
# Search using multiple indices
|
||||
similar = exp.get_similar(idx=[1, 10], limit=10)
|
||||
print(similar.head())
|
||||
```
|
||||
|
||||
### Plotting Similar Images
|
||||
|
||||
You can also plot the similar images using the `plot_similar` method. This method takes the same arguments as `get_similar` and plots the similar images in a grid.
|
||||
|
||||
!!! example "Plotting Similar Images"
|
||||
|
||||
=== "Using Images"
|
||||
|
||||
```python
|
||||
from ultralytics import Explorer
|
||||
|
||||
# create an Explorer object
|
||||
exp = Explorer(data="coco128.yaml", model="yolo26n.pt")
|
||||
exp.create_embeddings_table()
|
||||
|
||||
plt = exp.plot_similar(img="https://ultralytics.com/images/bus.jpg", limit=10)
|
||||
plt.show()
|
||||
```
|
||||
|
||||
=== "Using Dataset Indices"
|
||||
|
||||
```python
|
||||
from ultralytics import Explorer
|
||||
|
||||
# create an Explorer object
|
||||
exp = Explorer(data="coco128.yaml", model="yolo26n.pt")
|
||||
exp.create_embeddings_table()
|
||||
|
||||
plt = exp.plot_similar(idx=1, limit=10)
|
||||
plt.show()
|
||||
```
|
||||
|
||||
## 2. Ask AI (Natural Language Querying)
|
||||
|
||||
This feature lets you filter your dataset using natural language, without writing SQL. The AI-powered query generator converts your prompt into a query and returns matching results. For example, you can ask: "show me 100 images with exactly one person and 2 dogs. There can be other objects too" and it will generate the query and show you those results.
|
||||
Note: This feature uses LLMs, so results are probabilistic and may be inaccurate.
|
||||
|
||||
!!! example "Ask AI"
|
||||
|
||||
```python
|
||||
from ultralytics.data.explorer import plot_query_result
|
||||
|
||||
from ultralytics import Explorer
|
||||
|
||||
# create an Explorer object
|
||||
exp = Explorer(data="coco128.yaml", model="yolo26n.pt")
|
||||
exp.create_embeddings_table()
|
||||
|
||||
df = exp.ask_ai("show me 100 images with exactly one person and 2 dogs. There can be other objects too")
|
||||
print(df.head())
|
||||
|
||||
# plot the results
|
||||
plt = plot_query_result(df)
|
||||
plt.show()
|
||||
```
|
||||
|
||||
## 3. SQL Querying
|
||||
|
||||
You can run SQL queries on your dataset using the `sql_query` method. This method takes a SQL query as input and returns a pandas DataFrame with the results.
|
||||
|
||||
!!! example "SQL Query"
|
||||
|
||||
```python
|
||||
from ultralytics import Explorer
|
||||
|
||||
# create an Explorer object
|
||||
exp = Explorer(data="coco128.yaml", model="yolo26n.pt")
|
||||
exp.create_embeddings_table()
|
||||
|
||||
df = exp.sql_query("WHERE labels LIKE '%person%' AND labels LIKE '%dog%'")
|
||||
print(df.head())
|
||||
```
|
||||
|
||||
### Plotting SQL Query Results
|
||||
|
||||
You can also plot the results of a SQL query using the `plot_sql_query` method. This method takes the same arguments as `sql_query` and plots the results in a grid.
|
||||
|
||||
!!! example "Plotting SQL Query Results"
|
||||
|
||||
```python
|
||||
from ultralytics import Explorer
|
||||
|
||||
# create an Explorer object
|
||||
exp = Explorer(data="coco128.yaml", model="yolo26n.pt")
|
||||
exp.create_embeddings_table()
|
||||
|
||||
# plot the SQL Query
|
||||
exp.plot_sql_query("WHERE labels LIKE '%person%' AND labels LIKE '%dog%' LIMIT 10")
|
||||
```
|
||||
|
||||
## 4. Working with Embeddings Table
|
||||
|
||||
You can also work with the embeddings table directly. Once the embeddings table is created, you can access it using the `Explorer.table`
|
||||
|
||||
!!! tip
|
||||
|
||||
Explorer works on [LanceDB](https://lancedb.github.io/lancedb/) tables internally. You can access this table directly, using `Explorer.table` object and run raw queries, push down pre- and post-filters, etc.
|
||||
|
||||
```python
|
||||
from ultralytics import Explorer
|
||||
|
||||
exp = Explorer()
|
||||
exp.create_embeddings_table()
|
||||
table = exp.table
|
||||
```
|
||||
|
||||
Here are some examples of what you can do with the table:
|
||||
|
||||
### Get raw Embeddings
|
||||
|
||||
!!! example
|
||||
|
||||
```python
|
||||
from ultralytics import Explorer
|
||||
|
||||
exp = Explorer()
|
||||
exp.create_embeddings_table()
|
||||
table = exp.table
|
||||
|
||||
embeddings = table.to_pandas()["vector"]
|
||||
print(embeddings)
|
||||
```
|
||||
|
||||
### Advanced Querying with pre- and post-filters
|
||||
|
||||
!!! example
|
||||
|
||||
```python
|
||||
from ultralytics import Explorer
|
||||
|
||||
exp = Explorer(model="yolo26n.pt")
|
||||
exp.create_embeddings_table()
|
||||
table = exp.table
|
||||
|
||||
# Dummy embedding
|
||||
embedding = [i for i in range(256)]
|
||||
rs = table.search(embedding).metric("cosine").where("").limit(10)
|
||||
```
|
||||
|
||||
### Create Vector Index
|
||||
|
||||
When using large datasets, you can also create a dedicated vector index for faster querying. This is done using the `create_index` method on LanceDB table.
|
||||
|
||||
```python
|
||||
table.create_index(num_partitions=..., num_sub_vectors=...)
|
||||
```
|
||||
|
||||
## 5. Embeddings Applications
|
||||
|
||||
You can use the embeddings table to perform a variety of exploratory analysis. Here are some examples:
|
||||
|
||||
### Similarity Index
|
||||
|
||||
Explorer comes with a `similarity_index` operation:
|
||||
|
||||
- It tries to estimate how similar each data point is with the rest of the dataset.
|
||||
- It does that by counting how many image embeddings lie closer than `max_dist` to the current image in the generated embedding space, considering `top_k` similar images at a time.
|
||||
|
||||
It returns a pandas DataFrame with the following columns:
|
||||
|
||||
- `idx`: Index of the image in the dataset
|
||||
- `im_file`: Path to the image file
|
||||
- `count`: Number of images in the dataset that are closer than `max_dist` to the current image
|
||||
- `sim_im_files`: List of paths to the `count` similar images
|
||||
|
||||
!!! tip
|
||||
|
||||
For a given dataset, model, `max_dist` & `top_k` the similarity index once generated will be reused. In case, your dataset has changed, or you simply need to regenerate the similarity index, you can pass `force=True`.
|
||||
|
||||
!!! example "Similarity Index"
|
||||
|
||||
```python
|
||||
from ultralytics import Explorer
|
||||
|
||||
exp = Explorer()
|
||||
exp.create_embeddings_table()
|
||||
|
||||
sim_idx = exp.similarity_index()
|
||||
```
|
||||
|
||||
You can use similarity index to build custom conditions to filter out the dataset. For example, you can filter out images that are not similar to any other image in the dataset using the following code:
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
|
||||
sim_count = np.array(sim_idx["count"])
|
||||
sim_idx["im_file"][sim_count > 30]
|
||||
```
|
||||
|
||||
### Visualize Embedding Space
|
||||
|
||||
You can also visualize the embedding space using the plotting tool of your choice. For example here is a simple example using matplotlib:
|
||||
|
||||
```python
|
||||
import matplotlib.pyplot as plt
|
||||
from sklearn.decomposition import PCA
|
||||
|
||||
# Reduce dimensions using PCA to 3 components for visualization in 3D
|
||||
pca = PCA(n_components=3)
|
||||
reduced_data = pca.fit_transform(embeddings)
|
||||
|
||||
# Create a 3D scatter plot using Matplotlib Axes3D
|
||||
fig = plt.figure(figsize=(8, 6))
|
||||
ax = fig.add_subplot(111, projection="3d")
|
||||
|
||||
# Scatter plot
|
||||
ax.scatter(reduced_data[:, 0], reduced_data[:, 1], reduced_data[:, 2], alpha=0.5)
|
||||
ax.set_title("3D Scatter Plot of Reduced 256-Dimensional Data (PCA)")
|
||||
ax.set_xlabel("Component 1")
|
||||
ax.set_ylabel("Component 2")
|
||||
ax.set_zlabel("Component 3")
|
||||
|
||||
plt.show()
|
||||
```
|
||||
|
||||
Start creating your own CV dataset exploration reports using the Explorer API. For inspiration, check out the [VOC Exploration Example](explorer.md).
|
||||
|
||||
## Apps Built Using Ultralytics Explorer
|
||||
|
||||
Try our [GUI Demo](dashboard.md) based on Explorer API
|
||||
|
||||
## Coming Soon
|
||||
|
||||
- [ ] Merge specific labels from datasets. Example - Import all `person` labels from COCO and `car` labels from Cityscapes
|
||||
- [ ] Remove images that have a higher similarity index than the given threshold
|
||||
- [ ] Automatically persist new datasets after merging/removing entries
|
||||
- [ ] Advanced Dataset Visualizations
|
||||
|
||||
## FAQ
|
||||
|
||||
### What is the Ultralytics Explorer API used for?
|
||||
|
||||
The Ultralytics Explorer API is designed for comprehensive dataset exploration. It allows users to filter and search datasets using SQL queries, vector similarity search, and semantic search. This powerful Python API can handle large datasets, making it ideal for various [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) tasks using Ultralytics models.
|
||||
|
||||
### How do I install the Ultralytics Explorer API?
|
||||
|
||||
To install the Ultralytics Explorer API along with its dependencies, use the following command:
|
||||
|
||||
```bash
|
||||
pip install ultralytics[explorer]
|
||||
```
|
||||
|
||||
This will automatically install all necessary external libraries for the Explorer API functionality. For additional setup details, refer to the [installation section](#installation) of our documentation.
|
||||
|
||||
### How can I use the Ultralytics Explorer API for similarity search?
|
||||
|
||||
You can use the Ultralytics Explorer API to perform similarity searches by creating an embeddings table and querying it for similar images. Here's a basic example:
|
||||
|
||||
```python
|
||||
from ultralytics import Explorer
|
||||
|
||||
# Create an Explorer object
|
||||
explorer = Explorer(data="coco128.yaml", model="yolo26n.pt")
|
||||
explorer.create_embeddings_table()
|
||||
|
||||
# Search for similar images to a given image
|
||||
similar_images_df = explorer.get_similar(img="path/to/image.jpg")
|
||||
print(similar_images_df.head())
|
||||
```
|
||||
|
||||
For more details, please visit the [Similarity Search section](#1-similarity-search).
|
||||
|
||||
### What are the benefits of using LanceDB with Ultralytics Explorer?
|
||||
|
||||
LanceDB, used under the hood by Ultralytics Explorer, provides scalable, on-disk embeddings tables. This ensures that you can create and reuse embeddings for large datasets like COCO without running out of memory. These tables are only created once and can be reused, enhancing efficiency in data handling.
|
||||
|
||||
### How does the Ask AI feature work in the Ultralytics Explorer API?
|
||||
|
||||
The Ask AI feature allows users to filter datasets using natural language queries. This feature leverages LLMs to convert these queries into SQL queries behind the scenes. Here's an example:
|
||||
|
||||
```python
|
||||
from ultralytics import Explorer
|
||||
|
||||
# Create an Explorer object
|
||||
explorer = Explorer(data="coco128.yaml", model="yolo26n.pt")
|
||||
explorer.create_embeddings_table()
|
||||
|
||||
# Query with natural language
|
||||
query_result = explorer.ask_ai("show me 100 images with exactly one person and 2 dogs. There can be other objects too")
|
||||
print(query_result.head())
|
||||
```
|
||||
|
||||
For more examples, check out the [Ask AI section](#2-ask-ai-natural-language-querying).
|
||||
127
docs/en/datasets/explorer/dashboard.md
Executable file
127
docs/en/datasets/explorer/dashboard.md
Executable file
@@ -0,0 +1,127 @@
|
||||
---
|
||||
comments: true
|
||||
description: Unlock advanced data exploration with Ultralytics Explorer GUI. Utilize semantic search, run SQL queries, and ask AI for natural language data insights.
|
||||
keywords: Ultralytics Explorer GUI, semantic search, vector similarity, SQL queries, AI, natural language search, data exploration, machine learning, OpenAI, LLMs
|
||||
---
|
||||
|
||||
# Explorer GUI
|
||||
|
||||
!!! warning "Community Note ⚠️"
|
||||
|
||||
As of **`ultralytics>=8.3.10`**, Ultralytics Explorer support is deprecated. Similar (and expanded) dataset exploration features are available in [Ultralytics Platform](https://platform.ultralytics.com/).
|
||||
|
||||
Explorer GUI is built on the [Ultralytics Explorer API](api.md). It allows you to run semantic/vector similarity search, SQL queries, and natural language queries using the Ask AI feature powered by LLMs.
|
||||
|
||||
<p>
|
||||
<img width="1709" alt="Ultralytics Explorer GUI main dashboard interface" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/explorer-dashboard-screenshot-1.avif">
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<br>
|
||||
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/3VryynorQeo?start=306"
|
||||
title="YouTube video player" frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
<br>
|
||||
<strong>Watch:</strong> Ultralytics Explorer Dashboard Overview
|
||||
</p>
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
pip install ultralytics[explorer]
|
||||
```
|
||||
|
||||
!!! note
|
||||
|
||||
The Ask AI feature uses OpenAI, so you will be prompted to set the OpenAI API key when you first run the GUI.
|
||||
Set it with `yolo settings openai_api_key="..."`.
|
||||
|
||||
## Vector Semantic Similarity Search
|
||||
|
||||
[Semantic search](https://www.ultralytics.com/glossary/semantic-search) is a technique for finding similar images to a given image. It is based on the idea that similar images will have similar [embeddings](https://www.ultralytics.com/glossary/embeddings). In the UI, you can select one or more images and search for the images similar to them. This can be useful when you want to find images similar to a given image or a set of images that don't perform as expected.
|
||||
|
||||
For example, in this VOC Exploration dashboard, the user selects a few airplane images:
|
||||
|
||||
<p>
|
||||
<img width="1710" alt="Explorer selecting airplane images for similarity search" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/explorer-dashboard-screenshot-2.avif">
|
||||
</p>
|
||||
|
||||
After running the similarity search, you should see similar results:
|
||||
|
||||
<p>
|
||||
<img width="1710" alt="Ultralytics Explorer semantic similarity search" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/explorer-dashboard-screenshot-3.avif">
|
||||
</p>
|
||||
|
||||
## Ask AI
|
||||
|
||||
This feature lets you filter your dataset using natural language, without writing SQL. The AI-powered query generator converts your prompt into a query and returns matching results. For example, you can ask: "show me 100 images with exactly one person and 2 dogs. There can be other objects too" and it will generate the query and show you those results. Here is an example output when asked: "Show 10 images with exactly 5 persons":
|
||||
|
||||
<p>
|
||||
<img width="1709" alt="Explorer Ask AI results for images with 5 persons" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/explorer-dashboard-screenshot-4.avif">
|
||||
</p>
|
||||
|
||||
Note: This feature uses [Large Language Models](https://www.ultralytics.com/glossary/large-language-model-llm), so results are probabilistic and may be inaccurate.
|
||||
|
||||
## Run SQL queries on your CV datasets
|
||||
|
||||
You can run SQL queries on your dataset to filter it. It also works if you only provide the WHERE clause. For example, the following WHERE clause returns images that contain at least one person and one dog:
|
||||
|
||||
```sql
|
||||
WHERE labels LIKE '%person%' AND labels LIKE '%dog%'
|
||||
```
|
||||
|
||||
<p>
|
||||
<img width="1707" alt="Explorer SQL query filtering images with person and dog" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/explorer-dashboard-screenshot-5.avif">
|
||||
</p>
|
||||
|
||||
This demo was built using the Explorer API, which you can use to create your own exploratory notebooks or scripts for gaining insights into your datasets. To get started, check out the [Explorer API documentation](api.md).
|
||||
|
||||
## FAQ
|
||||
|
||||
### What is Ultralytics Explorer GUI and how do I install it?
|
||||
|
||||
Ultralytics Explorer GUI is a powerful interface that unlocks advanced data exploration capabilities using the [Ultralytics Explorer API](api.md). It allows you to run semantic/vector similarity search, SQL queries, and natural language queries using the Ask AI feature powered by [Large Language Models](https://www.ultralytics.com/glossary/large-language-model-llm) (LLMs).
|
||||
|
||||
To install the Explorer GUI, you can use pip:
|
||||
|
||||
```bash
|
||||
pip install ultralytics[explorer]
|
||||
```
|
||||
|
||||
Note: To use the Ask AI feature, you'll need to set the OpenAI API key: `yolo settings openai_api_key="..."`.
|
||||
|
||||
### How does the semantic search feature in Ultralytics Explorer GUI work?
|
||||
|
||||
The semantic search feature in Ultralytics Explorer GUI allows you to find images similar to a given image based on their embeddings. This technique is useful for identifying and exploring images that share visual similarities. To use this feature, select one or more images in the UI and execute a search for similar images. The result will display images that closely resemble the selected ones, facilitating efficient dataset exploration and [anomaly detection](https://www.ultralytics.com/glossary/anomaly-detection).
|
||||
|
||||
Learn more about semantic search and other features by visiting the [Feature Overview](#vector-semantic-similarity-search) section.
|
||||
|
||||
### Can I use natural language to filter datasets in Ultralytics Explorer GUI?
|
||||
|
||||
Yes, with the Ask AI feature powered by large language models (LLMs), you can filter your datasets using natural language queries. You don't need to be proficient in SQL. For instance, you can ask "Show me 100 images with exactly one person and 2 dogs. There can be other objects too," and the AI will generate the appropriate query under the hood to deliver the desired results.
|
||||
|
||||
### How do I run SQL queries on datasets using Ultralytics Explorer GUI?
|
||||
|
||||
Ultralytics Explorer GUI allows you to run SQL queries directly on your dataset to filter and manage data efficiently. To run a query, navigate to the SQL query section in the GUI and write your query. For example, to show images with at least one person and one dog, you could use:
|
||||
|
||||
```sql
|
||||
WHERE labels LIKE '%person%' AND labels LIKE '%dog%'
|
||||
```
|
||||
|
||||
You can also provide only the WHERE clause, making the querying process more flexible.
|
||||
|
||||
For more details, refer to the [SQL Queries Section](#run-sql-queries-on-your-cv-datasets).
|
||||
|
||||
### What are the benefits of using Ultralytics Explorer GUI for data exploration?
|
||||
|
||||
Ultralytics Explorer GUI enhances data exploration with features like semantic search, SQL querying, and natural language interactions through the Ask AI feature. These capabilities allow users to:
|
||||
|
||||
- Efficiently find visually similar images.
|
||||
- Filter datasets using complex SQL queries.
|
||||
- Utilize AI to perform natural language searches, eliminating the need for advanced SQL expertise.
|
||||
|
||||
These features make it a versatile tool for developers, researchers, and data scientists looking to gain deeper insights into their datasets.
|
||||
|
||||
Explore more about these features in the [Explorer GUI Documentation](#explorer-gui).
|
||||
277
docs/en/datasets/explorer/explorer.md
Executable file
277
docs/en/datasets/explorer/explorer.md
Executable file
@@ -0,0 +1,277 @@
|
||||
---
|
||||
comments: true
|
||||
description: Dive into advanced data exploration with Ultralytics Explorer. Perform semantic searches, execute SQL queries, and leverage AI-powered natural language insights for seamless data analysis.
|
||||
keywords: Ultralytics Explorer, data exploration, semantic search, vector similarity, SQL queries, AI, natural language queries, machine learning, OpenAI, LLMs, Ultralytics Platform
|
||||
---
|
||||
|
||||
# VOC Exploration Example
|
||||
|
||||
<div align="center">
|
||||
|
||||
<a href="https://www.ultralytics.com/events/yolovision" target="_blank"><img width="100%" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/ultralytics-yolov8-banner.avif" alt="Ultralytics YOLO banner"></a>
|
||||
<a href="https://docs.ultralytics.com/zh/">中文</a> |
|
||||
<a href="https://docs.ultralytics.com/ko/">한국어</a> |
|
||||
<a href="https://docs.ultralytics.com/ja/">日本語</a> |
|
||||
<a href="https://docs.ultralytics.com/ru/">Русский</a> |
|
||||
<a href="https://docs.ultralytics.com/de/">Deutsch</a> |
|
||||
<a href="https://docs.ultralytics.com/fr/">Français</a> |
|
||||
<a href="https://docs.ultralytics.com/es">Español</a> |
|
||||
<a href="https://docs.ultralytics.com/pt/">Português</a> |
|
||||
<a href="https://docs.ultralytics.com/tr/">Türkçe</a> |
|
||||
<a href="https://docs.ultralytics.com/vi/">Tiếng Việt</a> |
|
||||
<a href="https://docs.ultralytics.com/ar/">العربية</a>
|
||||
<br>
|
||||
|
||||
<br>
|
||||
<a href="https://github.com/ultralytics/ultralytics/actions/workflows/ci.yml"><img src="https://github.com/ultralytics/ultralytics/actions/workflows/ci.yml/badge.svg" alt="Ultralytics CI"></a>
|
||||
<a href="https://clickpy.clickhouse.com/dashboard/ultralytics"><img src="https://static.pepy.tech/badge/ultralytics" alt="Ultralytics Downloads"></a>
|
||||
<a href="https://zenodo.org/badge/latestdoi/264818686"><img src="https://zenodo.org/badge/264818686.svg" alt="Ultralytics YOLO Citation"></a>
|
||||
<a href="https://discord.com/invite/ultralytics"><img alt="Ultralytics Discord" src="https://img.shields.io/discord/1089800235347353640?logo=discord&logoColor=white&label=Discord&color=blue"></a>
|
||||
<a href="https://community.ultralytics.com/"><img alt="Ultralytics Forums" src="https://img.shields.io/discourse/users?server=https%3A%2F%2Fcommunity.ultralytics.com&logo=discourse&label=Forums&color=blue"></a>
|
||||
<a href="https://www.reddit.com/r/ultralytics/"><img alt="Ultralytics Reddit" src="https://img.shields.io/reddit/subreddit-subscribers/ultralytics?style=flat&logo=reddit&logoColor=white&label=Reddit&color=blue"></a>
|
||||
<br>
|
||||
<a href="https://console.paperspace.com/github/ultralytics/ultralytics"><img src="https://assets.paperspace.io/img/gradient-badge.svg" alt="Run Ultralytics on Gradient"></a>
|
||||
<a href="https://colab.research.google.com/github/ultralytics/ultralytics/blob/main/examples/tutorial.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open Ultralytics In Colab"></a>
|
||||
<a href="https://www.kaggle.com/models/ultralytics/yolo26"><img src="https://kaggle.com/static/images/open-in-kaggle.svg" alt="Open Ultralytics In Kaggle"></a>
|
||||
<a href="https://mybinder.org/v2/gh/ultralytics/ultralytics/HEAD?labpath=examples%2Ftutorial.ipynb"><img src="https://mybinder.org/badge_logo.svg" alt="Open Ultralytics In Binder"></a>
|
||||
<br>
|
||||
</div>
|
||||
|
||||
Welcome to the Ultralytics Explorer API notebook. This notebook introduces the resources available for exploring datasets with semantic search, vector search, and SQL queries.
|
||||
|
||||
Try `yolo explorer` (powered by the Explorer API)
|
||||
|
||||
Install `ultralytics` and run `yolo explorer` in your terminal to run custom queries and semantic search in your browser.
|
||||
|
||||
!!! warning "Community Note ⚠️"
|
||||
|
||||
As of **`ultralytics>=8.3.10`**, Ultralytics Explorer support is deprecated. Similar (and expanded) dataset exploration features are available in [Ultralytics Platform](https://platform.ultralytics.com/).
|
||||
|
||||
## Setup
|
||||
|
||||
Install `ultralytics` and the required [dependencies](https://github.com/ultralytics/ultralytics/blob/main/pyproject.toml), then check software and hardware.
|
||||
|
||||
```bash
|
||||
!uv pip install ultralytics[explorer] openai
|
||||
yolo checks
|
||||
```
|
||||
|
||||
## Similarity Search
|
||||
|
||||
Utilize the power of vector similarity search to find the similar data points in your dataset along with their distance in the embedding space. Simply create an embeddings table for the given dataset-model pair. It is only needed once, and it is reused automatically.
|
||||
|
||||
```python
|
||||
exp = Explorer("VOC.yaml", model="yolo26n.pt")
|
||||
exp.create_embeddings_table()
|
||||
```
|
||||
|
||||
Once the embeddings table is built, you can run semantic search in any of the following ways:
|
||||
|
||||
- On a given index/list of indices in the dataset, e.g., `exp.get_similar(idx=[1, 10], limit=10)`
|
||||
- On any image/ list of images not in the dataset - exp.get_similar(img=["path/to/img1", "path/to/img2"], limit=10) In case of multiple inputs, the aggregate of their embeddings is used.
|
||||
|
||||
You get a pandas DataFrame with the limit number of most similar data points to the input, along with their distance in the embedding space. You can use this dataset to perform further filtering.
|
||||
|
||||
|
||||
|
||||
```python
|
||||
# Search dataset by index
|
||||
similar = exp.get_similar(idx=1, limit=10)
|
||||
similar.head()
|
||||
```
|
||||
|
||||
You can also plot the similar samples directly using the `plot_similar` util
|
||||
|
||||
|
||||
|
||||
```python
|
||||
exp.plot_similar(idx=6500, limit=20)
|
||||
exp.plot_similar(idx=[100, 101], limit=10) # Can also pass list of idxs or imgs
|
||||
|
||||
exp.plot_similar(img="https://ultralytics.com/images/bus.jpg", limit=10, labels=False) # Can also pass external images
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Ask AI: Search or Filter with Natural Language
|
||||
|
||||
You can prompt the Explorer object with the kind of data points you want to see, and it will try to return a DataFrame with those results. Because it is powered by LLMs, it does not always get it right. In that case, it will return `None`.
|
||||
|
||||
|
||||
|
||||
```python
|
||||
df = exp.ask_ai("show me images containing more than 10 objects with at least 2 persons")
|
||||
df.head(5)
|
||||
```
|
||||
|
||||
To plot these results, you can use the `plot_query_result` utility. Example:
|
||||
|
||||
```python
|
||||
plt = plot_query_result(exp.ask_ai("show me 10 images containing exactly 2 persons"))
|
||||
Image.fromarray(plt)
|
||||
```
|
||||
|
||||
|
||||
|
||||
```python
|
||||
# plot
|
||||
from PIL import Image
|
||||
from ultralytics.data.explorer import plot_query_result
|
||||
|
||||
plt = plot_query_result(exp.ask_ai("show me 10 images containing exactly 2 persons"))
|
||||
Image.fromarray(plt)
|
||||
```
|
||||
|
||||
## Run SQL Queries on Your Dataset
|
||||
|
||||
Sometimes you might want to investigate certain entries in your dataset. For this, Explorer allows you to execute SQL queries. It accepts either of the following formats:
|
||||
|
||||
- Queries beginning with "WHERE" will automatically select all columns. This can be thought of as a shorthand query.
|
||||
- You can also write full queries where you can specify which columns to select.
|
||||
|
||||
This can be used to investigate model performance and specific data points. For example:
|
||||
|
||||
- let's say your model struggles on images that have humans and dogs. You can write a query like this to select the points that have at least 2 humans AND at least one dog.
|
||||
|
||||
You can combine SQL query and semantic search to filter down to specific type of results
|
||||
|
||||
```python
|
||||
table = exp.sql_query("WHERE labels LIKE '%person, person%' AND labels LIKE '%dog%' LIMIT 10")
|
||||
exp.plot_sql_query("WHERE labels LIKE '%person, person%' AND labels LIKE '%dog%' LIMIT 10", labels=True)
|
||||
```
|
||||
|
||||
|
||||
|
||||
```python
|
||||
table = exp.sql_query("WHERE labels LIKE '%person, person%' AND labels LIKE '%dog%' LIMIT 10")
|
||||
print(table)
|
||||
```
|
||||
|
||||
Just like similarity search, you also get a util to directly plot the sql queries using `exp.plot_sql_query`
|
||||
|
||||
|
||||
|
||||
```python
|
||||
exp.plot_sql_query("WHERE labels LIKE '%person, person%' AND labels LIKE '%dog%' LIMIT 10", labels=True)
|
||||
```
|
||||
|
||||
## Working with embeddings Table (Advanced)
|
||||
|
||||
Explorer works on [LanceDB](https://lancedb.github.io/lancedb/) tables internally. You can access this table directly, using `Explorer.table` object and run raw queries, push down pre- and post-filters, etc.
|
||||
|
||||
```python
|
||||
table = exp.table
|
||||
print(table.schema)
|
||||
```
|
||||
|
||||
### Run raw queries¶
|
||||
|
||||
Vector Search finds the nearest vectors from the database. In a recommendation system or search engine, you can find similar products from the one you searched. In LLM and other AI applications, each data point can be presented by the embeddings generated from some models, it returns the most relevant features.
|
||||
|
||||
A search in high-dimensional vector space, is to find K-Nearest-Neighbors (KNN) of the query vector.
|
||||
|
||||
Metric In LanceDB, a Metric is the way to describe the distance between a pair of vectors. Currently, it supports the following metrics:
|
||||
|
||||
- L2
|
||||
- Cosine
|
||||
- Dot Explorer's similarity search uses L2 by default. You can run queries on tables directly, or use the lance format to build custom utilities to manage datasets. More details on available LanceDB table ops in the [docs](https://lancedb.github.io/lancedb/)
|
||||
|
||||
|
||||
|
||||
```python
|
||||
dummy_img_embedding = [i for i in range(256)]
|
||||
table.search(dummy_img_embedding).limit(5).to_pandas()
|
||||
```
|
||||
|
||||
### Interconversion to popular data formats
|
||||
|
||||
```python
|
||||
df = table.to_pandas()
|
||||
pa_table = table.to_arrow()
|
||||
```
|
||||
|
||||
### Work with Embeddings
|
||||
|
||||
You can access the raw embedding from lancedb Table and analyze it. The image embeddings are stored in column `vector`
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
|
||||
embeddings = table.to_pandas()["vector"].tolist()
|
||||
embeddings = np.array(embeddings)
|
||||
```
|
||||
|
||||
### Scatterplot
|
||||
|
||||
One of the preliminary steps in analyzing embeddings is by plotting them in 2D space via dimensionality reduction. Let's try an example
|
||||
|
||||
|
||||
|
||||
```python
|
||||
import matplotlib.pyplot as plt
|
||||
from sklearn.decomposition import PCA # pip install scikit-learn
|
||||
|
||||
# Reduce dimensions using PCA to 3 components for visualization in 3D
|
||||
pca = PCA(n_components=3)
|
||||
reduced_data = pca.fit_transform(embeddings)
|
||||
|
||||
# Create a 3D scatter plot using Matplotlib's Axes3D
|
||||
fig = plt.figure(figsize=(8, 6))
|
||||
ax = fig.add_subplot(111, projection="3d")
|
||||
|
||||
# Scatter plot
|
||||
ax.scatter(reduced_data[:, 0], reduced_data[:, 1], reduced_data[:, 2], alpha=0.5)
|
||||
ax.set_title("3D Scatter Plot of Reduced 256-Dimensional Data (PCA)")
|
||||
ax.set_xlabel("Component 1")
|
||||
ax.set_ylabel("Component 2")
|
||||
ax.set_zlabel("Component 3")
|
||||
|
||||
plt.show()
|
||||
```
|
||||
|
||||
### Similarity Index
|
||||
|
||||
Here's a simple example of an operation powered by the embeddings table. Explorer comes with a `similarity_index` operation-
|
||||
|
||||
- It tries to estimate how similar each data point is with the rest of the dataset.
|
||||
- It does that by counting how many image embeddings lie closer than max_dist to the current image in the generated embedding space, considering top_k similar images at a time.
|
||||
|
||||
For a given dataset, model, `max_dist` & `top_k` the similarity index once generated will be reused. In case, your dataset has changed, or you simply need to regenerate the similarity index, you can pass `force=True`. Similar to vector and SQL search, this also comes with a util to directly plot it.
|
||||
|
||||
```python
|
||||
sim_idx = exp.similarity_index(max_dist=0.2, top_k=0.01)
|
||||
exp.plot_similarity_index(max_dist=0.2, top_k=0.01)
|
||||
```
|
||||
|
||||
|
||||
|
||||
Let's look at the plot first
|
||||
|
||||
```python
|
||||
exp.plot_similarity_index(max_dist=0.2, top_k=0.01)
|
||||
```
|
||||
|
||||
Now let's look at the output of the operation
|
||||
|
||||
```python
|
||||
sim_idx = exp.similarity_index(max_dist=0.2, top_k=0.01, force=False)
|
||||
|
||||
sim_idx
|
||||
```
|
||||
|
||||
Let's create a query to see what data points have similarity count of more than 30 and plot images similar to them.
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
|
||||
sim_count = np.array(sim_idx["count"])
|
||||
sim_idx["im_file"][sim_count > 30]
|
||||
```
|
||||
|
||||
You should see something like this
|
||||
|
||||
|
||||
|
||||
```python
|
||||
exp.plot_similar(idx=[7146, 14035]) # Using avg embeddings of 2 images
|
||||
```
|
||||
113
docs/en/datasets/explorer/index.md
Executable file
113
docs/en/datasets/explorer/index.md
Executable file
@@ -0,0 +1,113 @@
|
||||
---
|
||||
comments: true
|
||||
description: Discover Ultralytics Explorer for semantic search, SQL queries, vector similarity, and natural language dataset exploration.
|
||||
keywords: Ultralytics Explorer, CV datasets, semantic search, SQL queries, vector similarity, dataset visualization, python API, machine learning, computer vision
|
||||
---
|
||||
|
||||
# Ultralytics Explorer
|
||||
|
||||
!!! warning "Community Note ⚠️"
|
||||
|
||||
As of **`ultralytics>=8.3.10`**, Ultralytics Explorer support is deprecated. Similar (and expanded) dataset exploration features are available in [Ultralytics Platform](https://platform.ultralytics.com/).
|
||||
|
||||
<p>
|
||||
<img width="1709" alt="Ultralytics Explorer dataset visualization GUI" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/explorer-dashboard-screenshot-1.avif">
|
||||
</p>
|
||||
|
||||
<a href="https://colab.research.google.com/github/ultralytics/ultralytics/blob/main/docs/en/datasets/explorer/explorer.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a>
|
||||
|
||||
Ultralytics Explorer is a tool for exploring CV datasets using semantic search, SQL queries, vector similarity search, and natural language prompts. It also provides a Python API for accessing the same functionality.
|
||||
|
||||
<p align="center">
|
||||
<br>
|
||||
<iframe loading="lazy" width="720" height="405" src="https://www.youtube.com/embed/3VryynorQeo"
|
||||
title="YouTube video player" frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
<br>
|
||||
<strong>Watch:</strong> Ultralytics Explorer API | Semantic Search, SQL Queries & Ask AI Features
|
||||
</p>
|
||||
|
||||
## Installation of Optional Dependencies
|
||||
|
||||
Explorer depends on external libraries for some of its functionality. These are automatically installed when you use Explorer. To manually install these dependencies, use the following command:
|
||||
|
||||
```bash
|
||||
pip install ultralytics[explorer]
|
||||
```
|
||||
|
||||
!!! tip
|
||||
|
||||
Explorer works on embedding/semantic search & SQL querying and is powered by [LanceDB](https://lancedb.com/) serverless vector database. Unlike traditional in-memory DBs, it is persisted on disk without sacrificing performance, so you can scale locally to large datasets like COCO without running out of memory.
|
||||
|
||||
## Explorer API
|
||||
|
||||
This is a Python API for exploring your datasets. It also powers the GUI Explorer. You can use this to create your own exploratory notebooks or scripts to get insights into your datasets.
|
||||
|
||||
Explore the full capabilities and usage examples in the [Explorer API documentation](api.md).
|
||||
|
||||
## GUI Explorer Usage
|
||||
|
||||
The GUI demo runs in your browser allowing you to create [embeddings](https://www.ultralytics.com/glossary/embeddings) for your dataset and search for similar images, run SQL queries and perform semantic search. It can be run using the following command:
|
||||
|
||||
```bash
|
||||
yolo explorer
|
||||
```
|
||||
|
||||
!!! note
|
||||
|
||||
The Ask AI feature uses OpenAI, so you'll be prompted to set the API key for OpenAI when you first run the GUI.
|
||||
You can set it like this - `yolo settings openai_api_key="..."`
|
||||
|
||||
<p>
|
||||
<img width="1709" alt="Ultralytics Explorer OpenAI Integration" src="https://cdn.jsdelivr.net/gh/ultralytics/assets@main/docs/ultralytics-explorer-openai-integration.avif">
|
||||
</p>
|
||||
|
||||
## FAQ
|
||||
|
||||
### What is Ultralytics Explorer and how can it help with CV datasets?
|
||||
|
||||
Ultralytics Explorer is a powerful tool designed for exploring [computer vision](https://www.ultralytics.com/glossary/computer-vision-cv) (CV) datasets through semantic search, SQL queries, vector similarity search, and even natural language. This versatile tool provides both a GUI and a Python API, allowing users to seamlessly interact with their datasets. By leveraging technologies like [LanceDB](https://lancedb.com/), Ultralytics Explorer ensures efficient, scalable access to large datasets without excessive memory usage. Whether you're performing detailed dataset analysis or exploring data patterns, Ultralytics Explorer streamlines the entire process.
|
||||
|
||||
Learn more about the [Explorer API](api.md).
|
||||
|
||||
### How do I install the dependencies for Ultralytics Explorer?
|
||||
|
||||
To manually install the optional dependencies needed for Ultralytics Explorer, you can use the following `pip` command:
|
||||
|
||||
```bash
|
||||
pip install ultralytics[explorer]
|
||||
```
|
||||
|
||||
These dependencies are essential for the full functionality of semantic search and SQL querying. By including libraries powered by [LanceDB](https://lancedb.com/), the installation ensures that the database operations remain efficient and scalable, even for large datasets like [COCO](../detect/coco.md).
|
||||
|
||||
### How can I use the GUI version of Ultralytics Explorer?
|
||||
|
||||
Using the GUI version of Ultralytics Explorer is straightforward. After installing the necessary dependencies, you can launch the GUI with the following command:
|
||||
|
||||
```bash
|
||||
yolo explorer
|
||||
```
|
||||
|
||||
The GUI provides a user-friendly interface for creating dataset embeddings, searching for similar images, running SQL queries, and conducting semantic searches. Additionally, the integration with OpenAI's Ask AI feature allows you to query datasets using natural language, enhancing the flexibility and ease of use.
|
||||
|
||||
For storage and scalability information, check out our [installation instructions](#installation-of-optional-dependencies).
|
||||
|
||||
### What is the Ask AI feature in Ultralytics Explorer?
|
||||
|
||||
The Ask AI feature in Ultralytics Explorer allows users to interact with their datasets using natural language queries. Powered by [OpenAI](https://www.ultralytics.com/blog/openai-gpt-4o-showcases-ai-potential), this feature enables you to ask complex questions and receive insightful answers without needing to write SQL queries or similar commands. To use this feature, you'll need to set your OpenAI API key the first time you run the GUI:
|
||||
|
||||
```bash
|
||||
yolo settings openai_api_key="YOUR_API_KEY"
|
||||
```
|
||||
|
||||
For more on this feature and how to integrate it, see our [GUI Explorer Usage](#gui-explorer-usage) section.
|
||||
|
||||
### Can I run Ultralytics Explorer in Google Colab?
|
||||
|
||||
Yes, Ultralytics Explorer can be run in Google Colab, providing a convenient and powerful environment for dataset exploration. You can start by opening the provided Colab notebook, which is pre-configured with all the necessary settings:
|
||||
|
||||
<a href="https://colab.research.google.com/github/ultralytics/ultralytics/blob/main/docs/en/datasets/explorer/explorer.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a>
|
||||
|
||||
This setup allows you to explore your datasets fully, taking advantage of Google's cloud resources. Learn more in our [Google Colab Guide](../../integrations/google-colab.md).
|
||||
Reference in New Issue
Block a user