diff --git a/docs/data-sources/Ollama/linux-arm64.md b/docs/data-sources/Ollama/linux-arm64.md index c51b348..88c58b7 100644 --- a/docs/data-sources/Ollama/linux-arm64.md +++ b/docs/data-sources/Ollama/linux-arm64.md @@ -28,7 +28,7 @@ In order to use the plugin, the user must set any required environment variables {: .warning } This table's structure is determined at runtime, meaning its columns and properties will vary based on the data source. Use the 'desc' command to view the current table structure. -### ollama.llm(string model, float temperature) +### ollama.llm(string model, decimal temperature) Gives the access to Ollama api diff --git a/docs/data-sources/Ollama/linux-x64.md b/docs/data-sources/Ollama/linux-x64.md index fdf461e..582c80e 100644 --- a/docs/data-sources/Ollama/linux-x64.md +++ b/docs/data-sources/Ollama/linux-x64.md @@ -28,7 +28,7 @@ In order to use the plugin, the user must set any required environment variables {: .warning } This table's structure is determined at runtime, meaning its columns and properties will vary based on the data source. Use the 'desc' command to view the current table structure. -### ollama.llm(string model, float temperature) +### ollama.llm(string model, decimal temperature) Gives the access to Ollama api diff --git a/docs/data-sources/Ollama/macos-x64.md b/docs/data-sources/Ollama/macos-x64.md index febdce4..a6f5cec 100644 --- a/docs/data-sources/Ollama/macos-x64.md +++ b/docs/data-sources/Ollama/macos-x64.md @@ -28,7 +28,7 @@ In order to use the plugin, the user must set any required environment variables {: .warning } This table's structure is determined at runtime, meaning its columns and properties will vary based on the data source. Use the 'desc' command to view the current table structure. -### ollama.llm(string model, float temperature) +### ollama.llm(string model, decimal temperature) Gives the access to Ollama api diff --git a/docs/data-sources/Ollama/windows-x64.md b/docs/data-sources/Ollama/windows-x64.md index 47598a9..0cb4319 100644 --- a/docs/data-sources/Ollama/windows-x64.md +++ b/docs/data-sources/Ollama/windows-x64.md @@ -28,7 +28,7 @@ In order to use the plugin, the user must set any required environment variables {: .warning } This table's structure is determined at runtime, meaning its columns and properties will vary based on the data source. Use the 'desc' command to view the current table structure. -### ollama.llm(string model, float temperature) +### ollama.llm(string model, decimal temperature) Gives the access to Ollama api diff --git a/docs/data-sources/OpenAI/linux-arm64.md b/docs/data-sources/OpenAI/linux-arm64.md index 57d5fd1..6a9d62f 100644 --- a/docs/data-sources/OpenAI/linux-arm64.md +++ b/docs/data-sources/OpenAI/linux-arm64.md @@ -64,7 +64,7 @@ In order to use the plugin, the user must set any required environment variables {: .warning } This table's structure is determined at runtime, meaning its columns and properties will vary based on the data source. Use the 'desc' command to view the current table structure. -### openai.gpt(string model, int maxTokens, double temperature) +### openai.gpt(string model, int maxTokens, decimal temperature) Gives the access to OpenAI api @@ -82,7 +82,7 @@ In order to use the plugin, the user must set any required environment variables {: .warning } This table's structure is determined at runtime, meaning its columns and properties will vary based on the data source. Use the 'desc' command to view the current table structure. -### openai.gpt(string model, int maxTokens, double temperature, double frequencyPenalty) +### openai.gpt(string model, int maxTokens, decimal temperature, decimal frequencyPenalty) Gives the access to OpenAI api @@ -100,7 +100,7 @@ In order to use the plugin, the user must set any required environment variables {: .warning } This table's structure is determined at runtime, meaning its columns and properties will vary based on the data source. Use the 'desc' command to view the current table structure. -### openai.gpt(string model, int maxTokens, double temperature, double frequencyPenalty, double presencePenalty) +### openai.gpt(string model, int maxTokens, decimal temperature, decimal frequencyPenalty, decimal presencePenalty) Gives the access to OpenAI api diff --git a/docs/data-sources/OpenAI/linux-x64.md b/docs/data-sources/OpenAI/linux-x64.md index cf8d197..700c6e4 100644 --- a/docs/data-sources/OpenAI/linux-x64.md +++ b/docs/data-sources/OpenAI/linux-x64.md @@ -64,7 +64,7 @@ In order to use the plugin, the user must set any required environment variables {: .warning } This table's structure is determined at runtime, meaning its columns and properties will vary based on the data source. Use the 'desc' command to view the current table structure. -### openai.gpt(string model, int maxTokens, double temperature) +### openai.gpt(string model, int maxTokens, decimal temperature) Gives the access to OpenAI api @@ -82,7 +82,7 @@ In order to use the plugin, the user must set any required environment variables {: .warning } This table's structure is determined at runtime, meaning its columns and properties will vary based on the data source. Use the 'desc' command to view the current table structure. -### openai.gpt(string model, int maxTokens, double temperature, double frequencyPenalty) +### openai.gpt(string model, int maxTokens, decimal temperature, decimal frequencyPenalty) Gives the access to OpenAI api @@ -100,7 +100,7 @@ In order to use the plugin, the user must set any required environment variables {: .warning } This table's structure is determined at runtime, meaning its columns and properties will vary based on the data source. Use the 'desc' command to view the current table structure. -### openai.gpt(string model, int maxTokens, double temperature, double frequencyPenalty, double presencePenalty) +### openai.gpt(string model, int maxTokens, decimal temperature, decimal frequencyPenalty, decimal presencePenalty) Gives the access to OpenAI api diff --git a/docs/data-sources/OpenAI/macos-x64.md b/docs/data-sources/OpenAI/macos-x64.md index 2f43fc9..bc6a301 100644 --- a/docs/data-sources/OpenAI/macos-x64.md +++ b/docs/data-sources/OpenAI/macos-x64.md @@ -64,7 +64,7 @@ In order to use the plugin, the user must set any required environment variables {: .warning } This table's structure is determined at runtime, meaning its columns and properties will vary based on the data source. Use the 'desc' command to view the current table structure. -### openai.gpt(string model, int maxTokens, double temperature) +### openai.gpt(string model, int maxTokens, decimal temperature) Gives the access to OpenAI api @@ -82,7 +82,7 @@ In order to use the plugin, the user must set any required environment variables {: .warning } This table's structure is determined at runtime, meaning its columns and properties will vary based on the data source. Use the 'desc' command to view the current table structure. -### openai.gpt(string model, int maxTokens, double temperature, double frequencyPenalty) +### openai.gpt(string model, int maxTokens, decimal temperature, decimal frequencyPenalty) Gives the access to OpenAI api @@ -100,7 +100,7 @@ In order to use the plugin, the user must set any required environment variables {: .warning } This table's structure is determined at runtime, meaning its columns and properties will vary based on the data source. Use the 'desc' command to view the current table structure. -### openai.gpt(string model, int maxTokens, double temperature, double frequencyPenalty, double presencePenalty) +### openai.gpt(string model, int maxTokens, decimal temperature, decimal frequencyPenalty, decimal presencePenalty) Gives the access to OpenAI api diff --git a/docs/data-sources/OpenAI/windows-x64.md b/docs/data-sources/OpenAI/windows-x64.md index 22e2677..9cfe51f 100644 --- a/docs/data-sources/OpenAI/windows-x64.md +++ b/docs/data-sources/OpenAI/windows-x64.md @@ -64,7 +64,7 @@ In order to use the plugin, the user must set any required environment variables {: .warning } This table's structure is determined at runtime, meaning its columns and properties will vary based on the data source. Use the 'desc' command to view the current table structure. -### openai.gpt(string model, int maxTokens, double temperature) +### openai.gpt(string model, int maxTokens, decimal temperature) Gives the access to OpenAI api @@ -82,7 +82,7 @@ In order to use the plugin, the user must set any required environment variables {: .warning } This table's structure is determined at runtime, meaning its columns and properties will vary based on the data source. Use the 'desc' command to view the current table structure. -### openai.gpt(string model, int maxTokens, double temperature, double frequencyPenalty) +### openai.gpt(string model, int maxTokens, decimal temperature, decimal frequencyPenalty) Gives the access to OpenAI api @@ -100,7 +100,7 @@ In order to use the plugin, the user must set any required environment variables {: .warning } This table's structure is determined at runtime, meaning its columns and properties will vary based on the data source. Use the 'desc' command to view the current table structure. -### openai.gpt(string model, int maxTokens, double temperature, double frequencyPenalty, double presencePenalty) +### openai.gpt(string model, int maxTokens, decimal temperature, decimal frequencyPenalty, decimal presencePenalty) Gives the access to OpenAI api diff --git a/docs/data-sources/Os/linux-arm64.md b/docs/data-sources/Os/linux-arm64.md index db67a89..6212d34 100644 --- a/docs/data-sources/Os/linux-arm64.md +++ b/docs/data-sources/Os/linux-arm64.md @@ -25,7 +25,7 @@ Compares two directories | SourceFileRelative | string | Relative path to source file | | DestinationFileRelative | string | Relative path to destination file | -### os.directories(string directory, boolean useSubdirectories) +### os.directories(string directory, bool useSubdirectories) Gets the directories @@ -72,7 +72,7 @@ Gets the dlls | Assembly | Assembly | Gets the Assembly object | | Version | FileVersionInfo | Gets the assembly version | -### os.files(string directory, boolean useSubdirectories) +### os.files(string directory, bool useSubdirectories) Gets the files diff --git a/docs/data-sources/Os/linux-x64.md b/docs/data-sources/Os/linux-x64.md index cdeff44..a0bf260 100644 --- a/docs/data-sources/Os/linux-x64.md +++ b/docs/data-sources/Os/linux-x64.md @@ -25,7 +25,7 @@ Compares two directories | SourceFileRelative | string | Relative path to source file | | DestinationFileRelative | string | Relative path to destination file | -### os.directories(string directory, boolean useSubdirectories) +### os.directories(string directory, bool useSubdirectories) Gets the directories @@ -72,7 +72,7 @@ Gets the dlls | Assembly | Assembly | Gets the Assembly object | | Version | FileVersionInfo | Gets the assembly version | -### os.files(string directory, boolean useSubdirectories) +### os.files(string directory, bool useSubdirectories) Gets the files diff --git a/docs/data-sources/Os/macos-x64.md b/docs/data-sources/Os/macos-x64.md index 1c3e2aa..3eeb2d9 100644 --- a/docs/data-sources/Os/macos-x64.md +++ b/docs/data-sources/Os/macos-x64.md @@ -25,7 +25,7 @@ Compares two directories | SourceFileRelative | string | Relative path to source file | | DestinationFileRelative | string | Relative path to destination file | -### os.directories(string directory, boolean useSubdirectories) +### os.directories(string directory, bool useSubdirectories) Gets the directories @@ -72,7 +72,7 @@ Gets the dlls | Assembly | Assembly | Gets the Assembly object | | Version | FileVersionInfo | Gets the assembly version | -### os.files(string directory, boolean useSubdirectories) +### os.files(string directory, bool useSubdirectories) Gets the files diff --git a/docs/data-sources/Os/windows-x64.md b/docs/data-sources/Os/windows-x64.md index 86abfd1..d8d7221 100644 --- a/docs/data-sources/Os/windows-x64.md +++ b/docs/data-sources/Os/windows-x64.md @@ -25,7 +25,7 @@ Compares two directories | SourceFileRelative | string | Relative path to source file | | DestinationFileRelative | string | Relative path to destination file | -### os.directories(string directory, boolean useSubdirectories) +### os.directories(string directory, bool useSubdirectories) Gets the directories @@ -72,7 +72,7 @@ Gets the dlls | Assembly | Assembly | Gets the Assembly object | | Version | FileVersionInfo | Gets the assembly version | -### os.files(string directory, boolean useSubdirectories) +### os.files(string directory, bool useSubdirectories) Gets the files diff --git a/docs/practical-examples-and-applications/turning-photos-to-hashtags-with-llms.md b/docs/practical-examples-and-applications/turning-photos-to-hashtags-with-llms.md new file mode 100644 index 0000000..184f06a --- /dev/null +++ b/docs/practical-examples-and-applications/turning-photos-to-hashtags-with-llms.md @@ -0,0 +1,94 @@ +--- +title: Turning Photos To HashTags with LLMs +layout: default +parent: Practical Examples and Applications +nav_order: 9 +--- + +## Turning Photos To HashTags with LLMs + +I'm going to create hashtags from images using SQL. We will use both Ollama for image description generation and OpenAI for hashtag generation. + +## How It's Going To Work +- We're going to get image files from a specified directory. +- Each image will be converted to Base64. +- The local LLM will generate a description based on a prompt. +- The result will be stored as a column. +- We're going to contact OpenAI to generate hashtags from the given photo description. + +This way, we can index photos and do something useful with them, like creating a thematic album. + +## First Try +Here's an example SQL query using the `llama3.2-vision:latest` model to generate image descriptions: + +```sql +select f.Name, l.AskImage('Describe this photo of my child in one sentence.', f.Base64File()) as description +from #os.files('/some/folder/with/photos', false) f +cross apply #ollama.llm('llama3.2-vision:latest') l +``` + +``` +┌──────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ +│ f.Name │ description │ +├──────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ +│ BEZN8348.JPG │ The baby in the image appears to be wearing a white long-sleeved shirt and brown pants, with a neutral expression on their face as they │ +│ │ are being held by an older man who is likely a grandparent. │ +│ AARD3200.JPG │ The baby in the image appears to be sleeping peacefully, lying on its back with its arms at its sides and a calm expression on its face. │ +│ CCLO3762.JPG │ The baby in the image appears to be sleeping peacefully in a stroller or bassinet, wrapped snugly in a white blanket and wearing a │ +│ │ matching hat. │ +└──────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ +``` + +## Another Try With Commercial Model +For comparison, the following query uses the `gpt-4o` model to generate image descriptions: + +```sql +select f.Name, l.AskImage('this is the photo of my little child I want you to describe. Be conscise, use only single statement.', f.Base64File()) as description +from #os.files('/some/folder/with/photos', false) f +cross apply #openai.gpt('gpt-4o') l +``` + +``` +┌──────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ +│ f.Name │ description │ +├──────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ +│ BEZN8348.JPG │ A man in a red sweater is holding a baby, with a decorative background featuring balloons and a banner. │ +│ CCLO3762.JPG │ A baby is peacefully sleeping in a stroller, bundled in warm white clothing, while an adult adjusts the blanket. │ +│ AARD3200.JPG │ A baby is peacefully sleeping, surrounded by patterned bedding and wearing a blue outfit with colorful vehicle prints. │ +└──────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ +``` + +## Final Try With Both Models Combined +As I don't want to send private images (e.g., pictures of your child) to external services. By generating photo descriptions locally, you keep sensitive data in-house. You can then send just these descriptions to external LLM services, minimizing your data exposure while still benefiting from features which for me is hashtags generation. + +```sql +with PhotosDescription as ( + select + f.Name as Name, + l.AskImage('this is the photo of my little child I want you to describe. Be conscise, use only single statement.', f.Base64File()) as Description + from #os.files('/some/folder/with/photos', false) f + cross apply #ollama.llm('llama3.2-vision:11b-instruct-q4_K_M') l +) +select + p.Name, + p.Description, + l.LlmPerform('this is the description of the photo I want you generate hashtags for. It comes from my child photo album. Return only hashtags separated with comma (#something, #somethingElse). Comma is very important to separate hashtags. Dont forget about it. No description or explanation.', p.Description) as HashTags +from PhotosDescription p cross apply #openai.gpt('gpt-4o', 4096, 0.0) l +``` + +``` +┌──────────────┬─────────────────────────────────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────┐ +│ p.Name │ p.Description │ HashTags │ +├──────────────┼─────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤ +│ AARD3200.JPG │ The baby in the image appears to be sleeping peacefully, lying on │ #BabySleep, #PeacefulBaby, #SleepingBaby, #CalmBaby, #AdorableBaby, │ +│ │ its back with its arms at its sides and a calm expression on its │ #BabyAlbum, #SweetDreams, #InnocentSleep, #BabyMemories, │ +│ │ face. │ #CherishedMoments │ +│ BEZN8348.JPG │ The baby in the image appears to be wearing a white long-sleeved │ #BabyMemories, #FamilyLove, #GrandparentBond, #ChildhoodMoments, │ +│ │ shirt and brown pants, with a neutral expression on their face as │ #FamilyAlbum, #PreciousMemories, #GenerationsTogether, #FamilyTime, │ +│ │ they are being held by an older man who is likely a grandparent. │ #BabyFashion, #CherishedMoments │ +│ CCLO3762.JPG │ The baby in the image appears to be sleeping peacefully in a │ #BabySleep, #PeacefulBaby, #SleepingBaby, #SnugAsABug, │ +│ │ stroller or bassinet, wrapped snugly in a white blanket and wearing │ #BabyInBlanket, #AdorableBaby, #BabyInStroller, #Newborn, #BabyHat, │ +│ │ a matching hat. │ #SweetDreams, #InnocentSleep, #BabyAlbum, #CherishedMoments, │ +│ │ │ #BabyMemories, #CozyBaby │ +└──────────────┴─────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────┘ +```