Google Gemini via API

Google Gemini Pro API ~~Unfortunately Gemini via API has gotten too big (too many actions) and I'm no longer able to edit it in Shortcuts. Could be a while before there's any further development. Shortcuts really is a terrible piece of software!~~ Thanks to @gluebyte's Shortcut Source Tool , the shortcut is fixed and development can continue! Direct download link Here you'll find the main shortcut and some auxiliary shortcuts that work by calling the main shortcut. Note: iCloud syncing sucks and can cause issues. To reduce those issues: On iOS/iPadOS: open the Files app, navigate to Shortcuts folder, then long-press on the GeminiAPI folder and select "Keep downloaded". On macOS, go to your iCloud Drive/Shortcuts folder, then right-click/secondary-click on the GeminiAPI folder and select "Keep downloaded". These issues can cause the shortcut to fail with "There was a problem running shortcut" or complaining that a file can't be found, e.g. "Couldn't find file: Default.txt" (Free) API key(s) required You need a free API key to use the shortcut at all, in order to use the Gemini models. Get one here 🔗 You also need a free API key to use Google's AI text-to-speech (TTS). Get one following these directions 🔗 You can use the same key for both purposes if you enable both the Gemini and TTS APIs for the key. TODO Multiple attachments: mix and match multiple images/videos/documents/audio files using the Gemini Cloud YouTube URLs: include a youtube URL to chat based on the audio/video/transcript context Overhaul to optimize performance Features Drop-in replacement for the "Ask ChatGPT" shortcut action, or Runs standalone for voice/text/PDF/ Document/URL/ image/video-based Gemini requests Grounding with Google search Enable in settings and then the model will perform a Google search in order to improve the accuracy and recency of its response if appropriate Support for various input file types on iOS/iPadOS only (won’t work on macOS): Word, Excel, Powerpoint Pages, Numbers, Keynote XML, CSV Quick back and forth voice/text conversations when run standalone on WatchOS/iOS/iPadOS/macOS When PDF is used for input, you can select specific page(s) to include, or use entire document Settings menu : Configure shortcut settings by running shortcut and selecting "Configure settings", or typing/speaking "Settings", "Config", or "Configuration" as your input (case insensitive) Select from built-in system prompts to guide Gemini model output, or define your own Text replacement in system prompts: Use "{date}" in your system prompt to include date and time with each request Use "{location}" to include city/state/country with each request (iOS/iPadOS only) Beautiful rich text display of conversations , with several built-in color themes , or you can design your own Logs conversations with attachments to Files App at /Shortcuts/GeminiAPI Chat manager to view/export/continue/ search/create/ delete/copy codeblocks from Gemini conversations Open chat manager from main menu, or by typing/speaking "Chats", "Chat manager", "Chat history", or "History" as your input (case insensitive) Currently, Gemini-Pro API access is FREE Uses Google AI TTS for output speech (optional; requires free API key with TTS access, which can be the same as the one you get above; see here for instructions for enabling TTS for your API key) Code execution Enable in settings and then request to do something that can be solved with code, and Gemini will generate and run code to solve the problem Google's documentation for code execution For example: "What is the 23rd value in the Fibonacci sequence? Generate and run code to calculate your answer." Example response: def fibonacci(n): """Calculates the nth Fibonacci number.""" if n <= 1: return n else: return fibonacci(n-1) + fibonacci(n-2)

# Calculate the 23rd Fibonacci number fib_23 = fibonacci(23)

print(f"The 23rd value in the Fibonacci sequence is: {fib_23}") Output: The 23rd value in the Fibonacci sequence is: 28657 Explanation: The code defines a recursive function fibonacci(n) that calculates the nth Fibonacci number. • Base Cases: If n is 0 or 1, the function returns n. • Recursive Step: Otherwise, the function recursively calls itself to calculate the (n-1)th and (n-2)th Fibonacci numbers and returns their sum. The code then calls the fibonacci(23) function to calculate the 23rd Fibonacci number and prints the result. Acknowledgements Thanks to u/No-List2632’s AI Assistant shortcut for the inspiration for the "show/hide conversation" button Feature infographics This shortcut is design to be run standalone or from other Shortcuts. It passes the input prompt to Google via their API and returns the response if successful. Logs conversations to Files/Shortcuts/GeminiAPI.

Input: Text (a prompt, possibly via Share Sheet, though no dialog will be shown when used this way, as this is how to use the shortcut as a drop-in replacement for the Ask ChatGPT action of the ChatGPT app). Model temperature: Can also set a different temperature any time you send a message to Gemini by prefacing your message with “temperature” followed by a number. This setting will persist for the rest of the conversation or until you change it again. Can also use “temp” or just “t”. For example, “t 0.9 Do people make furniture from avocado husks?” OR An image, PDF, other file, or URL, which will cause additional dialogs to show to collect a prompt and determine other options, OR a valid Dictionary with optional keys, “prompt”, “messages”, “image”, “model”, “temperature”, “apikey”, “repeat”, “speak”, “datetime”, “checkupdates”, and “disablelogging” OR nothing. If no prompt is provided, user will be asked to provide one before continuing.

Output: A text string containing the bot response, OR an output dictionary containing “messages, “output”, and other keys (see below). Dictionary input arguments: “prompt” a text prompt from the user "system prompt" a text prompt to use as the system prompt “model” a valid model name to use. “temperature” a number between 0 and 2 “messages” a JSON array passed as the “contents” to the API. You don’t have to build it manually, since it’spart of the output. Just maintain it between calls to the shortcut. (see https://ai.google.dev/tutorials/rest_quickstart#multi-turn_conversations_chat). “image” is a text string that is a base64-encoded image. “repeat” is 0 or 1 to control if this shortcut repeats automatically (ongoing conversation). 1 to repeat, 0 by default. “speak” is 0 or 1 to control if output is read aloud. 1 to speak, 0 by default. “datetime” used to save image/json files. “checkupdates” set to 0 to disable shortcut update check, which are enabled by default “disablelogging” set to “1” to disable logging (to Files/Shortcuts/GeminiAPI)

Set temperature in prompt by prefacing prompt with “temperature” followed by a number. Can also use “temp” or just “t”. For example, “t 0.9 how do we define life?”

https://ai.google.dev/docs/concepts#model_parameters https://ai.google.dev/tutorials/rest_quickstart#configuration

Dictionary output content: When a dictionary is used as input, this shortcut will output a dictionary with “output” string, “messages” JSON array, “title” automatically made by Gemini, and “datetime” string that can be passed back into the shortcut. Performance comparison All the features of this shortcut make it slower than more barebones Gemini API shortcuts like my Gemini chat lite . It turns out this is largely due to the number of actions in the shortcut, which increases the time it takes to initially load and parse the shortcut before running. By removing all recursive shortcut calls, I've managed to speed things up considerably in this shortcut. This shortcut is still over 22x the number of actions (892) as Gemini chat lite (40). Let’s see how much it affects runtime. First with a short message, so that more time is spent processing than waiting on Google: Prompt: What is the sum of seven and three? (35 characters) 5 iterations Method Speedup Avg Time Avg Chars Gemini via API full 6.4s 41 Gemini via API (update checking and logging disabled) 1.7x 3.8s 41 Gemini chat lite 10.7x 0.6s 42 The shortcut runs 1.7x faster with conversation logging and update checks disabled, and the Gemini chat lite shortcut is nearly 11x faster! Now with a longer message: Prompt: Write six paragraphs about the benefits, drawbacks, and clever tips for living vegan. (86 characters) 5 iterations Method Speedup Avg Time Avg Chars Gemini via API full 9s 2973 Gemini via API (update checking and logging disabled) 1.5x 6.2s 2508 Gemini chat lite (standalone) 2.6x 3.4s 2934 Again the shortcut is 1.5x faster with conversation logging and update checks disabled, but the Gemini chat lite shortcut is only 2.6x faster; much less of a speedup than with the shorter response. Conclusion : If very short responses are expected, the Gemini chat lite shortcut is considerably faster. For longer responses, it’s still more than twice as fast. If you want conversation logging, image/video/document input support, and beautiful rich text output, it slows things down a bit. Shortcuts using this Shortcut A. What is this? 🔗 A simple example of a shortcut utilizing the “Gemini via API” shortcut This takes a picture and has gemini describe it. This is intended to be used via Siri or through a widget Same thing as selecting image input in "Gemini via API", but with fewer taps B. I have a question 🔗 This shortcut allows you to have a voice-only, back-and-forth conversation with Google Gemini Pro, using the “Gemini via API” shortcut You can also pass this shortcut an image/photo via the share sheet and the image will be included with every request so you can query the same image repeatedly. (No previous messages are retained when an image is used, due to Google API constraints) Basically the same thing as selecting dictation as input in "Gemini via API", but can be used from Siri more easily and requires no tapping B1. Gemini Chat 🔗 This shortcut allows you to have a back-and-forth text conversation with Google Gemini Pro, using the “Gemini via API” shortcut You can also pass this shortcut an image/photo via the share sheet and the image will be included with every request so you can query the same image repeatedly. (No previous messages are retained when an image is used, due to Google API constraints) Basically the same as selecting text input in "Gemini via API", but with fewer taps C. Create calendar events 🔗 Suggest/create one or more calendar events from a block of text shared using the Share Sheet or copied text, or a copied screenshot (using OCR) Because Gemini Pro is used, you get contextually named event titles, notes, and it automatically suggests which calendar the event should be added to, as well as auto populating most event fields You get a chance to modify events before they're added