Skip to content

Image Input

Inputting the images to your agents is quite simple and follows the same logic as text with a few minor differences. There are currently three ways to supply an agent with an image:

  1. Path to a local image file
  2. Path to a public image online
  3. Byte64 encoded data of an image.

Below you can find all three methods in play. Simply provide the attachment variable to your UserMessage.

attachment accetable values

The attachment can be a str or a list[str]

message_history = rt.llm.MessageHistory(
    [
        rt.llm.UserMessage(
            content="What is the image below showing?",
            attachment="https://cdn.britannica.com/39/226539-050-D21D7721/Portrait-of-a-cat-with-whiskers-visible.jpg",
        )
    ]
)
message_history = rt.llm.MessageHistory(
    [
        rt.llm.UserMessage(
            content="What is the image below showing?",
            # attachment="path/to/local/image.jpg", # Uncomment and provide a valid local file path
        )
    ]
)
import base64

# 1. Read and encode image
with open("random.png", "rb") as f:
    img_bytes = f.read()
    img_b64 = base64.b64encode(img_bytes).decode("utf-8")

# 2. Create message history with attachment
message_history = rt.llm.MessageHistory([
    rt.llm.UserMessage(
        content="What is the image below showing?",
        attachment=img_b64
    )
])
message_history = rt.llm.MessageHistory(
    [
        rt.llm.UserMessage(
            content="What is the image below showing?",
            attachment=[
                "https://cdn.britannica.com/39/226539-050-D21D7721/Portrait-of-a-cat-with-whiskers-visible.jpg",
                "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAkGBwgHBgkIBwgKCgkLDRYPDQwMDRsUFRAWIB0iIiAdHx8kKDQsJCYxJx8fLT0tMTU3Ojo6Iys/RD84QzQ5OjcBCgoKDQwNGg8PGjclHyU3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3N//AABEIALcAwwMBIgACEQEDEQH/xAAcAAABBQEBAQAAAAAAAAAAAAACAAMEBQYBBwj/xAA8EAACAQMCAwUFBwQBAwUAAAABAgMABBESIQUxQQYTIlFhMnGBkaEUI0JSscHwFTPR4QdDU/EWJDRjov/EABkBAQEBAQEBAAAAAAAAAAAAAAEAAgMEBf/EACMRAQEBAAICAgICAwAAAAAAAAABEQISAyExURNBBGEUInH/2gAMAwEAAhEDEQA/APWzzphruCO4Fuz/AHrbgVRcT4ynBuNxi+uYVtbnCqhJ1qfPlyrPf8hcWXh95Cy2EcpddcVzqOUYdB5fSlPRhSrK9iO1UXHrTuJ9K3sQw6j8Y8xWpOc7nJoTtKudPjVbwvii3t5f2/4reXRUlnSrh510cqkrrrii21/HaGJm7zHiHTNWIx0z8ayfajjUfAuLW9zLB3gZMA+W9WPDb3iF9MkpTurZt/Z5j31ntnprPS7pUtunKlWmSpUqVSKlSpVIqVKlUipE0EsiRqWZsYrJdo+0jWzBbY5otwya1ks0cS6nbFV8/G7SI/3FHvrzm67QXU6kGRgD0FVb3jsxJck+tZ7tdXqF52kt4wBE6lsVl+Ido5hdKQ+d+QrIPeNqO+aDvdZ1FsUd1kb+PtGWQF/a60qxEfEgiBTvjrSp7LHp3a3gFjx2w7y8MqtDkq8C6nHpjfPyrB8NurDimrs7xG6eZE2tLmRTGdXQNv8A4rUdju3FvxRxY8Sxa3vTUcCT0Gaqv+Q+x7GQ8V4ckzHYfZ4LcZB/NlTn6E10YYy/s+K9k+MRl/u5EOqORTs4HXGc4Neu9l+0UHH+HrPG4EqYEsZ/Ca8+tOM23F7D+hdrEeC5j/8Aj3bqVZD0DZ3xWeu4eKdlOJKY7gpJ7SyRE6XHywfhQXvjsdXhIOnoK834RdzcG7f3UU8bJBfyMFYghSx3GD8TU/sl2+teKlLW/wBNtdkYVvwv/g1nP+QbZkvk4hallKsCza8BSDscdatT1xWw/PPrT2cisz2P4ynGOGpdk/e8pU/Ka0Jk1oKgxH/JUAleydvZDYPuyDW6hx3KaeWkYrF/8ioV4ZFMeStitdwyQS8OtpByaMH6UQpNKuFhg6uQqtuuKRQNikLImlms1xXjK9yhVsYO9VE/aZ0icwNuKLYcbzO1A0qqcGvNh2nvpDl5MaaAdorhUMhkyAeVHaHq9NSQNyqNxK9WyhZ36cqxvD+0zaNWrBNRONdoJbmEpnNO+lI5xvtRNK5SOTAPSstc3ZkJLPkmmZ5AcludQ3YY2rla2cefG2c0HfjHi5VDnkC8uZphpiDpPOgamTXaatI5UaXCnaqid87UCyaWA8qmdXLSjNdqCLhAMHnSpLS3/Aorzh0nFeChpkU6pCG3QeuTn0wBmp/Zn/kO94XpseMJJc2wXQWJIljH6n3GqBb49mOItJwO8uu9VtLd7HpjYeRXmfpT13d8C46e8kU8JvjjWVGq3I6sABnUd9uXrXULrtH2dtOMw/1LsiYrmHnLbxs7XAY9SGJ+W3xrO8Qu+N8J4aeEcVjkWGQBkjm2dR6DOcVVFpbG472yuWUgnRIraWx8DtQXFzLcyNJdzSySN7UjvqzR8pGYN7ag4z/N60PCu0rrH9j4sj3Nq40a/wASjy9aoAh5xvrHl/qu7McMukeWmjU1fZvj8PZ3tG6W9wtxwyZsOy9B0PvFetS8UhjiEhkDK26spyCK+fTFCgVndVU7AkHY1puz/FGt9NlNeI8WoCNmf2c9Pcau6ntvO3twk3ANny+oECrbsxxeD+gWod1DKgBB9Nq8d4v2kluoGtXGiQOcFeWByH88qvOByfa+GxywgnzBPXrV3z9Hrr0binaOBYWVHBb0rE3nGJdbMrKQfM8qYkti6tlMHNR57Dwj7vNXfT1wEl/NISGnXFNGQHZrlRRf0/I/tUacJ1Jq7r61n0fZlHt43ybkEnoKOKW3ZHVXb1xQNY6R/aqXw6zRxpaNQS2Mmq4pqKt7ANkWRseVcaQzPtA+D51pJuFRWkJl2254FOWdpaXVsZEbDchvirZfhMndLIjaRB7VAscjDHcL8atbgJHK4uHwFOAaYaS0T/rKfeaFiGLUk5aCPPrTq2gx4oIcetPrfWEe7Srmgl41w8kjvAcdBVic+xw43iiA9BXBZWmf7IPoFqNN2htFH3UTMaiN2kmY/d2xA9aPRxam0sxt9jpVSPx27LHKL8aVOwYur3tFw3ii/Z+O8PltZsahIqnZuWcc6oL/AIbBF44L23uYW9kxtuvvXpTfGrOSykEjMZlkOe8Jyc+tVmoNy1YrowdMKqcpIp646V0nA8aL71oYdS5wxGPMUZWQgYKn31IKFVcPESHHLOalO9zNmWa4DufCGlcfqaZUPq30/MUckUZUNNIE929ZqCtplCGvINI5+Pf60zdypD93bSxzR8yDHsD6EnnUySCD+nmV55I0L6NoM5HnzH8FU+MuVQMyn2aMRwEyDxPu309KtOBXdxZ3IhV+7jnOxPLUKpZYZISqzqyl91zmijYrcq8GrUWXZhywR/iqzYpcrcNxK+j1HWpz186j/wBXvlIyVI8q73YlxJGQUbcEGu3Fs2F8OaxOP9unb+h/+oLoOAYVI86TdornB0QD4VIh4eZLZSVqQeFrIoBGMCja3JGbuuN38zYC93j61bdmb+eWVhMcsN8US8IXxe+nY7dbC7j/APs2pgsbDht8bq5EM6rgjIBpm84U6TmSKRUGrUFqts0tre8+3PdMrBcaOgp48UhnkIMjuegANavOy5IpPSl4/E8kcqyIMkcxWTWwlZsHVj0rbcUuI2kBOpW0+y1V6IqON1OazYooY+FMT7JPvqVHwonYhR76vJ0XQUyBkcx0rJyHuuIRaLl5PFuDyqxatDwjBxt8KTcK0RaqvoQrFT6U5GgkJQKSD58qcWs/9iQ8+dKtEOEJ+LQDSqyFWS2C3tk8Uo5HY9RWW4haPYThHi8B5HbFbuBiwOmI49ah3/CV4jGFlxGV9k7ZPpWfyTj+3O4xkbaiTpG3qK7pUHBjY1rbfs5aRxHXI5PnkDH0qPdw8Jj1RwOZXA30Y+rH9qePll9QZKzsUTPJiOFienXHr7qNQM4bQV643zVle8SSCHurZVUsviCDbHLnzJqmMoQZkIA6Z6/Ct/IuJwngQd5OHuZeSKwxGq/v7hVfeu80wKLCgH/aUL867JOs1uRCHM+7BVXko6n/AF86Rl79IZJmKxQx4JU7k52AH771A+83e2hhkkLDIJOMk1HeEQgSSwkAKSCGAHp9cU7apJKryKiRRA5LKuT7hUyEJbhZWwN9jJuzY8h0rFtjSNZXtzbIywsCin2HGcH9qkS8QWfBm76En/t+MfI7/WmLtYmh71VcSa9bsW9rPJfT386iSXOrDMvdkbBPOqVNlwC9tEtjE16rylthJlTV7kYDgAgjmDXmXfd2oeVDvyOKkWXEryIn7DJP6oPEPfilrs3oCsrDSdWegqt45IsgjGSjRHOph+1Ulr2ouAGDCMPnBZRgmoN1f3F8zd4xIzkMelR1Nmv3mYqcmLO+53+VTbf+mO/fXF8wAX+wpfwn3msqZJQ5LjCsPaPT/dXHZqyS8vt3GmJNeW5A7Y+O9HKW/sStJxHjEnEEtocPPbIQPHjIG24JxUWaayXuysLFhyLVOXh0eD3l2B7qaNjYKVMk2sjrVPgw6t0JQdUagsOWNqobq6ijmA+xRltXPb/FaTXZLpzcZAGMVFkseEyZZ3bUTnYmlHI7nREp8CZHSuNdgjxXAxnkKcWz4bKoGqTw7bZp+24TwzVqLPnkM5rLSRFZQyxq+s7jrSqZGkEaBEY6RsM4pVbQgRtE7LqZgCPMUUjWyEDJwTzLCum0jxvHk0BsoyM9xmvJefDc6uXr6Yzj/EL0yNEyGOFW8I6N/mq9rhmBbV4SuDqGPp1rfGwjJINupHNc/pWO41wi4gt5+I8SAR2bTFEG5En/ABnavR4+W/rEpmlVmHdcgMam6e6mJpe98KZ25k/rXZCH8Tf2yfjTLnpvz616Ihd8Y43RD4W3bH4h/inJGeNFOoaZF2TYjnv7tsU2kTE7Ic888x8a7MFxoRQAu+ejE+VTK8tL1IuHiKOISSrqZdvZb8x9wqPbrNAkNxOSyzAlQN2EYwC3LYGoCTHxC41FWxrGcagOny/arAzLPIY5nHdKA9xKux0jkg9P3rOGAM2tWK5CZyc1Guu7kGUOH6etHNcfbLiW5YiOPIKxg8hjAH0p17VdH/uCe8Y+CJTy5bn5n+CjMa+TaRM5XB1nHPnius0tovjxGjbeHGo+tE9yyFu6A1dWA2FUjSvNIzSPkk0yK1NunjCq8RCyN0ToPOihlULgt8zUWGEyErGurzxvREFDpLYIrQ05cPlfuzlyffVhwziDcOYBFDZ9oE7fOqxXABUHJNKNgcknD75Ppjas1a3NheQXaMVURkDJVzmpUkJguzG8S6ce1jnWEt7meHaKbK55edXUnErlLaH7SJ4pBnuznBI/cZrnd+2taMQI5+7CjHXTRKjZP3g+FZ2w7VTRaVnhinQea6T9KvrPtZwuRgJoJIM82wGrn18n2J/0RikbYXDj0WhNjI52u5wavLS+4TdELbXUDsehIU/Wp0kUMKa5GVEXfU1XTl9tZftjpLK7Ryq3E5A60q2fcJjY7HlSp68vs9Vb/WbEkgM2R1yTmmv62jNiKCVh0whp2Oe2WMfdoPSui9twfCMH0rl+bjPhjvDR4jO4yljL8RiqXtFFecXgWNYHjZd/ENh6860BvkJwGxQNdr55o/yZBecef3XZu9jt1eJZJMZJwv8ADT9twX+mxS3nFYNcajQkbHGonzrb/aj+FsCoXFUg4ham3uydJOxBwQa1P5c/Y7R53dXjXJYhsINgoAChfTFR4xjR8RhuVaHifZttbHh5VowBhNXiJx5naod1wC7ijhWJWlmkGXK8o/IV6OPm4cp6oU7N+HAAHPJpxZNNu0f5mB3z0z5e/rTbwSwqDKjKGHgBG7dKcuLeS3cRaT3pQOwHTrn5YrqnI2LARrgEtkk8/T5U/PdK87GEHScDBz4sDGaj2tpd3m9tBNKq7BlUneuyWz2zkXAKMvtA88+WP3o2fBPghY206c4O5251d8M7Kw3HCGuTJqmFsZQpHhGPP4frVJb2k0xjaQaIZNgRgE+4fvW2s7qJbKG3kl+zRFQk2g5JABI+dJkQOFcGge3e3glfvdkaZR4RtuBkjb/FYucES924IkB8WfwkbYrfx3VjZRBLbLOzF2Kv7OM4Hr6+8VnOM2L3PGlkgGr7WBIzeTE4bJ9+/wAaDYesuz6TcAubwyMbhPGIQNmQHxEnzHOqyGIRoASCBvuvWthDPNw+SFBKJNSiOZGUHSuNwB161Rcdgt47hfs6hIW1NtyOT09Bj60WasxFgMCuHdVIB1HbG9dnvopp2dhnyGcimMxxo3hDOwwCx5fzzriGNVClAxK42HL1rPVnQiZFBKxJ3nTUmce6oj3BMhJ8Rqc0QVQY8MCpBG+QajXVtGrao3Ixtp6nA5+m9Xwh2nds6tMXSMncZODjp/uvSOz93wOW2SK3ypJ1CK4kJJ92Tg1guIRhLO1jRUKqviG2QdqiidU9lQzfmJou0y49qE0JHNficUq8WPFbpPCtxcqB0EpxSo6/213endwooWhX8uakacDOnNDjO+MV8rY86OIo+oxSMSdKkFMb0HhPOnYjHdAjAoDbqDvUsFRXCATledWpEMC0DW61MkSTAxp+NA0bnn9KdSsueH280iO8CySJ7LN+Ghn4eko7wIgm5F8DOPI+lWYhOfFypq+nSxtHnYMVXkF69P4a68fJz3I1OVno3FGyBVRVUL+Ws5xmLhr3jTKGnmHNA/3Sty3OMnlyB+VR7zit7drqk+5hbIWNTjV7+p/SobNqXDbY2xjGB1/X616vF4LL3takSrlklWE20rtKpy5xsCRsB0FRbjiEeyqjBcc2OT1zT/DYftcN00cwiCMSq433/wBCoXCrNprrxRbYBLN065r0t6eErB3KncnY/U1Lku2EMUqsVMTh+uy7focVVzS+KQR5EZkJUHoancLngljkt50f7wYUod9/5yqp+Vva8RiuZYnn3hkfS+GxoYYPwH+Kg30ZeOSK3OtreQ+Neo5cv51qDb2d3bSM0cb3EG+to0JGM7E4G29TbSSJZe/jl0DAVw59o4PI8sVncKk1ZGwA9K4TnC+f6VcXyWDQu4hZLokYVNhjz3q0m7L8NURyJxqNGaIP3MkWSu2cZzj9KOXl4cJ/sPx2/DLl2XYcgM0JlYbasVZcQ4bNbWIuCgaGY4SVSCNjuPQ++qqVDqBHIrtTLL8MWYRcsTlq7a2s11J3cCNIw6LTcZBK94rFQdyPLqK2NhwG2S4ju4pJREQCiIcYOOpzWPJznGM2n7LgtqtrGJ7CEyY8WJT+9KrnT5Uq8d8gWHej2RQMVzu/woG01wAdK8t4siLBRt1pllbOzYp1UYt7OVopE9MVZiNKhAyTmultuePWhwoODXChZvDyFZtqdLeufWhB3pLq1GuqrHJq9gQZRzpi+uUgt5ZXUOiL7JGc+VGEfrypuWJGDK6kgkZGMg/CtcbllpjBzd407T3CgSMQQgGAgOwpqTEatltRxqJPn0ArTXvDUia6vrhwAFxGM7A8gd/LA2rOy27PGo5O51lDg6v4P0r6/j8k5T06Rx2jtLaHuHbTLp7xM5HIbf8A6IqVxWa6tZ2XEcB06mCc9/3qELabOv8AEDkpq3HSuXBmubp2ZmcsQCfy74/et7M2tWmImDQ56I5z8etCyNDOFBwTy351Yz2f2Piz2bIO7lTcD3dPjUOVQbU5KiSLbbn8aZdg1d8JknktooIhpTvSzjUMseW2SM/tv51tbi24LeKktzwsaH07xlkPpkZH861g+zMF4OJQFUc2j+22PD8/MVsphM53lJUAKuTghR08iK+d/M28pONx04+WcZlituuDWipcQWS5TUGQktjB/Cfh0z5b1IaNLqW2lZXjmt4RCUB2kUfmHX/dTBlht4QPkaDXNrbSqgfrXG87fn5c/wAnJluJdn7/AFSrZSK1u5z3OojfOdhyqDw/gt5cpOSpjKbaZE5+6tuGfHiAB8hR/wDSJPt/hrrP5POTMY7VirHgVzdW1xbTQSQzI4aJ5FwD0IrWcLtfsNgls7a2UYLflp7XIzYogxBwzYNc+fm58mdGNAGMZ9aVGrHSN1PvpVz7VHNSk5xii1qooXC4XIxThSNhvVUBpl0586DwkZrpgjU5VcmuNqGwGKInAq5ogEycc64jHrXXZtS6PjT1+iHDdFzQtlNwu5oyd+v7V3csM4x6VQGdT43XekMnmtOd5pyEXNNB3Ztxg0UFNEk0ZjlQMp30FQRt1qkXhqxTSXMsbbagN+f64/nlV9gqclc01dK1xCUAIxyxXTxeS8bka41lHmikt5kVss0mnDpjUM/+aV+k8TK06giRz4Y/ZweZHT+fKdwnh88ktxLPy1MsYbqc8/dR3vDr2Sa0jySqtuU5YxuP1+de3l5eNtkbt0z2os3723vUUvp8D+47iot1wieaQSpGO5uF8QHTArY9wzKFYAgryNHHEqLp0YwOleafyeXGYzqn7OW09vw2KK4Uo2SwU9BViTkeBc0+WVF5KPU02t5GpIJU1x585bo9AALbMMGutb5wfKi+2RNsAAfMUSurnnmsSxBYbDw5os/h00RAk5UIjZTtShpHsdseZpruLfvCxOT0p3WdODXAGztUiwo28PxrtItKDgLkUqvQO6VbSTjfyoUSNcsTkjkKbjUjOTkD1rquCSwOQOlbRZIbJbRXSdW59jrXCrFcoMedB001ek6yRtgHV8K6wEQwucetdjGARjHrmgAIkJJyNNPqIOhc786cjCjPhz5UCS4zooRJJq8TYFCP6XC5IxmhGFbDc6B3YLlDn0plXmc+IYFGJNBHSlk755VHQshz5U4ZCWBNM4+iAAE5HKiCMTs/woe8bbGn40pMscqoLelHVDfwAZpxGLLqPI7CoTNIwy60UOScAYA6fvViK48WVcZx0pmO3hG5TANTD94cfiFNsdB8dY6TQYktIACUjY+6nUTu98EZHLFdDh911YHlXSNA9phmtZxJKDjdgB6Vx8KBpJJPQmh7oNswY+Rrhtc820Y+dGUHUVjkYxilrYRsKiNrjYhQ49TXY1nZvGMr50XdB3vZByXNKnVVVGNNKnqReEFts+pocsBlBgcq5SrU+ARkkAyfF5b1yKaRye8RfTrSpVRC1ndSMY32pEHUSOR50qVKEAgzp50CAu2DyFKlRUeXTr00goVDilSqKPqZzhRy506UZ4SABgUqVCNaWBUMcD0rseT4Vbk1KlWkeRnD4Jz50dxLGijHM+lKlTEimbScedce4Dx70qVYAknC5AXfFCWbzx6UqVCdSV/zcqLW0ZznOaVKun6Qllzu48A6edGMsduZ2pUqyYLTKNs8qVKlW8L/2Q=="
                ],
        )
    ]
)

The rest of the invocation (tool_calling, structured_output, etc) will remain the same.

File Types

Supported file types will correspond to the file types supported by the underlying LLM used.

Image Output

We're currently not natively supporting outputing images. You can however wrap any image generation logic within a tool and provide your agent with that tool's specifications to achieve this behaviour.