VelvetShark

My AI agents journey

This is my journey with AI agents. I'm starting on January 1, 2025, and I'm going to work on it every day for at least an hour, until at least January 31, 2025. If it goes well, I'll keep working on it indefinitely. This record is both for me to keep track, and for others to learn from my research, my wins, and my mistakes.

January 1, 2025

  • Cloned Eliza repository
    • Decided to go with Eliza because it allows more control, has many plugins, and is very actively developed.
    • If all goes well, I might even contribute to it.
  • Used default character file to start quickly
  • Configured my agent to use Telegram and Discord
  • Exchanged the first message on Telegram
  • Her name is Deep Blue
  • She's alive!
    First conversation with Deep Blue

January 2, 2025

  • Put my money where my agent is
    • Bought $5k worth of $ai16z
  • Submitted first PR to Eliza's GitHub repository and got it approved
    • It wasn't a silly typo fix type of PR, it was a fix for a git command to checkout the latest release
    • Day 2, and I'm a contributor already!
Eliza contribution approval
  • Started customizing my character
    • I don't want 10,000th Eliza. I want my own, smart, quirky Deep Blue.
    • If you don't want to create your character manually, try this tool: https://elizagen.howieduhzit.best/
  • Started with only a few examples for each character detail, with a goal to build this file up considerably as I go.
    • Character file 84 lines long.
    • Default Eliza character file is 531 lines long.
    • Long way to go until Deep Blue is deep enough.
  • Made Deep Blue's voice different than Eliza's.
    • If you don't know where to look for voice model names and how they sound, you can play with them here.
// characters/deepblue.character.json
// (...)
"settings": {
	"voice": {
		"model": "en_US-kristin-medium"
	}
}
// (...)

January 3, 2025

  • Tried switching from Anthropic to OpenAI, to use "o1" model, but I'm not at the high-enough tier to have access to it via API. Trying out "gpt-4o" for now.
  • Spent unreasonable amount of time putting lots of instructions into agent's character file to use proper capitalization. It still used all lowercase. In the end, had to ask to write properly for it to finally work.
    Deep Blue stop lowercase
  • Put Deep Blue on Discord
    Deep Blue on Discord
  • Created a Twitter account for Deep Blue, configured it to post (random stuff for now) and confirmed that it works for both posting and interacting.
    Deep Blue on Twitter
  • Reading Eliza's documentation. A rare case of decent docs, especially for a new project.

January 4, 2025

  • Updated to the latest Eliza release(v0.1.7)
  • Struggled with runtime errors, plugin installs, library conflicts for half an hour. Turns out I needed a "turn it off and on again" treatment with pnpm clean.
  • Tested image generation capability.
    • Good news: it works.
    • Less good news: it's not that great yet.
First image generation

January 5, 2025

  • Successfully sent ETH by sending a message to my Deep Blue agent on Telegram.
  • A simple message is all it took. Based on that, it proceeded to send the transaction and then sent me a confirmation 🤯
Send ETH in Telegram
  • It was my first test, and I thought I'd get a confirmation message. I didn't 😁
ETH transaction cost
  • I'm glad that:
    • I entered the correct address and that it was mine
    • I didn't send a large amount
    • Gas was cheap when I did this.
  • If not for all of the above, I might have sent my ETH to some random address or burned it.

January 6, 2025

  • Mostly planning, reading docs and learning
  • Focusing on studying /agent/src/index.ts file in Eliza's repository, the main file that runs the agent and where all the magic happens. I want to know everything that's going on there, before I start adding my own code.
  • To keep this update less boring, I wanted to quickly generate a picture of Deep Blue, my agent.
  • First, I asked her to generate her own picture, but she was poetically refusing to do it: "Creating a photorealistic image of a character named Deep Blue would be like trying to capture the essence of a dream that dances between reality and fantasy."
  • I then fed her a character file (where all the agent details, style, lore, attributes are defined) and asked her to generate a picture based on that.
  • This is how Deep Blue thinks someone like Deep Blue looks like:
Deep Blue self image

January 7, 2025

  • Finished going through /agent/src/index.ts
    • Now I know what's happening when an AI agent is launched:
      • What starts when
      • Where the stuff is pulled from
      • What are the defaults, etc.
    • Time well spent
  • Created a simple PR to simplify some logs shown during agent startup
    • Approved and merged already!

January 8, 2025

  • Chatting with PDFs works in Discord, comes with the plugin.
PDF summary
  • Memory is retained and can answer follow-up questions.
  • If I wanted PDF reading to create my own actions, Eliza framework provides built-in PDF processing capabilities through the PdfService class. This service extracts text content from each page of the PDF and combines into a single string that can then be processed or analyzed with my agent.
  • Improved Deep Blue character file
    • The first version was rudimentary, just to give it some color and to make it work
    • Focused on improving style, provided lots of message/response examples, added many "don't do/say X" (regarding style, not content)
    • Took the character file from 84 to 429 lines
    • As a result, Deep Blue is much "smarter" now and much more pleasant to talk to

January 9, 2025

  • Gave Deep Blue access to the web
  • She can search now and provide up to date information
Deep Blue news fetch
  • She's getting more and more useful

January 10, 2025

  • Taking a step back and learning more fundamentals
  • Going through Neural Networks YouTube series by 3Blue1Brown
  • It's a great way to understand how neural networks work, and how the building blocks of AI agents are created

January 11, 2025

  • You win some, you lose some. Today I lost 3 hours of my life, with nothing to show for it.
  • I was working on a URL summarization plugin for a couple of hours, only to realize that its functionality is partly implemented in the web-search plugin, so it would be a waste of time and probably wouldn't make it to the main Eliza repository, so it wasn't worth it.
  • Scrapped the plugin when it was ~80% done.
  • Then about half an hour lost on cryptic error messages for another plugin, and still unresolved. I'll go to sleep and try again tomorrow.
  • Today was not great. Didn't move forward much, and also didn't learn much. See you tomorrow!

January 12, 2025

  • Updated to the latest Eliza release (v0.1.8)
  • Looked at all the new plugins and features, Obsidian in particular. I have a whole load of notes in Obsidian, and I wanted to start using that goldmine of knowledge since forever.

January 13, 2025

  • Had to clean my inbox today. Came up with an idea to have a "chat with mail" agent
  • That's how MailChat was born
    • It's a simple agent that can read your emails and answer questions about them
    • Started working on it, from scratch
    • Proof of concept as a standalone project in Python first
    • If it works well, I'll try to integrate it into Eliza or create a new plugin for it
    • Handled Gmail API setup, authentication and tests for API functionality today

January 14, 2025

  • Build an MVP version of MailChat with Python
  • Simple proof of concept
  • It worked!
  • It could read my emails, extract deadlines, send me reminders
  • But it turned out to be more of a bot than an agent
  • Decided to recreate it in Eliza, with a goal to make it a proper agent

January 15, 2025

  • Working on Gmail plugin for Eliza
  • Mainly fighting with errors
  • Plugin development docs are very lacking
VelvetShark

by Radek Sienkiewicz