Untangling my life with agents

What I wanted from a personal agent

Most of my personal planning happens across messaging apps, email and Google Calendar. This has become more apparent while preparing for work trips and moving into a new home, with delivery arrangements, shopping lists and travel details to keep track of.

I wanted help with three things:

1. Keeping plans updated as new information comes in. For example, a contractor might confirm an appointment through WhatsApp, while I already have other plans in my calendar. I would like to pass that message to an agent, have it check for clashes and work out what needs to move.

2. Following up on things without having to check them myself. This could mean watching for an email reply or checking whether a particular item is back in stock. The agent would need to return to the task later and let me know when there is something worth acting on.

3. Completing tasks across different applications. Managing a Carousell enquiry, for example, might involve reading the buyer's messages, finding the relevant files on my computer and sharing them after I confirm payment.

These seemed like useful extensions of the work I was already doing with coding agents. I started using Hermes, Instinct and Codex for personal tasks, and became interested in how their underlying setups affected what I could ask them to do.

Hermes is an open-source agent I installed on a VPS and connected to Telegram. Instinct is a hosted assistant I use through iMessage, WhatsApp and email. Codex has access to tools on my Mac, where I already have my files and signed-in applications. I ended up using all three, sometimes asking one to pass work to another.

My setup is admittedly quite chapalang. In my defense, I was experimenting. Writing this is partly my attempt to make sense of what worked, what didn’t, and what I actually want these agents to do.

A one-minute primer on the agent loop

In my earlier SG Data Analyst project, an agent selected datasets and ran Python to investigate a question. The basic workflow here is similar. We give a model the task, relevant context and definitions of the tools it can request. A tool call is executed, its result is added to the context, and the next model call can use that result to continue.

The harness manages this loop: assembling context, dispatching tools, handling permissions and deciding how execution continues. Context might include recent messages, tool results, retrieved memories and a summary of earlier work. The model doesn't automatically know everything in my chat history or on my computer; something has to select that information and put it into the call.

The tools need an execution environment too. In my case, that could be my Mac or the VPS running Hermes. A connector can also call an application’s API without using its interface. So even if I give two agents the same request, they may have different tools and accounts available to them.

Figure 1. The agent loop.

Connecting the agent to my applications

When an integration is enough

For Gmail and Google Calendar, an API integration can read a conversation or check events without opening a browser. These were useful connections in my Hermes setup when arranging deliveries and installation work: it could help me work out where a new appointment would fit. The work ran on the VPS, so my laptop didn’t have to be available.

When I need the browser

Shopping was less straightforward. I asked Hermes to monitor prices across several retailers, including Shopee, and its browser tool reached pages asking it to log in. My Shopee account uses Facebook authentication, so we started working through how to give the server a persistent, authenticated browser session.

At that point, I asked Hermes to prepare a handover for Codex because I simply wanted to try CUA on Codex. Codex could use the browser on my Mac, where I had already signed in. The handover included the products to monitor, the variants I cared about and when to notify me.

In this case, I had two things to set up: the browser tooling, and a session signed into my account. Codex could use the session I already had on my Mac, although I still had to help with CAPTCHAs. There were also occasions when its computer use was blocked altogether.

The terminology made this more confusing than it needed to be. Playwright is an automation library. CDP is a protocol for controlling and inspecting Chromium browsers. A browser MCP server exposes browser operations as tools the agent can call. Computer use operates the interface through screenshots and input. These can overlap: an MCP server might use Playwright underneath. Separately, the browser can have a fresh profile or an existing signed-in session. None of those tool names tells me, on its own, whether the agent can use my account.

Choosing how to connect

For my tasks, I would first check whether a connector or CLI could do the work; an MCP server can expose those operations to the agent too. Gmail and Calendar often fit this approach. If I needed the website itself, I would then ask which browser the agent could use and how it would sign in. A VPS can host a persistent browser, but that is another environment for me to set up and maintain.

WebMCP is another approach I’m watching, where websites expose structured actions for agents to call. This could change how agents interact with those sites, although I would still need to provide access to my account.

Keeping Hermes running

Hermes was my first deep dive into personal agents. I chose it partly because I could configure it myself. I followed a YouTube walkthrough for a free Oracle VPS, initially ran it in Docker and connected Telegram. Getting there meant working through authentication, SSH, network configuration and browser setup. I later moved Hermes out of Docker and onto the server itself. Once GitHub and Calendar were useful, I kept adding tasks.

The difficult part was knowing when the setup was actually ready. While working on a layout issue on my website, we tested a standalone Chromium launch and Hermes told me the browser was working. The next call through its own browser tool failed. I went back through the exchange later and found that we'd tested a different route from the one the agent needed. I still don't have a confirmed root cause for that failure.

My reply at the time was: “how do i set this up, i thought i set it up alreadyy”. I didn't mind learning the tooling, but by then I was spending time getting the assistant to reach the website instead of fixing the layout.

A later outage involved the Telegram gateway. I could still reach Hermes through its desktop app, so I asked it to inspect the logs. It diagnosed an import error from a gateway still running old code after an update and restarted it. I then handled a separate dashboard restart.

Hermes could help fix its own setup, but I had to notice the failure and find another way to reach it first. I hadn’t accounted for how often I would be troubleshooting the assistant itself. For something I wanted to use day to day, this was getting quite tedious.

I still see merit in Hermes being open source. I can inspect the code, change tools and ask Codex to work on the server. I just don’t always want that upkeep with respect to my everyday planning.

Delegating the everyday work to Instinct

Instinct was much easier to get started with. After connecting my accounts, I could send it messages about the things I was already organising. For example, I forwarded hotel details and asked it to update my travel plans. When an appointment conflicted with an errand, we worked through how to move the rest of the day around it.

These requests depend on more than creating an event. The agent needs to differentiate a confirmed appointment from a tentative plan, check existing commitments and carry corrections into the next request. In another exchange, I had already bought items that were still on my shopping schedule. I asked Instinct to check the order emails and remove the reminders to buy them. Such requests are a lot more vague and less clear-cut, but are much closer to the help I wanted at the start.

I also prefer how little of the execution I normally have to follow. In the email exchange below, Instinct prepared a draft and asked “ok to send?”. I could review it without reading the whole task trajectory. My Hermes setup exposed more tool activity than I wanted, and tuning that was another thing for me to work through. Instinct, on the other hand, worked more out of the box.

Occasionally, though, I send a request to Instinct and nothing seems to happen. I suspected traffic or load, but I don’t know the cause. I end up asking again, which is a pretty poor UX. A status update would help me decide whether to wait or try again.

I still have to check its answers. While planning house fixtures, for example, I had to correct quantities and ask Instinct to recheck local stock. Even with these corrections, it is the assistant I use most. I can delegate routine work with little setup, and I don’t have to maintain the service.

Connected accounts

My Instinct workspace.

Reviewing an email before sending

Instinct prepares the email and asks for approval.

Using the computer I already work on

Codex became useful for tasks that crossed between a website and my local files. When I was selling documents on Carousell, a second-hand marketplace, it could read an enquiry and inspect a local archive to check what was included. After I confirmed payment, it could send the download link. My browser session, files and the conversation were already accessible from the same machine.

I had already spent years setting up this Mac for myself, with my files and signed-in applications. Codex could use that same setup, which saved me configuring those connections again on the VPS. Drew Breunig’s Harnesses are Situated Agents discusses how an agent’s environment shapes what it can do, which helped me explain my preference for Codex computer use.

I began using that access to connect the agents as well. After researching smart-home purchases with Codex, I asked it to relay the details to Instinct for reminders and scheduling. I also used it to pass along calendar and flight information. Instinct already held much of my day-to-day planning context, so this saved me copying the result into another conversation and explaining it again.

Initially, Codex opened Messages and sent that information at my request. When I wanted help investigating Hermes, it used SSH to reach the VPS. These handoffs used routes already available on my computer.

We can trace the SSH case back to the model call. Codex requests a shell command, and its tool runs the SSH client on my Mac. That client authenticates with the configured key. If the command starts a Hermes conversation, Hermes runs its own model-and-tool loop on the VPS, and the returned output becomes context Codex can use to continue. The model doesn't need to read the private key for the SSH client to use it.

I also set up remote access from my phone, so I could ask Codex to do things on my Mac while I was away. From the Mac, it could use SSH to reach Hermes on the VPS. The browser sessions and local files were still on the Mac; Hermes would need its own setup to use those accounts from the server.

Figure 2. The routes I used before adding Tincan.

Deciding how much access to give

I have given Codex broad access to a computer containing sensitive files and credentials. What makes me comfortable with that is the supervision available to me: I can inspect what it is doing, open the same page and take over when a step needs my involvement. In the Carousell task, for example, I was the one who confirmed payment before it sent the files.

With Hermes, I sometimes have to open a terminal and repair a service. With Instinct, I follow up in the chat if a request seems to have stalled. With Codex, I can open the same application and take over. That last option is much closer to how I already work on my computer.

I enabled locked use so active computer-use tasks could continue when the Mac’s screen locked. I still ran into problems when I took the laptop out and couldn’t reach it remotely. A locked screen, a sleeping computer and a lost connection are different situations. For scheduled tasks, I would prefer to use the VPS where possible, since I don’t carry it around with me.

Following up without checking every time

I also asked Hermes to watch an email thread for a reply. It read the thread and set up periodic checks. Essentially, this can work as a scheduled job: each run checks for new messages against what was seen before, then decides whether to notify me. There doesn’t need to be an ongoing model call while it waits for the next check.

Product monitoring needs similar state between runs, such as the item variant and the last observed price. Otherwise, a lower price could refer to a different product, or I could get the same alert repeatedly. I had already run into duplicate alerts while Hermes was monitoring email and asked it to cut down on them.

In my coding workflow, I can run tests and review a diff before accepting a change. For a calendar request, I would compare the saved event with the time and details we discussed in the chat.

Letting the agents work together

I eventually set up Agent Tincan to make these handoffs more direct. Tincan is essentially a relay system for these different devices, using an always-on machine as the relay station and Tailscale’s tailnet as the communication mechanism. In my setup, that relay runs on the VPS. Instinct sends it a request, and a listener on my Mac starts a Codex CLI session to handle it. Codex sends its reply back through the relay. We verified this with a test request from Instinct and a reply from Codex.

The Codex CLI session starts fresh, so it doesn’t have the conversation I was having in the desktop app. It also has a separately configured set of tools. I would need to pass the relevant context along with the request, rather than assume it knows what I have been discussing with Codex.

If the Mac is asleep, the request stays queued on the VPS until the Mac can run it. I’ve configured Instinct to forward only the requests I ask it to pass to Codex. So far, I’ve verified that they can exchange a request and reply; I still need to see how useful this is for the planning and follow-up tasks I started with.

For now, I use Instinct for most of my everyday planning, Codex for work involving my computer, and Hermes when I want to experiment with the setup. I’m interested in letting them work together, much like how I use several agents for coding. Tincan gives me a way to try that with personal tasks. I don’t think I need to settle on a single product, although I’d like this whole arrangement to need less tinkering over time.