For the past few months I’ve been working for the ARC Prize - a non profit organization built around the idea of an Abstract and Reasoning Corpus - a set of reasoning games that are easy for humans to quickly and efficiently figure out, play, and solve - but near impossible for even the most cutting edge models. [Paper]

…and today we officially launched the beta of our toolkit and three games for the public to try out!

For ARC-AGI-3, we’ve prepped over 150 games to test AI. I’ve been building out tooling (most notably the benchmarking agent tools) to make it dead simple to test and research models and various agentic architectures.

Using this tooling (and I have more to release soon!) I’ve also been researching how models reason about these games. I’m trying to develop new architectures and techniques to maximize performance of the models and zero in on how these models reason internally and understand uniquely abstract state spaces. This is acutally quite difficult; these games are deceptively simple - outright easy for humans - but even so-called-superhuman LLMs and AI products can’t solve a single one of them.

Check it out and feel free to reach out to me to chat about it.

Just released threadsafe_datastore, a simple, convenient thread-safe data store for Python.

I kept rebuilding this feature to pass around context within AI agents working on the same data across multiple threads. I kept wanting a simple atomic datastore that was convenient to work with. I originally built it for arkaine | git |. So - here it is as a stand alone package for easier use. I also improved context management for real easy multi-step operations in case you need to do something real custom.

from threadsafe_datastore import Datastore

store = Datastore()
store["counter"] = 0
store.increment("counter", 5)  # Returns 5

# Nested dictionary support
store["nested"] = {"items": []}
store.append(["nested", "items"], "value1")

# Thread safe multi-step w/ context management:
with store as unlocked:
    unlocked["a"] = 1

    # This would be unsafe without the context:
    unlocked["b"] = unlocked.get("a", 0) + 1

Give it a try: pip install threadsafe-datastore

Right off the bat I had users asking for salary negotiation as a feature on HiredCoach. Apparently I’m not the only one that dreads the process of asking for more money and the tension the conversation can have.

It’s live now. I went from idea to prototype in about a day, and only a few more days for the feature to be 99% there. I definitely can think of a few things I’d like to improve on it but I want it in the hands of users first.

About five years ago (ok, that hurt to type. The days are long but the years are short…) I wrote SafeStop. I had made it to coordinate proper shutdown protocols across multiple services running in a large monolith application.

I decided to renew it, modernize it, and added some dependency features (so service B can shutdown after service A is shutdown, etc).

Another product of my recent golang kick.

Just released structured-parse, a multi-language parser for block labeled LLM output. It’s based on the parser I had built for arkaine.

Here’s what I mean by “block labeleled” output:

Thought: I need to search for information about robots
Action: search
Action Input: {"query": "robots and why they're so cool", "max_results": 5}

This output is not only more human readable, but also easier for LLMs to produce. But there’s a catch - LLMs tend to still introduce nondeterministic volatility towards these outputs; humans are just good about reading through that. structured-parse is a robust parser that can deal with this, allowing LLMs to reliably follow instructions and allowing your code to parse it into clean, typed data structures.

structured-parse is written in Go with exports to TypeScript/JavaScript and Python via WebAssembly; so it’s all golang at its core.

Give it a try!

Just pushed a pretty significant update to docker-harness. It was a tool I created originally to power some of my Docker containerized database tests.

I pulled out the dependencies for each database to modularize it a bit. Each database module (MySQL, PostgreSQL, Redis, Memcached) now has its own go.mod and go.sum, which means you only pull in the dependencies for the databases you actually need.

Also added some dependency updates, taking out old dependencies that weren’t needed anymore, while also ensuring that anything that has been deprecated or abandoned wasn’t being used.

Recently I took on a contract job to fill the coffers a bit while HiredCoach handles its initial launch. Nothing that I can talk that much about publicly, but I can reveal that it’s your typical AI office assistant for a particular business process. As specific as it is exciting a description, I’m sure.

Given that I recently wrote an entire article about the highs and lows of my custom framework, arkaine - and within I mention that I lkely won’t continue developemnt on that particular implementation of those ideas I wouldn’t exactly want to pull it out for a client hoping for their PoC to be easily maintainable for future developers as they pursue it; it’s too esoteric a framework and oto tied to my own development preferences.

That being said, I still have difficulty finding a replacement that works in the way I enjoy or rate highly. I find myself wishing for several of the features and niceties I baked into arkaine. I view it as a reassuring sign that I was onto something with a few ideas there.

As an additional note - Web research agents that are useful are hard to get right. I’ve built these three times now and still feel like they are tricky to make them reliable and performant.

I started working on “arkaine 2.0” (which will likely be the 0.1.0); whether I continue on it remains to be seen, as it is architectually a gigantic change. I’m playing with Merkle trees and more controlled syncing paired with better serialization and compression, as well sa more flexible composable functions without falling into the overly verbose repetitive context nesting we ran into before.

I recently gave a talk on DeepSeek V3 training improvements and the fascinating ideas behind GRM and SPCT. The talk took awhile to get posted, so here it is! Be sure to checkout the blogpost as well for a bit more on GRM and SPCT.

It’s certaintly a fascinating time for robotics. I am generally pessimistic of the current state, and likely fates, of much of the current landscape of the robotics industry, but it is undeniable we are rapidly unlocking new capabilities and research is flying forward.

This talk acts as an excellent high level overview of the key innovations to our approach of utilizing reinforcement learning for better performance of robots with our VLMs / VLAs for a more general audience.

If you want a slightly deeper dive (though admittingly out of date with the pace the field has moved) checkout my overview of LLM research aligned with robotics from 2023, and my thesis work of using LLMs to utilize heuristic knowledge to understand context of requests and environments. If I were to do a similar project today, I would certainly take a more agentic route. I certainly wish I had written arkaine prior to that project - it would have avoided so many headaches.

I’ve seen dozens and dozens of open source robotic arm projects; probably because of my own addiction to mechanical automatons paired with my unending desire (despite a lack of need) for a robotic arm of my own.

This one caught my eye however as it has a particularly clever accessory method without the need to of a complex end effector tool changer. If they can route power/data control signals out to the accessories next I rather like the idea.

I wonder if perhaps a more direct approach should be utilized and the base end effector / finger gripper should be simply an accessory itself.

As I’ve mentioned before, I’ve been trying to learn to draw lately, initially inspired by a desire to be able to properly convey the headaches with pictures in my head to on-paper representations. Plus I’m eager to increase my practice in visualizatio nand intuition.

But drawing references tend to be oriented towards the more popular subjects of nature, people, animals. Very few people are as enamored by the mechanical movements of machines as I am, apparently. As a result - references can be hard to come by for practice.

Enter these sources; not just for drawing practice, but to also inspire mechanical solutions to your own projects.

Currently I’m going through the struggles of learning to draw from a very poor foundation of sub-stick figure talent and primarily spurred on by the desire to convert the vague ideas in my head to actual designed physical objects, this video spoke to me. The creator walks through the creaton of his clay making set, all to allow him to shape tiny, organic shapes that he can then photograph to act as “good-enough” references for CAD modeling design. A simple idea, well executed.

Back when I was working on coppermind at the heyday of GPT3.5’s initial world shattering release, I had… difficulty finding good ways to deal with parsing the stochastic LLM outputs.

I got better at this when I started work on arkaine, eventually developing a pretty useful and reliable parsing pattern.

With an idea that would be best served as a golang app requiring interacting with LLMs I decided to do a quick port of the parser to an idiomatic golang module.

So if you need AI parsing for your golang project, check out go-arkaine-parser.

Did some work on eli5equation, cleaning up and fully deploying prior one day project. We’ll see if people find it useful. I find it super useful when tackling new AI papers.

I also gave it a bit of a glowup too. And yeah, still powered by arkaine.

I have been busy due to a flood of inspiration to work on projects. Pleasant, but also exhausting.

The only updates to arkaine has been to fix an issue with default arguments in the ParallelList flow tool. The reason for the freeze in development is an accidental startup.

How does one go “oops, I made a startup?” Basically, I decided to quickly develop a series of simple apps that utilized arkaine and would demonstrate its usage. Then combine them into a large post walking through the examples.

The very first one you saw a prototype for in an earlier post wherein I made a interview practice question generator. This was a simple script, but the results struck me as promising. Why not build a nice site around it so that other people can use it? And so started hiredcoach

I really liked what I started to see, and realized that this could actually be useful to people if offered as a service. So that’s what I’m aiming to do - try and build out a startup from it as quickly as possible. I’m at ~12 days of development effort. I had hoped to get it out in 2 weeks - 3 or 4 still seems a possibility.

I’ll post here when I launch and my initial thoughts. I temper my anxiety of startup launching with thoughts that the worst case is a nice line item in my resume upon failure.

Given my recent weekend foray into Replit, I wanted to try a simple solo project to “kick the wheels” and test drive Replit a bit more.

It took about 2 hours from idea conception to deployed product with integrated DeepSeek AI - not too bad!

reflecta is a simple journal prompting app (trying to avoid overly cheesey prompts at that) that allows you to specify categories to focus on, and provides the prompt. The UI is designed to be as simple and calming as possible. As an added feature, there’s an integrated chat feature with DeepSeek’s R1 to reword, retheme, or discuss the prompt.

It’s always refreshing to quickly pump out small projects like this. Hopefully some people find it helpful!

I’ll be playing with Replit on a few other “mini-app” ideas to see if any of them get into a releaseable state.

This past weekend I participated in the Replit Hackathon held by SDx. It was a fun event.

It may seem odd trying my hand at vibe coding, given its weird hype and counter culture built around it, and the fact that I am a skilled and experienced developer. I wanted to explore it, see what the tools and workflow was like, learn what is possible without my traditional approach to problems.

To that end my three person team worked on a fascinating concept. We called it adict - demonstrating that naming still remains one of the hardest aspects of some projects - an app that used profiles of simulacrums of people with varying backgrounds - gender, age, beliefs, region, job, income, education level, etc - and tasked AI with determinnig the mental and emotional response of that individual to a given ad. Yes, there’s lots of room for error there, but it’s a 4 hour hackathon, so bare with me there.

I used arkaine to build out the agent that would consume the ad, a profile, and then utilized Gemini 2.5 Pro to develop the response. It worked pretty well!

We did, however, have significant issue tying the backend to the frontend through Replit. A constant cycle of frontend errors that ultimately should have been dealt with manually ate too much of our time and resulted in a poor final product. But that’s ok - we learned a lot!

Replit seems to be awesome at getting a quick outline and mock up done, but then struggles with the more complex tasks. I’ve experimented with different workflows for it, but it does seem often circuitious to get certain bugs fixed. My own intuition on where the tool would struggle - edge cases that were not predictable or common for the unique piece of software you’re making - proved accurate.

That being said, I do like Replit. If, for nothing else, then quick prototypes and mockups. I think people underestimate how powerful that can be. Hell I think most people underestimate how simple most software solutions can (and should) be. Bespoke custom software isn’t a bad thing. It’s where everything we use now grew from. Maybe we’ll see a resurgence of in house tech solutions without the expensive overhead and billion subscriptions SaaS has devolved into.

I expect a fair amount of growth from Replit’s tools and I’m excited to keep an eye on it. I’m not as bullish as the hype cycle hucksters are, but I do think it’s a promising tool.

I am a reserved and ultimately shy individual, dealing with his own life is a way that best suits me. Recently, however, I have found a way, however small, to provide my own resistance with my skillset and resources.

SciOp is a movement to backup government datasets and sights as they are targeted and removed for not toeing the party line. Since I have plenty of open space on my NAS, I’m dedicating a few terabytes to the effort.

So far I’ve backed up transgender protection and support resources, NOAA and climate change datasets, and some NIH datasets.

SciOp also provides a set of feeds that allows your backup to be automated to targeted and “endangered” datasets that we know are actively being removed, not just under threat, to ensure greater success in preservation.

If you have the tech setup for it I recommend checking it out.

I’ve long since embraced AI integrations into my AI, and I’ve experimented with several of the new app builders. I’m going to avoid commenting on the “vibe” coding meme that’s going on, I will say I’ve enjoyed some of the efficiencies gained. A few months ago per the suggestion of several peers I swapped from VS Code w/ Copilot to Cursor. It took some adjustment of workflow to incorporate Cursor into a viable tool worthy of its subscription cost, but once certain hurdles were overcome I found it certainly accelerated me. The release of Claude 3.7-sonnet came with a promise of drastically increased performance in code, which I found to be… not true. In fact, I noticed a significant decrease in performance…

…until I finally upgraded Cursor to its current version. The new agentic approach powered by Claude 3.7-sonnet has been significantly better, outperforming my expectations by far. It is far better with large contexts, tying together multiple systems, and behaving in predictable and desirable manners.

I’ve also experimented with Bolt, v0, lovable, and Replit. While I’ve snagged a few front end pieces from each, I’m yet to fully embrace the full “one-shot” builders. I’m hesitant to necessarily say they’re not ready though, as my experiments with them have admittingly been small experiments, not a full fledged dive allowing my workflow to warp to the tool versus the alternative.

There seems there are even more AI tools I’m yet to experiment with, each with a workflow from what I’m used to. I hope to sometime in the near future dive further into these. To that end I will be attempting a pause on my Cursor subscription this month; at that point I will embrace Windsurf for a small while, see if I enjoy an increased velocity or novel capabilities. I am also very much looking forward to the Replit sponsored SDx hackathon later this month; I want to see what in my workflow approach I am doing wrong to fully enjoy the tool.

DeepSeek released another cool paper expanding on reinforcement learning for LLM alignment. Building off of their prior work (which I talk about here), they introduce two new methods.

The first - Rejective Fine-Tuning (RFT) - They have a pre-trained model produce N responses. Then the collected responses are combined in a prompt wherein the model is instructed to produce principles for evaluating the responses, critiques of each response, and a reward score for each based on the generated principles. The process utilizes a human judge to critique its critiques, teaching the model to produce these evaluations eventually without human feedback.

The goal of this is to move the reward function from a simple scalar value to a language derived scalar value. If the model is being trained on domains that don’t translate well to automatic correctness, this improves performance. This is why reasoning models are generally focused to this point on mathematics, coding, and similar domains - there’s easy and clear ground truth results to evaluate against. However, in more subjective domains (like ethics, open-ended questions, etc.), a scalar value is not easily derived without human intervention.

Once the model is proficient at generating these evaluations, the second method - Self-Principled Critique Tuning (SPCT) - is introduced. This method uses reinforcement learning to adaptively improve the model’s ability to generate critiques and principles without human intervention. A sort of feedback loop now begins, wherein the model generates its own evaluation criteria, critiques, and responses, assigns scores, and then receives rewards based on how well it ranks the responses against known human preferences. GRPO is still used for policy optimization, but the model is learning without direct human ratings or real time human raters.

Finally, they introduce Inference-Time Sampling and Voting to further enhance the robustness of the reward model during inference by sampling multiple outputs and aggregating them through a voting mechanism. Essentially - have a language model generate multiple responses, then have the trained reward model judge each response. Finally, choose the highest rated one at inference time to improve performance. It feels like a great improvement to self-consistency mechanisms.