This is a collection of some of my projects I’ve completed over the year that I’d like to share; some are projects I completed during my Master’s studies, others are fun projects I completed on my own time. If the project has an associated blog post, it’ll be listed within. If you’re interested to learn more, feel free to reach out to me - I’m always happy to talk shop.

• Robotics
  • AI
  • Reinforcement Learning
  • Deep Learning
  • Computer Vision
• LLM Work
• Reinforcement Learning
• Hardware Hacking Fun
• Pure Software
  • Golang
  • nodejs
• Startups

Robotics

AI

• LLM Powered Contextually Aware Task Planning - My Master’s capstone project, we built out a modified home environment with a robotic agent to act as an object retriever for a human user. The robot had a vision-powered understanding of world state, paired with a set of actionable APIs to build higher level behavior to control movement and grasping/interaction. LLM agents considered world state with human-language requests to generate multiple task plans, rate them, and execute the chosen one to accomplish the given task.

Reinforcement Learning

• PPO Reinforcement Learning Controlling a Robotic Arm - Blog post - Proximal Policy Optimization (PPO) and reinforcement learning controls a robotic arm trying to slide various shapes into specific sorting bins.

• Behavioral Cloning - final project for the Udacity Self Driving Car Nanodegree. A CNN observes simulated car’s manual driving and then manages to reproduce the behavior of steering to staying in-lane on a race course.

Computer Vision

• Semantic Segmentation for Self Driving Cars - Final project for the WPI RBE549 graduate computer vision course, our team automated the CARLA car simulator to generate a dataset and train a semantic segmentation network for driving applications.

• birdseye - birdseye is a work in progress. I’m trying to utilize 4 orthogonal cameras on a vehicle to recreate the overhead view looking down of the vehicle and its cityscape, as well as create a semantically segmented view.

• Vehicle Detection - assignment for the Udacity Self Driving Car Nanodegree. Within is a classical computer vision vehicle detector utilizing a histogram of gradients (HoG) to identify vehicles in a video.

• Lane Detection - assignment for the Udacity Self Driving Car Nanodegree. Utilizes thresholding and sobel edge edge detection to determine lane curvature.

• personable - a deprecated hack experiment from years ago. I wanted to make human identification and tracking easier for future projects, so I mashed together human pose estimation with face recognition and skeletal tracking. The idea was more natural contextually smart IoT projects (Keith is in the living room on the couch reading and it is night, so let’s turn on the lamp above his head).

Navigation

• Urban Delivery Robot w/ Local + Global Planner - In this project we created within PyGame a homebrewed simulation of an urban environment with static (trash can, traffic cones, etc) and dynamic (moving cars) obstacles. We then created a street-map aware global planner and a kinematic modeled local planner to asynchronously path the robot from a central grocery store to assigned delivery addresses.

• Motion Planning with Kinematic Constraints - Here I explore three kinematic models - a skid drive robot, an Ackermann drive robot, and an Ackermann drive robot with attached tow trailer (pictured). In each I utilize the kinematic models and constraints of the given robot model to utilize A* to plan across a continuous space.

• LQR Control of an Intercept and Return Drone Airspace Protection System - This group project for RBE502 (Robot Controls) had us creating a non linear controller for an interceptor drone system. The system would react to another drone entering its airspace. Utilizing a Linear Quadratic Regulator (LQR) controller, we feed the position of the target drone into the controller. Once the drone gets within a set proximity of the target, it is considered “captured” and is dragged, fighting, back to the launch site for inspection.

• RRT Path Search for Object Manipulation - Utilizing RRT path planning across a high dimensionality space (6 dimensions consisting of 3 translations and 3 rotations), remove the top shaft out of a tight, high collision environment of a transmission.

LLM Work

• coppermind - coppermind is an importable or stand alone backend for empowering LLM-powered AI agents written in Go. It injects personalities, memories, and knowledge into agents via generative embeddings and conversation self-summarization.

• Tree of Thoughts Engine - I wanted to experiment with using the Tree of Thoughts approach for higher level abstract reasoning. This is my attempt to build a generic engine for powering the approach and see what I could learn from it.

• State of the Art in LLMs + Robotics 2023 - This was a research article written and shared here to collect the current best research on applying LLMs and transformers to robotic applications. It acted as an excellent primer for diving into my LLM context planner project.

Reinforcement Learning

• Evolving Cars - To experiment with evolutionary algorithms, I created a self made a race car game in PyGame where I evolve neural networks to control cars that can complete a given track successfully.

• Deep Q Networks - a self-study exploration in deep reinforcement learning with Deep Q Networks - applied to a few OpenAI gyms.

• Evolutionary Algorithms to Solve Chicago Traffic - Using the SUMO traffic simulator, I used its Python API to automate the simulator and train an evolutionary algorithm trained AI to control traffic lights, aiming to reduce travel time throughout the city for simulated cars.

Hardware Hacking Fun

• hivenote - Work in progress - hivenote is a physical interaction media player, meant to be a mix of an in-home art installation and media device.

• Serial Synapse - A microcontroller that utilizes the serial-synapse defined messaging protocol can be connected to a loosely described node.js serial-synapse object. serial-synapse would then instantly create an interactive node API object to easily automate the hardware and programmatically control it. Asynchronous control, reacting to MCU feedback, and node->MCU hardware control is made effortless. This powered dozens of weird hardware builds throughout the years.

• Serial Synapse Socket - Takes the above and instantly exposes the hardware as a web socket API

• desk-bling - A Slackbot/Particle Photon app to bling out my desk with Slack-controllable RGB LED strips. Reacted to incoming messages, keyword triggers, and certain events for special lighting effects (lunchtime? disco mode!)

• mqtt-scheduler - A CLI and config file daemon process that allowed one to easily schedule MQTT broadcasts with human understandable timing (“turn the garden pump on at 7:00pm on Tuesdays”).

• garden-relay - An MQTT embedded controller with LCD and button control that automated lights and pumps for my wife’s indoor garden

• lifx-mqtt - a program that bridges the WiFI light bulbs made by LIFX to an MQTT server, exposing core functionality as subscribable and publishable topics

• doorbell - DIY raspberry pi video camera / MMS based doorbell project

• mlx90621-electron - Electron app w/ thermal sensor for low res DIY thermal camera

Pure Software

golang

• docker-harness - Dead simple golang docker harness for testing. Sometimes you just need to test the SQL query in the actual database. I have used this for many complicated tests and creating tooling for coordinating load tests.

• SafeStop - SafeStop is a service that coordinates safely shutting down and halting services and daemonized processes for larger applications when a SIGINT, SIGTERM, or equivalent is triggered.

nodejs

This stuff is old, but included because why not?

• CanThey - CanThey was an ACL based authorization module with optional express integration. This powered authorization on a lot of projects I’ve worked on.
• needle-swap - needleswap allowed unit and integration tests to override node’s require for injecting mocks into key components. This made testing significantly easier for code that integrated with third party services
• express-walker - an express service walker that would traverse directories and automatically import routes for an express server in a predictable manner. This allowed an easy standard to be followed by a team and avoided a lot of boilerplate code.
• controlled-merge - allowed merging of multiple javascript objects into a singular object, with an optional function parameter for custom rules on handling conflicts.
• node-packer - a node wrapper library for Packer for programmatically building images in node.
• pokemon-tracker - When Pokemon Go just launched and was a craze for all of a few weeks, an unofficial and bannable API was created to access and control the game. To help keep people focused at work, pokemon-tracker would make use of this API and create a slack bot with a fake account that would “walk” around the immediate vicinity of work and alert coworkers to the presence of any interesting pokemon.

Startups

On top of my work history I’ve co-founded a few startups in the past that have not worked out. These were great learning experiences. They are presented in reverse chronological order (most recent first).

• Wondervent - a crowdfunding site specifically for events. It allowed organizers test out interest in events prior to making significant investments. Works privately for friend-group events (purchasing movie tickets ahead of time, buying enough art kits for everyone, renting a lodge for a ski trip) and publicly for artists and entrepreneurs (hosting a conference, small concerts, art galleries).

• DateSonic - a dating website/app that held live events where people approved/disapproved profiles simultaneously. If a match occurred, both people would be immediately dropped into a voice chat to see if they were interested. Speed dating for the web era.

• LFG - Looking For Group was an Android app that created a social network of location-aware meetups on the fly. Users could glance at a map and see available hangouts and spontaneous events popping up near them - needing an extra for bar trivia, pickup sports, hiking partners, and more. Similarly allowed threaded discussions pinned to a set physical location and range. Eventually pivoted to entirely a location-based HTML5 and Android chat rooms called ZoneChat.