every day carry screenshot: picture of a couple of smart rings and a monocle device on top of an apple iPad Pro

Over on the Muse Discord, shared some thoughts about it what LLM/AI hardware could look like to my Muse-infused working style. Sharing that here for some future exploration.

LLM/AI Hardware and Muse - Discord

29 Mar

As I am close to receiving my Ai Pin (and sorta of the assumption a few here might also have the Rabbit R1 on order), def wondering how to integrate such hardware/LLM approaches with Muse. Been scribbling thoughts/ideas, but don’t really have an idea of how to mesh such bits… yet

4 Apr

I am very much a sketch-ink-in-Muse-first kind of user. It being purposed for the iPad’s modalities is what attracted me to the initial Ink and Switch research before Muse even came about. For various reasons, I find typed text and outlines conflicting to how I “think and connect the dots.” Spatial connections between data elements (logical, relational, informed, and remembered) compromise the bulk of how my Muse corpus exists. Before the connections and linked boards features, there was a lot of “hey, at the end of the week, let’s review boards. Copy boards into others where symmetry exists, and draw lots of lines between items on the same boards.”

Analogous-ly speaking, LLMs more or less function in the same way - between typed text. Generative models do have some sense of recognizing patterns, shapes, and form, but not yet efficiently in the “there’s a mix between characters and scribbles which denote the same thing and can be associated to different things.” This is not a Muse limitation, or a JSON/SVG one, it is simply inherent to however we might say “the stuff in the pipes” is designed to be fit. Any LLM attached to Muse will be fine with many of my boards with links, images, and typed text… but those illustrative scribbles - nah. In the future, maybe… but not right now.

When I interact with the world around me, I’m making notes of way too many things. Sometime, it bugs me that I cannot pull out the iPad and sketch in Muse what I’m connecting. I see my past boards, I see content across the two spaces I manage (one is a collaborative bit). If I could describe it, it’s like Muse is my own Minecraft world, and the shapes and structures I don’t expect any/many LLMs to be able to deal with, let alone offer insight across the (admittedly wide and deep nature of) what I process daily.

And yet, Muse offered me hope that the “spatial content type” could be cracked for such bits. The .canvas element is something of proof that there’s others who get that machines reading data isn’t the only possibility here. Yes, behind .canvas is our trusty outline that more or less seems indicative of the format that’s most persistent (Dave Winer kinda keeps proving this). Yet if it gets ink… and we stop scribing in the constraints of an outline, then there’s possibility of more.

An LLM (or several) which can take the coordinate mapping structures, and aid its user in composing, remixing, and revealing aspects inside and outside of that coordinate model seems most appropriate to the shape of “tools for thought.” It is in this framing that I am wondering about Ai Pin, Rabbit R1, Tab, etc… not “can it tell me about what I know.” But, can it aide in sharpening where I’ve been instinctively going? And if so… it should affirm scribble, not demote it.

This is how I am thinking of “what do LLMs mean to Muse” and “how best to think about what one alongside Muse/TLDraw/Miro/etc aptly offer those who already know how to think… they just wanna focus just a bit more with a tool.

I realize that what I just typed might be rambling-ish for some, and “wait, what’s his context again” for others… and I’m ok with some of that ambiguity. Some of the nature of “processing our thoughts” just comes across in this way.

A bit more context for how Muse and hardware works on my daily - this is my ‘everyday carry” or EDC

I don’t carry the iPhone with me. Watch is sufficient. Monocle is currently more of a learning-tech, than a productivity-helper (most non-local LLMs can’t touch what I deal with in the identity, credential, access management space).

I spend a significant amount of time straddling collaborative and deep work paradigms (latter defined here). Muse is dang near perfect for the latter… it is rightly deficient for the former. The threads between these is where I see LLM/AI hardware best suited. At least for my working and creative styling.