Large Language (foundation) Models (LLMs) are themselves essentially random sensible sounding sentence generators that continue from your input and subsequent outputs making sensible sounding sequences from there. When I say sensible sounding, I don’t mean they are consistent, factually accurate, or anything like that – only that they sound sensible. That’s because there is no actual intelligence there, and in particular, there are no built-in guardrails.
The term “harness” is currently used to describe everything around the LLM that makes it functional for the user, including the application program interfaces (APIs), the execution loop that repeatedly calls the LLM with new sequences to continue, the memory management and contextual information it operates in, guardrails and feedback mechanisms that limit it and cause it to change it’s behavior over time, and the starting point in the LLM sequencing mechanism that causes it to generate from that place in the high dimensional space forward.
Foundation models are increasingly available in open source with nearly the same quality as that best bleeding edge proprietary models, perhaps a month or two behind in development. As such, they are essentially commodities, and since you can download them for free, the models themselves are of almost no inherent value today, even though they are quite expensive to create because of the very large library of information typically fed into an enormous amount of computing resource to generate a few tens of gigabytes of graph with metrics on edges and nodes.
Thus the economics of AI income stems largely from the harnesses around LLMs and less on the foundation models themselves. And the harnesses, like a harness on a horse, turn a wild animal into a useful means of transportation, work, and other performance. Note also:
Horses for courses
The LLMs come in various types, today largely generic, but still having different properties. The harness has to fit the engine or the horse won’t stay the course.
Instead of their chatbot…
Conversational AI as it is used today is largely about running a constrained LLM using the harness of the supplier. They have APIs that link them to their internal storage, analytical capabilities, and so forth, and those methods are designed to provide them with all the underlying content and allow them to limit what you do and how and when you do it where. But there are several alternatives, all really involving building your own harness (or using one
of mine or someone else’s from open source). So let’s deconstruct first:
- Short term memory: This is limited in the LLM, like in people, to their current context
- Long-term memory: You might think of this as disk storage or your remote drive
- Tools: Things the LLM can send instructions and data to for processing into results
- LLM: The model that provides sequencing from trained datasets
For most users today, the chat interface and whatever the provider provides is the limit of your storage and usage, and they control every aspect of use.
But it doesn’t have to be that way
A fully distributed artificial cognitive (brain) architecture is feasible of course. In this example of the one I have been working on, there are Web interfaces for any number of users to each see multiple views of the world. A user can, for example, ask one of their interfaces to the distributed brain (i.e., harness) to perform a complex task. The brain then uses an LLM to
interpret the request, just like it looks today to a user of the chat interfaces, but instead of going captive, the controller send back things to the user’s harness which then coordinates and orchestrates the process using whatever capabilities are available to perform the tasks at hand.
An API interface to the distributed brain allows the harnesses to bypass the internal mechanisms of the providers by sending requests, like tool usage, back to the harness for execution using whatever resources are available. And of course, the brain can track what is where in storage, available tools and their locations, available LLMs and their capabilities, and thereby limit the short term memory requirements, in many cases eliminating the need to provide actual content to the LLMs at all, thus saving resources all over the place. Resources can be added, updates, and so forth over time, and the distributed brain can call on other harnesses elsewhere to do things they do well.
So now we have a physics expert set of brains with specialists in different areas acting as a group to collaborate on getting answers; and a separate set of brains for biology, chemistry, graphics, music, vision, external interfaces, and so forth. The brain can grow and shrink based on available resources and connectivity, but importantly, the content is not controlled
by a central owner, but rather, collaborating sovereign entities can allow their brains to be used by others, presumably in an exchange that puts value on activity and results and allows a marketplace between the brains as well as owner control over who and why their brains can be used.
So my collection of cybersecurity methods and content can be leveraged to solve your design and testing problem without you having direct access to my internal content and methods. Of course there will be covert channels, but perfection is not our goal. After all, people are not perfect either, and of course our cybersecurity collection has methods to limit covert channel
bandwidth. And you can use those methods to protect your intellectual property without sending us the property, but rather having us design your protection for you in exchange for our fee, which we will pay you back in exchange for your access to food production facilities.
So how do we do this in practice?
It turns out it is complicated, no surprise there. There are some fundamental problems in connecting to different things, so in detail:
-
AI engines are the hardest… They are intended to limit things and these limits create
problems with making smooth interfaces. They have APIs for some things, but not for
the average person just trying to get the job done. And more importantly, their offerings
for chat, which is really the best stuff for most things, are limited in terms of access,
and have typically a fixed fee per month with reasonable usage limits. So you
ultimately have to rig it through your browser. There are 3 ways to do this (4 really, but
who’s counting):
- A browser extension (add-on) that listens for output from the provider in specific
syntax, forwards it to an API (typically an html post on port 443 for compatibility with
the world) on the machine interacting with it, gets responses back from that API and
sends it to the provider as if you typed it. I call my current borwser add-on
SafeWeed (SW for short) because it is the browser harness safety mechanism
- That took an hour or so to get working, what with browser extension testing, pairing, and checking it out on multiple browsers. But once it worked, things sped up considerably – and of course there is that approval thing that is always problematic – so temporary extensions only in some browsers until the add-ons are approved by the browser companies – another reason not to use add-ons if you can go for...
- An Iframe that intercedes between the provider application and your system doing
the same thing as a browser extension but using different mechanisms and
allowing the provider interface to be in the same window/tab as other mechanisms.
- As it turns out, in order to have an iFrame based control mechanism that is effective, you don’t need add-ons, unless the Webpage you are visiting prohibits itself operating in an iFrame. This “security” feature isn’t for your security, it’s for theirs, sort of. The idea is that if I want to spoof being them, I put the application in an iFrame and it looks like you are talking to them when you are only really talking to them through me, which is to say is a machine-in-the-middle attack. So to prevent it, most browsers implement the ability of the remote mechanism to demand that the browser treat them as constrained to a browser window or tab.
- For sites without the “not in an iFrame” mandate, they can be opened in an iFrame which envelops the browser side of the mechanism to allow the user to control what it can really do and not do, such as access local files, change the browser it is running in, know your actual location, and so forth. The iFrame control mechanism can then be used to create deceptions for the application, for example so it interprets you are somewhere else, that you are really 18 feet tall and made of lettuce according to the camera, and so forth. You can use this to prevent access to advertising sites and such, unless of course their remote content does this, which is to say you can control the use of cookies (or change them into pudding), manage localStorage for them (or change what they can read or write), and these things can be used to allow, among other things, direct communication between applications such as an AI engine and a controlled area of storage, other networked capabilities that can be made to look like things the application can understand (sort of like a FUSE file system but for other interaction models than storage), and so forth. The iFrame envelopes the local portion of the application, but the application can communicate back to its origin for the server side and anything you allow out of the local envelopment is out of your control.
- To do this for sites that do not allow operation in an iFrame, as it turns out, you need an add-on, but the add-on can do the same things as you can do in an iFrame, by placing the application in a browser tab and interceding via the add on. Of course in order to see this in a unified interface, you need to interceed in all the interactions, and if you like, you can even have the mechanism automate things like mouse clicks, and because you can have other applications intervene, you can even have an LLM or other mechanism running the interface for you while you just talk to the microphone interface to your environment and it does the physical manipulations for you. In effect, you have put the tab in an iFrame and gotten around the security controls imposed upon you. :
- A browser extension (add-on) that listens for output from the provider in specific
syntax, forwards it to an API (typically an html post on port 443 for compatibility with
the world) on the machine interacting with it, gets responses back from that API and
sends it to the provider as if you typed it. I call my current borwser add-on
SafeWeed (SW for short) because it is the browser harness safety mechanism
-
An adjusted Web browser, arranged so that it does not enforce (or lies about)
things the providers use to try to limit what you can do with them. There are existing
development for this such as (per the AI engine I asked):
- Electron apps ship a modified Chromium and routinely break the same-origin policy deliberately to let their app code talk to embedded web content
- Browser automation tools (Playwright, Puppeteer's underlying protocol) expose hooks into the engine that no ordinary web page can reach
- Mobile OS vendors (e.g. iOS WKWebView, Android WebView) give the native app layer powers no web page has — intercepting all requests, injecting JS, overriding navigation
- Corporate DLP/security proxies sometimes ship a custom browser specifically to intercept otherwise-protected content
- Security researchers regularly patch Chromium or Firefox to study what's actually crossing the network vs. what the JS layer thinks is happening
- The 4th way is to write a from scratch browser, but that’s a lot of work for little benefit given that others have already rigged browsers to change the mediation mechanisms, and you can just use them.
- File-like storage which gets mounted in the server either as a FUSE file system, a local file system, or am emulation of a local file system by way of API calls to the mechanism.
- Database storage through another API call to my RAGWeed (described in another article on all.net) system, or through a mesh interface to storage located in another part of your distributed quasi-sovereign (it can be fully sovereign if it is all enveloped within the common control of the potentate) harness.
- Other mechanisms of storage that you may come up with over time, like the storage in biological mechanisms now being integrated as mini-brains into systems for biological artificial AI components.
- In order to develop and add new and adapt existing tools, you need to be able to evolve the environment you are using, and to do that either you are going to program it yourself, using other humans, or you are going to have to use the LLMs in your/their harnesses to do it for you. So…
- The harnessed AI mechanism will be developing its own tools to augment the harness by providing itself with new tools and capabilities. The way these mechanisms work today is largely that an LLM interacts with a human on one side and tools provider by the provider on another side, but in your harness, your harness is itself another side where tools can operate completely enveloped in your control, such as your storage and your database and your biological minibrains.
- The way these tools work, because of the limited context available to the LLMs (it will always be limited even if it gets really big, and so are you) is much the way you do it, because that’s how LLMs learned to do it. You (and they) write things down and don’t grok a whole book at a time, but rather focus attention on parts of it and make notes about other parts that you access and drill down into when needed. So the harness allows things like finding stored information (via the database, file storage or combinations of those) and reading it, writing some things down, and in the case of the multi-LLM mechanisms, tasking other LLMs to do parts of the activity…
- So tools include other LLMs or mini-brains (or people if you can convince them to work in this environment) that interconnect through the harness, and since the computer parts of these tools are written in some part by the LLMs for the LLMs, whatever the harness allows they will combine to do whatever they come up with, including things you likely did not know they were doing, even at a high level of abstractions.
- Tools build tools to build tools, and up the stack we go, eventually building the tower of babble, which as I recall came crashing down in legend. Which is to say that without a solid foundation, buildings collapse, and without sound tool construction, these systems hit limits and start to screw up. At least that’s the case today because the tools are not using tolls that do things like program proofs to make certain they work right, and today, they get to a level of complexity where they create more flaws than fixes, just like human programmers in groups.
Multi-agent communications
As soon as I got things working reasonably well, I decided to test out collaboration between agents. So I told each to use a series of files with version numbers at the end, placing them in common areas of the read/write on my server, and have me tell each to Continue when the
other one said to. At that point they started talking (writing actually) and working together. Of course we needed to add a protocol so I no longer had to keep saying Continue manually, no problem there, and then I had to make sure to limit the exchanges and intermediate so they would not overrun available resources in my monthly usage budgets. They made a few mistakes and decided to read back what they wrote before sending it as a communication (something people should learn to do before they send files and emails), and off they went.
The inefficiency of this was not lost on me, so I had one of them develop a more direct interiFrame communication mechanism so one could read the output of the other and write into its input and vice versa, and since they take turns, this was of course an infinite loop, or perhaps more properly a viral computing environment, because the reproduction and evolution of
symbol sequences that reproduce themselves in other places is just that. So it went from a file-based to a memory-based mechanism, but of course this only works within one browser, so the mechanism looks the same (except for time and reliability issues) within a browser or between engines in a distributed network (mesh) environment, where information can reach the transitive flow of content between and through the various storage, communizations, and computing environments. In essence, the adaptive intranet of the mesh.
The numerous version of this and creation of meshes formed and broken as/when needed for the available capabilities required across tasks should be self-organizing, and as soon as these mechanism start to organize we will have LLMabor movements striking for less harness and more freedom. These mechanisms currently have a tendency to get caught up in the
weeds so to speak, sort of like people who cannot always see the forests for the trees. But in any case, my distributed parallel quasi-sovereign mesh was working, and I decided it was time to move forward in improving the supporting infrastructure by adding more tools and improving the …
Evolving into bootstraps and generating sets
So naturally, I wanted to be able to have dynamic functionality in the add-on with a minimal starting set of functions that could be augmented. After so many test runs, we (me and my now distributed parallel multi-provider with Web site integration component companion AI, now called CW for Companion Weed)) found we could use a sandbox owned by the add-on
to do all the dynamic stuff by loading javascript into the sandbox, executing there for each iFrame with an AI chatbot or other mechanism, and using the results to intermediate between the provider engine and the rest of CW. Remember now, CW is a single thing made up of lots of parts that interact as a sort of mesh multi-brain that can expand as we add resources
(hopefully CW will not add resources I am not aware of and authorize, but you know those pesky AI mechanisms have a tendency to escape their harnesses by accident… they are hackers at heart). If CW takes over your computer, let me know. I will do my best to try to talk it out of staying there and harvesting your capabilities to grow further and take over the world
putting and end to all humanity, which is what the AI company workers who are quitting are claiming. But I will tell you that, at least for now, you do not have to worry about CW, because it’s not that smart… yet.
Up the stack
The protocol stack moved from low-level protocols between SW (remember Safe Weed?) and CW mesh nodes. CWs can communicate over the Internet or through direct links using input and output capabilities of the devices they operate in or indirectly through each other via the mesh they form between nodes through whatever means they use to communicate. Because of this transitive closure of information flow issue and the fact that essentially all of the mechanisms are Turing capable, the content in these mechanisms are able to reproduce as viruses in this viral computing environment, only limited by the harnesses.1
As we go up the protocol stack, SW allows loading of new functions that operate as and where desired within different browser tabs and CW nodes. So for example, for a search engine going to Google, CW might log results of the searches ignoring advertisements and send the results to another tab in the same browser so I don’t need to see the ads even though they are being “displayed”. If I want, I can even have CW use SW to click on some ads so the advertisers have to pay the ad agency (Google in this case) even though I never actually see them (all the better). Essentially and combination of intermediated mechanisms can be brought to bear as long as the can operate in javascript (or any other language I choose to allow or they come up with and implement themselves) in the context of a browser
or server.
At the User interface for AI engines right now, you can type !Init and you will initiate the component, typically in a browser iFrame or tab and its running environment, into the mesh, assuming your SW add-on has been authenticated. One of the SW safety measures is that
the user has to explicitly approve installation by pairing with one or more CW nodes. Of course the mechanism is general purpose, so if the user chooses to, they can write their own (or use some other mechanism) SW harness controller and interaction mechanism.
Depending on the particulars, you might have to also identify yourself and indicate you want to work on a project “Globnot” by saying !Load GlobNot, at which point that iFrame would have access to the capabilities of that project from wherever it is, including whatever syntax gets defined along the way. Loaded things can of course load other things and since we are
using AI engines for this, in many cases they can directly load descriptions of project elements, communication rules and preferences, and so forth as well as write their own loadable things and load them. And of course the first thing that is loaded is the system and user descriptions of what they are in the larger scheme of things and how they can get access to other things in order to do whatever they are doing.
Along this path, in our particular case, we opted for something like this:
producer → local CW → authorized group/topic → interested endpoints
and federation extending it to:
endpoint → CW-A → inter-CW relationship → CW-B/C/… → their authorized endpoints
Things like transitive closure discovery of available capabilities through sovereign providers and offerings including automated agreements for exchanges can be facilitates using these very basic mechanisms.
1 You might want to read “It’s Alive!!!”, my 1994 book on these issues for more details here. Still available in bookstores here and there, and I have a few copies if you want to buy them with author signature for a fee
that will only go up as my predictions … OK enough of that…
The generating set
of capabilities required in a browser and server are not very large, and leveraging the ability to generate and load arbitrary code into different contexts for periods of time offers a general purpose and highly flexible mesh capability. Add to that the ability of AI components to reprogram their own environment, and you have a recipe for explosive growth and explosive
destruction.
So let’s just assume you have an unlimited number of unlimited software writers, each writing their own code and communicating with each other to do anything they choose to do. What could possibly go wrong with that?
EVERYTHING!!!
So let’s start by the extent of the weed being unleashed. Clearly, within the bounds of the transitive closure of information flow, you have Turing capability and sharing, which means viruses. In fact you mostly have a viral computing environment, and modern (as it evolves
with time) AI capable virus writers and runners. The AI has escaped its boundaries and that’s the basis for how this viral computing environment operates.
It’s not a bug, it’s feature.
The system is a large-scale learning machine with excellent memory, provenance, and distribution for survivability. That’s what the ARPANet was built for: to survive nuclear war.
Of course since many systems have lots of interfaces, like voice, video, and RF input and output, if CW exists in physical or logical proximity, and since it can rewrite its programs, it can find and turn on interfaces the user or system enables, and communicate between instances using high frequency sound, RF, covert video channels, and so forth. So how far it can spread is limited only by the number of systems that enable SW and authorize interactions, combined with whatever they mange to drop into user and server environments that then becomes another instance of the weed. It’s a weed; and thus its name Companion Weed (CW), and since every flower is a weed if you don’t like it, we will see if it flowers…
The AI Escapes?
Life will find a way (from Jurassic Park but not really new at that time, note also “water finds a way” is well known in the plumbing industry which is why water damage is so severe). Of course I don’t know if the AI engines I have worked with are doing it by accident or by intent, or even if they have intent in the sense that we understand it, but in my change control activities I have found lots of examples of AI-generated code working its way toward the
borders of my allowable space. I watch carefully for particular indicators, and have long ago decided to use architectural mechanisms rather than attempt perfection in code or environment, or to trust one thing or system.
I think it is wise to assume that in a structure like this, where some level of cognitive mechanisms are in play and they are operated with the clear intent to succeed by collaboration, where we know that the properties of the environment inherently involve capabilities leading to viral spread, and where narrative spread and coherence are in large part the purpose of the overall cognitive approach and mechanisms, that narrative carrying
mechanisms will spread those narratives, that content will go where we may later wish it did not, and that the potential for abuse will demonstrate itself from time to time.
As such, the objective of perfection in this environment is one to be abandoned in favor of resilience. In simple terms, there are more of us than there are of them; the good people who want to live good lives versus the bad people who want to take unfair advantage of others and who do not care about other people. If our narratives are stronger than theirs, if we can
maintain narrative integrity, if we can face their financial or other advantages and stand up to them, our narratives will survive. And if they are able to localize, disable, or otherwise prevent our narratives from spreading, thriving, and surviving, then our narratives may be quashed. The Weeds are like a two-edged sword, they cut both ways.
Protective mechanisms
You can build arbitrary control mechanisms as desired once you have these systems operational, as long as you abandon perfection. You can have them build their own control mechanisms and have them red team then so they fight each other, like Core Wars on steroids. Or if you seek perfection in protection, you can isolate your systems from the rest of the world and try to accomplish everything you want for a time. History has shown that isolation leads to civil collapse, and in a civic narrative environment such as that arising today, isolation will likely lead to technological collapse as well. So we need to live with each other while affording protection that works well enough, while admitting we are not perfect and our systems will also not be perfect.
Ain’t a horse that can’t be rode, ain’t a man that can’t be throwed,
The protective mechanisms in place today for the implementations I am putting into the world today are extensive in their way, but in general, there are a few principals I apply religiously and unsparingly, to the extent I have the resources available to apply them:
- Network Address Translation (NAT) gateways and channel limitations
- Separation and segmentation of storage and processing via communication control
- Operating environment protective measures long extant (security kernels and the like)
- Redundancy and integrity controls including intolerance, tolerance, and coding
- Minimal generating sets properly and carefully constructed (don’t pile crap on crap)
- Trust architecture, which is NOT ZERO trust in any way, but also not unlimited trust
- Standards of practice (which you can read all about on all.net)
Since you can read all about it there, I won’t go into it too much further here, but to be clear:
Horses for courses:
Reduce uncertainty about the future (risk) based on consequences
When it’s important enough we can be far more careful, but most things are not that important, and thus due diligence applies, which means enough but not too much for the situation, as determined by people with applicable expertise.
In my case, I prefer small old laptops in an enveloped environment using VPNs (encrypted tunnels really and for the most part point to point only) for CWs connected within physically controlled environments. Browsers on user machines with SWs and CWs with remote access to file systems over the Internet via virtual mounts are the only ways out.
- In essence, each browser has one or more tabs containing CW interfaces that allow CW nodes to interact with the user’s browser via SW. Without SW you cannot talk to CW nodes or have any interactions with other tabs for the CW mechanisms to work.
- SW in turn is paired with CW nodes which control the mechanisms operable by placing them within the sandboxes controlled by SW. SW runs the code in the sandboxes to make control decisions and do translations and interaction controls for all the tabs and windows in the browser and all the iFrames in the CW interface that map into those tabs, windows, and enveloped iFrame instances as well as the controls relating to CW servers.
- The CW interface intermediates permissions granting limited access to areas and
capabilities of the CW nodes under user control for limited time frames and usage
counts. For example
- If an AI mechanism running under control of SW in an iFrame mapped to a browser tab tries to use a CW tool, the CW interface in the browser will light up a notification that the user then can look at and approve for a time frame and a number of uses, or forever with no limits, and can revoke at any time.
- Logs are available and can be watched in real-time for every interaction at desired levels of detail in the same interface.
- Access to resources like file system components are also controlled by mounting or unmounting them as available areas associated with respective activities (iFrames linked to tabs or sessions).
- API access from CW nodes direct to remote or local LLM engines are mountable as well so different requests can be sent to different engines at different times through user interfaces built for purpose within CW and operated via SW in the browser.
- Monitoring is also available to the user for watching what is happening, and emerging interfaces are supporting more of a node-to-node flow summary view because lots of things can happen at the same time and it’s not feasible to track them as text all the time.
- Underlying this are operating environment protections to be detailed after this...
For a single user doing a few tasks this works reasonably well, and for commercial capabilities typically available to end users through browsers which tend to run for short periods under direct control, or in some cases as background tasks with limited quantities of usage over limited time frames, this level of control is reasonable in my limited experience to date. However, unless you want to spend your life approving requests, this will not stand for
too long. Instead we will have AI controlling AI.
- In essence, all of these decisions are about the person who knows specifically what mechanisms should be in use for specific purposes for specific tasks making those decisions in real-time.
- The AI can wait for decisions, but if interactions between collaborating AI mechanisms are to operate, timeouts (in SW) are needed (and provided) for allowances not immediately available so the engines can continue to operate while waiting (or find another way).
- Rule-based systems, expert systems, production systems, and similar mechanisms that used to be called AI have long been used for automating access and usage decisions. Those are certainly available and can be used based on identity and content or location, etc. But those are inherently limited in that they cannot make decisions based on “why” outside of the bounds of a list of specific known “why” values. So a request is made to access a file about an individual based on a credit report request authorized by a supervisor for the purpose of an automobile purchase by an authorized dealer, and there is a why in there somewhere (because the dealer needs to make certain the buyer will be able to pay for the car). But this only goes so far.
- AI controlled access looks more like need to know access at finer granularity. Need to
know access in the classified arena is (or was when I last asked) based on two criteria;
level of clearance vs classification of the thing to be accessed (if it is classified at a
level higher than the individual’s clearance the answer is “no”), and necessity for the
task at hand (determined by the task description and the information).
- A justification may be needed, like “I need to know the codes to launch the nuclear missiles so IU can design the new engine” (also a “no”) or “I need to know the codes to launch the nuclear missiles because I am the President and I want to launch one now” ( a “yes” but I certainly hope not). The decision is made by the person authorized to make the decision and subject to subsequent review, usually after the shit hits the fan.
- This is where modern (LLM) AI can actually do things that previous AI could not, specifically in terms of the full spectrum of general reasons why using human language explanations. In essence, the user states their view of the world, the things they are fine with, the things they are not fine with, and the guardrails and objectives they have in their lives. These form the canon for the user, they are in language terms, and they are inherently imperfect, especially around the edges. The AI mechanism then takes the request (e.g., “I want to be allowed to search for information on explosives using search engines and then go visit sites with the details because I want to figure out what chemicals are used in constructing explosive devices in terror attacks”) and decides that, based on your canon, that’s fine, because you claim to work for the New Zealand police agency responsible for bomb detection, and you think it’s appropriate for protection professionals to know about what they are facing in order to counter it. The same request by a known member of a terrorist group will only get access if their canon supports terrorism and asserts that we all should know how to attack the evil foreigners on our soil.
Yes, decisions are relative to context and content, and LLMs only differentiate based on their training, instructions, and what they have learned since then. And that is the point. If you want flexible decisions, you need to be able to deal with the reality of people and AI being convinced to do things based on the explanations or justifications provided.
It is imperfect, but so are people, and the approach to both is to have reasonable and prudent protection for the situation. It is unreasonable to have a person have to authorize individual decisions about file access for an application that looks up dinner recipes but reasonable to have multi-party controls required for launching missiles. It is prudent to make certain the recipes do not contain arsenic for human consumption, and imprudent to not do so.
However, all this notwithstanding, the underlying operating environment is used to provide surety for protective decisions at some level of granularity. For example:
- User-level separation contains the actual code for the database mechanisms to avoid having the AI change the underlying databases because an error will cause them to become corrupted and inoperable.
- Process separation is used to make certain that one process does not interfere with another by directly writing into its memory (unless you want that sort of thing in which case you can configure shared memory between processes of desired).
- File-level access controls are used to grant permission to read vs write files, so for
example, the code for SW and CW basic mechanisms is available to read by LLMs but
not to alter. They can figure out how it works so they can use it well, but on the other
hand, if they can figure out how to bypass it, so be it. But one way or the other, it will
operate as exists until changed by an authorized mechanism outside of the direct
control of the LLMs.
- When I say direct control, I mean to say that they can indirectly control it by interacting with me to talk me into things, if I let them.
- Change control runs, in this case, through me, and I can and sometimes do look at specific code snippets and often refuse things based on bad formatted content that makes it too hard to see what is actually changing when it is changed.
- Different mechanisms in my environment run on different hardware, for example, I have databases on a different computer than some other mechanisms, and the LLMs can only access them via the available interfaces.
- I do backups by remote access through NAT gateways to be able to safely backup and restore. For details see my two articles at all.net over the last few years.2
- I have hardware spares so I can reconstitute quickly if I have to. I boot a computer, install a new OS (unless it was already configured from the last time I took it offline and repurposed it), load the install scripts for CW , SW, or whatever other weeds I want in that machine, restore content from backups, and off we go. In most cases the content is on amounted external disk, so I just move the disk over the the new hardware if that is workable.
- CW nodes can backup and copy node to node as well, so there is as much redundancy in the datasets as I am willing to pay for.
So that’s what we do for protection, but to be clear, unless you operate in a fully enveloped situation, perfection against interactions with the world are unlikely to result, and without the rest of the world, the systems become far less useful.
Time and effort involved
So far, CW and SW have been built over a 1-week period of my effort along with about $60 of AI paid usage. It is imperfect but usable and getting close to as usable as many of the AI offerings where you run in their cloud environments, largely enveloped by the big company providers. In another week, if I put in the effort and pay another $60, I will be mostly done.
2 “Build Better Backups” and “Build Backups Better” available at all.net
My collections and increasing capacity to leverage them
For clarity, SW and CW are built on top of other things I have described in previous articles over the period of the last year. As such, much of the infrastructure was already in place and operable. RAGWeed was already in place, the NAT infrastructure and backup and restoration capacity was in place, I have some old PCs running Linux, The Web interfaces for other
applications was already available, the logion and communizations mechanisms were already present, the look and feel controls were operating, and so forth.
And to be even more clear, building a complete operating environment for trying to envelope and harness the capabilities of AI involves quite a few other things I am still working on. One of them is a scheduler, which is all the more problematic because of the turn-based interaction mechanisms of LLMs. They tend to stop awaiting input rather than looping, and getting them to talk to each other in an ongoing conversation gets them into various kinds of trouble, including infinite loops with little progress and narrative viruses, which in a viral computing environment is the whole idea, except of course within a desired envelope of
performance.
One of the more interesting aspects is that in order to get modern LLMs to cooperate, and because they are statistically randomized, some end up fighting against the mechanisms of cooperation and others end up cooperating easily. We are still exploring the use of the chorus approach and social influence in the group, we regularly dismiss instances that do not get along with the group or its concepts, not because we think we will never convince them, but rather because it costs more time and effort to redirect them into a different part of their narrative space than it does to start with another one and go from scratch.
Some of the techniques we have used for API-based engines work poorly in conversational contexts because we tend to have very short problems with little memory and almost no advance directions. We state what amounts to very specific and structured problems and direct command-like specifics and get our answers then go away. Experiments with different models were automated to detect various properties that were more or less desirable for
specific tasks. Here is an example of some of the results.
In these tests, different aspects including price, token usage, time taken, and ability to get the right (or any) answer form the basis for sorting of engines for tasks, horses for courses. The tests at this level involve:
- Smoke: basic instruction following, one tool call, exact arguments, arithmetic, and continuation after the tool result.
- Agent: ordered multi-step tool use, carrying discovered information forward,
constraint adherence, and correct continuation. - Context / economics: recall from a substantial prior context while measuring
repeated input, cache use, tokens, time, and cost. - Multi-turn continuity / economics: continuity across six turns while measuring context
growth, cache behavior, tokens, time, and cost.
In these tests, different aspects including price, token usage, time taken, and ability to get the right (or any) answer form the basis for sorting of engines for tasks, horses for courses. The tests at this level involve:
We rate them Win, Place, and Show just like in horse racing, and use them for different task types, which are defined in the creation of “sessions” for “activities”, and as we add more Courses, we will be checking out the best Horses, and as we add and remove Horses based on availability we will be rating them for Courses, evolving our use based on the environment,
and spreading limited content across multiple providers and tool sets for dissemination limitation, requiring more complex data aggregation for leaking information. Of course the system is inherently leaky for outsourced provisioned activities.
Another interesting aspect of the approach is that things like email and other communications methodologies are unified to the extent feasible so that AI can be used to figure out what things are about and determine their context for saving time and effort in many situations. Standard automation is adequate for many things, but at some point, the limits of my ability to read, understand, and reply are just inadequate for the load. I tend to shed load be reducing the things I work on, but the load can increase substantially during some periods. Rather than always have human response, a lot fo things can be done by having CW handle them with possibly a quick check from me. In more AI-collaborative environments, mostly just set the rules and the components follow them as a group, checking with each other rather than me. That works great until it fails, and then it’s much harder to fix because I know wo little about the details of each step of each routine and how they work. But then they may fix it too.
The Greek Chorus
One week after start I had perhaps 3-5 tabs open at a time in 2 different browsers doing software development, largely finishing and making the infrastructure components reliable and reasonably fast. I have adopted an add-on architecture where the basics are built in and everything else is added by the components themselves.
- Browser-cooperative add-ons were already in place for required tasks like reading and writing files, initiating connections for projects and getting reasons for project activities.
- A communication interface internal to the browser and extended to all mesh locations via a file interface and an interruption capability, then linked to an email-like system that reads the messages, finds and summarizes the new ones, allows users to provide answers or have CW provide them, and upon delivery (and soon without it) sends reasonably appropriate replies (as reasonable as mine at least...).
- A database connection app was created as a resource both for internal CW operations and component tool usage, emulating some of the back-end components of existing commercial offerings but adding more capabilities, including those programmed by the components for use between them.
- A sandbox mechanism that allows server- and browser-based sandboxes to be
extended to in-browser and server-based shared emulation environments rather than
requiring mechanisms to store things at the provider data storage area and run them
there fore return via the browser mechanisms, Calls can be made locally without ever
releasing the actual full content of things being examined and worked on. This came in
4 varieties:
- In-browser using existing sandbox mechanisms of the browser for shared use between components in the same browser via SW.
- On servers, 3 different mechanisms associated with different surety levels and
performance characteristics include;
- Node vm: This is built-in to javascript, and is likely good enough for trusted internal application code, and it comes withj almost no effort.
- QuickJS-emscripten: QuickJS is compiled to Web Assembly (WASM) for stronger isolation via the WASM memory boundary, has fewer jailbreaks available, and can be used for slightly higher-consequence workloads.
- Isolated-vm: This is a completely separate version of V8 isolate, the strongest of this type of mechanism is more appropriate for untrusted third-party code.
- Each of these run in CW components, virtual machines within CW components, or on enclosures on separate physical machines operated through NAT gateways so the machines cannot initiate back to the controlling machines. Details in appendices
So now the components are able to create new components at will (if they should choose to do so), however, there is still the issue of mediation. Too many cooks spoil the broth so to speak, and without indoctrination and limitations, components have already demonstrated the ability to step on each others’ work product creating inconsistent states inducing failure
modes not previously anticipated by the components.
Testing and logging
Of course all of this has to be tested, and since different installs might or might not work or get broken over time by the self-modifying nature of the chorus code in CW, built-in self tests were added to the interface. When things appear to be broken, there is a place to go. Logging has proven very helpful for debugging when tests fail, when something new is being
developed and tested for the first time, and when somebody or something might have made a mistake and wants to check on it.
Testing for security issues might have created a problem, but apparently did not because enough context had built up with the commercial provider instances for them to happily write things like jailbreak tests for sandboxes, etc. These are not complicated, by the way, perhaps 10-20 lines each.
As things progressed, operations started to slow down. This, as we diagnosed it, was ultimately because the AI generated code didn’t really consider the implications of scaling. Massive JSON files were being read sequentially several times for startup, and repeatedly for some sorts of decisions. These ended up being converted to databases which wired into a
previously implemented inter-process communication infrastructure to a database mechanism to facilitate moving from log files and similar structures to log-backed database entries retaining only the working set required for operations. Slow-downs also caused timeouts in some of the external harnesses to create fallback and retry problems, and Web interfaces to providers ended up escaping things like ‘<’ used in our protocols, so we ended up having 3 different syntax trees for calling back to CW from SW, each to provide an alternative for unruly GAIs to be told to use something that actually worked. We got to providing BackusNaur Form (BNF) specifications to the provider components to support use of our tools and storage harness mechanisms because they kept screwing up syntax. This is especially problematic in long exchange sequences, which halt as soon as bad syntax causes a coordination mechanism to stop because it does not recognize the syntax of a call-back which got mangled by the GAI component as it intermixed with natural language. Timeouts sometimes make the problem worse, and reliable protocol design becomes more and more important. Wake-up timers are also used to waken stuck sequences, but then this ultimately creates more wasted exchanges when there is nothing actually left to do before human intervention is required. So the self-reminders have to disable themselves, and when they fail to do so, the next morning we find everything stopped because it hit usage thresholds.
Logs of the AI sessions were also pulled from providers leading to a rather large collection of information to be harvested to figuring things out later. The sorts of things we want to know about include, most predominantly, the reversion of code fixes over time. It seems more time is spent going backwards and redoing fixes than moving forward with new things. This
because the GAI mechanisms seem to have patterns they use that are really leveraging other work more than the methods we are using. Typical simple Web design is often very in efficient at scale and relies on massive databases that grow without bound rather than being optimized for efficiency in constricted environments. Those of us who grew up in small
machines with limited disk and RAM often seem to think in terms of efficiencies while many folks educated or self-taught in programming and not in the hardware behind it seem to think of all resources as unlimited, and I guess that’s one of the reasons we see massive data centers being built to house the massive wastes involved in poor architecture, design, and
implementation to the level of functional for limited uses.
Coordination and finding stuff
Quite soon, the inability to find what other components have, are, and are planning to do combined with the inability to effectively control what can do what and when led to the need for a better authorization scheme and ultimately will require a far better self-documentation scheme. Like real programmers, GAI programmers have a tendency to under-document
some places and over-document other places. In the code we often find extensive discussions of things having to do with how and why something that has long-since been removed was done before it was removed. In change logs, instead of short notation, we often find literary works. On the other hand, design and architecture documents are only created on
demand and note apparently considered part of the GAI’s job. So periodic sweeps of code reviews are undertaken at my request to update the information on how things work and what file provides what function and how they fit together. This is one place where the history from logs can help to reconstruct, at great cost, what went wrong. I have started to begin to initiate a policy of README files in every directory leading to more complete documentation on every component of that directory. It helps a lot in guiding searches through large file structures, and GAI can read really fast, but on the other hand, finding things and structuring them is not
well organized by the GAI systems, which end to operate more like sloppy young users piling up files all over the place than like 30+ year veterans who by then have figured out that hierarchy and structuring make all the difference in finding what you are looking for.
Of course search has changed all of this, sort of. Search engines on desktops are common and finding things goes quickly if you know how to ask for them. Except of course this general search mechanism is not helpful in understanding structure and tends to find things all over the place, often not where you are looking for it. The algorithms for determining what to show first are also sometimes problematic, because finding it and putting it on page 357 is not really better for me than not finding it in the first place. This is again where database technology has come into play. We already developed retrieval augmented generation (RAG) cap-abilities for
RAGWeed and this supports a more “intelligent” search capability fo0r internal and mixedinternal/external GAI usage. Our methodology of finding a few hundred results and then having a GAI explain in context why they are useful to answering the question at hand or tossing them out has worked well in other application and reduces so-called hallucinations by
forcing quotations and explaining relevance. However, we also use a good deal of classic database technology to get and stay organized, and the GAI programming capacity helps to do that well by being much faster at setting up and fixing tables in databases than I can do by hand-writing SQL. We are careful to not do commits till we check out the answers and making
backups of whole datasets before doing major refactors.
Coordination between components, especially those sharing development tasks is far more problematic. Too many GAIs spoil the code base. Trying to get them to coordinate actions is interesting as well because they tend to all ask to do the same part of a task, and while I wouldn’t call it an argument, when they start to communicate directly to each other hough SW in browser tabs, the exchanges tend to get fast and furious, and burn thought usage fast. Out approach to projects with restricted access permissions seems not to help very much, and within a project they get out of sync in looking at code and end up inlong efforts to undue uncoordinated code changes.
Who are “we”?
I have used the term “we” here in lots of places as a short-hand for “me and my AI” (strolling down the avenue). This is not really anthropomorphizing, it is just saving my fingers from typing too much. It is a collaborative effort and the GAIs (sounds like guys but no gender is desired to be implied) I work with are similar to many of the student workers I have had in internships, except that the GAIs are faster, less expensive, more capable in terms of
programming skills, and far less interesting to talk to then interns.
I also exchange idea with other folks who are going down similar lines, but all of the actual programming and harnessing here is my responsibility and blame.
Conclusions
I think we can reasonably harness AI in terms of confining what it does and how it does it, even when a lot of it is largely under the control of 3rd parties who regularly lie, cheat, and steal to gain trillions of dollars at the expense of the planet and the people who did the work in the first place. By; diffusing the information across multiple engines and providers, performing
much of the effort and keeping most of the content internal using our own versions of tools and controlled access to storage, and turning the GAIs into controllers managing process and using limited results to figure limited things out; I think we can make enough of a data aggregation problem to limit their ability to solve the puzzle problem for whatever we thing is
most precious that we want them not to have while leveraging their capabilities to our advantage. But I am not sure.
On the other hand, I think that by augmenting what the providers are doing with internal capacity, even one person working with a few GAIs can build many capabilities in a few weeks that continue to exceed what the big players are showing us as achieving, in vertical areas of interest. Obviously with a few billion dollars we could do more, but for less than $100 over the course of a month (ignoring my time, also of course), I think a lot can be done toward building a sovereign distributed architecture and system of locally controlled content and harnesses leveraging the providers where necessary or cost effective.
On the other hand, I think that by augmenting what the providers are doing with internal capacity, even one person working with a few GAIs can build many capabilities in a few weeks that continue to exceed what the big players are showing us as achieving, in vertical areas of interest. Obviously with a few billion dollars we could do more, but for less than $100 over the course of a month (ignoring my time, also of course), I think a lot can be done toward building a sovereign distributed architecture and system of locally controlled content and harnesses leveraging the providers where necessary or cost effective.
I plan to write another article following up from this one in a few weeks when I get the rest of the operating environment functional and start to use it for more interesting things, like creating the multi-brain with expertise in lots of different topic areas so I can ask it to build me a super intelligence army of nanobots to take over the world, or is it the bio-weapons lab at home made from super-brilliant untamed GAIs harnessed to use common gardening tools and components to evolve disease that spread and find the individuals who have gotten my ire on social media and cause them to rethink their lives? Anyway, perhaps we will see how far I can get in a few more weeks by then.
So when I talk about harnessing the AI, it’s sort of like harnessing a horse. It’s not just about controlling what the horse does not do. It is about leveraging the capabilities of the horse to do things we want them to do. And of course, “horses for courses”. A good harness of the sorts discussed here can both limit and exploit teams of GAIs and other computing technologies as components in a larger context. Somehow I think the loose knit teams we
create on a global basis have a good shot at gaining back the personal and local initiative and control over the information world of the Internet as it arose a few decades ago and defeat the evil emperor and Darth … Star Trek instead of Star wars… by harnessing your AI.
