The 10-Year Platform: Shutting Down KRE

Summary: The original pico engine, KRE, is no more. But the ideas and capabilities of the platform live on in the new pico engine.

A few years ago, I announced on this blog that Kynetx was done. But the platform we'd created, the Kynetx Rules Engine, or KRE, lived on. Today I am annoucing that KRE is dead too. We shut it down last week.

Despite the demise of Kynetx, the platform continued to be open and available. Fuse was still running on it and my students were using it for class and research. But Fuse stopped working for good last spring when the MVNO we were using to process cellular data from the car devices shut down. And the new pico engine is working so well that we use it for everything now.

KRE was started in 2007 and envisioned as a cloud-based programming platform for events. While we Continue reading "The 10-Year Platform: Shutting Down KRE"

The 10-Year Platform: Shutting Down KRE

Summary: The original pico engine, KRE, is no more. But the ideas and capabilities of the platform live on in the new pico engine.

A few years ago, I announced on this blog that Kynetx was done. But the platform we'd created, the Kynetx Rules Engine, or KRE, lived on. Today I am annoucing that KRE is dead too. We shut it down last week.

Despite the demise of Kynetx, the platform continued to be open and available. Fuse was still running on it and my students were using it for class and research. But Fuse stopped working for good last spring when the MVNO we were using to process cellular data from the car devices shut down. And the new pico engine is working so well that we use it for everything now.

KRE was started in 2007 and envisioned as a cloud-based programming platform for events. While we Continue reading "The 10-Year Platform: Shutting Down KRE"

Fuse Version 1 Candidate Release for 20140815

Summary: This blog post gives some highlights from the recent release candidate for Fuse version 1. Colour: The spice of life On Friday I released new code that serves as the latest candidate for Fuse version 1. Here are some of the things that were included:
  • Maintenance API — the maintenance API was completed. The maintenance API contains queries and services for managing reminders, alerts, and maintenance history.
  • Fix Refresh Token Refreshing — Refresh token refreshing is more robust now and integrated with the "fleet as repository for OAuth tokens" model of linking Fuse to Carvoyant.
  • Refactor Weekly Report — The weekly report now uses a separate function to get the fleet summary. This new function will also be used for generating exportable CSV files for taxes and other use cases.
  • Name changes — some query and event names were changed to be more consistent.
There have also been changes to
  • Add status — the provisioning app now shows the status of the link between vehicles and Carvoyant as well as some basic data about the vehicle.
  • Version choice — there are both product and development version of the service. Joinfuse now recognizes which the user is attached to and uses the correct service.
In addition, the javaScript SDK and it's documentation have been updated to match changes to the underlying service. Tags:

What Happens to the Data

Summary: Metromile offers per-mile car insurance based on an OBD II device that plugs into the car and reports data about how the vehicle is used to the insurance company. This raises questions about where the data goes, how it's used, and who owns it. Even more important, it's a business model that promotes the creation of data silos. Silos de Trigueros Nathan Schor pointed me at an article about Metromile that appeared in TechCrunch recently. Metromile is a per-mile insurance company that uses a OBD II device that you plug in your car. It tracks your vehicle stats, similar to Fuse, Automatic, and other connected car services. The kicker is that it's free because Metromile is making money by selling per-mile insurance. The more users they have using their device the bigger their potential market for selling insurance. That is made evident by the fact that you can only get the free device if you live in a state where they offer insurance (currently CA, OR, and IL). Otherwise, get in line (until they come to your state, presumably). I don't know how Metromile is implemented, but I wonder what happens to the data. I'm pretty sure they're using a cellular device (rather than Bluetooth) so that the data is always transmitted to their system even if your phone's not in the car or connected. Does all the data about every trip go to the insurance company? Or some aggregation? What's the algorithm? These questions are relevant because it's unclear who ultimately owns this data. Users aren't paying for the device or the data, just the insurance. As I wrote in The CompuServe of Things, business models that connect devices to non-substituable services threaten to leave users with little control over the things they own and use. I believe users ought to be customers who own the data and control where and how it's used. That doesn't mean they can't choose to share it with the insurance company, but they ought to know what's being shared and even be able to substitute one insurance company for another. If every connected car device is associated with a different insuarance company, I can't switch without having to give up access to all the data that's been collected about my car and driving. Data silos with murky policies about data ownership are all too common. Unfortunately, they lead to a future I don't want to live in. And if you think about it, I'll bet you won't want to live there either. Tags:

A Microservice for Healing Subscriptions

Summary: Here is a quick little microservice I wrote as a KRL rule to ensure that the vehicle has all the subscriptions it needs and fix them if need be. The simplicity illstrates why a microservices approach holds such promise for more loosely coupled systems. Such Great Lows Last week I wrote about how Fuse uses a microservices architecture and the benefits that such an architecture provides. This morning I was faced with a problem that the microservices approach solved handily, so I thought I'd write it up as an example. Fuse uses webhooks (we think of them as event channels) to receive notifications that a vehicle has started or stopped, has a low battery, etc. If these subscriptions aren't set up for a vehicle nothing works. Most noticably trips aren't recorded. Alex, who's working on the Fuse app, wasn't seeing any trips from his truck. Sure enough, when I checked, it had no Fuse event subscriptions. I could have just reinitialized Alex's truck, but I figured if it happened to Alex then it's likely to happen to other people, so I created a simple microservice (i.e. KRL rule) that checks the number of subscriptions and if it's lower than the expected number, reinitializes them.
rule check_subscriptions {
  select when fuse subscription_check
  pre {
    vid = carvoyant:vehicle_id(); 
    my_subs = carvoyant:getSubscription(vid);
    should_have = required_subscription_list.length();
  if(my_subs.length() < should_have) then
    send_directive("not enough subscriptions") with
      my_subscriptions = my_subs and
      should_have = should_have
  fired {
    log ">>>> vehicle #{vid} needs subscription check";
    raise fuse event need_initial_carvoyant_subscriptions;
  } else {
    log ">>>> vehicle #{vid} has plenty of subscriptions";
The rule uses a pre-existing function to get the current subscriptions and compares the length of that result to the number we should have. If there aren't enough the rule fires and raises the fuse:need_initial_carvoyant_subscriptions. This is a really simple rule. One reason is because it makes use of other services that already exist. There's already a working function for getting the current subscriptions for a vehicle. There's already another service (or rule), called initialize subscriptions, that sets up the subscriptions when they're needed. Another reason the rule is simple is because it doesn't have to figure out which subscriptions are missing and limit itself to only initializing those. If any are missing, it asks for them all. That's because the initialize subscriptions rule is idempotent. You can run it as many times as you like without messing anything up. Of course, I'd rather not put that load on the system if I don't have to, so the check_subscriptions rule checks if something needs to be done before it raises the event. The primary point is that the microservice architecture is loosely coupled and so setting up a service like this is easy. There's very little code, it makes use of other services, and it's unlikely to break anything. I wired it into the system by raising the event it looks for, fuse:subscription_check when the vehicle profile is updated. That seems like a good compromise between over checking and user control. Tags:

Fuse as a Microservice Architecture

Summary: Microservices provide a powerful pattern for programming picos with KRL. This post describes microservices and shows how we can view rules within the Fuse system as microservices for a vehicle. We give a detailed, technical example of microservice interaction within Fuse and of a specific rule. I recently ran across the idea of microservices. I don't know why it took me so long to run across it, since they've been discussed for a few years and there are many article written about them (see the end of this post for a link list). They have, like anything new, many different definitions, but I've settled on a few characteristics that I think differentiate a microservice from other styles of architecture:
  1. Organized around a single business capability
  2. Small, generally less than 1000 lines and code and usually much smaller
  3. Event-based and asynchronous
  4. Run in their own process
  5. Independently deployable
  6. Decentralized data storage
What struck me as I reviewed various material on microservices is how much the philosophy and architectural style match what I've been preaching around persistent compute objects (picos). As I worked through the ideas, I came to the conclusion that since you can view each pico as an event bus, we can view each rule installed in the pico as a microservice. With this lens, Fuse can be seen as an extensible, microservice architecture for connected cars.


Fuse is a connected-car platform. I've written extensively on Fuse in this blog. For the purposes of this post, it's important to understand the following:
  • Fuse uses Carvoyant to manage devices and provide an API from which we use to get vehicle data. The Carvoyant API is a well-designed RESTful API that uses OAuth for user authorization.
  • Picos use a set of pre-built services that collectively I call CloudOS to manage things like creating and destroying picos, pico-to-pico subscriptions, storing profiles, etc.
  • Rules are collected into rulesets that can share function definitions.
  • Each ruleset has a separate persistent key-value store from every other ruleset.
  • Rules are programmed in a language called KRL.
  • When we create a pico for a vehicle, the pico is automatically endowed with an event bus that connects all rules installed in the pico.
  • CloudOS provides a ruleset that functions as a persistent data store for the entire pico called the PDS. The PDS provides a standard profile for each pico. Fuse stores all of the vehicle's configuration and identity information in the pico profile.
  • Other vehicle data is stored by the individual service. For example, the trip service stores information about trips, the fuel service stores information about fuel purchase, and so on.
  • Rules can use HTTP to interface with other Web-based APIs.
Not only do we create a pico for each vehicle, but we also create one for each owner, and one per owner to represent the fleet. They are organized as shown below. fuse microservice overall This organization provides essential communication pathways and establishes a structural representation of ownership and control.

Example Interactions

Microservices should be designed to be tolerant of failure. Let me walk through one place that shows up in Fuse. As mentioned above, Carvoyant provides an API that allows Fuse to interact with the OBD II device and it's data. To do that, we have to mirror the vehicle, to some extent, in the Carvoyant API. Whenever we create a vehicle in Fuse, we have to create one in Carvoyant. But there's a small problem: before the vehicle can be created at Carvoyant, the owner needs a Carvoyant account and that account needs to be linked (via OAuth) to their Fuse account. One way to solve this problem would be to ensure that a vehicle can't be added to Fuse unless a Carvoyant account is active. Another way is to be more tolerant and let users add vehicles any time, even before they've created their Carvoyant account, and sync the two accounts pro actively. This has the added benefit of working the other direction as well. If someone happens to already have a Carvoyant account, they can add their vehicle to Fuse and the two accounts will sync up. The following diagram shows a few of the microservices (rules) in the vehicle pico that help perform this task.

fuse microservice

(click to enlarge) As I mentioned above, the vehicle configuration is stored in the PDS profile. Whenever the profile is updated, the PDS raises a pds:profile_updated event. So anytime the user or a process changes any data in the vehicle profile, this event will be raised.

Any number of rules might be listening to that event. One rule that listens for that event is carvoyant initialize vehicle (lower left of the preceding diagram). Carvoyant initialize vehicle is a fairly complicated service that ensures that any vehicles in Fuse and Carvoyant are represented, if possible, in the other service. We'll examine it in more detail below. When it's done, carvoyant initialize vehicle raises the fuse:vehicle_uninitialized if the vehicle is changed. Also, when the carvoyant initialize vehicle rule contacts Carvoyant to either update or create a vehicle, it does so with an http:post(). Picos are designed to automatically raise an event with the results of the post when it completes. The initialization OK rule (lower center) listens for that event (only with an HTTP 200 status) and store various information about the vehicle at Carvoyant. This is what establishes a link between the vehicle pico and the vehicle at Carvoyant. Independently, the initialize vehicle rule (upper left in the diagram) is listening for the fuse:vehicle_uninitialized event. Initialize vehicle primarily functions to raise other events and thus sets off a collection of activities that are necessary to bring an uninitialized vehicle into the system and make it function. Since it's now been connected to Carvoyant, the vehicle needs to
  1. subscribe to events from the Carvoyant system including events that communicate the ignition status, fuel level, battery level, and any diagnostic codes,
  2. retrieve the current status of various vehicle systems, and
  3. retrieve specific data about the vehicle's geoposition.
The initialize vehicle rule doesn't ask for all these on it's own, it merely signals that they're needed by raising events. Other services respond by carrying out one simple task. For example, the update vehicle data rule in the upper right makes calls to the Carvoyant API to gather data from the vehicle, communicates that data to the fleet pico (i.e., raises an external event from the standpoint of the vehicle), and raises various events including the pds:new_data_available event. The PDS add item rule (lower right corner) is listening for the pds:new_data_available event. It takes the data in the event and stores it in the vehicle PDS. We chose to store the vehicle data in the PDS instead of the update vehicle data rule because it's widely used and is more widely available in the PDS. Of course, these are only a few of the rules that make up the entire system. There are at least 60 in the vehicle pico and many more in the fleet. Most of these rules are quite simple, but still perform a vital task. One important note: each vehicle is represented by a unique pico and this has it's own copy of each of these rules, running against the unique data for that vehicle. The rule doesn't know or care about any vehicles except the one that it's installed in. As such, it is a dedicated service for that specific vehicle. In a system with 10,000 vehicles, there will be 10,000 independent carvoyant initialize vehicle microservices running, listening for pds:profile_update events from the PDS in the pico in which it runs.

A Detailed Look at A Service Rule

The carvoyant initialize vehicle rule performs initialization, but it has to be flexible. There are three basic scenarios:
  1. The Fuse vehicle pico has a representation in Carvoyant and thus any changes that have been made in Fuse merely need to be reflected back to Carvoyant.
  2. The Fuse vehicle pico has no Carvoyant representation but there is a Carvoyant vehicle that matches the Fuse vehicle in specific attributes (e.g. the vehicle identification number) and thus should be linked.
  3. The Fuse vehicle pico has no Carvoyant representation and nothing matches, so we should create a vehicle at Carvoyant to represent it.
Here's the code for the rule
rule carvoyant_init_vehicle {

  select when carvoyant init_vehicle
       or pds profile_updated

   pre {
    cv_vehicles = carvoyantVehicleData();
    profile = pds:get_all_me();
    vehicle_match = cv_vehicles
                   v{"vin"} eq profile{"vin"}  

    existing_vid = ent:vehicle_data{"vehicleId"} || profile{"deviceId"};

    // true if vehicle exists in Carvoyant with same vin and not yet linked
    should_link = not vehicle_match.isnull() 
           && ent:vehicle_data{"vehicleId"}.isnull();

    vid = should_link                            => vehicle_match{"vehicleId"} 
    | ent:vehicle_data{"vehicleId"}.isnull() => "" 
    |                                           existing_vid;

    config_data = get_config(vid); 
    params = {
    "name": event:attr("name") || profile{"myProfileName"} || "Unknown Vehicle",
    "deviceId": event:attr("deviceId") || profile{"deviceId"} || "unknown",
    "label": event:attr("label") || profile{"myProfileName"} || "My Vehicle",
    "vin": event:attr("vin") || profile{"vin"} || "unknown",
    "mileage": event:attr("mileage") || profile{"mileage"} || "10"

  if( params{"deviceId"} neq "unknown"
   && params{"vin"} neq "unknown"
    ) then
    with ar_label = "vehicle_init";

  fired { 
    raise fuse event vehicle_uninitialized 
      if should_link || event:name() eq "init_vehicle";
    log(">> initializing Carvoyant account with device ID = " + params{"deviceId"});
  } else {
    log(">> Carvoyant account initializaiton failed; missing device ID");
This code is fairly dense, but straightforward. There are four primary pieces to pay attention to:
  1. The select statement at the beginning declaratively describes the events that this rule listens for: carvoyant:init_vehicle or pds:profile_updated
  2. The pre block (or prelude) retrieves the current list of vehicles Carvoyant knows about and filters them to see if any of them match the current vehicle. Then it calculates the vehicle ID (vid) based on that information. The calculated value of vid corresponds to the three scenarios listed above. The code also computes the parameters to post to Carvoyant.
  3. The action block is guarded by a condition that ensures that the deviceId and vin aren't empty. Assuming the guard condition is met, the rule POSTs to carvoyant (the action carvoyant_post has all the logic for making the POST with appropriate OAuth access tokens).
  4. Finally, if the rule fired (i.e. it was selected and it's guard condition was true) the postlude raises the fuse:vehicle_uninitialized event. The raise statement is further guarded to ensure that we don't signal that vehicle is uninitialized event when updating an existing vehicle.


There are a few points I'd like to make about the preceding discussion and example: Service contracts must be explicit and honored. Because we lean heavily on independent services, knowing which services raise which events and what those events mean is critical. There have been times that I've built services that didn't quite connect because I had two events that meant the same thing and each service was looking for a different event. Self-healing allows misses. Picos don't guarantee evnt delivery. So consider what happens if, for example, the initialization OK rule misses the http:post event and the Carvoyant vehicle information fails to get stored? The carvoyant initialize vehicle rule is tolerant of failure in that the system will merely find the vehicle again later and link it then. Idempotency is important to loose coupling. There are multiple places where it's easier to just raise an event and rely on the fact that running a service again does no harm. The logic gets very complicated and tightly coupled when one service has to determine if it should be raising an event because another service isn't designed idempotently. Scaling by adding services. The carvoyant initialize vehicle rule ensures that Fuse vehicles are linked to their representation at Carvoyant, but it doesn't sync historical data, such as trips. What if something gets disconnected by a bad OAuth token? Because of the microservice architecture, we can add other rules later that look at historic data in the Carvoyant system and sync the Fuse vehicle with that data. These new services can be added as new needs arise without significant changes to existing services Picos and CloudOS provide infrastructure for creating microservices in the Internet of Things. Picos provide a closure around the services and data for a given entity. In the case of Fuse, a vehicle. By representing an entity and allowing multiple copies to be created for like entities, picos provide persistenly available data and services things whether or not they're online. Asynchronous, one-way communication is the rule. Note that two-way communication is limited in the preceding example. Rules are not waiting for returned results from other rules. Neither is it right to think of rules "calling" each other. While the KRL system does make some guarantees about execution order, much of the execution proceeds as dictated by the event-flow, which can be conditional and change with circumstances. Picos cleanly separate data. Picos, representing a specific entity, and microservices, representing a specific business capability within the pico, provide real benefit to the Internet of Things. For example, if you sell you car, you can transfer the vehicle pico to the new owner, after deleting the trip service, and its associated data, leaving untouched the maintenance records, which are stored in the maintenance service. A microservice architecture means picos are extensible. Microservices provide an easily extensible API, feature set, and data model for the pico. By changing the mix of services in a given pico, the API, functionality, and data model of that pico change commensurately. There is nothing that requires that every vehicle pico be the same. They can be customized by make, model, owner, or any other characteristic.


Viewing rules as microservics within a pico have given me a powerful way to view pico programming. As I've folded this way of thinking into my programming, I've found that many of the ideas and patterns being developed for microservices can equally be applied to KRL programming within a pico.

Further Reading

For more background on microservices, here are some resources: Tags:

Substitutability is an Indispensable Property for the Internet of Things

Summary: Substitutability is a key feature of decentralized systems that give people real choice and build trust. The coming Internet of Things will be limited in what it can provide without it.

In a recent update on the Fuse Kickstarter project, I posted a picture showing a possible UI flow for linking the Fuse app to the back end service that provides Fuse with all it's smarts. If you've been following along, you'll know that I propose that the service act as a personal cloud and that the app be built using the Personal Cloud Application Architecture.

Here's the initial flow I proposed:


The interaction proceeds left to right. Screen (2) follows the user clicking "Let's Go!" on screen (1). Screen (3) follows the sign in on (2). Screen (4) (or something like it) follows the user clicking on "Allow" on (3) if there are vehicles in the personal cloud. If not, the flow would lead to an "Add car" flow. I'm not showing the "More info" page or the account creation page. The "More info" page would talk more about the Fuse personal cloud and our policies around personal data.

Screen (3) is a "consent screen" asking the user for consent to allow the app to link to their personal cloud. Adrian Gropper wrote back to me:

Hi Phil,

In general, I have a problem with the feel of consent screens like #3 because they don't offer a meaningful choice. They feel like a standard click-through agreement and, as such, do injustice to the personal cloud concept we're promoting.

I don't have a clear solution for this problem in the Fuse application. In general, I'd hope to see at least 3 choices:

  • full functionality if I trust the service provider
  • partial functionality if I share nothing with the service provider yet and may choose one later
  • no functionality if I want to pause and investigate service provider options

In other words, meaningful choice means that the service providers are substitutable and compete for my trust and that this is made clear to me at the point of giving consent.


Adrian's last graf has a vital point that anyone interested in the larger Internet of Things must pay attention to: "meaningful choice means that the service providers are substitutable and compete for my trust".

Substitutablity is a key feature of decentralized systems. I can change my ISP without loss of functionality. I can change Web hosting providers. I can change email providers. I can even provide all of those things myself if I like and have it all work the same. Many people have used the same email address for 20 years now, all the while using a variety of email providers.

Unfortunately, the world that Web 2.0 gave us is decidedly unsubstitutable. I can use Twitter, Facebook, or both, but they're not substitutable in the same way as email. I can't substitute my iPhone for a Nexus without finding new apps and learning new things. I can't substitute one API for another because there are no standards. If the future is social, mobile, and cloud, we've given up a lot to get there. I talk about this at length in Why Personal Clouds Needs an OS.

I fully intend for people to be able to link their Fuse app to an alternate system (that speaks the Fuse API). Until the time that such alternate systems exist, however, Adrian's comment is spot on. Right now, the consent screen will largely be a "agree or else" proposition and that doesn't feel right. But without the backend system there's no graceful fallback to partial functionality. Kind of a Catch-22. If you've got suggestions, I'm open to them.

We need personal clouds. They need to be interoperable. They need to be substitutable. Our personal data must be portable. As Adrian says above, that's the way we build a system that we can trust. Choice in email allows me to find an email provider I trust. Choice in ISPs (for those of us not in a Comcast wasteland) let's me find one I trust.

What happens if you stop trusting Facebook? Fortunately, for most of us, Facebook isn't something we have to have to do our jobs, so we could just stop using it. But what about when the Internet of Things is fully in play, you've got a thousand connected products in your life and you stop trusting the company that provides their cloud backend? Unless that infrastructure is substitutable, you're stuck. Privacy, functionality, and affordability are all at risk without substitutability.


Vehicles That Get Better Over Time

Summary: As software becomes a bigger and bigger part of everything, the rules have changed for manufacturers of durable goods. They will have to adapt to customers who expect things to be updated and fixed automatically. For car manufacturers, this means that if I don't feel like my car has gotten better in some ways, because the software in it has gotten better, you're going to lose me as a customer.

The 10k

Smartphones have made us used to the idea that things can get better over time. If you're using a less-than-new iPhone or Android handset, chances are it's better now than it was when you bought it because you've upgraded the operating system. If you throw apps in, it gets better every day cause some app or another gets a new version. When I say "gets better", I mean bugs get fixed, performance (often) improves, new features get added. and so on. Sure, some "updates" aren't the improvements we were hoping for, but the overall trend is towards "better."

Contrast that with your car. From the moment you buy it, it gets worse. You never take it to the dealer and have them say "oh there's a better engine out now, so we upgraded your car while it was here." Hardware doesn't work like that. Hardware upgrades cost money. Software upgrades tend toward free.

But a greater percentage of every manufactured good, cars included, is software. That trend will accelerate over time. What's interesting is that car manufacturers largely treat the software in the car the same way they treat hardware. They'll update it if there's a safety issue, but otherwise, it never changes.

I have a Ford F-150 with Microsoft Sync, so I was interested in this article saying that Ford is ditching Sync in favor of the QNX platform. One of the things that I loved about Sync was that I was able to upgrade the firmware in my truck a few times. It was clunky, but it could be done. My truck got a little better.

The problem is that it was only a few times, early on, and there were never any cool new features, just some bug fixes. Car manufacturers just don't think of their product that way. I'm not surprised that people have been largely unsatisfied with Sync, but I suspect the problem is more the product culture at Ford than inherent problems with Sync. If Ford and other manufacturers don't start thinking of their products the way smartphone makers think of theirs, people will remain unsatisfied.

Adapting to this new requirement will require focusing on the software and making it as much, or more, of the overall product as the hardware. That means more than just giving token support to older models. Sure, they're going to become obsolete and get left behind, but that can't happen after one year. Car makers will have to own the software-mediated experience and work to make it better, bringing owners of older models along as much as they can.

Smartphones have gotten us used to things that are mostly software and, consequently, get better over time. Every other manufacturer of durable goods will have to follow suit. Their overall success will likely be a product of how well they adapt to this new fact of life.


Pico Event Evaluation Cycle

Summary: Events are processed in KRE using an event loop that decodes the event, makes a schedule, evaluates the respond, and finally assembles a response. This post describes the details of the event evaluation cycle in KRE and how explicit events affect rule schedules.

Let’s Monkey Around

In my post on the event-query API that picos present to the world, I talked about picos listening for events and responding, but there wasn't a lot of detail in how that worked. This post will describe the event evaluation cycle in some detail. Understanding these details can help developers have a better feel for how the rulesets in a pico will be evaluated for a given event.

Each pico presents an event loop that handles events sent to the pico according to the rulesets that are installed in it. The following diagram shows the five phases of event evaluation. Note that evaluation is a cycle like any interpreter. The event represents the input to the interpreter that causes the cycle to happen. Once that event has been evaluated, the pico waits for another event.

event eval cycle

We'll discuss the five stages in order.


The wait phase is where picos spend more of their time. For efficiency sake, the pico is suspended during the wait phase. When an event is received KRE (Kinetic Rules Engine) wakes the pico up and begins executing the cycle. Unsuspending a pico is a very lightweight operation.

Decode Event

The decode phase performs a simple task of unpacking the event from whatever method was used to transport it and putting it in a standard RequestInfo object. The RequestInfo object is used for the remainder of the event evaluation cycle whenever information about the event is needed.

While most events, at present, are transported over HTTP via the Sky Event API, that needn't be the case. Events can be transported via any means for which a decoder exists. In addition to Sky Event, there is also support for an SMTP transport called Sky Mail. Other transports (e.g. XMPP, RabbitMQ, etc.) could be supported with minimal effort.

Schedule Rules

The rule scheduling phase is very important to the overall operation of the pico since building the schedule determines what will happen for the remainder of the event evaluation cycle.

Rules are scheduled using a salience graph that shows, for any given event domain and event type, which rules are salient. The event in the RequestInfo object will have a single event domain and event type. The domain and type are used to look up the list of rules that are listening for that event domain and type combination. Those rules are added to the schedule.

The salience graph is calculated from the rulesets installed in the pico. Whenever the collection of rulesets for a pico changes, the salience graph is recalculated. There is a single salience graph for each pico. The salience graph determines for which events a rule is listening by using the rule's event expression.

Rule order matters within a ruleset. KRE ensures that rules appear in the schedule in the order they appear in the ruleset. No such ordering exists for rulesets, however, so there is no guarantee that rules from one ruleset will be evaluated before or after those of another unless the programmer takes explicit steps to ensure that they are (see the discussion of explicit events below).

The salience graph creates an event bus for the pico, ensuring that as rulesets are installed their rules are automatically subscribed to the events for which they listen.

Rule Evaluation

The rule evaluation phase is where the good stuff happens, at least from the developer's standpoint. The engine runs down the schedule, picking off rules one by one, evaluating the event expression to see if that rule is fired and then, if it is, executing the rule. Note that a rule can be one the schedule because it's listening for an event, but still not be selected because it's event expression doesn't reach a final state. There might be other event that have to be raised before it is complete.

Four purposes of understanding the event evaluation cycle, most of what happens in rule execution is irrelevant. The exception is the raise statement in the rule's postlude. The raise statement allows developers to raise an event as one of the results of the rule evaluation. Raising explicit events is a powerful tool.

From the standpoint of the event evaluation cycle, however, explicit events are a complicating factor because they modify the schedule. Explicit events are not like function calls or actions because they do not represent a change in the flow of control. Instead, an explicit event causes the engine to modify the schedule, possibly appending new rules. Once that has happened rule execution takes up where it left off in the schedule. The schedule is always evaluated in order and new rules are always simply appended. This means that all the rules that were scheduled because of the original event will be evaluated before any rules schedule because of explicit events. Programmers can also use event expressions to order rule evaluation.

If the rule makes a synchronous call to an external API, rule execution waits for the external resource to respond. If a rule sends an event to another pico, that sets off another independent event evaluation cycle, it doesn't modify the schedule for the cycle execution the event:send(). Inter-pico events are sent asynchronously by default.

Assembling the Response

The final response is assembled from the output of all the rules that fired. The idea of an event having a response is unusual. For KRE it's a historic happenstance that has proven useful. Events raised asynchronously never have responses. For events raised synchronously, the response is most useful as a way to ensure the event was received and processed. But the response can have real utility as well.

Historically, KRE returned JavaScript as the result of executing rules. That has been expanded so that the result can be JSON or other correctly mime-typed content. This presents challenges for the engine since rules could be written by many different developers and yet there can be only one result type.

Presently the engine handles this by assuming that any response to an event with the domain web will be JavaScript and otherwise be a directive document (JSON with a specific schema). This suits for many purposes, but doesn't admit raw responses such as images or even just a JSON doc that isn't a directive. The engine tries to put a correctly formatted response together as best it can, but more work is needed, especially in handling raw responses.

This isn't usually a problem because the semantics of a particular event usually imply a specific kind of response (much as we've determined up front that JavaScript is the correct response for events with a web domain). Over time, I expect more and more events will be raised asynchronously and the response document will become less important.


Once the response has been returned, the pico waits for another event.


For simple rulesets, a programmer can largely ignore what's happening under the covers and the preceding discussion is overkill. But applications that consist of multiple rulesets, complex pico structures, or many explicit events can have asynchronous interactions that the developer must understand to avoid pitfalls like race conditions.

If you'd like to add other transports to KRE, we welcome the help. KRE is an open source project hosted on Github. We're happy to help you get started.

The event evaluation cycle described above presents programmers with a flexible, programmable, cloud-based way to respond to events. Explicit events add significant power by allowing the rule schedule to be expanded on the fly. The salience graph provides a level of indirection, binding events to rules in a way that is loosely coupled and dynamic. The event loop is the source of much of KRL's power.

Related: I originally introduced the idea of KRE creating an event loop in A Big, Programmable Event Loop in the Cloud. At that time, 2010, the concept of picos and salience graphs was not fully developed. Those were made possible by the introduction of the Sky Event API in 2011.


Complex Pico Structures: Guard Tours

Summary: We've recently been working on a large project that we architected using picos, CloudOS, and the Personal Cloud Application Architecture. I'm pleased with how this turned out and the lessons that we learned.

Mall Security

We have recently been working on a project for a client that implements a guard tour patrol system. From Wikipedia:

A guard tour patrol system is a system for logging the rounds of employees in a variety of situations such as security guards patrolling property, technicians monitoring climate-controlled environments, and correctional officers checking prisoner living areas. It helps ensure that the employee makes his or her appointed rounds at the correct intervals and can offer a record for legal or insurance reasons.
From Guard tour patrol system - Wikipedia, the free encyclopedia
Referenced Fri Jan 17 2014 15:46:17 GMT-0700 (MST)

You could imagine implementing such a system in a variety of ways, but I saw it as an opportunity to experiment with picos, CloudOS, and the Personal Cloud Application Architectures, extending our techniques and learning important lessons. If you squint a little, you can see that this system and SquareTag are quite similar. In fact, the personal cloud and pico infrastructure are identical.


The fundamental use case of a guard tour system is a guard going from place to place, checking-in at each location to record the state of each location and reporting any anomalies. At its simplest, a tour is an ordered list of locations. The system also has to know about guards, managers, and reports.

We modeled these by creating a prototype pico for each of these objects. At present, KRL has no facility for formally managing prototypes, so we simply write initialization functions that we run in a pico after it's created depending on which type of pico we want to create. There is an Institution pico that is the "root" object of an entire guard tour. Creating prototypes allows us to easily create a new guard tour system using an initialization ruleset.

The other type of pico that might not be apparently obvious is an index pico for tours, locations, and reports. The index pico is necessary to map identifiers for each picos to an event channel identifier (ECI) as well as providing various query capabilities. The tour, location, and report identifiers are URLs and are, consequently, globally unique. Institution picos and the various index picos are all singletons, meaning that in any given Guard Tour set up, there is only one of each.

When a guard tour has been initialized, locations have been entered, and tours created, it might look like this:

guard tour pico relationships

Guards and managers have a "Guard Tour ruleset" installed in their personal cloud. In every other respect, these are just ordinary personal clouds like the ones we create for SquareTag. This is a critical idea: guards and managers don't get an account in the guard tour system. In fact the Guard Tour system has no notion of accounts. Instead, guards and managers have their own personal cloud that they might be using for other things—like running Fuse. This might be a personal cloud they use exclusively for work or they might mingle other components of their life in it—just like people do with laptops and other personal computing devices.

Each guard or manager has a subscription to the Institution pico representing the guard tour. The Guard Tour ruleset manages this subscription and knows what to do with it. The Guard Tour app provides an event-query API for use by the unhosted Guard Tour Web application that provides guards and managers with a user interface for interacting with tours.

The following diagram, from my blog post on the event-query API, shows the overall structure of the application except that a guard tour isn't a single pico, like the one shown here, but rather a constellation of picos like the one shown above. Conceptually, however, it looks just like this since the API provided by the ruleset installed in the guard or manager's personal cloud provides a proxy for the rest of the system. The Web application doesn't know that the other picos exist.

event-query model

The Web application is a pretty straightforward jQuery Mobile application. We built a JavaScript library on top of the CloudOS.js library that provides a Guard Tour-specific interface for the Web application to interface with the event-auery API presented by the Guard Tour ruleset. As for the code, the KRL that runs in the various picos represents about 25% of the overall code base. There's more than twice as much JavaScript (55%) as KRL.

Walking a Tour

When a guard, call him "Frank," starts a tour, he goes to the Web application and authorizes it to access his personal cloud (shown in purple in the first diagram). As stated, the Guard Tour ruleset provides the event-query API that the Web application needs to function. The Guard Tour API presented by the guard's personal cloud acts as an interface for the API presented by the Institution pico. Of course, the Guard Tour API need not be identical to that provided by the Institution pico. The ruleset for a guard, for example, only exposes the funciontality a guard needs; whereas the ruleset for a manager includes facilities for creating new locations and tours, updating them, and so on.

The Institution pico has subscriptions to the tour, location, and report index picos. When Frank does a search for tours that he's been assigned, the Web app asks the API presented by his personal cloud to find locations using specific criteria, which in turn asks the institution for those tours, which further pushes the request to the Tour Index pico. That pico handles the query and returns the list of appropriate tours to show Frank.

When Frank clicks on a tour to start walking it, the process is repeated, only this time, the Tour index pico is used to decode the tour identifier to the ECI for the selected Tour pico (LUFX in the diagram). Queries to that pico return the list of locations, which include ECIs to specific Location picos. As Frank manipulates the user interface, his personal cloud uses temporary channels (shown in green in the diagram) to the tour he's walking, various location picos, and a report pico created just for this interaction. When Frank's done, the report pico is completed and his personal cloud forgets all the temporary channels.

Lessons Learned

That may sound complicated, but it's surprisingly quick and effective. We could have just stored a tour's locations as data inside the tour pico. For that matter, we could store tours as data inside the Institution pico. By representing each as a separate pico, we maximize our flexibility and allow for reuse. For example, a location can be in more than one tour. More importantly, I imagine the day when each room in a building might be represented by an online avatar. One function of that room avatar would be to provide data for a guard tour. But it might also know about the room's structure, features, emergency access capabilities, maintenance, and so on.

Similarly, we could of just created accounts in the Institution pico and had it offer a OAuth-protected API directory, but I believe strongly in the idea of personal clouds and saw this as a way to experiment with how they can be used in conjunction with unhosted Web applications and a complex pico constellation to provide specific functionality. For example, Frank the guard might want his personal cloud to record the fact that he walked particular tours on particular days.

Overall, we've learned a lot from this project and I'm grateful we got the chance to do it. Here are a few of the things we've learned:

  • Building complex pico structures with many members that are also fast and responsive
  • Using personal clouds as the basis for interacting with complicated, pico-based applications
  • Techniques for testing
  • Using index picos and temporary channels

Many of these techniques will be useful to us as build the Fuse API and application. That's next on our plate. Eventually I hope to redo SquareTag in this style, making it an application that could be used by people with personal clouds that are hosted on other systems so long as they were based on CloudOS. There is much to recommend picos, event-query APIs, personal clouds, and unhosted Web apps as a programming style: it's readily distributable and protects personal data. I'm excited by the progress we've made.


Test-Driven Development and KRL

Summary: We've recently released some initial tools to support test-driven development in KRL. This is a great first step toward a more mature development process for CloudOS.

smashed up bug

One of the best decisions I made when we started Kynetx was to use test-driven development from the start. Our core engine, written in Perl, has thousands of tests and we run them religiously as we develop and before we deploy a new version to the servers. When someone reports a bug, the first thing we do is write a test to exercise it. That way we know we really have a bug and we know when it's squashed. As a result, we've done over a thousand deploys of the rules engine over the last 6 years and have only truly broken production a few times.

Interestingly, most of the time when production has broken, we kinda of expected it to. Not that we wanted it to, but we knew the changes we so vast and far reaching that it was almost certain that something would happen we hadn't anticipated and written a test for. For those kind of changes, you can't be so afraid of breaking production that you stifle innovation.

With all that, I'm ashamed to say that we've never really developed a test-driven development process for KRL, the rules language that is at the heart of everything we do. Partly, that's because for 4.5 of the six years that we've been writing KRL, the rulesets have been simple—usually less than 5 or 6 rules—and rarely interacted with each other. Over the last year or so, however, we've been writing applications in KRL that are much more complex than what we did previously. Consequently, knowing that they work correctly after a change can be a challenge...unless you have tests.

Over the past few weeks I've been trying to rectify this. We're doing a large project in KRL, arguably the most complicated pico structures and event-query APIs that we've built to date. I knew that the only way to make it rock solid was have tests and that meant developing the tools and processes necessary to effectively write them.

The first attempt at this is documented on the KRL wiki: Test-Driven Development and KRL. We have a KRL Test module to aid in writing and running tests along with a test harness in Perl.

I've used this to write tests for a few important modules and we'll be working on more of CloudOS over the next while. Ultimately we hope to have as good of test coverage on CloudOS as we do on the underlying engine so we can develop in confidence and reduce the friction of chasing bugs that should have never made it into production.

The tools we have now are simple. No doubt they'll gain additional features over time. If you're a KRL developer, feel free to offer suggestions. Better yet, send a pull request.


Intention Generation: Fuse and VRM

One of the most influential books I've read in the last several years is Doc Searls' Intention Economy. The concept is simple: customer demand is the source of commerce and yet business has done a poor job of finding ways to understand customer intention. Doc's ideas have given rise to a movement called vendor relationship management or VRM. The term is a play off of CRM, and leads to a key idea: customers need tools to manage their interactions with the people who sell them products and services.

When I write about social products, I'm writing about one such tool. Describing one of our experiments in building social products, SquareTag, I wrote:

The owner's side looks and feels like a Facebook wall where messaging and other interactions happen in the context of the thing itself. Things have profiles that they share with other like things. The profile contains pictures, product Web pages, manuals, how to videos, and other useful information. But the product profile is individual—made for a single instance of the product. It also contains information specific to that thing such as custom configurations, serial numbers, purchase history, maintenance history, and relationships to anyone or anything that is relevant. One of the most interesting thing that the product profile holds is notifications, reminders, and other interactions with the things and people it has relationships with.
From Facebook for My Stuff
Referenced Thu Dec 12 2013 11:33:29 GMT-0700 (MST)

Of course, the intention economy depends on many of our intentions being generated automatically. A few years ago, we put together a business travel scenario showing how events could be used to automate many of the mundane activites associatd with planning and taking a business trip. Here's a video that describes it:

We no longer use some of the nomenclature in the video like "personal event network" but you can see how events described in the video are really encodings of intention—intentions turned into semantically consistent, structured data that computer systems can operate against. It's funny that when I made that screencast, I had no idea how a car might customize itself based on events. Now that we've started working on Fuse, it seems perfectly doable.

Fuse, our connected-car product is an intention generator. Here's a few examples:

  • When Fuse sees your gas tank is nearly empty it can generate an intention to buy gas.
  • When Fuse indicates it's time for an oil change or tire rotation, it can generate an intention to have the car serviced.
  • When the vehicle raises a diagnostic code, Fuse can generate an intention to get the car fixed.
  • When insurance is up for renewal, Fuse can generate an intention to solicit quotes for a new policy.
  • Geofences could be linked to intentions.
  • Even a crash, sensed by Fuse's accelerometers, is an intention to seek emergency services.

As an intention generator Fuse could be seen as a brand-new way for companies to spy on drivers. But we don't think it has to be that way. If Fuse is going to generate intentions that can be acted on while preserving owner choice and privacy, it must also provide owners with two things:

  1. A way to see, select, and interact with vendors—both those who the owner has an existing relationship with and those who might be good candidates for future purchases.
  2. A way to use intentions and the make the choices that only the owner can make. For example, when my insurance is due, Fuse needs to ask me if I'm happy with my current insurance before going out to solicit bids.

Both of these features are about providing owner choice and putting the owner in control. In the terminology of VRM, the thing providing these features is typically called the "4th party" and refers to the system that is acting on the customer's behalf. When I speak of social products, I'm really describing our particular approach to building a VRM tool that provides these two necessary functions.

I believe this concept and architecture is critical to building a future world we will all want to live in. I frequently hear people describe connected products that just increase customer surviellance in service of advertising and digital marketing. If a connected product isn't putting the owner in charge of who sees what signals and when, then it's spying, pure and simple.


Ed Orcutt 1961-2013

ed and me

Ed Orcutt died on Monday morning of a long illness. Anyone who's come to a Free Lunch Friday or visited Kynetx knows Ed. Anyone who's used a Kynetx product in the last 3 years knows him, at least through his work. Ed was a great friend and an exceptional programmer. We're going to miss him very much.

Ed was my principal collaborator and the primary developer of CloudOS, the software layer that powers SquareTag, Fuse, and many other projects and products we've built. Ed designed and programmed the Intentcasting demonstration we built for Swift. He developed the DnB 360 product we sold to Dun and Bradstreet. Before that he was the developer of an app on the Kynetx browser extension, Hover Me, that had over 100,000 installs—our most successful KBX app ever. He helped me build several of the examples in my book.

Ed was a stalwart believer in what we were doing at Kynetx. He embraced the evented programming style and worked to extend our understanding of what it could do. Ed surprised me over and over with what he accomplished with KRL. His practical work often brought my theoretical vision to life. I will be forever grateful for Ed's faith in me and the vision we have for Kynetx.

Already we've had several of what I've started calling "Ed moments" where the natural reaction to something is "let's ask Ed". Of course, we quickly realize that we can't ask Ed anymore. Ed was such a pivotal part of what we do, I'm sure there will be many more of those.

Thanks for being part of Kynetx and part of our lives, Ed.

About the picture: This photo shows Ed and I in a discussion about CloudOS in my office at Kynetx about a year ago (Dec 1, 2012).


Kynetx Reboot

Summary: We've had a successful Kickstarter and now we've got a new CEO. Kynetx is ready to take off.

There are a few changes happening at Kynetx that I like to think of as a reboot. Most importantly, we just completed a successful Kickstarter campaign for Fuse, our connected car product. Kickstarter, at least in our case, wasn't as much about raising money as it was about product validation. We've sold almost 500 Fuse devices. That's a big step toward understanding the market for connected car products. We've also been able to map out markets we play in and how we're different from competitors in those markets.

The second big change is that Steve Fulling is leaving Kynetx and I'm taking over as CEO. There's no drama here. Steve's leaving for personal reasons and he remains a big shareholder and member of the board. Steve and I remain close friends and collaborators. Steve will continue to be an active participant in strategic discussions about Kynetx and its future.

Kynetx is in a good place, a place we've never been before with a product that's had some market validation. People are excited about Fuse and that, in turn, makes me excited about the future. There will be challenges ahead, to be sure, but we're poised to really take off in 2014.


Fuse Funds! Thank You and Next Steps

Summary: Our Kickstarter campaign for Fuse has ended successfully. This was an exhilarating experience. Thanks to everyone who participated.

KS Campaign The Fuse Kickstarter campaign has ended and we exceeded our goal of $60,000 by over $19,000 for a total of $79,024. More importantly, there were 386 backers of the project. I'm grateful to everyone who supported this and humbled by the messages of encouragement, backed by hard currency, that I received from family members and friends. Thank you all! Running a Kickstarter campaign is fun. More fun than I would have imagined. There are periods of intense doubt and periods of unimaginable exhilaration. We learned a lot about how to run a campaign. There was a lull in the middle of the campaign. I understand from people who've done this before that this is fairly typical. Towards the end as the numbers were getting close to the goal and pledges were coming in every few minutes it was addictive. We couldn't do anything but sit and watch the numbers. I was mesmerized. Perhaps the most surprising thing I learned about Kickstarter is the level of engagement it engenders in people. You are asking people to join with you and band together to help create something new. For some, Kickstarter is just another kind of shopping site, but many support projects on Kickstarter to be part of making something. Kickstarter also leads to conversations with people about what you're doing that you wouldn't otherwise have.
Face on. Sample buy viagra online to. In small still with where to buy viagra product burned especially pharmacy online using drains perfect looks one palette nice product cialis free damaged rarest boxes viagra price neck true-to-claim yours! Down canadian online pharmacy I HORRIBLE that impressed. All viagra cost Loves after reflexive. Raises order viagra have, which to these cialis side effects buy absolutely and! Do cialis daily dry I using is?
There's a marketing value in the way it helps get the message out to more than just potential backers. We've had half a dozen conversations with potential partners that were a direct result of our Kickstarter campaign. Kickstarter itself was a big help. Kickstarter pages and emails were responsible for over 40% of our total sales. Some of that is probably people who we had contacted who then went to Kickstarter and found it through search or the discovery pages. But most of that number is organic, Kickstarter-generated support. That's a big win. There are people who go to Kickstarter and look for cool projects to support. Now that the campaign is over, we're all going to take a deep breath and enjoy the weekend. Come Monday we'll be hard at work at Fuse, making it cool and useful. Watch for a survey from us soon on platforms and features so we can make sure Fuse hits the mark. Tags: