New Apple Patent. Siri: “Jim the plumber is at your front door. What should I do?

Brian Roemmele

Published in

multiplex

7 min readMay 18, 2017

🔊 Have this story read to you

This New Apple Patent Talks About A Voice First Home Hub And More.

On May 11th, 2017 Apple the USTPO released an Apple patent application that is stunning. This patent is a landmark patent that describes new embodiments that Apple has never before presented. I just produced a Multiplex Memorandum for paid subscribers (just $5 bucks a month, you can afford it) on the notable aspects of this patent. I will present two astonishing new futures.

In this patent Apple makes it clear that the user will be interacting with Siri over a number of devices. One technique they use is called a blackboard system. A blackboard system is an artificial intelligence approach based on the blackboard architectural model, where a common knowledge base, the “blackboard”, is iteratively updated by a diverse group of specialist knowledge sources, starting with a problem specification and ending with a solution. Each knowledge source updates the blackboard with a partial solution when its internal constraints match the blackboard state. In this way, the specialists work together to solve the problem. The blackboard model was originally designed as a way to handle complex, ill-defined problems, where the solution is the sum of its parts.

The embodiments of this new patent goes into deep detail of how Apple will use the blackboard technology for a new version of Siri. The patent describes two or more Voice First systems interacting with each other. In some embodiments the Voice First devices can owned by other users. This can allow for mediation and orchestration of a number of situations from calendars to solving delegation tasks.

Apple describes one embodiment that is stunning:

[0296] Returning to the first example, Pierre’s digital assistant 700a generated a plan to let Tomas into his apartment. During execution of that plan, the digital assistant 700b associated with Pierre’s apartment awaits contingent input associated with Tomas’ arrival. Referring to FIG. 8Z, as one example of an exception, the digital assistant 700b recognizes that someone who is not Tomas has arrived at the door. The digital assistant 700b requests the name of the visitor, who replies “Jim the plumber.” The digital assistant 700b is expecting Tomas at approximately 8:00 p.m. However, a visitor other than Tomas has arrived. This is an exception to the plan. Referring also to FIG. 7D, according to some embodiments, the digital assistant 700b associated with Pierre’s apartment generates a message that a person who has identified himself as “Jim the plumber” is at the door; that message travels through the blackboard 772 and message gateway 770 to Pierre’s digital assistant 700a, which in this example is shown as digital assistant 700a. Pierre’s digital assistant 700a recognizes the identity discrepancy between Tomas and “Jim the Plumber,” identifying the exception. Optionally, Pierre’s digital assistant 700a first determines if a rule is present in the plan template 802 for handling the arrival of a different visitor; in this example, there is not. Referring to FIG. 8Z, finding no rule, Pierre’s digital assistant 700a handles the exception by causing a message 850 to be displayed on the display 212 of Pierre’s electronic device 200: “`Jim the plumber` is at your front door. What should I do?” Pierre is not expecting a plumber, and is surprised one would want entry to his apartment at 8:00 p.m. Referring to FIG. 8AA, Pierre tells the digital assistant 700a, “Don’t let him in. Also, call the building manager”; the digital assistant 700a displays that message 852 on the electronic device 200 according to some embodiments. Pierre’s digital assistant 700a transmits this message to the message gateway 770, which forwards the message 830 to the blackboard 772. The digital assistant 700b associated with Pierre’s apartment is forwarded the message 830 by the blackboard 772, or retrieves the message 830 from the blackboard 772. The digital assistant 700b then leaves the door locked. Further, the digital assistant 700b calls the building manager, and informs the building manager that “Jim the plumber” is at the door. The plan generated to let Tomas in when he arrives is not complete, because Tomas has not yet arrived. However, the exception has been handled by refusing entry to the apartment to “Jim.”

You read this correctly, in this embodiment Siri is answering the front door and mediating interactions with guests. This is a far more sophisticated Siri that has high context and using a shared blackboard system to interact. We can see Apple diving far deeper into the home with ambient Voice First technology. There is a lot of embodiments that go even deeper into how the new Siri platform will extend in this patent.

In this patent Apple for the first time describes a stand-alone Voice First device they are calling a “human automation system hub”:

[0258] Referring to FIG. 7D, a block diagram architecture 766 is shown for virtual assistant interactions according to various examples. A first digital assistant 700a is shown, substantially as described above with regard to the digital assistant 700. The first digital assistant 700a is accessed by the user through a portable electronic device 200, according to some embodiments. A second digital assistant 700b is shown, substantially as described above with regard to the digital assistant 700. The second digital assistant 700b is accessed by a user through a human automation system hub, according to some embodiments. According to other embodiments, the second digital assistant 700b is accessed by a user through a portable electronic device 200, a television interface such as the Apple TV.RTM. digital media extender of Apple Inc. of Cupertino, Calif., a vehicle, or any other device through which a digital assistant 700 can be accessed.
…
The user digital assistant #700a is separate from the hub digital assistant #700b; the two digital assistants 700a, 700b are discrete entities.
…
Each separate digital assistant 700a, 700b is associated with the same user, according to some embodiments; for example, the user digital assistant 700a may be accessed by the user through his or her portable electronic device 700a, and the hub digital assistant 700b may be accessed by the user through his or her home automation system. “

It is very clear Apple is directly speaking about the continuity between a number of access points to this new Siri environment.

This new patent offers some of the most interesting aspects of how Apple will move the new Siri forward. I called this a landmark patent for many reasons.

Patents offer an aperture into the future. I have studied Apple patents for over 30 years and can say with certainty, this patent and a series of others are pointing to a massive shift for Siri, Messages App and the new #VoiceFirst system I have been calling HomeBase.

~ — ~

[adrotate group=”1"]

~ — ~

I dive in to the way this patent will change the direction of Apple in a number of ways and present the opportunities it will create for startups and legacy companies. The Multiplex Memorandum is a feature of the Multiplex Magazine subscription. You are on the web based blog for the Multiplex App. This plus much more can be accessed exclusively through the Multiplex app found only on the iOS app store. Though the Multiplex app you can choose to subscribe. Subscriber features include: