General purpose executable graphs and diagrams
My previous story introduced a capability of loading behavior (ability to do something) of arbitrary complexity from URIs. Behavior can be implemented in Java, scripting languages such as Groovy, and Drawio diagrams.
This story focuses on using this new capability with graphs and diagrams — associating one or more behaviors with graph/diagram elements.
Table of Contents
· Demo
∘ Person
∘ System
∘ Client code — dynamic proxy
∘ Wrapping into an invocable URI
· Observations
· Applications
∘ Documentation
∘ Generation
∘ CLI
∘ Semantic mapping
· Ecosystem
∘ Diagrammer
∘ Processor Developer
∘ Librarian
∘ Mapper
∘ Consumer
· Conclusion
Demo
In this section we will take a look at a very simple demo depicted below:
Please consult the online documentation for details.
Person
Person.groovy
import java.util.concurrent.CompletionStage
import java.util.function.BiConsumer
import java.util.function.Consumer
import java.util.function.Supplier
import org.nasdanika.capability.CapabilityFactory.Loader
import org.nasdanika.common.Invocable
import org.nasdanika.common.ProgressMonitor
import org.nasdanika.drawio.Node
import org.nasdanika.graph.Element
import org.nasdanika.graph.processor.OutgoingEndpoint
import org.nasdanika.graph.processor.ProcessorConfig
import org.nasdanika.graph.processor.ProcessorElement
import org.nasdanika.graph.processor.ProcessorInfo
// Script arguments for reference
Loader loader = args[0];
ProgressMonitor loaderProgressMonitor = args[1];
Object data = args[2]; // From fragment
ProcessorConfig config = args[3];
BiConsumer<Element, BiConsumer<ProcessorInfo<Invocable>, ProgressMonitor>> infoProvider = args[4];
Consumer<CompletionStage<?>> endpointWiringStageConsumer = args[5];
ProgressMonitor wiringProgressMonitor = args[6];
new org.nasdanika.common.Invocable() {
/**
* Diagram element is injected into this field
*/
@ProcessorElement
public Node element;
private amountSupplier
@OutgoingEndpoint
public void setAmountSupplier(Supplier<String> amountSupplier) {
System.out.println("Amount supplier " + amountSupplier);
this.amountSupplier = amountSupplier;
}
/**
* This method is invoked by the dynamic proxy's Function.apply() method.
* The first argument is the proxy object - it can be used to resolve state or call other methods of the proxy.
* The second is the apply()'s argument.
*/
def invoke(Object... args) {
System.out.println(args);
System.out.println(element.getProperty("greeting") + " I have " + amountSupplier.get() + " dollars in my bank and just got " + args[1] + " from you. I'm so happy!!!");
}
}System
SystemProcessor.java
package org.nasdanika.demos.diagrams.proxy;
import java.util.concurrent.CompletionStage;
import java.util.function.BiConsumer;
import java.util.function.Consumer;
import java.util.function.Supplier;
import org.nasdanika.capability.CapabilityFactory.Loader;
import org.nasdanika.common.Invocable;
import org.nasdanika.common.ProgressMonitor;
import org.nasdanika.drawio.Node;
import org.nasdanika.graph.Element;
import org.nasdanika.graph.processor.ConnectionProcessorConfig;
import org.nasdanika.graph.processor.IncomingHandler;
import org.nasdanika.graph.processor.NodeProcessorConfig;
import org.nasdanika.graph.processor.ProcessorConfig;
import org.nasdanika.graph.processor.ProcessorElement;
import org.nasdanika.graph.processor.ProcessorInfo;
/**
* This processor is not interacted with by the client code and therefore does not implement the processor type - {@link Invocable}.
*/
public class SystemProcessor {
private String amount;
@ProcessorElement
public void setElement(Node element) {
this.amount = element.getProperty("amount");
}
/**
* This is the constructor signature for graph processor classes which are to e instantiated by URIInvocableCapabilityFactory (org.nasdanika.capability.factories.URIInvocableCapabilityFactory).
* Config may be of specific types {@link ProcessorConfig} - {@link NodeProcessorConfig} or {@link ConnectionProcessorConfig}.
* @param loader
* @param loaderProgressMonitor
* @param data
* @param fragment
* @param config
* @param infoProvider
* @param endpointWiringStageConsumer
* @param wiringProgressMonitor
*/
public SystemProcessor(
Loader loader,
ProgressMonitor loaderProgressMonitor,
Object data,
String fragment,
ProcessorConfig config,
BiConsumer<Element, BiConsumer<ProcessorInfo<Invocable>, ProgressMonitor>> infoProvider,
Consumer<CompletionStage<?>> endpointWiringStageConsumer,
ProgressMonitor wiringProgressMonitor) {
System.out.println("I got constructed " + this);
}
@IncomingHandler
public Supplier<String> getAmountSupplier() {
return () -> amount;
}
}Client code — dynamic proxy
Function<URI, InputStream> uriHandler = null;
Function<String, String> propertySource = Map.of("my-property", "Hello")::get;
Document document = Document.load(
new File("diagram.drawio"),
uriHandler,
propertySource);
ProgressMonitor progressMonitor = new PrintStreamProgressMonitor();
DocumentInvocableFactory documentInvocableFactory = new DocumentInvocableFactory(document, "processor");
java.util.function.Function<Object,Object> proxy = documentInvocableFactory.createProxy(
"bind",
null,
progressMonitor,
java.util.function.Function.class);
System.out.println(proxy.apply(33));Result
Hello World! I have 385 dollars in my bank and just got 33 from you. I’m so happy!!!
In the result Hello comes from the property source. 385 is obtained from the System where it is computed from the amount property. And 33 comes from the apply() argument.
Wrapping into an invocable URI
YAML specification
Below is a specification wrapping the diagram into an invocable URI:
diagram:
location: diagram.drawio
processor: processor
bind: bind
interfaces: java.util.function.Function
properties:
my-property: Hola,The same specification, but with the diagram inlined, so the spec is self-sufficient:
diagram:
source: |
<mxfile ...abridged... </mxfile>
processor: processor
bind: bind
interfaces: java.util.function.FunctionClient code
CapabilityLoader capabilityLoader = new CapabilityLoader();
ProgressMonitor progressMonitor = new PrintStreamProgressMonitor();
URI specUri = URI.createFileURI(new File("diagram-function.yml").getCanonicalPath()).appendFragment("my-property=Hello");
Invocable invocable = capabilityLoader.loadOne(
ServiceCapabilityFactory.createRequirement(Invocable.class, null, new URIInvocableRequirement(specUri)),
progressMonitor);
Function<String,Object> result = invocable.invoke();
System.out.println(result);
System.out.println(result.apply("15"));Result
Hello World! I have 385 dollars in my bank and just got 15 from you. I’m so happy!!!
In the result Hello comes from the spec fragment, without it it would have been Hola coming from the spec my-property. 385 is obtained from the System where it is computed from the amount property. And 15 comes from the apply() argument.
If we had my-property value as ${env.GREETING} then the final value would be taken from the GREETING environment variable. And if we had it as ${env.GREETING|Hola,} then Hola, would be used as the default value if the GREETING variable was not set.
Observations
- Multiple processors (aspects/behaviors) can be associated with a single diagram element using properties
- URI fragments can be used parameterize processors
- Property values can have placeholders to customize URIs
- Diagrams can be created and maintained by non-technical people, Subject Matter Experts (SMEs) who know their problem domain, but not necessarily technical
- Processors can be implemented in Java, scripting languages, and diagrams
- Processors can be configured using YAML or JSON specifications, which includes Maven dependencies
- Diagram processors can be wrapped into a dynamic proxy, abstracting client code from the implementation and allowing multiple swappable implementations
Applications
Documentation
Documentation generation is already implemented in Nasdanika CLI drawio html-app command. An example of a site generated using this command — Internet Banking System. Declarative Command Pipelines and Visual Communication Continuum stories provide more information about the existing functionality.
Ability to associate multiple processors with a single diagram element and parameterize processor URIs with fragments and placeholders opens the following possibilities:
- Multiple documentation sets for different target audiences. Documentation may have multiple dimensions — language, area of focus, level of details, …
- Filtering of documentation including diagram elements for different audiences and using external data. For example, an R&D/POC infrastructure diagram with all elements for the R&D team, but only elements used in a particular POC for the POC stakeholders. And modifying style of elements based on their status. E.g. a red border for elements with problems and reduced opacity for elements unavailable due to planned maintenance. Another example is development activities — change element appearance/style based on status of issues associated with the element in the current sprint. E.g. a green border for Done, no border for no issues, blue for In Progress, grey for not started, … Status may roll-up from child elements to parent elements. The same diagram can be used to generate both flavors — runtime status for operations, development status for product owners.
- Creating libraries of elements with pre-set processors. For example, in the case the Internet Banking System the mainframe might be used by multiple other systems and it may have DEV/Test environments/regions with planned unavailability. Creating a shared library with a mainframe shape which “knows” how to pull status and update itself might be very useful for development teams.
Generation
Documentation generation is one type of generation. Another is code generation with multiple flavors — business logic code, client code for different languages, … Infrastructure is yet another.
With code generation one interesting feature is to be able to merge new generated code with manually modified previously generated code preserving manual changes. It would allow to generate high-level structure and manually code low-level details. More about it in Solution Instantiation.
Generation can be triggered by pushes to the source repository — as it is done with documentation generation for the Internet Banking System using GitHub Actions. Generation may be configured to be triggered only on specific changes. E.g. trigger code generation if the diagram file changes or files in some sub-directory change, but not if files under src folder change. At the same time, don’t trigger a build for changes in the diagram file. This would allow to have a generation chain diagram -> sources -> build.
If generated sources need to be reviewed by developers before incorporating them into the codebase, then the generation process may create a branch and a pull/merge request.
CLI
Processing logic, including generation, can be wrapped into CLI commands. Commands can be organized into “pipelines”. E.g. there is already a drawio command which reads a diagram file and makes it available to subcommands. Similarly there might be a process sub-command which takes processor property name as an argument, and a dynamic-proxy command which takes processor property and bind property.
Yet another opportunity, similar to the dynamic proxy, is to have processors which implement HttpServerRouteBuilder interface — this would allow to serve diagrams as web applications!
Semantic mapping
I already explained Semantic Mapping in the Visual Communication Continuum story — it is the process of creating model elements, e.g. Architecture Model or C4 Model, from diagram elements.
Semantic mapping as it is implemented now works well! It supports namespaces to allow mapping of a diagram element to multiple semantic elements. However, in cases where there is a considerable number of semantic targets per diagram element in addition to other processors, or the mapping is complex, it may be more convenient to define semantic mappers as processors.
Semantic mapping can be used in command pipelines — load a diagram with drawio command, convert it to a model with a semantic mapping processor command, and then use sub-commands of the model command — save, html-app, … Generation of multiple flavors of documentation can be implement this way.
Ecosystem
Diagramming is a process with multiple participants, activities, and artifacts. As it was mentioned in my previous story, it is an enterprise with a mission to bind decisions to make them executable.
Let’s take a look at how the process might be implemented. Please note that one person can represent multiple participants (roles) and vice versa.
The sources of the below process can be found here. You may clone or fork the repository and customize to your needs. For example, modify descriptions of process steps and other elements to your contexts. E.g. “bind” source repository to GitHub, CDN to JSDELIVR, and Binary repository to Maven Central.
Below I’ll provide a brief overview of the roles and steps in the above ecosystem. I already covered some of it in my previous story, along with artifacts and repositories, at a more general level. This section and the online documentation is a specialization for diagrams.
Click on the image caption to navigate to the online documentation with more detailed information. Over there you may click on the diagram elements to navigate to their respective documentation pages.
Diagrammer
Diagrammer is a party authoring diagrams — a human, an organization or a system.
They may use libraries of diagram elements with pre-set properties.
Diagrams can be created using one of many Drawio editors:
- Desktop app
- Web app
- Plug-ins
- VS Code
- Confluence
- …
They can also be create programmatically using Drawio Java API or other means.
Processor Developer
Processors can be created in Java or scripting languages.
Java processors are published to binary (Maven) repository.
Scripts can be stored into a version control system like Git, or be bundled into jars and published to a binary repository. They can also be encoded into data URL’s. Groovy scripts can be compiled to Java classes.
Librarian
Librarians create YAML/JSON specifications for Java, script and diagram processors. They may specify properties/bindings and, in the future releases, inheritance.
Librarians also create and publish catalogs of available building blocks — Java/Maven modules, scripts, diagrams, URIs. Librarians can use Wikis, GitHub pages, …
Catalogs may have multiple dimensions — technology, organization, geography, …
For diagrams librarians also may create libraries of reusable diagram elements, including groups of elements (patterns). Libraries can be stored in a source repository system. With GitHub, libraries can be published using GitHub Pages or JSDELIVR CDN. Drawio browser app can load libraries from a multitude of sources, including GitHub.
Librarians may also create and publish URI construction wizard applications where a URI is constructed based on user choices. Something similar to Drawio custom link tool.
Mapper
Mappers set processor URIs in diagrams. They use URI catalogs and wizards to find or construct URIs. Mapped diagrams can be wrapped into specs and be added to catalogs or used by consumers in Java or CLI applications.
Consumer
Consumers (Operators/Users) execute Java/CLI applications which leverage scripts, diagrams, and invocable URIs.
Operators may customize Maven configuration for loading dependencies. E.g. point to a corporate binary repository.
They also have control over system properties and environment variables which may be used in invocable URIs or be invocable URIs themselves.
Conclusion
In my opinion the current state of diagramming has a huge gap or even multiple gaps!
On the left side of the design/development journey there are “bare diagrams” — just shapes with connections. They are good to capture and discuss high level ideas. However, they become deficient in a very short time because they don’t support adding details beyond a few lines of comments.
Somewhere in the middle there are specialized diagramming solutions tailored for a specific purpose and often being a lot of burden — you may need a tiny fraction of the whole notation and the rest becomes noise.
The approach explained here may help you fill the gaps, shall be willing to do so. So, you feel the pain of inefficiencies — this might be a remedy!
