Deep dive into writing macros in Dart 3.5
I spent a month’s vacation full-time playing with macros, and here’s all you need to start fast.
In the first part, we set up a beta version of Dart to experiment with macros, tried @JsonCodable macro that the Dart team released to showcase the features, and wrote a hello-world macro.
In this second part, I will explain my full-fledged macro that produces a parser for command line arguments. I will use it as an example to explain all aspects of writing and testing a macro.
This implies that you have read the first part.
Warning: This whole Dart feature is currently in preview and will see many breaking changes before it matures. It just was too fun for me to wait.
Begin with the code you want
With any code generation, begin with manually writing an example of the code you want generated and then work your way to write the generator.
In our case, we want to apply a macro to the following class:
@Args()
class HelloArgs {
final String name;
final int count;
}And we want this generated for us:
class HelloArgsParser {
final parser = ArgParser();
HelloArgsParser() {
_addOptions();
}
void _addOptions() {
parser.addOption("name", mandatory: true);
parser.addOption("count", mandatory: true);
}
HelloArgs parse(List<String> argv) {
final wrapped = parser.parse(argv);
return HelloArgs(
name: wrapped.option("name")!,
count: int.parse(wrapped.option("count")!),
);
}
}
augment class HelloArgs {
HelloArgs({
required this.name,
required this.count,
});
}Here is the full code of my simplified macro that does this minimal job (check out this specific branch). I will first explain every part of it, and then we will dive into a storm of edge cases with error handling, optional arguments, boolean flags, lists, enums, default values, help messages, etc., but let’s begin with just this.
Phases of macro application
When you apply a macro to a class, it can’t just run and generate all the code in one pass. This is because you can potentially have multiple macros all modifying a single class. Can one macro see and modify the code generated by another one? In what order? To resolve those issues, phases were introduced. There are 3 phases: type creation, declaration, and definition.
Type creation phase
This is the first one. At this phase, a macro can generate code that introduces new types: classes, mixins, enums, typedefs, etc. At this phase, a macro can see other types in the program but can’t inspect them because a type can be shadowed by another type created by another macro after that.
Declarations phase
By the time when the declarations phase begins, all types have been created and no new types can be introduced. This allows us to see what exactly each type name resolves to. With this capability, a macro can now see into fields and methods of any class. It can create new fields and methods in existing types. However, there are still things it can’t see in the program. For instance, it can’t infer the types of variables because in this code
final a = b;b can still be shadowed if another macro introduces a getter named b.
Definitions phase
By the time when the definitions phase begins, all declarations in all types have been done, and nothing more can be shadowed. At this point, all macros enjoy the maximum visibility into the program. However, all they can do is replace the bodies of existing methods and the initializers of existing variables.
So the phases run from the maximum power with the least visibility to the least power with maximum visibility. These constraints allow multiple macros to be unaware of each other, run independently, and largely be agnostic to the order of their execution.
A macro can decide at which phases it wants to run. It does so by implementing one or more of the predefined interfaces.
Choosing the phases and the interfaces
In our case, a macro should be applicable to such a class:
@Args()
class HelloArgs {
final String name;
final int count;
}It should generate a class named HelloArgsParser which will instantiate ArgParser object from the standard args package and create an option in it for each of the fields.
This means that the macro must run in 2 of the 3 available phases:
- It should create
HelloArgsParserclass, so it needs to run in the types phase. - It needs to introspect the fields of the class it’s applied to, so it also needs to run in the declarations or definitions phase. The latter is tricky as I will explain later, so we choose the declarations phase.
This means the macro class must implement the following two interfaces:
macro class Args implements ClassTypesMacro, ClassDeclarationsMacro {
@override
Future<void> buildTypesForClass(
ClassDeclaration clazz,
ClassTypeBuilder builder,
) async {
// Create the parser class here.
}
@override
Future<void> buildDeclarationsForClass(
ClassDeclaration clazz,
MemberDeclarationBuilder builder,
) async {
// Populate the parser with options and parse the data here.
}
}The interface for the third phase is called ClassDefinitionsMacro, and we don’t need it for now.
If you wanted your macro to be applicable to enums, you would use the interfaces EnumTypesMacro, EnumDeclarationsMacro, and EnumDefinitionsMacro. There are many more sets of interfaces for macros applicable to other things in your code. You can find all of them in the API.
You can see that each method takes two parameters:
- The declaration of the class the macro was applied to.
- The builder object specific to the given phase to introspect and modify the program. This is how the visibility and the power of a phase are enforced.
Creating a class programmatically
It’s easy:
@override
Future<void> buildTypesForClass(
ClassDeclaration clazz,
ClassTypeBuilder builder,
) async {
final name = clazz.identifier.name;
final parserName = _getParserName(clazz);
builder.declareType(
name,
DeclarationCode.fromString('class $parserName {}\n'),
);
}
String _getParserName(ClassDeclaration clazz) {
final name = clazz.identifier.name;
return '${name}Parser';
}Initially, we create this class empty. We can potentially create methods and properties right here, but we heavily rely on the data that we will only learn in the second phase, so it makes more sense to generate all methods there in one place.
Augmenting the classes
At the declarations phase, we need to do the following:
- Learn all about the fields in the current class.
- Add a constructor.
- Augment the empty parser class so it parses the options.
@override
Future<void> buildDeclarationsForClass(
ClassDeclaration clazz,
MemberDeclarationBuilder builder,
) async {
final intr = await _introspect(clazz, builder);
await _declareConstructor(clazz, builder);
_augmentParser(builder, intr);
}Introspection is a long story. First, let’s just assume we have all the information about the fields that we need in intr variable and set up other things.
Adding a constructor
Our data class only has fields but not a constructor. Let’s add one.
I created @Constructor() macro that does just that and cares for a lot of special cases. In Args macro, we only need to do this:
Future<void> _declareConstructor(
ClassDeclaration clazz,
MemberDeclarationBuilder builder,
) async {
await const Constructor().buildDeclarationsForClass(clazz, builder);
}This is how a macro can directly invoke another macro to do a part of its job. We pass the same builder to that macro, so all the code it generates is as good as what we generate directly.
Setting up the iterations
We need to iterate the fields more than once:
- To add an option to the standard
ArgParserfor each field. - To construct the data object with the parsed values for options.
On each iteration, the behavior for each field depends on its type, so it’s a good case for the Visitor pattern. This pattern extracts the behavior nicely into classes and gives a compile-time guarantee we have all types covered every time we iterate the fields.
We will start with this visitor class:
abstract class ArgumentVisitor<R> {
R visitInt(IntArgument argument);
R visitString(StringArgument argument);
}And the two classes for the argument types:
sealed class Argument {
Argument({
required this.intr,
required this.optionName,
});
final FieldIntrospectionData intr;
final String optionName;
R accept<R>(ArgumentVisitor<R> visitor);
}
class IntArgument extends Argument {
IntArgument({
required super.intr,
required super.optionName,
});
@override
R accept<R>(ArgumentVisitor<R> visitor) {
return visitor.visitInt(this);
}
}
// The same for StringArgument.Introspection
Building an introspection helper object
We need to gather a lot of information about things out there: the fields of the data class, the standard identifiers that we will use when generating the code, etc.
A good pattern is to create a class with all the information the macro needs and then pass it around to our methods as a single argument instead of all those separate things.
We will call this class IntrospectionData:
Future<IntrospectionData> _introspect(
ClassDeclaration clazz,
MemberDeclarationBuilder builder,
) async {
final fields = await builder.introspectFields(clazz);
final ids = await ResolvedIdentifiers.resolve(builder);
final arguments = await _fieldsToArguments(fields, builder);
return IntrospectionData(
arguments: arguments,
clazz: clazz,
fields: fields,
ids: ids,
);
}This begins with the introspection of all the fields in the data class. It takes a lot of work to do it right. This is so common and tedious that I extracted this job to macro_util package. In the args macro, we only need this one line to get a map from field names to detailed information about each of them:
final fields = await builder.introspectFields(clazz);Let’s dive into how this extension method works, so you can safely forget it and just use the package.
Iterating the fields
If you have a ClassDeclaration, you can iterate its fields like this:
final List<FieldDeclaration> fields = await builder.fieldsOf(type);
for (final field in fields) {
// Do something.
}FieldDeclaration
This class contains the information about a field that the builder gives you out of the box. Generally, it’s everything a field is declared with. It has properties like hasConst, hasFinal, hasStatic, hasInitializer, etc. The most important for us is that it has type property that returns TypeAnnotation object.
TypeAnnotation
This is the base class for whatever is known about the type of a field without a dedicated deeper look. It’s abstract, and what you get depends on what the field declaration looks like.
When you introspect a field likeFoo foo;
you get NamedTypeAnnotation which holds the reference to Foo without knowing what it is.
When you introspect a field likevar a = 1;
you get OmittedTypeAnnotation which contains nothing useful but is a handle to infer the type if you are at the third phase.
There are many more subclasses for exotic declarations likefinal (a, b) = getRecord();
which we don’t care about.
We just make sure that every field’s type is NamedTypeAnnotation and break otherwise.
NamedTypeAnnotation
The most important thing that NamedTypeAnnotation adds to TypeAnnotation is the identifier property. For the declaration ofFoo foo;
it returns the Identifier object.
Identifier
Objects of this class represent each identifier in the code. Basically, it’s a name and the implication that it will be resolved to something.
In the declaration ofFoo foo;
the namedTypeAnnotation.identifier is “Foo”, the name and the idea that there’s something it points to.
For code likeprint();
the first thing is the identifier print that refers to the function in the core library.
You get the idea of what an identifier is.
So when we see a NamedTypeAnnotation with an Identifier with a name of “String” or “int”, we’ve found what we wanted, at least for our first version.
There’s more to it if a typedef is involved, and we will support it later. For now, let’s stop here. To recap:
Again, macro_util package does this for you.
Resolving the identifiers
Remember the trick we had to do in the first article to reference the print function? We first needed to resolve the identifier print from the core library and only then could we use it in the code we generate.
Here we need to do more of the same. Our generated code must reference ArgParser, List, String, and int classes. It’s easiest to group all those identifiers into a single structure:
class Libraries {
static final arg_parser = Uri.parse('package:args/src/arg_parser.dart');
static final core = Uri.parse('dart:core');
}
class ResolvedIdentifiers {
ResolvedIdentifiers({
required this.ArgParser,
required this.int,
required this.List,
required this.String,
});
final Identifier ArgParser;
final Identifier int;
final Identifier List;
final Identifier String;
static Future<ResolvedIdentifiers> resolve(
MemberDeclarationBuilder builder,
) async {
final (
ArgParser,
int,
List,
String,
) = await (
builder.resolveIdentifier(Libraries.arg_parser, 'ArgParser'),
builder.resolveIdentifier(Libraries.core, 'int'),
builder.resolveIdentifier(Libraries.core, 'List'),
builder.resolveIdentifier(Libraries.core, 'String'),
).wait;
return ResolvedIdentifiers(
ArgParser: ArgParser,
int: int,
List: List,
String: String,
);
}
}Ideally, we want to use public libraries when resolving identifiers, so we should use package:args/args.dart for ArgParser. However, there is a bug preventing this, so here we must use the private library:'package:args/src/arg_parser.dart'
You may look at this long class ResolvedIdentifiers and wonder if we really need all of this. It has the fields, the constructor that lists all of them, and the same operation for all the fields. Does this use case ring a bell? If only…
Right! Let’s apply what we are learning right now to simplify what we are doing right now!
I made a macro that can simplify it down to something like this:
@ResolveIdentifiers()
class ResolvedIdentifiers {
@Resolve('package:args/args.dart')
final Identifier ArgParser;
final Identifier int;
final Identifier List;
final Identifier String;
}
// ...
final ids = ResolvedIdentifiers.resolve(builder);It’s in the same macro_util package. But it has a problem. While we can make a dictionary for core types like int, List, and String, anything from a custom package like ArgParser needs an explicit link to the package, and a macro cannot yet read annotations on fields.
So while you can use @ResolveIdentifiers() macro for trivial things, we can’t use it now. We must get back to the full redundant code of ResolvedIdentifiers class. But I’m glad it will be so much simpler in the future.
Turning the fields into the Argument objects
To make our visitor classes work, we need to create StringArgument and IntArgument objects:
Map<String, Argument> _fieldsToArguments(
Map<String, FieldIntrospectionData> fields,
DeclarationBuilder builder,
) {
return {
for (final entry in fields.entries)
entry.key: _fieldToArgument(
entry.value as ResolvedFieldIntrospectionData,
builder: builder,
),
};
}
Argument _fieldToArgument(
ResolvedFieldIntrospectionData fieldIntr, {
required DeclarationBuilder builder,
}) {
final typeDecl = fieldIntr.deAliasedTypeDeclaration;
final optionName = _camelToKebabCase(fieldIntr.name);
final typeName = typeDecl.identifier.name;
switch (typeName) {
case 'int':
return IntArgument(
intr: fieldIntr,
optionName: optionName,
);
case 'String':
return StringArgument(
intr: fieldIntr,
optionName: optionName,
);
}
throw Exception();
}Augmenting the parser
Now we need to add a lot of code to the parser class. We start with this:
void _augmentParser(
MemberDeclarationBuilder builder,
IntrospectionData intr,
) {
final parserName = _getParserName(intr.clazz);
builder.declareInLibrary(
DeclarationCode.fromParts([
//
'augment class $parserName {\n',
' final parser = ', intr.ids.ArgParser, '();\n',
..._getConstructor(intr.clazz),
...AddOptionsGenerator(intr).generate(),
...ParseGenerator(intr).generate(),
'}\n',
]),
);
}Here we have a few methods each returning a list of code parts to later be glued into the generated code. Remember from the first article that code parts are mixtures of string literals and identifiers.
First, we declare parser public field to hold the instance of the standard parser which we will populate soon. Then some more interesting things.
Populating the parser with options
AddOptionsGenerator is a visitor for arguments. It creates a private _addOptions() method:
List<Object> _getConstructor(ClassDeclaration clazz) {
final parserName = _getParserName(clazz);
return [
//
parserName, '() {\n',
' _addOptions();\n',
'}\n',
];
}
class AddOptionsGenerator extends ArgumentVisitor<List<Object>> {
AddOptionsGenerator(this.intr);
final IntrospectionData intr;
List<Object> generate() {
return [
//
'void _addOptions() {\n',
for (final argument in intr.arguments.values)
...[...argument.accept(this), '\n'],
'}\n',
];
}
@override
List<Object> visitInt(IntArgument argument) =>
_visitStringInt(argument);
@override
List<Object> visitString(StringArgument argument) =>
_visitStringInt(argument);
List<Object> _visitStringInt(Argument argument) {
return [
//
'parser.addOption(\n',
' ${jsonEncode(argument.optionName)},\n',
' mandatory: true,\n',
');\n',
];
}
}This _addOptions() method is then called in the constructor of this parser class.
Filling the data class with the parsed data
This is what ParseGenerator class does, another argument visitor:
class ParseGenerator extends ArgumentVisitor<List<Object>> {
ParseGenerator(this.intr);
final IntrospectionData intr;
List<Object> generate() {
final name = intr.clazz.identifier.name;
final ids = intr.ids;
return [
//
'$name parse(', ids.List, '<', ids.String, '> argv) {\n',
' final wrapped = parser.parse(argv);\n',
' return $name(\n',
for (final argument in intr.arguments.values) ...[
...argument.accept(this),
',\n',
],
' );\n',
'}\n',
];
}
@override
List<Object> visitInt(IntArgument argument) {
return [
argument.intr.name,
': ',
intr.ids.int,
'.parse(wrapped.option(${jsonEncode(argument.optionName)})!)',
];
}
@override
List<Object> visitString(StringArgument argument) {
return [
argument.intr.name,
': wrapped.option(${jsonEncode(argument.optionName)})!',
];
}
}At this point, you can use the macro!
Create main.dart:
import 'macro.dart';
@Args()
class HelloArgs {
final String name;
final int count;
}
void main(List<String> argv) {
final parser = HelloArgsParser(); // Generated class.
final HelloArgs args = parser.parse(argv);
for (int n = 0; n < args.count; n++) {
print('Hello, ${args.name}!');
}
}And run it:
$ dart run --enable-experiment=macros lib/main.dart --name=Alexey --count=3
Hello, Alexey!
Hello, Alexey!
Hello, Alexey!And let the crazy storm of edge cases and features begin!
Handling errors
Our macro only handles the successful scenario throwing cryptic errors if it doesn’t like anything. Try adding a field of an unsupported type to the data class:
@Args()
class HelloArgs {
final Map map;
}_fieldToArgument function will throw an exception which will break the macro. That in turn will raise more errors because the class no longer has a constructor.
Failing gracefully is essential to the art of macros because users have no idea what went wrong inside of it. This involves two habits:
- Never throw an exception in a macro, report compiler diagnostic messages instead.
- Generate syntactically correct code even when reporting an error, to avoid secondary errors.
Reporting compiler diagnostic messages
This is how you report a diagnostic message:
builder.report(
Diagnostic(
DiagnosticMessage(
'My error',
target: fieldDeclaration.asDiagnosticTarget,
),
Severity.error,
),
);This will break the compilation and show the message to the user. It looks like built-in compiler errors:
$ dart run --enable-experiment=macros lib/min.dart --name=Alexey --count=3
lib/min.dart:5:16: Error: My error
final String name;
^
lib/min.dart:6:13: Error: My error
final int count;
^macro_util package has an extension method to make it shorter:
builder.reportError(
'My error',
target: fieldDeclaration.asDiagnosticTarget,
);The cases we want to handle
- Show errors for unsupported types.
- Show errors for omitted types.
- Show errors for fields of classes shadowing
intandString. - Show errors for _private fields.
- Show errors for fields with initializers.
An improved version that handles all of that is here. I suggest you diff these two branches to see what has changed:
Let’s go through the changes.
Silencing errors for unsupported types
We have a constructor which takes every public field as a named parameter. If for any reason we can’t parse a value for a field, we still must pass something to the constructor. Otherwise, the constructor call will have a syntax error, and we must not show that to the user.
We trick the compiler by making this variable:
static var _silenceUninitializedError;Its type is dynamic, so it can be passed to any constructor parameter without a compile error. Sure it would break if run, but we never get there because for an invalid field we produce an error diagnostic that prevents the execution.
We could also pass something like null as dynamic instead, but this would require to resolve the identifier dynamic, and I didn’t want to do that.
Next, how do we know to pass this dummy value to a constructor if we only have StringArgument and IntArgument? We need to create a dedicated class InvalidTypeArgument (and also make a base valid argument class above int and String):
class InvalidTypeArgument extends Argument {
InvalidTypeArgument({
required super.intr,
}) : super(
optionName: '',
);
@override
R accept<R>(ArgumentVisitor<R> visitor) {
return visitor.visitInvalidType(this);
}
}Then we add its handlers to both visitors. When adding options to the parser, we don’t need to do anything:
class AddOptionsGenerator extends ArgumentVisitor<List<Object>> {
// ...
@override
List<Object> visitInvalidType(InvalidTypeArgument argument) {
return const [];
}
}When constructing a data object, we pass our silencer to such a field:
class ParseGenerator extends ArgumentVisitor<List<Object>> {
// ...
@override
List<Object> visitInvalidType(InvalidTypeArgument argument) {
return [
argument.intr.name,
': _silenceUninitializedError',
];
}
}Then we need to produce an instance of InvalidTypeArgument on any error, and they are plenty:
Argument _fieldToArgument(
FieldIntrospectionData fieldIntr, {
required DeclarationBuilder builder,
}) async {
final field = fieldIntr.fieldDeclaration;
final target = field.asDiagnosticTarget;
final type = field.type;
void reportError(String message) {
builder.reportError(message, target: target);
}
void unsupportedType() {
if (type is OmittedTypeAnnotation) {
reportError('An explicitly declared type is required here.');
return;
}
reportError('The only allowed types are: String, int.');
}
if (fieldIntr is! ResolvedFieldIntrospectionData) {
unsupportedType();
return InvalidTypeArgument(intr: fieldIntr);
}
final typeDecl = fieldIntr.deAliasedTypeDeclaration;
final optionName = _camelToKebabCase(fieldIntr.name);
if (field.hasInitializer) {
reportError('Initializers are not allowed for argument fields.');
return InvalidTypeArgument(intr: fieldIntr);
}
if (typeDecl.library.uri != Libraries.core) {
unsupportedType();
return InvalidTypeArgument(intr: fieldIntr);
}
final typeName = typeDecl.identifier.name;
switch (typeName) {
case 'int':
return IntArgument(
intr: fieldIntr,
optionName: optionName,
);
case 'String':
return StringArgument(
intr: fieldIntr,
optionName: optionName,
);
}
unsupportedType();
return InvalidTypeArgument(intr: fieldIntr);
}This code gracefully handles all errors except private fields. We will get to them shortly after a quick tour of the type declarations which we have just used.
Checking TypeDeclaration of a field
We should tell a field of a built-in int type from a shadowing one:
class int {}
@Args()
class HelloArgs {
final int count; // Not the built-in 'int'.
}This means it’s not enough to just check the name of the type. We should also learn what library it’s from. If it’s dart:core, we’re good, otherwise we need to show an error. The code above does that because macro_util package resolves that for you. Let’s see how it’s done.
Remember the memo on the types? There’s more to it:
TypeDeclaration
Remember we had an Identifier that held ‘String’ or ‘int’. It didn’t hold just that name, it also was a handle that could pull the actual type declaration from the library where it was declared. Here is how to pull it:
final typeDecl = await builder.typeDeclarationOf(
namedTypeAnnotation.identifier,
);This will return TypeDeclaration. It’s different from TypeAnnotation which was just a handle. Now we have the actual type as it was declared in the library. This class is abstract, and you get a subclass depending on what the type is.
In the simplest case, you get ClassDeclaration. Remember, this is the class we started this journey from when the macro was applied to our class. This time, we get the ClassDeclaration for a field. If you repeat this long journey recursively, you can introspect almost everything in the program.
Another subclass that works for us is EnumDeclaration. Remember, we want to support enums for the fields so the user can only supply a value from a fixed set for a command-line option. But enums are complicated, so let’s make simple types work first.
We can also get TypeAliasDeclaration for a typedef. In this case, we need to look at the aliased type and repeat the process because we may still end up at a type we support. So the whole de-aliasing looks like this:
TypeDeclaration typeDecl = await builder.typeDeclarationOf(
namedTypeAnnotation.identifier,
);
while (typeDecl is TypeAliasDeclaration) {
final aliasedType = typeDecl.aliasedType;
if (aliasedType is! NamedTypeAnnotation) {
// Error. The typedef has led us to something weird like
// a record: final (a, b) = getRecord();
// or a function: final void Function() function;
throw Exception('...');
}
typeDecl = await builder.typeDeclarationOf(aliasedType.identifier);
}This is why the field that we were checking earlier was called FieldIntrospectionData.deAliasedTypeDeclaration.
Library
When you have TypeDeclaration, it links to the library where the declaration was loaded from. And this is how we can finally tell a built-in int from a custom shadowing one.
All of this is done by macro_util package. If everything was resolved fine, you get ResolvedFieldIntrospectionData with de-aliased type. Otherwise, you get FieldIntrospectionData with just TypeAnnotation.
This is also why we named the based valid argument type ResolvedTypeArgument.
Handling private fields
What if our data class has a private field?
@Args()
class HelloArgs {
final String _name;
}In Dart, named parameters to function can’t begin with an underscore, and so the constructor macro that we used turns them into positional parameters and not named ones.
Private fields make no sense for a data class, so we need to show an error for them.
First, we need to produce an InvalidTypeArgument for such a field and show a diagnostic:
Argument _fieldToArgument(
FieldIntrospectionData fieldIntr, {
required DeclarationBuilder builder,
}) {
final field = fieldIntr.fieldDeclaration;
final target = field.asDiagnosticTarget;
if (fieldIntr.name.contains('_')) {
builder.reportError(
'An argument field name cannot contain an underscore.',
target: target,
);
return InvalidTypeArgument(intr: fieldIntr);
}
// ...Then we need to separate the named parameters from the positional ones. We pass our silencer to all positional parameters:
class ParseGenerator extends ArgumentVisitor<List<Object>> {
ParseGenerator(this.intr);
final IntrospectionData intr;
List<Object> generate() {
final name = intr.clazz.identifier.name;
final ids = intr.ids;
final arguments = intr.arguments.values.where(
(a) =>
a.intr.constructorHandling ==
FieldConstructorHandling.namedOrPositional,
);
return [
//
name, ' parse(', ids.List, '<', ids.String, '> argv) {\n',
' final wrapped = parser.parse(argv);\n',
' return $name(\n',
for (final param in _getPositionalParams()) ...[...param, ',\n'],
for (final argument in arguments) ...[
...argument.accept(this),
',\n',
],
' );\n',
'}\n',
];
}
List<List<Object>> _getPositionalParams() {
final result = <List<Object>>[];
final fields = intr.fields.values.where(
(f) => f.constructorHandling == FieldConstructorHandling.positional,
);
for (final _ in fields) {
result.add([
'_silenceUninitializedError',
]);
}
return result;
}Here we use the convenience getterFieldIntrospectionData.constructorHandling
It’s either of the two:
positionalif the field begins with an underscore and thus can only be a positional parameter to a constructor.namedOrPositionalotherwise.
With this change, the macro can finally handle all errors gracefully. It only reports its own diagnostics and no other compile errors.
Supporting lists and inspecting type parameters
Our macro will support both lists and sets of int and String. For this, we will expand the argument types like this:
The full code that implements that is here. Diff it against the previous branch to see what has changed:
Most of the changes just add argument classes and visitors, we will skip those. Here is what you need to know from the macro perspective.
When you bump into a List or a Set, you check its typeArguments. That’s only available if you are dealing with NamedTypeAnnotation, so you add a check for that and discard other cases as invalid arguments.
When you get the type argument, you run the whole process of de-aliasing and checking types on it, and you are done:
Future<Argument> _fieldToArgument(
FieldIntrospectionData fieldIntr, {
required DeclarationBuilder builder,
}) async {
// ...
if (type is! NamedTypeAnnotation) {
unsupportedType();
return InvalidTypeArgument(intr: fieldIntr);
}
// ...
switch (typeName) {
// ...
case 'List':
case 'Set':
final paramType = type.typeArguments.firstOrNull;
if (paramType == null) {
reportError(
'A $typeName requires a type parameter: '
'$typeName<String>, $typeName<int>.',
);
return InvalidTypeArgument(intr: fieldIntr);
}
if (paramType.isNullable) {
reportError(
'A $typeName type parameter must be non-nullable because each '
'element is either parsed successfully or breaks the execution.',
);
return InvalidTypeArgument(intr: fieldIntr);
}
if (paramType is! NamedTypeAnnotation) {
unsupportedType();
return InvalidTypeArgument(intr: fieldIntr);
}
final paramTypeDecl = await builder.deAliasedTypeDeclarationOf(paramType);
if (paramTypeDecl.library.uri != Libraries.core) {
unsupportedType();
return InvalidTypeArgument(intr: fieldIntr);
}
switch (paramTypeDecl.identifier.name) {
case 'int':
return IterableIntArgument(
intr: fieldIntr,
iterableType: IterableType.values.byName(typeName.toLowerCase()),
optionName: optionName,
);
case 'String':
return IterableStringArgument(
intr: fieldIntr,
iterableType: IterableType.values.byName(typeName.toLowerCase()),
optionName: optionName,
);
}
// ...Supporting Enums
An enum argument allows a user to specify only a value from the given set:
@Args()
class HelloArgs {
final Fruit fruit;
}
enum Fruit { apple, banana, mango, orange }When you introspect such a field, you are supposed to get an EnumDeclaration instead of ClassDeclaration. However, it’s not implemented as of writing. You still get ClassDeclaration. You should either wait until the Dart team implements that or use a workaround as I did.
This is a huge red flag because once the API changes to actually return EnumDelcaration as expected, your workaround with classes will suddenly break. So only use this as a playground.
The workaround is to check if the class you have implements the built-in Enum interface. A macro can’t do this with just the introspection routines we saw earlier, so let’s take a look at another one and add it to the memo:
StaticType
This is the representation of a type that allows it to be compared with other StaticType objects for inheritance. It’s hard to say why they needed another class and why we can’t just compare ClassDeclaration objects for inheritance. Likely a StaticType takes some other kind of a more expensive introspection. However, it does not replace what we had before.
You get a StaticType by asking the builder to resolve NamedTypeAnnotationCode, which is just a container for an identifier of the type name:
final staticType = await builder.resolve(
NamedTypeAnnotationCode(name: namedTypeAnnotation.identifier),
);
if (await staticType.isSubtypeOf(enumStaticType)) {
// ...
}Now that you get the idea take a look at the version that supports enums. Diff it against the previous version:
Let’s take a closer look at the changes.
Resolving static types
macro_utils package takes care of resolving a StaticType object for each field we introspect. We only need to resolve it for the built-in Enum class to compare other fields to it.
We take the same approach as with ResolvedIdentifiers class that contains all identifiers the macro will ever need. We store static types in this class, even though we only have one:
import 'package:macros/macros.dart';
import 'resolved_identifiers.dart';
class StaticTypes {
StaticTypes({
required this.Enum,
});
final StaticType Enum;
static Future<StaticTypes> resolve(
MemberDeclarationBuilder builder,
ResolvedIdentifiers ids,
) async {
final Enum = await builder.resolve(NamedTypeAnnotationCode(name: ids.Enum));
return StaticTypes(
Enum: Enum,
);
}
}We add this object to our topmost introspection data class:
class IntrospectionData {
IntrospectionData({
required this.arguments,
required this.clazz,
required this.fields,
required this.ids,
required this.staticTypes, // NEW
});
final Map<String, Argument> arguments;
final ClassDeclaration clazz;
final Map<String, FieldIntrospectionData> fields;
final ResolvedIdentifiers ids;
final StaticTypes staticTypes; // NEW
}And then we fill it with the resolved data:
Future<IntrospectionData> _introspect(
ClassDeclaration clazz,
MemberDeclarationBuilder builder,
) async {
final ids = await ResolvedIdentifiers.resolve(builder);
final (fields, staticTypes) = await (
builder.introspectFields(clazz),
StaticTypes.resolve(builder, ids), // NEW
).wait;
final arguments = await _fieldsToArguments(
fields,
builder: builder,
staticTypes: staticTypes, // NEW
);
return IntrospectionData(
arguments: arguments,
clazz: clazz,
fields: fields,
ids: ids,
staticTypes: staticTypes, // NEW
);
}Creating enum arguments
When we find a field of a non-core type, it can now potentially be an enum, so check for that:
Future<Argument> _fieldToArgument(
FieldIntrospectionData fieldIntr, {
required DeclarationBuilder builder,
required StaticTypes staticTypes,
}) async {
// ...
if (typeDecl.library.uri != Libraries.core) {
if (await fieldIntr.nonNullableStaticType.isSubtypeOf(staticTypes.Enum)) {
return EnumArgument(
enumIntr:
await builder.introspectEnum(fieldIntr.deAliasedTypeDeclaration),
intr: fieldIntr,
optionName: optionName,
);
}
unsupportedType();
return InvalidTypeArgument(intr: fieldIntr);
}
// ...
switch (typeName) {
// ...
case 'List':
case 'Set':
// ...
if (paramTypeDecl.library.uri != Libraries.core) {
final paramStaticType = await builder.resolve(paramType.code);
if (await paramStaticType.isSubtypeOf(staticTypes.Enum)) {
return IterableEnumArgument(
enumIntr: await builder.introspectEnum(paramTypeDecl),
intr: fieldIntr,
iterableType: IterableType.values.byName(typeName.toLowerCase()),
optionName: optionName,
);
}
unsupportedType();
return InvalidTypeArgument(intr: fieldIntr);
}
// ...Introspecting the enum values
In the code above we used our extension method on a builder:
class EnumIntrospectionData {
EnumIntrospectionData({
required this.deAliasedTypeDeclaration,
required this.values,
});
final TypeDeclaration deAliasedTypeDeclaration;
final List<EnumConstantIntrospectionData> values;
}
class EnumConstantIntrospectionData {
const EnumConstantIntrospectionData({
required this.name,
});
final String name;
}
extension EnumIntrospectionExtension on DeclarationBuilder {
Future<EnumIntrospectionData> introspectEnum(
TypeDeclaration deAliasedTypeDeclaration,
) async {
final fields = await fieldsOf(deAliasedTypeDeclaration);
final values = (await Future.wait(fields.map(introspectEnumField)))
.nonNulls
.toList(growable: false);
return EnumIntrospectionData(
deAliasedTypeDeclaration: deAliasedTypeDeclaration,
values: values,
);
}
Future<EnumConstantIntrospectionData?> introspectEnumField(
FieldDeclaration field,
) async {
final type = field.type;
if (type is NamedTypeAnnotation) {
return null;
}
return EnumConstantIntrospectionData(
name: field.identifier.name,
);
}
}Normally, we should use a safe dedicated method builder.valuesOf(enumDeclaration), but we can’t get EnumDeclaration. So we use a naive approach of just taking every field of the class. Luckily, all enum constants are returned as the fields of the class. We only need to skip the values collection, and so we check if a field’s type is NamedTypeAnnotation, which happens to be false for values. This will behave unexpectedly if the enum contains other data, but I’m not that much of a nerd to tailor this temporary workaround any better.
Also, this is not in macro_util because this is not reliable enough.
Anyway, we get EnumIntrospectionData object and pass it to EnumArgument:
class EnumArgument extends ResolvedTypeArgument {
EnumArgument({
required super.intr,
required super.optionName,
required this.enumIntr,
});
final EnumIntrospectionData enumIntr;
@override
R accept<R>(ArgumentVisitor<R> visitor) {
return visitor.visitEnum(this);
}
}Then we use the values when populating the options:
class AddOptionsGenerator extends ArgumentVisitor<List<Object>> {
// ...
@override
List<Object> visitEnum(EnumArgument argument) {
final values =
argument.enumIntr.values.map((v) => v.name).toList(growable: false);
return [
//
'parser.addOption(\n',
' ${jsonEncode(argument.optionName)},\n',
' allowed: ${jsonEncode(values)},\n',
' mandatory: true,\n',
');\n',
];
}
// ...Parsing the data is easy:
class ParseGenerator extends ArgumentVisitor<List<Object>> {
// ...
@override
List<Object> visitEnum(EnumArgument argument) {
final valueGetter = _getOptionValueGetter(argument);
return [
argument.intr.name,
': ',
argument.intr.deAliasedTypeDeclaration.identifier,
'.values.byName($valueGetter!)',
];
}
// ...It’s much the same for IterableEnumArgument.
Field initializers and nullable fields
We want our parser to support default values for arguments. A great idiomatic expression of that is a field with a default value:
@Args()
class HelloArgs {
final int count = 1;
}According to the specification of the augmentation feature, we should be able to augment initializers like this:
augment class HelloArgs {
augment final int count = _parse_count(augmented);
static int _parse_count(int defaultValue) {
// Handle the parsing.
}
}However, as of writing this is not yet implemented and we can’t augment initializers yet. So instead of waiting for the proper solution, we will use a workaround again.
The next best thing is to give up final for such fields, instantiate an object with required fields only, and then overwrite those for which the default values were overridden.
This isn’t as clean as it should be because we can’t have a const constructor this way, so this should be changed once we can augment initializers.
However, there’s a bigger problem. We need to access the default values to feed them to ArgParser.addOption(defaultsTo: ...) so they appear in the usage help text. We could do this if we could read the field initializer as a Code object, but we can’t do that. The API only allows us to check if there is an initializer but not to read it. Upvote this feature request if you think that reading the initializer is useful.
So the workaround is the following:
- Create a second constructor with only the fields without defaults.
- Instantiate a mock object with this constructor so that the fields with initializers retain their default values. Put dummy values in the required fields.
- Read that object’s fields and feed them to
addOption(defaultsTo: ...)
This is also a good time to support nullable fields because an initializer and nullability are both ways to handle a missing value, and it’s easier to implement them in one stride.
Here is the version that handles default values and nullable options. Diff it against the previous version to see what has changed:
Let’s walk through the changes.
Adding another constructor
Let’s add MockDataObjectGenerator class and let it do the job:
Future<void> _declareConstructors(
ClassDeclaration clazz,
MemberDeclarationBuilder builder,
) async {
await Future.wait([
//
const Constructor().buildDeclarationsForClass(clazz, builder),
MockDataObjectGenerator.createMockConstructor(clazz, builder), // NEW
]);
}const _constructorName = 'withDefaults';
class MockDataObjectGenerator {
/// Creates the constructor on the data class which does not have
/// parameters for fields that have initializers
/// thus keeping them from being overwritten.
static Future<void> createMockConstructor(
ClassDeclaration clazz,
MemberDeclarationBuilder builder,
) async {
return const Constructor(
name: _constructorName,
skipInitialized: true,
).buildDeclarationsForClass(clazz, builder);
}
}The @Constructor macro we used has optional parameters to do just what we need. We can name the second constructor withDefaults and not add the fields that have initializers. This way, they are not overwritten.
Creating the mock object
Let’s turn this new class into a third ArgumentVisitor:
/// Generates code to create an instance of the data class that
/// does not overwrite the fields that have initializers.
///
/// This mock object is then used as the source of data
/// for options that were not passed.
///
/// The required fields are filled with dummy values of the respective types
/// since they will never be used because the actual values for them
/// will be parsed when constructing the end-instance of the data class.
class MockDataObjectGenerator extends ArgumentVisitor<List<Object>>
with PositionalParamGenerator {
// ...
List<Object> generate() {
final name = intr.clazz.identifier.name;
final arguments = intr.arguments.values.where(
(a) =>
a.intr.constructorOptionality ==
FieldConstructorOptionality.required &&
a.intr.constructorHandling ==
FieldConstructorHandling.namedOrPositional,
);
return [
'static final $fieldName = $name.$_constructorName(\n',
for (final param in getPositionalParams()) ...[...param, ',\n'],
for (final parts in arguments.map((argument) => argument.accept(this)))
...[...parts, ',\n'],
');\n',
];
}
@override
List<Object> visitEnum(EnumArgument argument) {
return [
argument.intr.name,
': ',
argument.intr.deAliasedTypeDeclaration.identifier,
'.values.first',
];
}
@override
List<Object> visitInt(IntArgument argument) {
return [
argument.intr.name,
': 0',
];
}
@override
List<Object> visitInvalidType(InvalidTypeArgument argument) {
return [
argument.intr.name,
': _silenceUninitializedError',
];
}
@override
List<Object> visitIterableEnum(IterableEnumArgument argument) =>
_visitIterable(argument);
@override
List<Object> visitIterableInt(IterableIntArgument argument) =>
_visitIterable(argument);
@override
List<Object> visitIterableString(IterableStringArgument argument) =>
_visitIterable(argument);
@override
List<Object> visitString(StringArgument argument) {
return [
argument.intr.name,
': ""',
];
}
List<Object> _visitIterable(IterableArgument argument) {
switch (argument.iterableType) {
case IterableType.list:
return [
argument.intr.name,
': const []',
];
case IterableType.set:
return [
argument.intr.name,
': const {}',
];
}
}
}Note that we now need to get positional parameters in two of the argument visitors, so we extracted that method into PositionalParamGenerator mixin.
So much effort just to produce valid code to not distract the user with irrelevant compile errors and let them focus on our diagnostics. This is one of the most important things in macros.
After that, we call this from our macro:
void _augmentParser(
MemberDeclarationBuilder builder,
IntrospectionData intr,
) {
final parserName = _getParserName(intr.clazz);
builder.declareInLibrary(
DeclarationCode.fromParts([
//
'augment class $parserName {\n',
' final parser = ', intr.ids.ArgParser, '();\n',
' static var _silenceUninitializedError;\n',
...MockDataObjectGenerator(intr).generate(), // NEW
..._getConstructor(intr.clazz),
...AddOptionsGenerator(intr).generate(),
...ParseGenerator(intr).generate(),
'}\n',
]),
);
}Handling the default values
First, let’s tell the standard ArgParser about the default values. Every method that adds an option needs to be changed like this:
class AddOptionsGenerator extends ArgumentVisitor<List<Object>> {
// ...
List<Object> _visitStringInt(Argument argument) {
final field = argument.intr.fieldDeclaration;
return [
//
'parser.addOption(\n',
' "${argument.optionName}",\n',
if (field.hasInitializer) ...[ // CHANGED
' defaultsTo: ', // CHANGED
MockDataObjectGenerator.fieldName, // CHANGED
'.', // CHANGED
argument.intr.name, // CHANGED
'.toString()', // CHANGED
',\n', // CHANGED
] else if (!field.type.isNullable) // CHANGED
' mandatory: true,\n', // CHANGED
');\n',
];
}
// ...Then every parsing method should account for nulls:
class ParseGenerator extends ArgumentVisitor<List<Object>>
with PositionalParamGenerator {
// ...
@override
List<Object> visitInt(IntArgument argument) {
final valueGetter = _getOptionValueGetter(argument);
return [
argument.intr.name,
': ',
if (argument.intr.fieldDeclaration.type.isNullable) // CHANGED
'$valueGetter == null ? null : ', // CHANGED
intr.ids.int,
'.parse($valueGetter!)',
];
}
// ...We don’t need to check if a value was passed to overwrite the default value in the data object. The standard ArgParser will just return either the explicitly passed value or the default we fed to it earlier.
Better error diagnostics
We now can have more than one error per field. For example, a lot of things are wrong in this example:
@Args()
class HelloArgs {
final List<int?>? list = [];
}For this code, we need to show three error diagnostics at once:
- A field with an initializer cannot be final because it needs to be overwritten when parsing the argument.
- A List cannot be nullable because it is just empty when no options with this name are passed.
- A List type parameter must be non-nullable because each element is either parsed successfully or breaks the execution.
If we break at the first error, the user will fix it just to be puzzled with another one, and then another one. They will think the errors are infinite and the macro is poor and unpredictable in its demand. Therefore, the art of macros includes showing every wrong thing upfront. So as the user will be fixing the code toward this
@Args()
class HelloArgs {
List<int> list = [];
}the errors will disappear one by one, which is very satisfying.
To enable this, we need to:
- Make
isValidlocal flag when parsing an argument. - When anything is wrong, report a diagnostic and set the flag to
false. - In the end, check the flag to produce
InvalidTypeArgumentor a valid one.
Here are some changes, check the full diff because there are many of them:
bool isValid = true;
if (field.hasInitializer && field.hasFinal) {
reportError(
'A field with an initializer cannot be final '
'because it needs to be overwritten when parsing the argument.',
);
isValid = false;
}
// ...A good macro evolves to heavy branching regarding the errors.
Supporting help messages
When you run a command with --help flag, it should print the usage. For that, we need some help text for each field. How do we store that?
Idiomatically, a doc comment is the best:
@Args()
class HelloArgs {
/// Your name to print.
final String name;
/// How many times to print your name.
final int count = 1;
}$ dart run --enable-experiment=macros lib/min.dart --help
Usage: [arguments]
--name (mandatory) Your name to print.
--count (defaults to "1") How many times to print your name.
-h, --help Print this usage information.Unfortunately, a macro can’t read doc comments on a field. Upvote this feature request if you find this beautiful.
The next best thing is to convey this help text in an annotation on a field:
@Args()
class HelloArgs {
@Arg(help: 'Your name to print.')
final String name;
@Arg(help: 'How many times to print your name.')
final int count = 1;
}Not as beautiful, but there should still be such annotation in the future to support overriding an option name, restricting a range of an int, etc.
However, a macro cannot yet read annotations in code.
The workaround I came up with is to store the help text in a separate static field:
@Args()
class HelloArgs {
final String name;
static const _nameHelp = 'Your name to print.';
int count = 1;
static const _countHelp = 'How many times to print your name.';
}Here is the version that supports it. Diff it against the previous one:
The change is simple:
- Ignore static fields (the previous version used to break on them).
- Reference the constant field’s name directly in the code.
Inferring types
It’s tempting to support this:
@Args()
class HelloArgs {
final String name;
var count = 1; // Inferred type
}It will be even more tempting when the augmentation feature is fully released and we can make the field final:
@Args()
class HelloArgs {
final String name;
final count = 1; // Inferred type
}Inferring types is only possible in the third phase of a macro application. Unfortunately, while in the second phase any macro can write global code like this
builder.declareInLibrary(
DeclarationCode.fromParts([
'augment class $parserName {',
// ...
]),
);in the third phase a macro is limited to replacing method bodies and field initializers in the class it was applied to.
This means we can take two approaches:
- Create a new macro named something like
@ArgsParserand have our main macro apply it to the parser class we generate. We should then shift most of the work to it. The spec allows that, but it’s not implemented yet. The downside is that this macro will be an exported symbol in the library without a meaningful thing a user can do with it. - Give up the separate parser class and convert all our work into static methods on the data class:
@Args()
class HelloArgs {
final String name;
final int count;
}
void main(List<String> argv) {
final args = HelloArgs.parse(argv);
}This can be done right now, but I don’t like the mixed concerns here. The clients of the data object should not have the access to parse method that created it.
I have a third idea in mind: take the first approach with two macros but reuse the same @Args macro and make it do a different thing when it’s applied to a parser class. This is messy, but the API to the user is the cleanest possible.
I’m not sure of it yet and should see what the final macro API will look like. For now, we will leave this thing unsolved and force the user to specify explicit types.
If you have a cleaner case and want to infer a type, it’s done like this:
final typeAnnotation = await builder.inferType(omittedTypeAnnotation);Testing macros
Unit tests
The first approach you can take is unit testing:
- Mock
ClassDeclarationand a builder for the given phase. - Instantiate the macro as a regular object and call its interface method.
- Verify that the invocation resulted in all the necessary calls to the builder methods to produce the code you want.
This takes a lot of work since the mock builder must resolve a lot of things: identifiers, StaticType, etc. Perhaps we should wait for a suite from the Dart team to simplify that.
Integration tests
In the case of @Args macro, these are easy:
- Write various programs that expect command line arguments.
- Run them.
- Check the output and error messages.
Check out the main branch of the args_macro package. Let’s see how the tests are made there.
By the way, that version adds all the neat things that were still missing: double arguments, bool flags, and better readable error messages. They don’t have much to do with macros, so we will skip them here.
Check out the diff:
All tests are in args_macro_test.dart
In example/lib, there are many sample programs that we expect to either run successfully or break.
Testing the runtime errors
main.dart is a crazy-long one with every feature and the 30 different arguments the package supports.
It can be run successfully or with errors if the command line arguments are messed up. The tests for it look like this:
const _executable = 'lib/main.dart';
const _experiments = ['macros'];
const _workingDirectory = 'example';
const _usageExitCode = 64;
// ...
group('int', () {
test('missing required', () async {
final arguments = {..._arguments};
arguments.remove(_requiredInt);
final result = await dartRun(
[_executable, ...arguments.values],
experiments: _experiments,
workingDirectory: _workingDirectory,
expectedExitCode: _usageExitCode,
);
expect(
result.stderr,
'Option "$_requiredInt" is mandatory.\n\n$_helpOutput',
);
});
// ...These tests are not specific to macros, so we won’t study them. However, they use a helper dartRun function I created to easily run Dart programs and expect things from them. It’s in my test_util package.
Testing the macro error diagnostics
These tests are more interesting:
const _compileErrorExitCode = 254;
// ...
test('error_iterable_nullable', () async {
await dartRun(
['lib/error_iterable_nullable.dart'],
experiments: _experiments,
workingDirectory: _workingDirectory,
expectedExitCode: _compileErrorExitCode,
expectedErrors: const [
ExpectedFileErrors('lib/error_iterable_nullable.dart', [
ExpectedError(
'A List cannot be nullable because it is just empty '
'when no options with this name are passed.',
[7],
),
ExpectedError(
'A Set cannot be nullable because it is just empty '
'when no options with this name are passed.',
[8],
),
]),
],
);
});The same dartRun function allows you to expect specific compile error messages at specific line numbers.
It’s not super-reliable because it does not use the compiler’s machine-readable output but just parses its regular output. It may fail in some edge cases or if the format changes.
A few things to remember:
- A compiler prints up to 10 error messages when you run a program. This means you can test each of your sample programs for up to 9 messages (if you have 10, you don’t know if there was an 11th one discarded). That’s one of the reasons we have many small programs under tests.
- You should use the output of the compiler and not the analyzer’s result. This is because they happen to differ for unstable features like macros are now. See this bug as an example of them differing. This is sad because analyzer’s results are well-structured with Language Server Protocol and are not limited to 10 errors.
Conclusion
This is far from being ready for production. Not only the API will change, but a lot of things are not implemented to the spec yet.
Think of this as an exercise in the ingeniousity of your workarounds.
I enjoyed the time I spent playing with this, although if I knew it would take a month I would rather do something for profit. Well, at least you may have learned something from it.
Never miss a story, follow me on Telegram, which is my primary media. Then also Twitter, LinkedIn, and GitHub.
Recap of the goodies
Dart resources:
- Macro feature specification.
- Augmentation feature specification.
- macros package.
- json package, the
@JsonCodableexample macro. - The macros API.
My packages:
- args_macro — the subject of this article.
- macro_util — field introspection.
- show_augmentation — shows augmentation if your IDE can’t.
- test_util —
dartRunfunction. - common_macros —
@Constructormacro. - enum_map — another good macro example, v0.4+, currently pre-release.
