Writing a Code Generator

In the previous section we’ve seen how to process Xtext files in general (e.g. with Java). If you need late binding, loading models dynamically and interpreting them is a good idea. If you do not need late binding, code generation is a viable option which is widely used to make models executable.

You could of course use Java to do the code generation, but unfortunately Java doesn’t support this task very well. Therefore we use a language specialized on code generation: M2T Xpand.

Xpand and MWE2

Xpand is also part of Eclipse and ships with its own documentation, so we don’t need to explain all the details here, but rather want to point you to the existing documentation. Instead this chapter is about providing some information about how to use Xtext and Xpand together.

The other technology we are going to use here is the Modeling Workflow Engine 2. It is used to describe the generation workflow, that is how and which models should be loaded, how they should be validated post-processed and where and how code generation should take place.

This section is based on the Getting Started section and uses the code generator project which is created by Xtext’s project wizard.

The Empty Generator Project

Previously when we created the projects to develop the domain model language, we had checked the option Create a generator project in the wizard, which as a result created a third project for us in the workspace. If you’ve followed along you should have a project called ‘org.eclipse.xtext.example.domainmodel.generator’ in your workspace. It should look like shown in the following screenshot:

Replacing the Example Model File (1)

The four files shown in the image above are very simple stubs generated for the ‘Hello World’-language the wizard creates as a starting point. We’ll have to adapt them to match the domain model language we have developed in the getting started section. The first thing we should do is open up the example model file (1) and replace its contents with an instance of the domain model language. To do so copy and paste the following into the editor:

package java.lang {
  datatype String
  datatype Boolean
}

package my.entities {
  
  import java.lang.*
  
  entity Session {
    title: String
    isTutorial : Boolean
  }

  entity Conference {
    name : String
    attendees : Person*
    speakers : Speaker*
  }

  entity Person {
    name : String
  }

  entity Speaker extends Person {
    sessions : Session*
  }


The Modeling Workflow Engine File (2)

As mentioned MWE2 is used to define and configure the generation process. Actually, it is much more general and can be useful for a lot of other things as well. Let’s have a look at the MWE2 file as generated by the wizard:

module workflow.DomainmodelGenerator

import org.eclipse.emf.mwe.utils.*

var targetDir = "src-gen"
var fileEncoding = "ISO-8859-1"
var modelPath = "src/model"

Workflow {

component = org.eclipse.xtext.mwe.Reader {
// lookup all resources on the classpath
// useJavaClassPath = true

// or define search scope explicitly
path = modelPath

// this class will be generated by the xtext generator 
register = org.eclipse.xtext.example.DomainmodelStandaloneSetup {}
load = {
slot = "greetings"
type = "Greeting"
}
}

component = org.eclipse.xpand2.Generator {
expand = "templates::Template::main FOREACH greetings"
outlet = {
path = targetDir
}
fileEncoding = fileEncoding
}
}

The first line just declares a name for this workflow file. Any qualified Java identifier is allowed, but it should match the file name of the MWE2 file (and in future versions we might make this mandatory). It is followed by an import statement. MWE2 references Java classes and to make that convenient, you can specify imports after the module declaration. Next up we see a couple of vars are declared. Such vars can be overridden when invoking a workflow file.

The important part is the Workflow part. There we declare an instance of org.eclipse.emf.mwe.utils.Workflow and add two instances to the component property. The first one is a Reader which is used to initialize Xtext languages, read in Xtext files, and fetch specific elements from those models in order to make them available to following workflow components (such as the declared Generator).

Xtext language initialization is done by specifying any number of ISetup implementations. In our case the generated DomainModelStandaloneSetup is registered, which makes sure that the infrastructure of your domain model language is set up properly. If you have multiple languages just add additional assignments for each language.

You have to tell the Reader which paths to scan for model files. In this particular case we just specified one path. Another convenient option is to reuse the Java classpath, as suggested in the comments.

The load section specifies which elements to fetch from the loaded resources. In this case we state that we want all elements of type Greeting. Note that this is completely file agnostic, it will provide you with all elements from all files the Reader could find on the specified paths. The slot name is the name by which other workflow components can access the stored elements.

The second workflow component is an instance of org.eclipse.xpand2.Generator, which is the MWE2 facade to the Xpand template language. The Xpand generator needs to know which template to invoke for which models. Qualified names in Xpand are separated by a double colon '::'. That is the name 'templates::Template::main' points to the definition main in the file templates/Template.xpt on the Java classpath. The second part FOREACH greetings references the greetings slot which has previously been populated by the reader component.

An Outlet describes where to put the generated sources. In Xpand there is a file-statement which refers to outlets. If you only have one outlet you don’t have to give it a name, but you need to declare where the root folder for that outlet can be found in the file system. I.e. you specifiy where the generated code should go. Oulets allow you to specify a couple of other things as well. As MWE2 just instantiates Java objects, you can go to the Java code of org.eclipse.xpand2.output.Outlet in order to find out (see the adder and setter methods). about the different options.

Now that we understood the most important things in the workflow we have to adapt it to match our domain model language. We only have to change the type which is used to fetch and filter the elements in the load section. We want to generate code for entities, therefore we change it to Entity. We should change the slot’s name, too:

module workflow.DomainmodelGenerator

import org.eclipse.emf.mwe.utils.*

var targetDir = "src-gen"
var fileEncoding = "ISO-8859-1"
var modelPath = "src/model"

Workflow {

component = org.eclipse.xtext.mwe.Reader {
// lookup all resources on the classpath
// useJavaClassPath = true

// or define search scope explicitly
path = modelPath

// this class will be generated by the xtext generator 
register = org.eclipse.xtext.example.DomainmodelStandaloneSetup {}
load = {
slot = "entities" //changed to entities
type = "Entity"   //changed to Entity
}
}

component = org.eclipse.xpand2.Generator {
expand = "templates::Template::main FOREACH entities" //changed to entities
outlet = {
path = targetDir
}
fileEncoding = fileEncoding
}
}

Using Xpand (3) and Xtend (4) for Code Generation

The MWE2 file invokes an Xpand definition called main located in templates/Template.xpt. Please open that file and replace its contents with the following Xpand code:

«IMPORT org::eclipse::xtext::example::domainmodel»
«EXTENSION templates::Extensions»

«DEFINE main FOR Entity-»
«FILE qualifiedName().replaceAll("\\.","/")+".java"-»
package «packageName()»;

public class «name» {
  «EXPAND property FOREACH features»
}
«ENDFILE»
«ENDDEFINE»

«DEFINE property FOR Feature»
  private «type.referenced.qualifiedName()» «name»;
  
  public void set«name.toFirstUpper()»(
       «type.referenced.qualifiedName()» «name») {
    this.«name» = «name»;
  } 
  public «type.referenced.qualifiedName()» get«name.toFirstUpper()»() {
    return «name»;
  } 
«ENDDEFINE»

You might get a couple of error markers, because the Xpand file references another file ( Extensions.ext (4)) which has not yet been updated. Let’s ignore this for a moment and have a look at the general structure of an Xpand template file. In the first line we import the namespace (i.e. Java package) of the generated AST classes, that is Entity and Feature, we want to refer to in the generator. Next up the previously mentioned Xtend file is imported. An Xtend file defines functions which can be used in Xpand.

The rest of the file contains two so called definitions. A definition in Xpand is similar to a function in that it can be called, it has a name and it is defined for one or more arguments. The general syntax is as follows:

«DEFINE name(ArgType1 arg1, ArgType2 arg2) FOR MainArgType»

Where MainArgType is bound to the variable name this which can like in Java be omitted when referring to.

The first definition main is defined for Entity and is the one invoked from the previously discussed MWE2 file (2). The first thing the definition does is opening a file using the qualifiedName() of the given Entity. The function qualifiedName() should be declared in the referenced Xtend file (4). To do so open that file and replace its current contents by the following snippet:

import org::eclipse::xtext::example::domainmodel;

packageName(Type this) :
qualifiedName(eContainer());

packageName(Void this) : null;

String qualifiedName(Object this) : null;

String qualifiedName(Type this) : 
packageName()==null? name : packageName()+"."+name;

String qualifiedName(PackageDeclaration this) :
if qualifiedName(eContainer())==null 
then name
else qualifiedName(eContainer())+'.'+name;

It defines the qualifiedName() function not only for Entity but also generally for types. Note how it computes the qualified name by calling qualifiedName() recursively on its eContainer(). Both Xpand and Xtend are based on the same expression language, which is statically typed and allows for very convenient navigation over object graphs and collections.

Back to the Xpand file (3) we see some static text which goes directly into the opened file. The EXPAND statement calls the other definition defined in this file. It will effectively generate a Java field and two accessor-methods for each Feature.

You’ll now able able to run the generator by opening the context menu on the *.mwe2 file (2) and choose Run As->MWE2 Workflow. For more information on Xpand and Xtend see the corresponding language documentation (available through Eclipse Help). MWE2 is explained in detail in a later section.