documentation/aql.textile - gerrit/www.eclipse.org/acceleo - Git at Google

 h1. Acceleo Query Language (AQL)

 h2. Introduction

 The Acceleo Query Language, AQL, is a new query interpreter created to replace the Acceleo MTL engine when used as a simple interpreter. AQL is small, simple, fast, extensible and it brings a richer validation than the MTL interpreter.

 For those looking for a simple and fast interpreters for your EMF models, AQL can provide you with a lot of features, including:

 * Support for static and dynamic Ecore models, no query compilation phase required.
 * The least possible overhead at evaluation time. During this phase, the evaluation goes forward and will not even try to validate or compile your expressions. Errors are tracked and captured along the way.
 * Strong validation: types are checked at validation time and the metamodels used are analyzed to do some basic type inference and avoid false positive errors.
 * Union types: In some context, a variable in a given query can have N potential types with N being often greater than one. AQL can embrace this fact without having to fall back to EObject as soon as there is more than one type.
 * A simple and straightforward implementation easily extensible with Java classes providing extension methods.
 * A very narrow dependency surface: AQL uses the very central parts of EMF, Guava and Antlr so that we could easily deploy AQL outside of Eclipse in a server or a standalone scenario.

 The AQL interpreter is used in Sirius with the prefix "aql:".

 h2. Language Reference

 These sections are listing all the services of the standard library of AQL.

 * "Services available for all types":./reference/aql_service_anyservices.html
 * "Services available for Booleans":./reference/aql_service_booleanservices.html
 * "Services available for Collections":./reference/aql_service_collectionservices.html
 * "Services available for Comparables":./reference/aql_service_comparableservices.html
 * "Services available for EObjects":./reference/aql_service_eobjectservices.html
 * "Services available for Resources and URIs":./reference/aql_service_resourceservices.html
 * "Services available for Strings":./reference/aql_service_stringservices.html


 h2. Using AQL programmatically

 This section provide information and code snippet. It will help you to integrate AQL in your own tool.

 Simple overview of AQL:

 !../images/aql/AQL_overview.png!

 h3. Type validation

 For each node of the AST we create a set of possible types as follow:
 * for a VarRef we ask the environment for its possible types
 * for a FeatureAccess we look up the type of the feature in the registered metamodels
 * for a Call we look up the the service in the service registry according to the possible types of its parameters (receiver is the first parameter). At this point there is a conversion from EMF to Java. The return type of the service is given by the IService.getType() method. At this point the there is a conversion form Java to EMF, one Java type can correspond to more than one EMF EClassifier. If no service can be found we try to find a corresponding EOperation if the receiver is an EObject.

 A special type NothingType is used to mark a problem on a given node of the AST. Those NothingTypes are then used to create validation messages. If an AST node has only NothingTypes validation messages will be set as errors for this node, otherwise they are set as warnings.

 h3. Completion

 The completion rely on the AST production and the type validation.
 The identifier fragments preceding (prefix) and following (remaining) the cursor position are removed from the expression to parse. The prefix and remaining are used later to filter the proposals. Many filters can be implemented: filter only on prefix, filter on prefix and remaining, same strategies with support for camel case, ...

 Completion on the AST:
 * if there is no error node in the AST the completion provide any symbols that can follow an expression ("+", "-", ...).
 * if there is an ErrorExpression node in the AST the completion provides anything that can prefix an expression ("not", "-", variable name, type name, ...).
 * if there is an ErrorFeatureAccesOrCall node in the AST the completion provides feature and service names corresponding to the receiver possible types. It also possible to add symbols that follow an expression if the prefix and remaining are already a valid feature or service name for the receiver possible types.
 * if there is an ErrorCollectionCall node in the AST the completion provides collection service names. It also possible to add symbols that follow an expression if the prefix and remaining are already a valid service name.
 * if there is an ErrorTypeLiteral node in the AST the completion provides EClassifier, EEnumLiteral names according to the state of the type description.

 h3. Creating and setting the environment

 To get a fresh environment you can use one of the following snippet:

 bc. IQueryEnvironment queryEnvironment = Query.newEnvironmentWithDefaultServices(null);

 To get an environment with predefined services.

 or

 bc. IQueryEnvironment queryEnvironment = Query.newEnvironment(null);

 To get an environment with no predefined services. It can be useful to create your own language primitives.

 Note that you can also provide a CrossReferenceProvider to define the scope of cross references in your environment. See CrossReferencerToAQL for more details.

 You can register new services Class as follow:

 bc. ServiceRegistrationResult registrationResult = queryEnvironment.registerServicePackage(MyServices.class);

 The registration result contains information about services overrides.

 You can also register your EPackages. Only registered EPackages are used to validate and evaluate AQL expression.

 bc. queryEnvironment.registerEPackage(MyEPackage.eINSTANCE);

 In some cases you might also want to create custom mappings between an EClass and its Class. A basic case is the use of EMap:

 bc. queryEnvironment.registerCustomClassMapping(EcorePackage.eINSTANCE.getEStringToStringMapEntry(), EStringToStringMapEntryImpl.class);

 By default the EClass is mapped to Map.Entry which is not an EObject. This prevents using services on EObject.

 h3. Building an AQL expression

 The first step is building your expression from a String to an AST:

 bc. QueryBuilderEngine builder = new QueryBuilderEngine(queryEnvironment);
 AstResult astResult = builder.build("self.name");

 h3. Evaluating an AQL expression

 To evaluate an AQL expression you can use the QueryEvaluationEngine

 bc. QueryEvaluationEngine engine = new QueryEvaluationEngine(queryEnvironment);
 Map<String, Object> variables = Maps.newHashMap();
 variables.put("self", EcorePackage.eINSTANCE);
 EvaluationResult evaluationResult = engine.eval(astResult, variables);

 Here we only use one variable for demonstration purpose.

 h3. Validating an AQL expression (optional)

 This step is optional for evaluation. You can evaluate an AQL expression without validating it in the first place.

 bc. Map<String, Set<IType>> variableTypes = new LinkedHashMap<String, Set<IType>>();
 Set<IType> selfTypes = new LinkedHashSet<IType>();
 selfTypes.add(new EClassifierType(queryEnvironment, EcorePackage.eINSTANCE.getEPackage()));
 variableTypes.put("self", selfTypes);
 AstValidator validator = new AstValidator(queryEnvironment, variableTypes);
 IValidationResult validationResult = validator.validate(astResult);

 h3. Completing an AQL expression

 To do this use the QueryCompletionEngine, it will build the query and validate it for you. It will also compute needed prefix and suffix if any:

 bc. Map<String, Set<IType>> variableTypes = new LinkedHashMap<String, Set<IType>>();
 Set<IType> selfTypes = new LinkedHashSet<IType>();
 selfTypes.add(new EClassifierType(queryEnvironment, EcorePackage.eINSTANCE.getEPackage()));
 variableTypes.put("self", selfTypes);
 QueryCompletionEngine engine = new QueryCompletionEngine(queryEnvironment);
 ICompletionResult completionResult = engine.getCompletion("self.", 5, variableTypes);
 List<ICompletionProposal> proposals = completionResult.getProposals(new BasicFilter(completionResult));

 Here 5 is the offset where the completion should be computed in the given expression.

 h2. Migrating from MTL queries

 As languages, AQL and MTL are very close yet there are some notable differences:

 h3. Implicit variable references

 There is no implicit variable reference. With this change, you can easily find out if you are using a feature of an object or a string representation of said object. As a result, instead of using "[something/]", you must use "self.something" if you want to access the feature named "something" of the current object or "something" if you want to retrieve the object named something.

 In a lambda expression, you must now define the name of the variable used for the iteration in order to easily identify which variable is used by an expression. In Acceleo MTL, you can write "Sequence{self}->collect(eAllContents(uml::Property))" and Acceleo will use the implicit iterator as a source of the operation eAllContents.

 The problem comes when using a lambda like "Sequence{self}->collect(something)", we can't know if "something" is a feature of "self" or if it is another variable.

 Using AQL, you will now have to write either "collect(m | m.eAllContents(uml::Property))" or "collect(m: uml::Model | eAllContents(uml::Property))".

 h3. Collect and flatten

 When a call or a feature acces is done on a collection the result is flattened for the first level. For instance a service returning a collection called on a collection will return a collection of elements and not a collection of collection of elements.

 h3.  Type literals & children EPackages

 Type literals can't be in the form someEPackage::someSubEPackage::SomeEClass but instead someSubEPackage::SomeEClass should be directly used. Note that the name of the EPackage is now mandatory.

 h3.  Type literals & children EPackages

 Enumeration literal should be prefixed with the name of the containing EPacakge for instance "myPacakge::myEnum::value".

 h3. Collections

 You can only have Sequences or OrderedSets as collections and as such the order of their elements is always deterministic. In Acceleo MTL, you had access to Sets, which are now OrderedSets and Bags, which are now Sequences. Those four kinds of collections were motivated by the fact that Sequence and OrderedSet were ordered contrary to Sets and Bags. On another side, OrderedSets and Sets did not accept any duplicate contrary to Bags and Sequences.

 By careful reviewing the use of those collections in various Acceleo generators and Sirius Designers we have quickly found out that the lack of determinism in the order of the collections Sets and Bags was a major issue for our users. As a result, only two collections remain, the Sequence which can contain any kind of element and the OrderedSet which has a similar behavior except that it does not accept duplicates.

 Previously in Acceleo MTL, you could transform a literal into a collection by using the operator "->" on the literal directly. In Acceleo MTL, the collection created was a Bag which is not available anymore. It is recommended to use the extension notation like "Sequence{self}" or "OrderedSet{self}". By default in AQL the created collection is an OrderedSet.

 h3. Renamed operations

 Some operations have been renamed. As such "addAll" and "removeAll" have been renamed "add" and "sub" because those two names are used by AQL in order to provide access to the operator "+" and "-". As a result we can now write in AQL "firstSequence + secondSequence" or "firstSet - secondSet".

 h3. Typing

 AQL is way smarter than MTL regarding to the types of your expressions. As a result, you can combine expressions using multiple types quite easily. For example, this is a valid AQL expression "self.eContents(uml::Class).add(self.eContents(ecore::EClass)).name". In Acceleo MTL, we could not use this behavior because Acceleo MTL had to fall back to the concept EObject which does not have a feature "name" while AQL knows that the collection contains objects that are either "uml::Class" or "ecore::EClass" and both of those types have a feature named "name".

 h3. null handling

 AQL handles null (OclVoid) differently from ocl, so "oclIsUndefined" should be mostly useless for AQL expressions. For example, "null.oclIsKindOf(ecore::EClass)" would have returned true for MTL/OCL, forcing users to use "not self.oclIsUndefined() and self.oclIsKindOf(ecore::EClass) instead. This is no longer true in AQL, where "null" doesn't conform to any type, so "null.oclIsKindOf(ecore::EClass)" will return false. Note that it's still possible to "cast" null in any given classifier. "null.oclAsType(ecore::EClass)" will not fail at runtime.

 h2. Migrating from Acceleo2 queries

 h3. EClassifier references

 All operations referencing a type are now using a type literal with the name of the EPackage and the name of the type instead of a string with the name of the type. As a result, "eObject.eAllContents('EClass')" would be translated using "eObject.eAllContents('ecore::EClass'). This allows AQL to now in which EPackage to look for the type and as such, it improves the quality of the validation.

 h3. Types and cast

 In order to test the type of an EObject, a common pattern in Acceleo 2 was to treat the EObject as a collection and filter said collection on the type desired to see if the size of the collection changed. In AQL, you have access to the operations oclIsTypeOf and oclIsKindOf. You can thus test the type of an EObject with the expression "eObject.oclIsKindOf(ecore::EStructuralFeature)" or "eObject.oclIsTypeOf(ecore::EAttribute)". You can use the operation oclIsKindOf to test if an object has the type of the given parameter or one of its subtype. On the other hand, you can use the operation oclIsTypeOf to test if an object has exactly the type of the given parameter.

 Casting in AQL is useless, since AQL is very understandable when it comes to types, it will always tries its best to evaluate your expression.

 Since AQL is very close to Acceleo MTL, you can find some additional documentation using the Acceleo equivalence documentation in the "Acceleo documentation":http://help.eclipse.org/mars/index.jsp?topic=%2Forg.eclipse.acceleo.doc%2Fpages%2Freference%2Fmigration.html&cp=5_3_4.
	h1. Acceleo Query Language (AQL)

	h2. Introduction

	The Acceleo Query Language, AQL, is a new query interpreter created to replace the Acceleo MTL engine when used as a simple interpreter. AQL is small, simple, fast, extensible and it brings a richer validation than the MTL interpreter.

	For those looking for a simple and fast interpreters for your EMF models, AQL can provide you with a lot of features, including:

	* Support for static and dynamic Ecore models, no query compilation phase required.
	* The least possible overhead at evaluation time. During this phase, the evaluation goes forward and will not even try to validate or compile your expressions. Errors are tracked and captured along the way.
	* Strong validation: types are checked at validation time and the metamodels used are analyzed to do some basic type inference and avoid false positive errors.
	* Union types: In some context, a variable in a given query can have N potential types with N being often greater than one. AQL can embrace this fact without having to fall back to EObject as soon as there is more than one type.
	* A simple and straightforward implementation easily extensible with Java classes providing extension methods.
	* A very narrow dependency surface: AQL uses the very central parts of EMF, Guava and Antlr so that we could easily deploy AQL outside of Eclipse in a server or a standalone scenario.

	The AQL interpreter is used in Sirius with the prefix "aql:".

	h2. Language Reference

	These sections are listing all the services of the standard library of AQL.

	* "Services available for all types":./reference/aql_service_anyservices.html
	* "Services available for Booleans":./reference/aql_service_booleanservices.html
	* "Services available for Collections":./reference/aql_service_collectionservices.html
	* "Services available for Comparables":./reference/aql_service_comparableservices.html
	* "Services available for EObjects":./reference/aql_service_eobjectservices.html
	* "Services available for Resources and URIs":./reference/aql_service_resourceservices.html
	* "Services available for Strings":./reference/aql_service_stringservices.html


	h2. Using AQL programmatically

	This section provide information and code snippet. It will help you to integrate AQL in your own tool.

	Simple overview of AQL:

	!../images/aql/AQL_overview.png!

	h3. Type validation

	For each node of the AST we create a set of possible types as follow:
	* for a VarRef we ask the environment for its possible types
	* for a FeatureAccess we look up the type of the feature in the registered metamodels
	* for a Call we look up the the service in the service registry according to the possible types of its parameters (receiver is the first parameter). At this point there is a conversion from EMF to Java. The return type of the service is given by the IService.getType() method. At this point the there is a conversion form Java to EMF, one Java type can correspond to more than one EMF EClassifier. If no service can be found we try to find a corresponding EOperation if the receiver is an EObject.

	A special type NothingType is used to mark a problem on a given node of the AST. Those NothingTypes are then used to create validation messages. If an AST node has only NothingTypes validation messages will be set as errors for this node, otherwise they are set as warnings.

	h3. Completion

	The completion rely on the AST production and the type validation.
	The identifier fragments preceding (prefix) and following (remaining) the cursor position are removed from the expression to parse. The prefix and remaining are used later to filter the proposals. Many filters can be implemented: filter only on prefix, filter on prefix and remaining, same strategies with support for camel case, ...

	Completion on the AST:
	* if there is no error node in the AST the completion provide any symbols that can follow an expression ("+", "-", ...).
	* if there is an ErrorExpression node in the AST the completion provides anything that can prefix an expression ("not", "-", variable name, type name, ...).
	* if there is an ErrorFeatureAccesOrCall node in the AST the completion provides feature and service names corresponding to the receiver possible types. It also possible to add symbols that follow an expression if the prefix and remaining are already a valid feature or service name for the receiver possible types.
	* if there is an ErrorCollectionCall node in the AST the completion provides collection service names. It also possible to add symbols that follow an expression if the prefix and remaining are already a valid service name.
	* if there is an ErrorTypeLiteral node in the AST the completion provides EClassifier, EEnumLiteral names according to the state of the type description.

	h3. Creating and setting the environment

	To get a fresh environment you can use one of the following snippet:

	bc. IQueryEnvironment queryEnvironment = Query.newEnvironmentWithDefaultServices(null);

	To get an environment with predefined services.

	or

	bc. IQueryEnvironment queryEnvironment = Query.newEnvironment(null);

	To get an environment with no predefined services. It can be useful to create your own language primitives.

	Note that you can also provide a CrossReferenceProvider to define the scope of cross references in your environment. See CrossReferencerToAQL for more details.

	You can register new services Class as follow:

	bc. ServiceRegistrationResult registrationResult = queryEnvironment.registerServicePackage(MyServices.class);

	The registration result contains information about services overrides.

	You can also register your EPackages. Only registered EPackages are used to validate and evaluate AQL expression.

	bc. queryEnvironment.registerEPackage(MyEPackage.eINSTANCE);

	In some cases you might also want to create custom mappings between an EClass and its Class. A basic case is the use of EMap:

	bc. queryEnvironment.registerCustomClassMapping(EcorePackage.eINSTANCE.getEStringToStringMapEntry(), EStringToStringMapEntryImpl.class);

	By default the EClass is mapped to Map.Entry which is not an EObject. This prevents using services on EObject.

	h3. Building an AQL expression

	The first step is building your expression from a String to an AST:

	bc. QueryBuilderEngine builder = new QueryBuilderEngine(queryEnvironment);
	AstResult astResult = builder.build("self.name");

	h3. Evaluating an AQL expression

	To evaluate an AQL expression you can use the QueryEvaluationEngine

	bc. QueryEvaluationEngine engine = new QueryEvaluationEngine(queryEnvironment);
	Map<String, Object> variables = Maps.newHashMap();
	variables.put("self", EcorePackage.eINSTANCE);
	EvaluationResult evaluationResult = engine.eval(astResult, variables);

	Here we only use one variable for demonstration purpose.

	h3. Validating an AQL expression (optional)

	This step is optional for evaluation. You can evaluate an AQL expression without validating it in the first place.

	bc. Map<String, Set<IType>> variableTypes = new LinkedHashMap<String, Set<IType>>();
	Set<IType> selfTypes = new LinkedHashSet<IType>();
	selfTypes.add(new EClassifierType(queryEnvironment, EcorePackage.eINSTANCE.getEPackage()));
	variableTypes.put("self", selfTypes);
	AstValidator validator = new AstValidator(queryEnvironment, variableTypes);
	IValidationResult validationResult = validator.validate(astResult);

	h3. Completing an AQL expression

	To do this use the QueryCompletionEngine, it will build the query and validate it for you. It will also compute needed prefix and suffix if any:

	bc. Map<String, Set<IType>> variableTypes = new LinkedHashMap<String, Set<IType>>();
	Set<IType> selfTypes = new LinkedHashSet<IType>();
	selfTypes.add(new EClassifierType(queryEnvironment, EcorePackage.eINSTANCE.getEPackage()));
	variableTypes.put("self", selfTypes);
	QueryCompletionEngine engine = new QueryCompletionEngine(queryEnvironment);
	ICompletionResult completionResult = engine.getCompletion("self.", 5, variableTypes);
	List<ICompletionProposal> proposals = completionResult.getProposals(new BasicFilter(completionResult));

	Here 5 is the offset where the completion should be computed in the given expression.

	h2. Migrating from MTL queries

	As languages, AQL and MTL are very close yet there are some notable differences:

	h3. Implicit variable references

	There is no implicit variable reference. With this change, you can easily find out if you are using a feature of an object or a string representation of said object. As a result, instead of using "[something/]", you must use "self.something" if you want to access the feature named "something" of the current object or "something" if you want to retrieve the object named something.

	In a lambda expression, you must now define the name of the variable used for the iteration in order to easily identify which variable is used by an expression. In Acceleo MTL, you can write "Sequence{self}->collect(eAllContents(uml::Property))" and Acceleo will use the implicit iterator as a source of the operation eAllContents.

	The problem comes when using a lambda like "Sequence{self}->collect(something)", we can't know if "something" is a feature of "self" or if it is another variable.

	Using AQL, you will now have to write either "collect(m \| m.eAllContents(uml::Property))" or "collect(m: uml::Model \| eAllContents(uml::Property))".

	h3. Collect and flatten

	When a call or a feature acces is done on a collection the result is flattened for the first level. For instance a service returning a collection called on a collection will return a collection of elements and not a collection of collection of elements.

	h3. Type literals & children EPackages

	Type literals can't be in the form someEPackage::someSubEPackage::SomeEClass but instead someSubEPackage::SomeEClass should be directly used. Note that the name of the EPackage is now mandatory.

	h3. Type literals & children EPackages

	Enumeration literal should be prefixed with the name of the containing EPacakge for instance "myPacakge::myEnum::value".

	h3. Collections

	You can only have Sequences or OrderedSets as collections and as such the order of their elements is always deterministic. In Acceleo MTL, you had access to Sets, which are now OrderedSets and Bags, which are now Sequences. Those four kinds of collections were motivated by the fact that Sequence and OrderedSet were ordered contrary to Sets and Bags. On another side, OrderedSets and Sets did not accept any duplicate contrary to Bags and Sequences.

	By careful reviewing the use of those collections in various Acceleo generators and Sirius Designers we have quickly found out that the lack of determinism in the order of the collections Sets and Bags was a major issue for our users. As a result, only two collections remain, the Sequence which can contain any kind of element and the OrderedSet which has a similar behavior except that it does not accept duplicates.

	Previously in Acceleo MTL, you could transform a literal into a collection by using the operator "->" on the literal directly. In Acceleo MTL, the collection created was a Bag which is not available anymore. It is recommended to use the extension notation like "Sequence{self}" or "OrderedSet{self}". By default in AQL the created collection is an OrderedSet.

	h3. Renamed operations

	Some operations have been renamed. As such "addAll" and "removeAll" have been renamed "add" and "sub" because those two names are used by AQL in order to provide access to the operator "+" and "-". As a result we can now write in AQL "firstSequence + secondSequence" or "firstSet - secondSet".

	h3. Typing

	AQL is way smarter than MTL regarding to the types of your expressions. As a result, you can combine expressions using multiple types quite easily. For example, this is a valid AQL expression "self.eContents(uml::Class).add(self.eContents(ecore::EClass)).name". In Acceleo MTL, we could not use this behavior because Acceleo MTL had to fall back to the concept EObject which does not have a feature "name" while AQL knows that the collection contains objects that are either "uml::Class" or "ecore::EClass" and both of those types have a feature named "name".

	h3. null handling

	AQL handles null (OclVoid) differently from ocl, so "oclIsUndefined" should be mostly useless for AQL expressions. For example, "null.oclIsKindOf(ecore::EClass)" would have returned true for MTL/OCL, forcing users to use "not self.oclIsUndefined() and self.oclIsKindOf(ecore::EClass) instead. This is no longer true in AQL, where "null" doesn't conform to any type, so "null.oclIsKindOf(ecore::EClass)" will return false. Note that it's still possible to "cast" null in any given classifier. "null.oclAsType(ecore::EClass)" will not fail at runtime.

	h2. Migrating from Acceleo2 queries

	h3. EClassifier references

	All operations referencing a type are now using a type literal with the name of the EPackage and the name of the type instead of a string with the name of the type. As a result, "eObject.eAllContents('EClass')" would be translated using "eObject.eAllContents('ecore::EClass'). This allows AQL to now in which EPackage to look for the type and as such, it improves the quality of the validation.

	h3. Types and cast

	In order to test the type of an EObject, a common pattern in Acceleo 2 was to treat the EObject as a collection and filter said collection on the type desired to see if the size of the collection changed. In AQL, you have access to the operations oclIsTypeOf and oclIsKindOf. You can thus test the type of an EObject with the expression "eObject.oclIsKindOf(ecore::EStructuralFeature)" or "eObject.oclIsTypeOf(ecore::EAttribute)". You can use the operation oclIsKindOf to test if an object has the type of the given parameter or one of its subtype. On the other hand, you can use the operation oclIsTypeOf to test if an object has exactly the type of the given parameter.

	Casting in AQL is useless, since AQL is very understandable when it comes to types, it will always tries its best to evaluate your expression.

	Since AQL is very close to Acceleo MTL, you can find some additional documentation using the Acceleo equivalence documentation in the "Acceleo documentation":http://help.eclipse.org/mars/index.jsp?topic=%2Forg.eclipse.acceleo.doc%2Fpages%2Freference%2Fmigration.html&cp=5_3_4.