Getting Started With ANTLR:Basics

Yeah! It's after a lapse of a month or so that there is a post in this blog! :)
Well, this post drives you towards the basics of ANTLR. Previously, we had learnt about setting up of ANTLR as an external tool.
RECAP! It's here:   ANTLR External Tool
:)
So, here we go....  
  

What is ANTLR?• ANother Tool for Language Recognition, is a language tool that provides a framework for constructing recognizers, interpreters, compilers, and translators from grammatical descriptions containing actions.  

What can be the target languages?    

• Action Script, Ada  

• C
• C#; C#2
• C#3
• D
• Emacs ELisp
• Objective C
• Java
• Java Script
• Python
• Ruby
• Perl6
• Perl
• PHP
• Oberon
• Scala
  

What does ANTLR support?• Tree construction
• Error recovery
• Error handling
• Tree walking
• Translation
  

What environment does it support?  

ANTLRWorks is the IDE for ANTLR. It is the graphical grammar editor and debugger, written by Jean Bovet using Swing.

What for ANTLR can be used?  
• ""REAL"" programming languages  

• domain-specific languages [DSL]
  

Who is using ANTLR?
• Programming languages :Boo, Groovy, Mantra, Nemerle, XRuby etc.  

• Other Tools: HIbernate, Intellij IDEA, Jazillian, JBoss Rules, Keynote(Apple), WebLogic(Oracle) etc.

  

Where is that you can look for ANTLR? 

You  can always follow here http://www.antlr.org • to download ANTLR and ANTLRWorks, which are free and open source

• docs,articles,wiki,mailing list,examples....

 You can catch everything here!

Row your Boat....

• Basic terms

• Lexer : converts a stream of characters to a stream of tokens.
• Parser : processes of tokens, possibly creating AST
• Abstract Syntax Tree(AST): an intermediate tree representation of the parsed input that is simpler to process than the stream of tokens. It can as well be processed multiple times.
• Tree Parser: It processes an AST
• String Template: a library that supports using templates with placeholders for outputting text   

• General Steps

• Write Grammar in one or more files
• Write string templates[optional]
• Debug your grammar with ANTLRWorks
• Generate classes from grammar
• Write an application that uses generated classes
• Feed the application text that conforms to the grammar


A Bit Further....

Lets write a simple grammar which consists of
• Lexer
• Parser
Lexer: Breaks the input stream into tokens
Lets take the example of simple declaration type in C of the form "int a,b;" or "int a;" and same with float.As we see we can write lexer as follows:


//TestLexer.g
grammar TestLexer;                                                                                                                                         
ID  : ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'0'..'9'|'_'|'.'|'a'..'z'|'A'..'Z')*;                                                                                           COMMA: ',';                                                                                                                                
SEMICOLON:';';                                                                                                                            
DATATYPE: 'int' | 'float';



As we could see, these were the characters that were to be converted to tokens.   
So, now lets write some rules which processes these tokens generated and may it create a parse tree accordingly.



//TestParser.g
grammar TestParser;
options {language : Java;}
decl:DATATYPE ID (',' ID)* ;      

Running ANTLR on the grammar just generates the lexer and parser,TestParser and TestLexer. To actually try the grammar on some input, we
need a test rig with a main( ) method as follows:

// Test.java
import org.antlr.runtime.*;
public class Test {

public static void main(String[] args) throws Exception {

// Create an input character stream from standard in
ANTLRFileStream input = new ANTLRFileStream("input"); // give path to the file input

// Create an ExprLexer that feeds from that stream
TestLexer lexer = new TestLexer(input);
// Create a stream of tokens fed by the lexer
CommonTokenStream tokens = new CommonTokenStream(lexer);
// Create a parser that feeds off the token stream
TestParser parser = new TestParser(tokens);
// Begin parsing at rule decl
parser.decl();
}}


We shall see how to create an AST and walk over the tree in the next blog post...
Happy learning....! :)

ANTLR as an external tool in eclipse on ubuntu

This tutorial tells how to setup the ANTLR in your eclipse.
STEP 1:
Download the jar file antlrworks-1.4.2.jar from http://www.antlr.org/download.
Further details about ANTLRWorks: The ANTLR GUI Development Environment, follow the link : http://www.antlr.org/works/index.html
STEP 2:
Create a java project in eclipse as follows:
File->New->Project
Select Java and Java project.
Click on Next.
Name the project as "TestANTLR"
Press Finish.
Add the antlrworks-1.4.2.jar to the project classpath.
Right click on "TestANTLR" project .
Select Properties->Libraries.
Click on "Add External jar"
Select the complete path of the "antlrworks-1.4.2.jar" and press Ok.
STEP 3: make it as an external tool
Goto Run->External Tools->Configure
Click on New.
Name: ANTLR Compiler
Tool Location: /usr/lib/jvm/java-6-sun-1.6.0.26/bin/java
// this must be the complete path to your java
Tool Arguments: -classpath complete_path_to_antlrworks-1.4.2.jar 
org.antlr.Tool ${resource_name}
Working Directory: ${container_loc}
Here, org.antlr.Tool is the main class which would take the ${resource_name} for processing.
${resource_name} and ${container_loc} can be selected with "Browse Variables" option too.

Going ahead :
*Creating a grammar file
Create a grammar file with .g extension. Say, Example.g

//sample code
grammar Example;
start : 'hello' ID ';' {System.out.println("hiii... "+$ID.text);} ;
ID: 'a'..'z' + ;
WS: (' ' |'\n' |'\r' )+
{$channel=HIDDEN;}

*Running the above code:
Run->External Tools->ANTLR Compiler
Press F5 or right click on the project and "refresh"
all you can see is a lexer and parser files generated with the tokens.

In our example,
ExampleLexer.java , ExampleParser.java and Example.tokens

Create Main.java program in the same project with the following code:

import org.antlr.runtime.*;
public class Main {
public static void main(String[] args) throws Exception {
// create a CharStream that reads from standard input
ANTLRInputStream input = new ANTLRInputStream(System.in);
// create a lexer that feeds off of input CharStream
ExampleLexer lexer = new ExampleLexer(input);
// create a buffer of tokens pulled from the lexer
CommonTokenStream tokens = new CommonTokenStream(lexer);
// create a parser that feeds off the tokens buffer
ExampleParser parser = new ExampleParser(tokens);
// begin parsing at rule start
parser.start();
}}  


Set the arguments in the Run configurations and click on Apply and Run.

Now you have the output at console.

:)