TestRig: use with package-ized lexer/parser?

[Graham Wideman]

Hi Terr and all,

I've been attempting to use the TestRig (part of the ANTLR 4 tools), and have fallen down a hole that I'm hoping someone just knows the answer to.

The short version is this:

How does one use TestRig with a grammar that specifies a package for the lexer/parser?

Longer version:

In the command line for TestRig we supply the name of the grammar, from which the TestRig class figures out the name of lexer and parser classes to dynamically load (using a ClassLoader). But TestRig fails to find the lexer/grammar classes if the grammar specifies a package.

Issue 1: How to load package-ized lexer?
-----------------------------------------
TestRig fails if the grammar specifies a package name. Or at least, I've not figured out how to get TestRig to find the lexer and parser classes when a package is specified. Any clues?

Issue 2: Why does TestRig find lexer class in current dir?
-----------------------------------------------------------
I attempted to create my own version of TestRig (with new name TestRig2, and using different package from the original) and though I can get it to compile and run, I can't get it to load even a non-packaged lexer... that is I can't even get my seemingly minimally modified TestRig2 to load a lexer class that the original TestRig is fine with.

Now it seems that in the original TestRig, the ClassLoader (from Thread.currentThread)getContextClassLoader) is able to find the (unpackaged) lexer class in the "current directory"... but I don't understand why it can do this -- how is it that it treats current dir as on the classpath?

Any clues on this?

Issue 3: TestRig exception improvement suggestion
--------------------------------------------------
A note for Terr: The exception handling around the class loading is a bit hosed up -- the second attempt at loading (without Lexer suffix) is in the catch block, but this may fail, in which case execution bails with an unhandled exception, instead of proceeding with lexerClass == null to the intended "Can't handle" message.

Thanks!

-- Graham


List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address

[Graham Wideman]

Hi Terr and all,

Update:

I've solved half of the puzzle of TestRig's dynamic class loading, but I'm still interested in whether there's already an intended way to use TestRig with a grammar that specifies a package.

The part I solved is how TestRig manages to find default-package lexer and parser class files under normal conditions. The short version is that TestRig relies on there being a "dot" (current dir) in the CLASSPATH environment variable. But there are a number of correlated details/pitfalls.

The keys are:

1) The user must invoke TestRig (runs grun shell/batch file) from the directory containing the lexer and parser class files, in such a way that TestRig (or my version TestRig2) sees that dir as the working dir. The subtle issue here is...

2) TestRig's own jar file must be in the CLASSPATH, so that it can be invoked with a reference to the class to run:
java org.antlr.v4.runtime.misc.TestRig %*
and NOT with the -jar option like this (with runnable class in the manifest):
java -jar "path-to-jarfile\TestRig2.jar" %*
... as this latter apparently changes the working directory.

3) The current working dir (".") must be included in the CLASSPATH (environment variable) that TestRig sees. (Note: the class loader for files does not treat the current working directory as an assumed place to look for classes. It only looks in cwd if there's a dot in CLASSPATH).

The appropriate CLASSPATH (including TestRig and dot can be set up in grun shell/batch file). However...

4) When debugging in NetBeans, I didn't find a way to set the debug/run CLASSPATH to include the current working dir (though there is a setting for current working dir itself). One work-around is to add the desired directory (the one containing the lexer and parser class) as a "Library" in Netbeans. Unfortunately there doesn't seem to be a way to set "." as a Library.

5) It's perhaps obvious, but the package name for a custom version of TestRig is not salient to finding the lexer/parser classes (though of course invocation of custom TestRig does have to refer to the custom package name).

(And by way of leaving clues: I wanted to look at the ClassLoader search algorithm, but the JRE 7 source from Oracle manages to omit crucial sun.misc.URLClassPath. However, more-comprehensive source is available here: http://jdk7src.sourceforge.net/)

------

OK, so now I can build and debug my own version of TestRig, and unless someone has an alternative bright idea, I hope to explore adding a command-line option to specify a package name for the grammar.

-- Graham


List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address

[Terence Parr]

Hi. A couple of thoughts:

1. It standard practice to put "." first in the class path.
2. java -jar limits the path to only those things inside the jar I think.
3. It should be clear that antlr's jar must be in the CLASSPATH to pick up TestRig

I'll lookat TestRig and packages.

Ter

Post by Graham Wideman
Hi Terr and all,
I've solved half of the puzzle of TestRig's dynamic class loading, but I'm still interested in whether there's already an intended way to use TestRig with a grammar that specifies a package.
The part I solved is how TestRig manages to find default-package lexer and parser class files under normal conditions. The short version is that TestRig relies on there being a "dot" (current dir) in the CLASSPATH environment variable. But there are a number of correlated details/pitfalls.
1) The user must invoke TestRig (runs grun shell/batch file) from the directory containing the lexer and parser class files, in such a way that TestRig (or my version TestRig2) sees that dir as the working dir. The subtle issue here is...
java org.antlr.v4.runtime.misc.TestRig %*
java -jar "path-to-jarfile\TestRig2.jar" %*
... as this latter apparently changes the working directory.
3) The current working dir (".") must be included in the CLASSPATH (environment variable) that TestRig sees. (Note: the class loader for files does not treat the current working directory as an assumed place to look for classes. It only looks in cwd if there's a dot in CLASSPATH).
The appropriate CLASSPATH (including TestRig and dot can be set up in grun shell/batch file). However...
4) When debugging in NetBeans, I didn't find a way to set the debug/run CLASSPATH to include the current working dir (though there is a setting for current working dir itself). One work-around is to add the desired directory (the one containing the lexer and parser class) as a "Library" in Netbeans. Unfortunately there doesn't seem to be a way to set "." as a Library.
5) It's perhaps obvious, but the package name for a custom version of TestRig is not salient to finding the lexer/parser classes (though of course invocation of custom TestRig does have to refer to the custom package name).
(And by way of leaving clues: I wanted to look at the ClassLoader search algorithm, but the JRE 7 source from Oracle manages to omit crucial sun.misc.URLClassPath. However, more-comprehensive source is available here: http://jdk7src.sourceforge.net/)
------
OK, so now I can build and debug my own version of TestRig, and unless someone has an alternative bright idea, I hope to explore adding a command-line option to specify a package name for the grammar.
-- Graham
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address

List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address

[Terence Parr]

I assume you have tried TestRig pack.Grammar? I don't remember if I thought of this so it might not work but I can't see why not. Should be a trivial string concatenation fix. I will look at the other things as well.
T

Post by Graham Wideman
Hi Terr and all,
I've been attempting to use the TestRig (part of the ANTLR 4 tools), and have fallen down a hole that I'm hoping someone just knows the answer to.
How does one use TestRig with a grammar that specifies a package for the lexer/parser?
In the command line for TestRig we supply the name of the grammar, from which the TestRig class figures out the name of lexer and parser classes to dynamically load (using a ClassLoader). But TestRig fails to find the lexer/grammar classes if the grammar specifies a package.
Issue 1: How to load package-ized lexer?
-----------------------------------------
TestRig fails if the grammar specifies a package name. Or at least, I've not figured out how to get TestRig to find the lexer and parser classes when a package is specified. Any clues?
Issue 2: Why does TestRig find lexer class in current dir?
-----------------------------------------------------------
I attempted to create my own version of TestRig (with new name TestRig2, and using different package from the original) and though I can get it to compile and run, I can't get it to load even a non-packaged lexer... that is I can't even get my seemingly minimally modified TestRig2 to load a lexer class that the original TestRig is fine with.
Now it seems that in the original TestRig, the ClassLoader (from Thread.currentThread)getContextClassLoader) is able to find the (unpackaged) lexer class in the "current directory"... but I don't understand why it can do this -- how is it that it treats current dir as on the classpath?
Any clues on this?
Issue 3: TestRig exception improvement suggestion
--------------------------------------------------
A note for Terr: The exception handling around the class loading is a bit hosed up -- the second attempt at loading (without Lexer suffix) is in the catch block, but this may fail, in which case execution bails with an unhandled exception, instead of proceeding with lexerClass == null to the intended "Can't handle" message.
Thanks!
-- Graham
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address

List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address

[Terence Parr]

Hi. it works for me:

$ cat foo/O.g4
grammar O;

a : ID+ ;

ID : [a-z]+ ;
WS : [ \r\t\n]+ -> skip ;
$ antlr4 -package foo foo/O.g4
$ javac -d . foo/O*.java
$ grun foo.O a -tree
x y z
(a x y z)

Ter

Post by Terence Parr
I assume you have tried TestRig pack.Grammar? I don't remember if I thought of this so it might not work but I can't see why not. Should be a trivial string concatenation fix. I will look at the other things as well.

Well, yes, I did go down that path (so to speak) unsuccessfully, but as I now learn, I missed a trick.
So, here are the details on how to exercise package-ized ANTLR grammars with TestRig as currently designed.
1. Class directory structure must correspond to package
--------------------------------------------------------
[MyProject]/build/classes/com/abc/myparser/mygrammarLexer.class etc
[see note A]
2. cd to _root_ of classes
--------------------------
[MyProject]/build/classes
... so that java runtime classloader can find the class in question from dot (cwd) using the classname com.abc.myparser.mygrammarLexer .
3. Specify grammar name qualified by package
--------------------------------------------
grun com.abc.myparser.mygrammar startelement -gui otherargs
[See note B]
------------
A) This probably requires that, upstream, you have ANTLR generate the java source files in a similarly appropriate source directory structure -- ANTLR itself doesn't create or require package-coordinated directories when generating the lexer/parser code. While ANTLR and javac can be used with simple commands assuming default package, once non-default packages become involved one probably needs a build system or IDE to automate the coordination of packages and directories.
B) Of course, since you have to set the cwd to the classes root for TestRig's benefit, you probably have to adjust the argument that specifies the location of input files.
C) An IDE issue: When working on a custom version of TestRig in NetBeans, and attempting to run or debug it from the IDE, I still haven't found a good way to add "current dir" to the runtime classpath. You can add an explicit dir to the Libraries list, but that's a poor substitute, so not a final answer, IMO. (This is only an issue if you're trying to work on TestRig with its dynamic class loading, not a general problem with ANTLR and packages.)
Hope that helps others.
-- Graham

List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address