First sorry for perhaps a bad title - I imagine a lot of the difficulty I'm experiencing relates to not knowing the correct terminology for what I'm trying to achieve.
In Go, I wish to have a program which when run can dynamically create a secondary binary. To illustrate with a basic hello world example - in pseudo code since I don't know how to achieve it.
generator.go
-> Read in statement from statement.txt (i.e. "Hello World")
-> Insert this statement into the following program...
package main
import (
"fmt"
)
func main(){
fmt.Println([dynamic statement inserted here])
}
-> compile this code into subprogram
Whenever go run generator.go is executed a subprogram binary is created. Running this would output Hello World. Changing statement.txt to something else and executing go run generator.go again would once more create subprogram which when run would execute the new statement.
In summary
With Go, how can I create a program which can dynamically create a compiled child program as output.
Thanks.
So you have 2 sub-tasks which together do what you want:
Perform a text substitution to acquire the final source code.
Compile the final source code into executable binary.
1. Text substitution
The first can be easily done using the text/template package. You can either have the source templates as separate, individual files, or embedded in the main Go source (e.g. as consts).
You can build / parse the source code templates and get a Template with e.g. template.ParseFiles() or using template.New().Parse(). Once you have your template, assemble (e.g. load from file) the values you want to include in the source template and execute it e.g. with Template.Execute(). You have the final source now.
The text/template package gives you a powerful template engine which is capable to a lot more than just text substitution.
Note: There is also a Go parser implemented and available in the standard library at your disposal in the subpackages of go, but using the text/template package is much simpler and looks it's enough and perfectly fine for your case.
2. Compile
To compile the final source into an executable binary, you need the help of the compiler. You can use the os/exec package to invoke the compiler which will produce the binary. See the exec.Command() function to acquire a Cmd, and Cmd.Run() and Cmd.Start() to execute it.
Related
I am using TCL-C API for my program.
and I read and created test program that is similar to this C++ example.
But I have a problem with this example. when I use this example in the shell (by loading it with load example.o) every input automatically invokes the interpreter of the API and run the command that is related to the input string.
But suppose that I want that the input will invoke tcl procedure that is inside a package required by me , this procedure will check the parameters and will print another message and only after this will invoke TCL-C API related function (kind of wrapper), In this case how can I do it?
I read somewhere that the symbol # is the symbol should be used for invoking external program but I just can't find where it was.
I will give a small example for make things more clear.
somepackage.tcl
proc dosomething { arg1 , arg2 , arg3 } {
# check args here #
set temp [ #invoke here TCL-C API function and set it's result in temp ]
return $temp
}
package provide ::somepackage 1.0
test.tcl
package require ::somepackage 1.0
load somefile.o # this is the object file which implements TCL-C API commands [doSomething 1 2 3 ]
...
But I have a problem with this example. when I use this example in the
shell (by loading it with load example.o) every input automatically
invokes the interpreter of the API and run the command that is related
to the input string.
Provided that you script snippets represent your actual implementation in an accurate manner, then the problem is that your Tcl proc named doSomething is replaced by the C-implemented Tcl command once your extension is loaded. Procedures and commands live in the same namespace(s). When the loading order were reversed, the problem would remain the same.
I read that everything is being evaluated by the tcl interperter so in
this case I should name the tcl name of the C wrap functions in
special way for example cFunc. But I am not sure about this.
This is correct. You have to organise the C-implemented commands and their scripted wrappers in a way that their names do not conflict with one another. Some (basic) options:
Use two different Tcl namespaces, with same named procedures
Apply some naming conventions to wrapper procs and commands (your cFunc hint)
If your API were provided as actual Itcl or TclOO objects, and the individual commands were the methods, you could use a subclass or a mixin to host refinements (using the super-reference, such as next in TclOO, to forward from the scripted refinement to the C implementations).
A hot-fix solution in your current setup, which is better replaced by some actual design, would be to rename or interp hide the conflicting commands:
load somefile.o
Hide the now available commands: interp hide {} doSomething
Define a scripted wrapper, calling the hidden original at some point:
For example:
proc doSomething {args} {
# argument checking
set temp [interp invokehidden {} doSomething {*}$args]
# result checking
return $temp
}
This is a fairly niche problem, but I'm currently trying to write a conventions-based settings storage library with golang. It would be a great API boon if I could programmatically determine the running package name that wants to store something (eg "github.net/author/projectname/pkg") calling my library function.
With Python a similar thing could be achieved with the inspect module, or even with __main__.__file__ and a look at the file system.
You can get similar information if you use the following functions:
runtime.Caller
runtime.FuncForPC
The code may look like this:
pc, file, line, ok := runtime.Caller(1)
if !ok { /*failed*/ }
println(pc, file, line, ok)
f := runtime.FuncForPC(pc)
if f == nil { /*failed*/ }
println(f.Name())
If I put the above code (with the 1st line changed into runtime.Caller(0)) into a (randomly chosen) Go library which I have installed in GOROOT, it prints:
134626026 /tmp/go-build223663414/github.com/mattn/go-gtk/gtk/_obj/gtk.cgo1.go -4585 true
github.com/mattn/go-gtk/gtk.Init
Or it prints:
134515752 /home/user/go/src/github.com/mattn/go-gtk/example/event/event.go 12 true
main.main
The filename on the 1st line, and the 2nd line, seem to contain the information you are looking for.
There are two problems:
It may give incorrect result if functions are automatically inlined by the compiler
For any function F defined in package main, the function name is just main.F. For example, if runtime.Caller(0) is called from main(), the function name is main.main even if the main() function is defined in a Go file found in GOROOT/src/github.com/mattn/go-gtk/.... In this case, the output from runtime.Caller is more useful than the output from runtime.FuncForPC.
I'm learning SciLab and I need to figure out the equivalent from MATLAB for running user-defined functions.
I'm used to MATLAB, where when you type foo(27), it looks for the foo.m script in the current directory and then the MATLAB path, and if it finds one, it calls that function with argument 27.
What's the equivalent to SciLab? It doesn't seem to want to look in the current directory for the appropriate .sci file.
In Scilab, you need to explicitly load the script that contains the function. Assuming you've changed your directory to the directory where the function file is loaded, which can be done in Scilab using the menu buttons or the following command:
cd("path/to/working/directory")
Now you load the function file. Assuming the function foo is stored in a file called foo.sci, you need to load this script using the following command:
exec("foo.sci")
Now you should be able to use your function as you would be able in MATLAB.
foo(27)
In my language I can use a class variable in my method when the definition appears below the method. It can also call methods below my method and etc. There are no 'headers'. Take this C# example.
class A
{
public void callMethods() { print(); B b; b.notYetSeen();
public void print() { Console.Write("v = {0}", v); }
int v=9;
}
class B
{
public void notYetSeen() { Console.Write("notYetSeen()\n"); }
}
How should I compile that? what i was thinking is:
pass1: convert everything to an AST
pass2: go through all classes and build a list of define classes/variable/etc
pass3: go through code and check if there's any errors such as undefined variable, wrong use etc and create my output
But it seems like for this to work I have to do pass 1 and 2 for ALL files before doing pass3. Also it feels like a lot of work to do until I find a syntax error (other than the obvious that can be done at parse time such as forgetting to close a brace or writing 0xLETTERS instead of a hex value). My gut says there is some other way.
Note: I am using bison/flex to generate my compiler.
My understanding of languages that handle forward references is that they typically just use the first pass to build a list of valid names. Something along the lines of just putting an entry in a table (without filling out the definition) so you have something to point to later when you do your real pass to generate the definitions.
If you try to actually build full definitions as you go, you would end up having to rescan repatedly, each time saving any references to undefined things until the next pass. Even that would fail if there are circular references.
I would go through on pass one and collect all of your class/method/field names and types, ignoring the method bodies. Then in pass two check the method bodies only.
I don't know that there can be any other way than traversing all the files in the source.
I think that you can get it down to two passes - on the first pass, build the AST and whenever you find a variable name, add it to a list that contains that blocks' symbols (it would probably be useful to add that list to the corresponding scope in the tree). Step two is to linearly traverse the tree and make sure that each symbol used references a symbol in that scope or a scope above it.
My description is oversimplified but the basic answer is -- lookahead requires at least two passes.
The usual approach is to save B as "unknown". It's probably some kind of type (because of the place where you encountered it). So you can just reserve the memory (a pointer) for it even though you have no idea what it really is.
For the method call, you can't do much. In a dynamic language, you'd just save the name of the method somewhere and check whether it exists at runtime. In a static language, you can save it in under "unknown methods" somewhere in your compiler along with the unknown type B. Since method calls eventually translate to a memory address, you can again reserve the memory.
Then, when you encounter B and the method, you can clear up your unknowns. Since you know a bit about them, you can say whether they behave like they should or if the first usage is now a syntax error.
So you don't have to read all files twice but it surely makes things more simple.
Alternatively, you can generate these header files as you encounter the sources and save them somewhere where you can find them again. This way, you can speed up the compilation (since you won't have to consider unchanged files in the next compilation run).
Lastly, if you write a new language, you shouldn't use bison and flex anymore. There are much better tools by now. ANTLR, for example, can produce a parser that can recover after an error, so you can still parse the whole file. Or check this Wikipedia article for more options.
Aside from getting any real work done, I have an itch. My itch is to write a view engine that closely mimics a template system from another language (Template Toolkit/Perl). This is one of those if I had time/do it to learn something new kind of projects.
I've spent time looking at CoCo/R and ANTLR, and honestly, it makes my brain hurt, but some of CoCo/R is sinking in. Unfortunately, most of the examples are about creating a compiler that reads source code, but none seem to cover how to create a processor for templates.
Yes, those are the same thing, but I can't wrap my head around how to define the language for templates where most of the source is the html, rather than actual code being parsed and run.
Are there any good beginner resources out there for this kind of thing? I've taken a ganer at Spark, which didn't appear to have the grammar in the repo.
Maybe that is overkill, and one could just test-replace template syntax with c# in the file and compile it. http://msdn.microsoft.com/en-us/magazine/cc136756.aspx#S2
If you were in my shoes and weren't a language creating expert, where would you start?
The Spark grammar is implemented with a kind-of-fluent domain specific language.
It's declared in a few layers. The rules which recognize the html syntax are declared in MarkupGrammar.cs - those are based on grammar rules copied directly from the xml spec.
The markup rules refer to a limited subset of csharp syntax rules declared in CodeGrammar.cs - those are a subset because Spark only needs to recognize enough csharp to adjust single-quotes around strings to double-quotes, match curley braces, etc.
The individual rules themselves are of type ParseAction<TValue> delegate which accept a Position and return a ParseResult. The ParseResult is a simple class which contains the TValue data item parsed by the action and a new Position instance which has been advanced past the content which produced the TValue.
That isn't very useful on it's own until you introduce a small number of operators, as described in Parsing expression grammar, which can combine single parse actions to build very detailed and robust expressions about the shape of different syntax constructs.
The technique of using a delegate as a parse action came from a Luke H's blog post Monadic Parser Combinators using C# 3.0. I also wrote a post about Creating a Domain Specific Language for Parsing.
It's also entirely possible, if you like, to reference the Spark.dll assembly and inherit a class from the base CharGrammar to create an entirely new grammar for a particular syntax. It's probably the quickest way to start experimenting with this technique, and an example of that can be found in CharGrammarTester.cs.
Step 1. Use regular expressions (regexp substitution) to split your input template string to a token list, for example, split
hel<b>lo[if foo]bar is [bar].[else]baz[end]world</b>!
to
write('hel<b>lo')
if('foo')
write('bar is')
substitute('bar')
write('.')
else()
write('baz')
end()
write('world</b>!')
Step 2. Convert your token list to a syntax tree:
* Sequence
** Write
*** ('hel<b>lo')
** If
*** ('foo')
*** Sequence
**** Write
***** ('bar is')
**** Substitute
***** ('bar')
**** Write
***** ('.')
*** Write
**** ('baz')
** Write
*** ('world</b>!')
class Instruction {
}
class Write : Instruction {
string text;
}
class Substitute : Instruction {
string varname;
}
class Sequence : Instruction {
Instruction[] items;
}
class If : Instruction {
string condition;
Instruction then;
Instruction else;
}
Step 3. Write a recursive function (called the interpreter), which can walk your tree and execute the instructions there.
Another, alternative approach (instead of steps 1--3) if your language supports eval() (such as Perl, Python, Ruby): use a regexp substitution to convert the template to an eval()-able string in the host language, and run eval() to instantiate the template.
There are sooo many thing to do. But it does work for on simple GET statement plus a test. That's a start.
http://github.com/claco/tt.net/
In the end, I already had too much time in ANTLR to give loudejs' method a go. I wanted to spend a little more time on the whole process rather than the parser/lexer. Maybe in version 2 I can have a go at the Spark way when my brain understands things a little more.
Vici Parser (formerly known as LazyParser.NET) is an open-source tokenizer/template parser/expression parser which can help you get started.
If it's not what you're looking for, then you may get some ideas by looking at the source code.