I'm writing my own compiler and I'm struggling to implement a module system.
Can someone guide me, how should this be done? how other languages tackle this?
Also I'm trying to avoid what c and c++ do (header files).
I do like the module system in Go/Golang though.
I don't know if this is relevant, but I'm using LLVM (maybe there's a magic way to import symbols).
my initial approach:
read and parse the entry point source file ie. main.mylang.
go through the imports of main.mylang
for each import: read, parse and resolve it's imports
...
this leads to a tree structure:
main.mylang: import1.mylang, import2.mylang, import3.mylang
import1.mylang: import4.mylang, import5.mylang
import2.mylang: import6.mylang
... etc.
then I would traverse each node and copy it's symbols (functions, global variables, etc.) to the parent node's symbol table. if a parent node is null, it's an entry point file and the compiler can start output object files.
why do I think that this is bad?
it's very slow, even when compiling 3-5 source files
it's easy to cause name collisions
you have to import the entire symbol table, because the imported file's exported symbols depend on the internal ones.
for example: imagine an exported function that modifies an internal global variable
Thanks in advance
I think your approach is really good. Compile time speed is not that important, usability is. To prevent name collisions you can use some kind of module-namespace (importname.foo() instead of just foo()) and whenever foo does not exist allow both methods. Alternatively you could insert a placeholder in the parents symbol table and whenever the user uses that name you throw a compile time error (something like ambiguous symbol).
that would look like this:
main.mylang
import module1
import module2
int main() {}
module1.mylang
import module2
void foo() {}
void bar() {}
module2.mylang
import module1
void bar() {}
void fun() {}
After finding loops, the tree would look like this:
main
├──module1
│ └──module2
└──module2
└──module1
And a graph like this:
main
├─>main()
├─>foo() (module1)
├─>bar() (defined twice, throw error when used)
├─>fun() (module2)
├─>module1<───────────┐
│ ├─>foo() (module1) │
│ └─>bar() (module1) │
└─>import2<───────────┘
├─>bar() (module2)
└─>fun() (module2)
I don't know much about llvm, but I am pretty sure normal tables are not enough to archive this. You will at least need nested tables if not even a graph like structure like I described. Also this is not possible with classical C/C++ architecture, except if you use unique identifiers as symbols and don't let the user know (like c++ function overloading). For example you could call one function __import1_bar and the other __import2_bar and whenever the user uses bar() you look up in this graph which one he wants to call. In the main function using import1.bar() will lead you to __import1_bar (follow the graph) and import2.bar() or import1.import2.bar() will lead you to __import2_bar.
Good Luck figuring that out. But it is certainly a interesting problem.
I need to test different implementations of the same function from one single unit test file, with different files holding different implementations.
The problem is that with normal requires, everything is imported without a name space, so functions with the same name cannot co-exist.
Is there a way to bind imported modules to a name space? Or to rename the elements of an imported module?
Something like
; All files contain a function named "fun" that needs to be tested.
(require
(rename "implementation-one.rkt" fun fun-one)
(rename "implementation-two.rkt" fun fun-two)
(rename "implementation-three.rkt" fun fun-three))
(check-equal? (fun-one 0) 0 "first implementation")
(check-equal? (fun-two 0) 0 "second implementation")
(check-equal? (fun-three 0) 0 "third implementation")
Here I'm importing the fun function from each implementation file, and renaming it so something else. This is not valid Racket though.
I looked into modules, but it seems that the same problem happens; when requiring a module everything inside is exposed without a name space.
Is there a way to specify the name of an imported module, or change the names of the contents of the module? Perhaps by renaming the module's bindings, or by encapsulating imports in a name space, or some sort of named import? How are name conflicts like this normally resolved in Racket?
Should modules contain structs and/or objects that encapsulate each implementation? Or is there a way to do it with simple modules that provide functions?
(require
(rename-in "implementation-one.rkt" [fun fun-one])
(rename-in "implementation-two.rkt" [fun fun-two])
(rename-in "implementation-three.rkt" [fun fun-three]))
This is not hard to find in the manual.
I have 4 files all in the same directory: main.rakumod, infix_ops.rakumod, prefix_ops.rakumod and script.raku:
main module has a class definition (class A)
*_ops modules have some operator routine definitions to write, e.g., $a1 + $a2 in an overloaded way.
script.raku tries to instantaniate A object(s) and use those user-defined operators.
Why 3 files not 1? Since class definition might be long and separating overloaded operator definitions in files seemed like a good idea for writing tidier code (easier to manage).
e.g.,
# main.rakumod
class A {
has $.x is rw;
}
# prefix_ops.rakumod
use lib ".";
use main;
multi prefix:<++>(A:D $obj) {
++$obj.x;
$obj;
}
and similar routines in infix_ops.rakumod. Now, in script.raku, my aim is to import main module only and see the overloaded operators also available:
# script.raku
use lib ".";
use main;
my $a = A.new(x => -1);
++$a;
but it naturally doesn't see ++ multi for A objects because main.rakumod doesn't know the *_ops.rakumod files as it stands. Is there a way I can achieve this? If I use prefix_ops in main.rakumod, it says 'use lib' may not be pre-compiled perhaps because of circular dependentness
it says 'use lib' may not be pre-compiled
The word "may" is ambiguous. Actually it cannot be precompiled.
The message would be better if it said something to the effect of "Don't put use lib in a module."
This has now been fixed per #codesections++'s comment below.
perhaps because of circular dependentness
No. use lib can only be used by the main program file, the one directly run by Rakudo.
Is there a way I can achieve this?
Here's one way.
We introduce a new file that's used by the other packages to eliminate the circularity. So now we have four files (I've rationalized the naming to stick to A or variants of it for the packages that contribute to the type A):
A-sawn.rakumod that's a role or class or similar:
unit role A-sawn;
Other packages that are to be separated out into their own files use the new "sawn" package and does or is it as appropriate:
use A-sawn;
unit class A-Ops does A-sawn;
multi prefix:<++>(A-sawn:D $obj) is export { ++($obj.x) }
multi postfix:<++>(A-sawn:D $obj) is export { ($obj.x)++ }
The A.rakumod file for the A type does the same thing. It also uses whatever other packages are to be pulled into the same A namespace; this will import symbols from it according to Raku's standard importing rules. And then relevant symbols are explicitly exported:
use A-sawn;
use A-Ops;
sub EXPORT { Map.new: OUTER:: .grep: /'fix:<'/ }
unit class A does A-sawn;
has $.x is rw;
Finally, with this setup in place, the main program can just use A;:
use lib '.';
use A;
my $a = A.new(x => -1);
say $a++; # A.new(x => -1)
say ++$a; # A.new(x => 1)
say ++$a; # A.new(x => 2)
The two main things here are:
Introducing an (empty) A-sawn package
This type eliminates circularity using the technique shown in #codesection's answer to Best Way to Resolve Circular Module Loading.
Raku culture has a fun generic term/meme for techniques that cut through circular problems: "circular saws". So I've used a -sawn suffix of the "sawn" typename as a convention when using this technique.[1]
Importing symbols into a package and then re-exporting them
This is done via sub EXPORT { Map.new: ... }.[2] See the doc for sub EXPORT.
The Map must contain a list of symbols (Pairs). For this case I've grepped through keys from the OUTER:: pseudopackage that refers to the symbol table of the lexical scope immediately outside the sub EXPORT the OUTER:: appears in. This is of course the lexical scope into which some symbols (for operators) have just been imported by the use Ops; statement. I then grep that symbol table for keys containing fix:<; this will catch all symbol keys with that string in their name (so infix:<..., prefix:<... etc.). Alter this code as needed to suit your needs.[3]
Footnotes
[1] As things stands this technique means coming up with a new name that's different from the one used by the consumer of the new type, one that won't conflict with any other packages. This suggests a suffix. I think -sawn is a reasonable choice for an unusual and distinctive and mnemonic suffix. That said, I imagine someone will eventually package this process up into a new language construct that does the work behind the scenes, generating the name and automating away the manual changes one has to make to packages with the shown technique.
[2] A critically important point is that, if a sub EXPORT is to do what you want, it must be placed outside the package definition to which it applies. And that in turn means it must be before a unit package declaration. And that in turn means any use statement relied on by that sub EXPORT must appear within the same or outer lexical scope. (This is explained in the doc but I think it bears summarizing here to try head off much head scratching because there's no error message if it's in the wrong place.)
[3] As with the circularity saw aspect discussed in footnote 1, I imagine someone will also eventually package up this import-and-export mechanism into a new construct, or, perhaps even better, an enhancement of Raku's built in use statement.
Hi #hanselmann here is how I would write this (in 3 files / same dir):
Define my class(es):
# MyClass.rakumod
unit module MyClass;
class A is export {
has $.x is rw;
}
Define my operators:
# Prefix_Ops.rakumod
unit module Prefix_Ops;
use MyClass;
multi prefix:<++>(A:D $obj) is export {
++$obj.x;
$obj;
}
Run my code:
# script.raku
use lib ".";
use MyClass;
use Prefix_Ops;
my $a = A.new(x => -1);
++$a;
say $a.x; #0
Taking my cue from the Module docs there are a couple of things I am doing different:
Avoiding the use of main (or Main, or MAIN) --- I am wary that MAIN is a reserved name and just want to keep clear of engaging any of that (cool) machinery
Bringing in the unit module declaration at the top of each 'rakumod' file ... it may be possible to use bare files in Raku ... but I have never tried this and would say that it is not obvious from the docs that it is even possible, or supported
Now since I wanted this to work first time you will note that I use the same file name and module name ... again it may be possible to do that differently (multiple modules in one file and so on) ... but I have not tried that either
Using the 'is export' trait where I want my script to be able to use these definitions ... as you will know from close study of the docs ;-) is that each module has it's own namespace (the "stash") and we need export to shove the exported definitions into the namespace of the script
As #raiph mentions you only need the script to define the module library location
Since you want your prefix multi to "know" about class A then you also need to use MyClass in the Prefix_Ops module
Anyway, all-in-all, I think that the raku module system exemplifies the unique combination of "easy things easy and hard thinks doable" ... all I had to do with your code (which was very close) was tweak a few filenames and sprinkle in some concise concepts like 'unit module' and 'is export' and it really does not look much different since raku keeps all the import/export machinery under the surface like the swan gliding over the river...
There are two packages I want to use: CorpusLoaders.jl, and WordNet.jl
CorpusLoaders.SemCor exports sensekey(::SenseTaggedWord)
WordNet exports sensekey(::DB, ::Synset, ::Lemma)
I want to use both sensekey methods.
Eg
for some mixed list of items: mixedlist::Vector{Union{Tuple{SenseTaggedWord},Tuple{DB, Synset,Lemma}}.
Ie the items in the list are a mixture of 1-tuples of SenseTaggedWord, and3 tuples of DB, Synset, and Lemma.
for item in mixedlist
println(sensekey(item...)
end
should work.
This example is a little facetious, since why would I be mixing them like this.
But, hopefully it serves for illustrating the problem in the general case.
Trying to using CorpusLoaders.SemCor, WordNet to bring in both results in WARNING: both WordNet and Semcor export "sensekey"; uses of it in module Main must be qualified.
Manually importing both: import CorpusLoaders.SemCor.sensekey; import WordNet.sensekey results in WARNING: ignoring conflicting import of Semcor.sensekey into Main
What can be done? I want them both, and they don't really conflict, due to multiple-dispatch.
Given that CorpusLoaders.jl is a package I am writing I do have a few more options, since I could make my CorpusLoaders.jl depend on WordNet.jl.
If I did do than then I could say in CorpusLoaders.jl
import WordNet
function WordNet.sensekey(s::SenseTaggedWord)...
and that would make them both work.
But it would mean requiring WordNet as a dependency of CorpusLoaders.
And I want to know how to solve the problem for a consumer of the packages -- not as the creator of the packages.
tl;dr qualify the functions when using them in your script via their module namespace, i.e. CorpusLoader.sensekey() and WordNet.sensekey()
Explanation
My understanding of your question after the edits (thank you for clarifying) is that:
You have written a package called CorpusLoaders.jl, which exports the function sensekey(::SenseTaggedWord)
There is an external package called WordNet.jl, which exports the function sensekey(::DB, ::Synset, ::Lemma)
You have a script that makes use of both modules.
and you are worried that using the modules or "importing" the functions directly could potentially create ambiguity and / or errors in your script, asking
how can I write my CorpusLoaders package to prevent potential clashes with other packages, and
how can I write my script to clearly disambiguate between the two functions while still allowing their use?
I think this stems from a slight confusion how using and import are different from each other, and how modules create a namespace. This is very nicely explained in the docs here.
In essence, the answers are:
You should not worry about exporting things from your module that will clash with other modules. This is what modules are for: you're creating a namespace, which will "qualify" all exported variables, e.g. CorpusLoaders.sensekey(::SenseTaggedWord).
When you type using CorpusLoaders, what you're saying to julia is "import the module itself, and all the exported variables stripped from their namespace qualifier, and bring them into Main". Note that this means you now have access to sensekey as a function directly from Main without a namespace qualifier, and as CorpusLoaders.sensekey(), since you've also imported the module as a variable you can use.
If you then try using the module WordNet as well, julia very reasonably issues a warning, which essentially says:
"You've imported two functions that have the same name. I can't just strip their namespace off because that could create problems in some scenarios (even though in your case it wouldn't because they have different signatures, but I couldn't possibly know this in general). If you want to use either of these functions, please do so using their appropriate namespace qualifier".
So, the solution for 2. is:
you either do
using CorpusLoaders;
using WordNet;
, disregarding the warning, to import all other exported variables as usual in your Main namespace, and access those particular functions directly via their modules as CorpusLoaders.sensekey() and WordNet.sensekey() each time you need to use them in your script, or
you keep both modules clearly disambiguated at all times by doing
import CorpusLoaders;
import WordNet;
and qualify all variables appropriately, or
in this particular case where the function signatures don't clash, if you'd really like to be able to use the function without a namespace qualifier, relying on multiple dispatch instead, you can do something like what FengYang suggested:
import CorpusLoaders;
import WordNet;
sensekey(a::SenseTaggedWord) = CorpusLoader.sensekey(a);
sensekey(a::DB, b::Synset, c::Lemma) = WordNet.sensekey(a, b, c);
which is essentially a new function, defined on module Main, acting as a wrapper for the two namespace-qualified functions.
In the end, it all comes down to using using vs import and namespaces appropriately for your particular code. :)
As an addendum, code can get very unwieldy with long namespace qualifiers like CorpusLoader and WordNet. julia doesn't have something like python's import numpy as np, but at the same time modules become simple variables on your workspace, so it's trivial to create an alias for them. So you can do:
import CorpusLoaders; const cl = CorpusLoaders;
import Wordnet; const wn = WordNet;
# ... code using both cl.sensekey() and wn.sensekey()
In this case, the functions do not conflict, but in general that is impossible to guarantee. It could be the case that a package loaded later will add methods to one of the functions that will conflict. So to be able to use the sensekey for both packages requires some additional guarantees and restrictions.
One way to do this is to ignore both package's sensekey, and instead provide your own, dispatching to the correct package:
sensekey(x) = CorpusLoaders.sensekey(x)
sensekey(x, y, z) = WordNet.sensekey(x,y,z)
I implemented what #Fengyang Wang said,
as a function:
function importfrom(moduleinstance::Module, functionname::Symbol, argtypes::Tuple)
meths = methods(moduleinstance.(functionname), argtypes)
importfrom(moduleinstance, functionname, meths)
end
function importfrom(moduleinstance::Module, functionname::Symbol)
meths = methods(moduleinstance.(functionname))
importfrom(moduleinstance, functionname, meths)
end
function importfrom(moduleinstance::Module, functionname::Symbol, meths::Base.MethodList)
for mt in meths
paramnames = collect(mt.lambda_template.slotnames[2:end])
paramtypes = collect(mt.sig.parameters[2:end])
paramsig = ((n,t)->Expr(:(::),n,t)).(paramnames, paramtypes)
funcdec = Expr(:(=),
Expr(:call, functionname, paramsig...),
Expr(:call, :($moduleinstance.$functionname), paramnames...)
)
current_module().eval(funcdec) #Runs at global scope, from calling module
end
end
Call with:
using WordNet
using CorpusLoaders.Semcor
importfrom(CorpusLoaders.Semcor, :sensekey)
importfrom(WordNet, :sensekey)
methods(sensekey)
2 methods for generic function sensekey:
sensekey(db::WordNet.DB, ss::WordNet.Synset, lem::WordNet.Lemma)
sensekey(saword::CorpusLoaders.Semcor.SenseAnnotatedWord
If you wanted to get really flash you could reexport the DocString too.
I have a static library say "A.lib" which contains a function int foo(). I have another dll say "B.dll" which consumes A.lib and uses the function foo() and also exports some other functions. Is it possible to export the function int foo() (imported from A.lib) from B.dll so that it can be consumed in a third dll say "C.dll".
I want to know whether it is possible or not, I dont want workarounds like making A.lib available to the C.dll. Also, I am not concerned if this is a bad design or not.
Thanks very much for your patience to read this through.
I had the same requirement - just found a different solution:
Assuming that A.lib has an A.h (that is consumed by source files used to build B.dll e.g. assuming that A.h contains prototypes for functions contained in A.lib), just add the following in A.h:
#pragma comment(linker, "/export:_foo")
This will instruct the linker to export foo() when building B.dll. Note the leading underscore - it is there because that's the true name of the symbol for foo() contained in A.lib (use dumpbin /symbols A.lib | findstr foo to see it). In my example foo() was using the __cdecl calling convention, but if you use __stdcall() or compile as C++, you'll get different name decoration, so you'll have to adjust the #pragma statement above as a result.
It doesn't matter if A.h gets included by many source files in B.dll - the linker doesn't complain if the exact same definition is made multiple times.
One "advantage" to this approach is that you don't even have to use the __declspec(dllexport) specifier on foo() in A.lib ...
Yes, it's possible but any code example is language dependent.
(for example in C you may simply export a function with the same name and C.dll will see it)