Not having come from a C/compiled languages background, I'm finding it hard to get to grips with using Go's packages mechanism to create modular code.
In Python, to import a module and get access to it's functions and whatnot, it's a simple case of
import foo
where foo.py is the name of the module you want to import in the same directory. Otherwise you can add an empty __init__.py into a subfolder and access the modules via
from subfolder import foo
You can then access functions by simply referencing them through the module name, e.g. y = foo.bar(y). This makes it easy to separate logical pieces of code from one another.
In Go however, you specify the package name in the source file itself, e.g.
package foo
at the top of the 'foo' module, which you can then supposedly import through
import (
"foo"
)
and then refer to it through that, i.e. y := foo.Bar(x) . But what I can't wrap my head around is how this works in practice. The relevant docs on golang.org seem terse, and directed to people with more (any) experience using makefiles and compilers.
Can someone please clearly explain how you are meant to modularise your code in Go, the right project structure to do so, and how the compilation process works?
Wiki answer, please feel free to add/edit.
Modularization
Multiple files in the same package
This is just what it sounds like. A bunch of files in the same directory that all start with the same package <name> directive means that they are treated as one big set of code by Go. You can transparently call functions in a.go from b.go. This is mostly for the benefit of code organization.
A fictional example would be a "blog" package might be laid out with blog.go (the main file), entry.go, and server.go. It's up to you. While you could write a blog package in one big file, that tends to affect readability.
Multiple packages
The standard library is done this way. Basically you create modules and optionally install them into $GOROOT. Any program you write can import "<name>" and then call <name>.someFunction()
In practice any standalone or shared components should be compiled into packages. Back to the blog package above, If you wanted to add a news feed, you could refactor server.go into a package. Then both blog.go and news.go would both import "server".
Compilation
I currently use gomake with Makefiles. The Go installation comes with some great include files for make that simplify the creation of a package or a command. It's not hard and the best way to get up to speed with these is to just look at sample makefiles from open source projects and read "How to Write Go Code".
In addition to the package organisation, Like pip in python, use dep https://github.com/golang/dep for go package management. if you use it on existing go package it will automatically build the dependency tree with versions for all the packages being used. when shifting to production server, dep ensure will use Gopkg.toml to install all the required packages.
Just use dep ensure -add , other commands for dep are:
Commands:
init Set up a new Go project, or migrate an existing one
status Report the status of the project's dependencies
ensure Ensure a dependency is safely vendored in the project
version Show the dep version information
check Check if imports, Gopkg.toml, and Gopkg.lock are in sync
Examples:
dep init set up a new project
dep ensure install the project's dependencies
dep ensure -update update the locked versions of all dependencies
dep ensure -add github.com/pkg/errors add a dependency to the project
Related
We're planning three similar vue projects. We already know that we will be able to reuse a lot of code (especially vue SFCs and simple js helper functions) in all of them and we're looking for a proper way to share the code between them.
Unfortunately the scope of the projects is rather different and a monorepo is not an option due to its limitations in read / write permission and visibility management. Therefore we're planning to handle the reusable parts as separate repos (and most likely private npm packages) which seems to be a straightforward approach. However, the question is: How can we create a convenient setup in which we are able to work on the shared components from within the scope of one of the parent projects?
Project A [project-repo-a]
project-specific stuff for A
private package A [package-repo-a] (conveniently editable from within project A)
private package B [package-repo-b] (conveniently editable from within project A)
Project B [project-repo-b]
project-specific stuff for B
private package B [package-repo-b] (conveniently editable from within project B)
private package C [package-repo-c] (conveniently editable from within project B)
In our PHP projects, there is a simple solution, we just require the reusable parts via composer with the prefer-source option which provides the full git repository which can be worked on right from within the parent application. However, as far as we understand there is no prefer-source thing in npm or yarn. So how can we achieve the desired setup? (Or are we overlooking a major downside of this setup in general?)
We already looked into / considered the following (without finding a suitable approach):
yarn / npm link: We understood, that we could use linking in general, but this seems to be a very inconvenient approach while constantly developing the shared components (and always having to publish them to reflect the latest changes).
yarn workspaces / lerna: Seem to be closest to what we want, however they seem to be (or are explicitly) designed for a monorepo approach. In the end they don't to provide a solution for actually getting the git source of a package (in a separate repo) into the parent project (since there is no --prefer-source thing) - do they?
using composer additionally: Just pulling the git sources down with composer and creating yarn workspaces from the composer vendor folder. However, this is obviously a hacky way and sounds quite error prone concerning the whole dependency management
using a yarn post-install script to pull down the git source of the required private packages, but as the composer way, this seems to be rather unpredictable in terms of module resolution, dependency management and so on.
using git submodules and yarn workspaces: Could be a solution. To be honest we're just completely unexperienced with git submodules and at a first glance it didn't look very intuitive. If there is no other way, we'll anyways consider to use this approach.
To be clear about this: We're not asking the taste question if one or the other of those approaches would be "best". We're feeling like none of them is the right one. The question is: Are we overlooking a technically clean and proven approach to our scenario, using npm, yarn or another package manager / dependency management solution?
Git X-Modules is a tool designed to do exactly what you were asking about. Here's a video that explains it. However, it's very new and therefore can't be really considered "proven" :-)
Yet, if you consider trying it, we would love to hear your feedback!
(As you may guess from the previous sentence, I am a part of the development team.)
you probably have already figured this out but have you looked into https://bit.dev/ ?
I'm currently considering it for a similar task to yours and it looks like it could do the job. Here's an article explaining how to use it https://blog.bitsrc.io/how-to-easily-share-vue-components-between-applications-1d30a1ad4e4d
I'm in the process of rewriting a legacy CMake setup to use modern features like automatic dependency propagation. (i.e. using things like target_include_directories(<target> PUBLIC <dir>) instead of include_directories(<dir>).) Currently, we manually handle all project dependency information by setting a bunch of global directory properties.
In my testing, I've found a few examples where a target in the new build will link to a library that the old build would not. I'm not linking to it explicitly, so I know this is coming from the target's dependencies, but in order to find which one(s) I have to recursively look through all of the project's CMakeLists.txts, following up the dependency hierarchy until I find one that pulls in the library in question. We have dozens of libraries so this is not a trivial process.
Does CMake provide any way to see, for each target, which of its dependencies were added explicitly, and which ones were propagated through transitive dependencies?
It looks like the --graphviz output does show this distinction, so clearly CMake knows the context internally. However, I'd like to write a tree-like script to show dependency information on the command line, and parsing Graphviz files sounds like both a nightmare and a hack.
As far as I can tell, cmake-file-api does not include this information. I thought the codemodel/target/dependencies field might work, but it lists both local and transitive dependencies mixed together. And the backtrace field of each dependency only ties back to the add_executable/add_library call for the current target.
You can parse dot file generated by graphviz and extract details which you want. Below is sample python script to do that.
import pydot
import sys
graph = pydot.graph_from_dot_file(sys.argv[1])
result = {}
for g in graph:
# print(g)
for node in g.get_node_list():
if node.get("label") != None:
result[node.get("label")] = []
for edge in g.get_edges():
result[g.get_node(edge.get_source())[0].get("label")].append(g.get_node(edge.get_destination())[0].get("label"))
for r in result:
print(r+":"+",".join(result[r]))
You can also add this script to run from cmake as custom target, so you can call it from you build system. You can find sample cmake project here
I have a simple program written in Golang. It's an API. So inside the project folder, there's a folder named cmd containing my main package (used to initialise the app and defines the endpoints for the API). There's also a folder named after my program, containing multiple files from a package also named after my program. This package serves as the model to do all the necessary queries and contains all the types I have defined.
I also created a folder called test. It contains all my test files under the package named test. The problem is that to run the tests, I have to access my main package ! Is there a way to do that in Golang ? I tried simply using import "cmd/main" but of course it doesn't work.
I also had an idea. Perhaps I could move all my initialising functions (in the cmd folder) to the package named after my program. This way I could do a regular import in test. And I create, inside of cmd, a main.go in the main package that serves as the entry point for the compiler.
I'm new to Go so I'm not really confident. Do you think it's the right way ?
Thanks.
EDIT : Apparently some people think this question is a duplicate, but it's not. Here's the explanation I gave in on of the comments :
I read this post before posting, but it didn't answer my question
because in that post the person has his tests in the main package. The
reason why I asked my question is because I don't want to have my
tests in the main package. I'd rather have them all in a test folder
inside the same package.
What you want to do is not not possible in GO (assuming you want to test private functions).
because I don't want to have my tests in the main package. I'd rather
have them all in a test folder inside the same package.
Your code belongs to different package if you move it into different folder.
This is how GO defines packages https://golang.org/doc/code.html#Organization:
Each package consists of one or more Go source files in a single
directory.
This is how your code structured:
main
| -- main.go (package main)
+ -- test
| -- main_test.go (package test)
It is idiomatic to keep tests in the same folder with code. It is normal if language or framework set some rules that developer has to follow. GO is pretty strict about that.
This is how you can organize your code:
main
| -- main.go (package main)
| -- main_test.go (package main_test)
| -- main_private_test.go (package main)
Often it makes sense to test code against its pubic interfaces. The best way to do that that, is to put tests into different package. GO convention is to keep tests in the same folder what leads to using the same package name. There is a workaround for that issue. You can add _test (package main_test) prefix to package name for your tests.
If, that is not possible to test your code using public interfaces, you can add another file with tests and use package main in that file.
I have a product that I'm working on, Foo. It has currently roughly the following filesystem structure. It's composed of several logically-distinct modules. I want to package each of those modules so that I can make dependencies a bit more explicit.
I'd also like to continue being able to do a single checkout, though, and have my single solution, single build-script, etc available to me.
Something like how rspec does it; the rspec package depends on a set of sub-packages that can be individually maintained.
Edit: How best to:
make the modules inter-dependent
make the work-on-many-things-at-once-from-source-control-checkout experience work, in the sense of not duplicating things like build-automation, etc. I want to keep having a single solution so that ReSharper can find unused code throughout (this is a big legacy codebase), for example.
** So changes to a set of modules would require that I increment all of their versions at once, to correctly advance the dependencies.
.
/Foo.git
/module1
/src
/module1
/module1.specs (tests)
/module1.sln
/module1.wrapdesc
/version
/module2
/src
/module2
/module2.specs
/module2.sln
/module2.wrapdesc
/version
/Foo.sln
/Rakefile.rb (I'm using ruby/rake to build)
/Gemfile
/Gemfile.lock
I'm starting to create a MSBuild scripts for my products, and I've encounter a dilema.
The code is divided into around 25 projects, some wll require obfuscation, some will require strong-name signing; others will require linking into a single file.
All these projects should result in 3 products, with 3 setups.
The question at hand is as follow: How do I divide the MSBuild scripts to make most sense?
Do I create a script for each product? do I create a script for each project? Do I have one script for building, another for obfuscation and so on?
I think this is good idea to have script per product.
To minimaze dublication create reusable "sub-scripts" and import them to main script (this could be done with Import directive).
<Import Project="..\Steps\Step1.proj" />
Script per product sounds like the way to go. You may want to consider having any number of shared or base scripts too, to import common build steps. Like Mike Chaliy already mentioned you can then use Import in your product's build script:
<Import Project="..\Shared\Base.proj" />
Another thing you might also want to take advantage of is target and property overriding. It's akin to overriding virtual methods in a .Net class. See the documentation and the MSBuild Team Blog for more details. I know I've taken advantage of this quite often by setting defaults in the included scripts then overriding them as necessary in the product build script in order to customize build behaviour. For instance I often have generated files that are required before the build so I hook those targets into the BuildDependsOn property group. This way my generated files are generated whenever I do an F5 from the IDE, call the build target from the command line or otherwise build the project or solution. Obviously if you have any build steps that run long or only need to be run in special circumstances (like building installers), you'll want to take care about exactly what gets hooked in.