I've inherited a collection of largely undocumented ssis packages. The entry point package (ie: the one that forks off in a variety of directions to call other packages) defines a number of variables. I would like to know how these variables are being used, but there doesn't seem to be an equivalent of "right click/Find All References"
Is there a reliable way to determine where these variables are being used?
A hackish way would be to open the dtsx file in a text editor/xml viewer and search for the variable name.
If it's being used in expressions, it should show it and you can trace the xml tree back up until you find the object it's being used on.
You can use the bids helper add-in thats gives you visual feedback on where variables are used in your package. Thats makes it very fast and easy to detect them.Besides that, it offers several other valueable features.
Check out: http://bidshelper.codeplex.com/
Related
I am trying to figure out a way to access the Items from the context without having to declare an ItemGroup explicitly.
Currently trying for the Copy task:
<Copy SourceFiles="C:\blabla\**\*.*" DestinationFiles="%(?.RecursiveDir)" />
What can I use in place of "?" to select the Items in context ?
The reason is that, I have an MSBuild project file being generated via XSLT and there are unknown number of folders & files (some of them follow a different structure under the destination folder - in that case I intend to use different meta data in place of RecursiveDir) in the input XML. Is it possible to achieve this without the need to declare loads of Itemgroups (or an Itemgroup with lots of Items) ?
I tried searching for this, but all I found were posts with Itemgroups declared.
#Alexey Shcherbak wrote:
You want to refer item metadata without explicitly declaring item itself, so I have doubts you will be able to do this. Also Copy task require that Source Files should be ITaskItem[] type (literally - it require item collection). Actually the msdn description of copy task has an exact example you could follow, but you should declare itemgroup with nested items clause inside.
You may wonder if you have an item with a lot of files, does it make MSBuild slow down.
The answer is: it depends =). What numbers you mean under huge fileset =). It's true that MSBuild engine emits and evaluates each item group in memory, and probably a huge fileset could lead to a bigger memory footprint. But MSBuild is not adapted to work as your scripting language of choice (even powershell has issues with 250K+ files in one dir, windows itself also). If you just need perform a copy without accessing to full meta (except recursive dir)- use Exec task and invoke robocopy.exe - it works waaaay better than anything else (considering available out of the box tools).
As an addition - huge numbers should be tested and evaluated before we declare that concrete tool isn't acceptable for that. I think as soon as MSBuild could deal with big solutions - it could probably deal with pretty big filesets. It just the resource/speed question. But any tool also have it's unsurmountable limits.
Actually I meant not robocopy extension, but robocopy.exe itself en.wikipedia.org/wiki/Robocopy, you can easily call it with Exec task. And surely hardlinks are unbeatable in terms of "copy" speed =). But keep in mind - it will work only across single disk volume (because it's not actual copy, it just adding another file name to same set of bytes =) ). In case you need actual copy to another drive or over the network - robocopy will shine again =).
PS: 20k files are far from my definition of huge ;) We dealt with ~280k-300k small files, summary volume around 80Gb. Powershell for plumbing and robocopy for actual bits-moving won that round.
As suggested by the Eclipse documentation, I have an org.eclipse.core.resources.IncrementalProjectBuilder that compiles each source file and separately I also have a org.eclipse.ui.editors.text.TextEditor that can edit each source file. Each source file is compiled into its own compilation unit, but it can reference types from other (already compiled) source files.
Two tasks for which this is important are:
Compiling (to make sure the types we're using actually exist)
Autocomplete (to look up the type so we can see what properties/methods are present on it)
To accomplish this, I want to store a representation of all the compiled types in memory (referred to below as my "type store").
My question is two fold:
Task one above is performed by the builder and task two by the editor. So that they both have access to this type store, should I create a static store somewhere that they both can have access to, or does Eclipse provide a neater way to deal with this problem? Note that it is eclipse, not me, that instantiates the builders and editors when they are needed.
When opening eclipse, I don't want to have to rebuild the whole project just so I can re-populate my type store. My best solution so far is to persist this data somewhere and then repopulate my store from that (perhaps upon project open). Is this how other incremental compilers typically do this? I believe Java's approach is to use a special parser that efficiently extracts this data from the class files.
Any insights would be really appreciated. This is my first DSL.
This is an interesting question and one that doesn't have a simple solution. I'll try to describe a potential solution and also describe in a little bit more detail how JDT accomplishes incremental compilation.
First, a bit about JDT:
Yes, JDT does read class files for some of its information, but only for libraries that don't have source code. And this information is really only used for editing assistance (content assist, navigation, etc).
JDT computes incremental compilation by keeping track of dependencies between compilation units as they are compiled. This state information is stored on disk and retrieved and updated after each compile.
As a more complete example, let's say that after a full build, JDT determines that A.java depends on B.java, which depends on C.java.
If there is a structural change in C.java (a structural change is a change that can affect outside files (e.g., adding/removing a non-private field or method)), then B.java will be recompiled. A.java will not be recompiled since there was no structural change in B.java.
After this bit of clarification on how JDT works, here are some possible answers to your questions:
Yes. This must be done through statically accessible global objects. JDT does this through the JavaCore and JavaModelManager objects. If you don't want to use global singletons, then you can access to your type store available through your plugin's Bundle activator instance. The e4 project does allow dependency injection, which is probably even better (but is not really a part of the core Eclipse APIs).
I think persisting the information on the file system is your best bet. The only real way to determine incremental compile dependencies is to do a full build, so you need to persist the information somewhere. Again, this is how JDT does it. The information is stored in your workspaces' .metadata directory somewhere in the org.eclipse.core.resources plugin. You can have a look at the org.eclipse.jdt.internal.core.builder.State class to see the implementation.
So, this may not be the answer you are looking for, but I think this is the most promising way to approach your problem.
Supposing I have a merge module that installs a file "MyFile.txt" to a certain location, and that I wish to use that merge module, however I want to supply a different copy of "MyFile.txt" from the one supplied with the merge module.
Is it possible to do this? (And for bonus points how can I do this using Wix)
Update: Roughly speaking MyFile.txt is part of a package up component of installable items that we provide to others, they then comine these components with their own to produce an installer.
In the ideal world they would only need to add new files to the output, however this is a replacement for an existing system where they currently have the ability to modify or even replace items (suce as MyFile.txt) in the end installer, and so without the ability to do the same with the merge module the migration path will be difficult.
The packaged up component doesn't need to be a merge module if there is a better solution, however merge modules seemed like the sensible choice and in all other respects provide a very nice re-usable package of installer logic.
It's possible but every technique that I know is a bit of a hack and doesn't scale very well. Can you tell me more about what type of file MyFile.txt is and what the intent of the different flavors of the file? Usually my goal is to never have the same filename twice ( darn component rules ) and then design variation points to support the needs. Sometimes upstream changes to the application are required to do this correctly.
What is a tool or technique that can be used to perform spell checks upon a whole source code base and its associated resource files?
The spell check should be source code aware meaning that it would stick to checking string literals in the code and not the code itself. Bonus points if the spell checker understands common resource file formats, for example text files containing name-value pairs (only check the values). Super-bonus points if you can tell it which parts of an XML DTD or Schema should be checked and which should be ignored.
Many IDEs can do this for the file you are currently working with. The difference in what I am looking for is something that can operate upon a whole source code base at once.
Something like a Findbugs or PMD type tool for mis-spellings would be ideal.
As you mentioned, many IDEs have this functionality already, and one such IDE is Eclipse. However, unlike many other IDEs Eclipse is:
A) open source
B) designed to be programmable
For instance, here's an article on using Eclipse's code formatting functionality from the command line:
http://www.peterfriese.de/formatting-your-code-using-the-eclipse-code-formatter/
In theory, you should be able to do something similar with it's spell-checking mechanism. I know this isn't exactly what you're looking for, and if there is a program for doing spell-checking in code then obviously that'd be better, but if not then Eclipse may be the next best thing.
This seems little old but seems to do a good job
Source Code Spell Checker
I'm producing a hex file to run on an ARM processor which I want to keep below 32K. It's currently a lot larger than that and I wondered if someone might have some advice on what's the best approach to slim it down?
Here's what I've done so far
So I've run 'size' on it to determine how big the hex file is.
Then 'size' again to see how big each of the object files are that link to create the hex files. It seems the majority of the size comes from external libraries.
Then I used 'readelf' to see which functions take up the most memory.
I searched through the code to see if I could eliminate calls to those functions.
Here's where I get stuck, there's some functions which I don't call directly (e.g. _vfprintf) and I can't find what calls it so I can remove the call (as I think I don't need it).
So what are the next steps?
Response to answers:
As I can see there are functions being called which take up a lot of memory. I cannot however find what is calling it.
I want to omit those functions (if possible) but I can't find what's calling them! Could be called from any number of library functions I guess.
The linker is working as desired, I think, it only includes the relevant library files. How do you know if only the relevant functions are being included? Can you set a flag or something for that?
I'm using GCC
General list:
Make sure that you have the compiler and linker debug options disabled
Compile and link with all size options turned on (-Os in gcc)
Run strip on the executable
Generate a map file and check your function sizes. You can either get your linker to generate your map file (-M when using ld), or you can use objdump on the final executable (note that this will only work on an unstripped executable!) This won't actually fix the problem, but it will let you know of the worst offenders.
Use nm to investigate the symbols that are called from each of your object files. This should help in finding who's calling functions that you don't want called.
In the original question was a sub-question about including only relevant functions. gcc will include all functions within every object file that is used. To put that another way, if you have an object file that contains 10 functions, all 10 functions are included in your executable even if one 1 is actually called.
The standard libraries (eg. libc) will split functions into many separate object files, which are then archived. The executable is then linked against the archive.
By splitting into many object files the linker is able to include only the functions that are actually called. (this assumes that you're statically linking)
There is no reason why you can't do the same trick. Of course, you could argue that if the functions aren't called the you can probably remove them yourself.
If you're statically linking against other libraries you can run the tools listed above over them too to make sure that they're following similar rules.
Another optimization that might save you work is -ffunction-sections, -Wl,--gc-sections, assuming you're using GCC. A good toolchain will not need to be told that, though.
Explanation: GNU ld links sections, and GCC emits one section per translation unit unless you tell it otherwise. But in C++, the nodes in the dependecy graph are objects and functions.
On deeply embedded projects I always try to avoid using any standard library functions. Even simple functions like "strtol()" blow up the binary size. If possible just simply avoid those calls.
In most deeply embedded projects you don't need a versatile "printf()" or dynamic memory allocation (many controllers have 32kb or less RAM).
Instead of just using "printf()" I use a very simple custom "printf()", this function can only print numbers in hexadecimal or decimal format not more. Most data structures are preallocated at compile time.
Andrew EdgeCombe has a great list, but if you really want to scrape every last byte, sstrip is a good tool that is missing from the list and and can shave off a few more kB.
For example, when run on strip itself, it can shave off ~2kB.
From an old README (see the comments at the top of this indirect source file):
sstrip is a small utility that removes the contents at the end of an
ELF file that are not part of the program's memory image.
Most ELF executables are built with both a program header table and a
section header table. However, only the former is required in order
for the OS to load, link and execute a program. sstrip attempts to
extract the ELF header, the program header table, and its contents,
leaving everything else in the bit bucket. It can only remove parts of
the file that occur at the end, after the parts to be saved. However,
this almost always includes the section header table, and occasionally
a few random sections that are not used when running a program.
Note that due to some of the information that it removes, a sstrip'd executable is rumoured to have issues with some tools. This is discussed more in the comments of the source.
Also... for an entertaining/crazy read on how to make the smallest possible executable, this article is worth a read.
Just to double-check and document for future reference, but do you use Thumb instructions? They're 16 bit versions of the normal instructions. Sometimes you might need 2 16 bit instructions, so it won't save 50% in code space.
A decent linker should take just the functions needed. However, you might need compiler & linke settings to package functions for individual linking.
Ok so in the end I just reduced the project to it's simplest form, then slowly added files one by one until the function that I wanted to remove appeared in the 'readelf' file. Then when I had the file I commented everything out and slowly add things back in until the function popped up again. So in the end I found out what called it and removed all those calls...Now it works as desired...sweet!
Must be a better way to do it though.
To answer this specific need:
•I want to omit those functions (if possible) but I can't find what's
calling them!! Could be called from any number of library functions I
guess.
If you want to analyze your code base to see who calls what, by whom a given function is being called and things like that, there is a great tool out there called "Understand C" provided by SciTools.
https://scitools.com/
I have used it very often in the past to perform static code analysis. It can really help to determine library dependency tree. It allows to easily browse up and down the calling tree among other things.
They provide a limited time evaluation, then you must purchase a license.
You could look at something like executable compression.