I know variants on this have been asked before (e.g. https://groups.google.com/forum/#!topic/snakemake/4kslVBX2kew), but I don't see a definitive solution.
If I run a long-running and complex Snakemake pipeline with '--notemp' (maybe because I'm debugging), it would be really nice to be able to subsequently run a 'cleanup' command to delete anything that would automatically have been deleted on the first run without --notemp. Is there any easy way of doing this?
The way I'm doing this right now is to re-run after using '--forceall --touch', without '--notemp', such that everything just gets touched, and the temp files then get removed at the end. But it's not ideal to change all the timestamps. Is there a better way?
Jon
Since v5.0.0, --delete-temp-output achieves this.
--delete-temp-output
Remove all temporary files generated by the workflow. Use together with –dry-run to list files without actually deleting anything. Note that this will not recurse into subworkflows.
Default: False
Related
Recently, I've been trying to write to a .PAK file while it is being used by another process in ring 0. This has been a problem for quite a while and i haven't had much success. I am able to use any programming language necessary to accomplish this, but C#/VB.net is preferred. I originally wanted to use a find and replace system when editing, but I will just choose and offset to write to and such instead.
No, I can't just terminate the process then edit; the process must be running. Yes, I obviously know the process with the file handle attached.
No, I can't just run as admin because the process is established in ring 0/the kernel.
I've tried multiple methods including setting the process speed temporarily to 0 to edit then revert, and changing the FileShare and other parameters, none with any success.
One approach which I have been told a lot and which I have no experience in is creating a "Kernel Driver". I'm not sure how to go about this and I cant find much info online so if you think that's is the best method please inform me on how to get started. Any help is appreciated!
Always create a temporary file (a copy of your original file). If you need to process a file within your codes, create a temp file, use the temp file and process that file. So if you need another process, there will be no problem.
So here's whats up,
I am a Bench Technician for an IT company. I find myself repeating the same task over and over when preforming system reloads. I want to write an application where I have all the programs for a reload in one spot, and call them by a button click event. I have tried adding them into the Resources and calling them by Environment.CurrentDirectory+"\Path" to no avail, I get " System cannot find the file specified. When the path is hard coded it works like a charm, but this will obviously not do as it needs to be able to move to any system. I am looking for a way to add the exe's I need and a generic way to call the path. I am not looking for handouts here, I have done my homework on this one and still not found a solution, If I could get someone to -point me in the right direction, it would be awesome.
Since what you have already tried is much saner and easier for the average user to work with than having the files embedded in another executable, I'll explain that method.
CurrentDirectory is where your executable is executed from, Like this:
C:\MyDir> MyOtherDir\MyProgram.exe
CurrentDirectory refers to C:\MyDir in this example.
What you need is the application directory; and according to top answer of this question the most reliable way to get that is using AppDomain.CurrentDomain.BaseDirectory
EDIT: Also consider using Path.DirectorySeparatorChar instead of \.
I know I can use gruntjs but it seems it's suited for simple tasks and I can't figure out to embed more logic into it. Basically, I want to have if this and that for compiling. So i want there to be one main.css, and multiple sectionA.css. And when server starts it pre compiles everything to one.css for each section.
This is just one scenario I felt i couldn't make gruntjs do, I can see doing it but it just makes it populated it doesn't seem right way to do it.
When using the magic function %edit from QtConsole with IPython, the call does not block, and does not execute the saved code. It does however save a temporary file...
I think this is intended behavior due to GUI editors and uncertainty, and whatever that reason is for not being able to communicate with subprocess (pyZMQ?).
What do you suggest as the best way to mix %edit/%run magics?
I would not mind calling two different commands (one to edit, and one after I have saved and execution is safe). But those commands need a way to synchronize this target file location, or someone to persist storage, and probably need some crude form of predicatably generating filenames such that you can edit more than one file at a time, and execute in arbitrarily. Session persistence is not a must.
Would writing my own magic do any good? Hope we can %edit macros soon, that would do well enough to make it work.
you shoudl be able to do %edit filename.py and %run filename.py. The non blocking behavior is expected, and IIRC due to technical reason. Not unsurmountable but difficult.
You could define your own magic if you wish, improvement are welcomed.
Hope we can %edit macros soon, that would do well enough to make it work.
For that too, PR are welcomed. I guess as a workaround/option you can %load macro which would put macro on input n+1 , edit it and redefine it, that might be a good extension for a cell magic %%macro macroname
If you have some executable code on your input (from QtConsole), you can type
%edit 1-5
This fires the editor, creates a temporarily file (automatically managed), and loads your input lines. This is nearly enough, now how to retrieve the name of that temp file pragmatically?
I see the print statement on Stdout, but its not visible to QtConsole AFAIK. Could maybe redirect stdout to catch that line, but that may not be an option anyway if your doing something else with stdout.
If I could retrieve the full pathname that was just created, this would be cake. Store it where some magics will know how to find it. Then issue a followup command when ready,pops the name off the stack, loads it into a macro, and run. All this with 2 input commands and no names to remember (unless you want to find and use that macro again, but for 1 shot stuff...)
How do I catch or retrieve the path of that temporary file?
As part of a release we run a load of PL/SQL scripts against a database. Recently someone left the ; off the end of a line in one script that was called another script so this meant that script did not get run. Because this did not cause an error, it just didn't get run, it took quite a while to track down what had happened.
I want to check the scripts before they are run for lines in them that are missing either a ; at the end or a / on the line after. This is made more complicated as 'lines' in the script could actually span more than one line if it is statement or block of code.
To me this seems like to do this I'm going to have to parse the scripts then check they meet the above.
I've found ANTLR and wonder if this might be a way to do it since there seem to be existing PL/SQL grammars but looks like that's going to be a step learning curve for what's just a simple check.
Does anyone know an easy way or any other tools, eclipse plugins etc that I can use to check for lines in the scripts that are missing either a ; at the end or a / on the line after?
Update
We already do most of the stuff Tom H suggested. The scripts are run into our test server and we have a version table that gets updated at the end. The problem was that the missing semi-colon in the container script meant one script did not get run but the rest including the one to update the version number ran without errors. Therefore the problem only got picked up quite a way into testing. This needed the database restored before running the scripts with the missing semi-colon added so basically resulted in half a day of testing time being lost. If there was a simple way to check this before running the scripts into the test server it could save quite a bit of time.
I agree with MattH that you may be going about this the wrong way. I would just add an insert statement to the end of all of your scripts which insert a "version" row into a table in the database. At the end of your deployment scripts it's then an easy task to check that the version table has all of the correct rows in it.
Also, you should have all of your release scripts being run exactly as they will be in production against your QA server. That's where all of the testing takes place. You never do anything to the server besides what is in your release steps - you only run the release scripts and if those release scripts are ever changed then you refresh the QA server with them and redo testing.
When you go to production your release process has then been fully tested. As a fail safe measure you can also use tools like Red Gate's SQL Compare and SQL Data Compare to check that production matches the QA server. The data compare would only be against certain tables (look-up tables, etc.). If you have data changes to major tables (1M rows, etc.) then you can right a custom script to check that they are correct.
Even if the scripts are different for every release (and not part of a defined source control structure that creates or replaces database objects) I would adopt a practice of breaking the scripts down into the most fundamental units of work per file and deploying them through Ant with the standard sql task. You probably have these types of scripts:
CREATE or REPLACE dbobject...
SQL DML scripts
Anonymous PL/SQL blocks
If you standardize on a consistent statement delimiter (I suggest using "/" since it works with all of the cases above) and set the deployment to fail on error, then Ant will either deploy all of the files or indicate why it couldn't.
I think it would be very difficult to otherwise parse files of one or more SQL and/or PLSQL statements and find missing delimiters if there are no standards on delimiter choice or statements per file.
Just a thought, but are you going about this the wrong way?
I assume, at the file-level, the lack of a semi colon in the file was not a problem? but it only became a problem when run via the batch processing? If that's the case maybe you can change your batch processing to cope with this.
If it was the file, then testing should have picked it up. You don't want to parse your input files to make sure they compile etc.