I use tcsh at work - one of the features I use extensively is command-line history completion at the shell prompt. Currently, I've limited the size of my history file to 2000 (as I don't want to slow down the shell too much). However at times I need a command I know I've used a month or two back , but by now has been erased. So I want a system wherein:
My history buffer stores 2000 lines only
Instead of older commands getting erased , they should be saved into a "master" history file, ordered chronologically i.e if two shells were opened , then the commands entered in the history should be sorted as per the datestamp (not the order in which the shells were closed)!
It would be perfect , if this master history file could be auto-backed up, say per week basis.
I'm sure many of avid shell users have faced a situation like this - I'm hoping to get the answer from one of such users !!
2000 is pretty low. You could raise that a fair amount without suffering too much.
Next you probably want to store the history on logout, since this is when new commands are added to the .history file.
Create a file called .logout in your $HOME (for bash users, this file is .bash_logout). In this, copy the contents of the history to a permanent store. For example:
cat $HOME/.history >> $HOME/.ancient_history
This will append the history to a file ".ancient_history". For bash users, the file to copy is called .bash_history.
Then create a cron job that creates a back up of this every now and again. For starters here is one that moves the file to a filename with a date stamp at 5 minutes past midnight every day.
5 0 * * * mv $HOME/.ancient_history $HOME/.ancient_history_`date +%s`
There are probably more things you could do with this, but this is enough to get started. It's a pretty good idea that I hadn't thought of doing before either :-)
never quite thought of doing this but the simplest way would be to write a cron job that appended the history file to another file. The problem with this would be that you would get duplicates unless you wrote the cron to clear the history file after it did the dump.
history is stored (as far as i am aware) by line number only so the numbers would repeat for each dump. but you cold add a marker line with the date of the dump.
Related
Apologies if something similar to the question I'm asking has already been addressed. I'm not even sure how to best frame my question but I haven't been able to find any posts that are obviously germane. I'm hoping someone has some experience with this and might be willing to offer some suggestions. My company has already contracted to have the bulk of our database converted to HTML for ETL purposes and we simply can't afford to double the already barely-manageable costs of this project by adding this additional requirement to the scope.
We have a SQL database from an EMR software vendor that our company has now left. Due to recent economic factors, we just just can't afford to stay with them any longer. When we left this ex-vendor begrudgingly provided us with a backup copy of our SQL database along with copies of all the scanned images our users have uploaded via their application GUI over the years. I was told they stored the uploads as BLOB data but it turns out not. They weren't actually storing the files in the database at all. Instead, they moved the image to a storage location and wrote the ID, DocType, Filename, DirPath and other document information to the Document table of the DB. It makes sense but leaves us in a bind. Mainly because the filename appears to have been randomly generated at upload. So we now have 50,000 image files with unintelligible filenames stored in a date-based folder structure with no way to correlate any of them with the patients to whom they belong. A couple of examples are as follows:
/root/2020/05102019/69353829-e46b-47e7-ab56-a1762424f0dd.pdf
/root/2014/09282017/385ba21d-e108-4cbb-9287-91110c16edb0.jpg
I compiled a list of attribs so I can make any of them available to the transform. I pulled:
SELECT * FROM document d
JOIN patients p ON d.PatientId = p.pid
JOIN users u ON d.PatientId = u.uid
WHERE u.UserType = '3' AND d.fileformat is NOT NULL AND d.dirpath LIKE 'm%'
ORDER BY u.ulname;
This gave me all patient and document attribs resulting in a list with 197 columns. The challenge is the new EMR vendor can only import these files if all the files for each patient are in a dedicated folder at the patient level so I need the files in a new folder structure. I am trying to do it without abandoning things like PatientID, Scan Date, Description (the customName column), Scanned By, and a possibly a couple other points.
I'll probably end up making the file name something like a concat of customName+docID for identification purposes. Then I'll just need to get the files in something like a /Patient/Docs.extension folder structure.
I went ahead and flattened all the files into a single folder figuring that would make it easier to manipulate. I batched them out like so:
md "D:\OneDrive\Documents\Assets\eClinicalworks\PID\FTP\mobiledoc\Documents\All\"
cd /d "D:\OneDrive\Documents\Assets\eClinicalworks\PID\FTP\mobiledoc\Documents\"
for /r %d in (*) do copy "%d" "D:\OneDrive\Documents\Assets\eClinicalworks\PID\FTP\mobiledoc\Documents\All\"
Now I have them all together.
Screenshot
I still have to figure out how to get them into the new folder structure by patient though.
Just to have it mentioned, I was originally considering using SQL so I could recreate the files and assign the desired attribs as file attribs in one step.
To answer the question asked about the HTML conversion, we have tons of Progress Notes, Doctors Notes, Prescriptions, etc in the database. The only way to get them to the new EMR is to export them to HTML and group them at the patient level so the new vendor can import them.
Honestly, after having to wrestle with all this garbage, I would prefer to avoid this situation in the future by refusing to upload them to the new EMR at all. Instead, just put all these documents on OUR file server and give the new EMR a hyper-link to insert into each patient's patient record that would open all the patient files. The new EMR is browser-based so it could be feasible but I doubt I'll be able to get them to write files to our file server moving forward so doing so would likely just end up making the end-user experience more disjointed.
I don't think your contractors did anything wrong tbh. Taking uploaded files with all their problem characters/duplicated names (got more than one patient called JohnSmith.jpg?) etc and renaming them to a GUID so they can coexist alongside other images without overwriting them is a) sensible and b) what I would do.
I also wouldn't store images inside a database as then the only thing you can do with them is get them out again; something you have to do every time you want to do anything with them. Being able to map an images folder to a url on your web server and then send html using just the file name means that the web server can sever the image without having to pull it out of the db; the db doesn't have to involve itself in pointless IO.
The way to correlate these images with the patients to whom they belong is done by the database. Somewhere else in the db structure will be eg a Patient record with a DocumentId column that links to this document record or a PatientDocuments table that has PatientId/DocumentId pairs.
If there is not, then storing the document bytes in the db won't have helped relate them to the patient, because this relation is not about where the bytes of an image are, it's about what other data was stored to make for a usable system. As it stands your thoughts on the matter, of uploading tens of thousands of images into a db just so you can... er.. get them all out again, would seem to indicate you haven't yet fully grasped the reasons behind why your contractors did what they did.
Because you're under the impression that you can do this, you seem to know how the db relates a document to a patient (if it doesn't then your proposed process will fail) and as such you can arrange for a suitable renaming process without needing to move the image data anywhere. In essence, you're failing to see that a file system storing file data against unique paths is no different to a database table storing file data against unique ids. Your database tables for documents clearly thus links to your file system/file system can be viewed as an extension to the documents table. You need the other tables in the db to make sense of the files, but you need the other tables in a db to make sense of any table in a db. These are key concepts of modelling related data
I don't recommend you undertake the process you propose, but I'm sure that won't dissuade. Consider then (because you didn't really post any details we can work with) this assumed scenario:
Patients
Name,DocumentId
John Smith,1
Jane Doe,2
Documents
Id,FilePath
1,'/root/2020/05102019/69353829-e46b-47e7-ab56-a1762424f0dd.pdf'
2,'/root/2014/09282017/385ba21d-e108-4cbb-9287-91110c16edb0.jpg'
SELECT CONCAT('REN ', d.filePath, ' "', p.Name, RIGHT(d.filePath, 4), '"')
FROM
Patients p
INNER JOIN Documents d ON p.DocumentId = d.DocumentId
The results of the query will essentially be a batch file full of rename commands that renames all the files into a single folder, organized by patient name.
And now all your multiple patients with the same names will overwrite each other and everything will end up in a mess
It also makes my point for me about "don't store files in the db" - look how easy it is to manipulate files when they're in a file system, using existing commands that understand filesystems and files and do things like rename files, or extract exif data, rotate, resize and print... if all those images were in your db the only thing you could do with them, is get them out again; sqlserver cannot rotate, resize, print etc BLOB data but there are thousands of tools out there that understand files and can convert them - those tools cannot understand your db so putting files into a db saddles you with the problem that they become useless until dug out again
Your contractors may not have been so daft as you think; pause a moment before you set about hacking apart all they did, and question whether your driver for doing so is actually correct. If Jane from reception needs to see a picture of John Smith with drivers license XY1234 to ID him, don't provide her with a shared drive full of everyone's pictures, and let her double click, drag and accidentally delete her way around the file system. Provide her with an app that looks in the db, gets the unintelligible but helpfully unique filename off disk and opens it for her to view. And make the file system read only to everyone other than the app, so that users can't break things
Technologies available: Autosys, Informatica, Unix scripting, Database (available via informatica)
How our batch currently works is with filewatchers looking for a file called "control.txt" which gets deleted when a feed starts processing. It gets recreated once completed which allows all "control" autosys jobs waiting, to have one pick up the control file and begin processing data feeds one by one.
However, the system has grown large, and some feeds have become more important than others, and we're looking at ways to improve our scheduler to prioritize feeds over others.
With the current design, of one a file deciding when the next feed runs, it can't be done, and I haven't been able to come up with a simple solution to make it happen.
Example:
1. Feed A is processing
2. Feed B, Feed C, Feed X, Feed F come in while Feed A is processing
3. Need to ensure that Feed B is processed next, even though C, X, F are ready.
4. C, X, F have a lower priority than A and B, but have the same priority and can process in any order
A very interesting question. One thing that I can think of is to have an extra Autosys job with a shell script that copies the file in certain order. Like:
Create input folder e.g. StageFolder
Let's call your current Autosys input folder "the InputFolder"
Have Autosys monitor it and for any file run a OrderedFileCopyScript.sh, every minute
OrderedFileCopyScript.sh should copy one file from StageFolder to InputFolder in desired order only if InputFolder is empty
I hope I made myself clear.
I oppose use of Autosys for this requirement ! Wrong tool !
I don't know all the details but considering an application with the usual reference tables.
In this case you should make use of feed reference table to include relative priorities.
I would suggest to create(or reuse) a table to loaded by the successor job of the file watcher.
1) Table to contain the unprocessed file with the corresponding priority and then use this table to process the files based on the priority.
2) Remove/archive the entries once done.
3) Have another job of this and run like a daemon with start_times/run_window.
This gives the flexibility to deal with change in priorities and keeps overall design simple.
This gives
I have a .RDL report which I designed in BIDS and have deployed to my report server. The report asks for three parameters before viewing report: Year, Month and Customer ID. The report works great and does exactly what it is supposed to.
While I used to run each report individually because there were 2-3 customers, now there are 30+ customers who receive the report, so I wanted to switch to a more automated fulfillment method to get the reports generated. After doing some research it appears that a using Report Manager to create a "Data Driven Subscription" (DDS) using the "Windows File Share" option gives me the capabilities I need.
As part of creating the DDS, I created a table called [Subscription] which is a table containing one row for each customer receiving the report and has the following columns:
Year
Month
CustomerID
FileName
FileLocation
Overwrite
Format
...so through using the DDS Wizard in Report Manager, I was able to successfully set up a Data Driven Subscription (which is linked to various columns in the [Subscription] table) which creates a new report for each customer in the [Subscription] table, saves [and overwrites, if necessary] it in a location of my choosing as a PDF (specified in [Subscription].[FileLocation], or the FileLocation column of my table for each row), and runs every minute (I plan on changing frequency to once a week, eventually).
This works flawlessly, giving me a new set of 30 reports in the directory of my choosing, with each report having a name I assigned in the FileName column of my table. Exactly what I was looking for.
HERE'S THE PROBLEM: When I update the FileLocation or FileName (or anything, really) in the [Subscription] table - it doesn't pick up the changes right away. Sometimes it doesn't even pick it up at all (for example I updated the [ReportName] column for one customer from Report_711622 to SpecialReport_711622, so that the output file for that customer should be named SpecialReport_711622 while all of the other reports should be called Report_XXXXX [no Special prefix]. But the file name of report for Customer 711622 remains the same!
It's almost like the job only see's what it needs to do once a day, and then does not go back and reference the [Subscription] table until I leave for the night, then when I come back in the morning it picks up the change.
Since I am about to scale this process out to a large customer-base using a different report, I need to be able to make edits to the [Subscription] table and have them get picked up by the Data Driven Subscription immediately (and if not immediately, at least a fixed interval of time that I can adjust, so that I can know 100% when the change will get picked up).
Does anyone know what's causing my lag? How do I change it so that updates to the Subscription table get picked up regularly? I'm also having issues with creating new DDS on other reports (following the exact process outlined above) - I've created the subscriptions, for every minute, and it says they are running and the number of outputs match the number of customers with 0 errors, but there are no files in the drive I specified (or anywhere else I've looked, for that matter).
Any help would be greatly appreciated!
I think the answer lies in the mechanism SSRS uses. There are a few places "lag" can occur.
The subscription is in fact an SQL Agent job which creates a record in the Event table. This table is a queue that SSRS checks to do scheduled tasks.
There is a small amount of time between the moment the subscription creates the Event record and the moment SQL reads it and starts creating the dataset for your DDS. The creation of the DDS dataset takes some time, too. In this time, the subscription will be in the Pending state. If you change anything in the data during this time, The subscription will still use the old data as report parameters. So obviously you will not notice your change until the next scheduled run.
Which brings me to the following: if a subscription is still being run and the next schedule kicks in (chances are, because yours runs every minute), the engine will not execute it, but wait for the next subscription schedule, and so on. So that's another possibility of lag - and cause of missing reports for a certain schedule minute. The subscription processes reports sequentially, one row from your DDS recordset at a time. Again, this takes some time. You can also see that in the subscription window when it says: # of # processed.
I suggest you look at the Event table in the database ReportServer during an execution. Also the ExecutionHistory views (there are 3) may be interesting. A scheduled run shows up as a RequestType = 1 and generates one record for each report. You can see the exact timing and parameters of each report that is run in the subscription. You may be able to extract the data you need to resolve your other issues.
EDIT: Here is a more elaborate guide to DDS data and events
http://blogs.msdn.com/b/deanka/archive/2009/01/13/diagnosing-and-troubleshooting-subscriptions.aspx
http://blogs.msdn.com/b/deanka/archive/2010/02/16/troubleshooting-subscriptions-part-ii-using-the-report-services-trace-log-file.aspx
Could this "Double-Hop" problem be the source of my issues? I'm so stuck on this one!
The Double-Hop Problem - MSDN Knowledgecast
I am using postgresql is it possible for me to change a table when a specific time happens. I would like to modify values when a specific date happens that is specified in the table being modified. For example:
a piece of artwork is located in a museum, after it's exhibition ends it is automatically placed back into storage, changing its location attribute. This occurs on a specified date.
It is not possible. See cron jobs, as suggested by Muleinik... but then, to expand on your example:
a piece of artwork is located in a museum, after it's exhibition ends it is automatically placed back into storage, changing its location attribute. This occurs on a specified date.
What happens if the piece of art is stolen (happens), or the museum it got sent to as part of a temporary exhibition decides to keep it (happens) or return it to its "rightful owner" (happens), or it's shelved in the wrong location (happens), etc.?
Don't just assume that things will go well -- they won't.
postgres does not have triggers on system-wide event (such as time).
What you can do, however, is have the OS's cron or at services do it for you by scheduling a statement like this:
echo "UPDATE artwork SET location='storage' WHERE name='Mona Lisa' | psql -u some_user -d some_database
Maybe you need a different data model, one that would allow you to store "historical" location. (in quotes because you'll keep future records there, too.)
Is there a way to automatically delete a row 24 hours after its creation in Transact-SQL?
I'm making a site (learning experience) where the user needs to click a validation link sent by e-mail once they register. I want the users to validate themselves within 24 hours.
I suppose what I'd need is a trigger, but I'm really not sure on the syntax, nor if it is even possible.
I'm not sure of your schema but I would do it a different way. I would have a date/time against the database record that corresponds to the validation link. When they click the link, verify that the date and time of the database record is within 24 hours of the current time. If so, allow it, otherwise reject it.
Q: Is there a way to automatically delete a row 24 hours after its creation in Transact-SQL?
A: Sure. Write a "sqlcmd" script, wrap it in a .bat file, and invoke it from Windows Scheduled Tasks:
http://windows.microsoft.com/en-US/windows7/schedule-a-task
Alternatively, depending on your version, you could schedule the same SQL script from SQL Server Agent:
http://msdn.microsoft.com/en-us/library/ms189237.aspx
Putting a different spin on things:
When the user clicks your link, you can check if the current time (with respect to MSSQL) is >> 24 hours. If so, you'll reply with a "Too late" message (rather than validating the entry).
In any case - you absolutely, completely, totally, do NOT want to use a trigger!