Needs to create a Rails App where I want to get the time in local time Zone i.e. if the location is Delhi the time zone should be IST and if the the location is San Fransisco the time zone should be PDT.
How to accomplish this in ruby on rails?
P.S. One line code that can set the time zone automatically according to the location.
try this Time.now.getlocal.zone
If you need the Olson time zone (because three-letter time zones are ambiguous, as are GMT offsets), it looks like there's no way to do it in pure Ruby/Rails. Ruby will only provide the short code (basically via date +%Z), and Rails uses the time zone of its configuration (default: UTC).
That said, shelling out can be made to work in combination with another answer:
def get_local_timezone_str
# Yes, this is actually a shell script…
olsontz = `if [ -f /etc/timezone ]; then
cat /etc/timezone
elif [ -h /etc/localtime ]; then
readlink /etc/localtime | sed "s/\\/usr\\/share\\/zoneinfo\\///"
else
checksum=\`md5sum /etc/localtime | cut -d' ' -f1\`
find /usr/share/zoneinfo/ -type f -exec md5sum {} \\; | grep "^$checksum" | sed "s/.*\\/usr\\/share\\/zoneinfo\\///" | head -n 1
fi`.chomp
# …and it almost certainly won't work with Windows or weird *nixes
throw "Olson time zone could not be determined" if olsontz.nil? || olsontz.empty?
return olsontz
end
Related
For my powerlevel10k custom prompt, I currently have this function to display the seconds since the epoch, comma separated. I display it under the current time so I always have a cue to remember roughly what the current epoch time is.
function prompt_epoch() {
MYEPOCH=$(/bin/date +%s | sed ':a;s/\B[0-9]\{3\}\>/,&/;ta')
p10k segment -f 66 -t ${MYEPOCH}
}
My prompt looks like this: https://imgur.com/0IT5zXi
I've been told I can do this without the forked processes using these commands:
$ zmodload -F zsh/datetime p:EPOCHSECONDS
$ printf "%'d" $EPOCHSECONDS
1,648,943,504
But I'm not sure how to do that without the forking. I know to add the zmodload line in my ~/.zshrc before my powerlevel10k is sourced, but formatting ${EPOCHSECONDS} isn't something I know how to do without a fork.
If I were doing it the way I know, this is what I'd do:
function prompt_epoch() {
MYEPOCH=$(printf "%'d" ${EPOCHSECONDS})
p10k segment -f 66 -t ${MYEPOCH}
}
But as far as I understand it, that's still forking a process every time the prompt is called, correct? Am I misunderstanding the advice given because I don't think I can see a way to get the latest epoch seconds without running some sort of process, which requires a fork, correct?
The printf zsh builtin can assign the value to a variable using the -v flag. Therefore my function can be rewritten as:
function prompt_epoch() {
printf -v MYEPOCH "%'d" ${EPOCHSECONDS}
p10k segment -f 66 -t ${MYEPOCH}
}
Thanks to this answer in Unix Stackoverflow: https://unix.stackexchange.com/a/697807/101884
I'm new to Splunk and I want to do a search that specifies the start and end time.
I am using curl to automate the process and calling it from a bash script.
I have the starttime and endtime values I want but they do not have the %d/%m/%Y:%H%M%S" format and converting my "%Y%m%d H:%M%S" in bash will be ugly.
Is there a way to use timeformat to make my life easier?
My curl command is made up like this:
curl -u "$user:$pass" -k https://$splunkserver/services/search/jobs/export \
-d search='search "'"$search"'" | search starttime="'"$STARTTIME"'" | search endtime="'$ENDTIME"'" | search index=$index sourcetype=$sourcetype'
If I use these values, it works:
STARTTIME=01/20/2022:23:59:00
ENDTIME=01/21/2022:01:00:00
But fails when use the values I have:
STARTTIME="20220120 23:59:00"
ENDTIME="20220121 01:00:00"
Feel free to suggest other improvements to my Splunk code.
You have to convert your human-readable timestamp to Unix epoch time - since _time is always in Unix epoch time
Check out strftime.org for the exact calls
Then you can send this in your curl call:
curl -u "$user:$pass" -k https://$splunkserver/services/search/jobs/export -d search='search "'"$search"'" | search index=$index sourcetype=$sourcetype' earliest=$STARTTIME latest=$ENDTIME'
Running a system with UTC time I'd like to be able to write out to log files with local time so reading them later and identifying the influence of external factors for issues needs less brain work.
The command to get the local date (e.g. Sydney, Australia) when running on UTC is
TZ=Australia/Sydney date
This would return
Sat 15 Sep 17:19:28 AEST 2018
A sample of my script is below. For now, please ignore the fact it is not the best script for the job it appears to be trying to do. My issue is the time being recorded in the log file. What this script does each loop is write the same date/time into the log file - the local time when the script was started.
#!/bin/sh
localdatetime=$(TZ=Australia/Sydney date)
while
do
nc -zw5 192.168.0.199 # IP of router
if [[ $? -eq 0 ]]; then
status="up"
else
status="down"
fi
echo "$localdatetime The router is now $status" >> /home/pi/userX/routerStatus.log
sleep 10
done
What I want is for the current local time to be stored into the log file each loop so that I know from reading the log file when the router was up or down. Is there a way to do this using a variable?
Thanks in advance for any good advice.
What I did in the end was the obvious solution: update the local time variable immediately before trying to write it to the log file.
For example:
#!/bin/sh
while
do
nc -zw5 192.168.0.199 # IP of router
if [[ $? -eq 0 ]]; then
status="up"
else
status="down"
fi
localdatetime=$(TZ=Australia/Sydney date)
echo "$localdatetime The router is now $status" >> /home/pi/userX/routerStatus.log
sleep 10
done
I have a 680 million rows (19gig) file that I need the datetime appended onto every line. I get this file every night and I have to add the time that I processed it to the end of each line. I have tried many ways to do this including sed/awk and loading it into a SQL database with the last column being defaulted to the current timestamp.
I was wondering if there is a fast way to do this? My fastest way so far takes two hours and that is just not fast enough given the urgency of the information in this file. It is a flat CSV file.
edit1:
Here's what I've done so far:
awk -v date="$(date +"%Y-%m-%d %r")" '{ print $0","date}' lrn.ae.txt > testoutput.txt
Time = 117 minutes
perl -ne 'chomp; printf "%s.pdf\n", $_' EXPORT.txt > testoutput.txt
Time = 135 minutes
mysql load data local infile '/tmp/input.txt' into table testoutput
Time = 211 minutes
You don't specify if the timestamps have to be different for each of the lines. Would a "start of processing" time be enough?
If so, a simple solution is to use the paste command, with a pre-generated file of timestamps, exactly the same length as the file you're processing. Then just paste the whole thing together. Also, if the whole process is I/O bound, as others are speculating, then maybe running this on a box with an SSD drive would help speed up the process.
I just tried it locally on a 6 million row file (roughly 1% of yours), and it's actually able to do it in less than one second, on Macbook Pro, with an SSD drive.
~> date; time paste file1.txt timestamps.txt > final.txt; date
Mon Jun 5 10:57:49 MDT 2017
real 0m0.944s
user 0m0.680s
sys 0m0.222s
Mon Jun 5 10:57:49 MDT 2017
I'm going to now try a ~500 million row file, and see how that fares.
Updated:
Ok, the results are in. Paste is blazing fast compared to your solution, it took just over 90 seconds total to process the whole thing, 600M rows of simple data.
~> wc -l huge.txt
600000000 huge.txt
~> wc -l hugetimestamps.txt
600000000 hugetimestamps.txt
~> date; time paste huge.txt hugetimestamps.txt > final.txt; date
Mon Jun 5 11:09:11 MDT 2017
real 1m35.652s
user 1m8.352s
sys 0m22.643s
Mon Jun 5 11:10:47 MDT 2017
You still need to prepare the timestamps file ahead of time, but that's a trivial bash loop. I created mine in less than one minute.
A solution that simplifies mjuarez' helpful approach:
yes "$(date +"%Y-%m-%d %r")" | paste -d',' file - | head -n "$(wc -l < file)" > out-file
Note that, as with the approach in the linked answer, you must know the number of input lines in advance - here I'm using wc -l to count them, but if the number is fixed, simply use that fixed number.
yes keeps repeating its argument indefinitely, each on its own output line, until it is terminated.
paste -d',' file - pastes a corresponding pair of lines from file and stdin (-) on a single output line, separated with ,
Since yes produces "endless" output, head -n "$(wc -l < file)" ensures that processing stops once all input lines have been processed.
The use of a pipeline acts as a memory throttle, so running out of memory shouldn't be a concern.
Another alternative to test is
$ date +"%Y-%m-%d %r" > timestamp
$ join -t, -j9999 file timestamp | cut -d, -f2-
or time stamp can be generated in place as well <(date +"%Y-%m-%d %r")
join creates a cross product of the first file and second file using the non-existing field (9999), and since second file is only one line, practically appending it to the first file. Need the cut to get rid of the empty key field generated by join
If you want to add the same (current) datetime to each row in the file, you might as well leave the file as it is, and put the datetime in the filename instead. Depending on the use later, the software that processes the file could then first get the datetime from the filename.
To put the same datetime at the end of each row, some simple code could be written:
Make a string containing a separator and the datetime.
Read the lines from the file, append the above string and write back to a new file.
This way a conversion from datetime to string is only done once, and converting the file should not take much longer than copying the file on disk.
In my quest to catch robots, I would like to find a way to extract IP addresses from our Apache access logs which have requests in every hour of a 24 hour period.
For example this will show me a list and a count of requests per IPs that requested /bla/
find . -name logfile.*00 -print0 | xargs -0 fgrep -h '"GET /bla/' | awk '{print $NF}' | sort | uniq -c
Would there be some kind of awk that would tell me which IPs are present in all hours of the day?
Alternatively some free weblog analyser that could do the same.
Some information about the situation:
Our robot.txt is blocking all non-static files but is ignored by the majority of our abusers
At the moment I am only interested in some tool or awk statement that can give me a list of IPs that access us 24/7 since a normal user will access us 6-9 hours per day but from different timezones.
we already have several methods to detect and blacklist IPs and ranges of IPs but I want to see how the spread are on robots that are just turned on and runs non-stop.
The awk statement above is providing
Req IP
3234 111.222.333.444
234 222.222.333.444
5234 333.222.333.444
and I am looking for
IP Hrs
111.222.333.444 24
222.222.333.444 24
333.222.333.444 24
or better:
IP Hrs Req
111.222.333.444 24 3234
222.222.333.444 24 234
333.222.333.444 24 5234
I keep recommending Piwik found at http://piwik.org/. It is one of the best log file analysis tools out there and it is free! This is an excellent tool!!
You will find that is it not access over 24 hours. Google and Bing fall into these categories. You will want to be looking for a few things:
Did they access an image?
Did they access robots.txt?
Are they accessing your site at a reasonable speed? Is it human or machine?
Are they accessing a normal/reasonable amount of pages? Is it human or machine?
There are more factors, but this is enough. You will be able to quickly determine human from machine very quickly with just these factors.
What you do not want to do is make too many assumptions. For example, throw out the concept of using any agent name as some kind of indication. It is junk data.
What you need to do is research domain names and IP addresses and begin to know who these people (lack of a better term) are. Of course there are the obvious such as Google, Bing, Yandex, and Baidu. Some will be legitimate SEO sites such as MOZ, Ahrefs, or Open Site Explorer. You may want to allow accesses for these. However, there are lot of SEO scraper sites and content scraper sites. You will find some accesses from China, Russia, even Poland. These are often junk. You may even see competitors using Screaming Frog to discover how your site competes with keyword usage. And of course, let us not forget the script-kiddies that are trying to landscape and hack your site. It will take time of course to get a feel for who is good for our site. However, the discovery of abusers will never end.
You will want to block bad accesses as far away from your web server as possible. This means using a hardware firewall. But if that is not an option for you, you will want to explore the various software firewall options and possibly use ModSecurity and other tools to protect your site. Of course there is always the .htaccess file (assuming Apache).
This is a common practice that webmasters have to perform daily. It is just the reality of things. If you are not sure of an IP address or domain name or even an access pattern, just post it here and I will evaluate it for you and try and help. I study these things as a smaller subject area of my research. I do not always have answers, but most of the time I have one at hand ready to go and of course I can always do some research.
I decided to solve it brute force - there are simply too many IPs to handle.
Just ran this in 24 steps and extracted the IPs that were in all files.
Added bonus I got to see how many requests they managed to do a day.
find . -name access_log.*00 -mtime -2 -print0 | xargs -0 zfgrep --no-filename -w "[13" | grep "2014:00" | awk '{print $NF}' | sort | uniq -c > $HOME/unique13_00.txt;
find . -name access_log.*00 -mtime -2 -print0 | xargs -0 zfgrep --no-filename -w "[13" | grep "2014:01" | awk '{print $NF}' | sort | uniq -c > $HOME/unique13_01.txt;
find . -name access_log.*00 -mtime -2 -print0 | xargs -0 zfgrep --no-filename -w "[13" | grep "2014:02" | awk '{print $NF}' | sort | uniq -c > $HOME/unique13_02.txt;
.
.