I am using Apache flume for log collection. This is my config file
httpagent.sources = http-source
httpagent.sinks = local-file-sink
httpagent.channels = ch3
#Define source properties
httpagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
httpagent.sources.http-source.channels = ch3
httpagent.sources.http-source.port = 8082
# Local File Sink
httpagent.sinks.local-file-sink.type = file_roll
httpagent.sinks.local-file-sink.channel = ch3
httpagent.sinks.local-file-sink.sink.directory = /home/avinash/log_dir
httpagent.sinks.local-file-sink.sink.rollInterval = 21600
# Channels
httpagent.channels.ch3.type = memory
httpagent.channels.ch3.capacity = 1000
My application is working fine.My problem is that in the log_dir the files are using some random number (I guess its timestamp) timestamp as by default.
How to give a proper filename suffix for logfiles ?
Having a look on the documentation it seems there is no parameter for configuring the name of the files that are going to be created. I've gone to the sources looking for some hidden parameter, but there is no one :)
Going into the details of the implementation, it seems the name of the file is managed by the PathManager class:
private PathManager pathController;
...
#Override
public Status process() throws EventDeliveryException {
...
if (outputStream == null) {
File currentFile = pathController.getCurrentFile();
logger.debug("Opening output stream for file {}", currentFile);
try {
outputStream = new BufferedOutputStream(new FileOutputStream(currentFile));
...
}
Which, as you already noticed, is based on the current timestamp (showing the constructor and the next file getter):
public PathManager() {
seriesTimestamp = System.currentTimeMillis();
fileIndex = new AtomicInteger();
}
public File nextFile() {
currentFile = new File(baseDirectory, seriesTimestamp + "-" + fileIndex.incrementAndGet());
return currentFile;
}
So, I think the only possibility you have is to extend the File Roll sink and override the process() method in order to use a custom path controller.
For sources you have execute commands to tail and pre-pend or append details, based on shell scripting. Below is a sample:
# Describe/configure the source for tailing file
httpagent.sources.source.type = exec
httpagent.sources.source.shell = /bin/bash -c
httpagent.sources.source.command = tail -F /path/logs/*_details.log
httpagent.sources.source.restart = true
httpagent.sources.source.restartThrottle = 1000
httpagent.sources.source.logStdErr = true
Related
I have a folder with my text files that can read and write and work in eclipse. But, when I export to jar, it fails because the files are not found, meaning they are not exported and I don't know how to make eclipse do that. I'm sure the solution is out there, but I don't know exactly what I'm searching for. Do I make a relative directory and how? Or another source folder? What exactly do I need to do?
This shows I have a folder called conf where my files are stored but it is not there on export.
Scanner in = new Scanner(new FileReader("conf/Admins.txt"));
FileWriter out = new FileWriter("conf/CurrentUser.txt");
int id = 0;
String name = "";
String pass = "";
boolean found = false;
while(in.hasNext()) {
id = in.nextInt();
name = in.next();
pass = in.next();
if(id == userID) {
out.write(id + " " + name + " " + pass + "\n");
found = true;
break;
}
}
All I had to do was once exported, put my conf file in the same place as the exported jar file. I don't know if, theres a better way but this is a win for me.
I am using org.apache.commons.net.ftp.FTPClient for retrieving files from a ftp server. It is crucial that I preserve the last modified timestamp on the file when its saved on my machine. Do anyone have a suggestion for how to solve this?
This is how I solved it:
public boolean retrieveFile(String path, String filename, long lastModified) throws IOException {
File localFile = new File(path + "/" + filename);
OutputStream outputStream = new FileOutputStream(localFile);
boolean success = client.retrieveFile(filename, outputStream);
outputStream.close();
localFile.setLastModified(lastModified);
return success;
}
I wish the Apache-team would implement this feature.
This is how you can use it:
List<FTPFile> ftpFiles = Arrays.asList(client.listFiles());
for(FTPFile file : ftpFiles) {
retrieveFile("/tmp", file.getName(), file.getTimestamp().getTime());
}
You can modify the timestamp after downloading the file.
The timestamp can be retrieved through the LIST command, or the (non standard) MDTM command.
You can see here how to do modify the time stamp: that: http://www.mkyong.com/java/how-to-change-the-file-last-modified-date-in-java/
When download list of files, like all files returned by by FTPClient.mlistDir or FTPClient.listFiles, use the timestamp returned with the listing to update timestemp of local downloaded files:
String remotePath = "/remote/path";
String localPath = "C:\\local\\path";
FTPFile[] remoteFiles = ftpClient.mlistDir(remotePath);
for (FTPFile remoteFile : remoteFiles) {
File localFile = new File(localPath + "\\" + remoteFile.getName());
OutputStream outputStream = new BufferedOutputStream(new FileOutputStream(localFile));
if (ftpClient.retrieveFile(remotePath + "/" + remoteFile.getName(), outputStream))
{
System.out.println("File " + remoteFile.getName() + " downloaded successfully.");
}
outputStream.close();
localFile.setLastModified(remoteFile.getTimestamp().getTimeInMillis());
}
When downloading a single specific file only, use FTPClient.mdtmFile to retrieve the remote file timestamp and update timestamp of the downloaded local file accordingly:
File localFile = new File("C:\\local\\path\\file.zip");
FTPFile remoteFile = ftpClient.mdtmFile("/remote/path/file.zip");
if (remoteFile != null)
{
OutputStream outputStream = new BufferedOutputStream(new FileOutputStream(localFile));
if (ftpClient.retrieveFile(remoteFile.getName(), outputStream))
{
System.out.println("File downloaded successfully.");
}
outputStream.close();
localFile.setLastModified(remoteFile.getTimestamp().getTimeInMillis());
}
I'm using the following code on Mac using Mono to unzip a zip file. The zip file contains entries under directories (for example foo/bar.txt). However, in the unzipped directory, instead of creating a directory foo with a file bar.txt, FastZip creates a file foo\bar.txt. How do I get around this?
FastZip fz = new FastZip();
string filePath = #"path\to\myfile.zip";
fz.ExtractZip(filePath, #"path\to\unzip\to", null);
This creates a file foo\bar.txt in path\to\unzip\to.
Apparently cannot use FastZip for this case so I ended up writing my own unzipping mechanism:
string filePath = #"path\to\myfile.zip";
string unzipDir = #"path\to\unzip\to";
using (var zipFile = new ZipFile(filePath))
{
foreach (var zipEntry in zipFile.OfType<ZipEntry>())
{
var unzipPath = Path.Combine(unzipDir, zipEntry.Name);
var directoryPath = Path.GetDirectoryName(unzipPath);
// create directory if needed
if (directoryPath.Length > 0)
{
Directory.CreateDirectory(directoryPath);
}
// unzip the file
var zipStream = zipFile.GetInputStream(zipEntry);
var buffer = new byte[4096];
using (var unzippedFileStream = File.Create(unzipPath))
{
StreamUtils.Copy(zipStream, unzippedFileStream, buffer);
}
}
}
use a forward slash to separate folders when creating the zip
How to take automatically backup of a log file(.txt) when it's size reached a threshold level, say 5MB. The backup file name should be like (log_file_name)_(system_date) and original log file should be cleaned(0 KB).
Please help. Thanks in advance.
Check your log file size using lenght().Then check if its bigger then 5mb call extendLogFile() func.
This is c# code u can easly convert to java
Size check:
if (size > 400 * 100 * 100)
{
extendLogFile(Path);
}
Copy old log file in archive directory and create new log file:
private static void extendLogFile(string lPath)
{
string name = lPath.Substring(0, lPath.LastIndexOf("."));
string UniquName = GenerateUniqueNameUsingDate(); // create a unique name for old log files like '12-04-2013-12-43-00'
string ArchivePath = System.IO.Path.GetDirectoryName(lPath) + "\\Archive";
if (!string.IsNullOrEmpty(ArchivePath) && !System.IO.Directory.Exists(ArchivePath))
{
System.IO.Directory.CreateDirectory(ArchivePath);
}
string newName = ArcivePath + "\\" + UniquName;
if (!File.Exists(newName))
{
File.Copy(lPath, newName + ".txt");
using (FileStream stream = new FileStream(lPath, FileMode.Create))
using (TextWriter writer = new StreamWriter(stream))
{
writer.WriteLine("");
}
}
}
I am trying to upload a file via a form and then save in in SQL as a blob.
I already have my form working fine, my database is fully able to take the blob and I have a controller that take the file, saves it in a local directory:
[AcceptVerbs(HttpVerbs.Post)]
public ActionResult FileUpload(int id, HttpPostedFileBase uploadFile)
{
//allowed types
string typesNonFormatted = "text/plain,application/msword,application/pdf,image/jpeg,image/png,image/gif";
string[] types = typesNonFormatted.Split(',');
//
//Starting security check
//checking file size
if (uploadFile.ContentLength == 0 && uploadFile.ContentLength > 10000000)
ViewData["StatusMsg"] = "Could not upload: File too big (max size 10mb) or error while transfering the file.";
//checking file type
else if(types.Contains(uploadFile.ContentType) == false)
ViewData["StatusMsg"] = "Could not upload: Illigal file type!<br/> Allowed types: images, Ms Word documents, PDF, plain text files.";
//Passed all security checks
else
{
string filePath = Path.Combine(HttpContext.Server.MapPath("../Uploads"),
Path.GetFileName(uploadFile.FileName)); //generating path
uploadFile.SaveAs(filePath); //saving file to final destination
ViewData["StatusMsg"] = "Uploaded: " + uploadFile.FileName + " (" + Convert.ToDecimal(uploadFile.ContentLength) / 1000 + " kb)";
//saving file to database
//
//MISSING
}
return View("FileUpload", null);
}
Now all I am missing is putting the file in the database. I could not find anything on the subject... I found some way to do it in a regular website but nothing in MVC2.
Any kind of help would be welcome!
Thank you.
This could help: http://byatool.com/mvc/asp-net-mvc-upload-image-to-database-and-show-image-dynamically-using-a-view/
Since you have HttpPostedFileBase in your controllers method, all you need to do is:
int length = uploadFile.ContentLength;
byte[] tempImage = new byte[length];
myDBObject.ContentType = uploadFile.ContentType;
uploadFile.InputStream.Read(tempImage, 0, length);
myDBObject.ActualImage = tempImage ;
HttpPostedFileBase has a InputStream property
Hope this helps.
Alright thanks to kheit, I finaly got it working. Here's the final solution, it might help someone out there.
This script method takes all the file from a directory and upload them to the database:
//upload all file from a directory to the database as blob
public void UploadFilesToDB(long UniqueId)
{
//directory path
string fileUnformatedPath = "../Uploads/" + UniqueId; //setting final path with unique id
//getting all files in directory ( if any)
string[] FileList = System.IO.Directory.GetFiles(HttpContext.Server.MapPath(fileUnformatedPath));
//for each file in direcotry
foreach (var file in FileList)
{
//extracting file from directory
System.IO.FileStream CurFile = System.IO.File.Open(file, System.IO.FileMode.Open);
long fileLenght = CurFile.Length;
//converting file to a byte array (byte[])
byte[] tempFile = new byte[fileLenght];
CurFile.Read(tempFile, 0, Convert.ToInt32(fileLenght));
//creating new attachment
IW_Attachment CurAttachment = new IW_Attachment();
CurAttachment.attachment_blob = tempFile; //setting actual file
string[] filedirlist = CurFile.Name.Split('\\');//setting file name
CurAttachment.attachment_name = filedirlist.ElementAt(filedirlist.Count() - 1);//setting file name
//uploadind attachment to database
SubmissionRepository.CreateAttachment(CurAttachment);
//deleting current file fromd directory
CurFile.Flush();
System.IO.File.Delete(file);
CurFile.Close();
}
//deleting directory , it should be empty by now
System.IO.Directory.Delete(HttpContext.Server.MapPath(fileUnformatedPath));
}
(By the way IW_Attachment is the name of one of my database table)