How to write/serialize lucene's ByteBuffersDirectory to disk? - lucene

How one would write a Lucene 8.11 ByteBuffersDirectory to disk?
something similar to Lucene 2.9.4 Directory.copy(directory, FSDirectory.open(indexPath), true)

You can use the copyFrom method to do this.
For example:
You are using a ByteBuffersDirectory:
final Directory dir = new ByteBuffersDirectory();
Assuming you are not concurrently writing any new data to that dir, you can declare a target where you want to write the data - for example, a FSDirectory (a file system directory):
Directory to = FSDirectory.open(Paths.get(OUT_DIR_PATH));
Use whatever string you want for the OUT_DIR_PATH location.
Then you can iterate over all the files in the original dir object, writing them to this new to location:
IOContext ctx = new IOContext();
for (String file : dir.listAll()) {
System.out.println(file); // just for testing
to.copyFrom(dir, file, file, ctx);
}
This will create the new OUT_DIR_PATH dir and populate it with files, such as:
_0.cfe
_0.cfs
_0.si
segments_1
... or whatever files you happen to have in your dir.
Caveat:
I have only used this with a default IOContext object. There are other constructors for the context - not sure what they do. I assume they give you more control over how the write is performed.

Meanwhile I figured it out by myself and created a straight forward method for it:
#SneakyThrows
public static void copyIndex(ByteBuffersDirectory ramDirectory, Path destination) {
FSDirectory fsDirectory = FSDirectory.open(destination);
Arrays.stream(ramDirectory.listAll())
.forEach(fileName -> {
try {
// IOContext is null because in fact is not used (at least for the moment)
fsDirectory.copyFrom(ramDirectory, fileName, fileName, null);
} catch (IOException e) {
log.error(e.getMessage(), e);
}
});
}

Related

Arquillian ShrinkWrap how to add an asset to the file system path

I am importing a library that reads from the file system instead of my web archive's resource folder. I want to be able to essentially mock that file by adding an asset with that path using ShrinkWrap, so I can run tests on my build server without guaranteeing the file system has all these files. I tried to add a String Asset in the appropriate path, but the code can't find that asset. Here's an example of what I'm trying to achieve.
Rest Resource
#Path("/hello-world")
public class HelloWorldResource {
#GET
public Response getHelloWorld(){
return Response.ok(getFileContent()).build();
}
private String getFileContent() {
StringBuilder builder = new StringBuilder();
try {
BufferedReader bufferedReader = new BufferedReader(
new FileReader(
"/usr/myFile.txt"));
String line = bufferedReader.readLine();
while (line != null) {
builder.append(line);
line = bufferedReader.readLine();
}
}
catch (Exception e) {
e.printStackTrace();
}
return builder.toString();
}
}
Test
#RunWith(Arquillian.class)
public class HelloWorldResourceTest {
#Deployment
public static WebArchive createDeployment()
{
WebArchive webArchive = ShrinkWrap
.create(WebArchive.class)
.addPackages(true,
HelloWorldApplication.class.getPackage(),
HelloWorldResource.class.getPackage(),
Hello.class.getPackage())
.add(new StringAsset("Blah"),"/usr/myFile.txt")
.addAsWebInfResource(EmptyAsset.INSTANCE, "beans.xml");
System.out.println("WebArchive: " + webArchive.toString(true));
return webArchive;
}
#Test
#RunAsClient
public void testHello(
#ArquillianResteasyResource("hello-world") final WebTarget webTarget)
{
final Response response = webTarget
.request(MediaType.APPLICATION_JSON)
.get();
String hello = response.readEntity(String.class);
System.err.println("Hello: " + hello);
Assert.assertEquals("Status is not OK", response.getStatus(), 200);
}
}
Web Archive toString
/WEB-INF/
/WEB-INF/classes/
/WEB-INF/classes/com/
/WEB-INF/classes/com/
/WEB-INF/classes/com/
/WEB-INF/classes/com/helloworld/
/WEB-INF/classes/com/helloworld/application/
/WEB-INF/classes/com/helloworld/application/HelloWorldApplication.class
/WEB-INF/classes/com/helloworld/resource/
/WEB-INF/classes/com/helloworld/resource/HelloWorldResourceTest.class
/WEB-INF/classes/com/helloworld/resource/HelloWorldResource.class
/WEB-INF/classes/com/helloworld/dataobjects/
/WEB-INF/classes/com/helloworld/dataobjects/Hello.class
/WEB-INF/beans.xml
/usr/
/usr/myFile.txt
I get the following error:
java.io.FileNotFoundException: /usr/myFile.txt (No such file or
directory)
Seems like ShrinkWrap is adding /usr/myFile.txt as a relative path within the archive instead of making it seem like /usr/myFile.txt is at the root directory of my file system. Is there any way I can get ShrinkWrap to do what I want?
Shrinkwrap is intended to create archives, so the API is scoped to create assets within the archive you are creating. If you want to have resources created in the regular filesystem simply use JDK, there is nothing Shrinkwrap could help you with.
Alternatively, if possible, change your resource to read resources from the classpath, not filesystem path. With this approach, you can easily swap content for the test using Shrinkwrap as you are trying now with your example.

Different project settings for user Intellij Idea

How can I do different project settings for users in the same project. I have a project with some .properties file like email.properties which contains user-specific settings.I need something like user-specific enviroment settings, something like this: email..properties, and variable contains in OS enviroment or, maybe, in project file
Here's how I've done this in some projects. I create a System Property to the path of the property file. The path is outside of the project so the properties file is never committed. But to help others who are running the project for the first time, I commit a template properties file with default options.
In intellij, I start the project with a different -D option than my coworkers do and the startup options don't get committed (because we don't commit the .idea folder).
As a result a do the same: at first I created properties folder in propject root, add there file env.properties which contains current environment name (or user can add it as JVM start paramenter -Denv=<env name>. I add static class and method Properties getProperty(String fileName) which receive property file name as parameter and return all records in file as java.util.Properties.
public static Properties loadProperties(String fileName)
{
Properties properties = null;
if (propertiesMap.containsKey(fileName)) {
properties = (Properties)properties.get(fileName);
} else {
String environment = getEnvironment();
try {
properties = (new PropertiesLoaderImpl()).LoadAllPropertiesForUtilAndEnv(environment, fileName);
} catch (FileNotFoundException exception) {}
}
return properties;
}
private static String getEnvironment() {
// Проверка на наличие параметра при запуске java-машины
String environment = System.getProperty("env");
if (environment == null) {
try {
// Попытка найти файл env.properties
ResourceBundle bundle = ResourceBundle.getBundle("properties/env");
environment = bundle.getString("env");
} catch (MissingResourceException exception) {
environment = "";
}
}
return environment;
}
Property search implements that way:
1. look in ..proeprties;
2. if there is no suitable property, then look in .property;
3. else return empy.

Best way to extract .zip and put in .jar

I have been trying to find the best way to do this I have thought of extracting the contents of the .jar then moving the files into the directory then putting it back as a jar. Im not sure is the best solution or how I will do it. I have looked at DotNetZip & SharpZipLib but don't know what one to use.
If anyone can give me a link to the code on how to do this it would be appreciated.
For DotNetZip you can find very simple VB.NET examples of both creating a zip archive and extracting a zip archive into a directory here. Just remember to save the compressed file with extension .jar .
For SharpZipLib there are somewhat more comprehensive examples of archive creation and extraction here.
If none of these libraries manage to extract the full JAR archive, you could also consider accessing a more full-fledged compression software such as 7-zip, either starting it as a separate process using Process.Start or using its COM interface to access the relevant methods in the 7za.dll. More information on COM usage can be found here.
I think you are working with Minecraft 1.3.1 no? If you are, there is a file contained in the zip called aux.class, which unfortunately is a reserved filename in windows. I've been trying to automate the process of modding, while manipulating the jar file myself, and have had little success. The only option I have yet to explore is find a way to extract the contents of the jar file to a temporary location, while watching for that exception. When it occurs, rename the file to a temp name, extract and move on. Then while recreating the zip file, give the file the original name in the archive. From my own experience, SharZipLib doesnt do what you need it do nicely, or at least I couldnt figure out how. I suggest using Ionic Zip (Dot Net Zip) instead, and trying the rename route on the offending files. In addition, I also posted a question about this. You can see how far I got at Extract zip entries to another Zip file
Edit - I tested out .net zip more (available from http://dotnetzip.codeplex.com/), and heres what you need. I imagine it will work with any zip file that contains reserved file names. I know its in C#, but hey cant do all the work for ya :P
public static void CopyToZip(string inArchive, string outArchive, string tempPath)
{
ZipFile inZip = null;
ZipFile outZip = null;
try
{
inZip = new ZipFile(inArchive);
outZip = new ZipFile(outArchive);
List<string> tempNames = new List<string>();
List<string> originalNames = new List<string>();
int I = 0;
foreach (ZipEntry entry in inZip)
{
if (!entry.IsDirectory)
{
string tempName = Path.Combine(tempPath, "tmp.tmp");
string oldName = entry.FileName;
byte[] buffer = new byte[4026];
Stream inStream = null;
FileStream stream = null;
try
{
inStream = entry.OpenReader();
stream = new FileStream(tempName, FileMode.Create, FileAccess.ReadWrite);
int size = 0;
while ((size = inStream.Read(buffer, 0, buffer.Length)) > 0)
{
stream.Write(buffer, 0, size);
}
inStream.Close();
stream.Flush();
stream.Close();
inStream = new FileStream(tempName, FileMode.Open, FileAccess.Read);
outZip.AddEntry(oldName, inStream);
outZip.Save();
}
catch (Exception exe)
{
throw exe;
}
finally
{
try { inStream.Close(); }
catch (Exception ignore) { }
try { stream.Close(); }
catch (Exception ignore) { }
}
}
}
}
catch (Exception e)
{
throw e;
}
}

How to get the name of a temporary file created by File.tmpfile in D2?

I need to generate a temporary file, fill it with some data and feed it to an external program. Based on description of D available here I'm using File.tmpfile() method:
auto f = File.tmpfile();
writeln(f.name());
which doesn't provide a way to get the generated file name. It's documented that name might be empty. In Python I would do that like this:
(o_fd, o_filename) = tempfile.mkstemp('.my.own.suffix')
Is there a simple, safe and cross-platform way to do that in D2?
Due to how tmpfile() works, if you need the name of the file you can't use it. However, I have already created a module to work with temporary files. It uses conditional compilation to decide on the method of finding the temporary directory. On windows, it uses the %TMP% environment variable. On Posix, it uses /tmp/.
This code is licensed under the WTFPL, so you can do whatever you want with it.
module TemporaryFiles;
import std.conv,
std.random,
std.stdio;
version(Windows) {
import std.process;
}
private static Random rand;
/// Returns a file with the specified filename and permissions
public File getTempFile(string filename, string permissions) {
string path;
version(Windows) {
path = getenv("TMP") ~ '\\';
} else version(Posix) {
path = "/tmp/";
// path = "/var/tmp/"; // Uncomment to survive reboots
}
return File(path~filename, permissions);
}
/// Returns a file opened for writing, which the specified filename
public File getTempFile(string filename) {
return getTempFile(filename, "w");
}
/// Returns a file opened for writing, with a randomly generated filename
public File getTempFile() {
string filename = to!string(uniform(1L, 1000000000L, rand)) ~ ".tmp";
return getTempFile(filename, "w");
}
To use this, simply call getTempFile() with whatever arguments you want. Defaults to write permission.
As a note, the "randomly generated filenames" aren't truely random, as the seed is set at compile time.

context path for file upload without HttpRequest in REST application

I am building REST application. I want to upload a file and I want to save it for example in /WEB-INF/resource/uploads
How can I get path to this directory ? My Controller looks like this
#RequestMapping(value = "/admin/house/update", method = RequestMethod.POST)
public String updateHouse(House house, #RequestParam("file") MultipartFile file, Model model) {
try {
String fileName = null;
InputStream inputStream = null;
OutputStream outputStream = null;
if (file.getSize() > 0) {
inputStream = file.getInputStream();
fileName = "D:/" + file.getOriginalFilename();
outputStream = new FileOutputStream(fileName);
int readBytes = 0;
byte[] buffer = new byte[10000];
while ((readBytes = inputStream.read(buffer, 0, 10000)) != -1) {
outputStream.write(buffer, 0, readBytes);
}
outputStream.close();
inputStream.close();
}
} catch(Exception ex) {
ex.printStackTrace();
}
model.addAttribute("step", 3);
this.houseDao.update(house);
return "houseAdmin";
}
Second question...what is the best place to upload user files ?
/WEB-INF is a bad place to try to store file uploads. There's no guarantee that this is an actual directory on the disk, and even if it is, the appserver may forbid write access to it.
Where you should store your files depends on what you want to do with them, and what operating system you're running on. Just pick somewhere outside of the webapp itself, is my advice. Perhaps create a dedicated directory
Also, the process of transferring the MultipartFile to another location is much simpler than you're making it out to be:
#RequestMapping(value = "/admin/house/update", method = RequestMethod.POST)
public String updateHouse(House house, #RequestParam("file") MultipartFile srcFile, Model model) throws IOException {
File destFile = new File("/path/to/the/target/file");
srcFile.transferTo(destFile); // easy!
model.addAttribute("step", 3);
this.houseDao.update(house);
return "houseAdmin";
}
You shouldn't store files in /WEB-INF/resource/uploads. This directory is either inside your WAR (if packaged) or exploded somewhere inside servlet container. The first destination is read-only and the latter should not be used for user files.
There are usually two places considered when storing uploaded files:
Some dedicated folder. Make sure users cannot access this directory directly (e.g. anonymous FTP folder). Note that once your application runs on more than one machine you won't have access to this folder. So consider some form of network synchronization or a shared network drive.
Database. This is controversial since binary files tend to occupy a lot of space. But this approach is a bit simpler when distributing your application.