I am using ServiceStack.Redis library to work with Redis. To start with, I have implemented this solution. The get/set methods work fine for plain text/string.
Now when I save a string with quotes (with escape char), it saves properly (I verify the same in redis-cli). But the Get method returns string having all the double quotes removed.
For example saving this string - "TestSample" is saved and get as expected. Also,
saving "TestSample \"with\" \"quotes\"" is fine and shows same in redis-cli. But the output of Get method becomes "TestSample with quotes"
public bool SetDataInCache<T>(string cacheKey, T cacheData)
{
try
{
using (_redisClient = new RedisClient(_cacheConfigs.RedisHost))
{
_redisClient.As<T>().SetValue(cacheKey, cacheData, new TimeSpan(0,0,300));
}
return true;
}
catch (Exception ex)
{
return false;
}
}
public T GetDataFromCacheByType<T>(string cacheKey)
{
T retVal = default(T);
try
{
using (_redisClient = new RedisClient(_cacheConfigs.RedisHost))
{
if (_redisClient.ContainsKey(cacheKey))
{
var wrapper = _redisClient.As<T>();
retVal = wrapper.GetValue(cacheKey);
}
return retVal;
}
}
catch (Exception ex)
{
return retVal;
}
}
Usage:
cacheObj.SetDataInCache("MyKey1","TestSample");
cacheObj.SetDataInCache("MyKey2","TestSample \"with\" \"quotes\"");
string result1 = Convert.ToString(cacheObj.GetDataFromCacheByType<string>("MyKey1"));
string result2 = Convert.ToString(cacheObj.GetDataFromCacheByType<string>("MyKey2"));
Actual : "TestSample with quotes"
Expected : "TestSample \"with\" \"quotes\""
The Typed Generic API is only meant for creating a generic Redis Client for serializing complex types. If you're implementing a generic cache you should use the IRedisClient APIs instead, e.g:
_redisClient.Set(cacheKey, cacheData, new TimeSpan(0,0,300));
Then retrieve back with:
var retVal = _redisClient.Get<T>(cacheKey);
Alternatively for saving strings or if you want to serialize the POCOs yourself you can use the IRedisClient SetValue/GetValue string APIs, e.g:
_redisClient.SetValue(cacheKey, cacheData.ToJson());
var retVal = _redisClient.GetValue(cacheKey).FromJson<T>();
Note: calling IRedisClient.ContainsKey() performs an additional unnecessary Redis I/O operation, since you're returning default(T) anyway you should just call _redisClient.Get<T>() which returns the default value for non-existing keys.
Related
I am trying to use Google protocol buffer library for serializing and deserialzing process in C#. I can deserialize using following codeNotificationSet.Parser.ParseJson(json); And this is working fine.
NotificationSet is auto generated file by .proto.
But here you can see it is not generic. So, instead of specif type i need to make a method in generic way. Can you please advice on this?
Example:
public async Task<TResult> Deserialize<TResult, TValue>(TValue value)
{
TResult.Parser.ParseJson(value.ToString());
}
Problem is TResult is generic type, so unable to get Parser method from that.
Found an answer.
Try with is code to achieve generic deserialization process using google protocol buffer library.
public async Task<TResult> Deserialize<TResult,TValue>(TValue value)
{
try
{
System.Type type = typeof(TResult);
var typ = Assembly.GetExecutingAssembly().GetTypes().First(t => t.Name == type.Name);
var descriptor = (MessageDescriptor)typ.GetProperty("Descriptor", BindingFlags.Public | BindingFlags.Static).GetValue(null, null);
var response = descriptor.Parser.ParseJson(value.ToString());
return await Task.FromResult((TResult)response);
}
catch (Exception ex)
{
throw ex;
}
}
}
I have a method that try use WebClient to return a Mono
#GetMapping("getMatch")
public Mono<Object> getMatch(#RequestParam Long matchId) {
return WebClient.create(OpenDotaConstant.BASE_URL).get()
.uri("/matches/{matchId}", matchId)
.accept(MediaType.APPLICATION_JSON)
.retrieve()
.bodyToMono(Object.class);
}
It can return result that I expected.
Then I try to create another method to support List as params
#GetMapping("getMatches")
public Flux<Object> getMatches(#RequestParam String matchesId) {
List<Long> matchesList = JSON.parseArray(matchesId, Long.class);
return Flux.fromStream(matchesList.parallelStream().map(this::getMatch));
}
But this time return a weird result.
[
{
"scanAvailable": true
},
{
"scanAvailable": true
}
]
I'm new to reactive-programming, What is the correct way to combine Stream and Mono,and then convert to the Flux?
Probably, what you need is the following:
#GetMapping("getMatches")
public Flux<Object> getMatches(#RequestParam String matchesId) {
List<Long> matchesList = JSON.parseArray(matchesId, Long.class);
return Flux.fromStream(matchesList.stream())
.flatMap(this::getMatch);
}
Instead of:
#GetMapping("getMatches")
public Flux<Object> getMatches(#RequestParam String matchesId) {
List<Long> matchesList = JSON.parseArray(matchesId, Long.class);
return Flux.fromStream(matchesList.parallelStream().map(this::getMatch));
}
Notes:
Basically, you expect getMatches endpoint to return Flux<Object>. However, as it is written - it actually returns Flux<Mono<Object>>, therefore you see the strange output. To get Flux<Object>, I suggest, first, create Flux<Long> that are match ids, and then flatMap the result of calling getMatch (that returns Mono<Object>), this finally gives Flux<Object>.
Also, there is no need to use parallelStream(). Because you're already using reactor, everything will be performed concurrently on reactor scheduler.
I have a script which is loading some data about venues:
venues = LOAD 'venues_extended_2.csv' USING org.apache.pig.piggybank.storage.CSVLoader() AS (Name:chararray, Type:chararray, Latitude:double, Longitude:double, City:chararray, Country:chararray);
Then I want to create UDF which has a constructor that is accepting venues type.
So I tried to define this UDF like that:
DEFINE GenerateVenues org.gla.anton.udf.main.GenerateVenues(venues);
And here is the actual UDF:
public class GenerateVenues extends EvalFunc<Tuple> {
TupleFactory mTupleFactory = TupleFactory.getInstance();
BagFactory mBagFactory = BagFactory.getInstance();
private static final String ALLCHARS = "(.*)";
private ArrayList<String> venues;
private String regex;
public GenerateVenues(DataBag venuesBag) {
Iterator<Tuple> it = venuesBag.iterator();
venues = new ArrayList<String>((int) (venuesBag.size() + 1)); // possible fails!!!
String current = "";
regex = "";
while (it.hasNext()){
Tuple t = it.next();
try {
current = "(" + ALLCHARS + t.get(0) + ALLCHARS + ")";
venues.add((String) t.get(0));
} catch (ExecException e) {
throw new IllegalArgumentException("VenuesRegex: requires tuple with at least one value");
}
regex += current + (it.hasNext() ? "|" : "");
}
}
#Override
public Tuple exec(Tuple tuple) throws IOException {
// expect one string
if (tuple == null || tuple.size() != 2) {
throw new IllegalArgumentException(
"BagTupleExampleUDF: requires two input parameters.");
}
try {
String tweet = (String) tuple.get(0);
for (String venue: venues)
{
if (tweet.matches(ALLCHARS + venue + ALLCHARS))
{
Tuple output = mTupleFactory.newTuple(Collections.singletonList(venue));
return output;
}
}
return null;
} catch (Exception e) {
throw new IOException(
"BagTupleExampleUDF: caught exception processing input.", e);
}
}
}
When executed the script is firing error at the DEFINE part just before (venues);:
2013-12-19 04:28:06,072 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: <file script.pig, line 6, column 60> mismatched input 'venues' expecting RIGHT_PAREN
Obviously I'm doing something wrong, can you help me out figuring out what's wrong.
Is it the UDF that cannot accept the venues relation as a parameter. Or the relation is not represented by DataBag like this public GenerateVenues(DataBag venuesBag)?
Thanks!
PS I'm using Pig version 0.11.1.1.3.0.0-107.
As #WinnieNicklaus already said, you can only pass strings to UDF constructors.
Having said that, the solution to your problem is using distributed cache, you need to override public List<String> getCacheFiles() to return a list of filenames that will be made available via distributed cache. With that, you can read the file as a local file and build your table.
The downside is that Pig has no initialization function, so you have to implement something like
private void init() {
if (!this.initialized) {
// read table
}
}
and then call that as the first thing from exec.
You can't use a relation as a parameter in a UDF constructor. Only strings can be passed as arguments, and if they are really of another type, you will have to parse them out in the constructor.
I was wondering if anybody find a way to stub/mock a logic inside a lambda without making the lambda's visibility?
public List<Item> processFile(String fileName) {
// do some magic..
Function<String, List<String>> reader = (fileName) -> {
List<String> items = new ArrayList<>();
try (BufferedReader br = new BufferedReader(new FileReader(fileName))) {
String output;
while ((output = br.readLine()) != null) {
items.add(output);
}
} catch (IOException e) {
e.printStackTrace();
}
return items;
};
List<String> lines = reader.apply("file.csv");
// do some more magic..
}
I would say the rule is that if a lambda expression is so complex that you feel the need to mock out bits of it, that it's probably too complex. It should be broken into smaller pieces that are composed together, or perhaps the model needs to be adjusted to make it more amenable to composition.
I will say that Andrey Chaschev's answer which suggests parameterizing a dependency is a good one and probably is applicable in some situations. So, +1 for that. One could continue this process and break down the processing into smaller pieces, like so:
public List<Item> processFile(
String fileName,
Function<String, BufferedReader> toReader,
Function<BufferedReader, List<String>> toStringList,
Function<List<String>, List<Item>> toItemList)
{
List<String> lines = null;
try (BufferedReader br = toReader.apply(fileName)) {
lines = toStringList.apply(br);
} catch (IOException ioe) { /* ... */ }
return toItemList.apply(lines);
}
A couple observations on this, though. First, this doesn't work as written, since the various lambdas throw pesky IOExceptions, which are checked, and the Function type isn't declared to throw that exception. The second is that the lambdas you have to pass to this function are monstrous. Even though this doesn't work (because of checked exceptions) I wrote it out:
void processAnActualFile() {
List<Item> items = processFile(
"file.csv",
fname -> new BufferedReader(new FileReader(fname)),
// ERROR: uncaught IOException
br -> {
List<String> result = new ArrayList<>();
String line;
while ((line = br.readLine()) != null) {
result.add(line);
}
return result;
}, // ERROR: uncaught IOException
stringList -> {
List<Item> result = new ArrayList<>();
for (String line : stringList) {
result.add(new Item(line));
}
return result;
});
}
Ugh! I think I've discovered new code smell:
If you have to write a for-loop or while-loop inside a lambda, you're doing something wrong.
A few things are going on here. First, the I/O library is really composed of different pieces of implementation (InputStream, Reader, BufferedReader) that are tightly coupled. It's really not useful to try to break them apart. Indeed, the library has evolved so that there are some convenience utilities (such as the NIO Files.readAllLines) that handle a bunch of leg work for you.
The more significant point is that designing functions that pass aggregates (lists) of values among themselves, and composing these functions, is really the wrong way to go. It leads every function to have to write a loop inside of it. What we really want to do is write functions that each operate on a single value, and then let the new Streams library in Java 8 take care of the aggregation for us.
The key function to extract here from the code described by the comment "do some more magic" which converts List<String> into List<Item>. We want to extract the computation that converts one String into an Item, like this:
class Item {
static Item fromString(String s) {
// do a little bit of magic
}
}
Once you have this, then you can let the Streams and NIO libraries do a bunch of the work for you:
public List<Item> processFile(String fileName) {
try (Stream<String> lines = Files.lines(Paths.get(fileName))) {
return lines.map(Item::fromString)
.collect(Collectors.toList());
} catch (IOException ioe) {
ioe.printStackTrace();
return Collections.emptyList();
}
}
(Note that more half of this short method is for dealing with the IOException.)
Now if you want to do some unit testing, what you really need to test is that little bit of magic. So you wrap it into a different stream pipeline, like this:
void testItemCreation() {
List<Item> result =
Arrays.asList("first", "second", "third")
.stream()
.map(Item::fromString)
.collect(Collectors.toList());
// make assertions over result
}
(Actually, even this isn't quite right. You'd want to write unit tests for converting a single line into a single Item. But maybe you have some test data somewhere, so you'd convert it to a list of items this way, and then make global assertions over the relationship of the resulting items in the list.)
I've wandered pretty far from your original question of how to break apart a lambda. Please forgive me for indulging myself.
The lambda in the original example is pretty unfortunate since the Java I/O libraries are quite cumbersome, and there are new APIs in the NIO library that turn the example into a one-liner.
Still, the lesson here is that instead of composing functions that process aggregates, compose functions that process individual values, and let streams handle the aggregation. This way, instead of testing by mocking out bits of a complex lambda, you can test by plugging together stream pipelines in different ways.
I'm not sure if that's what you're asking, but you could extract a lambda from lambda i.e. to another class or as is and pass it as a parameter. In an example below I mock reader creation:
public static void processFile(String fileName, Function<String, BufferedReader> readerSupplier) {
// do some magic..
Function<String, List<String>> reader = (name) -> {
List<String> items = new ArrayList<>();
try(BufferedReader br = readerSupplier.apply(name)){
String output;
while ((output = br.readLine()) != null) {
items.add(output);
}
} catch (IOException e) {
e.printStackTrace();
}
return items;
};
List<String> lines = reader.apply(fileName);
// do some more magic..
}
public static void main(String[] args) {
// mocked call
processFile("file.csv", name -> new BufferedReader(new StringReader("line1\nline2\n")));
//original call
processFile("1.csv", name -> {
try {
return new BufferedReader(new FileReader(name));
} catch (FileNotFoundException e) {
throw new RuntimeException(e);
}
});
}
I've read this answer (How to store complex objects into hadoop Hbase?) regarding the storing of string arrays with HBase.
There it is said to use the ArrayWritable Class to serialize the array. With WritableUtils.toByteArray(Writable ... writable) I'll get a byte[] which I can store in HBase.
When I now try to retrieve the rows again, I get a byte[] which I have somehow to transform back again into an ArrayWritable.
But I don't find a way to do this. Maybe you know an answer or am I doing fundamentally wrong serializing my String[]?
You may apply the following method to get back the ArrayWritable (taken from my earlier answer, see here) .
public static <T extends Writable> T asWritable(byte[] bytes, Class<T> clazz)
throws IOException {
T result = null;
DataInputStream dataIn = null;
try {
result = clazz.newInstance();
ByteArrayInputStream in = new ByteArrayInputStream(bytes);
dataIn = new DataInputStream(in);
result.readFields(dataIn);
}
catch (InstantiationException e) {
// should not happen
assert false;
}
catch (IllegalAccessException e) {
// should not happen
assert false;
}
finally {
IOUtils.closeQuietly(dataIn);
}
return result;
}
This method just deserializes the byte array to the correct object type, based on the provided class type token.
E.g:
Let's assume you have a custom ArrayWritable:
public class TextArrayWritable extends ArrayWritable {
public TextArrayWritable() {
super(Text.class);
}
}
Now you issue a single HBase get:
...
Get get = new Get(row);
Result result = htable.get(get);
byte[] value = result.getValue(family, qualifier);
TextArrayWritable tawReturned = asWritable(value, TextArrayWritable.class);
Text[] texts = (Text[]) tawReturned.toArray();
for (Text t : texts) {
System.out.print(t + " ");
}
...
Note:
You may have already found the readCompressedStringArray() and writeCompressedStringArray() methods in WritableUtils
which seem to be suitable if you have your own String array-backed Writable class.
Before using them, I'd warn you that these can cause serious performance hit due to
the overhead caused by the gzip compression/decompression.