How does Lettuce dynamic command interface work with Redis modules? - redis

Trying to send RedisTimeSeries commands thru Lettuce (Java) to Redis. It worked for simple commands such as TS.Create but I couldn't get slightly more complicated commands to work (such TS.ADD which takes key, score, value as args) or TS.Range (which takes args and returns List) to work.
Redis is running on Linux (Ubuntu running on Windows 10 thru WSL), RedisTimeSeries is installed onto Redis. Redis and RedisTimeSeries commands have been tested using Redis-cli on linux, they work fine. I use VS Code + JDK 13.0 + Maven to build and test a Java client for Redis. So far Redis commands supported by Lettuce are working thru the client, plus some simple RedisTimeSeries commands.
Code snippet:
RedisCommands<String, String> syncCommands = MyRedisClient.getSyncCommands(connection);
// this works:
RedisCodec<String, String> codec = StringCodec.UTF8;
String result = syncCommands.dispatch(TS.CREATE, new StatusOutput<>(codec),new CommandArgs<>(codec).addKey("myTS"));
System.out.println("Custom Command TS.CREATE " + result);
//custom command definition:
public enum TS implements ProtocolKeyword{
CREATE;
public final byte[] bytes;
private TS() {
bytes = "TS.CREATE".getBytes(StandardCharsets.US_ASCII);
}
#Override
public byte[] getBytes() {
return bytes;
}
}
But when I switched everything to test TS.ADD, it's doesn't work even though I added additional args accordingly.
e.g.
String result = syncCommands.dispatch(TS.ADD, new StatusOutput<>(codec),new CommandArgs<>(codec).addKey("myTS").addValue("1000001").addValue("2.199"));
Here is the exception from the run:
Exception in thread "main" io.lettuce.core.RedisException: java.lang.IllegalStateException
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:129)
at io.lettuce.core.FutureSyncInvocationHandler.handleInvocation(FutureSyncInvocationHandler.java:69)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy0.dispatch(Unknown Source)
at MyRedisClient.main(MyRedisClient.java:72)

Sorry for seeing this so late. If you haven't already found a solution I had initially implemented RediSearch commands using the dynamic commands.
public interface RediSearchCommands extends Commands {
#Command("FT.SUGADD ?0 ?1 ?2")
Long sugadd(String key, String string, double score);
...
}
public void testSuggestions() {
RedisCommandFactory factory = new RedisCommandFactory(client.connect());
RediSearchCommands commands = factory.getCommands(RediSearchCommands.class);
commands.sugadd(key, "Herbie Hancock", 1.0);
}
Full source: https://github.com/RediSearch/lettusearch/commit/567de42c147e0f07184df444cd1ae9798ae2514e

Related

The way to use Redis in Apache Flink

I am using the Flink and want to insert the result value into the Redis.
When I googled the Redis, I found the redis-connector included in the Apache bahir.
So I am able to insert the result value into the redis using the reids-connector in the Apache bahir.
However, I think that I am also able to connect to the Redis using the Jedis.
I had the experiment showing that I was able to connect the redis and found the value inserted into the redis using jedis as shown in the code below.
DataStream<String> messageStream = env.addSource(new FlinkKafkaConsumer<>(flinkParams.getRequired("topic"), new SimpleStringSchema(), flinkParams.getProperties())).setParallelism(Math.min(hosts * cores, kafkaPartitions));
messageStream.keyBy(new KeySelector<String, String>() {
#Override
public String getKey(String s) throws Exception {
return s;
}
}).flatMap(new RedisConnector());
In the RedisConnector module, without the redis-connector in the Apache bahir, I also successfully connected to the redis and found the message processed after the Flink.
The example code is shown as below
public class ProcessorCommon {
private static final Logger logger = LoggerFactory.getLogger(ProcessorCommon.class);
private Jedis jedis;
private Set<DummyPair> dummy;
public ProcessorCommon(String redisServerHostName) {
this.jedis = new Jedis(redisServerHostName);
}
public void writeToRedis(String key, String value) {
this.jedis.set(key, value);
}
public String getFromRedis(String key) {
return this.jedis.get(key);
}
public void close() {
this.jedis.close();
}
}
So I am wondering that there is a difference between using redis-connector in the bahir and Jedis.
There is currently no real Redis connector maintained by the Flink community. The Redis connector in Bahir is rather outdated. There is a new Redis Streams connector in the works, which can be found at https://github.com/apache/flink-connector-redis-streams

Sending DataStream in Flink using sockets; serialization issue

I want to send Stream of data from VM to host machine and I am using method writeToSocket() as shown below:
joinedStreamEventDataStream.writeToSocket("192.168.1.10", 6998) ;
Here joinedStreamEventDataStream is of type DataStream<Integer,Integer>.
Can someone please tell me how should I pass serializer to above method.
Thanks in Advance
It depends a little bit on how you would like to read the data from the socket. If you expect it to be the String representation of the data, then you could do it via:
joinedStreamEventDataStream.map(new MapFunction<Type, String>() {
#Override
public String map(Type value) throws Exception {
return value.toString();
}
}).writeToSocket(hostname, port, new SimpleStringSchema());
If you want to keep Flink's serialization format, then you can do write:
joinedStreamEventDataStream.writeToSocket(
hostname,
port,
new TypeInformationSerializationSchema<>(
joinedStreamEventDataStream.getType(),
env.getConfig()));
If you want to output it in your own serialization format, then you have to implement your own SerializationSchema as pointed out by Alex.
The writeToSocket() method takes 3 arguments: a socket host and port and also an implementation of SerializationSchema interface which used to serialize your data. So your implementation maybe like this:
joinedStreamEventDataStream.writeToSocket(
"192.168.1.10", // host name
6998, // port
new SerializationSchema<Integer>() {
#Override
public byte[] serialize(Integer element) {
return ByteBuffer.allocate(4).putInt(element).array();
}
}
);
It's true if joinedStreamEventDataStream has DataStream<Integer> type.

Spring-data-redis with redis after a while get Exception: could not get a resource from the pool

I'm using spring-data-redis to access redis(one machine) with xml config file, on beginning, everything is ok, but after some minutes, i run my
test again, i got "could not get a resource from the pool" exception, i haved searched some answers, i guess this reason is the connections did
not return to the pool, how to solve this problem, why this problem can be occur, i'm using redis-3.2.6 spring-data-redis1.8 jedis2.9, below is my config
#Redis settings
redis.host=27.57.100.3
redis.port=6379
redis.pass=
maxTotal=5
maxIdle=3
minIdle=1
maxWaitMillis=10000
testOnBorrow=true
testOnReturn=true
testWhileIdle=true
timeBetweenEvictionRunsMillis=30000
numTestsPerEvictionRun=10
minEvictableIdleTimeMillis=60000
softMinEvictableIdleTimeMillis=10000
blockWhenExhausted=true
And here is my code :
#Autowired
StringRedisTemplate stringRedisTemplate
#Test
public void test(){
ValueOperations<String, String> vop = stringRedisTemplate.opsForValue();
String k = "k";
String v = "v";
vop.set(k, v);
String value = vop.get(k);
}
maxTotal=5, I thought 5 is too small, you can set it to e.g,20.

Redis on Appharbor - Booksleeve GetString exception

i am trying to setup Redis on appharbor. I have followed their instructions and again i have an issue with the Booksleeve API. Here is the code i am using to make it work initially:
var connectionUri = new Uri(url);
using (var redis = new RedisConnection(connectionUri.Host, connectionUri.Port, password: connectionUri.UserInfo.Split(new[] { ':' }, 2)[1]))
{
redis.Strings.Set(1, "greeting", "welcome to remember your stuff!");
try
{
var task = redis.Strings.GetString(1, "greeting");
redis.Wait(task);
ViewBag.Message = task.Result;
}
catch (Exception)
{
// It throws an exception trying to wait for the task?
}
}
However, the issue is that it sets the string correctly, but when trying to retrieve the same string from the key value store, it throws a timeout exception waiting for the task to eexecute. However, this code works on my local redis server connection.
Am i using the API in a wrong way? or is this something related to Appharbor?
Thanks
Like a SqlConnection, you need to call Open() (otherwise your messages are queued for delivery).
Unlike SqlConnection, you should not fire up a RedisConnection each time you need it - it is intended to be used as a shared, thread-safe, multiplexer - i.e. a single connection is held somewhere and used by lots and lots of unrelated callers. Unless of course you only need to do one thing!

Very slow performance deserializing using datacontractserializer in a Silverlight Application

Here is the situation:
Silverlight 3 Application hits an asp.net hosted WCF service to get a list of items to display in a grid. Once the list is brought down to the client it is cached in IsolatedStorage. This is done by using the DataContractSerializer to serialize all of these objects to a stream which is then zipped and then encrypted. When the application is relaunched, it first loads from the cache (reversing the process above) and the deserializes the objects using the DataContractSerializer.ReadObject() method. All of this was working wonderfully under all scenarios until recently with the entire "load from cache" path (decrypt/unzip/deserialize) taking hundreds of milliseconds at most.
On some development machines but not all (all machines Windows 7) the deserialize process - that is the call to ReadObject(stream) takes several minutes an seems to lock up the entire machine BUT ONLY WHEN RUNNING IN THE DEBUGGER in VS2008. Running the Debug configuration code outside the debugger has no problem.
One thing that seems to look suspicious is that when you turn on stop on Exceptions, you can see that the ReadObject() throws many, many System.FormatException's indicating that a number was not in the correct format. When I turn off "Just My Code" thousands of these get dumped to the screen. None go unhandled. These occur both on the read back from the cache AND on a deserialization at the conclusion of a web service call to get the data from the WCF Service. HOWEVER, these same exceptions occur on my laptop development machine that does not experience the slowness at all. And FWIW, my laptop is really old and my desktop is a 4 core, 6GB RAM beast.
Again, no problems unless running under the debugger in VS2008. Anyone else seem this? Any thoughts?
Here is the bug report link: https://connect.microsoft.com/VisualStudio/feedback/details/539609/very-slow-performance-deserializing-using-datacontractserializer-in-a-silverlight-application-only-in-debugger
EDIT: I now know where the FormatExceptions are coming from. It seems that they are "by design" - they occur when when I have doubles being serialized that are double.NaN so that that xml looks like NaN...It seems that the DCS tries to parse the value as a number, that fails with an exception and then it looks for "NaN" et. al. and handles them. My problem is not that this does not work...it does...it is just that it completely cripples the debugger. Does anyone know how to configure the debugger/vs2008sp1 to handle this more efficiently.
cartden,
You may want to consider switching over to XMLSerializer instead. Here is what I have determined over time:
The XMLSerializer and DataContractSerializer classes provides a simple means of serializing and deserializing object graphs to and from XML.
The key differences are:
1.
XMLSerializer has much smaller payload than DCS if you use [XmlAttribute] instead of [XmlElement]
DCS always store values as elements
2.
DCS is "opt-in" rather than "opt-out"
With DCS you explicitly mark what you want to serialize with [DataMember]
With DCS you can serialize any field or property, even if they are marked protected or private
With DCS you can use [IgnoreDataMember] to have the serializer ignore certain properties
With XMLSerializer public properties are serialized, and need setters to be deserialized
With XmlSerializer you can use [XmlIgnore] to have the serializer ignore public properties
3.
BE AWARE! DCS.ReadObject DOES NOT call constructors during deserialization
If you need to perform initialization, DCS supports the following callback hooks:
[OnDeserializing], [OnDeserialized], [OnSerializing], [OnSerialized]
(also useful for handling versioning issues)
If you want the ability to switch between the two serializers, you can use both sets of attributes simultaneously, as in:
[DataContract]
[XmlRoot]
public class ProfilePerson : NotifyPropertyChanges
{
[XmlAttribute]
[DataMember]
public string FirstName { get { return m_FirstName; } set { SetProperty(ref m_FirstName, value); } }
private string m_FirstName;
[XmlElement]
[DataMember]
public PersonLocation Location { get { return m_Location; } set { SetProperty(ref m_Location, value); } }
private PersonLocation m_Location = new PersonLocation(); // Should change over time
[XmlIgnore]
[IgnoreDataMember]
public Profile ParentProfile { get { return m_ParentProfile; } set { SetProperty(ref m_ParentProfile, value); } }
private Profile m_ParentProfile = null;
public ProfilePerson()
{
}
}
Also, check out my Serializer class that can switch between the two:
using System;
using System.IO;
using System.Runtime.Serialization;
using System.Text;
using System.Xml;
using System.Xml.Serialization;
namespace ClassLibrary
{
// Instantiate this class to serialize objects using either XmlSerializer or DataContractSerializer
internal class Serializer
{
private readonly bool m_bDCS;
internal Serializer(bool bDCS)
{
m_bDCS = bDCS;
}
internal TT Deserialize<TT>(string input)
{
MemoryStream stream = new MemoryStream(input.ToByteArray());
if (m_bDCS)
{
DataContractSerializer dc = new DataContractSerializer(typeof(TT));
return (TT)dc.ReadObject(stream);
}
else
{
XmlSerializer xs = new XmlSerializer(typeof(TT));
return (TT)xs.Deserialize(stream);
}
}
internal string Serialize<TT>(object obj)
{
MemoryStream stream = new MemoryStream();
if (m_bDCS)
{
DataContractSerializer dc = new DataContractSerializer(typeof(TT));
dc.WriteObject(stream, obj);
}
else
{
XmlSerializer xs = new XmlSerializer(typeof(TT));
xs.Serialize(stream, obj);
}
// be aware that the Unicode Byte-Order Mark will be at the front of the string
return stream.ToArray().ToUtfString();
}
internal string SerializeToString<TT>(object obj)
{
StringBuilder builder = new StringBuilder();
XmlWriter xmlWriter = XmlWriter.Create(builder);
if (m_bDCS)
{
DataContractSerializer dc = new DataContractSerializer(typeof(TT));
dc.WriteObject(xmlWriter, obj);
}
else
{
XmlSerializer xs = new XmlSerializer(typeof(TT));
xs.Serialize(xmlWriter, obj);
}
string xml = builder.ToString();
xml = RegexHelper.ReplacePattern(xml, RegexHelper.WildcardToPattern("<?xml*>", WildcardSearch.Anywhere), string.Empty);
xml = RegexHelper.ReplacePattern(xml, RegexHelper.WildcardToPattern(" xmlns:*\"*\"", WildcardSearch.Anywhere), string.Empty);
xml = xml.Replace(Environment.NewLine + " ", string.Empty);
xml = xml.Replace(Environment.NewLine, string.Empty);
return xml;
}
}
}
This is a guess, but I think it is running slow in debug mode because for every exception, it is performing some actions to show the exception in the debug window, etc. If you are running in release mode, these extra steps are not taken.
I've never done this, so I really don't know id it would work, but have you tried just setting that one assembly to run in release mode while all others are set to debug? If I'm right, it may solve your problem. If I'm wrong, then you only waste 1 or 2 minutes.
About your debugging problem, have you tried to disable the exception assistant ? (Tools > Options > Debugging > Enable the exception assistant).
Another point should be the exception handling in Debug > Exceptions : you can disable the user-unhandled stuff for the CLR or only uncheck the System.FormatException exception.
Ok - I figured out the root issue. It was what I alluded to in the EDIT to the main question. The problem was that in the xml, it was correctly serializing doubles that had a value of double.NaN. I was using these values to indicate "na" for when the denominator was 0D. Example: ROE (Return on Equity = Net Income / Average Equity) when Average Equity is 0D would be serialized as:
<ROE>NaN</ROE>
When the DCS tried to de-serialize it, evidently it first tries to read the number and then catches the exception when that fails and then handles the NaN. The problem is that this seems to generate a lot of overhead when in DEBUG mode.
Solution: I changed the property to double? and set it to null instead of NaN. Everything now happens instantly in DEBUG mode now. Thanks to all for your help.
Try disabling some IE addons. In my case, the LastPass toolbar killed my Silverlight debugging. My computer would freeze for minutes each time I interacted with Visual Studio after a breakpoint.