I'm using spring-data-redis to access redis(one machine) with xml config file, on beginning, everything is ok, but after some minutes, i run my
test again, i got "could not get a resource from the pool" exception, i haved searched some answers, i guess this reason is the connections did
not return to the pool, how to solve this problem, why this problem can be occur, i'm using redis-3.2.6 spring-data-redis1.8 jedis2.9, below is my config
#Redis settings
redis.host=27.57.100.3
redis.port=6379
redis.pass=
maxTotal=5
maxIdle=3
minIdle=1
maxWaitMillis=10000
testOnBorrow=true
testOnReturn=true
testWhileIdle=true
timeBetweenEvictionRunsMillis=30000
numTestsPerEvictionRun=10
minEvictableIdleTimeMillis=60000
softMinEvictableIdleTimeMillis=10000
blockWhenExhausted=true
And here is my code :
#Autowired
StringRedisTemplate stringRedisTemplate
#Test
public void test(){
ValueOperations<String, String> vop = stringRedisTemplate.opsForValue();
String k = "k";
String v = "v";
vop.set(k, v);
String value = vop.get(k);
}
maxTotal=5, I thought 5 is too small, you can set it to e.g,20.
Related
I am using the Flink and want to insert the result value into the Redis.
When I googled the Redis, I found the redis-connector included in the Apache bahir.
So I am able to insert the result value into the redis using the reids-connector in the Apache bahir.
However, I think that I am also able to connect to the Redis using the Jedis.
I had the experiment showing that I was able to connect the redis and found the value inserted into the redis using jedis as shown in the code below.
DataStream<String> messageStream = env.addSource(new FlinkKafkaConsumer<>(flinkParams.getRequired("topic"), new SimpleStringSchema(), flinkParams.getProperties())).setParallelism(Math.min(hosts * cores, kafkaPartitions));
messageStream.keyBy(new KeySelector<String, String>() {
#Override
public String getKey(String s) throws Exception {
return s;
}
}).flatMap(new RedisConnector());
In the RedisConnector module, without the redis-connector in the Apache bahir, I also successfully connected to the redis and found the message processed after the Flink.
The example code is shown as below
public class ProcessorCommon {
private static final Logger logger = LoggerFactory.getLogger(ProcessorCommon.class);
private Jedis jedis;
private Set<DummyPair> dummy;
public ProcessorCommon(String redisServerHostName) {
this.jedis = new Jedis(redisServerHostName);
}
public void writeToRedis(String key, String value) {
this.jedis.set(key, value);
}
public String getFromRedis(String key) {
return this.jedis.get(key);
}
public void close() {
this.jedis.close();
}
}
So I am wondering that there is a difference between using redis-connector in the bahir and Jedis.
There is currently no real Redis connector maintained by the Flink community. The Redis connector in Bahir is rather outdated. There is a new Redis Streams connector in the works, which can be found at https://github.com/apache/flink-connector-redis-streams
We are connecting our microservices to aws keyspace(Cassandra) through dbaas.
Getting error
cloud.dbaas.client.exceptions.CreateDbException: MicroserviceRestClientResponseException{message=404 Not Found: "No physical database known of type cassandra
Even getting same error from dbaas pods logs.
I already configured below parameters
spring.data.cassandra.ssl
spring.data.cassandra.contact-points
spring.data.cassandra.local-datacenter
spring.data.cassandra.port
spring.data.cassandra.password
spring.data.cassandra.username
You will want to reference the external configuration. See the following Amazon Keyspaces spring example.
https://github.com/aws-samples/amazon-keyspaces-spring-app-example/
#Configuration
public class AppConfig {
private final String username = System.getenv("AWS_MCS_SPRING_APP_USERNAME");
private final String password = System.getenv("AWS_MCS_SPRING_APP_PASSWORD");
File driverConfig = new File(System.getProperty("user.dir")+"/application.conf");
#Primary
public #Bean
CqlSession session() throws NoSuchAlgorithmException {
return CqlSession.builder().
withConfigLoader(DriverConfigLoader.fromFile(driverConfig)).
withAuthCredentials(username, password).
withSslContext(SSLContext.getDefault()).
withKeyspace("keyspace_name").
build();
}
}
Trying to send RedisTimeSeries commands thru Lettuce (Java) to Redis. It worked for simple commands such as TS.Create but I couldn't get slightly more complicated commands to work (such TS.ADD which takes key, score, value as args) or TS.Range (which takes args and returns List) to work.
Redis is running on Linux (Ubuntu running on Windows 10 thru WSL), RedisTimeSeries is installed onto Redis. Redis and RedisTimeSeries commands have been tested using Redis-cli on linux, they work fine. I use VS Code + JDK 13.0 + Maven to build and test a Java client for Redis. So far Redis commands supported by Lettuce are working thru the client, plus some simple RedisTimeSeries commands.
Code snippet:
RedisCommands<String, String> syncCommands = MyRedisClient.getSyncCommands(connection);
// this works:
RedisCodec<String, String> codec = StringCodec.UTF8;
String result = syncCommands.dispatch(TS.CREATE, new StatusOutput<>(codec),new CommandArgs<>(codec).addKey("myTS"));
System.out.println("Custom Command TS.CREATE " + result);
//custom command definition:
public enum TS implements ProtocolKeyword{
CREATE;
public final byte[] bytes;
private TS() {
bytes = "TS.CREATE".getBytes(StandardCharsets.US_ASCII);
}
#Override
public byte[] getBytes() {
return bytes;
}
}
But when I switched everything to test TS.ADD, it's doesn't work even though I added additional args accordingly.
e.g.
String result = syncCommands.dispatch(TS.ADD, new StatusOutput<>(codec),new CommandArgs<>(codec).addKey("myTS").addValue("1000001").addValue("2.199"));
Here is the exception from the run:
Exception in thread "main" io.lettuce.core.RedisException: java.lang.IllegalStateException
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:129)
at io.lettuce.core.FutureSyncInvocationHandler.handleInvocation(FutureSyncInvocationHandler.java:69)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy0.dispatch(Unknown Source)
at MyRedisClient.main(MyRedisClient.java:72)
Sorry for seeing this so late. If you haven't already found a solution I had initially implemented RediSearch commands using the dynamic commands.
public interface RediSearchCommands extends Commands {
#Command("FT.SUGADD ?0 ?1 ?2")
Long sugadd(String key, String string, double score);
...
}
public void testSuggestions() {
RedisCommandFactory factory = new RedisCommandFactory(client.connect());
RediSearchCommands commands = factory.getCommands(RediSearchCommands.class);
commands.sugadd(key, "Herbie Hancock", 1.0);
}
Full source: https://github.com/RediSearch/lettusearch/commit/567de42c147e0f07184df444cd1ae9798ae2514e
I am working on a usecase where I am supposed to poll S3 -> read the stream for the content -> do some processing and upload it to another bucket rather than writing the file in my server.
I know I can achieve it using S3StreamingMessageSource in Spring aws integration but the problem I am facing is that I do not know on how to process the message stream received by polling
public class S3PollerConfigurationUsingStreaming {
#Value("${amazonProperties.bucketName}")
private String bucketName;
#Value("${amazonProperties.newBucket}")
private String newBucket;
#Autowired
private AmazonClientService amazonClient;
#Bean
#InboundChannelAdapter(value = "s3Channel", poller = #Poller(fixedDelay = "100"))
public MessageSource<InputStream> s3InboundStreamingMessageSource() {
S3StreamingMessageSource messageSource = new S3StreamingMessageSource(template());
messageSource.setRemoteDirectory(bucketName);
messageSource.setFilter(new S3PersistentAcceptOnceFileListFilter(new SimpleMetadataStore(),
"streaming"));
return messageSource;
}
#Bean
#Transformer(inputChannel = "s3Channel", outputChannel = "data")
public org.springframework.integration.transformer.Transformer transformer() {
return new StreamTransformer();
}
#Bean
public S3RemoteFileTemplate template() {
return new S3RemoteFileTemplate(new S3SessionFactory(amazonClient.getS3Client()));
}
#Bean
public PollableChannel s3Channel() {
return new QueueChannel();
}
#Bean
IntegrationFlow fileStreamingFlow() {
return IntegrationFlows
.from(s3InboundStreamingMessageSource(),
e -> e.poller(p -> p.fixedDelay(30, TimeUnit.SECONDS)))
.handle(streamFile())
.get();
}
}
Can someone please help me with the code to process the stream ?
Not sure what is your problem, but I see that you have a mix of concerns. If you use messaging annotations (see #InboundChannelAdapter in your config), what is the point to use the same s3InboundStreamingMessageSource in the IntegrationFlow definition?
Anyway it looks like you have already explored for yourself a StreamTransformer. This one has a charset property to convert your InputStreamfrom the remote S3 resource to the String. Otherwise it returns a byte[]. Everything else is up to you what and how to do with this converted content.
Also I don't see reason to have an s3Channel as a QueueChannel, since the start of your flow is pollable anyway by the #InboundChannelAdapter.
From big height I would say we have more questions to you, than vise versa...
UPDATE
Not clear what is your idea for InputStream processing, but that is really a fact that after S3StreamingMessageSource you are going to have exactly InputStream as a payload in the next handler.
Also not sure what is your streamFile(), but it must really expect InputStream as an input from the payload of the request message.
You also can use the mentioned StreamTransformer over there:
#Bean
IntegrationFlow fileStreamingFlow() {
return IntegrationFlows
.from(s3InboundStreamingMessageSource(),
e -> e.poller(p -> p.fixedDelay(30, TimeUnit.SECONDS)))
.transform(Transformers.fromStream("UTF-8"))
.get();
}
And the next .handle() will be ready for String as a payload.
Here is the situation:
Silverlight 3 Application hits an asp.net hosted WCF service to get a list of items to display in a grid. Once the list is brought down to the client it is cached in IsolatedStorage. This is done by using the DataContractSerializer to serialize all of these objects to a stream which is then zipped and then encrypted. When the application is relaunched, it first loads from the cache (reversing the process above) and the deserializes the objects using the DataContractSerializer.ReadObject() method. All of this was working wonderfully under all scenarios until recently with the entire "load from cache" path (decrypt/unzip/deserialize) taking hundreds of milliseconds at most.
On some development machines but not all (all machines Windows 7) the deserialize process - that is the call to ReadObject(stream) takes several minutes an seems to lock up the entire machine BUT ONLY WHEN RUNNING IN THE DEBUGGER in VS2008. Running the Debug configuration code outside the debugger has no problem.
One thing that seems to look suspicious is that when you turn on stop on Exceptions, you can see that the ReadObject() throws many, many System.FormatException's indicating that a number was not in the correct format. When I turn off "Just My Code" thousands of these get dumped to the screen. None go unhandled. These occur both on the read back from the cache AND on a deserialization at the conclusion of a web service call to get the data from the WCF Service. HOWEVER, these same exceptions occur on my laptop development machine that does not experience the slowness at all. And FWIW, my laptop is really old and my desktop is a 4 core, 6GB RAM beast.
Again, no problems unless running under the debugger in VS2008. Anyone else seem this? Any thoughts?
Here is the bug report link: https://connect.microsoft.com/VisualStudio/feedback/details/539609/very-slow-performance-deserializing-using-datacontractserializer-in-a-silverlight-application-only-in-debugger
EDIT: I now know where the FormatExceptions are coming from. It seems that they are "by design" - they occur when when I have doubles being serialized that are double.NaN so that that xml looks like NaN...It seems that the DCS tries to parse the value as a number, that fails with an exception and then it looks for "NaN" et. al. and handles them. My problem is not that this does not work...it does...it is just that it completely cripples the debugger. Does anyone know how to configure the debugger/vs2008sp1 to handle this more efficiently.
cartden,
You may want to consider switching over to XMLSerializer instead. Here is what I have determined over time:
The XMLSerializer and DataContractSerializer classes provides a simple means of serializing and deserializing object graphs to and from XML.
The key differences are:
1.
XMLSerializer has much smaller payload than DCS if you use [XmlAttribute] instead of [XmlElement]
DCS always store values as elements
2.
DCS is "opt-in" rather than "opt-out"
With DCS you explicitly mark what you want to serialize with [DataMember]
With DCS you can serialize any field or property, even if they are marked protected or private
With DCS you can use [IgnoreDataMember] to have the serializer ignore certain properties
With XMLSerializer public properties are serialized, and need setters to be deserialized
With XmlSerializer you can use [XmlIgnore] to have the serializer ignore public properties
3.
BE AWARE! DCS.ReadObject DOES NOT call constructors during deserialization
If you need to perform initialization, DCS supports the following callback hooks:
[OnDeserializing], [OnDeserialized], [OnSerializing], [OnSerialized]
(also useful for handling versioning issues)
If you want the ability to switch between the two serializers, you can use both sets of attributes simultaneously, as in:
[DataContract]
[XmlRoot]
public class ProfilePerson : NotifyPropertyChanges
{
[XmlAttribute]
[DataMember]
public string FirstName { get { return m_FirstName; } set { SetProperty(ref m_FirstName, value); } }
private string m_FirstName;
[XmlElement]
[DataMember]
public PersonLocation Location { get { return m_Location; } set { SetProperty(ref m_Location, value); } }
private PersonLocation m_Location = new PersonLocation(); // Should change over time
[XmlIgnore]
[IgnoreDataMember]
public Profile ParentProfile { get { return m_ParentProfile; } set { SetProperty(ref m_ParentProfile, value); } }
private Profile m_ParentProfile = null;
public ProfilePerson()
{
}
}
Also, check out my Serializer class that can switch between the two:
using System;
using System.IO;
using System.Runtime.Serialization;
using System.Text;
using System.Xml;
using System.Xml.Serialization;
namespace ClassLibrary
{
// Instantiate this class to serialize objects using either XmlSerializer or DataContractSerializer
internal class Serializer
{
private readonly bool m_bDCS;
internal Serializer(bool bDCS)
{
m_bDCS = bDCS;
}
internal TT Deserialize<TT>(string input)
{
MemoryStream stream = new MemoryStream(input.ToByteArray());
if (m_bDCS)
{
DataContractSerializer dc = new DataContractSerializer(typeof(TT));
return (TT)dc.ReadObject(stream);
}
else
{
XmlSerializer xs = new XmlSerializer(typeof(TT));
return (TT)xs.Deserialize(stream);
}
}
internal string Serialize<TT>(object obj)
{
MemoryStream stream = new MemoryStream();
if (m_bDCS)
{
DataContractSerializer dc = new DataContractSerializer(typeof(TT));
dc.WriteObject(stream, obj);
}
else
{
XmlSerializer xs = new XmlSerializer(typeof(TT));
xs.Serialize(stream, obj);
}
// be aware that the Unicode Byte-Order Mark will be at the front of the string
return stream.ToArray().ToUtfString();
}
internal string SerializeToString<TT>(object obj)
{
StringBuilder builder = new StringBuilder();
XmlWriter xmlWriter = XmlWriter.Create(builder);
if (m_bDCS)
{
DataContractSerializer dc = new DataContractSerializer(typeof(TT));
dc.WriteObject(xmlWriter, obj);
}
else
{
XmlSerializer xs = new XmlSerializer(typeof(TT));
xs.Serialize(xmlWriter, obj);
}
string xml = builder.ToString();
xml = RegexHelper.ReplacePattern(xml, RegexHelper.WildcardToPattern("<?xml*>", WildcardSearch.Anywhere), string.Empty);
xml = RegexHelper.ReplacePattern(xml, RegexHelper.WildcardToPattern(" xmlns:*\"*\"", WildcardSearch.Anywhere), string.Empty);
xml = xml.Replace(Environment.NewLine + " ", string.Empty);
xml = xml.Replace(Environment.NewLine, string.Empty);
return xml;
}
}
}
This is a guess, but I think it is running slow in debug mode because for every exception, it is performing some actions to show the exception in the debug window, etc. If you are running in release mode, these extra steps are not taken.
I've never done this, so I really don't know id it would work, but have you tried just setting that one assembly to run in release mode while all others are set to debug? If I'm right, it may solve your problem. If I'm wrong, then you only waste 1 or 2 minutes.
About your debugging problem, have you tried to disable the exception assistant ? (Tools > Options > Debugging > Enable the exception assistant).
Another point should be the exception handling in Debug > Exceptions : you can disable the user-unhandled stuff for the CLR or only uncheck the System.FormatException exception.
Ok - I figured out the root issue. It was what I alluded to in the EDIT to the main question. The problem was that in the xml, it was correctly serializing doubles that had a value of double.NaN. I was using these values to indicate "na" for when the denominator was 0D. Example: ROE (Return on Equity = Net Income / Average Equity) when Average Equity is 0D would be serialized as:
<ROE>NaN</ROE>
When the DCS tried to de-serialize it, evidently it first tries to read the number and then catches the exception when that fails and then handles the NaN. The problem is that this seems to generate a lot of overhead when in DEBUG mode.
Solution: I changed the property to double? and set it to null instead of NaN. Everything now happens instantly in DEBUG mode now. Thanks to all for your help.
Try disabling some IE addons. In my case, the LastPass toolbar killed my Silverlight debugging. My computer would freeze for minutes each time I interacted with Visual Studio after a breakpoint.