For my usage of NServiceBus, I really do not need to get an updated list of subscribers once for each publish. I only need to get it one time at startup, so as to avoid the wasted time on querying this over and over again.
I found discussion about caching the subscriber list - here, but in the end it does not provide a solution. Can anyone provide a concrete solution? I am using NSB 4.6.4.
Unless you are seeing an actual performance problem with fetching subscribers, I would see this as premature optimization. If you are using Raven, the database fetch should be very, very fast.
It sounds like they will implement caching in NHibernate in 5.0, so you may have to wait for that if you are using SQL. I'm not sure how subscription data is stored in SQL, so I can't comment on how fast/slow it is currently.
Related
I am reading events from Kafka and depositing into REDIS. Then, we read events using Python and in case we don’t find events we drop/re-create the index.
However, I noticed at times even after re-creating the index I still don’t find events.
I have couple of questions -
[Q1] Is re-indexing a good approach where we are continuously getting a huge flow of events?
[Q2] Also, I noticed during REDIS search at times I do get events and then at another instance query does not return results, can this be related to dropping / re-creating index?
[Q3] Is there a better standard approach to ensure JSON are deposited / retrieved consistenly.
[Q4] Is there an explanation as to why at times everything just seems to work fine continuously for several hours and then does not work at all for few hours.
I would appreciate alternate approaches for this simple use case as I am fairly new to REDIS
Here are some points. I hope it would be helpful.
[Q1]
Re-indexing can be a good approach if you are getting a huge flow of events, as it will help keep your data up-to-date. However, you also need to make sure that your Redis instance is able to handle the increased load otherwise Redis can become very slow when it is re-indexing and may not be able to keep up with the flow of event. And in that case, you may need to scale your Redis instance to handle the load or incremental indexing may be a better option.
[Q2]
There could be many reasons why search results vary, but it is possible that the index is being dropped and re-created, which would cause the event to not be found. There are a few other things that could be happening:
There could be an issue with the search algorithm, which would cause search results to not be returned.
The data in the database could be changing, which would cause search results to not be returned.
[Q3]
There is no one standard approach to ensure JSON are deposited or retrieved consistently. However, some best practices you may consider include using a library or tool that supports serialization and deserialization of JSON data, validating input and output data, and using an editor that highlights errors in JSON syntax. You can also use locking mechanism to ensure that only one process can write to the Redis instance at a time, or using a queue to buffer writes to Redis. Also, different developers may have different preferences and opinions on the best way to handle JSON in Redis. Some possible methods include using commands such as JSET or JGET to manage JSON objects, or using a library such as JRedis to simplify the process.
[Q4]
Redis can be temperamental, and its behavior can vary depending on the specific configuration and usage scenarios. There is no specific explanation for this behavior, but it could be due to various factors such as load on the Redis server, network conditions, or other applications using the same Redis instance. In that case, server will not be able to handle requests properly and will stop responding. If everything is working fine for a few hours and then suddenly stops working, you can try restarting Redis or checking the logs for any errors that may have occurred.
I have a project with requirements that response time should be under 0.5sec under load as 3000 concurrency users;
I have few API which use some aggregation from SQL Server.
when we testing it with 3000CCU average response time about 15 second. And also 500 error due to SQL can't handle so many requests. Actually requests to SQL Server interrupt with timeout)
Our current instance is r4.2xlarge 8CPU and 61GB Memory.
All code is asynchronous without blocking operations.
We running our app behind load balancer with 10 instances 300 CCU per instance in this case. utilization on instances about 30%. The bottleneck currently is SQL server.
I see few solution. Setup some big SQL, Cluster or Sharding, I'm not really sure. I'm not strong in that.
Or use cache for request. We have mostly read only data, which we can cache after aggregation.
Update:
I need solution to cache exactly sql responses. To order work with it late with LINQ.
But it seems there is no ready solution for that.
I found good try for that called CacheManager. But there are few problems exist with this.
It works with Redis only in sync mode, means use sync command instead of async.
There is no implementation of concurrency lock, which can occur in our case because we have 10 instances. We need solution which work with as distributed cache.
There are few bugs which utilize Redis multiplexor wrong. And you constantly will have connection problem.
Please advice how to overcome this issue. How you solve it. I'm sure there are people who already solve it somehow.
I enable query store on sql and monitor all missing indexes. Ef core generates some of request absolutely different way that expected. After creating missing indexes performance became much better. But i still have problem to handle required CCU. I explore all existing solution which extend ef core to cache. Most of them was written in sync version. Which just can’t utilize all benefits of async. As well i did’t found any distributed cache that implement distributed lock. Finally I create this lib which extend ef core and and distributed cache in redis .cache allow us scale a lot more. And now everything just flight;) leave it here, for someone who have performance issue like me. https://github.com/grinay/Microsoft.EntityFrameworkCore.DistributedCache
So the question is mostly in the title but after some research I can't really find any deeper information about this. Mostly I want to know if a deadlock situation occurs does Breeze automatically reattempt the commit or does it just return an error back to the front end to try saving again? Any documentation or articles going deeper into this would be appreciated!
To a certain extent this depends on the server backend that you are using. But in general breeze will NOT attempt to retry a deadlock failure and will instead return an exception indicating that a deadlock occurred. You can then retry the save yourself by handling the client side exception and reexecuting the query.
Note that because of the way that most breeze servers automatically toposort the entities in a save request, deadlocks are much less likely than if such a sort was not performed. The idea here is that by ensuring that multiple instances of a program use the same ordering when updating the same set of tables, we reduce the possibility of a deadlock.
This toposorting is part of any Entity Framework based backend as well as the Breeze Node/Sequelize (MySQL, Postgress) provider, and is likely to be added to the Breeze NHibernate and MongoDb providers in the near future.
I have a database question. I am developing an application where users sends some request and gets an answer from a vendor. I have a server receiving the request (through a rest call or a running web service, haven't decided which yet).
Whenever a new request comes in it should be logged in a database and when the vendor responds the record should be updated indicating whether it was accepted or not and stuff like that. The only reason for this storage of transactions is for reporting and logging purposes. So now that I have stated my requirement I need help from someone with more expertise in this.
What I've come up with so far is that it would be best to use a structured database since all records will have one type and the same information, so there's no need to waste space using a semi-structured database with each record containing both structure and information.
But I don't know if there are any databases that are particularly good for this kind of "create/update operations only" ?? As I said I only need to read the data perhaps once a month or so.
Any inputs are appreciated!
You can use any open source database like postgreSql as you are mostly going to do inserts and not much other features needed. My suggestion will try to put logging process in separate threads rather than the one you are using for processing to have better performance for your api calls.
I'm developing a application with a lot of create/update queries and currently using Neo4j.
It's fast and really good with j2E and php. NoSQL is really fast to learn with it, and the web interface is really user friendly :)
We are working with a .NET 3.5 app which is fast approaching legacy status. We have an existing SOAP service which reads records from our database and saves them to a third party MS SQL database, sending all the data rows in a single batch.
This has always worked fine, but recently we've taken on a much larger client than any we've had before, and they are transmitting much larger batches, so much so that they have begun to fail. We've upped the time out and max memory sizes in IIS, and maxed out the maxRequestLength in the web.config, but we are still bumping up against size problems.
So, I understand that long term, we should consider moving away from SOAP and into WCF, and plans for that are in the works. But in the mean time, we need a short term fix for this new client. And of course, to make the business and sales people happy, we need it kinda quickly.
I'm wondering what the best-practice approach might be. Initially I'm thinking something like this, but I could be thinking inside the box too much:
Establish a bench mark of # of records over which we don’t want to attempt to sync all at once.
Before attempting to save the data, check the number of records against that bench mark
If it's above it, then break the transmission down into segments which are each below that benchmark. SELECT TOP 10000 * FROM table WHERE sent = false, etc., if the benchmark is 10000. Then update sent to true for those records once submitted. Repeat.
Obviously, this will slow the process down, so to handle the user experience, we may want to toss in a status bar so they can see the progress.
Am I on the right track?
In addition to the comments from John, you should consider if you are solving the problem in the most optimal way.
It looks like you are triggering a one way sync between 2 database by calling a web service. This approach leads to the time out and memory problems that you are experiencing.
If your goal is to do the one way sync, you could use a free framework such as Microsofts sync framework: http://msdn.microsoft.com/en-US/sync