I have a json Object and i am putting it into cache by using a thread which calls for every 5 sec,i want to remove cache data after 10 put oration perform and put that data into third party database.how can i do this,which are the techniques to do this.if have a sample example please share.Thanks
You can achieve a similar behaviour by using a cache store with write-behind along with an expiry policy.
But given the number of records, that you want to keep in the cache, I would do something like this:
private static final int BATCH_SIZE = 10;
private Map<K, V> batch = new HashMap<>();
public void addRecord(K key, V val) {
batch.put(k, v);
if (batch.size() == BATCH_SIZE) {
flush(batch); // Write data into the database.
batch.clear();
}
}
Related
i am using StackExchange.Redis (2.1.58) , i write below code by using cursor .
Dictionary<string, string> keyResult = new Dictionary<string, string>();
var Server = Connection.GetServer(Connection.GetEndPoints()[0]);
long current_cursor = 0;
int next_cursor =-1;
long page_size = 10;
while(next_cursor!=0)
{
var allkeys = Server.Keys(RedisCache.Database, argKeyPattern, 10, 0, next_cursor==-1 ? 0 : next_cursor);
var cursor = ((StackExchange.Redis.IScanningCursor)allkeys);
foreach (var key in allkeys)
{
if (current_cursor == cursor.Cursor)
{
keyResult.Add(key, RedisCache.StringGet(key));
}
else
{
next_cursor = Convert.ToInt32(cursor.Cursor);
current_cursor = next_cursor;
break;
}
}
}
this code works fine , my question is there any other approach to read keys from Redis batch wise in more efficient way ?
Thanks !!
From the documentation
The Keys(...) method deserves special mention: it is unusual in that it does not have an *Async counterpart. The reason for this is that behind the scenes, the system will determine the most appropriate method to use (KEYS vs SCAN, based on the server version), and if possible will use the SCAN approach to hand you back an IEnumerable that does all the paging internally - so you never need to see the implementation details of the cursor operations. If SCAN is not available, it will use KEYS, which can cause blockages at the server.
From first looks, KEYS command should be avoid. However, the library already fix that for you by using SCAN if the command is available. So I think you're good here.
if SCAN command is not available so Keys() will fallback to using KEYS command
SCAN command is available in version 2.8++
The question is simple but I don't know how use it.
For example there is a controller
public class MainController : Controller
{
private int a;
public IActionResult Index(bool set = true)
{
if (set) a = 10;
return View(a)
}
}
If I get in Index page at first time, I set a = 10. And I get in Index page again (for example refresh Index page or paging in Index page, i.e. move in same page) Actually, I get in Index page with url : ~Index?set=False after first access.
Then the a has 0 (default for int variable). I did not know the Controller page (Controller class) is always initialized when I gen in it even when I move to same page.
So, I want to use variable like global variable not using session.
Is there any way?
It sounds like you wish to persist a variable between requests.
Per user
If you wish to store a variable that persists but is only visible to the current user, use session state:
public int? A
{
get
{
return HttpContext.Current.Session["A"] as int?;
}
set
{
HttpContext.Current.Session["A"] = value;
}
}
Note that we are using int? instead of int in order to handle the case where the session variable has not yet been set. If you prefer to default to 0, you can simply use the coalesce operator, ??.
Truly global
If you wish to persist a variable in a manner where there is only one copy for all users, you can store it in a static variable or in an application state variable.
So either
static volatile public int a;
Or
public int? A
{
get
{
return HttpContext.Current.Application["A"] as int?;
}
set
{
HttpContext.Current.Application["A"] = value;
}
}
Obviously variables that are shared between users can change at any time (due to activity in other threads), so you should be careful about how you handle them. For variables that are int-sized or smaller, the processor will perform atomic reads and writes, but for variables larger than an int you may need to use Interlocked or lock to control access.
You do not need to worry about thread synchronization for session variables; the framework handles it for you.
Note: The above is just an example to help you find the right API. It does not necessarily demonstrate the best pattern-- accessing HttpContext via the static method Current is considered bad form, as it makes it impossible to mock the context. Please see this article for ways to expose it to your code via DI.
There's a variable persistence concept I have integrated multiple times:
// Standard initialiation
boolean save = true;
Map<String, Object> dataHolder;
// variables to persist
int number = 10;
String text = "I'm saved";
// Use the variables in various ways in the project
void useVariables() { ... number ... text ...}
// Function to save the variables into a datastructure and for example write them to a file
public Map<String, Object> getVariables()
{
Map<String, Object> data = new LinkedHashMap<String, Object>();
persist(data);
return(data);
}
// Function to load the variables from the datastructure
public void setVariables(Map<String, Object> data)
{
persist(data);
}
void persist(Map<String, Object> data)
{
// If the given datastructure is empty, it means data should be saved
save = (data.isEmpty());
dataHolder = data;
number = handleVariable("theNumber", number);
text = handleVariable("theText", text);
...
}
private Object handleVariable(String name, Object value)
{
// If currently saving
if(save)
dataHolder.put(name, value); // Just add to the datastructure
else // If currently writing
return(dataHolder.get(name)); // Read and return from the datastruct
return(value); // Return the given variable (no change)
}
The main benefit of this principle is that you only have a single script where you have to mention new variables you add during the development and it's one simple line per variable.
Of course you can move the handleVariable() function to a different class which also contains the "save" and "dataHolder" variables so they wont be in the main application.
Additionally you could pass meta-information, etc. for each variable required for persisting the datastructure to a file or similar by saving a custom class which contains this information plus the variable instead of the object itself.
Performance could be improved by keeping track of the order (in another datastructure when first time running through the persist() function) and using a "dataHolder" based on an array instead of a search-based map (-> use an index instead of a name-string).
However, for the first time, I have to document this and so I wondered whether this function-reuse principle has a name.
Does someone recognize this idea?
Thank you very much!
I am working on an app, and need to keep track of how any views a page has. Almost like how SO does it. It is a value used to determine how popular a given page is.
I am concerned that writing to the DB every time a new view needs to be recorded will impact performance. I know this borderline pre-optimization, but I have experienced the problem before. Anyway, the value doesn't need to be real time; it is OK if it is delayed by 10 minutes or so. I was thinking that caching the data, and doing one large write every X minutes should help.
I am running on Windows Azure, so the Appfabric cache is available to me. My original plan was to create some sort of compound key (PostID:UserID), and tag the key with "pageview". Appfabric allows you to get all keys by tag. Thus I could let them build up, and do one bulk insert into my table instead of many small writes. The table looks like this, but is open to change.
int PageID | guid userID | DateTime ViewTimeStamp
The website would still get the value from the database, writes would just be delayed, make sense?
I just read that the Windows Azure Appfabric cache does not support tag based searches, so it pretty much negates my idea.
My question is, how would you accomplish this? I am new to Azure, so I am not sure what my options are. Is there a way to use the cache without tag based searches? I am just looking for advice on how to delay these writes to SQL.
You might want to take a look at http://www.apathybutton.com (and the Cloud Cover episode it links to), which talks about a highly scalable way to count things. (It might be overkill for your needs, but hopefully it gives you some options.)
You could keep a queue in memory and on a timer drain the queue, collapse the queued items by totaling the counts by page and write in one SQL batch/round trip. For example, using a TVP you could write the queued totals with one sproc call.
That of course doesn't guarantee the view counts get written since its in memory and latently written but page counts shouldn't be critical data and crashes should be rare.
You might want to have a look at how the "diagnostics" feature in Azure works. Not because you would use diagnostics for what you are doing at all, but because it is dealing with a similar problem and may provide some inspiration. I am just about to implement a data auditing feature and I want to log that to table storage so also want to delay and bunch the updates together and I have taken a lot of inspiration from diagnostics.
Now, the way Diagnostics in Azure works is that each role starts a little background "transfer" thread. So, whenever you write any traces then that gets stored in a list in local memory and the background thread will (by default) bunch all the requests up and transfer them to table storage every minute.
In your scenario, I would let each role instance keep track of a count of hits and then use a background thread to update the database every minute or so.
I would probably use something like a static ConcurrentDictionary (or one hanging off a singleton) on each webrole with each hit incrementing the counter for the page identifier. You'd need to have some thread handling code to allow multiple request to update the same counter in the list. Alternatively, just allow each "hit" to add a new record to a shared thread-safe list.
Then, have a background thread once per minute increment the database with the number of hits per page since last time and reset the local counter to 0 or empty the shared list if you are going with that approach (again, be careful about the multi threading and locking).
The important thing is to make sure your database update is atomic; If you do a read-current-count from the database, increment it and then write it back then you may have two different web role instances doing this at the same time and thus losing one update.
EDIT:
Here is a quick sample of how you could go about this.
using System.Collections.Concurrent;
using System.Data.SqlClient;
using System.Threading;
using System;
using System.Collections.Generic;
using System.Linq;
class Program
{
static void Main(string[] args)
{
// You would put this in your Application_start for the web role
Thread hitTransfer = new Thread(() => HitCounter.Run(new TimeSpan(0, 0, 1))); // You'd probably want the transfer to happen once a minute rather than once a second
hitTransfer.Start();
//Testing code - this just simulates various web threads being hit and adding hits to the counter
RunTestWorkerThreads(5);
Thread.Sleep(5000);
// You would put the following line in your Application shutdown
HitCounter.StopRunning(); // You could do some cleverer stuff with aborting threads, joining the thread etc but you probably won't need to
Console.WriteLine("Finished...");
Console.ReadKey();
}
private static void RunTestWorkerThreads(int workerCount)
{
Thread[] workerThreads = new Thread[workerCount];
for (int i = 0; i < workerCount; i++)
{
workerThreads[i] = new Thread(
(tagname) =>
{
Random rnd = new Random();
for (int j = 0; j < 300; j++)
{
HitCounter.LogHit(tagname.ToString());
Thread.Sleep(rnd.Next(0, 5));
}
});
workerThreads[i].Start("TAG" + i);
}
foreach (var t in workerThreads)
{
t.Join();
}
Console.WriteLine("All threads finished...");
}
}
public static class HitCounter
{
private static System.Collections.Concurrent.ConcurrentQueue<string> hits;
private static object transferlock = new object();
private static volatile bool stopRunning = false;
static HitCounter()
{
hits = new ConcurrentQueue<string>();
}
public static void LogHit(string tag)
{
hits.Enqueue(tag);
}
public static void Run(TimeSpan transferInterval)
{
while (!stopRunning)
{
Transfer();
Thread.Sleep(transferInterval);
}
}
public static void StopRunning()
{
stopRunning = true;
Transfer();
}
private static void Transfer()
{
lock(transferlock)
{
var tags = GetPendingTags();
var hitCounts = from tag in tags
group tag by tag
into g
select new KeyValuePair<string, int>(g.Key, g.Count());
WriteHits(hitCounts);
}
}
private static void WriteHits(IEnumerable<KeyValuePair<string, int>> hitCounts)
{
// NOTE: I don't usually use sql commands directly and have not tested the below
// The idea is that the update should be atomic so even though you have multiple
// web servers all issuing similar update commands, potentially at the same time,
// they should all commit. I do urge you to test this part as I cannot promise this code
// will work as-is
//using (SqlConnection con = new SqlConnection("xyz"))
//{
// foreach (var hitCount in hitCounts.OrderBy(h => h.Key))
// {
// var cmd = con.CreateCommand();
// cmd.CommandText = "update hits set count = count + #count where tag = #tag";
// cmd.Parameters.AddWithValue("#count", hitCount.Value);
// cmd.Parameters.AddWithValue("#tag", hitCount.Key);
// cmd.ExecuteNonQuery();
// }
//}
Console.WriteLine("Writing....");
foreach (var hitCount in hitCounts.OrderBy(h => h.Key))
{
Console.WriteLine(String.Format("{0}\t{1}", hitCount.Key, hitCount.Value));
}
}
private static IEnumerable<string> GetPendingTags()
{
List<string> hitlist = new List<string>();
var currentCount = hits.Count();
for (int i = 0; i < currentCount; i++)
{
string tag = null;
if (hits.TryDequeue(out tag))
{
hitlist.Add(tag);
}
}
return hitlist;
}
}
It's more like theoretical question.
I have one table to hold dictionary items, and the next one for hold Users data.
User table contains a lot reference collumns of type many to one indicated on dictionary item table. It's looks like:
public class User
{
public int Id;
public Dictionary Status;
public Dictionary Type;
public Dictionary OrganizationUnit;
......
}
I want retrieve all dictionary on startup of aplication, and then when i retrieved user and invoke reference property to dictionary the dictionary object should be taken from cache.
I know i can use a 2nd level cache in this scenario, but i'm interested about other solution. Is there any?
It's posible to make my custom type and said that: use my custom cache to retrieved value of dictionary??
Across multiple session the second level cache is the best answer, the only other solutions to populate objects from a cache without using second level cache i can think of would be to use an onLoad interceptor (and simply leave your dictionaries unmapped) or do it manually somewhere in your application.
But why don't you want to use the seocondlevel cache? If your views on caching is very different from the storages there are providers for in hibernate it is possible for you to implement your own provider?
Why not store it in the session? Just pull the record set one time and push it into session and retrieve it each time you want it. I do something similar for other stuff and I believe my method should work for you. In my code I have a session manager that I call directly from any piece of code needs the session values. I choose this method since I can query the results and I can manipulate the storage and retrieval methods. When relying on NHibernate to do the Caching for me, I don't have the granularity of control to cause specific record sets to only be available to specific sessions. I also find that NHibernate is not as efficient as using the session directly. When profiling the CPU and memory usage I find that this method is faster and uses a little less memory. If you want to do it on a site level instead of session, look into HttpContext.Current.Cache.
The following example works perfectly for storing and retrieving record sets:
// Set the session
SessionManager.User = (Some code to pull the user record with relationships. Set the fetch mode to eager for each relationship else you will just have broken references.)
// Get the session
User myUser = SessionManager.User;
public static class SessionManager
{
public static User User
{
get { return GetSession("MySessionUser") as User; }
set { SetSession("MySessionUser", value); }
}
private static object GetSession(string key)
{
// Fix Null reference error
if (System.Web.HttpContext.Current == null || System.Web.HttpContext.Current.Session == null)
{
return null;
}
else
{
return System.Web.HttpContext.Current.Session[key];
}
}
private static void SetSession(string key, object valueIn)
{
// Fix null reference error
if (System.Web.HttpContext.Current.Session[key] == null)
{
System.Web.HttpContext.Current.Session.Add(key, valueIn);
}
else
{
System.Web.HttpContext.Current.Session[key] = valueIn;
}
}
}