Other answers give are outdated
The only option I found in the docs is ZUNIONSCORE which is great, but it forces me to save the results somewhere, and then fetch to retrieve it.
Is there a way to do the sum, on redis side, and not save the results? (since then I need to manually expire it)
Yes, you can use Lua scripting embedded in Redis.
Related
Thanks to the help in this forum, I got my SQL-conncection and inserts working now.
The following TR-formula is used to retrieve the data from Excel Eikon:
#TR($C3,"TR.CLOSEPRICE (adjusted=0);
TR.CompanySharesOutstanding;
TR.Volume;
TR.TURNOVER"," NULL=Null CODE=MULTI Frq=D SDate="&$A3&" EDate="&$A3)
For 100k RICs the formulas usually need between 30s and 120s to refresh. That would still be acceptable.
The problem is to get the same refresh-speed in a VBA-loop. Application.Run "EikonRefreshWorksheet" is currently used for a synchronous refresh as recommended in this post.
https://community.developers.refinitiv.com/questions/20247/can-you-please-send-me-the-excel-vba-code-which-ex.html
The syntax of the code is correct and working for 100 RICS. But already for 1k the fetching gets very slow and will freeze completely for like 50k. Even with a timeout interval of 5min.
I isolated the refresh-part. There is nothing else slowing it down. So is this maybe just not the right method for fetching larger data sets? Does anyone know a better alternative?
I finally got some good advice from the Refinitiv Developer Forum which I wanted to share here:
I think you should be using the APIs directly as opposed to opening a spreadsheet and taking the data from that - but all our APIs have limits in place. There are limits for the worksheet functions as well (which use the same backend services as our APIs) - as I think you have been finding out.
You are able to use our older Eikon COM APIs directly in VBA. In this instance you would want to use the DEX2 API to open a session and download the data. You can find more details and a DEX2 tutorial sample here:
https://developers.refinitiv.com/en/api-catalog/eikon/com-apis-for-use-in-microsoft-office/tutorials#tutorial-6-data-engine-dex-2
However, I would recommend using our Eikon Data API in the Python environment as it is much more modern and will give you a much better experience than the COM APIs. If you have a list of 50K instruments say - you could make 10 API calls of say 5K instruments using some chunking and it would all be much easier for you to manage - without even resorting to Excel - and then you can use any Python SQL tool to ingest into any database you wish - all from one python script.
import refinitiv.dataplatform.eikon as ek
ek.set_app_key('YOUR APPKEY HERE')
riclist = ['VOD.L','IBM.N','TSLA.O']
df,err = ek.get_data(riclist,["TR.CLOSEPRICE(adjusted=0).date","TR.CLOSEPRICE(adjusted=0)",'TR.CompanySharesOutstanding','TR.Volume','TR.TURNOVER'])
df
#df.to_sql - see note below
#df.to_csv("test1.csv")
1641297076596.png
This will return you a pandas dataframe that you can easily directly write into any SQLAlchemy Database for example (see example here) or CSV / JSON for example.
Unfortunately, our company policy does not allow for Python at the moment. But the VBA-solution also worked, even though it took some time to understand the tutorial and it has more limitations.
I have a few question about ordered lists in Aerospike:
How can I see in the DB, using aql, if the list is ordered or not?
Does ordered list means it’s sorted?
I want to scan the db and change all lists (in a specific bin) to be ordered. I want to do is using set_type, but I can’t seem to make it work. Is that possible? how can I do it?
Thanks
I'm posting my answer from your cross-posted question here https://discuss.aerospike.com/t/list-oprations/5282:
You could scan the namespace with a ScanPolicy.includeBinData=false and for each record digest you get back use operate() to wrap the following operations into a single transaction:
ListOperation.setOrder() to ListOrder.ORDERED
ListOperation.sort() with a ListSortFlags.DROP_DUPLICATES
You will only need to run this once to clean up your database.
The ordering type will stick for all future operations. You'd just continue to use the ListWriteFlags.ADD_UNIQUE list policy.
This is for the Java client, but all other clients have these operations and policies in them.
I don't think AQL is the right tool to exploit the full power of lists. Perhaps it is not yet updated to the full functionality of lists. It is built on top of the C client. At least AQL ver 3.15.2.1 that I checked with is not. You might want to write a java client application.
I'm looking for the best solution to store the ettings for a website, like the limit of posts for users, limit of users online, ranks, min. number of posts to be able to do something.
Like here, if you're new you can't thumbs up/down a post, or whatever, so how would you store all of these?
I thought of creating a table with constants in mysql but i think it's not the best solution to add a new mysql query on every page refresh.
Why not? After all, MySQL can handle a large number of requests and I've seen much more complex queries than checking access rights. A MySQL query is a query to a file just like checking an INI file, but optimized. I'm guessing that if you don't expect a huge amount of traffic, you'll be fine with a database.
Here it's a matter of preference. I prefer to do this in MySQL because I don't like to parse files and find querying a database easier. Also, editing rows is easier than changing values in a text file.
I'd say your first thought was spot-on. Put constants into a database.
OK, first let me state that I have never used this control and this is also my first attempt at using a web service.
My dilemma is as follows. I need to query a database to get back a certain column and use that for my autocomplete. Obviously I don't want the query to run every time a user types another word in the textbox, so my best guess is to run the query once then use that dataset, array, list or whatever to then filter for the autocomplete extender...
I am kinda lost any suggestions??
Why not keep track of the query executed by the user in a session variable, then use that to filter any further results?
The trick to preventing the database from overloading I think is really to just limit how frequently the auto updater is allowed to update, something like once per 2 seconds seems reasonable to me.
What I would do is this: Store the current list returned by the query for word A server side and tie that to a session variable. This should be basically the entire list I would think. Then, for each new word typed, so long as the original word A exists, you can filter the session info and spit the filtered results out without having to query again. So basically, only query again when word A changes.
I'm using "session" in a PHP sense, you may be using a different language with different terminology, but the concept should be the same.
This question depends upon how transactional your data store is. Obviously if you are looking for US states (a data collection that would not change realistically through the life of the application) then I would either cache a System.Collection.Generic List<> type or if you wanted a DataTable.
You could easily set up a cache of the data you wish to query to be dependent upon an XML file or database so that your extender always queries the data object casted from the cache and the cache object is only updated when the datasource changes.
RAM is cheap and SQL is harder to scale than IIS so cache everything in memory:
your entire data source if is not
too large to load it in reasonable
time,
precalculated data,
autocomplete webservice responses.
Depending on your autocomplete desired behavior and performance you may want to precalculate data and create redundant structures optimized for reading. Make use of structs like SortedList (when you need sth like 'select top x ... where z like #query+'%'), Hashtable,...
While caching everything is certainly a good idea, your question about which data structure to use is an issue that wasn't fully answered here.
The best data structure for an autocomplete extender is a Trie.
You can find a good .NET article and code here.
A Google search for "site:example.com" will tell you the number of pages of example.com that are currently in Google's index. Is it possible to find out how this number has changed over time?
HubSpot does this for you. It costs money but they do a lot of useful things like this.
If you don't mind waiting, you could have a cron parse your site:example.com results every day and wait for the data to build up.
I was going to suggest Google Webmaster Tools, but it doesn't appear to have this information. How irritating.
Anyway, to follow on from UltimateBrent's answer, this regular expression will extract the value from a google search:
\d+(?=</b> from <b>domain\.net</b>)
Obviously, changing domain\.net to whatever your domain is.
I set up a Python script as a cron job to parse the result from the Google results page and save it. I set it to run once per day for each of a set of sites. I wrote another script to produce a CSV spreadsheet from the data. I can open that in a spreadsheet program and quickly make charts to visualise trends.
I have similar scripts for monitoring PageRank.
This will still only give me data from the day I begin checking. I do not know of a way to access historical values.
you can use domaintools.com
Rahul
http://Valdot.com