Not sure if this is even the correct place to ask, but I couldn't find any relevant information on this topic (apart from an old forum post that was last answered a year ago).
Like the question says, does anyone know if it's possible to specify a AWS or GCP region that FaunaDB will use?
I saw that on the Pricing page, the Business Plan offers Data locality* and Premium regions* but they are marked as future feature, with no further information like a roadmap or planned release quarter.
Many of my clients are Canadian or from Europe and they are already asking me about hosting their data in their own country. I know that AWS and Google offer data center locations in Canada (for example), so I'm just looking for any further information on this and if/when it will be possible.
I really, really don't want to have to host my own database on a private server.
Thanks in advance!
It is not possible to specify an AWS region. Fauna database transactional replication involves all deployed regions.
We are working towards the data locality feature, but is not available yet nor does it have a finalized definition.
When the data locality feature is closer to completion, we'll be able to say more.
Hi ouairz As eskwayrd mentions, it's not possible to select individual regions in which to locate your data. We do plan to provide a a set of individual replication zones across the globe which you select to control your data residency needs. For example, there would be a EU zone which would you would use for data that must stay resident to EU member states. Other zones may include, for instance, Asia, Australia, North America, etc. We are considering a Canadian zone. Please feel free to reach out to product#fauna.com for more details
Related
For compliance requirements, we would like to move all our Bigquery data and GCS data from US region to EU region.
From my understanding, multi-region is either within US or within EU. There is no cross-region option as such.
Question 1: In order to move the data from US to EU or vice versa, our understanding is that we need explicitly move the data using a storage transfer service. And assuming a cost associated with this movement even though it is within Google cloud?
Question 2: We also think if we can maintain copies at both locations. In this case, Is there a provision for cross-region replication? If so, what would be the associated cost for the same?
Question1:
You are moving data from one part of the world to another one. So, yes you will pay the egress cost of the source location.
Sadly, today (November 28th 2023), I can't 100% commit on that cost. Indeed, I reached Google Cloud about a very similar question and my Google Cloud contact told me that the cost page was out of date. The Cloud Storage egress cost should apply (instead of the Compute Engine Networking egress cost as today in the documentation).
Question2:
You copy the data, so, you have, at the end, the volume of data duplicated in 2 dataset and you have your storage cost duplicated.
Every time that you want to sync the data, you perform a copy. It's only a copy, and not a smart delta update. So, be careful if you update directly the data in the target dataset: a new copy will override the data!
Instead, use the target dataset as a base to query, and duplicate (again) the data in an independent dataset, where you can add your region specific data
According to the docs, once the dataset is created, the location cannot be changed, but you can copy the dataset to a different location, or manually move (recreate) the dataset in a different location.
The easier approach is copy, you can learn more about the requirements, quotas and limitations here: https://cloud.google.com/bigquery/docs/copying-datasets
So:
There is no need for the transfer service, you can copy datasets to a different location.
There is no mechanism for automatic replication across regions. Even a disaster recovery policy will require cross-region datasets copies.
BigQuery does not automatically provide a backup or replica of your data in another geographic region. You can create cross-region dataset copies to enhance your disaster recovery strategy.
https://cloud.google.com/bigquery/docs/availability#:%7E:text=cross%2Dregion%20dataset%20copies
So in both cases you need to work with dataset copies and deal with data freshness in the second scenario.
I am trying to download as much information from Bloomberg for as many securities as I can. This is for a machine learning project, and I would like to have the data reside locally, rather than querying for it each time I need it. I know how to download information for a few fields for a specified security.
Unfortunately, I am pretty new to Bloomberg. I've taken a look at the excel add-in, and it doesn't allow me to specify that I want ALL securities and ALL their data fields.
Is there a way to blanket download data from Bloomberg via excel? Or do I have to do this programmatically. Appreciate any help on how to do this.
Such a request in unreasonable. Bloomberg has dozens of thousands of fields for each security. From fundamental fields like sales, through technical analysis like Bollinger bands and even whether CEO is a woman and if the company abides by Islamic law. I doubt all of these interest you.
Also, some fields come in "flavors". Bloomberg allows you to set arguments when requesting a field, these are called "overrides". For example, when asking for an analyst recommendation, you could specify whether you're interested in yearly or quarterly recommendation, you could also specify how do you want the recommendation consensus calculated? Are you interested in GAAP or IFRS reporting? What type of insider buys do you want to consider? I hope I'm making it clear, the possibilities are endless.
My recommendation is, when approaching a project like you're describing: think in advance what aspects of the security do you want to focus on? Are you looking for value? growth? technical analysis? news? Then "sit down" with a Bloomberg rep and ask what fields apply to this aspect. Then download those fields.
Also, try to reduce your universe of securities. Bloomberg has data for hundreds of thousands of equities. The total number of securities (including non equities) is probably many millions. You should reduce that universe to securities that interest you (only EU? only US? only above certain market capitalization?). This could make you research more applicable to real life. What I mean is that if you find out that certain behavior indicates a stock is going to go up - but you can't buy that stock - then that's not that interesting.
I hope this helps, even if it doesn't really answer the question.
They have specific "Data Licence" products available if you or your company can fork out the (likely high) sums of money for bulk data dumps. Otherwise, as has been mentioned, there are daily and monthly restrictions on how much data (and what type of data) is downloaded via their API. These limits are not very high at all and so, by the sounds of your request, this will take a long and frustrating time. I think the daily limits are something like 500,000 hits, where one hit is one item of data, e.g. a price for one stock. So if you wanted to download only share price data for the 2500 or so US stocks, you'd only managed 200 days for each stock before hitting the limit. And they also monitor your usage, so if you were consistently hitting 500,000 each day - you'd get a phone call.
One tedious way around this is to manually retrieve data via the clipboard. You can load a chart of something (GP), right click and copy data to clipboard. This stores all data points that are on display, which you can dump in excel. This is obviously an extremely inefficient method but, crucially, has no impact on your data limits.
Unfortunately you will find no answer to your (somewhat unreasonable) request, without getting your wallet out. Data ain't cheap. Especially not "all securities and all data".
You say you want to download "ALL securities and ALL their data fields." You can't.
You should go to WAPI on your terminal and look at the terms of service.
From the "extended rules:"
There is a daily limit to the number of hits you can make to our data servers via the Bloomberg API. A "hit" is defined as one request for a singled security/field pairing. Therefore, if you request static data for 5 fields and 10 securities, that will translate into a total of 50 hits.
There is a limit to the number of unique securities you can monitor at any one time, where the number of fields is unlimited via the Bloomberg API.
There is a monthly limit that is based on the volume of unique securities being requested per category (i.e. historical, derived, intraday, pricing, descriptive) from our data servers via the Bloomberg API.
I am currently building a website which sends SPARQL queries to get information from DBpedia (mostly name of cities which can then be displayed on a map, and further information about this places like number of inhabitants, prominent persons etc).
I would like your opinion about the general use of SPARQL queries and DBpedia:
is it preferable that I create a database specific for my website with the information that I need, and I use DBpedia queries only to update this database at regular intervals (e.g. everyday).
or is it ok if my website sends queries systematically to DBpedia when someone comes and visit this website?
The second option is much easier for me to implement because and I don't need to worry about the extra database,
but if everybody does the same it may surcharge the DBpedia servers?
DBPedia is frequently down or unresponsive for the reasons you cite - there can be unanticipated periods of high volume on the servers. So the first choice, caching to a local data store, is probably best.
The public DBpedia endpoint which is made available at no cost to the Web community at large may not be suited to such use as you describe -- being the backend to what seems likely to be a freemium service of some kind.
You can instantiate your own mirror of the DBpedia service on AWS or elsewhere, if you need a more consistently available instance.
I am looking for a laundry list of reasons why a large company with 24000 employees would NOT want to put their Primary SharePoint system for their internet into the cloud?
What are the limitations and challenges compared to operating your own farm servers.
Thank you for your thoughts.
Depending on the provider, they might limit your choice of webparts, addons, solutions, etc.
Check for:
Addons that contain unmanaged code (BPOS does not allow this, for example)
Addons that need to elevate privileges
Anything that needs to run in a full-trust environment
Ask about any other possible limitation. At 24k users, you probably are only looking at high-end providers, but ask, just in case.
You mean apart from the fact that hosting your entire sensitive company data, trade secrets, potential HR data at a third party that may or may not do a good job securing it from other customers on the same cloud may be just a tiny little risk?
Or that if the provider has an outage (like the Amazon S3 blackout yesterday) leaves you somewhat powerless and at the mercy of the provider?
We host about 20 separate web applications and counting, each having atleast one unique top level domain name. Almost all of which are .com.au. We also manage our own DNS name servers. All on Australian soil.
A couple times I have suggested moving our Australian VPS instances to overseas providers namely in the US as they provide much better value for money. We have decent enough caching strategies in place that the increased latency won't be a real issue.
However I'm often met with the response, "having our servers overseas affects SEO". We've since had a major issue that will lead to infrastructure changes and I would like to suggest once again that we move our servers to the US.
My question is, is there any weight to this statement? And if so, are there any ways around this affect?
Possible reverse proxying all requests through an Australian server? Just throwing ideas out there.
If I remember correctly, it does matter. Google and other search engines look to provide the most relevant content to searchers, and therefore, if I am in the US, it thinks that I probably don't want to look for information in China or Russia or Australia. Also you will need to be careful of speeds if you do reverse proxy. Outside of location based search indexing, Google will lower SEO if loading speeds are too high. If that is the route you take then I would suggest using a CDN within or near Australia to serve any static files, like JS CSS and images, which will reduce page speeds.