I have a requirement and I am not sure if I should use Analysis services or Reporting services or some other technique.
My client wants to show special deals from a database on their online website. They want to target users, i.e. if user is from UK; show deals for UK and in Pound. if user is from Canada; show deals for Canada and in canadian dollar etc.
Their database has multiple tables loaded with 1 to 2 million records in each table. Each table is for a different category of products and has a currency and a Country column to filter. I cannot restructure their schema as they have huge amount of development done to integrate with various buisness applications.
I need a solution which involves datawarehouse, can fetch data quickly and cache it for next 12 or 24 hours (Do not want to cache on web server). I do not have much experience in Analysis and Reporting services so I need your solution/suggestion and anything you can share from your good or bad experiences.
Analysis services is not what you want here: you do not need cubes that summerize info.
Nor is reporting services: you will want to display your data in plain HTML.
I would just query the existing data and display that data. If performance becomes an issue you could run an SSIS job every 12 hours to extract data to a specific database you created for this application. But consider tweaking your indexes first.
Related
Setup today
We currently have two SQL databases in our setup:
Production database
Develop database
We keep the table structures updated using Entity Framework Code First migrations, which is great because it's in version control.
Challenge: how to handle data.
We have never migrated production data from production to develop, simply because we have a lot of sensitive data we don't want (or are even allowed because of less security) on our development database.
This includes:
Private data on users
Emails on customers and users
However, the PROD and TEST website has started to differ a lot in content and looks, which is a problem because our environments diverge a lot.
What I am looking for an answer on:
My question: How do you easily migrate data from production to develop, but "hide" sensitive data?
In my imagination I am looking for something that:
Every week automatically migrate data
Scramble some fields (ie. move some values around, remove/add some letters in some tables)
But I have no idea how this is usually done in a professional environment.
Technology stack
We use Azure Database, .NET / Visual Studio and Entity Framework as an ORM.
I was contracted to a Fortune 50 company's finance department for a while, and faced this issue. We wrote a custom application which would connect to the production database, extract the data we needed to load into the development database, and selectively scramble certain fields. We used production data as the source due to the nature of the...data 'scenarios' that users had generated in the production data, and we wanted to represent these interrelationships as closely as possible in the development environment.
For example, meaningful textual names of certain elements were obfuscated randomly against a large table of strings in a text file, and numeric data values were multiplied against a number randomly chosen from within a limited range that was meaningful for that field. We did not obfuscate everything, but only those data elements that were considered sensitive.
Because these records required a certain level of data integrity against related records in other tables, we found that only a custom app would satisfy our needs.
In your situation (user names and email addresses) it would be relatively straightforward to set up some dummy data, from which an app could generate "fake" names and emails.
I'm building a talent management CRM application and I'm having trouble choosing between a SQL or NoSQL database for my data.
The application will only have a few 'core' entities (Person, Job, Company, Interview), and will rely heavily on 'tagging' of those entities. You can add Tags and Notes to a Person, a Job, a Company, and then sort/search data by those tags.
What I learned about NoSQL is that I can just have a Person object (document) with an array of Tags and Notes, where in SQL I would need separate Tags and Notes tables and construct joins to gather all my data for a Person.
Could anyone give me some pointers on what would be the way to go for my particular scenario?
Thanks!
Our ERP system is based on UniData (NoSQL), it is okay for performing the standard tasks needed to do business like entering in customers, creating sales orders, invoicing etc. But when it comes to creating reports that were not originally foreseen it is quite cumbersome. The system only lets you create reports off of one table, if you need data from another table you have two options: 1. Create what is called a virtual attribute for every field you need to look up from a different table, Or write a UniBasic program to retrieve the data needed.
To meet most of our business needs on the reporting front it is more beneficial for us to export the Data to SQL and then perform reports in SQL, the result is the reports run quicker from SQL and most of the time a reporting tool can be used to create the reports - this can usually be performed by a power user as opposed to someone that has to have quite a high level of programming abilities to just build a report.
It would have been nice if it had already been in SQL in the first place.
But maybe some other NoSQL database has better functionality than UniData, that said too usually 3rd party support for NoSQL database engines comes at a higher premium because there are less specialists available than 3rd party support for SQL engines.
Is it correct in my understanding that we can build SSAS cubes sourcing from the transaction Systems? I meant the not the live but copy of the Live.
I'm trying to see if there is any scope to address few reporting needs without the need to build a traditional Data Warehouse and then build cubes on top of the data warehouse, instead build cubes to do Financial monthly aggregated reporting needs sourcing from backup copy of the Transaction systems.
Alternatively, if you have any better way to proceed please suggest.
Regards,
KK
You can create a set of views on top of you transactional system tables and then build your SSAS cubes ontop of those views. This would be less effort than creating a fully fledged datawarehouse.
I am a data warehouse developer (and therefore believe in cubes), but not every reporting solution warrants the cost of building a cube. If your short to medium term reporting requirements are fixed and you don't have users requiring data to be sliced differently each week, then a series of fixed reports may suffice.
You can create a series of SQL Server Reporting Services reports (or extract to Excel) either directly against your copied transactional data, or against a series of summarised tables that are created periodically. If you decide to utilise a series of pre-formatted reporting tables, try to create tables that cover multiple similar reports (rather than 1 monthly report table = 1 report) for ease of ongoing maintenance.
There are many other important aspects to this that you may need to consider first. Like how busy is the transaction system, what is the size of the data, concurrency and availability issues etc.
It is absolutely fine to have a copy of your live data and then build a report on the top of it. Bear in mind that the data you see in the report will not be the latest and there will be a latency factor depending on the frequency of your data pull.
Excuse me if the question is simple. We have multiple medical clinics running each running their own SQL database EHR.
Is there anyway I can interface each local SQL database with a cloud system?
I essentially want to use the current patient data that one is consulting with at that moment to generate a pathology request that links to a cloud ?google app engine database.
As a medical student / software developer this project of yours interests me greatly!
If you don't mind me asking, where are you based? I'm from the UK and unfortunately there's just no way a system like this would get off the ground as most data is locked in proprietary databases.
What you're talking about is fairly complex anyway, whatever country you're in I assume there would have to be a lot of checks / security around any cloud system that dealt with patient data. Theoretically though, what you would want to do ideally is create an online database (cloud, hosted, intranet etc), and scrap the local databases entirely.
You then have one 'pool' of data each clinic can pull information from (i.e. ALL records for patient #3563). They could then edit that data and/or insert new records and SAVE them, exporting them back to the main database.
If there is a need to keep certain information private to one clinic only this could still be achieved on one database in a number of ways, or you could retain parts of the local database and have them merge with the cloud data as they're requested by the clinic
This might be a bit outdated, but you guys should checkout https://www.firebase.com/. It would let you do what you want fairly easily. We just did this for a client in the exact same business your are.
Basically, Firebase lets you work with a Central Database on the Cloud, that is automatically synchronised with all its front-ends. It even handles losing the connection to the server automagically. It's the best solution I've found so far to keep several systems running against one only cloud database.
We used to have our own backend that would try its best to sync changes, but you need to be really careful with inter-system unique IDs for your tables (i.e. going to one of the branches and making a new user won't yield the same id that one that already exists in any other branch or the central database). It becomes cumbersome very quickly.
CakePHP can automatically generate this kind of Unique IDs pretty easily and automatically, but you still have to work on sync'ing all the local databases with the central repository.
The company i am working for is implementing Share-point with reporting servers that runs on an SQL back end. The information that we need lives on two different servers. The first server being the Manufacturing server that collects data from PLCs and inputs that information into a SQL database, the other server is our erp server which has data for payroll and hours worked on specific projects. The i have is to create a view on a separate database and then from there i can pull the information from both servers. I am having a little bit of trouble with the syntax for connecting the two servers to run the View. We are running ms SQL. If you need any more information or clarification please let me know.
Please read this about Linked Servers.
Alternatively you can make a Data Warehouse - which would be a reporting data base. You can feed this by either making procs with linked servers or use SSIS packages if they're not linked.
It all depends on a project size and complexity, but in many cases it is difficult to aggregate data from multiple sources with Views. The reason is that the source data structure is modeled for the source application and not optimized for reporting.
In that case, I would suggest going with an ETL process, where you would create a set of Extract, Transform and Load jobs to get data from multiple sources (databases) into a target database where data will be stored in the format optimized for reporting.
Ralph Kimball has many great books on the subject, for example:
1) The Data Warehouse ETL Toolkit
2) The Data Warehouse Toolkit
They are truly worth the read if you are dealing with data