Friends i am working on a web project that is a free classified ad site. Its just like http://www.olx.com. I am not getting the idea of how http://www.olx.com.database is designed. can anyone help me in this regard. i just need the "big picture" not minor details. Means how they have structured their database in relation to their product/service categories and how data passes to database etc.
you can take reference from the below schema used in open source classified engine osclass
source:http://doc.osclass.org/Database_model_and_diagram
Related
I have very little Database management experience, I took a single class when I was in Undergrad. I wanted to see other's inputs on the best way to setup the database.
I have developed a docker application(Webscraping, PostGIS database). The webscraper scrapes from multiple websites everyday. Then uploads to the database, it also checks for duplicates before uploading to the database.
However, I don't want the Reasearch Assistants to be able to change things on the original tables, since lot of the webscraper depends on the structure of the original tables. I gave them SELECT access, but I want them to be able to share their data on the Database as this is a collaborative project.
My original thoughts was to create a new and empty database with full permission. And only SELECT access to the webscraper database. I don't know if this is the best way to do this.
What are your thoughts?
Also to note, this is a contract job for a university project under a grant so I won't be maintaining the database after the contract. Also the project isn't big enough to hire a person with Docker & Database experience just to maintain the database. So I am trying to bulletproof this as much as possible.
For a web application (with some real private data) we want to use privacy enhancing technology to prevent big risks when someone gets permission to our database.
The application is build with different layers, and we use (as said in the topic title) Fluent NHibernate to connect to our database and we've created our own wrapper class to create query's.
Security is a big issue for the kind of application we're building. I'll try to explain the setting by a simple example:
Our customers got some clients in their application (each installation of the application uses its own database), for which some sensitive data is added, there is a client table, and a person table, that are linked.
The base table, which links to the other tables (there will be hundreds of them soon), probably containing sensitive data, is the client table
At this moment, the client has a cleint_id, and a table_id in the database, our customer only knows the client_id, the system links the data by the table_id, which is unknown to the user.
What we want to ensure:
A possible hacker who would have gained access to our database, should not be able to see the link between the customer and the other tables by just opening the database. So actually there should be some kind of "hidden link" between the customer and other tables. The personal data and all sensitive other tables should not be obviously linked together.
Because of the data sensitivity we're looking for a more robust solution then "statically hash the table_id and use this in other tables", because when one of the persons is linked to the corresponding client, not all other clients data is compromised too.
Ultimately, the customer table cannot be linked to the other tables at all, just by working inside the database, the application-code is needed to link the tables.
To accomplish this we've been looking into different methods, but because of the multiple linked tables to this client, and further development (thus probably even more tables) we're looking for a centralised solution. That's why we concluded this should be handled in the database connector. Searching on the internet and here on Stack Overflow, did not point us in the right direction, perhaps we couldn't find this because of wrong search terms (PET, Privacy enhancing technology, combined with NHibernate did not give us any directions.
How can we accomplish our goals in this specific situation, or where to search to help us fix this.
We have a similar requirement for our application and what we ended up with using database schema's.
We have one database and each customer has a separate schema, where all the data for that customer is stored. It is possible to link from the schema to the rest of the database, but not to different schema's.
Security can be set for each schema separately so you can make the life of a hacker harder.
That being said I can also imagine a solution where you let NHibernate encrypt every peace of data it will send to the database and decrypt everything it gets back. The data will be store savely, but it will be very difficult to query over data.
So there is probably not a single answer to this question, and you have to decide what is better: Not being able to query, or just making it more difficult for a hacker to get to the data.
I'm working on an .NET MVC SQL application that will contain sensitive data, for example- HIV test results or income. I want to error-proof this privacy as much as possible so no one except the user can access it (think Joe the Plumber having his information hacked by a state employee).
I read hear that splitting the database in two doesn't seem reasonable:
Is splitting databases a legitimate security measure?
although I've heard of this being done. If we could just use two tables... better.
But when I say error-proofing, I mean impossible for ANYONE in our company to access both databases/tables. I'm thinking about putting access to the application code (which would access both databases) and to both databases in the hands of a deep-pockets third party (like PWC or EY) for when the government came calling or some other real need to see both data sources came along.
Anyone have any thoughts on the cleanest way to do this? We'd want to design the tables such that most queries would not require access to both data sources so the relative cost in throughput wouldn't be that much.
You can encrypt a column of data in SQL. So the columns which has the sensitive data e.g. HIV test results/income, you can encrypt the data while storing it in the DB.
Check the details here:
http://msdn.microsoft.com/en-us/library/ms179331.aspx
http://msdn.microsoft.com/en-us/library/bb964742.aspx
Let me know if it helps.
I'm trying out Microsoft Master Data Services and I would like to add data to the database programmatically. I'm starting to get the model/entity/member structure but I'm not yet sure. If you have a nice explanation for this structure, please share.
Say somebody added a new employee in an ERP system and I would like to send that to the MDS. How would I do that? Is the data that I want to add a new member? Because if I look at the following information (http://technet.microsoft.com/en-us/library/hh230995), the only way to import data is through entities?
Thanks in advance for any useful information about this!
Lets start with the basics.
Entities in Master Data Services (MDS) are roughly analogous to tables in a regular database.
Every entity must live in a model.
A model can contain any number of entities.
The Metadata* methods you see on that page can be used to create, read and update models and entities. Once you have modeled your ERP tables as an MDS model, you can use the EntityMembersCreate API (with the relevant model/entity information) to create a member (roughly analogous to a row in a table). You can use EntityMembersUpdate to update members and EntityMembersDelete to delete them.
Another way to get large amounts of data into MDS is by using Entity Based Staging. Entity Based Staging allows you to use tools like SSIS to get bulk data into MDS. A good primer here: http://msdn.microsoft.com/en-us/sqlserver/hh802433.aspx.
I hope this helps. Feel free to ask more questions.
I like using a generic data-access-object that classes in my model inherit from. Each class has a one to one relationship with tables in the database.
We're using SSIS to replicate data from our CRM (as well as other data sources) into our MDS (for the time-being). If you're not familiar with the tool, I'd recommend in terms of moving data around - it's relatively easy to pickup the basics. If you go this route, here's a great resource I followed to push data into our MDS system:
http://www.sqlchick.com/entries/2013/2/16/importing-data-into-master-data-services-2012-part-2.html
We're just starting Sharepoint development, and one of my first tasks is to build a data collection tool. It will be used across multiple sites, so there will be an admin area, and each site will pull in it's related questions, and record the data. I've gone through a bunch of tutorials on development, and have a fairly good idea of how to start. I just want to make sure I understand one thing. Do lists basically take the place of your database? If this was a regular app, I would create a question table, a link table that tells which questions are connected to which site, a table that stores the answer, linking to the site and question table.
Is this the basic pattern you follow, or should I be doing things differently for Sharepoint applications?
If the thought is to use an external databse, can anyone point me to some info on this?
In our Sharepoint project we stared with Lists. It was good to some point - till DB had only few relations between data. After adding tables and relations performance falls a lot and we had to switch to use standard DB in MS SQL Server. So I recomend to use DB.
Disadventages: you cannot use sharepoint controls to edit/view data and cannot restrict access to data from sharepoint level
Adventages: much faster access to data