I am new to NOSQL world and still comparing between nosql and sql databases,
I Just tried making few samples using mongodb.
I am asking about stored procedures when we send few parameters to one stored procedure and this procedure execute number of other stored procedures in the database, will get data from stored procedures and send data to others.
In other words, will make the logic happen on the database side using sequence of functions and stored procedures.
Is that behavior or something the same already exist on NOSQL databases, or its completely different and i am thinking in the wrong way?
Mongo uses stored Javascript in a few places including Map/Reduce, db.eval and where clauses. Checkout this blog post for a survey:
Working With Stored JavaScript in MongoDB
The key to storing your functions on the server and making them available in these three contexts is db.system.js.save:
db.system.js.save( { _id : "foo" , value : function( x , y ){ return x + y; } } );
More details in the Mongo docs
Depends whether you define "NOSQL" as "No SQL" or "Not Only SQL."
OpenLink Virtuoso [1] (produced by my employer) is the latter, and has stored procedures, and SPARQL-in-SQL, and SQL-in-SPARQL, among other neat tricks; including extensibility via in-process runtime hosting (Perl, PHP, Python, Ruby, JSP, and others), external libraries or helper apps, and more.
Other NoSQL DBs have other ways of handling this, such as Mongo's extensibility via JavaScript, described in that other answer.
[1] http://virtuoso.openlinksw.com/
Related
I am surprised why this is so hard to find.
I am trying to convert my monolith to a 3 layer architecture for my node express app with a propriety sql.
Repository
BEFORE architecture change
1 file entry.js
1 endpoint with business logic
Functions with raw sql that are called after validation of res.body objects
AFTER
📁 backend
📁 src
📁 services
📄 service1
📁 routes
📄 service1Route [ Route handling /service1 from entry.js ]
📁 models
-service1Model [ Contains sql functions, not schema ]
📄 entry.js [ Main express app]
Tech used
- Using .js not .ts
- nodejs
- express
-
Constraints
- I cannot use postgres or sequlize as they don't support the propriety db that I am using.
Assumptions
postgres or mongoose have popular ORMs and ODMs without which you cannot use them hence developers are ( as a good practice ) forced to create models.
Now I want to create my own models / schema with such validations.
Ask
How do I create models without ORM or ODM.
Is there a difference between schema and model ?
Writing sql functions in model folder : is that the right way to use this architectural pattern.
If schema/model is created in 📁 models folder then where do the sql queries reside ?
What I have tried?
For validating objects with required keys from res.body object
I'll have a go at answering. I'm a database engineer and have written node.js apps that connect directly to the database without using an ORM.
1: So long as you know the structure of the data that you wish to store as it resides in the database you can write a javascript class that has methods to do all of your updating and class instantiation etc. For instance, you can have a user class with a constructor and a number of methods for creating a new user in the database, retrieving a user from the database, updating a user etc. You can read more about javascript classes here and here.
So here your Model is just a class that knows how to interact with the database.
You could store all of your SQL here, but I would advise against that in favour of using Stored Procedures in the database. That way if you ever need to tune your query, or make changes, you can change just the stored procedure without having to create a whole release of your application. Your DBAs can also tinker round with your SPs for you as well.
2: Depends what you're referring to here. A schema in the database is a container for functionality, business areas, or whole applications. Like within a business you could have a Sales schema and a Marketing schema, or you could store all of your application logic in the MySalesApp schema.
A javascript model is one particular piece of functionality and its interactions with the database. Like a user, or a person, or an animal etc. In the database all of these objects (there would probably be a person/user/animal table, with a number of stored procedures for updating, creating, etc) would be stored inside a schema. In MySQL a schema can also be a whole database.
3: You could store your SQL there. But for the reasons mentioned before I'd do this with Stored Procedures. Basically because you can tune them for efficiency without having to redeploy your whole application.
4: This also answered by using Stored Procedures. So instead of having a bunch of code that creates a user, you have an SP that lives in the database and handles everything for you. Then you just end up with a call to a stored procedure that looks like this:
var query = "BB.sp_CreateUser";
var params = [
parseInt(user.userId),
user.firstName,
user.surname,
user.userInitials,
user.telephone,
user.username,
user.password
];
let result = await database.asyncQueryDynamic(query, params);
I've got a bit of abstraction going on here because I'm using connection pools etc. But you can read more about using Stored Procedures here.
I am designing a multi-tenant database. Many actions that are needed on a per-tenant basis have been written as stored procedures. Almost all of them use dynamic SQL execution since I have not found a way to differentiate between the schemas other than with dynamic statements.
EXEC( 'SELECT * FROM ' + #SchemaName + '.Contacts' )
Is there any way to create a variable representing the schema so I can call the select statements without building them dynamically?
SELECT * FROM #TheSchema.Contacts
SQL Server 2008
FWIW I have found it much more useful to separate customers by database than by schema. Reasons?
All schemas can have identical procedures, thus derive them from model instead of having to create anything. Deployment to multiple databases is no more complex than deployment to multiple schemas, actually less so I'd argue, and the only difference is the application would have to know which database to reference instead of which schema.
Different customers can have different recovery models, be backed up on different schedules, etc. This can improve your backup/restore plan especially as the data set gets larger.
Having each customer in a separate database makes it very easy to move a tenant to a different server if they get too big for the current one. Extracting the data and objects for their schema only will be quite convoluted.
Some customers may legally and/or contractually require that you not store their data in the same database as other customers.
A simpler solution is to keep the stored procedure in each of the tenant databases and execute it inside them.
This question is a bit old, but I am looking into this option as a cost savings measure - since many cloud providers charge per database and the costs associated with storage are less - especially useful for multi-tenant small databases.
From what I understand, a default schema can be set for the login, thus, setting a schema per login would automatically use that schema for all queries and sprocs. Something to consider for those that comes across this question.
I'm developing an iOS application that's a manager/viewer for another project. The idea is the app will be able to process the data stored in a database into a number of visualizations-- the overall effect being similar to cacti. I'm making the visualizations fully user-configurable: the user defines what she wants to see and adds restrictions.
She might specify, for instance, to graph a metric over the last three weeks with user accounts that are currently active and aren't based in the United States.
My problem is that the only design I can think of is more or less passing direct SQL from the iOS app to the backend server to be executed against the database. I know it's bad practice and everything should be written in terms of stored procedures. But how else do I maintain enough flexiblity to keep fully user-defined queries?
While the application does compose the SQL, direct SQL is never visible or injectable by the user. That's all abstracted away in UIDateTimeChoosers, UIPickerViews, and the like.
Is all of the data in the database available to all of the users, or do you only permit each user to access a subset of the data? If the latter, simply restricting the database login to read-only access isn't enough to secure your data.
As a trivial example, a user could compromise your interface in order to submit the query SELECT password, salt FROM users WHERE login = 'admin', hijack the response to get at the raw data, and brute force your admin password. As the popularity of an app grows, the pool of malicious users grows more than linearly, until eventually their collective intelligence exceeds that of your team; you shouldn't put yourself in a situation where success will be your downfall.
You could take the SQL query sent by the client application and try to parse it server-side in order to apply appropriate restrictions on the query, to fence the user in, so to speak. But getting there would require you to write a mini SQL parser in your server code, and who wants to do all that work? It's much easier to write code that can write SQL than it is to write code that can read it.
My team solved a similar problem for a reporting interface in a rather complex web application, and our approach went something like this:
Since you already intend to use a graphical interface to build the query, it would be fairly easy to turn the raw data from the interface elements into a data structure that represents the user's input (and in turn, the query). For example, a user might specify, using your interface, the condition that they want the results to be confined to those collected on May 5, 2010 by everyone but John. (Suppose that John's UserID is 3.) Using a variant of the JSON format my team used, you would simply rip that data from the UI into something like:
{ "ConditionType": "AND",
"Clauses": [
{ "Operator": "Equals",
"Operands": [
{ "Column": "CollectedDate" },
{ "Value": "2010-05-05" }
]
},
{ "Operator": "NotEquals",
"Operands": [
{ "Column": "CollectedByUserID" },
{ "Value": 3 }
]
}
]
}
On the client side, creating this kind of data structure is pretty much isomorphic to the task of creating an SQL query, and is perhaps somewhat easier, since you don't have to worry about SQL syntax.
There are subtleties here that I'm glossing over. This only represents the WHERE part of the query, and would have to live in a larger object ({ "Select": ..., "From": ..., "Where": ..., "OrderBy": ... }). More complicated scenarios are possible, as well. For example, if you require the user to be able to specify multiple tables and how they JOIN together, you have to be more specific when specifying a column as a operand in a WHERE clause. But again, all of this is work you would have to do anyway to build the query directly.
The server would then deserialize this structure. (It's worth pointing out that the column names provided by the user shouldn't be taken dirty – we mapped them onto a list of allowed columns in our application; if the column wasn't on the list, deserialization failed and the user got an error message.) With a simple object structure to work with, making changes to the query is almost trivial. The server application can modify the list of WHERE clauses to apply appropriate data access restrictions. For example, you might say (in pseudo-code) Query.WhereClauses.Add(new WhereClause(Operator: 'Equals', Operands: { 'User.UserID', LoggedInUser.UserID } )).
The server code then passes the object into a relatively simple query builder that walks the object and splits back an SQL query string. This is easier than it sounds, but make sure that all of the user-provided parameters are passed in cleanly. Don't sanitize – use parameterized queries.
This approach ultimately worked out really nicely for us, for a few reasons:
It allowed us to break up the complexity of composing a query from a graphical interface.
It ensured that user-generated queries were never executed dirty.
It enabled us to add arbitrary clauses to queries for various kinds of access restrictions.
It was extensible enough that we were able to do nifty things like allowing users to search on custom fields.
On the surface, it may seem like a complex solution, but my team found that the benefits were many and the implementation was clean and maintainable.
EDIT: I have come to dislike my answer here. I agree with some of the commenters below, and I would like to recommend that you build "Query" objects on the client and pass those to a web service which constructs the SQL statement using prepared statements. This is safe from SQL injection because you are using prepared statements, and you can control the security of what is being constructed in the web service which you control.
End of Edit
There is nothing wrong with executing SQL passed from the client. Especially in query building situations.
For example, you can add as many where clauses by joining them with "AND". However, what you should not do is allow a user to specify what the SQL is. You should instead provide an interface that allows your users to build the queries. There are a couple reasons this is advantageous:
Better user experience (who wants to write SQL other than developers?)
Safer from injection. There is just no way you could possibly filter out all dangerous SQL strings.
Other than that, it's absolutely fine to execute dynamic SQL instead of using a stored procedure. Your view that everything should be written in terms of stored procedures seems misguided to me. Sure, stored procedures are nice in a lot of ways, but there are also many downsides to using them.
In fact, overuse of stored procs sometimes leads to performance problems since developers reuse the same stored procedure in multiple places even when they don't need all the data it returns.
One thing you might want to look into though is building the SQL on the server side and passing over some kind of internal representation of the built query. If you have some kind of web service which is exposed and allows your client to run whatever SQL it wants to run, then you have a security concern. This would also help in versioning. If you modify the database, you can modify the web service with it and not worry about people using old clients building invalid SQL.
I see this fully user-configurable visualizations more like building blocks.
I wouldn't pass direct sql queries to the back-end. I would make the user send parameters (wich view to use, filters in the where clause, so on). But letting the user inject sql it's a potential nightmare (both for security and maintenance)
If you want to let users send over actual sql, try filtering words like "drop and truncate." If you have to allow deletes, you can enforce that they use a primary key.
There is nothing wrong about an application sending SQL commands to a database, as long as you are aware of injection issues. So don't do this in you're code:
(Pseudocode)
String sqlCommand = "SELECT something FROM YOURTABLE WHERE A='" + aTextInputFieldInYourGui + "'";
cmd.execute(sqlCommand);
Why not? See what happens if the user enters this line into aTextInputFieldInYourGui
' GO DELETE * FROM YOURTABLE GO SELECT '
(assuming your DB is MS SQL Server here, for other RDBMS slightly different syntax)
Use prepared statements and Parameterbinding instead
(Pseudocode)
String sqlCommand = "SELECT something FROM YOURTABLE WHERE A=?";
cmd.prepare(sqlCommand);
cmd.bindParam(1, aTextInputFieldInYourGui);
cmd.execute();
Regards
Without going into specifics...I have a large SQL Server 2005 database with umpteen stored-procedures.
I have multiple applications from WinForm apps to WebServices all of which use this DB.
My simple objective now is to create a meta-database...a prospective data-dictionary where I can maintain details of which specific app. file uses which SP.
For example, My application Alpha which has a file Beta.aspx...uses 3 SPs which are physically configured for usage in BetaDAL.cs
You might have inferred by now,it will make life easier for me later when there is a migration or deprecation....where I can just query this DB SP-wise to get all Apps/Files that use the DB or vice-versa.
I can establish this as a single de-normalized table..or structure it in a better way.
Does some schema already exist for this purpose?
SQL Server supports what are called extended properties, basically a key-value dictionary attached to every object in the catalog. You can add whatever custom information about the catalog (comments on tables, columns, stored procedures, ...) you wish to store as extended properties and query them along with the normal catalog views.
Here's one overview (written for SQL Server 2005, but roughly the same techniques should apply for 2000 or 2008).
I have a 'reference' SQL Server 2005 database that is used as our global standard. We're all set up for keeping general table schema and data properly synchronized, but don't yet have a good solution for other objects like views, stored procedures, and user-defined functions.
I'm aware of products like Redgate's SQL Compare, but we don't really want to rely on (any further) 3rd-party tools right now.
Is there a way to ensure that a given stored procedure or view on the reference database, for example, is up to date on the target databases? Can this be scripted?
Edit for clarification: when I say 'scripted', I mean running a script that pushes out any changes to the target servers. Not running the same CREATE/ALTER script multiple times on multiple servers.
Any advice/experience on how to approach this would be much appreciated.
1) Keep all your views, triggers, functions, stored procedures, table schemas etc in Source Control and use that as the master.
2) Failing that, use your reference DB as the master and script out views and stored procedures etc: Right click DB, Tasks->Generate Scripts and choose your objects.
3) You could even use transactional replication between Reference and Target DBs.
I strongly believe the best way is to have everything scripted and placed in Source Control.
You can use the system tables to do this.
For example,
select * from sys.syscomments
The "text" column will give you all of the code for the store procedures (plus other data).
It is well worth looking at all the system tables and procedures. In fact, I suspect this is what RedGate's software and other tools do under the hood.
I have just myself begun experimenting with this, so I can't really be specific about all the gotcha's and what other system tables you need to query, but this should get you started.
Also see:
Query to list SQL Server stored procedures along with lines of code for each procedure
which is slightly different question than yours, but related.
I use (and love) the RedGate tools, but when Microsoft announced Visual Studio 2010, they decided to allow MSDN subscribers who get Visual Studio 2008 Team System to also get Visual Studio 2008 Database Edition (which has a schema compare tool).
So if you or your organization has an MSDN subscription, you might want to consider downloading and installing the Database Edition over your Team System to get all the features now.
More details at http://msdn.microsoft.com/en-us/vsts2008/products/cc990295.aspx
Take a look at ScriptDB on Codeplex (http://www.codeplex.com/ScriptDB)
It is a console C# app that creates scripts of the SQL Database objects using SMO. You can use that to compare scripts generated on two servers. Since its open source, adjust it if you need it.
Timur