I'm building this website which internally calls some APIs to interact with data in a server,
but not planning to make those APIs officially public.
Even in this case, should I make those RESTful?
There are lots of trade-offs. You say you're prototyping, but may need to implement "seriously".
Firstly, even for prototyping, you get a lot of benefit from sticking with a consistent API approach, ideally based on client and server libraries in your framework of choice. Common choices are "synchronous/asynchronous", "function-based/resource-based", "JSON/XML" etc. Mixing and matching those choices just makes everything much harder.
Some business domains are great for resource-based API structures. Order management systems, social networks, question-and-answer web sites all work well. Some are not so easy to represent as resources - real-time/IoT applications, chat/messaging systems, etc.
If you decide that "synchronous" and "resource based" are a good fit for your business domain, you may as well take advantage of the libraries that exist to build and consume RESTful APIs. You can decide for yourself how "pure" and "future-proof" you want to make those APIs. You may not care about versioning, for instance.
If "synchronous" and "resource-based" are not a good fit, I'd not try to shoe horn them into a RESTful API design.
The REST interface is designed to be efficient for large-grain hypermedia data transfer, optimizing for the common case of the Web, but resulting in an interface that is not optimal for other forms of architectural interaction. -- Fielding, 2000
REST is software design on the scale of decades: every detail is intended to promote software longevity and independent evolution. Many of the constraints are directly opposed to short-term efficiency. -- Fielding, 2008
That doesn’t mean that I think everyone should design their own systems according to the REST architectural style. REST is intended for long-lived network-based applications that span multiple organizations. If you don’t see a need for the constraints, then don’t use them. -- Fielding, 2008
For a private API, you probably control both the clients and the server, and its unlikely that you are going to be facing Web Scale[tm] traffic levels where you will need to offload work to caches.
Which means that you aren't likely to get full return on your investment.
I was recently tasked with developing automated build and release pipelines for one of my company's legacy applications. After some investigation, I keep hearing from managers and other devs that certain application layers and architectures don't lend themselves to automation, particularly automated testing. Therefore, it's often suggested I shouldn't bother trying to apply DevOps principles and AT unless I want to re-architect the whole app.
The common cited example would be PL/SQL backends or monolithic architectures. I asked why these were not suitable, but never got a really clear answer. Does anyone have any insight on when automated test should not be used in favor of dumping the old architecture and starting fresh?
Short answer - ones that suffer from testability issues.
For a more in depth one, let's first admit that many software systems are untestable, or not immediately testable. So that, the effort of
trying to apply DevOps principles and AT
is far greater than the ROI. Such notorious example is Google's ReCAPTCHA, which causes some pain for the automation testing folks (like me). The devs are actually right to say that it will take be a
re-architect the whole app
journey, as testabilty is highly related to other key software qualities such as encapsulation, coupling, cohesion, and redundancy.
common cited example would be PL/SQL backends or monolithic architectures
Now, that is totally not the case. The firt one is more data-centric and requires a deeper understanding, but there are solutions to that as well. As to, single-tiered software applications - one can argue that in contrast to the mSOA, monolithic applications are much easier to debug and test. Since a monolithic app is a single indivisible unit, you can run end-to-end testing much faster/easier.
Put simply - if your app is highly testable, is highly usable. In case, the architecture and design were aligned to a very, very specific company needs - no wonder, is usable only up to a point.
can someone please explain the difference between system design and object oriented design?
Object oriented design involves object modeling and uses object oriented concepts such as Abstraction, Encapsulation, Decomposition and Generalization. Both of the design involves Architectural design and conceptual design.
Is one of the design paradigm subset of other?
System design is the designing the software/application as a whole [high level] that may include analysis, modelling, architecture, Components, Infrastructure etc. whereas the objected-oriented design is the set of defined rules/concepts to implement the functionalities within a software.
Take an analogy, a football game.
So the System design involves the design of football ground, goal poles, grass on the ground, location of the ground, length/breadth of the ground, putting line marks on the ground, scoreboard, defining the playing teams, number of players to play etc.
Now, take object-oriented design: how the games need to be played is defined by a set of rules that need to be followed. Thus the players need to play the game within the defined rules. So the more the player knows those rules the better they can play the game without making fouls. Similarly, the rules for the object-oriented concepts are inheritance, composition, abstraction, encapsulation. Thus the better we know these concepts the better design we can make.
#Prashant, hope it explains a bit
Let's say you want to design an online shopping web Application (like amazon.com). So before making that web application you have to think of :
load balancing to handle the user requests,
database whether to use SQL or NOSQL,
whether to use cache like Redis or not,
making microservices for the different tasks ( like notification service, order processing service etc.)
monitoring your web application (like memory, cpu etc.),
logging (say in sumoLogic).
CI/CD
So all these will fall under system design as they are required for any system to work.
Now once you finalized all these things and went to implement (writing code) any part of the system then you can follow the Object-oriented design to make classes, interfaces etc.
For example, you started implementing the notification microservice ( say in Java), so now you have to design classes, interfaces etc., and for this, you can use the object-oriented design principles like SOLID.
So we can say that object-oriented design is a part of system design which comes into picture when we start working on the actual implementation for any part of the system.
System in System Theory can be everything. (Facets of Systems Science, George Klir, 1991) System is on TOP of TREE, Superclass of other categories.
In overall, System Development included 2 major phases:
1- System Analysis: including Planning, requirements, analysis and etc. These items related to specific methodology in system theory.
2- System Design: including design, implementation, test, deploy, maintenance and etc.
As I said, System can be everything. For example:
Mechanical systems, Psychological systems, Social systems, Aircraft Systems and so on. Each category may have detailed and specific analysis and design steps based on mentioned 2 major phases.
In computer world, Software Systems is one of System categories.
Each software Analysis and Design method is based on mentioned 2 major phases too.
Additionally, In Software Systems, we have some paradigms to analysis and design like:
Structural/Process Centered
Data Centered
Object Oriented
Service Oriented
and etc.
Each of them has it's own Analysis and Design Steps. These steps are based on 2 major phases as well. But in details, they have some differences.
To sum up, Systems Analysis and Design is a big picture to all other type of systems. Learning System Analysis and Design helps to understand all other systems analysis and design and specially helps to compare and evaluate them.
Objects are parts of a system. You can think Object Oriented Design is a part of System Design. You can design a system in high level (with overview) or in low level (with details). It will highlight
Infrastructure
Data flow management
Services
Cache management
Request/response management etc
When doing object oriented design you can think that you have to use some objects and making relations among them. But before making object you need to design some classes. So actually OOD (object oriented design) is
Designing class diagram (optional)
Making some classes
Making relations among classes
Using classes for creating objects
Implementing the purpose (basic functions and logics) etc
I just saw the lecture:
Breaking apart conceptions
He keeps saying that the 3 layers paradigm started because of the problem with connection pooling.
And NOT because of architectural considerations.
Seems like a mind blowing theory.
Can anybody proove or disproove this claim?
This is kind of a hard question to properly answer. The relevant piece of the presentation states that because of the databases not being designed for dozens of connections, there had to be a business layer introduced between the applications and the stored procedures in the database.
I beg to differ. The idea of separating business logic, presentation and storage of state originates much earlier, namely in the 1970 or 1980, and according to Wikipedia was developed at Xerox PARC.
Back then, of course, servers were way more expensive than they are today. However, that still didn't keep them from thinking about maintainability and other aspects of software development, namely testing, debugging and assigning to different teams parts of the development of a full product. It makes a lot of sense to have tightly defined interfaces within an integrated application not for the sake of connection pooling (which is just working around limitations of an ingredient in the entire mixture), but more importantly for keeping the application maintainable and simplifying the development process.
I like the talk though. From practice, I know that blindly subscribing to design patterns can lead you into undesirable situations. You can always overdo everything.
In short though, the multi-tier model (be it on individual servers or just as a development paradigm) is older than the rise of the Internet. It has been kinda commonplace since at least the 1980.
From what I understand, OOP is the most commonly used paradigm for large scale projects. I also know that some smaller subsets of big systems use other paradigms (e.g. SQL, which is declarative), and I also realize that at lower levels of computing OOP isn't really feasible. But it seems to me that usually the pieces of higher level solutions are almost always put together in a OOP fashion.
Are there any scenarios where a truly non-OOP paradigm is actually a better choice for a largescale solution? Or is that unheard of these days?
I've wondered this ever since I've started studying CS; it's easy to get the feeling that OOP is some nirvana of programming that will never be surpassed.
In my opinion, the reason OOP is used so widely isn't so much that it's the right tool for the job. I think it's more that a solution can be described to the customer in a way that they understand.
A CAR is a VEHICLE that has an ENGINE. That's programming and real world all in one!
It's hard to comprehend anything that can fit the programming and real world quite so elegantly.
Linux is a large-scale project that's very much not OOP. And it wouldn't have a lot to gain from it either.
I think OOP has a good ring to it, because it has associated itself with good programming practices like encapsulation, data hiding, code reuse, modularity et.c. But these virtues are by no means unique to OOP.
You might have a look at Erlang, written by Joe Armstrong.
Wikipedia:
"Erlang is a general-purpose
concurrent programming language and
runtime system. The sequential subset
of Erlang is a functional language,
with strict evaluation, single
assignment, and dynamic typing."
Joe Armstrong:
“Because the problem with
object-oriented languages is they’ve
got all this implicit environment that
they carry around with them. You
wanted a banana but what you got was a
gorilla holding the banana and the
entire jungle.”
The promise of OOP was code reuse and easier maintenance. I am not sure it delivered. I see things such as dot net as being much the same as the C libraries we used to get fro various vendors. You can call that code reuse if you want. As for maintenance bad code is bad code. OOP did not help.
I'm the biggest fan of OOP, and I practice OOP every day.
It's the most natural way to write code, because it resembles the real life.
Though, I realize that the OOP's virtualization might cause performance issues.
Of course that depends on your design, the language and the platform you chose (systems written in Garbage collection based languages such as Java or C# might perform worse than systems which were written in C++ for example).
I guess in Real-time systems, procedural programming may be more appropriate.
Note that not all projects that claim to be OOP are in fact OOP. Sometimes the majority of the code is procedural, or the data model is anemic, and so on...
Zyx, you wrote, "Most of the systems use relational databases ..."
I'm afraid there's no such thing. The relational model will be 40 years old next year and has still never been implemented. I think you mean, "SQL databases." You should read anything by Fabian Pascal to understand the difference between a relational dbms and an SQL dbms.
" ... the relational model is usually chosen due to its popularity,"
True, it's popular.
" ... availability of tools,"
Alas without the main tool necessary: an implementation of the relational model.
" support,"
Yup, the relational model has fine support, I'm sure, but it's entirely unsupported by a dbms implementation.
" and the fact that the relational model is in fact a mathematical concept,"
Yes, it's a mathematical concept, but, not being implemented, it's largely restricted to the ivory towers. String theory is also a mathematical concept but I wouldn't implement a system with it.
In fact, despite it's being a methematical concept, it is certainly not a science (as in computer science) because it lacks the first requirement of any science: that it is falsifiable: there's no implementation of a relational dbms against which we can check its claims.
It's pure snake oil.
" ... contrary to OOP."
And contrary to OOP, the relational model has never been implemented.
Buy a book on SQL and get productive.
Leave the relational model to unproductive theorists.
See this and this. Apparently you can use C# with five different programming paradigms, C++ with three, etc.
Software construction is not akin to Fundamental Physics. Physics strive to describe reality using paradigms which may be challenged by new experimental data and/or theories. Physics is a science which searches for a "truth", in a way that Software construction doesn't.
Software construction is a business. You need to be productive, i.e. to achieve some goals for which someone will pay money. Paradigms are used because they are useful to produce software effectively. You don't need everyone to agree. If I do OOP and it's working well for me, I don't care if a "new" paradigm would potentially be 20% more useful to me if I had the time and money to learn it and later rethink the whole software structure I'm working in and redesign it from scratch.
Also, you may be using another paradigm and I'll still be happy, in the same way that I can make money running a Japanese food restaurant and you can make money with a Mexican food restaurant next door. I don't need to discuss with you whether Japanese food is better than Mexican food.
I doubt OOP is going away any time soon, it just fits our problems and mental models far too well.
What we're starting to see though is multi-paradigm approaches, with declarative and functional ideas being incorporated into object oriented designs. Most of the newer JVM languages are a good example of this (JavaFX, Scala, Clojure, etc.) as well as LINQ and F# on the .net platform.
It's important to note that I'm not talking about replacing OO here, but about complementing it.
JavaFX has shown that a declarative
solution goes beyond SQL and XSLT,
and can also be used for binding
properties and events between visual
components in a GUI
For fault tolerant and highly
concurrent systems, functional
programming is a very good fit,
as demonstrated by the Ericsson
AXD301 (programmed using Erlang)
So... as concurrency becomes more important and FP becomes more popular, I imagine that languages not supporting this paradigm will suffer. This includes many that are currently popular such as C++, Java and Ruby, though JavaScript should cope very nicely.
Using OOP makes the code easier to manage (as in modify/update/add new features) and understand. This is especially true with bigger projects. Because modules/objects encapsulate their data and operations on that data it is easier to comprehend the functionality and the big picture.
The benefit of OOP is that it is easier to discuss (with other developers/management/customer) a LogManager or OrderManager, each of which encompass specific functionality, then describing 'a group of methods that dump the data in file' and 'the methods that keep track of order details'.
So I guess OOP is helpful especially with big projects but there are always new concepts turning up so keep on lookout for new stuff in the future, evaluate and keep what is useful.
People like to think of various things as "objects" and classify them, so no doubt that OOP is so popular. However, there are some areas where OOP has not gained a bigger popularity. Most of the systems use relational databases rather than objective. Even if the second ones hold some notable records and are better for some types of tasks, the relational model is unsually chosen due to its popularity, availability of tools, support and the fact that the relational model is in fact a mathematical concept, contrary to OOP.
Another area where I have never seen OOP is the software building process. All the configuration and make scripts are procedural, partially because of the lack of the support for OOP in shell languages, partially because OOP is too complex for such tasks.
Slightly controversial opinion from me but I don't find OOP, at least of a kind that is popularly applied now, to be that helpful in producing the largest scale software in my particular domain (VFX, which is somewhat similar in scene organization and application state as games). I find it very useful on a medium to smaller scale. I have to be a bit careful here since I've invited some mobs in the past, but I should qualify that this is in my narrow experience in my particular type of domain.
The difficulty I've often found is that if you have all these small concrete objects encapsulating data, they now want to all talk to each other. The interactions between them can get extremely complex, like so (except much, much more complex in a real application spanning thousands of objects):
And this is not a dependency graph directly related to coupling so much as an "interaction graph". There could be abstractions to decouple these concrete objects from each other. Foo might not talk to Bar directly. It might instead talk to it through IBar or something of this sort. This graph would still connect Foo to Bar since, albeit being decoupled, they still talk to each other.
And all this communication between small and medium-sized objects which make up their own little ecosystem, if applied to the entire scale of a large codebase in my domain, can become extremely difficult to maintain. And it becomes so difficult to maintain because it's hard to reason about what happens with all these interactions between objects with respect to things like side effects.
Instead what I've found useful is to organize the overall codebase into completely independent, hefty subsystems that access a central "database". Each subsystem then inputs and outputs data. Some other subsystems might access the same data, but without any one system directly talking to each other.
... or this:
... and each individual system no longer attempts to encapsulate state. It doesn't try to become its own ecosystem. It instead reads and writes data in the central database.
Of course in the implementation of each subsystem, they might use a number of objects to help implement them. And that's where I find OOP very useful is in the implementation of these subsystems. But each of these subsystems constitutes a relatively medium to small-scale project, not too large, and it's at that medium to smaller scale that I find OOP very useful.
"Assembly-Line Programming" With Minimum Knowledge
This allows each subsystem to just focus on doing its thing with almost no knowledge of what's going on in the outside world. A developer focusing on physics can just sit down with the physics subsystem and know little about how the software works except that there's a central database from which he can retrieve things like motion components (just data) and transform them by applying physics to that data. And that makes his job very simple and makes it so he can do what he does best with the minimum knowledge of how everything else works. Input central data and output central data: that's all each subsystem has to do correctly for everything else to work. It's the closest thing I've found in my field to "assembly line programming" where each developer can do his thing with minimum knowledge about how the overall system works.
Testing is still also quite simple because of the narrow focus of each subsystem. We're no longer mocking concrete objects with dependency injection so much as generating a minimum amount of data relevant to a particular system and testing whether the particular system provides the correct output for a given input. With so few systems to test (just dozens can make up a complex software), it also reduces the number of tests required substantially.
Breaking Encapsulation
The system then turns into a rather flat pipeline transforming central application state through independent subsystems that are practically oblivious to each other's existence. One might sometimes push a central event to the database which another system processes, but that other system is still oblivious about where that event came from. I've found this is the key to tackling complexity at least in my domain, and it is effectively through an entity-component system.
Yet it resembles something closer to procedural or functional programming at the broad scale to decouple all these subsystems and let them work with minimal knowledge of the outside world since we're breaking encapsulation in order to achieve this and avoid requiring the systems to talk to each other. When you zoom in, then you might find your share of objects being used to implement any one of these subsystems, but at the broadest scale, the systems resembles something other than OOP.
Global Data
I have to admit that I was very hesitant about applying ECS at first to an architectural design in my domain since, first, it hadn't been done before to my knowledge in popular commercial competitors (3DS Max, SoftImage, etc), and second, it looks like a whole bunch of globally-accessible data.
I've found, however, that this is not a big problem. We can still very effectively maintain invariants, perhaps even better than before. The reason is due to the way the ECS organizes everything into systems and components. You can rest assured that an audio system won't try to mutate a motion component, e.g., not even under the hackiest of situations. Even with a poorly-coordinated team, it's very improbable that the ECS will degrade into something where you can no longer reason about which systems access which component, since it's rather obvious on paper and there are virtually no reasons whatsoever for a certain system to access an inappropriate component.
To the contrary it often removed many of the former temptations for hacky things with the data wide open since a lot of the hacky things done in our former codebase under loose coordination and crunch time was done in hasty attempts to x-ray abstractions and try to access the internals of the ecosystems of objects. The abstractions started to become leaky as a result of people, in a hurry, trying to just get and do things with the data they wanted to access. They were basically jumping through hoops trying to just access data which lead to interface designs degrading quickly.
There is something vaguely resembling encapsulation still just due to the way the system is organized since there's often only one system modifying a particular type of components (two in some exceptional cases). But they don't own that data, they don't provide functions to retrieve that data. The systems don't talk to each other. They all operate through the central ECS database (which is the only dependency that has to be injected into all these systems).
Flexibility and Extensibility
This is already widely-discussed in external resources about entity-component systems but they are extremely flexible at adapting to radically new design ideas
in hindsight, even concept-breaking ones like a suggestion for a creature which is a mammal, insect, and plant that sprouts leaves under sunlight all at once.
One of the reasons is because there are no central abstractions to break. You introduce some new components if you need more data for this or just create an entity which strings together the components required for a plant, mammal, and insect. The systems designed to process insect, mammal, and plant components then automatically pick it up and you might get the behavior you want without changing anything besides adding a line of code to instantiate an entity with a new combo of components. When you need whole new functionality, you just add a new system or modify an existing one.
What I haven't found discussed so much elsewhere is how much this eases maintenance even in scenarios when there are no concept-breaking design changes that we failed to anticipate. Even ignoring the flexibility of the ECS, it can really simplify things when your codebase reaches a certain scale.
Turning Objects Into Data
In a previous OOP-heavy codebase where I saw the difficulty of maintaining a codebase closer to the first graph above, the amount of code required exploded because the analogical Car in this diagram:
... had to be built as a completely separate subtype (class) implementing multiple interfaces. So we had an explosive number of objects in the system: a separate object for point lights from directional lights, a separate object for a fish eye camera from another, etc. We had thousands of objects implementing a few dozen abstract interfaces in endless combinations.
When I compared it to ECS, that required only hundreds and we were able to do the exact same things before using a small fraction of the code, because that turned the analogical Car entity into something that no longer requires its class. It turns into a simple collection of component data as a generalized instance of just one Entity type.
OOP Alternatives
So there are cases like this where OOP applied in excess at the broadest level of the design can start to really degrade maintainability. At the broadest birds-eye view of your system, it can help to flatten it and not try to model it so "deep" with objects interacting with objects interacting with objects, however abstractly.
Comparing the two systems I worked on in the past and now, the new one has more features but takes hundreds of thousands of LOC. The former required over 20 million LOC. Of course it's not the fairest comparison since the former one had a huge legacy, but if you take a slice of the two systems which are functionally quite equal without the legacy baggage (at least about as close to equal as we might get), the ECS takes a small fraction of the code to do the same thing, and partly because it dramatically reduces the number of classes there are in the system by turning them into collections (entities) of raw data (components) with hefty systems to process them instead of a boatload of small/medium objects.
Are there any scenarios where a truly non-OOP paradigm is actually a
better choice for a largescale solution? Or is that unheard of these
days?
It's far from unheard of. The system I'm describing above, for example, is widely used in games. It's quite rare in my field (most of the architectures in my field are COM-like with pure interfaces, and that's the type of architecture I worked on in the past), but I've found that peering over at what gamers are doing when designing an architecture made a world of difference in being able to create something that still remains very comprehensible at it grows and grows.
That said, some people consider ECS to be a type of object-oriented programming on its own. If so, it doesn't resemble OOP of a kind most of us would think of, since data (components and entities to compose them) and functionality (systems) are separated. It requires abandoning encapsulation at the broad system level which is often considered one of the most fundamental aspects of OOP.
High-Level Coding
But it seems to me that usually the pieces of higher level solutions
are almost always put together in a OOP fashion.
If you can piece together an application with very high-level code, then it tends to be rather small or medium in scale as far as the code your team has to maintain and can probably be assembled very effectively using OOP.
In my field in VFX, we often have to do things that are relatively low-level like raytracing, image processing, mesh processing, fluid dynamics, etc, and can't just piece these together from third party products since we're actually competing more in terms of what we can do at the low-level (users get more excited about cutting-edge, competitive production rendering improvements than, say, a nicer GUI). So there can be lots and lots of code ranging from very low-level shuffling of bits and bytes to very high-level code that scripters write through embedded scripting languages.
Interweb of Communication
But there comes a point with a large enough scale with any type of application, high-level or low-level or a combo, that revolves around a very complex central application state where I've found it no longer useful to try to encapsulate everything into objects. Doing so tends to multiply complexity and the difficulty to reason about what goes on due to the multiplied amount of interaction that goes on between everything. It no longer becomes so easy to reason about thousands of ecosystems talking to each other if there isn't a breaking point at a large enough scale where we stop modeling each thing as encapsulated ecosystems that have to talk to each other. Even if each one is individually simple, everything taken in as a whole can start to more than overwhelm the mind, and we often have to take a whole lot of that in to make changes and add new features and debug things and so forth if you try to revolve the design of an entire large-scale system solely around OOP principles. It can help to break free of encapsulation at some scale for at least some domains.
At that point it's not necessarily so useful anymore to, say, have a physics system encapsulate its own data (otherwise many things could want to talk to it and retrieve that data as well as initialize it with the appropriate input data), and that's where I found this alternative through ECS so helpful, since it turns the analogical physics system, and all such hefty systems, into a "central database transformer" or a "central database reader which outputs something new" which can now be oblivious about each other. Each system then starts to resemble more like a process in a flat pipeline than an object which forms a node in a very complex graph of communication.