I know that executable files (eg: .exe for Windows) are binaries. I know about hex files and assembly and so on. I also know about OS API's. Theoretically, I could write a web browser entirely in assembly code that uses the OS API's, assemble it with NASM and get an executable. But my question is, how do operating systems control the applications? For example, I could have a executable on Windows that writes to the video memory, and fills the screen with random stuff. I've tried this (actually) and Windows halts the application. How does it control the application? Moreover, if I have a linux executable and I attempt to run it on a Windows machine with the exact same hardware, theoretically, it should work (though it won't use any Windows API's) since the processor architecture is the same, but yet It cant. How exactly does the OS 'control' the binaries?
The operating system, in particular windows and linux protect other applications and the operating system itself from applications. So there is a protection layer that an application ideally cannot punch through running as a user.
You want to put pixels on the screen, you have to ask the operating system to do it for you. If the operating system allows a way to punch through (calling mmap perhaps having to run as root) then you can trash the computer at will, yes. The plan/design is to not let you have direct access, you are given a virtual space, a sandbox, for your application to play in, you have a virtual address space with some ram you can read and write at will, but you go outside that, you need to be shut down or at least handled in some way.
As far as what language you write applications in yes, if it is a normal windows program or linux program, then anything you can do to generate machine code for that target, asm, ada, C, etc. If you are talking specific virtual machines (java, python) then your choices get more limited. If you want to do more than play in your space, like have some sort of output other than a return value, then you have to make system calls in the way that the system requires them, which is target and operating system specific. And again, you get the right instructions or registers or memory structure or whatever is required by the operating system for that call in whatever language you are using or libraries you can link with then the rest of your program can be in whatever language, yes absolutely you want to write a web browser in assembly language using nasm, completely doable. have to create a binary format the os supports, and make the system calls as needed in the way that the system requires.
Related
JVM and CLR are virtual machines. Similarly to bare metal computer machines, they provide virtual machine languages.
On real computer machines, we have operating systems, which provide system calls and APIs. For example the famous book Advanced Programming Unix Environment describes the APIs provided by Unix and Linux. Windows, however, provide different APIs.
On top of virtual machines like JVM and CLR, is there something which
plays the role of an operating system, and
provides programming APIs?
If there is nothing playing the role of an OS on virtual machines like JVM and CLR, what provides programming APIs (such as those in Java, C#, ...) similar to OS APIs?
Note: I am asking about VMs and on top of them, instead of what is underlying them. Do VMs not run some virtual OS on top of them? If there is no, why is there no such a need?
Thanks.
There are two kinds of virtual machines: system virtual machines and process virtual machines. System virtual machines provide a virtualization of complete instructions sets including user-mode instructions and kernel-mode instructions and therefore they can run operating systems. Process virtual machines virtualizes user-mode instructions and, usually, some system calls (such as those for managing threads, memory, and files) and therefore can only run applications or processes. That is, on top of a single process virtual machine a single app or process can run. The JVM and CLR are process virtual machines.
While in theory it is indeed possible to develop an OS to run on a process virtual machine, this is practically not useful because the performance of the programs that will run on that OS will be terrible due to the excessive layering in software.
Generally, system and process virtual machines themselves are not considered to be operating systems. However, process virtual machines do not necessarily require an OS to run on and may run on a bare-metal computer. The .NET Micro Framework is an example of such VM. Such VMs are sometimes called operating systems. Some virtual ISAs or a subset thereof have been implemented completely in hardware similar to x86 and ARM. One could develop operating systems to run on them. They are almost never used in industry because of their low performance.
An "operating system" is a a large, fuzzy ball of hairs. You got the hal, kernel, userspace... do we count some userspace libraries (typically libc) too?
With squinting you can find some comparable concepts in the JVM/JRE but generally a JVM runs on top of a bona-fide operating system and thus does not reimplement all aspects and instead simply provides platform-independent abstractions over facilities that you can find on almost all systems.
For example these days Thread usually is just java representation of native OS threads, but a JVM could choose to implement thread scheduling in userspace, and in sun's JVM did back during 1.1 times and some other JVMs still do today.
I'll answer this from the perspective of Java since I have no in-depth knowledge of .Net. I would assume, however that the CLR and JVM are similar from this point of view.
Let's start with an operating system. The purpose of this is to abstract away the hardware specific interfaces as well as providing a runtime environment for processes.
The OS uses device drivers to provide a uniform interface to similar devices (like the processor, memory, disk drives, network cards and so on). The OS uses system calls to allow user-level code to interact with these devices. If you wrote code in C you will call 'open' then 'read' and 'write' to the device before calling 'close'. The 'ioctl' (IO control) system call is also used a lot for device control. Each OS provides a standard set of these system calls (you can run Linux on an Intel or ARM processor, but you have the same set of system calls for each distro). Incidentally, this is also how Docker works by using a standard set of system calls to enable containers to be moved from one platform to another without problem.
The OS also provides the ability to run multiple processes simultaneously. With newer, multi-core machines this really can happen in parallel but the OS also uses scheduling to share a CPU between multiple processes or threads. By switching processes or threads very quickly this gives the impression that things are happening simultaneously, even on a single processor.
Now let's look at the JVM, which is a user-level process (from the OS point of view so just like any other user application). This has been designed to abstract away CPU and operating specific functionality from the Java application. The bytecodes generated by the Java compiler do not contain any system calls. If you look at the bytecode instruction set (defined in the Java Virtual Machine Specification) you will find that the instructions provide many familiar low-level features such as loading a register, bit manipulation and so on. In addition, there are many instructions that are higher level and relate more specifically to Java; things like invokestatic that invokes a static method on a class, monitorenter. monitorexit for locking, newarray and so on.
The JVM takes these bytecodes and converts them from a CPU- and OS-independent form (that of a Virtual Machine) into the instructions for the specific CPU architecture and OS that the JVM is running on. In some cases this can be a one-to-one mapping (for things like bitwise operators), but can often be much more complex and involve the use of system calls to open files, access network interfaces, etc. The JVM also uses the OS to deal with threads created by the application. In the very early days of Java operating systems like Windows 95 did not have the concept of threads within a process so the JVM had to provide it's own implementation (this was called green threads and performed pretty badly).
To summarise the JVM takes the platform neutral bytcodes of the class files it is executing and converts them to the apprporiate native CPU instructions and system calls to make the application run. The JVM does not provide any traditional OS services, it just uses them.
I'm interested in PC firmware programming, and am just studying the UEFI spec. To my surprise, it seems like a spec for an entire OS which is embedded in firmware. You can even write UEFI "applications", which run directly using the UEFI boot services, without any other OS present.
I've found blog posts which show how to create a "Hello, world!" application which can run in the UEFI preboot environment. This is... interesting, and bizarre at the same time. I'll run my "Hello, world" programs on a regular OS, thank you.
What kind of use cases are UEFI applications actually good for? Fancy boot configuration screens? Does any "real", commercially available PC firmware use UEFI applications to implement anything more than just boot loaders and boot configuration utilities?
Anything that isn't PEI/DXE/SMM core or driver is an application, so any "real" PC have them, because BIOS Setup is actually an UEFI application. Some vendors include various other apps like firmware updaters, diagnostic and troubleshooting utilities, etc. UEFI 2.4 makes possible to add your own application with a properly filled BootXXXX/KeyXXXX variable pair and then run it by pressing a key combination during POST.
Most console applications written in C can be compiled as UEFI application by using StdLib package of current EFI Development Kit and then run in UEFI shell.
Major examples of useful UEFI apps (besides bootloaders, shell and Linux kernel, of course) are Intel ME System Tools, Read Universal, Python 2.7 and many more.
Eventually, when legacy boot will not be available anymore, all currently useful DOS utilities must either be made UEFI applications or go extinct.
Despite many valuable answers here, because I wrote couple UEFI applications myself I will try to add my 2 cents. First, what is UEFI application to just give ground what we talking about:
UEFI Specificatin v2.5:
Section 2.1.1
The major differences between image types are the memory type that the firmware
will load the image into, and the action taken when the image’s entry point exits or returns. An
application image is always unloaded when control is returned from the image’s entry point.
Section 2.1.2
When the application returns from
its entry point, or when it calls the Boot Service EFI_BOOT_SERVICES.Exit(), the application
is unloaded from memory and control is returned to the UEFI component that loaded the application.
Groups of applications that make sense in UEFI:
Configuration tools - Configuration interface for Option ROMs (ie. for storage controllers), out of band management (ie. AMT configuration tools), manufacturer performance tweaking tools
Provisioning tools - used by administrators to preload specific BIOS setting, manually setting all options in BIOS setup would be inefficient
Diagnostics tools - mostly for tests that cannot be performed in OS (DRAM tests, full storage scan, storage R/W tests, etc.). In some districts specific diagnostics tools are required in UEFI BIOS, so those can be sold to government.
Security applications - HDD encryption/decryption, antivirus scanner and anti thief applications
BIOS capability enhancement - Power Over Ethernet extensions, DRAM discovery, patching and modification of system tables (SMBIOS, ACPI)
Display tools - for displaying complex animations while running, splash screen displaying
Bootloaders - this is special type of application, which can call EFI_BOOT_SERVICES.ExitBootServices() causing termination of all memory management and passing control to Operating System.
Note that very important feature of UEFI application is that it can be added to boot order and be executed each boot time. Also UEFI application do not have to be delivered with BIOS image it can be stored in connected device memory, which is common for Option ROM configuration tools.
Here it is an example of a full blown UEFI Pre-boot Application;
There are SED SSD/HDD drives. As soon as SSD/HDD loses its power it goes into locked state (hardware-based encryption) There is no way you can get access to drive's data and all partitions on the drive are no longer even visible. Only small read-only partition (ShadowMBR) is available. UEFI firmware boots an UEFI application from that only available partition (UEFI app is written on that partition during the initialization process and when the ownership of an SED is taken). It securely authenticate user and if credentials are valid it unlocks the drive. When the drive is unlocked Shadow MBR disappears and all partitions on the drive becomes available. Then the App chain-boots the installed OS.
So if you don't have credentials you cannot even boot the OS and you cannot access the data on the drive by any means.
Here's a couple of examples:
https://github.com/NikolajSchlej/CrScreenshotDxe
UEFI DXE driver to take screenshots from GOP-compatible graphic console (yes, you can make PNG screenshots of your BIOS and save them)
http://ruexe.blogspot.com/
RU.EFI is quite an advanced tool for debugging the BIOS
Well, there are the OS loaders - both the more heavyweight ones (Windows, GRUB, BSD Loader) and the "present a menu" ones (rEFInd, Gummiboot). Shim, which enables UEFI Secure Boot for Linux platforms, consists of an application as well as installing a protocol for use by other applications.
Then you have things like the Linux kernel, which when compiled with CONFIG_EFI_STUB becomes a valid UEFI application, with the awareness of booting itself.
And firmware updates can also be shipped as UEFI applications.
The UEFI shell itself is an application.
Then there are things like factory production testing utilities, development diagnosis tools, ...
Windows 7 - 8 have UEFI installer. I'm not fully aware of the details, but I'm pretty sure this new environment gives a lot more flexibility to the developers than traditional boot environment on DVD.
Some motherboards have "instant on" features that allows you to get to a desktop screen within a few seconds. This is usually a stripped down flavor of some linux that allows you to access a web browser and play music/video. ASUS have such boards.
I've seen download manager programs including IDM taking control of downloads in browsers without having extensions in them and they are calling it (Advanced Browser Integration).
I was wondering if anyone can suggest an approach for a similar situation?
IDM only works on Windows and does his Advanced Browser Integration tricks using Windows Filtering Platform which is a windows specific service.
If you want to do something similar on Windows, you should study that platform.
On unix systems, as far as I know, there isn't anything like the Windows Filtering Platform. Packet filtering and other firewall like functionality happens in the kernel and there are multiple implementations of that: which is running (if any) depends on how the user decided to configure the system (even if ipf is almost guaranteed to be the used one on BSD and BSD derived systems).
On Mac Os X specifically you probably want to check Network Kernel Extensions. I'm not sure they are sufficient to do what you want to do, but I suspect they are.
I've seen programs like magicdisc create virtual cd drives and mount them on the machine. How do these programs trick the operating system into thinking there is a new hardware device attached to it?
I imagine I would have to write a driver for the virtual hardware, and I am comfortable in C so that doesn't sound terrible, but how do I make the OS think there is a piece of hardware attached to it that isn't?
Thanks!
Usually the operating system has different layers and libraries, at some point there is a library that sits between something above it and the actual hardware, you fake it there, if there is some sort of read sector call, you pretend to read a sector using that sector address, read it from a file, whatever. Each operating system (windows, linux, etc) may do things a different way.
Probably you are familiar with C as application level, which is above OS.
The virtual driver is a piece a software, but need to write in proper locations the settings: Registry at Windows, some config file at some Linux. You need to handle OS level calls and callbacks at Kernel level. Than you will desire the software too to communicate with your device, probably integrated into OS Shell. At least 2 layes will be , if not 3 or more.
For eg windows xp you can make a virtual graphic card(and intercept various things), for Vista not, just with a trick :)
It is very-very OS specific.
I want to understand how this applies to an operating system and also to those things that are not infact operating systems. I can't understand the difference between the three and their essence. API is the functions we can call but what is Shell? If we have an API than what exactly is the Kernel of the operating system? I understand the an operating system has a Core that is not going to change and this core does the fundamental Job of a typical OS while we may have different user interfaces like GUI or command line with the same Kernel. So the problem is I am confused how these things are different. Aaaaaaarhg!
Can the functions like printf and fopen in C be called API calls?
A command-line interface (CLI) shell is a command interpreter, i.e. the program that either processes the command you enter in your command line (aka terminal) or processes shell scripts (text files containing commands) (batch mode). In early Unix times, it used to be the unique way for users to interact with their machines. Nowadays, graphical user interfaces (GUIs) are becoming the preferred type of shell for most users.
A kernel is a low level program interfacing with the hardware (CPU, RAM, disks, network, ...) on top of which applications are running. It is the lowest level program running on computers although with virtualization you can have multiple kernels running on top of virtual machines which themselves run on top of another operating system.
An API is a generic term defining the interface developers have to use when writing code using libraries and a programming language. Kernels have no APIs as they are not libraries. They do have an ABI, which, beyond other things, define how do applications interact with them through system calls. Unix application developers use the standard C library (eg: libc, glibc) to build ABI compliant binaries. printf(3) and fopen(3) are not wrappers to system calls but (g)libc standard facilities. The low level system calls they eventually use are write(2) and open(2) and possibly others like brk, mmap. The number in parentheses is a convention to tell in what manual the command is to be found.
The first volume of the Unix manual pages contains the shell commands.
The second one contains the system call wrappers like write and open. They form the interface to the kernel.
The third one contains the standard library (including the Unix standard API) functions (excluding system calls) like fopen and printf. These are not wrappers to specific system calls but just code using system calls when required.
The Shell is the way to communicate with the OS and kernel by command line. The Shell does this by also calling the API.
The kernel is indeed the core of the OS and does memory management, task scheduling, handles with filesystems, I/O handling,...
All the things that the kernel does, can in some way be invoced by the API the OS provides.
printf and fopen are wraps around the system calls (API) provided by the OS and kernel
Shell: It is like a command line interface to your operating system. Commands like ls, ps, kill and many more can be used to request to complete the specific operation to the OS. It is like "cmd" on windows.
Kernel: It is the main code of any operating system. Any request you give on shell or through GUI (like memory allocation, opening a file etc) are finally fulfilled by Kernel.
And yes, the calls you mentioned are regarded as API calls. The request to these calls are also handled by Kernel. Please go the below link to find API calls in unix.
http://www.mkssoftware.com/docs/api_index.asp
This is the overall picture in a unix os:
Applications => (shell+library routines) => system calls => kernel
look the final request handler is the Kernel.
Thx!
Consider an example, You are watching the movie is on shell and actually process done over hardware is the kernel. shell is approximately work as that of os for user and software interface and kernel work as that of os for software and hardware.