I have with interest studied the hardware virtualization extensions added by Intel and AMD to the x86 architecture (known as VMX and SVM, respectively). While this is still a relatively recent addition to x86 CPU's my understanding is that the mainframe architecture made extensive use of virtualization since the 70's-80's for instance in the form of the venerable z/VM operating system. Even nested virtualization has been used.
My question is, is there a public documentation of the hardware facilities provided by the z/Architecture used by the z/VM operating system to implement this virtualization? I.e. the control registers and data structures that the hardware implements to allows the hypervisor to simulate the guest state and trap necessary instructions? Another thing I am curious about is if the z/Architecture supports second-level address translation (which was added later to VMX and SVM).
Just to get it out of the way, System/370 and all its descendants support virtualization as is (they satisfy virtualization requirements). In that sense, no special hardware support has ever been needed, as opposed to Intel architecture.
The performance improvements for VM guests on System/370, XA, ESA etc. all the way through z/Architecture have been traditionally implemented using DIAG (diagnose) instruction as well as microcode (now millicode) assist. In modern terms, it is more of paravirtualization. The facilities are documented, you can start here for instance.
Update - after reading extensive comments, a few notes and clarifications.
S/370 and its descendants never needed specialized hardware virtualization support to correctly run guest operating systems - not because the virtualization was part of the initial design and requirements - it wasn't, but because the architecture was properly designed to support secure multiuser environment. Popek and Goldberg's virtualization requirements are actually very weak - in essence, that only privileged instructions can affect system configuration. This requirement was part of even S/370's predecessor, System/360, well before first virtualized systems appeared.
Performance imporvements of VM guests proceeded along two lines.
First, paravirtualization approach - essentially developing well-architected API for guest-hypervisor communication. It's been used not only for performance, but for a wide variety of other services such as inter-VM communication. The API is documented in the manual referred to above.
Second, microcode extensions (VM microcode assist) that performed some performance sensitive hypervisor logic at microcode level, essentially hardware level. That is not paravirtualization, it is hardware virtualization support proper. But in early 370 machines this support was not architected, meaning it was model-dependent and subject to change. With 370/XA, IBM introduced a proper architectural way to support high-performance virtualization, the Start Interpretive Execution (SIE) instruction. This instruction is not documented in Principles of Operation, but rather in a separate publication, IBM System/370 XA Interpretive Execution. (This document is referenced multiple times in Principles of Operation. The link refers to the first version of the document, you can download version 2 here. I am not sure if that publication was ever updated - probably this is the latest version.) Additionally, I/O subsystem provided VM assist too.
I failed to mention the SIE instruction and the manual that documented it in my original answer, which is a crucial part of the story. I am grateful to the author of the question and the extensive comments that proded me to check my memory and figure out that I skipped an important bit of technical background. This presentation provides an excellent overview of z/VM core facilities that covers additional aspects including memory management, I/O, networking etc.
The SIE instruction is how virtualization software accesses the z/Architecture Interpretive Execution Facility (IEF). The exact details of the interface have not been published since the early 1990s.
This is a hardware-based capability. IEF provides two levels of virtualization. The first level is used by firmware (via the SIE instruction) to create partitions. In each partition you can run an operating system. One of those operating systems is z/VM. It uses the SIE instruction (running within the context of the first level SIE instruction) to run virtual machines. You can run any z/Architecture operating system in a virtual machine, including z/VM itself.
The SIE instruction takes as input the description of a virtual server (partition or virtual machine). The hardware then runs the virtual server's instruction stream, stopping only when it needs help from whatever issued the SIE instruction, whether it be the partition hypervisor or the z/VM hypervisor.
Related
What is the connection between microprogramming and embedded systems?
Is microprogramming a machine language?
Microprogramming it is the same as microcode?
Are embedded systems manufactured using microprogramming only?
Or isn't it an exclusivity of an embedded system that is using microprogramming?
If possible, please exemplify. Thanks!
Microprogramming / microcoding is an implementation technique for processors — as such dates back pretty far.
A processor implements an instruction set; programs using these instructions are generated by a compiler or assembly language programmer and stored in program files, later loaded into memory to execute the program.
A microcoded processor is like having another, different processor-within-the-processor that is used to interpret the instruction stream (sequences of machine language) of the program. This processor within the processor has its own instructions set and its own program. Unlike the externally visible instruction set (which can load & run any program), the processor within the processor generally only runs one dedicated program (the instruction set interpreter), which is stored in a ROM (or re-writable flash) inside the processor.
(In some such systems, the processor within the processor has instructions that are very wide (as in horizontal microcode), and impractical (regarding code size) for general use by regular programs.)
What is the connection between microprogramming and embedded systems?
There is no particular relationship between microcoding and embedded. Yes/no on either can be mixed with each other.
Is microprogramming a machine language?
Yes, I would say it is, but it is generally not accessible to operating systems and user programs.
Microcoding was particularly popular when virtually all instructions each executed in multiple cycles. Later techinques removed the indirection of the microcoded machine in favor of direct hardwired execution, with single cycle approaches. This publication sheds some light on some of the thinking of the day during the transition of the state of the art from microcoding to hard wiring. See also IBM 801.
Most processors these days are not microprogrammed; however, the very advanced techniques applied by x86 processors may mimic microprogramming techniques here and there.
Embedded systems are simply processors used in devices that are not seen as "computers", for example, a thermostat, a microwave, or a car (which might have numerous embedded systems). Considerations here are that these systems are dedicated: they tend to run a single program (rather than running an operating system capable of running any program the user directs); they have low power requirements, disconnected requirements (disconnected from user terminal/screen/keyboard, perhaps from network, etc..). Still, embedded system keep getting even more powerful.
As I understand, a BSP (Board Support Package) contains bootloader, kernel and device driver which help OS to work on HW. But I'm confused because OS also contains a kernel. So what is the difference between the kernel in OS and the kernel in BSP?
What a BSP comprises of depends on context; generically it is code or libraries to support a specific board design. That may be provided as generic code from the board supplier for use in a bare-metal system or for integrating with an OS, or it may be specific to a particular OS, or it may even include an OS. In any case it provides board specific support for higher-level software.
A kernel is board agnostic (though often processor architecture specific), and makes no direct access to hardware not intrinsic to the processor architecture on which it runs. Typically an OS or application will require a Hardware Abstraction Layer (HAL); the HAL may well be built using the BSP, or the BSP may in fact be the HAL. A vendor may even package a HAL and OS and refer to that as a BSP.
The term means what it means to whoever is using it - context is everything. For example in VxWorks, WindRiver use the term BSP to refer to the layer that supports the execution of a VxWorks based application on a specific hardware design. A board vendor on the other hand may provide a complete Linux distribution ported to the board and refer to that as a BSP.
However and to what extent a particular vendor or developer chooses to support a board is a board support package regardless of how much or how little it may contain.
BSP definition is broad. It is a supporting software package for a specific board. BSP for a tiny microcontroller probably just contains HW drivers for its peripherals. On the other hand, for an embedded CPU it may contain HW drivers, bootloader and OS kernel and what not.
So the kernel in a BSP (board support package) is just a specific version of an OS kernel that has been ported to your board.
Im probably just saying the same things already said.
You have a chip and/or board product you want to sell to other (software) developers. A reference design (board) with the chip(s) in question are used. The BSP is a vague term to mean the software that is provided to you as a software developer to ideally make your life easier in using that product (chip and/or board) and adding your software to it or developing for it. So if it is a linux or rtos or other operating system capable platform and the vendor (providing the bsp) believes that users want an operating system and a specific operating system, then instead of you having to port the os to that target, they do it for you. If something like linux that is open source, then you either are told which linux sources to download then the patches made by the bsp are added and/or the bsp contains the complete sources for the whole thing already patched. Drivers, applications as deemed necessary by the vendor etc. Multiple operating systems may be supported if the vendor feels that is needed in order to attract customers to buy that board/chip product.
The whole package of software that you get from them to make that chip/board into your own product, is the BSP.
vxWorks kernel which you can run on a Board contains vxWorks core kernel and "other components" which may change from one environment.
Core kernel contains essential programs such as Scheduler, Memory manager, Basic File systems, security features etc.
These "other components" which are part of BSP may be optional or may vary from system to system, and helps the core kernel features.
In simple words, the image dislays the defination of BSP. Please correct me if I'm wrong
I would say for a well structured code base, the application layer should be abstracted from lower layers by the HAL layer. This would allow the app layer to be portable if we want to migrate the system to a new board. If you see you have board/CPU specific logic in your app layer, you know you have broken the portability.
The HAL layer functions' bodies should contain board specific code, here is where the BSP layer code comes into play. When we want to port the system to a new board, code changes should happens in the HAL functions' bodies, while the HAL functions' declaration should not change, which leads to the app layer remains the same.
A colleague of mine is getting a quote from a software developer that involves serial communication, and in their quote, the developer makes the the following statement:
...Windows 7 operating system, which uses a non-real-time serial communication setup.
Is it true that Windows 7 does not support real-time serial communication? To clarify on what it is meant by "real-time," the project deals with robot automation and any delays in communication (such as from buffering) could cause damage to the product, or stop the production line. I can not find any evidence to either support or deny this claim. I don't believe it to be true though, and I think it probably has more to do with them using VB.Net for development.
The 'real-time' term used here does not actually refer to anything in the serial communications bus.
However, it does have to do with the fact that the windows multitasking scheduler is not designed to allow for realtime tasks which have hard deadlines.
See this question for some info Why is Windows not considered suitable for real time systems/high performance servers?
Lets pretend you have a particle accelerator hooked up to your computer and you have to ensure that every 10 microseconds the magnet train switches to power the next set of cells but windows decides that it's time to apply some Windows Update patches. Your photon stream wouldn't get redirected properly and could cause damage to the system.
It is a fairly nonsensical statement, Windows itself is not a real-time operating system. It cannot provide hard guarantees that user mode code is going to respond quick enough. Other than thread scheduling delays, a simple mishap like getting the pages of the process swapped to the paging file is enough to cause arbitrary delays in getting it running again. An attribute of any demand-paged virtual memory operating system. So of course a "serial communications setup" cannot be either, assuming you are not contemplating writing ring 0 kernel code. Nobody does.
It is not a practical problem, the only point of using a serial port is to talk to the controller for the robot. Which provides the real-time guarantee.
You could only get in trouble when you command the robot to make an unrestricted move and use an external sensor to get it to stop. Not uncommon when you need to find an object whose location you don't know. A decent controller knows how to do that, avoid implementing it in your Windows code. Solid overtravel protection built into the robot itself that triggers an e-stop is necessary anyway, you can't trust that sensor either.
No, Windows 7 (and in fact all of the mainstream Windows releases) are not Real-time operating systems. To clarify what is meant by a real-time operating system:
A real-time operating system (RTOS) is an operating system (OS)
intended to serve real-time application requests. It must be able to
process data as it comes in, typically without buffering delays.
Processing time requirements (including any OS delay) are measured in
tenths of seconds or shorter.
A key characteristic of an RTOS is the level of its consistency
concerning the amount of time it takes to accept and complete an
application's task; the variability is jitter.[1] A hard real-time
operating system has less jitter than a soft real-time operating
system. The chief design goal is not high throughput, but rather a
guarantee of a soft or hard performance category. An RTOS that can
usually or generally meet a deadline is a soft real-time OS, but if it
can meet a deadline deterministically it is a hard real-time OS. [2]
An RTOS has an advanced algorithm for scheduling. Scheduler
flexibility enables a wider, computer-system orchestration of process
priorities, but a real-time OS is more frequently dedicated to a
narrow set of applications. Key factors in a real-time OS are minimal
interrupt latency and minimal thread switching latency; a real-time OS
is valued more for how quickly or how predictably it can respond than
for the amount of work it can perform in a given period of time.[3]
Note that most of the time real-time operating systems are less efficient (i.e. have lower throughput), which is why none of the mainstream operating systems are real-time (e.g. real-time editions of Linux use completely different kernels) - its only worth it in cases where timing at a very precise level is absolutely critical.
Windows CE is a real-time operating system Real-Time Systems with Microsoft Windows CE 2.1
First the background, specifics of my question will follow:
At the company that I work at the platform we work on is currently the Microchip PIC32 family using the MPLAB IDE as our development environment. Previously we've also written firmware for the Microchip dsPIC and TI MSP families for this same application.
The firmware is pretty straightforward in that the code is split into three main modules: device control, data sampling, and user communication (usually a user PC). Device control is achieved via some combination of GPIO bus lines and at least one part needing SPI or I2C control. Data sampling is interrupt driven using a Timer module to maintain sample frequency and more SPI/I2C and GPIO bus lines to control the sampling hardware (ie. ADC). User communication is currently implemented via USB using the Microchip App Framework.
So now the question: given what I've described above, at what point would I consider employing an RTOS for my project? Currently I'm thinking of these possible trigger points as reasons to use an RTOS:
Code complexity? The code base architecture/organization is still small enough that I can keep all the details in my head.
Multitasking/Threading? Time-slicing the module execution via interrupts suffices for now for multitasking.
Testing? Currently we don't do much formal testing or verification past the HW smoke test (something I hope to rectify in the near future).
Communication? We currently use a custom packet format and a protocol that pretty much only does START, STOP, SEND DATA commands with data being a binary blob.
Project scope? There is a possibility in the near future that we'll be getting a project to integrate our device into a larger system with the goal of taking that system to mass production. Currently all our projects have been experimental prototypes with quick turn-around of about a month, producing one or two units at a time.
What other points do you think I should consider? In your experience what convinced (or forced) you to consider using an RTOS vs just running your code on the base runtime? Pointers to additional resources about designing/programming for an RTOS is also much appreciated.
There are many many reasons you might want to use an RTOS. They are varied & the degree to which they apply to your situation is hard to say. (Note: I tend to think this way: RTOS implies hard real time which implies preemptive kernel...)
Rate Monotonic Analysis (RMA) - if you want to use Rate Monotonic Analysis to ensure your timing deadlines will be met, you must use a pre-emptive scheduler
Meet real-time deadlines - even without using RMA, with a priority-based pre-emptive RTOS, your scheduler can help ensure deadlines are met. Paradoxically, an RTOS will typically increase interrupt latency due to critical sections in the kernel where interrupts are usually masked
Manage complexity -- definitely, an RTOS (or most OS flavors) can help with this. By allowing the project to be decomposed into independent threads or processes, and using OS services such as message queues, mutexes, semaphores, event flags, etc. to communicate & synchronize, your project (in my experience & opinion) becomes more manageable. I tend to work on larger projects, where most people understand the concept of protecting shared resources, so a lot of the rookie mistakes don't happen. But beware, once you go to a multi-threaded approach, things can become more complex until you wrap your head around the issues.
Use of 3rd-party packages - many RTOSs offer other software components, such as protocol stacks, file systems, device drivers, GUI packages, bootloaders, and other middleware that help you build an application faster by becoming almost more of an "integrator" than a DIY shop.
Testing - yes, definitely, you can think of each thread of control as a testable component with a well-defined interface, especially if a consistent approach is used (such as always blocking in a single place on a message queue). Of course, this is not a substitute for unit, integration, system, etc. testing.
Robustness / fault tolerance - an RTOS may also provide support for the processor's MMU (in your PIC case, I don't think that applies). This allows each thread (or process) to run in its own protected space; threads / processes cannot "dip into" each others' memory and stomp on it. Even device regions (MMIO) might be off limits to some (or all) threads. Strictly speaking, you don't need an RTOS to exploit a processor's MMU (or MPU), but the 2 work very well hand-in-hand.
Generally, when I can develop with an RTOS (or some type of preemptive multi-tasker), the result tends to be cleaner, more modular, more well-behaved and more maintainable. When I have the option, I use one.
Be aware that multi-threaded development has a bit of a learning curve. If you're new to RTOS/multithreaded development, you might be interested in some articles on Choosing an RTOS, The Perils of Preemption and An Introduction to Preemptive Multitasking.
Lastly, even though you didn't ask for recommendations... In addition to the many numerous commercial RTOSs, there are free offerings (FreeRTOS being one of the most popular), and the Quantum Platform is an event-driven framework based on the concept of active objects which includes a preemptive kernel. There are plenty of choices, but I've found that having the source code (even if the RTOS isn't free) is advantageous, esp. when debugging.
RTOS, first and foremost permits you to organize your parallel flows into the set of tasks with well-defined synchronization between them.
IMO, the non-RTOS design is suitable only for the single-flow architecture where all your program is one big endless loop. If you need the multi-flow - a number of tasks, running in parallel - you're better with RTOS. Without RTOS you'll be forced to implement this functionality in-house, re-inventing the wheel.
Code re-use -- if you code drivers/protocol-handlers using an RTOS API they may plug into future projects easier
Debugging -- some IDEs (such as IAR Embedded Workbench) have plugins that show nice live data about your running process such as task CPU utilization and stack utilization
Usually you want to use an RTOS if you have any real-time constraints. If you don’t have real-time constraints, a regular OS might suffice. RTOS’s/OS’s provide a run-time infrastructure like message queues and tasking. If you are just looking for code that can reduce complexity, provide low level support and help with testing, some of the following libraries might do:
The standard C/C++ libraries
Boost libraries
Libraries available through the manufacturer of the chip that can provide hardware specific support
Commercial libraries
Open source libraries
Additional to the points mentioned before, using an RTOS may also be useful if you need support for
standard storage devices (SD, Compact Flash, disk drives ...)
standard communication hardware (Ethernet, USB, Firewire, RS232, I2C, SPI, ...)
standard communication protocols (TCP-IP, ...)
Most RTOSes provide these features or are expandable to support them
I know little about MCUs and embedded systems.
A year ago we made contract with a company to design a special purpuse MP4 device based on the SigmaTel STMP 3650 kit. Now we have all the source code for the firmware (code, resource around 1G).
My questions are
Can we use this code to run on other STMP 3xxx famliy based devices (with acceptable modification, of course)? What about other ARM9 based devices?
ARM9 defines the processor core (but even then there are variants; yours being ARM962EJ-S), but most on-chip peripherals and support hardware including clocks, PLLs, and interrupt controller are vendor specific, you you would have to port your hardware initialisation and device driver code, and make sure that you choose a device with a comparable peripherals set to the ones your current code uses.
Moreover if the code is written in C or C++ rather than assembler, much of it may be prtable to other architectures, particularly if the application layer and hardware abstraction layer are well defined.
Another issue may be whether your existing implementation relies on any particular OS or RTOS; you may need to select a device that supports the same OS in order to reduce the porting effort.
Finally, a non-programming point, but just to keep you out of trouble; you need to be sure that you own the rights to the code you intend to reuse, and that the original client has no claim on it.
your logical successor chip is iMX233 from Freescale for a couple of reasons.
STMP3650 lead to STMP3780 by SigmaTel - same CPU core (ARM9EJ-S), mostly same architecture and registers. Then, SigmaTel was sold to Freescale and they simply copied STMP3780 to ...iMX233. Identical silicon.
We have a fully fledged MP3/MP4 player based on STMP3650 (see bones.ch website) and transfer our R&D now to iMX233. What is your project doing by now? How "good" was the design based on STMP3650 running? Do you have any chip stock left?