Important Unix Concepts

IPC mechanisms:
http://mia.ece.uic.edu/~papers/WWW/multi-process/multi-process.html

Shared memory is the fastest of all IPC schemes. The memory to be shared is mapped into the address space of the processes (that are sharing). The speed achieved is attributed to the fact that there is no kernel involvement. But this scheme needs synchronization.

One advantage by using pipe as IPC,It does not support a flush operation. If all of the data writtenin a `PIPE:' device is not read out, it will stay there, buffered. It would be useful remote server connection gets lost in the middle, atleast Clients can retrieve data available in PIPE, not like sockets.

Synchronisation mechanisms:
http://mia.ece.uic.edu/~papers/WWW/multi-process/multi-process.html

* A semaphore is useful for managing an object that is shared by several threads or processes, but should be limited. A mutex is useful for managing access to an object that should only be accessed by a single thread at a time. * mutex is essentially a semaphore with its max count=1

Sheduling:

On DNP we schedule some important processes as Realtime and some other non-important processes as Time Shared.The realtime process will have more prority than the Time Share process.Whenever Realtime process goes for I/O,then the time shared processes will be scheduled.The time shared process will have the number of clock-ticks it needs,prior to scheduling.

Sockets, Client-Server:
http://users.actcom.co.il/~choo/lupg/tutorials/internetworking/internet-theory.html

* When you are finished using a socket, you can simply close its file descriptor with close. If there is still data waiting to be transmitted over the connection, normally close tries to complete this transmission. You can control this behavior using the SO_LINGER socket option to specify a timeout period; The shutdown function shuts down the connection of socket socket. The argument how specifies what action to perform:

Real Time OS:

* An RTOS will have a deterministic scheduler. For any given set of tasks your process will always execute every number of microseconds or miliseconds exactly, and the same number from schedule to schedule

* In UNIX and Windows the scheduler are usually soft-realtime (as opposed to some hard-realtime RTOS). Soft-realtime means that the scheduler tries to assure your process runs every X number of miliseconds, but may fail to do so on occassion. * Modern RTOSs simply make sure that a) no interrupt is ever lost, and b) no interrupt can be blocked by a lower priority process.

* The real difference between an RTOS and a general purpose OS is that with an RTOS the designers have taken care to ensure that the response times are known

fork:

The `fork()' function is used to create a new process from an existingprocess. The new process is called the child process, and the existingprocess is called the parent. You can tell which is which by checking thereturn value from `fork()'. The parent gets the child's pid returned tohim, but the child gets 0 returned to him.

Childs inherits from the parent the following stuff:
* process credentials (real/effective/saved UIDs and GIDs)
* environment
* stack
* memory
* open file descriptors (note that the underlying file positions are shared between the parent and child, which can be confusing)
* close-on-exec flags
* signal handling settings
* nice value
* scheduler class
* process group ID
* session ID
* current working directory
* root directory
* file mode creation mask (umask)
* resource limits
* controlling terminal

Childs wont inherit the following stuff:
* process ID
* different parent process ID
* Own copy of file descriptors and directory streams.
* process, text, data and other memory locks are NOT inherited.
* process times, in the tms struct
* resource utilizations are set to 0
* pending signals initialized to the empty set
* timers created by timer_create not inherited
* asynchronous input or output operations not inherited

Signals:

*A signal is a message which can be sent to a running process.Signals can be initiated by programs, users, or administrators.

*The signal() system call is used to set a signal handler for a single signal type. signal() accepts a signal number and a pointer to a signal handler function, and sets that handler to accept the given signal

*For example, to the proper method of telling the Internet Daemon (inetd) to re-read its configuration file is to send it a SIGHUP signal.

*By default, the kill command sends the SIGTERM signal. If SIGTERM fails, we can escalate to using the SIGKILL signal to stop the process:

*Two signals are unable to be redefined by a signal handler. SIGKILL always stops a process and SIGSTOP always moves a process from the foreground to the background. These two signals cannot be "caught" by a signal handler.

*When a child process calls exit(), a SIGCHLD signal is sent by the system to the parent.If there is no explicit signal handler for SIGCHILD in the parent process, the child process will be kept in Zombie state,till the point of Parent process dies.

* what are synchronous and async signals??* How to generate a particular signal? by using kill command along with the signal number.* send of signals to a process which is blocking/waiting on system call will disturb the process's system call. For example, if a process ( waiting on accept ) gets a SIGUSR1, will come out of accept system call.* by sending signal zero to particular PID, we can know the process existance.If signal delivery gets succedeed, the kill command returns zero otherwise it will delivery non zero value.

* when kill() is called with PID as zero, the signal will be issued to all the processes in the same group* when kill() is called with PID as -1, the signal will be issued to all processes, except swapper(PID=0),init(PID=1) and current process

Select vs Poll:

*Note: `select()' was introduced in BSD, whereas `poll()' is an artifact ofSysV STREAMS. As such, there are portability issues; pure BSD systems maystill lack `poll()', whereas some older SVR3 systems may not have`select()'. SVR4 added `select()', and the Posix.1g standard defines both.

*`select()' and `poll()' essentially do the same thing, just differently.Both of them examine a set of file descriptors to see if specific eventsare pending on any, and then optionally wait for a specified time for anevent to happen.

*[Important note: neither `select()' nor `poll()' do anything useful whenapplied to plain files; they are useful for sockets, pipes, ptys, ttys &possibly other character devices, but this is system-dependent.]

Typical Process Layout:

Command-line arguments and environment
variables
Stack
Heap
Uninitialized data(bss)
Initialized data Text(code)

Differences between Process and Thread:

*fork() is an expensive system call. Creating a new process requires more system memory space thus it causes more load on the operating system in keeping track of active processes [Ste98]. In a multiple-process application, the only way for the processes to share resources is through an Interprocess Communication (IPC) object, which is maintained by and kept within the system. For the user program, usage of these IPC objects is normally simple and abstract but causes heavy system overhead. On the other hand, in a multiple-thread application, all the threads share resources within user address space and load on the operating system is reduced. However, the synchronization between threads is much more complicated. Moreover, using system call functions in multiple-thread application may have some unusual implications

* in the M X N model, there will be a user level schedular for taking care of CPU scheduling across threads

* in the 1 X 1 model, Kernel schedular will take care of scheduling fo CPU across threads.

* kernel threads are required in multi processor environment..where to address some of the scheduling complexities.

* from a thread you can do fork..but it will just inherit the code of thread..not all other threads..if you want to inherit other threads too..call pthread_atfork function.

* what exactly pthread_join does? the main thread will join to child threads,to make sure the main thread is alive till the end of child threads.

* An application can create both PTHREAD_SCOPE_SYSTEM and PTHREAD_SCOPE_PROCESS threads explicitly using the pthread_attr_setscope API. When no contention scope is specified, a thread created by pthread_create has a contention scope of PTHREAD_SCOPE_PROCESS. This change in default thread type is due to POSIX standards requirement.Refer http://docs.hp.com/en/5187-0701/ch07s15.html

clone system call:

clone creates a new process like fork(2) does. Unlike fork(2), __clone allows the child process to share parts of its execution context with its parent process, such as the memory space, the table of file descriptors, and the table of signal handlers. The main use of __clone is to implement threads: multiple threads of control in a program that run concurrently in a shared memory space.

signals and threads:

Synchronous signals, those caused by the thread itself (like SIGPIPE and SIGBUS), are delivered to the thread that caused them. Asynchronous signals (signals sent to the process by an external source) are delivered to an arbitrary thread within the process.At thread level to block the signal,a thread needs to set appropriate mask by calling function pthread_sigmask. Note thatsignal handlers are at process level and signal masks are at thread level.

Basic differences between Linux and Traditional Unix:

1.Linux kernel is monolithic.It is a large,composed of several logically different components.

2. Traditional Unix kernels are compiled and linked statically. Most modern kernels can dynamically load and unload some portions of the kernel code (typically, devicedrivers), which are usually called modules. Linux's support for modules is very good,since it is able to automatically load and unload modules on demand.

3. Kernel threading. Some modern Unix kernels, like Solaris 2.x and SVR4.2/MP, areorganized as a set of kernel threads. A kernel thread is an execution context that can be independently scheduled; it may be associated with a user program, or it may run only some kernel functions. Context switches between kernel threads are usually much less expensive than context switches between ordinary processes, since the former usually operate on a common address space. Linux uses kernel threads in a very limited way to execute a few kernel functions periodically; since Linux kernel threads cannot execute user programs, they do not represent the basic execution context abstraction

4. Multithreaded application support. Most modern operating systems have some kind ofsupport for multithreaded applications, that is, user programs that are well designed interms of many relatively independent execution flows sharing a large portion of theapplication data structures. A multithreaded user application could be composed ofmany lightweight processes (LWP), or processes that can operate on a common address space, common physical memory pages, common opened files, and so on.

Linux defines its own version of lightweight processes, which is different from thetypes used on other systems such as SVR4 and Solaris. While all the commercial Unixvariants of LWP are based on kernel threads, Linux regards lightweight processes asthe basic execution context and handles them via the nonstandard clone( ) systemcall.

5. Linux is a nonpreemptive kernel. This means that Linux cannot arbitrarily interleaveexecution flows while they are in privileged mode. Several sections of kernel codeassume they can run and modify data structures without fear of being interrupted andhaving another thread alter those data structures. Usually, fully preemptive kernels are associated with special real-time operating systems. Currently, among conventional,general-purpose Unix systems, only Solaris 2.x and Mach 3.0 are fully preemptive kernels. SVR4.2/MP introduces some fixed preemption points as a method to get limited preemption capability.

6. Multiprocessor support. Several Unix kernel variants take advantage of multiprocessor systems. Linux 2.2 offers an evolving kind of support for symmetric multiprocessing(SMP), which means not only that the system can use multiple processors but also that any processor can handle any task; there is no discrimination among them. However,Linux 2.2 does not make optimal use of SMP. Several kernel activities that could beexecuted concurrently.like filesystem handling and networking.must now beexecuted sequentially.

7. Filesystem. Linux's standard filesystem lacks some advanced features, such asjournaling. However, more advanced filesystems for Linux are available, although notincluded in the Linux source code; among them, IBM AIX's Journaling File System(JFS), and Silicon Graphics Irix's XFS filesystem. Thanks to a powerful objectoriented Virtual File System technology (inspired by Solaris and SVR4), porting a foreign filesystem to Linux is a relatively easy task.

8. On Linux 2.6, on every pthread_create call, kernel creates a corresponding LWP.

Linux kernel Mechanisms:

1. Bottom Half Handling:-There are often times in a kernel when you do not want to do work at this moment. A good example of this is during interrupt processing. When the interrupt was asserted, the processor stopped what it was doing and the operating system delivered the interrupt to the appropriate device driver. Device drivers should not spend too much time handling interrupts as, during this time, nothing else in the system can run. There is often some work that could just as well be done later on. Linux's bottom half handlers were invented so that device drivers and other parts of the Linux kernel could queue work to be done later on2. Task Queues:-Task queues are the kernel's way of deferring work until later. Linux has a generic mechanism for queuing work on queues and for processing them later. Task queues are often used in conjunction with bottom half handlers; the timer task queue is processed when the timer queue bottom half handler runs. 3. Timers:-4. Interrupts:-* In Linux, the system timer (or clock) is programmed to generate a hardware interrupt 100 times a second (as defined by the HZ system parameter). The interrupt is accomplished by sending a signal to a special chip on the motherboard called an interrupt controller. (We go into more detail about these in the section on hardware.) The interrupt controller then sends an interrupt to the CPU. When the CPU receives this signal, it knows that the clock tick has occurred and it jumps to a special part of the kernel that handles the clock interrupt. Scheduling priorities are also recalculated within this same section of code.

* When a page fault does happen when in kernel mode, the kernel panics. Special routines have been built into the kernel to deal with the panic to help the system shut down as gracefully as possible.

* When a process makes a system call, the behavior is similar to that of interrupts and exceptions. Like exception handling, the general purpose registers and the number of the system call are pushed onto the stack. Next, the system call handler is invoked, which calls the routine within the kernel that will do the actual work.

Linux boot sequence:

1.Processor comes out of reset and branches to the ROM startup code.

2.The ROM startup code initializes the CPU and memory controller, performing only minimal initialization of on-chip devices, such as the console serial port (typically SMC1 on 8xx devices) to provide boot diagnostic messages. It also sets up the memory map for the kernel to use in a format that is consistent across platforms, and then jumps to the boot loader.

3.The boot loader(Linux Loader(LILO)) decompresses the kernel(example vmlinuz.x.x.x) into RAM, and executes it.

4.The kernel program(example vmlinuz.x.x.x) sets up the caches, detects(via bus) & initializes each of the hardware devices via the init function in each driver, mounts the root filesystem and execs the init process, which is the ultimate parent of all user mode processes, typically /sbin/initd.

5.Executing the first program linked against the shared C runtime library (often init) causes the shared runtime library to be loaded.

6.In a typical Linux system, init reads /etc/inittab to execute the appropriate run control script from /etc/rc.d, which execute the start scripts to initialize networking and other system services. The rc scripts will start all network daemons.

7.At last init will spawn getty process,which is responsible for all user logins.

The main Differences between 2.6 kernel and 2.4 kernel:

1. 2.6 kernel scheduling algorithm got changed to O(1) algo.Will work better on multi processors

2. 2.6 kernel made partially preemptable,during the some part of kernel code exec

3. there were some changes made to VM in 2.6, for improving physical page removal,while many processes map the same page.

4. Linux threads removed in 2.6,in place of it Native Posix Thread Libs are introduced.NPTL works in 1:1 manner.and proved to be faster.Each thread has a separate PID.

5. 2.6 also got changed in terms of Max number of threads supported on the system to approx 2 billion,whereas 2.4 was supporting 8192 threads per processor.

6. Workqueue interface got introduced in 2.6 kernel,which will replace task queue interface(used to schedule kernel tasks)

Steps involved in calling main function:

1. GCC build your program with crtbegin.o/crtend.o/gcrt1.o And the other default libraries are dynamically linked by default. Starting address of the executable is set to that of _start.

2. Kernel loads the executable and setup text/data/bss/stack, especially, kernel allocate page(s) for arguments and environment variables and pushes all necessary information on stack.

3. Control is pased to _start. _start gets all information from stack setup by kernel, sets up argument stack for __libc_start_main, and calls it.

4. __libc_start_main initializes necessary stuffs, especially C library(such as malloc) and thread environment and calls our main.

Some interesting problems/topics:

1. If a file is open with read only..fclose() function will not flush the buffer..its better to use explicit system call fflush for doing the same.If fil buffers are not flushed properly, there could be unexpected file corruption problems,provided if the files are shared between parent and child processes.

2.How to find how many network interfaces are associated with a socket?THis is by using ioctl system call,by passing sockfd as the arguement.THe system call returns the list of interfaces.

3.How to effectivly kill children when parent gets abruptly terminated??This can be achieved by using system call prctl in children's logic,for notifying it with a signal whenever parents terminates.Note that it is supported by Linux only.

4.Unix wont allocate PIDs in sequential manner.Its because to avoid unexpected signals to the new processes

5..read. is a system call and fread is a C (glibc) linbrary function. fread uses buffer caches and internal buffers which will fasten the operation of reading. Fread will be more efficient while using on block device like disk. Read can be used for accessing a character device like a network device.

6.difference between system call and a library functionA system call is an entry point to the kernel. system calls are executed in kernel mode. Hence, whenever a system call is made from a user program a mode switching has to be done from "user mode" to "kernel mode". System calls are provided by each OS. They are not portable. example: read

A library function usually works in a user mode which may or may not require any OS services, for example, strlen() which calculates the length of a particular string.

7.A function is called thread safe, if multiple threads can call that function without any destructive results. A function can be made a thread safe by locking a mutex while entering the function and releasing the mutex while exiting the function. (or) locking the mutex while executing the critical section of the code (or) locking the mutex while accessing the critical data and unlocking once done with it.

8.The HP OpenCall SS7 APIs (except the TimerLib) are Posix.1c Thread-Restrictedlevel B. This means that the interface is not thread-safe, but that it can be used byany single .dedicated. thread of a multi-threaded application.

9. to see the individual size of various segments of a lib or of object file,use size command.That will give size of text,data and bss segments.

10. Kernel panics, are two types. 1. Hard Panic 2.Soft Panic. Hard panics are usually happens in interrupt handler...and soft panics generally happens in driver software For debugging the hard panic,KDB can be used to get extra information for finding the root c