Unlock all 21 lessons of Linux System ProgrammingFull course ₹40,000 · paid
How Shells and System Calls Work: fork, execve, Trap and open
A free lesson from Skill-Lync's Linux System Programming course, on Grow by Skill-Lync · Last updated 16 Nov 2026 · Session: “Week 1 Session 2”
What will you learn in this free Linux System Programming lesson?
The lesson explains how a shell runs a command using fork, execve and waitpid, with a page table and the trap instruction as background. It then compares normal calls with system calls, steps through a read call using register RAX and the interrupt service routine, mentions DPDK, and runs an open system call example.
- How a shell uses fork, execve and waitpid
- Trap instruction and page fault handling
- Steps of a read system call into the kernel
- Using the open system call in C
What comes after lesson 2?
The next free lesson is lesson 3, “Memory Hierarchy, CPU Registers, Paging, TLB and BIOS” (29 min). Later in Skill-Lync's Linux System Programming course (paid): lesson 4, Week 1 Session 4; lesson 5, Week 2 Session 1; lesson 6, Week 2 Session 2. The full course has 21 lessons (7 h 25 min of video); see the course overview or its price and syllabus.
Lesson transcript
Machine transcript of the lesson audio (lightly cleaned). Technical terms may contain errors; the video is the reference.
This is week one, session two. We were on shell. So we have already discussed types of shells. So now the shell working principle, how basically the kernel shell works. So if we remember from our previous session, shell basically is a program which sits over the Linux kernel space and it does the standard library calls. So it's next to the user. So basically what is shell? So this is the shell. So if I am on my Ubuntu terminal, so this is my Ubuntu machine. This is a Linux box running a GNU Linux. So this is a Ubuntu machine which is installed on the Windows operating system.
And this command prompt is a shell. So it actually interacts with the user. I can just type in the commands. For example, LS, it will just show me whatever the contents are, execute some of the system calls and just provide the output to the user. So this is the shell, the blinking cursor. This is a shell. So now how basically shell works inside the Linux kernel, we are going to discuss it. So the shell basically puts the arguments onto the register. So what does the argument mean here? So for example, let us consider I am on a shell and I have type LS minus LTR.
So in this case, the LTR will be the arguments. If I type CP test1 and test2. So it copies test1 to test2. So these are the two arguments in this case. So whenever we write the shell. So whenever we write the command onto the shell, for example, this is a shell dollar prompt. Here we have written this command. It puts the arguments onto the register. Then it switches the context from user mode to kernel mode using the trap instruction. So now we are going into the kernel mode using the trap instruction. So it executes a trap instruction.
So how the trap instruction basically works, let us look into that. Consider a 16-bit system. So in a 16-bit system, we have something as 64 KB of hard disk. These are actually very less, but for our study purpose, we are taking it low. And it will have somewhere around 16 pages. Then we can say the page size is 4 KB, every page size, which makes it up to 4096 bytes. But in real systems, page size is maybe up to as large as 64 KB. So if we consider 4 KB page sizes, so we have 8 frames here.
So before moving on to that, how the trap instruction works, let us look at the page table. So a page table in the Linux operating system is a mapping between two tables. So one is here and other is here. Sorry for my drawing again. So on the left-hand side, these are the virtual addresses. And we can say a virtual table and this is the physical table or the physical addresses. So page table is basically used for storing the virtual pages or the virtual addresses and map to the physical addresses.
So basically this virtual memory, we can say this is a hard disk. And this physical address or physical memory, we can say a RAM. So in all the systems, we'll have hard disk is very big and RAM is definitely small. So we have to map the pages. So every page here is a 4 KB size and every frame here is also 4 KB size. So the page table actually maps different pages, virtual pages onto the physical memory that is called, we call a page frame. So now let's go into the concept of trap.
So with this in mind, page table concept in mind, let's suppose program tries to access the virtual address 0. So in that case, the virtual address is sent to MMU for translation. So let's suppose the program's code segment. So we also see about the segments of a program. It says that why I want to access the virtual address 0. So it will have something like this in assembly. Move segment 0. So this will ask CPU. CPU and CPU will ask MMU to get the physical address of 0.
MMU will return it and it will say the physical address is at 8192. So it means that the page 0 maps to 8192. So here it means that the page 0 maps to somewhere here. So this is 4KB and this is something here. It maps to here. Right. So MMU maintains this mapping. So this physical address maps to this virtual address. Okay. We got it. Now, let me move on to here. So the concept of the trap actually comes into the picture when MMU does not have the virtual address of a physical page.
So what happens is in the code segment, our program says that I want the address of something, let's say X. Once it goes to CPU, CPU again checks MMU, MMU says, no, I don't have. Now our trap CPU calls this trap instruction. So what does this trap instruction does? It says map X and it executes some commands. Right. And what are those commands? Basically, CPU is moving to the kernel space. So here we have a RAM, which is our physical memory. And here we have our hard disk, which is our virtual memory.
So what happens now on RAM? So it says that move frame 0 to RAM. So this hard disk, from hard disk, page number X, that is some address, page number X is loaded in RAM. Right. Then what happens here? MMU updates its mapping. And the other page, which is not mapped, let's suppose some page Y was there, which was not mapped. It's discarded or dropped. And the virtual address of page 0 is sent back to the CPU. Virtual address of page X is sent back to the CPU.
So this whole process that when the mapping of the page is not found inside the MMU, that is when program tries to access the virtual address, its physical address is not found. In that case, CPU goes and executes a trap instruction and it does all this processing, that is changes the content in RAM, moves out the pages which are not required to hard disk. MMU updates its mapping and the actual physical address is returned to CPU is what the trap instruction does. So CPU switches the context from user to kernel mode using the trap instruction and gets the actual physical address of the virtual address which was typed in by the CPU.
And then the user types the command, shell extracts the first word. And why this happens? For example, if we have a command CP in code segment, we try to access the code for it or the pages for it. And the pages are not present in the RAM and the CPU checks in MMU. So in that case, the trap instruction happens so that the actual code which wants to be executed, which has to be executed for CP need to be copied back into the RAM. So then it assumes this as a program and it forks the child.
So child runs the program and shell waits on the wait PID. So this is the source code for the working shell. This is just for the demonstration purpose. This is not. execute or run. So what happens is we have a while loop. In while loop we are just showing this dollar sign and in the dollar sign which shell shows, it's the same. In the shell might have some other method to get input from the user but we have used get line. So from the command line we are getting the input and storing it in string.
Then we are using the fork system call. So we will discuss fork as well in detail. What fork does is it creates a child process of the parent process. So it's something like a parent actually creates a child and it returns twice. So if PID process ID is 0 that means it's a child and if it's not that is it's a parent that is if PID is not 0. So in case of a shell the parent waits on the process ID of the child that is minus 1 means in PID it waits for all the children. It collects the return status in status. The child actually executes exeve command. So now if we see what is exeve.
exeve. So exeve is a family of functions which are in Linux. We will also look into those in later weeks. So the purpose of exeve family functions is it replaces the current running process with the new process. So in this case we have a program. It creates a child and the child is basically replaced by the cp process. So if we see here this is a cp. Parameters we are going to collect in params. So it will be test1 and test2 in this case here. And the exeve the child will be replaced by cp. So now there is no existence of child. This cp process will do whatever task it want to do.
And the parent is waiting. So now what happens is cp will copy file test1 to test2 with going into the kernel mode by calling the different libraries and so on. And then on the waitpid the original parent. What is parent? The parent is shell is waiting. And when it returns the cp process or the cp program is already executed. So this is how it happens. So this is software demo. So we are into program. So this is again just for demonstration. So how it happens is we are running in a while loop. Then using the get line we are taking the input. We are forking. And then in exeveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveveve so the path here if you see it's the path of the command then args is the arguments basically it's an array of the path and arguments and environment we can just provide on the runtime we have not taken anything so this is somehow how the shell actually works in this case we have assumed that we are taking CP as only one command but in actual shell it will have a switch statement which will fall back on to the actual which is passed to the shell now we are going to discuss what are the system calls so what are the normal calls and what are the system calls what's the difference between them so system call is on the call into the kernel that is performed using the interrupt call while if we write a normal function for example we have our main program int main then we create a function fun so that is a normal call it does not go into the kernel for processing further system call the speed of the system call is slow because it needs switching from user to the kernel space while the normal call speed is comparably faster system calls execute using the trap instruction while the normal call doesn't need the trap instruction to be executed so what is the working principle of the system call or how the system call basically works we look into this one now so first the user process makes a system call the trap instruction is basically called and control moves to the kernel space kernel reads the system call number from the register we'll just see it in detail now kernel finds the address of the system call handler that is we say ISR interrupt service routine and then it completes the work by calling the device driver before moving ahead let's understand that how the system call basically works so consider there is a user process which calls the read system call so read takes a file descriptor a buffer and number of bytes so when the user process calls this read system call how basically it goes into the kernel and what all things happen let's see that so at the very first step the user pushes the parameters onto the stack in the reverse order so here the parameters are pushed onto the stack in the reverse order so it will be FD then it will be the buffer then it will be n bytes then it places the system call number so every system has a system call number so let's suppose our system call number is 100 it's placed onto the register called RAX in the third step the user process calls the trap instruction and it moves into the kernel space so for gravity I'm taking it here now we are into the kernel space so the fourth step kernel reads the system call number from RAX and it finds it 100 kernel finds the address of system call handler so it finds so it has a mapping kernel maintains a mapping of a system call number 100 versus an address where it can find the ISR or the system call routine it finds the ISR then at the sixth step the kernel already have the device device driver registered it has the device driver registered it has the file operation structure registered with it so in a file operation structure there is a mapping that if a read system call comes then for the device driver we have to call a XYZ function something like this we can say this is the step 6 now kernel from this address calls the system call handler that is we can say a function and and the system call handler will call this XYZ function because finally what this read will do is reading from a socket or reading from a file or reading from a port so it will call read and in turn it will call XYZ and the now call goes to a device driver now device driver now device driver generates an interrupt and it sends to the interrupt controller number 9 now interrupt controller decides on the priority of the interrupt and it forwards it forwards to the hard disk or maybe wherever the socket belongs to and it reads the data data so now this copies onto the private buffer and then it is copied to the kernel buffer and finally it reaches the application application is here inside the application buffer so this is how basically a system call works so how we can reduce the time spent on a system call we can keep the socket or the networking information in the application code rather than in the kernel so system call basically wants the control to go from the user space to kernel space and then if it is a networking application doing a send receive or a waiting on a socket we will also see about socket we will also see about socket programming later in this course in that case it has to wait for that particular point of time even the processing of the network packets or the network goes through the network stack which is there in kernel so that takes time some of the new techniques like DP DK and there is one new technique also you So, in DPDK, we have the data plane on the application while kernel handles the control plane and the advantages of this is, in terms of a packet scaling, Intel DPDK claims that it can process 80 million packets per second on a lightweight server.
So, that's how we can bypass the kernel from processing the networking or the network packets and we can scale. Now, we are going to see that how the open system call works. So, this is the source code of the open system call. So, initially, we have included the header files, studio.h, fcntl, error number. Then, uni std is for the sleep. And then, we have used the open. So, open is a system call which is provided by kernel. So, main purpose of the open system call is to establish connection between file and the file descriptor.
So, here file descriptor is fd and the file is a.txt. We are opening the file in read only and the create mode. If file is not present, it will be created. A file descriptor is returned. If file descriptor is minus one, in that case, we will print the error. Otherwise, we are just returning zero. So, let me just comment this sleep here. It was for some other next purpose. Let me run open since I am using cpp as a file extension. I will use g++ and I will see that the a.txt is created.
Oh, let me remove a.txt and then let me run the a.out. So, there is no a.txt now and it generates fd and a file called a.txt is created. So, this is zero but the open system call created a file in which we can write. So, this is the end of session two, week one. While we will come back, we will see about the memory management and interrupts in the week one. Thank you.