CIS-4020 Homework #1: System Calls

Due: Thursday, September 1, 2016

In the text you should review/skim the introductory chapters 1 and 2 to familiarize yourself with the material they contain. Our first topic is system calls, covered in chapter 5. Some of the questions below require you to browse the kernel source code. You can use the cscope-linux command on DevBox.

Can a process accidentally try to invoke a system call that does not exist? If not, why is it impossible? If so, how could it happen and what does the kernel do in response?
The system call numbers assigned to each system call can, in effect, never be changed once they are assigned. Why not?
What does the ptrace system call do? If you consult online resources, give appropriate references in your answer.
In some systems it is possible for a kernel module to "hook" a system call by replacing the call's entry in the system call table with a pointer to a function provided by the module. That function could then do some extra processing before (or after) the original system call. There are a number of applications were this is useful (debugging, security, auditing), however Linux does not allow modules to hook system calls in this way.
1. Would it be possible to modify the Linux kernel to allow modules to hook system calls? If not, why not? If so, how would it be done?
2. Why do you think the Linux developers chose not to support this feature? Hint: Consider the case where two independent modules try to hook the same system call. Consider especially what might happen when the modules are unloaded. Be specific about the problem(s) that might arise.
Traditionally Unix has had a system call creat for creating new files and a separate system call open for opening existing files. The POSIX standard requires that open be able to also create files given appropriate flags, making creat technically unnecessary. In Linux how does creat differ from open?

Hint: Look up the implementation of creat in the kernel source using cscope. Because system calls are defined in a special way you should search for uses of SYSCALL_DEFINE2 and SYSCALL_DEFINE3 to find all system calls with two or three parameters respectively. Then search those lists manually for the calls you are interested in studying.