A stack overflow or buffer overflow is a program error which causes the buffer of a server or PC to overflow with excessively large amounts of data. This means that the stored data will not only land in the buffer, but can also overwrite memory locations behind it. A stack overflow is a common security vulnerability utilized by hackers to install malicious software. Correcting this issue is therefore part of network security.
Each computer software has variables stored locally for their runtime. There are also parameters for the transfer of functions as well as return addresses, which are stored in the working memory, specifically in the stack segment. In a current operating system, each software program is assigned its own address space, which is virtual and variable in size. When the program is started, three segments are created, a code segment, a data segment (heap), and the stack segment. The latter is used as a buffer during the execution of the program and is located at the upper end of the address space. If older data segments are to be retrieved, the stack must first be deleted.
This is the moment when a stack overflow can take place. For example, a program can fill a local variable in the buffer with content or a string that is larger than the available space within the variable. In this case, the return address can be overwritten, for example. A program crash may result.
The consequence of a stack overflow is that the program used, crashes as a result of incorrectly entered variables or because a return address contains no reachable target. In the case of an exploit, the attacker manages to overwrite the stack with their own code, thereby inserting this code in the return address. A commonly used method to introduce such code is to camouflage it as user input.
Further consequences of a buffer overflow are the corruption of stored data or its modification.
The programming language C is particularly prone to a stack overflow, since it cannot monitor the limits for memory management. Moreover, the programs which store program data and other data in the same memory space are particularly vulnerable. Languages like Java are resistant to stack overflow. Compilers can also be used as a precaution against possible buffer overflows.