GSoC2008 Stefan Announcements

From Syslinux Wiki
Jump to: navigation, search

5 September 2008 - Google Code Project for My Contributions

Google created a project page for each organization on their hosting service so the students could showcase their work done during this program. Thus, I have uploaded on my organization's project page an archive containing only my specific contributions (patches and source files). This wiki page on the Google Code project shows more information regarding how to get the archive contents or even browse it online.

19 August 2008 - Preview "Release" 2, Official GSoC Release Package

This second preview "release" of my work on dynamic loadable module support also marks the official deadline of Google Summer of Code 2008. GSoC rules state that only the work done before the deadline to be considered for evaluation and be sent to Google later in September, thus I have also tagged the current snapshot of the repository as a GSoC Release. However, as far as the module system integration is concerned, this snapshot remains only a preview, as the final version of the code will be found in SYSLINUX 4.

This section will describe the changes occurred since the first preview "release". For an overall description of the project please consult the announcement on 11 August.

What's new?

This preview release comes with a couple of changes/new features:

  • The dynamic memory allocation system in the klibc was modified to permit memory block tagging. That is, each memory allocation using malloc() or posix_memalign() result in an allocated block that also has associated a tag (a void* value). The value of the tag is inherited from a global tag variable (also called the current allocation context), and can also be set and retrieved separately for each block by special functions that were added to <stdlib.h>.
  • A new layer of abstraction has been added on top of the module API: the execution API. While the module API is concerned with module loading and linking, module memory management and dependency information, the execution API deals with running the code included in the modules, module life cycle and the rules of module storage and access. The execution API divides the modules into two categories:
    • dynamic libraries are modules that are loaded in memory and provide functions (services) to other modules. They can also be later removed from memory if no other module requires them. The dynamic libraries can have associated initialization and termination functions that are executed at appropriate times in the library life cycle.
    • programs are modules that contain a main function. They are loaded in memory, have their main function executed, then they are immediately unloaded. A particular aspect of program modules are that they create allocation contexts. Each time a program is loaded, before the main function is ran, the global tag variable in the memory manager is set to the module structure pointer, and it is reset to its original value after main() terminates. Then the memory manager is asked to clean up all the memory tagged with module descriptor pointer, thus implementing a basic mechanism of garbage collection. Note that dynamic libraries do not create allocation contexts, thus any memory request made by a function in a library is associated with the innermost running program in the call stack.
  • The root COM32 module was rewritten in order to implement a basic CLI interface with commands that load/unload libraries and execute programs. One could also inspect the current state of the modules loaded at any given time, such as size and dependency information. Please run the help command in the module CLI to see the full list of available commands.
  • The size of the root COM32 module has been reduced from 35K to 25K. This size includes services not required by the module itself, but that might be needed by other dynamic modules that would link to the root module. This size also does not reflect the size of the code that will get into the SYSLINUX core at integration. It is estimated that the size of the module system in the SYSLINUX core will be as low as 10K. For a more in-depth analysis of the COM32 module components, please consult this thread in the mailing list archive.
  • The hello.dyn module has been modified to be a program module, and the sort.dyn module became a dynamic library. Thus, in order to test the functionality of those modules, one should enter in the CLI:
load sort.dyn
spawn hello.dyn

How to get it?

You can download from here a package containing this preview release. You can also clone the repository at

git://git.zytor.com/users/stefanb/syslinux-elf.git

The preview release is labeled elflink-preview-2 and gsoc-release. You can also browse the repository online, through the web interface:

http://git.zytor.com/?p=users/stefanb/syslinux-elf.git;a=summary

If you want to test the package in a virtual environment, you might prefer to use this testing framework based on qemu. Details about how to use it can be found in the previous announcement on this page.

What's next?

The work done until so far is going to be integrated into SYSLINUX next major release. More details about the integration plans will be available in the SYSLINUX 4 roadmap, which will be published in the near future. Every status update from now will be made as part of SYSLINUX 4 development.

11 August 2008 - Preview "Release" 1

As I'm on the last mile with my Google Summer of Code 2008 project (only one week left until the final deadline), I considered appropriate to announce a preview "release" with my work so far. I invite everyone interested to take a look, test it and even provide feedback on the mailing list.

The project is not completely finalized, nor the documentation for it, as the code will find its final place in SYSLINUX 4.0, which is planned to be release later this year. However, the vast majority of the features and inner logic has been implemented, and a small testing framework was written. As the documentation isn't fully available, I'm briefly presenting below, in a FAQ manner, the most relevant aspects of my work so far.

What does the project currently offer?

In the current implementation, it offers a way to load and execute additional code (modules) on the fly, and also resolve automatically any symbol dependency. The modules are files conforming to the ELF format specifications, that are built and linked using the standard GNU toolchain. The modules are loaded at runtime using a special API implemented as an extension to the already existing klibc. This API is currently used by a testing COM32 module that loads additional modules specified on the command line, at runtime.

Note that this is only for testing purposes; in the long run (i.e. for the next major SYSLINUX release), the dynamic linking code will be integrated directly into the SYSLINUX core, without the need of an intermediate COM32 module.

What differs from the official source tree?

There are a coupe additions/modifications with respect to the current SYSLINUX release:

  • The klibc library contains in com32/lib/sys/module/ an implementation of an ELF objects loader and linker, together with its corresponding interface that can be included from <sys/module.h>. Please see module.h for an extensive presentation of the API and its structures.
  • The klibc build system has been modified in order to split the library in two parts: a part that gets integrated into the testing COM32 module, and a part that is loaded dynamically, when requested. Other modules can be dynamically linked against the symbols found in the COM32 module. klibc is generated into both the original libcom32.a version (for the rest of the COM32 modules), and libcom32min.a to be included in the testing COM32 module.
  • A directory containing the testing COM32 module has been created separately from the rest of the modules: com32/elflink. It contains another subdirectory, called com32/elflink/modules, that contains the available dynamic modules. Feel free to study the source code, test it and hack on it :)

I want to test it. How can I get it?

All my development was done in a separate source tree that can be found at this address:

http://git.zytor.com/?p=users/stefanb/syslinux-elf.git;a=summary

You can download from here a package with the source code and the build of the preview; you may as well clone the repository and look for the git tag of the preview:

git://git.zytor.com/users/stefanb/syslinux-elf.git

For your convenience, I have also adapted a testing framework originally written by hpa, which now includes support for testing the dynamic ELF modules. It can be found here: http://zytor.com/~stefanb/syslinux/elftest.tar.gz Brief instructions:

  1. Get the already packaged binary, or pull the source tree from my repository, and build it (make clean && make).
  2. Extract elftest.tar.gz under the root of the source tree, move into elftest/ and run gen.sh (you may need root privileges).
  3. Run the generated image using qemu: $ qemu -M pc hda.img

Note that after each modification of the source code, you will need both to recompile the source tree and regenerate the testing image using gen.sh.

The generated image will contain a COM32 module called test_com32 that is capable of loading additional code from ELF dynamic modules. The image has also a coupe of dynamic modules (*.dyn, located in the dyn/ directory in the image root directory):

  • hello.dyn - a simple "Hello world" module that sorts a couple of numbers using a quick sort function found in a separate module.
  • sort.dyn - a module offering a simple quick sort function.
  • klibc.dyn - a module containing some portions from the klibc library (image manipulation - PNG, JPG, support for compressed data).
  • _root_.dyn - a stub (shallow) module loaded automatically by the COM32 module that maps over the COM32 code and is used to link other dynamic modules against the COM32 functions and data.

test_com32 takes as parameters the list of dynamic modules to be loaded in the specified order. Examples of testing:

  • test_com32 hello.dyn (generates an error as hello.dyn depends on sort.dyn)
  • test_com32 hello.dyn sort.dyn (also generates an error as sort.dyn must be available when hello.dyn is loaded and linked)
  • test_com32 sort.dyn hello.dyn (works like a charm :) and should display the original vector and the sorted vector)

What next?

In the following days I'm planning to implement the following:

  • a layer of abstraction on top of the current module API that would permit loading of two class of modules: executable modules (which are loaded, executed their main functions, and then removed), and library modules (that are loaded, initialized and kept in memory in order to offer services to other libraries and programs loaded afterwards).
  • a way of tracing dynamic memory allocation (using malloc()) requests made by executable modules and automatically free the unallocated memory after the module exits.
  • a better separation of the two parts of klibc: the one that goes into the COM32 module, and the one that is dynamically loaded at request. The separation must be done with the goal of minimizing the COM32 module size, that is, include only a minimal amount of code and data.

I appreciate any feedback / bug reports / use case suggestions that would help me better design the module subsystem. Even if I won't have the time to implement all the features in the wishlist before the GSoC official deadline (this sunday), I will continue to maintain and improve the code afterwards.

I will also announce on the mailing list any notable improvement that I will do.

See also