Google Summer of Code Mentors Summit 2011

Last week end from 21th to 23th October mentors meet in San Francisco / US to share our experiences in organising and mentoring students during Google Summer of Code 2011, talk about open source projects and of course get in contact with other teams and meet in person. The Mentors Summit was well organised and it was fun to be there. It was my first time in the south of North America. So chances where good to meet new people.

The VLC and FFMPEG people where interested in colour management. I tried to give them some idea, what colour managed applications need to know about media streams. It seems, that awareness rises about colour management in the open source video community, which is wonderful.

Two members of the Scribus team where present, Peter Linell and Malex. We could discuss quite some topics around distributions. It was great to meet them both.MountainView Google Summer of Code Mentors Summit 2011

The Open Source in Visual Effects was missed be me due to a swapping of rooms. Luckily Peter knew the OpenColorIO author and we could get in contact in a small four people meeting. We discussed Linux colour management, distribution and the relation of movie picture to ICC style colour management. Jeremy Selan pointed out the very difference of HDR scene referred imagery compared to the style of HDR which is typical in ICC workflows. I could convince Jeremy to look at the ICC floating point extension and it’s two open source implementations. He was interested and might work on implementing that in OCIO too. If that happens, it would be a great step toward joining movie with still graphics workflows. In the past the issues around round tripping and HDR handling had lead to the decision of movie studios to develop own colour management systems. OCIO supports already the export of colour correction tables to ICC profiles for use in photo and paint applications, which traditionally feature ICC workflows. Once the input side through ICC profiles is implemented in OCIO the same library could be used as a CMM inside ICC and movie workflows.

Thanks to Google and its OPSO team for sponsoring and organising a great event.SF_near_port_after_GSoC_summit

Colour Correction Concepts for Monitors

Colour management for the desktop is a long standing issue not only for Linux. The following text will concentrate on colour correction of monitors. This means the experience, when you switch your computer or handheld on and look on the screen.

Why colour correct the desktop?
People tend to compensate for quite a lot of different types of monitors. They are most often able to adapt to one full screen and see colours as they are intended to look like. That’s fine as long as they are concentrated on the visual event on that one display. But in this article we want to discuss how to compensate the monitor colours to our human visual needs in environments with various displays side by side, as is the typical situation for more and more people today. They take pictures with mobile phones or other digital cameras and look at them on laptops, tablets, picture frames, TV sets and printouts often side by side. Or we look at them over the internet and want to share the same visual impression with other people. Many people with uncorrected systems see quite a difference between various colour devices and try to figure out how to conveniently synchronise them. Colour management should help accomplish that task.

What is used today on Linux for colour improvements?
Different display devices feature different colour gamuts and colour appearance. We are especially sensible to the gray balance. This is a important property for our visual experience. Gray balance is since long time maintained by placing three linear correction curves into the graphics card. They are known as VCGT tag. Properly done, this helps in maintaining a equally stepped gradient from black to white with neutral shades in between. This calibration technic is fast and deployed since a long time by tools like xcalib. Calibration means in ICC terms, to setup a device driver in order to deliver a good and stable colour response. Strictly it is not part of the ICC specification. But for practical reasons it is usually embedded in ICC profiles. This basic calibration with linear curves is still in use on Windows, osX and X11 systems.

Acer Extensa 5630EZ laptop monitor

Different primary colours become more and more an issue through evolving LCD display technology. The above and below CIE shoe diagrams show quite different gamuts between the three primary colours red, green and blue of a typical laptop above and a wide gamut monitor below.

HP LP2481zx wide gamut monitor

It is generally desirable to use a screen, which is able to show deeply saturated colours. But uncorrected images show much shifted colour primaries to better cover human visible colours. Compared to traditional monitor primaries green typical becomes more cyan and red is moved toward the purple line. Of course the same RGB number triple looks quite different on booth devices. A calibration technic like VCGT linear per channel curves can not correct in any way for these saturation and hue shifts. Colours look strange not only on side by side comparisons of such devices. We need a colorimetric description of the monitor to properly colour correct the given device response.

Historically we have seen three approaches to cover the colour correction of the full screen.

  • X11 systems are colour management wise much like Windows. They predate ICC colour management and therefore have no historical APIs to enforce throughout colour correction. Maintainance of gray balance through VCGT is possible. The X11 XCMS extension was to our knowledge never in real use. Never APIs can bring a requirement for colour management. Few do today. Differences between corrected and non corrected content on one screen can be very irritating.
  • A very rigorous answer to the situation of missed colour management APIs is a fixed colour correction in hardware. But this can limit the capabilities of monitors to sRGB and thus the experience of wide gamut monitor technology not available. On the other side it provides a way to synchronise tightly controlled display devices.
  • On osX the whole rendering pipe line is colour managed. Each colour must have a colour space assigned to or it is not accepted by the systems graphic APIs. As a result all content from applications is converted on the fly by Apples ColorSync to the according monitor and looks correct. ColorSync takes over the work to bring colours from a source colour space like LStar-RGB to a native destination colour space like from a monitor. This helps programmers to concentrate on their actual task, which is most often not colour management related at all.

What is done so far to get the colour correction ball rolling?
In 2008 we started a project for OpenICC to do colour correction on the GPU on top of X11. The project allows to colour correct all non colour characterised desktop areas. Non colour characterised means by definition all window regions, except actively marked regions for opt out of system colour management. If no colour managed application is running, this means a full screen colour correction. These non colour characterised regions or areas are handled similar to other non colour characterised content. Such colours are considered to be in sRGB In the printing world for instance. sRGB is the internets default colour space. This approach is a combination of the above outlined ones and an answer to the historical X11 specifics. The advantage is, that legacy APIs and applications can benefit from a colour corrected desktop, while updated applications will be able to opt-out of colour correction. Thus a situation like on osX, with all content clearly characterised, can be reached gradually. During the project was a protocol designed for server client communication. The belonging spec is the X Color Management spec and is maintained by me over OpenICC. The spec was previously called net-color spec, but was renamed due to name space conflicts. For a smooth transition of legacy applications, the implementation of the spec in libXcm contains the xcm tool. This tool allows to set colour regions manually as a workaround. LibXcm covers just the communication protocol for applications and toolkits to talk to a colour server. The colour server, for instance CompICC, is responsible for doing the colour correction on GPU. If you want to add a colour server to your favorite compositing window manager or have questions around the spec, get in contact with me.

Can the X Color Management spec do compositing?
During a discussion on a Qt email list came out the question about compositing and colour correction. Most obviously the spec misses a freely selectable compositing colour space to do full blown up compositing. this would mean blurry region borders and other technics where transparency is involved. That is simply out of scope for the X Color Management spec. It is logical in the responsibility of the compositor to decide about the compositing colour space. Compositing window managers on Linux use today just the plain monitor colour space for their compositing. That is not perfect but seems to be good enough for simple transparency effects.  One correct way would be to meld the compositing of the window manager into the compositing of the client, which is in many cases the toolkit. But that is a bold adventure.taking the freedom of toolkits away and making traditionally modular stuff very monolithic. A architectural more sensible approach would be to use an intermediate blending space on top of the X Color Management spec. Thus the blending space is correctly selectable and modularity, interoperability and so on is much better. The toolkit can then rely on the compositing window manager to do all complicated transforms with the provided window and do colour correction as a final step. That would then essentially become a per window colour correction. From a today’s point of view the intermediate per window blending space needs a lot to do inside toolkits. It might take years to get there as toolkits just started about first steps. So the per region colour corrected desktop is still a valuable concept for today’s needs.

What is needed in the future?
Future developments have to take into account that the X11 architecture might change and other approaches become possible. The design of new architectures should cover colour management right from the beginning. With Wayland on the horizon and OpenGL-ES this is technically not a problem and would solve legacy issues like we see today in X11, on Windows and all major mobile platforms.

X Color Management 0.3 DRAFT1

The net-color spec from the libXcm repository is renamed. During a great conversation with Martin Gräßlin from KWin at oSC rwx³ he pointed out, that the _NET_ prefix is reserved inside the Xorg atom name space. So I decided to rename all atoms in the applications known to me, which use the net-color spec. The renaming went on fast and all apps work again as expected in git.

The new spec is called X Color Management. It contains the color regions in Xorg description and the used device profile for Linux desktop colour servers.
Affected are libXcm, Xcm, Oyranos, CompICC and CinePaint.

Oyranos @ rwx³

rwx³ alias openSUSE Conference (oSC) will be from 11.09. – 14.09.2011 in Nuremberg / Germany. Im going the to openSUSE Conference! The conference for openSUSE - and Free Software enthusiasts, September in Nuremberg, Germany. Are you? We will meet there and can discuss ICC colour management for the openSUSE distribution, KDE and Qt.

Sebastian Oliva will be there too. I hope we can hack together a Oyranos connection to his newly created ICC DB. The ICC DB project, done during GSoC 2011, shall be used to search for ICC profiles by terms of colour device configurations. This means a printer can obtain a fitting ICC profile for a special driver without the need to have all the canned profiles packaged. Independent vendors can easily upload their ICC data and get their optimised profile selected automatically – if all works. Sebastian and Joseph Simon have done fair bits to get there in a clean way, without hacking the whole system.

I will have colour measurement equipment with me and we can create new profiles for actual used gear, like laptop monitors. Of course these shall walk into the ICC DB.

It would be cool to meet with Compiz and KWin maintainers/packagers to get the GPU colour managed desktop project further. The net-color-spec appears pretty robust. But we need a critical mass to support the concept of colour correcting all windows on the desktop in compositing window managers.

Will be great to meet you all.

Oyranos Colour Management LiveCD II

After new second try, the driver license problem appears somewhat more relaxed. One important component needs good OpenGL support for plug and play full desktop colour correction. First choice for the OpenGL API on Nvidia and Ati hardware are the proprietary drivers from these manufacturers. For a LiveCD this did not work out due to legal reasons. The new CD can offer some basic OpenGL support to run Compiz and GPU accelerated colour conversions. The more and improving open source Nouveau driver comes to the rescue. Together with the experimental Mesa DRI it provides shader support. Some aspects work even better than the proprietary drivers, like backlight, obtaining monitor infos in a standard way through XRandR and automatic driver selection by Xorg. If you want to run on a daily base consider the Nvidia driver, as that provides power saving. It’s simply cooler.

There are some more changes like placing Krita on the CD. Krita is colour management wise a very interesting project. It supports floating point HDR images, comes with two own colour transformation modules in OpenGTL. The other new application is RawStudio. It implements the DCP spec of Adobe for DNG colour profiles. Both add to a very interesting colour software suite.

To read more about the LiveCD, please look at the old blog entry.

Oyranos Colour Management LiveCD

On OpenICC’s download area (edited on 16. February) on SourceForge is now a CD Live media available for 64-bit computers. It contains many colour management tools as available from openSUSE’s Build Service.
* littleCMS – widely used colour converter
* ArgyllCMS -1.3.0, dispcalGUI – cross platform colour management
* Oyranos – colour management system
* Compiz ICC colour server – or short CompIcc
* kolor manager – in KDE’s system settings panel
* ICC Examin – profile viewer
* Xcm/QCmsEvents – Xorg colour management event observer
* CinePaint – with net-color and other patches
* Scribus 1.3.8 – Layoutprogramm
* Cmyktool, Photoprint – imaging software
* SampleICC, IccXML – ICC sample implementation
* Nouveau/radeonhd + Mesa 3D drivers (edited 16. February)
… and more.

The CD should start on not too old nvidia graphics card hardware. Other systems are currently not support due to the requirement of a stable OpenGL driver with GPU Shader support for Compiz and CompIcc.

Once the live media runs, the desktop should appear colour managed. The trayicon, with the little horse shoe in it, should be coloured to show the colour server is correcting the the desktop. CompIcc is colour managing each monitor separately and acts on hotplug appropriately. Currently is no monitor ICC profile pre installed on the CD. So it must be generated on the fly. The colorimetry data comes from the monitor itself and contains the colour primaries, a white point and a single gamma value. This is enough to let strange primaries appear more natural, or detect a wide gamut monitor and compensate for its possibly very strong saturation.

kolor-manager device profile selection

To change the monitor profile one can use kolor-manager from KDE’s systemsettings panel. It contains as well policies and default profile selection. These settings are stored in a per user database. To see that CompIcc is working one might select the CIE*XYZ profile, with its headroom and gamma of 1.0 the monitor appearance should change dramatically. But thats only visible when the “Show only device related ICC profiles” box is deactivated.

On the desktop are three example images just for having some wide gamut media available. The two tiff files are raw camera developed images with a custom ICC profile assigned. The restaurant JPEG is in AdobeRGB as typical for some cameras. All are tagged with the respective profiles and can be loaded into the installed image tools. PhotoPrint is a very sensible application and Scribus of course. To show the whole image gamut on a wide gamut monitor only CinePaint can communicate with CompIcc to get a own hole in the screen to colour correct to the native screen colours. All other applications see sRGB as monitor colour space. Thats visible by again assigning a CIE*XYZ while CinePaint has opened a image. It will not be affected as it does not check monitor profile switching.

The advantages of complete desktop colour correct are:
* wallpapers and movies look like indented
* shopping via internet is more reliable colour wise
* content on different brand of monitor look more uniform
* wide gamut displays become useable
* non colour managed applications fall back reasonable to sRGB

Hope you can start the media. For ATI cards one further version is planed and maybe for other cards, like intel graphics, a separate version can follow.

sRGB = monitor?

CompIcc sees sRGB as its document colour space. Obviously setting sRGB as monitors device colour space would result in no colour conversion for untagged content. Conversion from one ICC profile to the same ICC profile does not alter the colour numbers nor their interpretation.

Currently untagged content is the majority of graphics on the Linux desktop, at least from the perspective of a colour server. The sRGB == sRGB solution is a real hack to skip colour corrections. This hack is too easily in conflict with the desktop colour space. And the right way is still to tag window regions as already prematched. The later method is a clear path to omit any colour correction in the context of a colour server. After fiddling around in CompIcc to support setting sRGB as a monitor colour space I gave up on that idea. It is simply too fragile and difficult to implement.

The newly released CompIcc with its monitor hotplug support appears much more stable without the sRGB hack.

GPU Graphics Processing

It is pretty normal that graphics applications use GPU power for their image processing needs. The graphics cards where primarily designed to better support that kind of processing and offload computational power requirements from the main processor the CPU.

GPU computing came not over night. It was first introduced for boosting 2D applications. Simple desktop operations like moving a regions like a window or scaling and help in fast video decoding where first targets. Somewhat later came support for 3D operations. The later is now widely used through OpenGL APIs. The capabilities of the graphic cards increased and OpenGL was extended to support specialised programming on the GPU. The fragment and vertex shaders appeared. Programming of them is in a assembler language. So do not expect much convenience reading, while looking at the assembler code. However, for hard coded graphic operations it plays quite nicely. Compiz, for instance, uses this language for its desktop effects. Long times the assembler code was only available through the proprietary OpenGL drivers from ATI, Nvidia or Intel. Now the open source drivers begin to mature in supporting GPU programming. So it is possible to run Compiz with newer versions of the Nouveau or Radeon drivers. These two types of shaders, vertex and fragment, where joined and later a more convenient expression form for shaders was specified through GLSL. GLSL is OpenGL Shader Language. Newer OpenGL API’s support programming of them in a more straight forward manner. GLSL is officially part of OpenGL since version 2.0.

The C-like GLSL language is much more pleasing at least to most developers eyes. So it is possible to use this language by non specialised programmers to do fast computing on the GPU. If you understand and can write C, imagine you create a graphics filter of your own and process your graphics data through OpenGL directly on the GPU. Its fast and slick and you do not have to wait for a programming hero to convert your favorite algorithm for the GPU to run reasonably fast. But its works only on GPU’s. So either there is a software fallback in case your driver does not support GLSL or you are limited to a subset of hardware and drivers. Thats not so good for sharing and wide spreading your nice ideas.

The Kronos Group, which is the formal body behind OpenGL, specified the new OpenCL standard to improve portability and allow for much more flexible programming. OpenCL has, like GLSL, a C-alike syntax. It is a subset of C with some extensions. OpenCL is now adopted by osX SL and available as vendor driver from Nvidia. If you read adopted by Apple’s osX, you guess right, it is as well available for phones and tablet computers. The AMD Stream SDK contains a OpenCL implementation for CPUs. OpenCL makes the GPU now scriptable with a easy fallback to CPU. Of course thats all very young and up to date exist no open source implementation.

ICC colour management in Xorg 2

After some talk with Xorg developers, they argued that needing a second layer for the colour correction would cause trouble for the missing path of higher precision than 8bpc into Xorg. This seems at least not a problem for compiz. Using OpenGL the desktop is placed as textures into the graphics buffer and the colour correction can happen on the GPU, right before putting on cable.

The next argument against ICC in Xorg came with a additional read/modify/write cycle and its inherent power and time consumption.

Last but not least synchronisation is not easier with colour conversions running here and something drawing the pixels there.

So it seems most appropriate to keep colour management inside the compositing WM, improve the specs and put some more useful code snippets into libXcm.

(edited september 10, 2010 at 15:46)

ICC colour management in Xorg

The CompIcc plugin is already a great tool for mixed multi monitor setups. However compiz is in few distributions the default window manager. So it would make sense for the Linux desktop to implement the net-color spec inside Xorg.

Hey, hey, you might ask now, Why do I need this compositing toy called compiz now in Xorg and what brings me that ICC stuff? The answer is simple. Look at Apple computers. They show most likely the colours the designer has intended. And at least basic stuff works. You shurely want one a small gamut monitor to see the same skin tones like on a wide gamut monitor. After all its the photo or picture or drawing you are interested in. A grayish photo from a baby on a small laptop monitor or a grand daddy with highly orange saturated face on a wide gamut screen is surely not intuitive. Supposed your camera works good enough, and most digitals are really good, its possible to exchange the photos from the one monitor to an other monitor with different sized gamut. There shall be not that high difference as we see on todays Linux desktops.

Is’nt ICC colour management expensive? No, not necessarily. The very basic colorimetric information comes straight from the monitor. It is embedded in a small binary block known as EDID. A ICC profile can be generated on the fly from this data block. E.g. Oyranos supports that as a fall back. So, ICC colour management can be made instant and needs no extra purchase of expensive equipment to improve the colour rendering.

What is needed in Xorg to get ICC support in? A layer, which sees the window regions defined by the net-color spec and which can use hardware shaders. Then it needs to put the monitors each on a own texture, apply the window regions and let the shader run on the GPU. The net-colour spec has some more requirements. But these should be manageable.

Why in Xorg? Its a technical question. The main reason is that each compositing window manager needs to implement the net-color spec. That is lots of work and you know there are lots of window managers for Xorg. Beside that repeated work, not all window managers will naturally be prepared to work with GPU shaders. From a user point of view its desirable that capable hardware is deployed for colour correction without the need of configuration. So it makes sense to have the colour management implemetation in one place. Of course there needs to happen some discussion with the Xorg people about the concept and the details in advance.

Will higher bit depth and different channels, e.g. Cmyk, be supported? The net-color spec does not talk about that. It works with whatever pixel storage format is provided. E.g. In compiz it is 16-bit integers ARGB.

Will a true HDR format not be better and simpler to be implemented? Let me ask back how long shall we wait for that to happen?

What weaknesses are actual in the net-color spec? It can not define a custom colour transform per monitor. However by putting the colour management into the application it is still possible to gain more control over the colour rendering process. As well, defining a document colour space per region is not yet supported but easy to extent and I guess to implement. The classical use case for that would be a video stream with a attached ICC profile. Currently the content is assumed to be sRGB only or the application has to colour manage itself.