Linux and its Collaboration Explained (Video)

I can recommend this video giving a quick intro to what Linux is and does today as well as how it's built, though there's probably not much new in there for regular readers.

The coming big 3.0 Kernel - First Impresions

After running the 3.0-rc2 for a few days I can say that I notice a higher responsiveness. I'm not sure right now what causes them, but something's definitely improved. On the downside my intel i965 chipset crashes even more. This started in 2.6.39 with video playback, but it's gotten worse now and includes switching to the framebuffer console and back - and a very hard freeze that is, too...

Using a Linux Driver for a Device that is not yet supported without recompiling

If you have  a new shiny webcam or some other device you want to use in Linux, here is a quick short guide how you can have it runnig within minutes with a bit of luck.

Always full of Surprises - Intel Linux Wireless Drivers

You always think that device support is very good in Linux for old drivers. You think that Intel wireless drivers are especially well supported. So an old intel wireless device should have eccelent support. But it turns out I was wrong in this case. Intel took from August 2010 until the end of Apri 2011 to fix a regression in their driver covering several devices introduced in 2.6.35, now in 2.6.39 it's fixed it seems.

Things you probably didn't know about the Kindle

As an outsider I had no idea that the kindle has a wide open linux root. You can install not only a port of FBreader to read epubs and other files, but there is also a port of mplayer to listen to music including internet radio via wifi, font patches, ssh, different screen savers and other things. I just wanted to test it for two weeks as you can in Europe with pretty much any mail ordered thing. But now I'll keep it!

Why Not Scale the CPU on Schedule?

Usually e.g. CPU frequency scaling works mostly by seeing how much of the CPU is utilized and then reacting by setting the frequency to a certain level to satisfy this need. But why not at least in addition introduce an interface for some software, e.g. media players, to inform the kernel up front how much CPU it will need. Then it the CPU will be available on time.

0.4 Watt Less During Audio Playback - (Updated:) Power Performance: Pulseaudio + Interrupt-Less Alsa

(Skip to the update)
Ok, so with some help from Pierre-Louis from Intel I've managed to get it working and do some performance/power tests. But let me start at the beginning: Recently, pulseaudio not only switched to a more power efficient (and otherwise) timing system, as far as I understand a callback API. It also provided the infrastructure to use ALSA devices without causing any interrupts ("period wakeup disabling"), so you CPU can stay longer in standby mode (e.g. "C6 residency"), saving you power and avoiding playback glitches at the same time. See here and here or more background information. With kernel 2.6.38 the first driver (snd-hda-intel) supports this infrastructure out of the box, the snd-hda-intel driver. This combination is what I tested for power efficiency...

The Linux Kernel Trojan Horse Gift

Linus describes how he prefers new features to be introduced to Linux:

In other words, every new crazy feature should be hidden in a nice solid "Trojan Horse" gift: something that looks _obviously_ good at first sight.

The fact that it may contain the germs for future features should be hidden so well that not only is it not used as an argument ("Hey, look at all those soldiers in that horse, imagine what you could do with them"), it should also not be obvious from the source code ("Look at all those hooks I sprinkled around, which aren't actually used by anything, but just imagine what you could do with them").
Linus

Pulseaudio Power Efficiency

Using pulseaudio in combination with alsa is already more power efficient and avoids many interrupts on current systems. In my experience the interrupts created by hda_intel decreased from 60-80 down to 6 per second. The sleep residency increased from about 3 ms to about 8 ms. This already makes a difference of about .25 Watts. I wonder if and how the difference will be with the new patches that went into 2.6.38, disabling even more interrupts.

Update: Also check how to save 0.4 Watts and more with pulseaudio and alsa.

Saving 2 W power by Making Sure Laptop_Mode oesn't destroy your HD

Laptop_mode is actually a great thing, as it activates a few powerful power saving modes in Linux, most importantly disk power saving and delayed writes. But if you use it with out of the box settings, there are two major bugs in Laptop Mode Tools, which could cause your disk to crash and are the reason it's normally disabled. Now let me show you how to work around them and easily save 2 W of power:

What can Governor Adjustments do for your Battery Mileage?

After my previous post about Android saving power with Linux governor tweaking, I did some governor tweak testing on my netbook with a tweaked phoronix test suide (pts) and came up with these results:

The first two show the power usage in Watts, the third shows the frequencies used during the test.

"Unusual" Linux Games

Linuxaria has a great list of fun, "different" Linux games, can't wait to try out some of them.

If you liked this article, you might want to subscribe for more.

Immediate Latency Improvement on Recent Linux Kernels

Check out this article on Webupd8. But I'm sure an easier version you can just install as a normal package will come out soon as well.

But it works excellently indeed. I can transcode now and I don't even notice any change in responsiveness.

Atom Power Usage Reduced with kernel 2.6.36

As expected, the intel_idle driver includes ATM-C4 and ATM-C6 CPU power saving modes in kernel 2.6.36, even if your BIOS doesn't announce them. Check it out!

I'm sure it'll lower your power consumption by at least 0.5W on average loads. Of course there might be no difference at all if your BIOS already supports them. But often the BIOS disables the states when the AC adapter is plugged in. They still work, though, when using intel_idle. This makes your netbook less power hungry and more importantly: quieter. For some netbooks this will definitely be a big difference in run time as well.

Let me know how it works for you in the comments.

Disabling Fsync in Laptop_Mode

There are several libraries that help you to disable fsync temporarily so your hard disk doesn't always spin up to save your notebooks battery. Of course that is not very flexible as you need to preload the libraries meaning you have to restart the programs.

Introducing (native!) µTorrent for Linux

If you've ever looked into the various programs available to download via bittorrent, you will probably have discovered µtorrent. Back then it was probably already tiny and full-featured at the same time. It even had customizations to better run inside WINE. But it didn't run natively on Linux. Of course that made the very low resource bittorrent client much more resource hungry. Now this all changed. (Screenshots inside)

Syncronized Folders across Windows and Linux - (Cloud-)Free Encrypted Cross-Platform Synchronisation with Unison

I always wanted to sync a few folders of my system across all my computers. E.g. the my scripts directory, where I constantly fiddle with my scripts and create new ones, my documents I always want to be able to have and edit everywhere, etc. I would never know where the most current version is without bothersome comparison of the dates. I would have to check every file and then copy the newer one. But that's now history, thanks to unison and my sync scripts.

What you need
The biggest problem is that you need a Linux server somewhere that is best constantly running and connected. I sync all my files with an online server. This is not a big deal because with SSH the connection is securely encrypted. Also due to compression and the fact that files are only transferred once, sync is pretty fast even with an ADSL connection. Of course this won't work for slow dial up connections in combination with large files. But as long as there are only small changes in certain files, even dial up might work for you. A Linux server is not really needed, but makes the setup much easier, which is why that's what I demonstrate here. And of course if you only want to sync two computers with each other, then there's no need for a constantly active server. It should work with Mac as a client or even server, but I don't own one so I can't help with that.

Downloads
You can get your unison downloads for Mac, Windows here and for Linux here (Ubuntu binaries i386 for 9.10 and 10.04 and more in this ppa. I recommend version 2.40.16, read below for why.

For Windows you also need to get Putty Agent (Pagent), Putty Puttygen and Plink get them at http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html. For Linux you also need openssh-server ssh-agent, ssh-keygen (in the openssh-client package) and keychain. You need to setup a working ssh connection with public key authentication for this to work without your interaction in the background. See here how to do this.

Bugs in the past
At the time I found it, over a year ago, it was still pretty buggy. It couldn't handle unicode, special characters or Umlauts properly. Especially between different system you'd end up with two differently named versions of a file unless you restricted yourself to normal a-z characters - very annoying.

But recently new version has come out, which fixes the problem. Now unison is not only a tool to synchronize flawlessly across different systems, it's also faster and prettier "GUI" than older versions and it still synchronized across encrypted ssh connections if you want it to. In my view this makes it the perfect tool for my needs. I've set it up to synchronize my laptop and my netbook with each other and my server system over ssh.

How it works
It's blazingly fast by only copying those parts of the files that have changed. And it doesn't transfer any files to check this. It runs both on the server and the client and only transfers the dates and hash coddes of the files via a secure compressed ssh connection. You can find out more about unison at LinuxJournal.

The easiest setup and the one I will show here is a star topology setup. This means there are several client systems synchronizing with one server. It works because unison knows which files are newer. The great side effect is that you automatically have a distributed backup on all your systems - the server and all clients of all your files.

The script runs automatically every X minutes in the background via cron on Linux (completely invisible) and is called by the Windows Task Manager on Windows (almost invisible).

Important to know
You need to use exactly the same version on all system you deploy it. There are easy to install packages for Debian and Ubuntu and precompiled versions for Windows. I haven't worked with OS X version yet, but they should work as well. I use a Linux server, Windows on the server side should be possible to set up but it's going to be much more difficult. Setting up the client on Windows is already non-trivial.

As you don't modify the same file on both systems at the same time before a server sync, then there won't be any problems. If you do, unison will not sync them. You need to call unison by itself and it will prompt you to chose how to deal with the situation.

The Linux version uses the symlink feature of Linux to let you configure which folders to sync on the go. In Windows this won't work, so all your folders you want to sync need to be inside one parent folder. I know no way around this, though the unison config file might have a solution somewhere...

Sync script:

#!/bin/bash

LOG="tee -a $HOME/logs/mysync2.log"
source $HOME/scripts/functions $HOME/scripts/variables

main ()
{
echo Starting at $(date)

# exit if running on battery
grep on-line /proc/acpi/ac_adapter/*/state >> /dev/null
if [ ! $? ]; # 1 if not "on-line"
then
echo Running on battery. Leaving now...
exit 0;
else
echo Running on AC Adapter.
fi

# use ssh-agent
source $HOME/.keychain/*-sh 2>&1
echo sock $SSH_AUTH_SOCK pid $SSH_AGENT_PID me $(whoami)
ssh-add -l || (echo "SSH Key not active\!"; ssh-add || exit 1)

cd $HOME

unison-2.40.16 sync -batch -maxbackups 2 ${@} 2>&1

echo Exiting at $(date)
}

main | $LOG

It checks and exits when the system is running on battery. To be able to work in the background, it uses ssh-agent. So you need to setup ssh key authentication.

Unison config in ~/.unison/sync.prf:

# Unison preferences file
root = /home/user/sync
root = ssh://user@syncserver:port/syncfolder
follow = Path *
ignore = Path {scripts/s3.sh}

This is a sample config file. It follows all symlinks in the $HOME/sync folder to syncfolder on syncserver via ssh. The ignore section shows you how to exclude certain files from synchronization.

Windows Setup
This should already work just fine in Linux. Now let's turn to the Windows client.

The Window setup, as you will see, it much more "fun". There are several steps you need: Putty Agent (Pagent), Putty Puttygen and Plink (see above for links) and then a few bash scripts.

You need to copy your private ssh key from your linux box and convert it to putty's format with puttygen. Then copy it into a safe folder. I have it in the same folder as the Unison.exe. Then setup an ssh connection to your server with Putty that uses the key and save it to profile name unisonssh. Copy the following bash files to your unison folder and adjust the path names accordingly.

Putty Agent.bat:

pageant.exe sshkey.ppk

This needs to be a script or otherwise it's not started in the right folder and won't find the key. Try it with the full path of the key file then. This must be linked into Autostart if your and will prompt you for your ssh key password if your key is password protected.

SSH Connect.bat

@plink -C -ssh -load unisonssh -i "C:\unison\sshkey.ppk" -l djtm -P 22 unison -server -auto

Change the path to where your ssh key lies and the port to your server's port if necessary.

Unison Sync.bat

C:
cd "C:\unison\"
unison2.4.exe -sshcmd sshconnect.bat -backups -backupdir unisonbackups -backuplocation central -batch -confirmbigdeletes -contactquietly

Testing it
Now is a good chance to try everything works fine. Try creating a file one one system and see how after two syncs it magically appears on the other. Edit it there to see how the edits are transferred back to the first system.

Scheduling it
Once everything works you can schedule it to run regularly without you needing to do anything. In Linux, you should install keychain for this to work in scripts and then enter the following into your cron. (To edit your cron type crontab -e)

*/15 * * * * $HOME/scripts/unisonsync

The */15 means to sync every 15 minutes. Don't worry, usually nothing will be done - I hardly ever notice anything happening. Of course the path should be where your unison sync script lies.

In Windows it's a tad more difficult again. The most difficult part of it all is to make it run in the background without annyoing you every 15 minutes or so. The following command looks a bit quirky, but that's the best I could do. Enter exactly this behind Execute: in the task scheduler.

cmd /C start /LOW /MIN "Unison Sync" "C:\unison\unison.bat"

and "C:\unison" next to execute in. Check both execute only when logged in and, under the settings tab, not to start the task if you're running on battery.

And now - finally - enjoy great, free, in sync folders!

Another older, less detailed guide is available here and here. Thanks to the authors of Unison.

New Low Power Record - Power Management Progress in 2.6.35

I just wanted to test how low the system can go in it's power usage with the power management framework of kernel 2.6.35 and the results amaze me. A test just showed that the system can save an additional 0.8 watts with the new kernel. Before the minimum battery usage without wifi was 5.6 W. Now in kernel 2.6.35 this has gone down another 0.8 W to 4.8 W - or 8.3 hours with just over 70 % battery. This is amazing. Even with wifi active the system now goes down to about 6 W - significantly increasing the amount of time you can be online surfing on battery. I wonder how much energy recent kernels save on other systems.

What Linux means for the Consumer - Drivers, Open Source and Support

When a consumer installs a Linux system, this has various consequences for him, which are sometimes hard to estimate at first. I've written a little summary touching the differences to Windows and Mac OS X in drivers, open source and support.

Drivers
Linux is not yet as far spread as Operating Systems like Windows or Mac OS X. This means that hardware corporations which have little money or skilled staff usually don't publish their own Linux driver and ship it along with the hardware when you buy it. In short: There currently is usually no driver disk for Linux.

But that's not really as a big a problem as it seems at first. One point is that - especially if the poor or skill lacking company is smart and publishes detailed product specifications - drivers are often written by the community. This has the disadvantage that it takes longer and when a device is first release, there is not usually no driver ready yet.

But it also has several advantages: Once drivers are written, they are usually open source. For the advantages, see below in my next point. And the drivers eventually are often better and more extensive than the drivers delivered by the hardware manufacturer. E.g. the bttv drivers of my TV card worked much better and longer than their Windows counterparts. And there was even an additional driver not present in Windows for a sound chip on the TV card to digitalize TV and line input sound up to a rate of 192 Khz -- for a TV card produced in 199X! My current sound card - using the snd-hda-intel driver - for some reason has a significantly better sound in Linux than in Windows, where it pretty much lacks any bass.

A huge advantage is the common API of drivers. In Windows every manufacturer writes his own drivers with their own API, and their own software to use the API. E.g. A DVB-T digital TV card has a Windows driver, and a special decoding and recording software for each and every single card. Sometimes the main chip and driver comes from the same manufacturer for several cards. But it's still often difficult if possible at all to use another software than the one supplied by the manufacturer.

In Linux, there is a common DVB interface. This means you can not only use the same software for all DVB-T receivers, you can even often use the same for DVB-T(errestrial), DVB-S(atellite), and DVB-C(abel). And it makes perfect sense that instead of tens of quickly written (and often buggy) different applications for different cards, you have a selection of a few really good programs. (E.g. Tvtime and others for analog, Kaffeine and others for digital reception -- and I mean for all supported ones).

In Windows, if there is a new group of popular hardware, like WIFI or Digital TV adapters, you need to wait for Microsoft to release a common API and hope they do. That's likely to happen only in the next Windows Version (e.g. XP, Vista). And then everything must be backwards compatible. This means that a better API might not be introduced for Windows XP ever, and advantages for new APIs are restricted by the requirement that all old APIs have to still work exactly the way they did. With open source drivers in the kernel tree, you can just modify the drivers along with the API. It's all in the same place and some changes are simply a matter of "search and replace".

Once a good open source driver exists, and especially if it gets integrated into one of the main driver projects, it usually works out of the box. This means you start Linux, even from a Live CD, and the hardware, e.g. a sound card, scanner or TV card just immediatelly works without any fuss like driver installation or configuration. It's not like installing Windows and then downloading the drivers for all your software, but if the driver is well supported, it just all works after installing the sytem. That's why Live CDs are so useful to test the Linux compatibility of a computer's hardware. And another great thing is that these integrated drivers are automatically updated for you - for Ubuntu even without the need to reboot (for other distributions you need to pay for this feature). This prevents security risks, which are much more rare in Linux, anyway.

Open Source
Open Source is a guarantee that a good product can have a long life-time. Drivers are one excellent example. If you have an open source driver, and it's of good quality and integrated into the Linux kernel or one of the main user land driver projects (SANE, CUPS, ALSA), you will have a driver for a very, very long time. This means that unlike with Microsoft and Mac OS X systems, you will not need to buy a new scanner, because you can't find a driver for Windows 7 or the "Snow Leopard", but quite the contrary the driver usually gets better and better and you often have no problems whatsoever with using a 10 years or older scanner, printer, sound card, TV card, etc with the newest Windows version.

My DVB-T adapter has lots of features it lacks in Windows, including recording up to 4 channels in one bouquet at once, 50 fps deinterlacing, hardware accelerated video decoding, automatic scheduled recording, etc. I just got a new sound card, which uses a lot less CPU and doesn't have a few of the Windows issues like skipping and beeping during playback. Even the pretty new Intel poulsbo chipset already works better for me in Linux now than in Windows (lower battery consumption, better video scaling and hardware accelerated video playback, much faster resume from suspend (feels like 1 second max, dmesg says 2.5 seconds; Windows takes a felt 4-15 seconds), ...). Thanks to Linux my Netbook is now a full featured surround sound FullHD video center and it consumes even less battery than in Windows.

But open source also means that things are customizable. This would of course also mean that if you can program yourself you can potentially modify and fix as many issues with software and drivers as your skills permit. And this is one of the main driving forces of Open Source. But it also means that if there is a problem, the fix can be applied where it makes the most sense. Let's make another example in comparison to Windows.

In Windows if there is a bug in the suspend to ram mechanism, which causes problems for a certain hardware, this is quite hard to figure out in the first place. Because you don't know how the Windows mechanism works. If it was publicly known, Linux would have a much easier way with ensuring suspend to ram compatibility for more hardware. Once you figure out where the problem is in Windows, you still have no way of fixing the problem where it occurs.

Because you can't modify any Windows components. That's completely different in Linux - here anyone can modify anything. I once wrote a small Linux kernel patch myself, and I'm really not that much of a programmer. But with the help of some kernel developers I fixed my own problem and then posted my first Linux kernel patch.

Another great example is powertop. It's a tool to check what decreases your battery time in Linux by looking into which applications wake up your CPU from power saving modes and how often. For this to work the kernel had to be modified. Once this was done you could find out which programs are the main cause. And then, because they are also usually open source, you could modify them as well so they don't wake up the processor as often. And that's how my netbook now consumes less power - in some scenarious a lot less - in Linux than in Windows. All this would not be possible in a closed source world. The same is now done for latency with latencytop btw.

Support
The disadvantage with support in Linux and Open Source is that if you don't pay for it, you won't get people who will treat you like you paid for it. This means it's very un-wise to contact driver or other software developers and say: "You've got to fix this!" Because they really, really don't have to. They are in no way obligated to you. You should always remember that. Only some people get paid for their work for open source at all. Many start a project and keep working it to scrach and itch (see here, and here). And I think it's how Linus started the Linux kernel project sometime long ago. If you want support that treats you like you've paid for it - get paid support!

But the big advantage in Linux support is that if you have a clue, you have access to incredibly smart people. You can talk directly to the developers and they usually know their stuff really, really well. Yes it can be a problem to get an answer sometimes, but if you've read the manual, you really know what you're doing, keep in mind that you didn't pay them and appropriately contact them very politely, you will be amazed how much is possible.

I still remember how helpful, friendly and patient the KDE plasma developers were when I wrote my first plasma-applet. I remember how the Linux kernel developers take every issue seriously and try to help you find the problem, especially with regressions.

And in my view it's so incredibly much better than most hardware manufacturers who you pay with buying their products to get support. You usually first get stuck with incompetent telephone support and are really lucky if you ever get to contact the driver developers or someone anywhere near them. And then those driver developers usually get their money for writing bleeding edge new drivers for new hardware, or they need to write a driver that's compatible with the newest Windows release (without really knowing how everything works on the Windows side).

They usually have little patience and understanding for a not working driver in some system and say that your supplier should fix the issues. (E.g. NVIDIA would say the graphics card manufacturer should fix the problem and Intel would say your Notebook manufacturer should deal with the problems.) But open source developers will usually understand you well, because that's often why they wrote the driver in the first place, so as many people can use it without problems as possible. Here the problem is usually not the lack of will, but of your politeness, your and their time and other resources.

Conclusion
You have to ensure that hardware is Linux compatible before you buy it. If you're lucky and you find open source driver integrated into the Linux kernel or one of the main driver projects, you are most likely to be able to really enjoy your hardware in Linux for a very, very, very long time and there will be no need miss out on the newest features and software coming for free with the newest release of your favorite distribution. If something doesn't work you're expected to do your part and google and try everything you can to fix it, including postings in a forum and on a user mailing list before contacting the developers. This way they have more time for and fun with actually fixing the problem.

But it also means that for not paying for Windows and the drivers, you can't expect the same support. If you want the same level of support, try offering money to the developers you want the support from (paypal helps) or buy a support service. I'm sure they're much more likely to help you in a way similar or better than Windows then.

I'm happy about your thoughts and comments. And remember I don't get paid for writing this either - as you can't even see ads on this page.

All trademarks belong to their respective owners, not to me.
(Windows, Windows XP and Windows Vista->Microsoft, Mac OS X -> Apple, Linux to Linus Torvalds etc.).

Must Read Article on the Issues of Ubuntu, Open Source and Linux

http://www.omgubuntu.co.uk/2010/06/many-hands-make-light-work-few-make-it.html