Showing posts with label netbooks. Show all posts
Showing posts with label netbooks. Show all posts

Syncronized Folders across Windows and Linux - (Cloud-)Free Encrypted Cross-Platform Synchronisation with Unison

I always wanted to sync a few folders of my system across all my computers. E.g. the my scripts directory, where I constantly fiddle with my scripts and create new ones, my documents I always want to be able to have and edit everywhere, etc. I would never know where the most current version is without bothersome comparison of the dates. I would have to check every file and then copy the newer one. But that's now history, thanks to unison and my sync scripts.

What you need
The biggest problem is that you need a Linux server somewhere that is best constantly running and connected. I sync all my files with an online server. This is not a big deal because with SSH the connection is securely encrypted. Also due to compression and the fact that files are only transferred once, sync is pretty fast even with an ADSL connection. Of course this won't work for slow dial up connections in combination with large files. But as long as there are only small changes in certain files, even dial up might work for you. A Linux server is not really needed, but makes the setup much easier, which is why that's what I demonstrate here. And of course if you only want to sync two computers with each other, then there's no need for a constantly active server. It should work with Mac as a client or even server, but I don't own one so I can't help with that.

Downloads
You can get your unison downloads for Mac, Windows here and for Linux here (Ubuntu binaries i386 for 9.10 and 10.04 and more in this ppa. I recommend version 2.40.16, read below for why.

For Windows you also need to get Putty Agent (Pagent), Putty Puttygen and Plink get them at http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html. For Linux you also need openssh-server ssh-agent, ssh-keygen (in the openssh-client package) and keychain. You need to setup a working ssh connection with public key authentication for this to work without your interaction in the background. See here how to do this.

Bugs in the past
At the time I found it, over a year ago, it was still pretty buggy. It couldn't handle unicode, special characters or Umlauts properly. Especially between different system you'd end up with two differently named versions of a file unless you restricted yourself to normal a-z characters - very annoying.

But recently new version has come out, which fixes the problem. Now unison is not only a tool to synchronize flawlessly across different systems, it's also faster and prettier "GUI" than older versions and it still synchronized across encrypted ssh connections if you want it to. In my view this makes it the perfect tool for my needs. I've set it up to synchronize my laptop and my netbook with each other and my server system over ssh.

How it works
It's blazingly fast by only copying those parts of the files that have changed. And it doesn't transfer any files to check this. It runs both on the server and the client and only transfers the dates and hash coddes of the files via a secure compressed ssh connection. You can find out more about unison at LinuxJournal.

The easiest setup and the one I will show here is a star topology setup. This means there are several client systems synchronizing with one server. It works because unison knows which files are newer. The great side effect is that you automatically have a distributed backup on all your systems - the server and all clients of all your files.

The script runs automatically every X minutes in the background via cron on Linux (completely invisible) and is called by the Windows Task Manager on Windows (almost invisible).

Important to know
You need to use exactly the same version on all system you deploy it. There are easy to install packages for Debian and Ubuntu and precompiled versions for Windows. I haven't worked with OS X version yet, but they should work as well. I use a Linux server, Windows on the server side should be possible to set up but it's going to be much more difficult. Setting up the client on Windows is already non-trivial.

As you don't modify the same file on both systems at the same time before a server sync, then there won't be any problems. If you do, unison will not sync them. You need to call unison by itself and it will prompt you to chose how to deal with the situation.

The Linux version uses the symlink feature of Linux to let you configure which folders to sync on the go. In Windows this won't work, so all your folders you want to sync need to be inside one parent folder. I know no way around this, though the unison config file might have a solution somewhere...

Sync script:
#!/bin/bash

LOG="tee -a $HOME/logs/mysync2.log"
source $HOME/scripts/functions $HOME/scripts/variables

main ()
{
echo Starting at $(date)

# exit if running on battery
grep on-line /proc/acpi/ac_adapter/*/state >> /dev/null
if [ ! $? ]; # 1 if not "on-line"
then
echo Running on battery. Leaving now...
exit 0;
else
echo Running on AC Adapter.
fi

# use ssh-agent
source $HOME/.keychain/*-sh 2>&1
echo sock $SSH_AUTH_SOCK pid $SSH_AGENT_PID me $(whoami)
ssh-add -l || (echo "SSH Key not active\!"; ssh-add || exit 1)

cd $HOME

unison-2.40.16 sync -batch -maxbackups 2 ${@} 2>&1

echo Exiting at $(date)
}

main | $LOG

It checks and exits when the system is running on battery. To be able to work in the background, it uses ssh-agent. So you need to setup ssh key authentication.

Unison config in ~/.unison/sync.prf:
# Unison preferences file
root = /home/user/sync
root = ssh://user@syncserver:port/syncfolder
follow = Path *
ignore = Path {scripts/s3.sh}

This is a sample config file. It follows all symlinks in the $HOME/sync folder to syncfolder on syncserver via ssh. The ignore section shows you how to exclude certain files from synchronization.

Windows Setup
This should already work just fine in Linux. Now let's turn to the Windows client.

The Window setup, as you will see, it much more "fun". There are several steps you need: Putty Agent (Pagent), Putty Puttygen and Plink (see above for links) and then a few bash scripts.

You need to copy your private ssh key from your linux box and convert it to putty's format with puttygen. Then copy it into a safe folder. I have it in the same folder as the Unison.exe. Then setup an ssh connection to your server with Putty that uses the key and save it to profile name unisonssh. Copy the following bash files to your unison folder and adjust the path names accordingly.

Putty Agent.bat:
pageant.exe sshkey.ppk
This needs to be a script or otherwise it's not started in the right folder and won't find the key. Try it with the full path of the key file then. This must be linked into Autostart if your and will prompt you for your ssh key password if your key is password protected.

SSH Connect.bat
@plink -C -ssh -load unisonssh -i "C:\unison\sshkey.ppk" -l djtm -P 22 unison -server -auto
Change the path to where your ssh key lies and the port to your server's port if necessary.

Unison Sync.bat
C:
cd "C:\unison\"
unison2.4.exe -sshcmd sshconnect.bat -backups -backupdir unisonbackups -backuplocation central -batch -confirmbigdeletes -contactquietly

Testing it
Now is a good chance to try everything works fine. Try creating a file one one system and see how after two syncs it magically appears on the other. Edit it there to see how the edits are transferred back to the first system.

Scheduling it
Once everything works you can schedule it to run regularly without you needing to do anything. In Linux, you should install keychain for this to work in scripts and then enter the following into your cron. (To edit your cron type crontab -e)
*/15 * * * * $HOME/scripts/unisonsync
The */15 means to sync every 15 minutes. Don't worry, usually nothing will be done - I hardly ever notice anything happening. Of course the path should be where your unison sync script lies.

In Windows it's a tad more difficult again. The most difficult part of it all is to make it run in the background without annyoing you every 15 minutes or so. The following command looks a bit quirky, but that's the best I could do. Enter exactly this behind Execute: in the task scheduler.
cmd /C start /LOW /MIN "Unison Sync" "C:\unison\unison.bat"
and "C:\unison" next to execute in. Check both execute only when logged in and, under the settings tab, not to start the task if you're running on battery.

And now - finally - enjoy great, free, in sync folders!

Another older, less detailed guide is available here and here. Thanks to the authors of Unison.

Moblin 2.1 IVI FC Release tested on the MSI Wind U110 (GMA 500/Poulsbo)

Phoronix hasn't gotten to it yet, but now I've done a quick test with the Moblin distribution IVI version with supposed GMA 500 support (moblin-2.1-preview-ivi-20091208.img) to see how well it's working on my netbook and what feature advantages its IEGD driver may have over the psb driver.

Well, it's not a lot:
  • The resolution was set to 800x600. No other resolutions were detected (xrandr -q)
  • The colors were distorted. It looked like 256 colors but randomly exchanged.
  • The touchpad was not working. The mouse moved in seemingly opposite or random directions and kept skipping back and forth. I had to use an external mouse to do anything at all.
  • The sound wasn't working, thought that might possibly be fixed by disabling power saving, just like in Ubuntu
  • Wifi wasn't working. I could not add networks or configure the wifi. " Join other Network" was greyed out. The iwconfig tools have to be started manually from /sbin. They found a network, but I didn't bother to try to set it up with wpa_supplicant manually. A path is not set. wpa_supplicant was already running with 99% cpu usage when I boot.
  • Switching to the console and back is impossible. It completely and irreversibly crashes the graphics until a reboot.
  • ACPI showed tons of errors in dmesg.
  • Trying to suspend crashed the system.
  • I wonder if there's an application in the distribution that can take advantage of VAAPI accelerated video decoding. There is a video player. But I had no easy access to the hard drive's partitions (ntfs).
The distribution as tested on my netbook is in two words:
utterly unusable.

The psb driver set up according to my howto works flawlessly compared to that! It interesting how difficult it seems to be for Intel to get its own chipset to work in its own distribution with its own driver.

But, yes. Moblin was very snappy. ;-) (I think that just shows just how cool XFCE is, though.)

Over 9 Hours of Video on Battery with the MSI Wind U110

After I've managed to set up the video acceleration, it was time to test if it's really as power efficient as claimed. I just tried that with some SD and HD video. HD video wouldn't play properly of course, so it's silly to compare battery times.

But with SD video and vaapi, I managed to get down to 6.1 watts (9.1 hours). Without vaapi it was around 6.3-6.5 watts (8.5 hours).

I used a few tricks to achieve this:
  1. disabled wifi and bluetooth.
  2. lowered the display brightness to minum (still good though). This has to be done before boot in 2.6.31.
  3. moved the system to an SDHC card. While the power in standby is the same (5.7 watts both with only SDHC or only hard disk), the active power is much lower with the SDHC (only +0.5-2 watts instead of 3-5 watts).
  4. used by powersave script to optimize the kernel settings for low power mode:
echo cpu
echo 1 | tee /sys/devices/system/cpu/sched_smt_power_savings
echo 1 | tee /sys/devices/system/cpu/cpu*/cpufreq/ondemand/ignore_nice_load
echo 95 | tee /sys/devices/system/cpu/cpu*/cpufreq/ondemand/up_threshold
echo 50 | tee /sys/devices/system/cpu/cpu0/cpufreq/ondemand/powersave_bias
echo

echo usb
echo 2 | tee /sys/module/usbcore/parameters/autosuspend
echo auto | tee /sys/bus/usb/devices/*/power/level
echo

echo misc
echo 1500 | tee /proc/sys/vm/dirty_writeback_centisecs
echo 0 | tee /sys/block/sdb/queue/rotational
# broken echo 10 | tee /sys/module/snd_hda_intel/parameters/power_save
# not better echo 1 | tee /sys/module/psb/parameters/disable_vsync

killall -q hald-addon-storage knotify4 NetworkManager
hciconfig hci0 down

MSI Wind U110 Linux Review

The MSI Wind U110 netbook has an excellent basic design: The new generation Intel hardware is made for long battery life and the MSI can really make it over 9 hours under Windows. The MSI's are one of the few netbooks of this generation which include a nice and bright non-glossy display.

The keyboard is excellent and can from my experience be used immediately without any issues. I actually like it better than my external keyboard. The only problem here was the tiny "'" key. But that's only important for English text and I'm sure in the version they sell in English speaking countries the keyboard doesn't have that flaw.

So they only thing that might be seen as negative is the lower display resolution compared to some other new netbooks that have an 11.6" display. But their display is usually glossy and thus disqualified for really using it anywhere outside and also of course costs battery time.

But for Linux it looks much worse. While the Poulsbo chipset was already roughly supported in Ubuntu 8.04, which was released over a year ago, the support for Poulsbo is still at least flaky. It needs binary blobs to work anywhere near well as far as I have read.

And unfortunately it's not yet working on the MSI Wind U110 or U115 netbooks. You can normally get the chipset to work pretty well - with the appropriate resolution and at least some 2D acceleration, but no hardware video acceleration (aka VDPAU or VA-API).

But on the MSI's the support isn't working yet. You get the message that the there was a null pointer dereference when trying to load the psb kernel modules that is "necessary for the driver to work" as the Xorg driver tells you. Though I've read it might actually work on a certain SuSe distribution.

That unfortunately means that the netbook can not currently be used for Linux. A small display combined with a bad resolution will definitely lead to headaches. Linux users will either have to use Windows until sometime - hopefully - Intel fixes it's drivers (and maybe even includes a working VA-API support!).

What else I've checked worked fine: Sound, USB, Brightness Adjustments. I haven't tested standby enough yet, but I think there were still problems. But that may very well also be related to the graphics chipset. And as far as I've read the standby to ram works fine at least with certain BIOS versions.

The MSI U110 comes with one RAM slot, which held 1 GB ram in this case and is extensible to 2 GB. There is of course an internal card reader included. A microphone and a headset jack and a USB port, a VGA and a 100 Mbit LAN port are on the right side, two more USB ports, the energy plug, a kensington lock connection and a fan on the left side.

Summary: For now only for Linux experts who are either willing to debug and fix the kernel module or figure out how to get SuSe running on it or people who are willing to use Windows until the problems are fixed (supposedly that might happen by the end of this year). But it's a great netbook with lots of battery time, a nice display and a great keyboard. Perfect for writing blog entries or a large word document.

Once the graphics work, I'll try installing and running Linux from an SDHC card to see it that saves even more power. But right now with the vesa driver the power savings aren't that good. I wonder if they are better with the real driver.

What works&What doesn't
  • Grafics (works pretty well now!)
  • Sounds works (needs a fix in Ubuntu 9.10)
  • SD-Card Adapter works (it's generally not very fast, though)
  • Standby (works in Ubuntu 9.10)
  • CPU low power mode (works for me, reportedly doesn't work in certain configurations)
  • Grafics low power mode (?)
  • Sound low power mode (creates problems and had to be deactivated in 9.10)
  • Webcam works with uvcvideo after pressing the short cut (Fn-F6).
  • Ethernet works in 9.10 after plugging in the cable (ASIX AX88772 USB 2.0 Ethernet).
  • Wifi works out of the box in 9.10. Needs additional drivers in 9.04.
If there are any questions feel free to ask.

UPDATE: X video works nicely now. Finally!!! :)

I even managed to get to work vaapi hardware video decoding up to FullHD 1080i.

Sound does not always work without problems. In Karmic 9.10 you have to comment out
#options snd-hda-intel power_save=10 power_save_controller=N
in /etc/modprobe.d/alsa-base.conf.
options snd-hda-intel position_fix=1 bdl_pos_adj=64
instead helped me as well.

Suspend to RAM works fine with newer kernels from Ubuntu 9.10 and the psb module active. But I think it crashed when I tried to use vaapi after that. It did. But that's fixed with the newest BIOS version (the most current one on 01. June 2010) for me now.

I've run powertop on a full battery(half a year old) to test how low I can get the power usage in X with it.

PowerTOP version 1.11 (C) 2007 Intel Corporation

Cn Avg residency P-states (frequencies)
C0 (cpu running) ( 1.8%) 1.60 Ghz 0.4%
C0 0.0ms ( 0.0%) 1333 Mhz 0.2%
C1 mwait 0.0ms ( 0.0%) 1067 Mhz 0.6%
C2 mwait 0.2ms ( 0.0%) 800 Mhz 98.7%
C4 mwait 0.3ms ( 0.0%)
C6 mwait 9.6ms (98.1%)

Wakeups-from-idle per second : 103.4 interval: 15.0s
Power usage (ACPI estimate): 5.6W (10.3 hours)

Top causes for wakeups:
53.7% (126.9) : acpi
26.4% ( 62.3) : psb@pci:0000:00:02.0
11.2% ( 26.4) : hrtimer_start_range_ns (tick_sched_timer)
2.3% ( 5.5) : Rescheduling interrupts
1.8% ( 4.3) : extra timer interrupt
1.3% ( 3.1) : hrtimer_start (tick_sched_timer)

I think the psb driver is not highly optimized for low power consumption yet, but the biggest problem is the pretty high number of "acpi" interrupts. We can see that with neither ethernet nor wifi active, the netbook can reach over 10 hours of run time. And this is with a battery that already lost 10% capacity. We should be able to get this down a lot more by finding the cause for the acpi interrupts and with some optimizations, mostly in psb.