Showing posts with label network. Show all posts
Showing posts with label network. Show all posts

Issues after Upgrade to Ubuntu 12.04 - Network Manager not starting and "Waiting for network configuration..." at boot

Both issues are related to your /etc/network/interfaces file. Comment out any automatic interface setup parameters you put in there and it should work fine. Otherwise, there are some other workarounds.

Learn the new Linux Network Commands

Doug Vitale wrote a wonderful summary of the new linux network commands on his blog. It includes typical commands as they were before as well as they replacements.

Copying Binary Files Via Telnet

Telnet really wasn't mean for copying at all, even less copying binary files. But you can do it with a fair amount of effort. You need two open terminals.
On Terminal 1:
1. telnet host > somefile.txt (then log in blind or look at tail -f somefile.txt)
2. cat somefile | busybox uuencode -m -
3. Once you see that you are logged in and the data stream is going just wait for it to finish.
4. Close Telnet (Ctrl-C, Ctrl-D)
Then once telnet is closed and you are back on your system enter:
5. busybox uudecode -o somefile.bin somefile.txt

There is no need to first remove the login protocol and other random shell noise, they are no problem. uudecode automatically detects where the binary data starts and ends if you do one file at a time.
Now you have your precious binary data, rejoice! ;-)

Saturating your Link with Lftp

I've been looking for a tool that downloads with several connections at the same time for a while, like GetRight for Windows used to be. Because without a tool like that, you can't really take advantage of fast connections with 20 and more Mbit. Now I've found a tool that does that and even surpasses my expectations: lftp. Lftp does not only support this, it supports the feature in http and ftp like many programs, but also in sftp, fish, ftps and https. And the usage is extremely simple: lftp pget url.

Eduroam with Linux (incl. Android!)

The guide my university supplied did not work after all with Linux. So I used an alternative guide, which also turned out to be working for Android:

Security: WPA2 Enterprise

Authentification Tunneled TLS (TTLS)

Anonymous Identity: none

CA-Certifikate: (none)

Inner Authentification: MSCHAP2

User: user@your-uni.com (as in the Windows settings guide of your home university)
password: your password, as in windows.

Syncronized Folders across Windows and Linux - (Cloud-)Free Encrypted Cross-Platform Synchronisation with Unison

I always wanted to sync a few folders of my system across all my computers. E.g. the my scripts directory, where I constantly fiddle with my scripts and create new ones, my documents I always want to be able to have and edit everywhere, etc. I would never know where the most current version is without bothersome comparison of the dates. I would have to check every file and then copy the newer one. But that's now history, thanks to unison and my sync scripts.

What you need
The biggest problem is that you need a Linux server somewhere that is best constantly running and connected. I sync all my files with an online server. This is not a big deal because with SSH the connection is securely encrypted. Also due to compression and the fact that files are only transferred once, sync is pretty fast even with an ADSL connection. Of course this won't work for slow dial up connections in combination with large files. But as long as there are only small changes in certain files, even dial up might work for you. A Linux server is not really needed, but makes the setup much easier, which is why that's what I demonstrate here. And of course if you only want to sync two computers with each other, then there's no need for a constantly active server. It should work with Mac as a client or even server, but I don't own one so I can't help with that.

Downloads
You can get your unison downloads for Mac, Windows here and for Linux here (Ubuntu binaries i386 for 9.10 and 10.04 and more in this ppa. I recommend version 2.40.16, read below for why.

For Windows you also need to get Putty Agent (Pagent), Putty Puttygen and Plink get them at http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html. For Linux you also need openssh-server ssh-agent, ssh-keygen (in the openssh-client package) and keychain. You need to setup a working ssh connection with public key authentication for this to work without your interaction in the background. See here how to do this.

Bugs in the past
At the time I found it, over a year ago, it was still pretty buggy. It couldn't handle unicode, special characters or Umlauts properly. Especially between different system you'd end up with two differently named versions of a file unless you restricted yourself to normal a-z characters - very annoying.

But recently new version has come out, which fixes the problem. Now unison is not only a tool to synchronize flawlessly across different systems, it's also faster and prettier "GUI" than older versions and it still synchronized across encrypted ssh connections if you want it to. In my view this makes it the perfect tool for my needs. I've set it up to synchronize my laptop and my netbook with each other and my server system over ssh.

How it works
It's blazingly fast by only copying those parts of the files that have changed. And it doesn't transfer any files to check this. It runs both on the server and the client and only transfers the dates and hash coddes of the files via a secure compressed ssh connection. You can find out more about unison at LinuxJournal.

The easiest setup and the one I will show here is a star topology setup. This means there are several client systems synchronizing with one server. It works because unison knows which files are newer. The great side effect is that you automatically have a distributed backup on all your systems - the server and all clients of all your files.

The script runs automatically every X minutes in the background via cron on Linux (completely invisible) and is called by the Windows Task Manager on Windows (almost invisible).

Important to know
You need to use exactly the same version on all system you deploy it. There are easy to install packages for Debian and Ubuntu and precompiled versions for Windows. I haven't worked with OS X version yet, but they should work as well. I use a Linux server, Windows on the server side should be possible to set up but it's going to be much more difficult. Setting up the client on Windows is already non-trivial.

As you don't modify the same file on both systems at the same time before a server sync, then there won't be any problems. If you do, unison will not sync them. You need to call unison by itself and it will prompt you to chose how to deal with the situation.

The Linux version uses the symlink feature of Linux to let you configure which folders to sync on the go. In Windows this won't work, so all your folders you want to sync need to be inside one parent folder. I know no way around this, though the unison config file might have a solution somewhere...

Sync script:
#!/bin/bash

LOG="tee -a $HOME/logs/mysync2.log"
source $HOME/scripts/functions $HOME/scripts/variables

main ()
{
echo Starting at $(date)

# exit if running on battery
grep on-line /proc/acpi/ac_adapter/*/state >> /dev/null
if [ ! $? ]; # 1 if not "on-line"
then
echo Running on battery. Leaving now...
exit 0;
else
echo Running on AC Adapter.
fi

# use ssh-agent
source $HOME/.keychain/*-sh 2>&1
echo sock $SSH_AUTH_SOCK pid $SSH_AGENT_PID me $(whoami)
ssh-add -l || (echo "SSH Key not active\!"; ssh-add || exit 1)

cd $HOME

unison-2.40.16 sync -batch -maxbackups 2 ${@} 2>&1

echo Exiting at $(date)
}

main | $LOG

It checks and exits when the system is running on battery. To be able to work in the background, it uses ssh-agent. So you need to setup ssh key authentication.

Unison config in ~/.unison/sync.prf:
# Unison preferences file
root = /home/user/sync
root = ssh://user@syncserver:port/syncfolder
follow = Path *
ignore = Path {scripts/s3.sh}

This is a sample config file. It follows all symlinks in the $HOME/sync folder to syncfolder on syncserver via ssh. The ignore section shows you how to exclude certain files from synchronization.

Windows Setup
This should already work just fine in Linux. Now let's turn to the Windows client.

The Window setup, as you will see, it much more "fun". There are several steps you need: Putty Agent (Pagent), Putty Puttygen and Plink (see above for links) and then a few bash scripts.

You need to copy your private ssh key from your linux box and convert it to putty's format with puttygen. Then copy it into a safe folder. I have it in the same folder as the Unison.exe. Then setup an ssh connection to your server with Putty that uses the key and save it to profile name unisonssh. Copy the following bash files to your unison folder and adjust the path names accordingly.

Putty Agent.bat:
pageant.exe sshkey.ppk
This needs to be a script or otherwise it's not started in the right folder and won't find the key. Try it with the full path of the key file then. This must be linked into Autostart if your and will prompt you for your ssh key password if your key is password protected.

SSH Connect.bat
@plink -C -ssh -load unisonssh -i "C:\unison\sshkey.ppk" -l djtm -P 22 unison -server -auto
Change the path to where your ssh key lies and the port to your server's port if necessary.

Unison Sync.bat
C:
cd "C:\unison\"
unison2.4.exe -sshcmd sshconnect.bat -backups -backupdir unisonbackups -backuplocation central -batch -confirmbigdeletes -contactquietly

Testing it
Now is a good chance to try everything works fine. Try creating a file one one system and see how after two syncs it magically appears on the other. Edit it there to see how the edits are transferred back to the first system.

Scheduling it
Once everything works you can schedule it to run regularly without you needing to do anything. In Linux, you should install keychain for this to work in scripts and then enter the following into your cron. (To edit your cron type crontab -e)
*/15 * * * * $HOME/scripts/unisonsync
The */15 means to sync every 15 minutes. Don't worry, usually nothing will be done - I hardly ever notice anything happening. Of course the path should be where your unison sync script lies.

In Windows it's a tad more difficult again. The most difficult part of it all is to make it run in the background without annyoing you every 15 minutes or so. The following command looks a bit quirky, but that's the best I could do. Enter exactly this behind Execute: in the task scheduler.
cmd /C start /LOW /MIN "Unison Sync" "C:\unison\unison.bat"
and "C:\unison" next to execute in. Check both execute only when logged in and, under the settings tab, not to start the task if you're running on battery.

And now - finally - enjoy great, free, in sync folders!

Another older, less detailed guide is available here and here. Thanks to the authors of Unison.

Fast, Cheap, Reliable VPN for Linux

I did some research into VPN services because even if I'm not in the US or UK for a while I don't want to be prevented from accessing certain websites there and signing up for services there.

You can sign up to the PureVPN shared IP services here. I have good experience (hulu with maximum quality streaming, iplayer work fine) with the cheapest version.

I recommend chosing the Monthly or Quarterly plan for tests ($14), but there is a three day test account ($3) available as well. It seems there is a bandwith limit now, and without you pay more.

Payment processing can take a few hours so be aware you can't start you fun and tests directly. Settings for network manager (gnome) can be found here. It works fine for me after installing network-manager-pptp-kde. Let me know in the comments if you need help. The most important part is to deselect everything in that one list except CHAP in the advanced settings and dselect MPPE. You can also try the nm-applet from Gnome, which you can of course also use in KDE. In my experience it often works better than the KDE version anyway unfortunately.

You may run into problems if your router doesn't support pptp forwarding. Then you either get a better router, or - in the long term probably the cheaper and better option for your speed - get a better router.

By signing up via a link from this blog you support the author.

Increase Speed and Save Traffic with Opera Turbo

If you have a traffic limitation I have some great advice: Install Opera.
Starting with Opera 10, there is the integrated "Opera Turbo". With the
help of that proxy you not only speed up the loading of websites by 4x and
more, but you also decrease the amount of data transfered, which makes
your volume last much longer.

After a whole day of 3G web surfing, Opera Turbo had saved me 2 GB of
traffic! (Only about 100 MB were left, that 20x less!) So the first thing
after setting up 3G should be setting up Opera with Opera Turbo!

Oh and you'll probably also want to put this list into your Opera urlfilter.ini. This stops Opera from loading ads, saving you even more traffic. But you can't use save as, you must use copy & paste to get rid of the html elements in the file.

Adjust Linux Network Receive Window for Fast Networks

Simply execute the following command in your shell to increase the receive window buffers. That should improve the performance, especially from servers with a high latency.
sudo sysctl -w \
net.core.wmem_max=1075200 \
net.core.rmem_max=1075200 \
net.ipv4.tcp_rmem="4096 87380 8388608" \
net.ipv4.tcp_wmem="4096 87380 8388608" \
net.core.rmem_default=1075200 \
net.core.wmem_default=1075200

Install Daily Fresh Ubuntu Directly over the Web

If you're doing a fresh installation and you don't have a CD downloaded yet, I can recommend the netboot installer. It let's you download and install the freshest packages from a mirror near you. All you need as preparation is e.g. a USB-Stick that boots into grub. Checkout this howto.

Basically all you need to do is copy the kernel "linux" and the initrd "initrd.gz" to your stick. Then boot them from Grub and you can start your installation.

The great thing about it: During the installation you can chose whether to install Kubuntu, Xubuntu, Ubuntu, etc. And you will automatically get the newest available packages right away. There's no need to fetch and install updates after the installation.

And getting the newest packages right away might save you a lot of trouble because of bad packages in the installation cd. And the network installer itself also gets updated regularly.

Bye Bye Privacy

The New Year begins with complete Connection Data Retention in Germany. This means e.g. your IP is logged every time you go online and many other things. The same happens more or less across the entire European Union. A sad day for citizen's rights such as privacy and anonymity.

Setting up NX

A tutorial how to set up the IMHO best remote desktop connection: NX.

WPA Encryption Cracked

If you haven't already read about it, it seems to be true: The Wifi encryption protocoll WPA has been compromised. The method will probably published pretty soon.

That means many people will need a new router and a new setup for it. And for a long time many networks will remain open again. If you want a long term solution it more and more looks like a VPN solution is the only way to go.

And if I'm not mistaken he who managed it is Erik Tews from the German University of Applied Sciences Darmstadt.

Okay, it turned out the hack is not as bad as thought. It will probably still take some time and combination with other approaches to really become useful. But I think this might help around the current restriction that you can only decypher clients that connected after you were "listening" with WPA.

A Tiny Backup Script with Rsync

This script lets you easily mirror a local directory to foreign computers that have rsync installed - securely via ssh. It assumes you have already set up a public key authentication with those servers. (Have a look at ssh-keygen and ssh-copy-id)
#!/bin/sh
# (c) 2008 linux-tipps.blogspot.com
#

FROM=/home/user
# the base path
DIRS="Desktop Documents"
# you can also use absolute paths

RSYNC="rsync -av --progress -e ssh"
# add -z for compression

ssh-add -l || ssh-add
# this adds ssh private keys if not added already
# for cron scripts use keychain instead:
# source ~/.keychain/*-sh
# see here for more

cd $FROM

$RSYNC -z $DIRS djtm@server1:/some/path/Backup/ &

$RSYNC $DIRS djtm@server2:/some/path/Backup/ &

wait && echo Sync Done. || echo Sync Failed or Aborted.

Find Bots in your LAN with BotHunter

BotHunter is a pretty cool software that helps you detect bots through passive network monitoring.

Their webpage has just gone offline. I'm sure that's because of all my readers visiting their web page right now. ;-)

Fixing Wake On LAN in Ubuntu 8.10

I've already spent a few hours to finally get this working again. There are several problems in Ubuntu 8.10 considering Wake on LAN:

1. The network cards' drivers are unloaded during suspend.

This often deactivates your card's wake on lan feature and completely powers it off.
You have to add your network cards driver module to the MODULES_WHITELIST in /etc/default/acpi-support. Also add the network card's interface(e.g. eth0) to SKIP_INTERFACES.

2. The network is not (properly) restarted after resume.

a. NetworkManager overrides settings in /etc/network/interfaces.

The easy way is to simply remove networkmanager: sudo aptitude remove network-manager.

b. The network setup script is not restarted after resume.

You can add the script (networking) to STOP_SERVICES in /etc/default/acpi-support.

3. Wake on Lan is not enabled by default

You need to issue e.g. sudo ethtool -s eth0 wol g. You can do that in an init script and you will likely have to do it again after resume. (I've put it a script to do that into both /etc/acpi/suspend.d/54-wol.sh and /etc/acpi/resume.d/99-wol.sh )

Broadcom Wireless Drivers - wl vs. b43 - Performance Comparison

I've compared the performance of both drivers with a simple iperf test over a few meters:

Broadcom Proprietary Driver:

TCP

------------------------------------------------------------
TCP window size: 85.3 KByte (default)  
------------------------------------------------------------
[ 4] local 192.168.178.24 port 5001 connected with 192.168.178.34 port 46362
[ 4] 0.0-10.2 sec 24.6 MBytes 20.2 Mbits/sec  
[ 5] local 192.168.178.24 port 5001 connected with 192.168.178.34 port 46363
[ 5] 0.0-10.2 sec 24.6 MBytes 20.2 Mbits/sec
[ 4] local 192.168.178.24 port 5001 connected with 192.168.178.34 port 46365
[ 4] 0.0-10.2 sec 24.0 MBytes 19.8 Mbits/sec

UDP
------------------------------------------------------------
Receiving 1470 byte datagrams
UDP buffer size: 375 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.178.24 port 5001 connected with 192.168.178.34 port 35141[ 3] 0.0-10.0 sec 24.2 MBytes 20.2 Mbits/sec 1.364 ms 0/17239 (0%)[ 3] 0.0-10.0 sec 1 datagrams received out-of-order[ 4] local 192.168.178.24 port 5001 connected with 192.168.178.34 port 38437[ 4] 0.0-10.0 sec 27.2 MBytes 22.8 Mbits/sec 1.837 ms 0/19418 (0%)[ 4] 0.0-10.0 sec 1 datagrams received out-of-order[ 3] local 192.168.178.24 port 5001 connected with 192.168.178.34 port 45356[ 3] 0.0-10.0 sec 28.8 MBytes 24.1 Mbits/sec 1.319 ms 0/20555 (0%)[ 3] 0.0-10.0 sec 1 datagrams received out-of-order
B43 Open Source Driver

TCP
------------------------------------------------------------
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.178.24 port 5001 connected with 192.168.178.34 port 42345
[ 4] 0.0-10.1 sec 19.6 MBytes 16.4 Mbits/sec
[ 5] local 192.168.178.24 port 5001 connected with 192.168.178.34 port 42346
[ 5] 0.0-10.2 sec 23.1 MBytes 19.0 Mbits/sec
[ 4] local 192.168.178.24 port 5001 connected with 192.168.178.34 port 42347
[ 4] 0.0-10.2 sec 21.7 MBytes 17.8 Mbits/sec

UDP

------------------------------------------------------------
Receiving 1470 byte datagrams
UDP buffer size: 375 KByte (default)
------------------------------------------------------------

[ 3] local 192.168.178.24 port 5001 connected with 192.168.178.34 port 57459
[ 3] 0.0-10.1 sec 14.0 MBytes 11.7 Mbits/sec 0.438 ms 152/10171 (1.5%)
[ 3] 0.0-10.1 sec 1 datagrams received out-of-order
[ 4] local 192.168.178.24 port 5001 connected with 192.168.178.34 port 55062
[ 4] 0.0-10.0 sec 15.7 MBytes 13.1 Mbits/sec 1.372 ms 205/11373 (1.8%)
[ 4] 0.0-10.0 sec 1 datagrams received out-of-order
[ 3] local 192.168.178.24 port 5001 connected with 192.168.178.34 port 49861
[ 3] 0.0-10.1 sec 23.4 MBytes 19.5 Mbits/sec 3.724 ms 292/17017 (1.7%)
[ 3] 0.0-10.1 sec 1 datagrams received out-of-order
[ 4] local 192.168.178.24 port 5001 connected with 192.168.178.34 port 42069
[ 4] 0.0-10.0 sec 21.5 MBytes 18.0 Mbits/sec 1.370 ms 239/15544 (1.5%)
[ 4] 0.0-10.0 sec 1 datagrams received out-of-order

As you can see, the proprietary driver is slightly faster in TCP mode. And somehow they're both really slow in UDP mode. That's something I'd like to find out more about.

Update: That was my fault. For some reason iperf restricts UDP tests to 1 Mbit/s. Now the results are more appropriate. The result is that the proprietary driver has a more stable connection speed.

But I think that's really not worth installing a proprietary driver. Broadcom really would better integrate the changes into the open source driver. Their driver is still relying on the deprecated wireless system anyway.

Update2: Further tests showed that, at least after a while of usage, the proprietary driver creates high network latencies. Ping times to the router went up from 2 ms to an average of 160ms (peaks of 500ms !).

Broadcom Proprietary Drivers with Linux 2.6.27

I've found a way to get the drivers to compile... I just downloaded the intrepid restricted modules package source from Ubuntu.

Then unpack, go to ubuntu-intrepid-lrm/ubuntu-restricted/broadcom
and make -C /lib/modules/`uname -r`/build M=`pwd`

They only work with TKIP for me, at least not with WPA2. (The open source b43 drivers does.)