Useful Scripts, Part 2: Linux "file sorter" script

The following Linux-based script (which you can download here) is also one I find myself using on occasion. I've made versions of it that run in just about every OS I've used at one time or another. It's a script that takes a large directory full of files, creates 37 subdirectories (A-Z, 0-9 and "other") and moves all the files in the folder it is run into their lettered folder according to their first character.

This is great when you have a large folder full of files that takes forever to "ls" or access over a network or a folder that is unmanageable for quickly finding particular files in a GUI. It's case-insensitive (I'm sorry, but DOS got this bit right, case-sensitivity can be a real hindrance at times, no matter how careful and experienced you are) so "Afile" and "anotherfile" both get put into the a/ folder.

I've used this on emulator rom folders when they got too many files in them, I've used it (or a Windows-"port" of this script) in user-profile folders on servers to help narrow the users down a bit and even move portions of them to a different storage medium in a logical manner (it's better to move A-L to another drive than it is to move the first 1000 users, from a "I'm only human" point of view). I've used it in archives of zip downloads, photographs, instruction manuals, all sorts. I hope someone else finds it useful.

#!/bin/sh
mkdir other
for LETTER in a b c d e f g h i j k l m n o p q r s t u v w x y z
do
        mkdir $LETTER
        FILESPEC=$LETTER\*
        UCASEFILESPEC=`echo $LETTER | tr a-z A-Z`*
        mv ./$FILESPEC ./$LETTER/
        mv ./$UCASEFILESPEC ./$LETTER/
done
for NUMBER in 0 1 2 3 4 5 6 7 8 9
do
        mkdir $NUMBER
        FILESPEC=$NUMBER\*
        mv ./$FILESPEC ./$NUMBER/
done
mv ./* ./other/

Useful Scripts, Part 1: Linux "screen" wrapper script

A few people in the past have asked me for a few of the hundreds of scripts that I have made for myself. My life is a one long session of script/batchfile writing (I use the terms interchangably because a list of commands to run is a list of commands to run no matter what language, OS, etc. it is written for). There are school networks that are running on some of my scripts!

I'm not so keen on letting the larger, more tedious scripts out without me doing a proper licensing thing on them (open-source of course) but the little ones that are just convenient wrappers etc. I don't mind giving away. They are so trivial you can hardly claim copyright on them, even if the only intention of that copyright is to give them away but ensure you get recognition. Anyway, here's the first one, for Linux. It's a wrapper for the "screen" command. You can download the script from here.

Screen is a powerful command. It's ALMOST as good as when you first use a Unix-based machine and are taught to use Virtual Consoles (pressing Alt-F1, Alt-F2 etc. lets you have a variety of commandlines open at the same time even if you aren't in a GUI). Screen lets you run tasks in the background, with their output hidden, in such a way that you can "resume" or connect back to the screen that the program is displaying, from anywhere. It's a kind of VNC for the console. So, for example, if you want to run a program like IPTraf or HTop, but want to be able to see its "screen" from an SSH session later when you're at work, you can do this.

The script itself is quite simple, and I have a habit of calling it my "do offline" script, so I normally name it do_offline.sh:

#!/bin/sh
if [ -z $1 ]
then
        echo "First arg is screen name, rest is command to run"
fi
if [ ! -z $1 ]
then
        SCREEN_NAME=$1
        shift
        echo "Running commands ($*) under screen name $SCREEN_NAME"
        screen -A -m -d -S $SCREEN_NAME nice $*
fi

Basically, you run it as:

do_offline.sh "screenname" "program and arguments..."

e.g.

do_offline.sh htop htop

do_offline.sh lare_file_copy cp -R *.* /mnt/backup/

do_offline.sh kernel_compile make

And it will happily run those commands in the background while putting the output into an invisible "screen". This is useful if you are logged in via SSH... rather than having to fudge access to the F1-F12 consoles you can start jobs in their own screens and then disconnect and reconnect from their screens as necessary without interrupting the job. You can start an 50Gb download remotely and not have to interrupt it when you disconnect your SSH session (the program will just carry on running in its screen and you can "resume" the screen from the local console when you get home to see if it finished).

To resume the screens, you just use:

 screen -r 

e.g.

screen -r kernel_compile

And the one annoyance... to "disconnect" from a screen without interrupting the program its running, just press Ctrl-A and then D. Horrible keyboard shortcut.

For example,

# do_offline.sh htop htop
# do_offline.sh kernel_download wget 
http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.24.3.tar.bz2
# screen -r kernel

Is it done yet? Nope?

Ctrl-A  D
# cd /

Go off and do something else, disconnect from the computer, go to work and log in to this computer remotely.

# screen -r kernel_download
There is no screen to be resumed matching kernel_download.

So it's finally finished and we can compile it. But I want the computer to have compiled it by the time I get home

# do_offline.sh kernel_compile make

Go home, log back into the machine at the local console

# screen -r kernel_compile

It shows up as still compiling, so I just watch the progress for a few minutes and then..

Ctrl-A  D
# screen -r htop

Have a look at the CPU usage, wait for it to dip

Ctrl-A D
# screen -r kernel_compile
There is no screen to be resumed matching kernel_compile.

My kernel is downloaded and compiled without me having to watch its progress, limit it's output to a single virtual console on a single physical machine, or keep the remote session connected constantly while it does it.

I hope it's useful to someone. It's already been useful to me and to at least a couple of people on the Linux Questions forum.

Essential Linux Utilities

Ever since setting up three Linux PC's in a row, I've realised that I've grown dependent on a few pieces of software for Linux, above and beyond what comes with a standard distro (or, at least, Slackware).

Beep - a tiny util that can beep the PC speaker in a variety of ways, perfect for headless systems. I use it to give a warning tones inside boot scripts and also to provide a rising or falling tone on the start or end of certain tasks, such as booting or shutting down. Because it uses the PC speaker, it doesn't interfere with ALSA, works on even the oldest of PC's, doesn't necessarily require an external set of speakers etc. Beware using it, however, on multi-user installations - I tend to keep it restricted to the audio group of users only to stop people messing about with it.

Ether-wake (available from various places, originally by Donald Becker) - the ultimate power-saving util... this is a Wake-on-LAN packet broadcaster to wake up computers that support WoL from their deep sleep (i.e. turn them on so long as they are plugged into the net and have a power cable in them). With this I keep my home network largely turned off and "wake up" (i.e. turn on) particular PC's as and when I need them. And larger scale experiments have shown that there's nothing better than the sound of a room full of PC's all booting up simultaneously at the click of a single button / cron job.

HTop - a better version of "top" that I find easier to use. Shows processes and RAM usage in a nice controllable text-mode GUI that allows you to kill individual processes, scroll up and down etc.

rc.firewall (See this post for a mirror) - a perfect, simple, one-file iptables firewall that works well as rc.firewall in Slackware. Works for single computers, NAT'ing routers, multiple network cards, multiple-networks-on-a-single-card, and lots of other configurations. It uses a simple syntax for even multi-port port-forwards, has many simple options for various things such as allowing or deny ping's or cross-network traffic, has a very strong default configuration and can be reloaded at the drop of a hat at which point all the detected network interfaces are re-firewalled.

x11vnc - This is one of those utilities that few people ever use. It's a vnc server for X. But it has a vital difference... it's a VNC server for EXISTING X sessions. Most people are familiar with xVNC which allows you to spawn an entire X-Windows system where each "screen" is actually a VNC session (thereby providing instant-VNC-thin-client) but that's not much use to someone that has a single-user Linux PC who wants to log onto their home PC and click on that link that they left showing in their browser. x11VNC does just that - the command-lines get horrid very quickly, you have to pay close attention to the security of the thing (because now connecting to the PC on port 5900 is the equivalent of logging in as yourself on the local PC!) but it's a great piece of software. The author is also working hard to make VNC-wrapped-in-SSH a cinch, even from Windows PC's, by extending the TightVNC clients to incorporate SSL tunnelling. Yeah, you can now do this with some things like KDE's Remote Desktop functionality but I've been using this particular utility for so long that I have scripts which build-on to it and it also has some features that just aren't present in other imitators.

knockd - a simple port-knocking daemon implementation which can be triggered remotely using either a tiny utility that works on Linux/Unix/Windows or by simpler tools such as telnet. Perfect for securing a server for remote access (and incidentally the best way to stop random port probes to your machine - my SSH logs were filling up until I found this) as you can just put the portknock client on a usb disk or a website and download it from wherever you happen to be or you can even "bodge" one in a real emergency. Also, the configuration basically consists of port-sequences and names of scripts to run. This means that it's easy to configure it to see port-hits on ports X,Y,Z as an instruction to run an "open" script and then you can hit ports Z,Y,X to run a "close" script. And because you can have multiple port sequences running, it's very easy to have all sorts of different things happening. See my article here for a bit more background on my use of this utility.

Mirror of Projectfiles.com / lfw.sourceforge.net rc.firewall

Having just completed a set of instructions for a group of Linux newbies on how to set up a firewall, I then discovered that my favourite Linux iptables firewall script has all-but gone from the Internet. I checked Google, both "official" websites (including the Freshmeat.net mirror) and archive.org. Still no joy. Luckily I had kept a copy of this GPL script, which I have mirrored.

For those people who have had trouble finding the script that's been hosted at both ProjectFiles.com and http://lfw.sourceforge.net you can download the rc.firewall script at the following address:

http://www.ledow.org.uk/linux/

This is the 2.0 "final" version. I have the documentation mirrored too. Oh, and I assume that the reason that the archive.org site has no mirror is that the author wants no more to do with it. So be polite if you do need to contact them (the above file has their email address etc.) and don't bother me for support, either! (You probably couldn't afford me!).

Making your own Firewall / Router / Fileserver / Print Server

As I have just done the above numerous times for myself and my brother, I thought that a little writeup might help. This is not designed to be complete or simple, but is for someone with a basic grasp of Linux and/or an advanced grasp of PC's in general.

I assume several things. You have a local Windows network. You have ADSL and an ADSL router that is connected to this local network. You know the basics of TCP/IP and are comfortable with command-line interfaces. You have a spare machine that has nothing of any value on it (e.g. Windows, documents etc.) that can safely be wiped and made to become a firewall/router/server. You have a spare parallel port laser printer (e.g. an HP Laserjet) that you want shared over the network. You know how to use Google and look things up for yourself if this article doesn't cover your problem.

First, get a computer and network it. I happen to have a source for old Pentium machines (233's) but for my brother's project, I gave him an old P400 with 64Mb RAM, two PCI network cards and an 80Gb hard drive - smaller harddrives are fine, down to about 4Gb, and may actually be more compatible with a machine with an old BIOS. 2Gb is an absolute minimum but it's then a struggle to keep up to date with such little free space.

[Author's Note: Yes, I'm being lazy and installing EVERYTHING from Slackware, I know perfectly well that you can do this from a single floppy but that's not the point. See Freesco if you want to be an expert and do it properly... I ran a Freesco install for about 4 years, making it do all of the above.]

The PC could really have done with a bit more RAM but it runs fine with just that much. Plug it into your network. At first, you can just add it onto an existing network by plugging into the hub/switch, to make installation easier. Later, we'll turn it into a router that can be put between your broadband connection and your computers to keep them much safer than they would normally be.

Next, blank the harddrive completely (or just be aware that you will be wiping everything on it, even Windows or whatever else is on there) and download Slackware. You can use any recent version but for me the version was 10.1. Download the first two ISO disk images and burn them to CD. CD Burner XP Pro is good for putting them onto CD if you don't have Nero or anything else to use, is free and runs fine under Windows 98 despite the name.

Next, boot the machine intended to be a firewall, file server etc. from the first CD. Follow the instructions until you get to the login prompt. Login as root and you'll be left hanging on a command line. Type cfdisk and delete any partitions on your hard drive (THIS WILL DELETE ALL YOUR DATA ON THAT HARD DRIVE, OBVIOUSLY). Then install one large partition over the whole disk and make it bootable. Exit out of cfdisk and reboot.

Boot back off of the CD again and login, then type setup. Go through the setup process (selecting a keyboard layout etc.), select the partition you just made to install into, don't set swap space, use the default kernel and make sure you install all packages, full install.

After a few hours, you should be done and rebooting without the CD in will take you straight into Slackware Linux. You would have been asked to set a root password. It's a bloody good idea and you'll need it to log in to this computer from then on. Now we have to set up your network card in order to connect to the internet and the rest of the network.

Login as root and type dmesg | more and scroll through, looking for anything that indicates that Slackware has found a network card. If it is a PCI or even an onboard card, it should find it without any assistance. Non-PNP ISA cards need a bit of a nudge beyond the scope of this article (Google isapnp).

If your network cards were successfully detected, type pico /etc/rc.d/rc.inet1.conf. This will allow you set IP's, network mask and (if need be) a gateway address (e.g. of an adsl router or any other device that all Internet-bound traffic should be directed through). You should set one card to an internal network address (e.g. 192.168.0.1 or 10.0.0.1) and the other card to an address which will be "external" (or enable DHCP for that card).

[For an example, my ADSL router gives out addresses in the 10.0.2.x / 255.255.255.0 range, which my server picks up as it's "external" address. The local network uses 10.0.1.x / 255.255.255.0 which the server has an address set for on its other card.]

Hit Ctrl-X when you're done.

After a reboot, both cards should now appear in the output of ifconfig as eth0 and eth1 (lo is a loopback interface that's always there, network or not).

Test your connection by ping'ing a website, ping www.example.com. This should come back with replies. Hit Ctrl-C to stop the pinging.

If you have two cards and the ping doesn't work, make sure you try this with a cable in both cards... sometimes the cards are chosen in a different order to what you might expect. Also, try pinging the address of other machines on the network, your ADSL router etc.

If you have a succesful website ping, it means that the Slackware machine is now able to connect to the internet. Now we can download a little program that will help us later on. Personally, I like Projectfiles.com's firewall... a simple script that enables all sorts of things without the complications of a GUI.

Type lynx http://projectfiles.com/firewall/. (Lynx is a text-based web browser that is simple enough to download files should you need them). Select the link for the latest stable release and hit d for download. Let it download it and then save it. It should have saved it in the home directory for root. Exit out of Lynx.

Copy it to /etc/rc.d (cp rc.firewall /etc/rc.d/)
Edit it to suit your network (pico /etc/rc.d/rc.firewall) using the output from ifconfig to remember which card (eth0 or eth1) has which IP address etc. There is a line in the file that tells you below which point you should not make any changes.

Make sure that you add the local network interface (e.g. eth0) to the list of internal interfaces. Any other network cards are considered external, i.e. anything that comes through them is NOT trusted, as if it were a direct connection to the Internet.

Press Ctrl-X when you are done.

Now, to turn the firewall on, you type chmod +x /etc/rc.d/rc.firewall and it will run at the next reboot. To turn it off, do the same with a -x instead and reboot. If you have done it correctly, you should get no errors and still be able to access the internet with ping etc.

While we're here, we need to turn a few other things on and off:

chmod +x /etc/rc.d/rc.dnsmasq
chmod +x /etc/rc.d/rc.samba
chmod +x /etc/rc.d/rc.cups

For these to work, however, they need some configuring. If in doubt, google for the name of the program or config file (e.g. dnsmasq.conf) and read their documentation.

DNSMasq
=======

This program forwards DNS lookup requests (i.e. resolving www.example.com to an address the computer can use) to an upstream DNS server. This is usually the same machine as your gateway if you have an ADSL connection (i.e. the IP address of your ADSL router) or it can be your ISP's DNS address (they will have these written down somewhere on their website but you may need to login to their website first... usually they can be found inside their instructions for connecting other types of computer to the internet, e.g. Mac, Linux, Playstation, XBox etc.)

Mostly, it needs no configuration at all but any can be done using pico /etc/dnsmasq.conf. It will get the IP of the upstream DNS server from /etc/resolv.conf which you can edit using pico too.

If this works properly, you should be able to set the other computers in your network to use your server's LOCAL ip address as a gateway and DNS server. E.g.

Internet (some address which your ISP has given you)
|
|
ADSL Router (10.0.2.1 / 255.255.255.0 and providing DHCP in that range)
|
|
10.0.2.100 / 255.255.255.0 on eth0
Server
10.0.1.1 /255.255.255.0 on eth1
|
|
Windows machines
10.0.1.x /255.255.255.0 using 10.0.1.1 as their gateway and DNS.

With this setup, the Windows machines should be able to connect to the internet as if the server wasn't even there. A quick trip to GRC to run their ShieldsUp! test should show all ports as Stealth, though.

That's routing, firewall and DNS-forwarding configured already!

Samba
=====

Samba is the piece of software that will allow you to produce network shares, which in Windows will appear as shares under Network Neighbourhood. pico /etc/samba/smb.conf to set it up. Generally, you should set the workgroup to the same for all computers on your local network, which you can do for your other computers using the settings inside Windows Control Panel. I use something like HOME or OFFICE for this so, for example, all the machines in the office have a workgroup of OFFICE. Also, for simple purposes, security=shareis fine.

You should also set a name for the server, which can be anything you like but which is probably better off as "server" or something simple. With just a name and workgroup configured, saving the file and rebooting (all this rebooting is *completely* unnecessary under Linux but it's quicker to type... :-)) the machine should appear under Network Neighbourhood of any Windows machine on the local network with the same workgroup.

If not, check whether you can ping the server from one of the Windows machines (type ping followed by the internal IP address in a command prompt) and also try disabling the firewall on the server (chmod -x /etc/rc.d/rc.firewall). If disabling the firewall works, check your firewall config... all machines on the local network should be able to have unrestricted access to the server if they are on a card which is listed in rc.firewall as being an internal interface.

Also, check any Windows firewall software you might have. Zonealarm, for instance, will sometimes through a wobbler when it sees file-sharing protocol traffic from the server and this may make it block the server... in this case sometimes the internet will work for five minutes or so and then stop. Add the server IP address as a "Trusted" IP address in Zonealarm on the client machines to fix this.

Now the hard part: read through the shares part at the bottom of the /etc/samba/smb.conf file and comment out any you aren't going to explicitly use (except for the printers one, leave that one there) and then make your own, following their example. If you want to create a basic, empty share using the hard drive space that you have left over on the server, that you can store stuff on from your Windows machines, it's best to do this:

Type mkdir /mnt/share. This makes a directory for us to use.
Type chown nobody:nogroup /mnt/share. This gives anyone permission to use that directory.
Type chmod +r +w /mnt/share. This lets anyone who has permission to read from and write to that directory.

Add the following share to smb.conf:-

[storage]
browseable = yes
comment = General storage
guest ok = yes
path = /mnt/share
read only = no

The share should now show up and be writable (i.e. you can save files to it) from the Windows machines.

If you install more hard drives into this machine, you can incorporate them into this share by putting a line for them into your /etc/fstab and making the directory inside /mnt/share.

An example fstab might be:

#
# /etc/fstab
#
#

/dev/hda1 / ext2 defaults 1 1
/dev/hdb1 /mnt/share/another ext2 defaults 0 0
1

To prepare this second harddrive and show it under the existing share, you can use:

mkdir /mnt/share/another (this creates the directory that it will "live" under).
Edit the /etc/fstab file as above
cfdisk /dev/hdb and delete any partitions on this new hard drive. THIS WILL LOSE YOU ANY DATA OBVIOUSLY. Type carefully. (Note that by using /dev/hdb, we are modifying the SECOND hard drive that the computer sees.)

Create one large Linux partition on it which should show as hdb1.

Similarly for hdc, hdd etc. as you add extra hard drives.

Printing
========

First pico /etc/rc.d/rc.modules and uncomment anything under the Parallel port sections (remove the # character from the beginning of the lines). When you reboot, typing lsmod should show up some stuff to do with "parport".

Then, use lynx to go to http://localhost:631 and in there you will find an interface that will let you add most laser printers. Any printers added should print a test page and should then appear under Network Neighbourhood as a shared printer.

If you have a non-laser printer or a strange standard one (e.g. Samsung), you can try with CUPS using Linux Printing.org's CUPS tutorial or use APSFilter:

/usr/lib/apsfilter/SETUP

(remember to cd back to the home directory when you are done).

APSFilter is more complicated and quite tricky to piece together.