FreeBSD
Systemadministration under FreeBSD
To make it short: There is absolutely NO real Cluster-Filesystem (like GFS or OCFS2) for FreeBSD at present. Also other projects for distributed filesystems like GlusterFS, PVFS or DRBD are not ported to FreeBSD, or the ports are very old.
Since I was in the need to have four identical data-filesystems (which have to be in sync just seconds after the upload), I wrote a little work-around for rsync, using the FreeBSD audit-system. The idea to use the audit-system for triggering the rsync I got from Luke Marsden, who is monitoring filesystem activity with audit_control and some python-scripts.
First of all, the audit_system must be activated and configured. The event-auditing is part of FreeBSD and has to be compiled into the kernel.
Add the following line to your kernel configuration:
options AUDIT
Then rebuild and reinstall your kernel as described in the FreeBSD Handbook
After this, add the following line to your /etc/rc.conf
auditd_enable=”YES”
The next step is, to configure the audit-system: Open the file /etc/security/audit_control and change the config to:
dir:/var/audit
flags:fc,fd,fw
minfree:20
naflags:lo
policy:cnt
filesz:0
That’s all for now. You can now start the audit-system by either calling
/etc/rc.d/auditd start
or by rebooting your system.
If rsync isn’t already installed on your system, you may do this by using the ports:
cd /usr/ports/net/rsync
make
make install
Installation of rsync should be no issue.
Next step is, to set an alternative path to your data-directory, using a symbolic link (I’ll explain later why).
ln -s /path/to/your/data/ /alternative_data_path/
Now we have to configure rsync to run as daemon. Therfor we create (or change) the config for rsync: /etc/rsyncd.conf
max connections = 5
log file = /var/log/rsync.log
timeout = 30[shareName]
comment = Name of this “Rsync mount”
path = /alternative_data_path/
read only = no
list = yes
uid = validUser
gid = validGroup
hosts allow = ,
hosts deny = *
To start the rsync daemon, you have to call:
/usr/local/bin/rsync –config=/etc/rsyncd.conf –daemon
It is perhaps a good idea to monitor the rsyncd with the daemontools to make sure, the rsync-service is always available (you have then to run it with the –no-detach option).
#!/usr/bin/perl ## # This software is published under the Apchae 2.0 licenses. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Author: Erik Scholtz # Web: http://blog.elitecoderz.net ### # We are strict, cauz we are elitecoderz! use strict; use threads qw(yield); use threads::shared; use Thread::Semaphore; # No caching $|=1; ################ # Configuration my $debug = 1; # 0/1 to enable logging to the console or disable my $path = '/path/to/your/data/'; # Path to sync my @cmds; # Syncer commands that should be executed $cmds[0] = '/usr/local/bin/rsync -raz --progress --size-only /path/to/symboliclink/data/<!--target--> rsync:///shareName/<!--target-->'; $cmds[1] = '/usr/local/bin/rsync -raz --progress --size-only /path/to/symboliclink/data/<!--target--> rsync:///shareName/<!--target-->'; $cmds[2] = '/usr/local/bin/rsync -raz --progress --size-only /path/to/symboliclink/data/<!--target--> rsync:///shareName/<!--target-->'; ############################################################################### # DO NOT CHANGE ANYTHING BELOW THIS LINE, UNLESS YOU KNOW WHAT YOU ARE DOING! # ############################################################################### ### # Set Threads yield threads->yield(); # SetUp some thread-shared variables my $commands :shared; $commands = &share([]); my $run :shared; $run = &share({}); $run->{'status'} = 1; my $sema :shared; $sema = &share({}); # Local array where all syncer-threads are stored my @threads; # Create a thread for each syncer my $maxid = -1; for (my $i=0;$i<=$#cmds;$i++) { print "Starting syncer $i\n" if $debug; $sema->{$i} = Thread::Semaphore->new(0); my $syncer = threads->create('syncJob',$run,$sema,$i,$commands,$path,$cmds[$i],$debug); push(@threads,$syncer); $maxid = $i; } # Create the Checker thread, which cleanup the jobs and ensures the function of all syncers $sema->{'checker'} = Thread::Semaphore->new(0); my $syncer = threads->create('JobChecker',$run,$sema,$commands,$maxid,$debug); push(@threads,$syncer); # Create the audit thread print "Starting audit\n" if $debug; my $auditthread = threads->create('audit',$run,$sema,$commands,$path,$maxid,$debug); print "Waiting for audi to terminatet\n" if $debug; $auditthread->join(); # If the audit-thread gets joinable, we have to terminate everything # Terminate all threads and cleanup $run->{'status'} = 0; while ($#threads >=0 ) { my $worker = shift(@threads); print "Shutdown of syncer ...\n" if $debug; $worker->join(); } print "Shutdown clean completed\n" if $debug; exit(0); ######################################################################################################################################## ######################################################################################################################################## #################################################################################################### # audit thread sub audit { my $r = shift; my $sp = shift; my $c = shift; my $p = shift; my $m = shift; my $d = shift; print " audit started ...\n" if $d; # open listener on the audit device open(STATUS, "/usr/sbin/praudit /dev/auditpipe |") || die "can't fork: $!"; while (<STATUS>) { my $line = $_; last if ($line eq '' || $r->{'status'}<=0); # Terminate if audit terminated if ($line =~ /path,$p(.+)/) { # Check if the changed file is in the observed path my $file = $1; print "Change detected on file: $file\n" if $d; my $hash :shared; # Create a command for the syncers $hash = &share({}); $hash->{'file'} = $file; $hash->{'status'} = ''; $hash->{'time'} = ''; for (my $j=0;$j<=$m;$j++) { # init job done charta $hash->{$j} = 'no'; } if (1) { lock($c); push(@{$c},$hash); print "Added new job for $file\n" if $d; } for (my $j=0;$j<=$m;$j++) { # wakeup syncers $sp->{$j}->up(); } $sp->{'checker'}->up(); } } close STATUS || die "audit not closed correctly: $! $?"; print " audit terminated ...\n" if $d; return(0); } #################################################################################################### # syncer thread sub syncJob { my $r = shift; my $sp = shift; my $id = shift; my $c = shift; my $p = shift; my $e = shift; my $d = shift; print " syncer $id started ...\n" if $d; while ($r->{'status'}>0) { if ($#{$c}>=0) { # if there are any jobs to be done for (my $j=0; $j<=$#{$c}; $j++) { next if ($c->[$j]->{$id} eq 'ok'); # if my job is already done skip this job and check next my $file = $c->[$j]->{'file'}; if (-e $p.$file) { # check if the file is existing $c->[$j]->{$id} = 'working'; # mark this job as being worked on my $dif = 1; while ($dif>0) { # check if the file is in upload and changes size within 1,5 secs print "Checking Filesize ...\n" if $d; my $ssize = -s $p.$file; sleep(1.5); my $eesize = -s $p.$file; $dif = $eesize - $ssize; print "Checking Filesize $ssize - $eesize = $dif\n" if $d; } my $cm = $e; $cm =~ s/<!--target-->/$file/g; system($cm); # rsync to other server } lock($c); $c->[$j]->{$id} = 'ok'; # mark job as done for me } } $sp->{$id}->down(); } print " syncer $id terminated ...\n" if $d; return(0); } #################################################################################################### # checker thread that checks if all jobs are done sub JobChecker { my $r = shift; my $sp = shift; my $c = shift; my $m = shift; my $d = shift; print " checker started ...\n" if $d; while ($r->{'status'}>0) { while ($#{$c} >= 0 && $r->{'status'}>0) { print " Checker loop ...\n" if $d; my $rem = 0; foreach my $job (@{$c}) { # loop through all jobs my $mem = 'ok'; for (my $j=0;$j<=$m;$j++) { # check job done charta if ($job->{$j} eq 'no') { # job not handled $mem = 'no' if ($mem ne 'working'); # job not handled (may never override a job in progress state) } elsif ($job->{$j} eq 'working') { # job in progress (always overrides not handled) $mem = 'working'; } } # Job not completed if ($mem eq 'no') { if ($job->{'time'} eq '') { # Set timestamp to know, how long this job is already waiting $job->{'time'} = time; } else { # Job already got a timestamp my $watch = time - $job->{'time'}; print "Job age: $watch\n" if $d; if (time - $job->{'time'} > 300) { # Job has waited for more than 5 minutes. terminate program print "TIME FOR JOB EXCEEDED - shutting down syncer"; $r->{'status'} = 0; for (my $j=0;$j<=$m;$j++) { # wakeup syncers $sp->{$j}->up(); $sp->{'checker'}->up(); # wakeup ourself } } } } elsif ($mem eq 'working') { # job in progress - just actualize the timestamp $job->{'time'} = time; } else { $job->{'status'} = 'complete'; # job is completely done and is marked for being removed $rem = 1; } } # Job to remove available if ($rem > 0) { lock($c); # lock the command-queue my @arr; for (my $j=0;$j<=$#{$c};$j++) { # store all not handled jobs / drop completed jobs my $ex = shift(@{$c}); if ($ex->{'status'} ne 'complete') { push(@arr,$ex); } } for (my $j=0;$j<=$#arr;$j++) { # put all stored (not finished) jobs back into the command queue push(@{$c},$arr[$j]); } } print " Checker reloop ...\n" if $d; sleep(1); } print " Checker sleeping (".$#{$c}.")...\n" if $d; $sp->{'checker'}->down(); } print " checker terminated ...\n" if $d; return(0); }
This script does the whole magic: It listens via the audit-system for files changed or added and then uses rsync to sync the file to the other systems. And here we come to the part why we need a symbolic link to the data directory: when the script uses rsync to copy a file to a second system, the audit-system of this second system will notify the script there about this change. So the script on the second system would start to copy the file back to the first system and so on. So if you do not use a symbolic link for the rsync, you will create an endless loop of copy and recopy-processes!
Copy this script to each system that should be kept in sync with the others. I recommend to observe this script via daemontools too. Then edit the script on each system as shown below:
$debug can be set to 0 (for no debug output) or 1 (for debugging output).
$path should be set to the physical path of your data.
For each system that should be kept in sync add the following line. Please keep in mind to increase the number in the square-brakets ($cmds["number"]) by 1 in each line:
$cmds[0] = ‘/usr/local/bin/rsync -raz –progress –size-only /path/to/symboliclink/data/ rsync:///shareName/‘;
Before changing anything on your system, make sure you have a complete backup of your system! The usage of this script and howto is at your very own risk. So if you suffer any data-losses by using this howto or the script you can not hold me responsible for this.
To get close to a “realtime sync”, the script starts an own thread for each volume to keep in sync. So you need to have a perl-installation that is thread-enabled.
After over one year working together with Danny Braniss and testing several thousands of options, settings and configurations, I managed to get the iStore iSCSI-device working together with FreeBSD.
Just to remember. The following error occured, when trying to write an UFS filesystem to the device:
newfs -O2 /dev/da0s1
/dev/da0s1: 782023.5MB (1601584044 sectors) block size 16384, fragment size 2048
using 4256 cylinder groups of 183.77MB, 11761 blks, 23552 inodes.
super-block backups (for fsck -b #) at:
160, 376512, 752864, … … …
1601377920
internal error: can’t find block in cyl 0
And in dangerously dedicated mode:
“# newfs -O2 /dev/da0“
Creating the lable in this mode fails with the message:
newfs -O2 /dev/da0
/dev/da0: 782023.5MB (1601584044 sectors) block size 16384, fragment size 2048
using 4256 cylinder groups of 183.77MB, 11761 blks, 23552 inodes.
super-block backups (for fsck -b #) at:
160, 376512, 752864, … … …
1601377920
internal error: cg 0: bad magic number
The important hint I got from a test with a PetaStor system, where everything worked perfectly. On the FreeBSD-FS mailinglist, I got the last part of the puzzle. Creating the filesystem works with these commands:
“# gpart create -s GPT da0"
# gpart show da0
# gpart add -b 34 -s 20971519 -t freebsd-ufs -l AnosLabel da0
# newfs -O2 /dev/da0p1
Important: Replace 20971519 by the size of your device, given by gpart show da0.
All you need to realize a SOCKS5 tunnel over SSH is an openSSH on your “SOCKS-Server” and the already used “Proxifier”, as well as PuTTY on Windows-based systems.
First of all, you need to build up the secure tunnel (type the command into your OS X console):
ssh -2 -N -D 8080 user@<ip of your SOCKS5-Host>
for example:
ssh -2 -N -D 8080 myuser@192.168.1.11
On Windows, the configuration of PuTTY is the following:
Add the Hostname (for example: 192.168.1.11 – you need to add your SSH/SOCKS-Server IP there) under Tab “Sessions”
Then go to Tab “Tunnels” and enter Source Port 8080
Select “Dynamic” from the Destination Port
Click Open to buildup the tunnel
Now start the Proxifier and add the following Setting:

I find it usefull (but it is not neccessary) to add a rule to the Proxification Rules:

That’s all: openSSH has an integrated SOCKS-Proxy. Proxifier sends all traffic through the SSH-Tunnel (which is encrypted) and the SOCKS-proxy in openSSH on the other side sends the traffic to the destination hosts and returns incomming traffic to you.
After asking google for SOCKS I found a recommended implementation of SOCKS5: Dante
There is a very nice overview of SOCKS-implementations on wikipedia: http://en.wikipedia.org/wiki/SOCKS
Installation was quite easy: On SuSE it is a rpm you simply install, on FreeBSD you find it in the ports (cd /usr/ports/net/dante/; make; make install ). More interesting got the configuration of Dante, and that is what I want to explain today (since I did not find a howto and had to read the documentation):
The config-file can be found in /etc/sockd.conf (/usr/local/etc/sockd.conf on FreeBSD). This you have to edit in the following way:
#define the logfile for dante
logoutput: /var/log/dante.log#define the IP/Port Dante should listen for connections
internal: <IP address of your dante server> port = 80#define the IP/interface Dante should use for outgoing connections
# Check the name of your interface using ifconfig
external: eth0
#Alternative: >external: <IP address that should be used>#authentication: deactivated, since I will use a static IP-adress – that is auth enough now!
method: username none#unprivileged user for Dante
user.notprivileged: nobody
Ok – that was the basic stuff – now the interesting part:
#Who can access this SOCKS Tunnel?
client pass {
from: <your ip here>/32 port 1-65535 to: 0.0.0.0/0
}#Loopback may also access the tunnel
client pass {
from: 127.0.0.0/8 port 1-65535 to: 0.0.0.0/0
}#Block all others
client block {
from: 0.0.0.0/0 to: 0.0.0.0/0
log: connect error
}# Once connected, who may be connected then?
# block connections from anywhere to loopback
block {
from: 0.0.0.0/0 to: 127.0.0.0/8
log: connect error
}# Allow connections from anywhere to client
pass {
from: <your ip here>/32 to: 0.0.0.0/0
protocol: tcp udp
}pass {
from: 127.0.0.0/8 to: 0.0.0.0/0
protocol: tcp udp
}#Block the rest
block {
from: 0.0.0.0/0 to: 0.0.0.0/0
log: connect error
}
Please note: this example will limit the access to one IP (/32), you can also allow more IPs. If you are not firm to subnetting, use the Subnet Cheat Sheet
After this simple configuration, your Dante-server should start without any problems, by typing:
/etc/rc.d/sockd start
(On FreeBSD first add sockd_enable=”YES” to your /etc/rc.conf, then type: /usr/local/etc/rc.d/sockd start ). Now you can watch your logfile under /var/log/dante.log to see what is going on.
Now let’s come to the complicated part: Make OSX work with the SOCKS5 tunnel we created.
The first (and unsuccessfull) idea was, to configure it in in the network setup in the System Preferences. (Go to the Network preference pane, then click on further options and go to the “Proxies” tab. Enable SOCKS Proxy and fill in the IP of your server and the correct port, save the changes and activate the setting). Unfortunatly, this setting only works only for Cocoa and WebKit-based applications (and since not all of these applications use the System Preferences, you are covered only by 95% there too).

Screenshot Systempreferences network german
Thunderbird and Firefox for example doe not use the System preferences. You can both configure them to make them use the SOCKS tunnel, but to be honest: I do not want to reconfigure my applications on every new place. So I looked for a general SOCKS-Proxy.
After some search I found Proxifier – a commercial product, that is easy to setup easy to use and does everything you need with just a few clicks. There is also a version for Windows, that I did not check out, but I’m sure it will work as good as the OS X-version does.
The SetUp is easy and does not need the really good documentation that is provided on the Proxifier-homepage. If you feel better by watching the dosumentation, here you will find it: http://www.proxifier.com/mac/documentation/ProxifierHelp.html
After starting and configuring Proxifier, I got back online to the world, bypassing the firewall of my customer over port 80.
ATTENTION: Dante and SOCKS may not be confused with VPN, even if it is the same feeling! The data is send clear-text and my be visualized with any Packet-Sniffer!