Re: Website/ftp mirror

[mRgOBLIN <goblin@scooter.co.nz>]

David Webster wrote:

>
> I was thinking that I'd probably use wget for the web/ftp sites. I've used
> it sucessfully for mirroring websites before, but not ftp sites, anybody
> know if this is the tool to use, of if a better one exists.
>

Well I use wget for mirroring and find it werks very well
Here is a script that I use to mirror my site.
Called "dundir.wget"
-------------------------CUT--------------------
#!/bin/sh
#
# Update Dunedin-direct
#
# JM 7/1998

# usage:     dundir.wget [anything]
# any arguments at all skips mirroring, moves right to cleanup.

site=ftp://USER:PASS@ftp.dunedin-direct.co.nz
sitedir=/home/dunedin-direct/www; cutdirs=3
#localdir=`basename $sitedir`
localdir=/home/dunedin-direct/www/
log=/var/log/dundir.log

excludes=""
for exclude in shop ; do
 [ "$excludes" ] && excludes="${excludes},"
 excludes="${excludes}${sitedir}/${exclude}"
done

# Do the mirroring:

if [ ! "$*" ]; then
 echo -n "Mirroring from $site (see $log) ... "
 wget -w 5 --mirror $site/$sitedir -o $log -nH --cut-dirs=$cutdirs
-X"$excludes" --directory-prefix=$localdir
 echo "done."
fi

# Remove old stuff
# (important, but wipes out extra stuff you might have added)

echo "Removing old stuff ..."
for d in `find $localdir -depth -type d`; do
 pushd $d > /dev/null
 for f in *; do
  grep -q "$f" .listing || { rm -rf "$f" && echo $d/$f; }
 done
 popd > /dev/null
done
echo "Done."
-------------------------CUT----------------


Just take care with the clean-up part of the script
Best to trial run it somewhere harmless first
P.S I didn't write it ... I just modified it for my own use :)



mRgOBLIN