s3fs is a FUSE filesystem that allows you to mount an Amazon S3 bucket as a local filesystem. It stores files natively and transparently in S3 (i.e., you can use other programs to access the same files).

The following instructions detail the steps to install the program s3fs on an Amazon EC2 running Debian 5.0.5.

1. Install libfuse
   First, you need to install the package libfuse manually as the one provided via apt-get is too old (s3fs needs a version greater than or equal to 2.8.4).

   wget http://downloads.sourceforge.net/project/fuse/fuse-2.X/2.8.7/fuse-2.8.7.tar.gz
   tar xzf fuse-2.8.7.tar.gz
   cd fuse-2.8.7
   ./configure --prefix=/usr
   make install
   

2. Install libxml
   You can simply install the package provided by apt-get:

   apt-get install libxml2-dev
   

3. Upgrade mount
   Because of a problem between fuse and mount, you need to upgrade the version of mount:

   wget http://www.kernel.org/pub/linux/utils/util-linux/v2.21/util-linux-2.21-rc1.tar.gz
   tar xzf util-linux-2.21-rc1.tar.gz
   ./configure --prefix=/usr --without-ncurses
   make install
   

   For more information about this issue, please go to the following page: http://code.google.com/p/s3fs/issues/detail?id=228

4. Install s3fs
   You can now install s3fs using the following commands:

   wget http://s3fs.googlecode.com/files/s3fs-1.61.tar.gz
   tar xvzf s3fs-1.61.tar.gz
   cd s3fs-1.61/
   ./configure --prefix=/usr
   make install
   

Note that I wanted to use s3fs to create incremental snapshot-style backups with rsync. Unfortunately, as mentioned on the following page, it didn’t work because s3fs doesn’t support hard links: http://code.google.com/p/s3fs/issues/detail?id=46

The post Install s3fs on Amazon Clouds first appeared on LogikDevelopment.

I will talk on this one about an error I got when running the S3 synchronisation:

S3 command failed:
list_bucket prefix /data max-keys 200 delimiter /
With result 403 Forbidden
S3 ERROR: #
s3sync.rb:290:in `+': can't convert nil into Array (TypeError)
	from s3sync.rb:290:in `s3TreeRecurse'
	from s3sync.rb:346:in `main'
	from ./thread_generator.rb:79:in `call'
	from ./thread_generator.rb:79:in `initialize'
	from ./thread_generator.rb:76:in `new'
	from ./thread_generator.rb:76:in `initialize'
	from s3sync.rb:267:in `new'
	from s3sync.rb:267:in `main'
	from s3sync.rb:735

As you can see, this error is not very human-friendly! The only thing we know is that the S3 command failed because of the error can't convert nil into Array. It looks to me like an internal error within s3sync…

But after some investigation, it appears it is simply because the system date on the server is not correct. I cannot tell you how much time I spent on this one! 

Anyway, if you are doing automatic backups as describe on John Eberly’s blog, you need to add the following code at the top of your upload.sh script:

# update the system date
/usr/sbin/ntpdate 3.uk.pool.ntp.org 2.uk.pool.ntp.org 1.uk.pool.ntp.org 0.uk.pool.ntp.org

NB: please find below the command lines I use to install ntpdate on a Debian server:

apt-get install ntpdate
dpkg-reconfigure tzdata

The post S3 command failed if the time is not synced first appeared on LogikDevelopment.

I am using this tool to automatically backup the important data from Debian servers to Amazon S3. I am not going to explain here how to install s3sync as it is not the purpose of this article. However, you can read this very useful article from John Eberly’s blog: How I automated my backups to Amazon S3 using s3sync.

If you followed the steps from John Eberly’s post, you should have an upload.sh script and a crontab job which executes this script periodically.

From this point, here is what you need to do to monitor the success of the synchronisation with Zabbix:

1. Add the following code at the end of your upload.sh script:

   # print the exit code
   RETVAL=$?
   [ $RETVAL -eq 0 ] && echo "Synchronization succeed"
   [ $RETVAL -ne 0 ] && echo "Synchronization failed"
   

2. Log the output of the cron script as follow:

   30 2 * * sun /path/to/upload.sh > /var/log/s3sync.log 2>&1
   

3. On Zabbix, create a new item which will check the existence of the sentence “Synchronization failed” in the file /var/log/s3sync.log:
   
   Item key: vfs.file.regmatch[/var/log/s3sync.log,Synchronization failed]
   
4. Still on Zabbix, define a new trigger for the previously created item:
   
   Trigger expression: {Template_AmazonCloud_Debian:vfs.file.regmatch[/var/log/s3sync.log,Synchronization failed].last(0)}=1

With these few steps, you should now receive Zabbix alerts when a backup on S3 fails. 🙂

The post Monitor s3sync with Zabbix first appeared on LogikDevelopment.