Category Archives: AmazonWS

cf_sesmail custom tag for Railo


Lately I have been using Amazon’s Simple Email Service quite a bit. I use the AmazonSES component for a few clients, but it still is not quite as natural as using a cfmail tag.  Running PostFix to relay messages through SES is not hard to setup, but it is a little overkill if you just need to use it from ColdFusion. With these things in mind, I remembered the post Todd Rafferty wrote about writing a cfc custom tag. I thought it would be pretty cool to write a custom tag to allow you to send an email through SES with the familiar feel of cfmail. If you would like to check it out, it is pretty easy to setup.

sesmail.cfc 
component {
/*
* Project: sesmail
* Author : Robert Zehnder
* Date   : 9/2/2011
* Purpose: cfmail-like implementation to make it easier to send emails through Amazon Simple Email Service
*/
 this.metaData.attributeType = "fixed";
 this.metaData.attributes = {
  from        : { required: true, type: "string" },
  to          : { required: true, type: "string" },
  cc          : { required: false, type: "string", default: "" },
  bcc         : { required: false, type: "string", default: "" },
  subject     : { required: true, type: "string" },
  mailerID    : { required: false, type: "string", default: "cfmailses" },
  endPoint    : { required: false, type: "string", default: "" },
  credentials : { required: true, type: "string" },
  name        : { required: false, type: "string", default: "sesResults" }
 };

 public void function init(required boolean hasEndTag, any parent) {

 }

 public boolean function onStartTag(struct attributes, struct caller) {
  return true;
 }

 public boolean function onEndTag(struct attributes, struct caller) {
  var results = {};
  var awsCredentials = createObject("java", "java.io.File").init(attributes.credentials);
  var creds = createObject("java", "com.amazonaws.auth.PropertiesCredentials").init(awsCredentials);
  var emailService = createObject("java", "com.amazonaws.services.simpleemail.AmazonSimpleEmailServiceClient").init(creds);
  var props = createObject("java", "java.util.Properties");
  var verifyRequest = createObject("java", "com.amazonaws.services.simpleemail.model.VerifyEmailAddressRequest").withEmailAddress(attributes.from);
  var sendHeaders = { "User-Agent" : attributes.mailerID };

  // Set properties for establishing connection
  props.setProperty("mail.transport.protocol", "aws");
  props.setProperty("mail.aws.user", creds.getAWSAccessKeyId());
  props.setProperty("mail.aws.password", creds.getAWSSecretKey());

  // Send email message
  var mailSession = createObject("java", "javax.mail.Session").getInstance(props);
  var mailTransport = createObject("java", "com.amazonaws.services.simpleemail.AWSJavaMailTransport").init(mailSession, JavaCast("null", 0));
  var messageObj = createObject("java", "javax.mail.internet.MimeMessage").init(mailSession);
  var messageRecipientType = createObject("java", "javax.mail.Message$RecipientType");
  var messageFrom = createObject("java", "javax.mail.internet.InternetAddress").init(attributes.from);
  var messageTo = listToArray(attributes.to);
  var messageCC = listToArray(attributes.cc);
  var messageBCC = listToArray(attributes.bcc);
  var messageSubject = attributes.subject;
  var messageBody = arguments.generatedContent;
  var verified = arrayToList(emailService.ListVerifiedEmailAddresses().getVerifiedEmailAddresses()).contains(attributes.from);
  var i = 0;

  try {

   // Is the sender verified
   if(!verified){
    var verifyRequest = createObject("java", "com.amazonaws.services.simpleemail.model.VerifyEmailAddressRequest").withEmailAddress(attributes.from);
    try{
     emailService.verifyEmailAddress(verifyRequest);
    }
    catch (any e){
    }
    throw("Email address has not been validated.  Please check the email on account " & attributes.from & " to complete validation.");
   }

   mailTransport.connect();

   messageObj.setFrom(messageFrom);
   for(i = 1; i <= arrayLen(messageTo); i++){
    messageObj.addRecipient(messageRecipientType.TO, createObject("java", "javax.mail.internet.InternetAddress").init(trim(messageTo[i])));
   }

   if(arrayLen(messageCC)){
    for(i = 1; i <= arrayLen(messageCC); i++){
     messageObj.addRecipient(messageRecipientType.CC, createObject("java", "javax.mail.internet.InternetAddress").init(trim(messageCC[i])));
    }
   }

   if(arrayLen(messageBCC)){
    for(i = 1; i <= arrayLen(messageBCC); i++){
     messageObj.addRecipient(messageRecipientType.BCC, createObject("java", "javax.mail.internet.InternetAddress").init(trim(messageBCC[i])));
    }
   }

   if(len(structKeyList(sendHeaders))){
    for(i in sendHeaders){
     messageObj.addHeader(i, sendHeaders[i]);
    }
   }

   messageObj.setSubject(messageSubject);
   messageObj.setContent(messageBody, "text/html");
   messageObj.saveChanges();

   mailTransport.sendMessage(messageObj, JavaCast("null", 0));

   mailTransport.close();

  }
  catch (Any e){
   throw("Error sending message.");
  }
  return false;
 }

}

Now we can send mail through the SES gateway like this:

<cf_sesmail from="user@domain" to="other@domain" subject="subject" credentials="/path/to/awscredentials.properties">
Hello from cf_sesmail!
</cf_sesmail>

It is self-explanatory.  You supply the basics such as the from address, to address and subject. Instead of passing a user name and password for authentication, you pass the full path to your awscredentials.properties file and the tag takes care of the rest. The code is still a little rough around the edges but it is usable. I will get something up on github soon.

Here is a link to the version of the AWS SDK I am using for development. It is a little dated, but it works perfectly with Railo without requiring you to update any additional jars in your installation. Just extact this file and place the sdk jar into your Railo classpath. For Linux systems, this will most likely be /opt/railo/lib and you should be good to go. If you want to really integrate it into your web context you can copy the component into your WEB-INF/railo/library/tags/ folder, this will allow you to use the custom tag just as if it was a built in function (i.e., <cfsesmail ..></cfsesmail>).

Updated 9/4/2011:
Project is now hosted on github here: https://github.com/robertz/cf_sesmail

Advertisements

Using PostFix to send emails using Amazon SES


Of all the Amazon Web Services I use,  the Simple Email Service would have to be the service I use the most. ColdFusion allows me to easily create a component that sends emails pragmatically but, however, it would be much nicer to use the CFMAIL tag and be done with it. As it turns out, Amazon SES allows you to send a raw email so you can run PostFix  to relay the message through Amazon SES using a perl script.

I found a good tutorial on getting this configured here. I did have an issue getting perl to find the SES.pm file, but this post details how to work around that issue. The great thing about handling it this way is other applications on the server can send messages through the gateway, not just ColdFusion applications.

Installing yasm on Amazon Linux


I am currently working on a project that requires me to build ffmpeg locally on an Amazon Linux instance.  I did a repo search and could find nasm, but ffmpeg didn’t like it at compile time.  Here is how to get yasm installed.  I am documenting this because I will probably need it again.  It is assumed you have already installed git-core.

git clone git://github.com/yasm/yasm.git
cd yasm
./autogen.sh
./configure --prefix=/usr 
make
sudo make install

Git is a handy little tool.

Installing s3fs on RHEL/Centos


Lately I have been doing a lot of work with AWS t1.micro instances running Amazon Linux which seem to be based on RHEL/Centos.  Both Railo and ACF do a good job of interacting with Amazon S3 storage which definitely makes our jobs as developers easier, but what if you wanted to mount your S3 storage locally to have access to your files at the system level so you could actually work with them?  Luckily there is an open-sourced s3fs project that will allow you to do just that.

At the time of this writing, the current file release is s3fs-1.59.tar.gz.  The unfortunate thing is, s3fs requires Fuse 2.8.4 and the newest version available in the package repos is Fuse 2.8.3.  The first step is to get the newest version of Fuse and get it on the server.

wget "http://downloads.sourceforge.net/project/fuse/fuse-2.X/2.8.4/fuse-2.8.4.tar.gz?r=&ts=1299709935&use_mirror=cdnetworks-us-1"

Once the download is completed, extract it:

tar -xzvf fuse-2.8.4.tar.gz
cd fuse-2.8.4

If you are still on a stock install of Amazon Linux, at this point we will need some tools to get everything configured and compiled.

sudo yum groupinstall "Development Tools"

This will install the tools we need for a moment.  However, we will need to install some more packages to get s3fs to compile.  We might as well get them now:

sudo yum install curl-devel libxml2-devel openssl-devel mailcap

Now we should still be in the the fuse-2.8.4 directory, so now it is time to configure and compile Fuse.

./configure --prefix=/usr
make
sudo make install
sudo ldconfig
export PKG_CONFIG_PATH=/usr/lib/pkgconfig
pkg-config --modversion fuse

If everything went as planned, pkg-config should return 2.8.4.

Next we need to download and install s3fs.  First we need to get and extract the archive:

cd
wget http://s3fs.googlecode.com/files/s3fs-1.59.tar.gz
tar -xzvf s3fs-1.59.tar.gz
cd s3fs-1.59
./configure --prefix=/usr
make
sudo make install

The installation should now be in working order.  The next step will be to decide how you would like to create your password file for s3fs.  You can either create a site wide password file /etc/passwd-s3fs or you can create one just for your user account ~/.passwd-s3fs.  The files are required to be secure, so if you go with the system wide password file be sure to chmod 640 /etc/passwd-s3fs or if you want to use your user account, chmod 600 ~/.passwd-s3fs.  The format for the files is the standard [AccessKey]:[SecretKey].

Finally, lets map the drive to a local directory.  In my home directory I created a folder name s3storage that will be my mount point.  We create the mount with

s3fs [bucketname] ~/s3storage -o default_acl=public-read

I have only set this up on one machine so I still do not have the install down completely, but I was able to get this up and running successfully.  All the steps above are more or less from memory so I apologize for any hazy steps.  I will correct as needed.

Load balancing a Railo cluster using AWS


Lately I have been playing around with load balancing Railo in a cluster.  Before Amazon Web Services this would have been a fairly expensive proposition, but AWS makes managing instances mostly painless.  Here I will be using EC2 instances, but this setup will can apply to just about any configuration.

The minimal requirements for this exercise will be to have three instances running.  First I created two instances of 64-bit Amazon Linux running configured with a basic Railo setup using the VivioTech installers and Apache as the front end.  I already have an AMI image created so I can easily create identically configured Railo instances with the click of a button.

Next we need to configure an instance to handle load balancing to each of our Railo instances and for this I will be using Nginx.   First, create a clean, new instance of Amazon 64-bit linux and SSH into the instance.  Amazon Linux uses the yum package manager so installing Nginx is as easy as:

yum install nginx

Once Nginx is installed, we just need to edit the config file and add in a few lines.  You can find the configuration file in /etc/nginx/nginx.conf.

#----------------------------------------------------------------------
# HTTP Core Module
#
#   http://wiki.nginx.org/NginxHttpCoreModule
#
#----------------------------------------------------------------------

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    upstream balancer {
        server 10.x.x.1;
        server 10.x.x.2;
    }

    #
    # The default server
    #
    server {
        listen       80;
        server_name  _;

        location / {
            proxy_pass http://balancer;
        }

        error_page  404              /404.html;
        location = /404.html {
            root   /usr/share/nginx/html;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }
    }

    # Load config files from the /etc/nginx/conf.d directory
    include /etc/nginx/conf.d/*.conf;

}

This is the complete listing for the nginx.conf file.  This configuration is pretty much stock, I have removed the comments, added the upstream directive and modified the default server to proxy requests to the upstream provider named balancer.

This approach will evenly distribute requests to both machines.  You can add the hash;  ip_hash; directive to the top of  your upstream block to make the requests “sticky” and send requests to the same server behind the load balancer each time.

I have just started to using this setup so I am sure things will probably change as I start seeing what it can do. If you are interested in setting up a similar system to mess around with I highly recommend AWS.

Edit:
Some helpful documentation: http://wiki.nginx.org/HttpUpstreamModule

AmazonSES and custom mail headers


A user pinged me this morning asking whether or not it was possible to set the fail-to email address when a message was bounced when using the AmazonSES API wrapper.  This is accomplished by setting the Return-Path header in the AmazonSES API wrapper.  This header is left empty by default but setting it is quite easy.  Here is an example of setting it when the component is initialized:

 public any function init(required String pathToCredentials) hint = "I initialize the gateway" {
  variables.instance = {};
  instance['awsCredentials'] = createObject("java", "java.io.File").init(arguments.pathToCredentials);
  instance['credentials'] = createObject("java", "com.amazonaws.auth.PropertiesCredentials").init(instance.awsCredentials);
  instance['props'] = createObject("java", "java.util.Properties");
  instance.props.setProperty("mail.transport.protocol", "aws");
  instance.props.setProperty("mail.aws.user", instance.credentials.getAWSAccessKeyId());
  instance.props.setProperty("mail.aws.password", instance.credentials.getAWSSecretKey());
  instance['lastUpdated'] = now();
  instance['statsRefreshTimeout'] = 5;
  instance['supportedHeaders'] = listToArray("Accept-Language,Bcc,Cc,Comments,Comment-Type,Content-Transfer-Encoding,Content-ID,Content-Description,Content-Disposition,Content-Language,Date,DKIM-Signature,DomainKey-Signature,From,In-Reply-To,Keywords,List-Archive,List-Help,List-Id,List-Owner,List-Post,List-Subscribe,List-Unsubscribe,Message-Id,MIME-Version,Received,References,Reply-To,Return-Path,Sender,Subject,Thread-Index,Thread-Topic,To,User-Agent");
  instance['endPoint'] = "";
  instance['sendQuota'] = getSendQuotaService();
  instance['sendStats'] = getSendStatisticsService();
  instance['verifiedEmailAddresses'] = listVerifiedEmailAddressesService();
  instance['sendHeaders'] = {'Return-Path'='my@email.com'};
  return this;
 }

You can also call the setHeaders(required struct headers) method to pass the default header you want to use with all emails.  If you want bounces sent to the sender of the message it can also be handled in the sendEMail() method like so:

messageObj.addHeader("Return-Path", createObject("java", "javax.mail.internet.InternetAddress").init(messageFrom).toString());

AmazonSES supports quite a few headers out of the box as well as letting set your own custom headers as well.  If there is enough interest I will go ahead and integrate the Return-Path header into the code base.   Should it be set by default or would you rather set it yourself?  Let me know.

AmazonSES API 0.1.6


I finally got around to installing the developer edition of ACF tonight.  I have gotten several emails about having to make modifications to AmazonSES to get it working correctly with ACF so I took a little bit of time and got all of this corrected.   The fixed version is now on github.

AmazonSES API 0.1.5


I finally had a few minutes tonight to finish up the modifications for version 0.1.5.

Improvements for this version:
* Now defaults to a persisted application-scoped component
* No hard error on demo if AwsCredentials.properties is not correctly configured
* Each new request initiates a new connection to the email service
* The setEndPoint function is now called every time the mail service is initiated (if it is not the default address)
* You can now set message headers for outgoing emails

I was hesitant to make this an application-scoped component because I was trying to make this as light-weight as possible.  I did try to make sure it was backwards-compatible with the previous versions so if you did decide that you did not want to create an instance in the application scope, it still works just fine when created in the variables scope.  If you wanted to set any headers or change the end point from the default though you will need to edit the init method and override the default values with the values you would like to use.

Enjoy.

https://github.com/robertz/AmazonSES

AmazonSES API updated


Tonight I completed writing the majority of the remaining methods for the AmazonSES API.  I added the ability to set your AWS end point if you are not using the defaut.  You can also get your send quota for the day and delete a previously verified email address.

There are a few methods that I did not implement.  The getSendStatistics method did not make it in just yet but I do plan on getting that added.  I would also like to add in getCachedResponseMetaData as well because the debugging information it provides may be useful.  On the other hand, sendRawEmail will not be added and the shutDown method is not needed.

If you are interested, here are the javadocs for the Amazon simple email service.

There were also a few changes to some of the existing methods.  I renamed the sendMail method to sendEmail to match with the javadocs.  You can also now pass in a CSV string for the recipient, CC and BCC fields.  In version 0.1.0 you could only pass it one recipient and the CC and BCC fields were not functional and this has all been corrected.  Most other changes were around error checking and exception handling.

AmazonSES API wrapper for Railo/ACF


As I alluded to this morning, I have been working on a wrapper API for the Amazon SES service.  It is not yet completed, but it is functional enough to get you sending some emails through the gateway. Currently it will allow you to get a listing of emails addresses verified with Amazon Web Services, create a verification email so that you can verify your email address with AWS as well as actually sending an email through Amazon SES.

Integration with Railo (and presumably ACF) is straight forward.  First, download the Java SDK for AWS.  Extract the file and in the ./lib folder you will find the jar file we will need to move into your classpath for Railo/ACF.  At the time of this writing the file is aws-java-sdk-1.1.8.jar. I generally use the Viviotech installer which places Railo in /opt/railo on Linux and I copied the SDK jar to /opt/railo/lib. Restart Railo so all the jars will be reloaded giving us access to the API.  I do not have access to an ACF server so your mileage here may vary.

The server configuration done, now you just need to go to AWS and get your access key and secret key.  These are required to authenticate your connection to Amazon and should be stored in the AwsCredentials.properties file.  You obviously want to keep this file protected so I would suggest storing outside of your web root or create some rules on your web server to ensure no one can view the file.  Once you have this setup it is time to send some emails!

Sending an email is extremely easy to do.  First create a new instance of the amazonSES component and then invoke the sendMail method passing in the required parameters (from, to, subject and the message body).  If you have not already verified your address with Amazon, the script will create a new verification email and send it to your email address and then throw an error letting you know you need to check your email for the verification.  If you have already verified your email address it will just send the email out.

That is about all I have for tonight.  As noted above, I have only tested this on Railo, I do not have an ACF box to verify against.  It will probably need to be modified to handle multiple message recipients (I have not tested this yet). I plan on working on some new features this weekend if I have a little free time.

I have included a simple application you can use to test the installation.  If you have any questions, problems or suggestions, please let me know.

Update: I have created a RIAForge project for the AmazonSES API wrapper.

Update 2: I have linked to the older version of the AWS API on my forum and you can download the file here if you are having issues.