ColdFusioned Security

Amazon HMAC with ColdFusion

2009-08-02T16:36:00.006-05:00

I recently received a work request to update our calls that we were making against Amazon's Web Services with a new authentication signature. This proved to be more complicated that it should have been. I thought I would share my experience, in hope that it will help remove a headache for someone else out there.

The purpose was to "sign" your requests with a secret key so Amazon could verify that the request had not been altered in transit (these are requests that are transmitted via REST calls (http). There have been questions about why this was needed. The biggest point of abuse that I can think of comes from the use of these webservices from a client side (AJAX) - or doing a DOS against another competitor since you can only make so many requests per minute. Of course, now AJAX calls will need to be designed so the secret key never ever ever get exposed to the client side.

Here is the "documentation" Amazon provided. They gave this Important note to us:
"You have until August 15, 2009 to authenticate requests sent to the Product Advertising API. After August 15, 2009, messages that aren't authenticated will be denied."
What they failed to mention was their "testing" schedule (aka planned outages) which was sent (and forwarded to me just a few days ago).
Here is the full schedule:

"These planned outages will help our developers test their signed requests implementation and also discover applications and code paths that they may have missed, giving them the opportunity to address these gaps before signed requests become mandatory on August 15."
Monday, Aug 3 at 2PM (ET) through Tuesday, Aug 4 at 2PM, a few requests (appx 20% of the requests) that do not implement this new method correctly will be rejected.
Monday Aug 10 from 2PM (ET) through 4PM, all requests that do not implement this new method correctly will be rejected.

Uhh, sorry what? Planned production outage 2 weeks earlier?!! Needless to say this led to an elevated blood pressure for many. I looked at some of the Sample REST requests, found some HMAC code already available on RIA Forge in several "amazon" related projects and thought "This will be easy enough".

Its not that it wasn't easy to reproduce the example they gave. That part WAS easy. The difficult / frustrating part was getting the REAL requests signed and authenticated. The good news for many is that you can simply check the status code of your original request and make it again if it has failed during this first round that lasts 24 hours and you will likely be fine.

Here is a checklist of what I would suggest looking at if you are having problems, this is from my own experiences and is bound to help someone. As I receive feedback, I'll update accordingly.

Make sure you are using your OWN secret key.
Use http://ecs.amazonaws.com - NOT http://webservices.amazon.com
Do NOT use a trailing slash at the end of /onca/xml?...
DO NOT put line breaks in your query string. Just use the normal & to separate your params.
Make sure you are only doing a line break chr(10) like the following:
'GET' & chr(10) & 'ecs.amazonaws.com' & chr(10) & '/onca/xml' & chr(10) & '#YourQueryStringHERE#'

For me, the big epiphany was after using the webservices URL, one type of responseGroup would work, but another would not. The error was coming from ecs.amazonaws.com, even though I was making my calls to webservices.amazon.com, just like their example.

Here is an example of the code that I used.

Puzzler: Which way to the runway?

2009-05-19T10:52:00.003-05:00

Here is a fun puzzle for those looking for a break from normalcy ...

When visiting and looking at my route via a popular map utility on my phone, I came across an interesting artifact.

I'm curious if anyone can find the shadow of this, or for bonus points, the equation needed to calculate the height of said artifact, once the shadow is found!

Power of SQL: Update Replace()

2009-03-02T02:23:00.005-06:00

Sometimes you realize after the fact that there's an issue with the way your data is being stored. In this case, we had a field that was storing names. Originally intended to be able to show an admin / show on reports the full person's name without concatenating data together, here is what was being built.
"firstname middlename lastname"
Unfortunately, when the middlename was not filled out, it contained two spaces between the first and last names. In addition to fixing the way that the the data is being updated, there's an easy fix for existing data.

update users
set fullname = replace(fullname,' ',' ')
where fullname like '% %'

For clarity, this finds and replaces two spaces with one space. Its easy to adapt to your specific situation and yes, its really that easy.

Tag Context

2008-12-24T12:02:00.003-06:00

Update: I had a formatting issue in the code example.

I found this method handy when trying to track down the tab Context of a page's execution. This throws an error, catches it and allows you to output (or email ... or post to a log) the needed information of where a page's execution is coming from.

<cftry>
<cfthrow>
<cfcatch>
<cfset e=duplicate(cfcatch)>
<cfdump var="#e.tagContext#">
</cfcatch>
</cftry>

One word of caution though, in my experience, it can over time (several days of try/throw/catching errors) lead to server instability. In my case, it was sending an email in the catch and was left in even after I fixed the issue to monitor / make sure it was fixed. The error you will see is:

An exception occurred when setting up mail server parameters.
This exception was caused by: coldfusion.mail.MailSessionException

For some reason it loses the connection to the mail server and ANY code that tries to send an email will fail. Not good. It was handy and served its purpose, but be warned on your method of implementation.

Power of SQL: Update Join Tables

2008-12-14T09:25:00.002-06:00

This is one of the most common update queries I do when dealing with multiple tables that needs data updated.

update o
set o.status = 'complete'
from orders o
join order_detail od
on o.id = od.order_id
where ... (criteria here)

I often see code that will do a select and then update the subsequent table where id in (list_of_values). Remember that (almost?) anytime you see an IN clause, it can be replaced with a more efficient join statement.

Power of SQL: Find and Delete Duplicates

2008-10-08T06:14:00.011-05:00

Most likely we have all had the situation, or will most likely encounter it, where we have duplicate data that needs to be removed from a system. For me this was experienced when moving data from a flat file / legacy system into SQL. This is the solution I came up with after getting it into SQL. Perhaps you have a more elegant way in dealing with this, if so, feel free to comment. The execution of this entire process against ~200,000 records took less than 15 seconds on my particular database. The creation of the index seems to be helpful if you are going to run several queries against the data while preparing the statement.

First, the background:
I have a table of notes. Those zero or more of those notes are related to a foreign key. If there is more than one note that is identical, for the same foreign key, it needs to be removed.

It is extremely difficult to compare a text field to another text field. With up to 2GB in length, it can be a daunting task. For my situation, most of the notes were less than a length of 8000, only a small number were longer. If they were longer, we could safely assume if they were not different for the first 8000 chars, they wouldn't be. You may choose to add additional constraints for your situation.

Basic Table layout:

noteId (identity field)
foreignId (integer / foreign key to other data)
noteText (text field)
... other unimportant data

ALTER Statements Before Starting: This adds a checksum and length based on the note. FYI - I did find duplicate checksums with different lengths. If you need to be precise, I suggest using the binary checksum function which is more sensitive, but understand, a checksum CAN have a collision. That means you can have the same checksum for different data.

ALTER TABLE notes ADD len_notes AS len(CAST(notetext AS Varchar(8000)))
ALTER TABLE notes ADD cs_notes AS checksum(CAST(notetext AS Varchar(8000)))
CREATE INDEX temp_index ON notes (foreignId,cs_notes,len_notes)

Now we have just added a column that represents the length of a note and the checksum of a note. If your data is changing, new records will contain NULL values and updated records will NOT recalculate the checksum, so keep that in mind. You may want to put criteria on your select statement that will only update the non-null values if your data changes, or the data that has a length <>

Select and Review part of your data that will be deleted:

SELECT top 100 n2.noteId, n2.cs_notes, n2.len_notes, n2.notetext
FROM notes n2
WHERE noteId in (
SELECT n2.noteId
FROM notes n1
JOIN notes n2
ON (n1.foreignId = n2.foreignId
AND n1.cs_notes = n2.cs_notes
AND n1.len_notes = n2.len_notes)
GROUP BY n2.noteId
HAVING n2.noteId > min(n1.noteId)
) ORDER BY n2.noteId

Delete your duplicates: To be clear here, this will preserve the lowest noteId and delete those with higher ids where the checksum AND length AND foreignId match.

DELETE from notes where note_id in
(
SELECT n2.noteId
FROM notes n1
JOIN notes n2
ON (n1.foreignId = n2.foreignId
AND n1.cs_notes = n2.cs_notes
AND n1.len_notes = n2.len_notes)
GROUP BY n2.noteId
HAVING n2.noteId > min(n1.noteId)
)

Put the table back the way it was:

DROP INDEX notes.temp_index
ALTER TABLE notes DROP column [len_notes]
ALTER TABLE notes DROP column [cs_notes]

I hope this helped.

We Need YOU!

2008-09-09T06:29:00.009-05:00

As a friend or family member - or even a stranger if you are reading this here, we are are asking for your help. A relative of ours was recently diagnosed with a highly aggressive form of lymphoma at the ripe old age of 33. His doctors at Baylor are working aggressively to eliminate the cancer, and part of his treatment is a stem cell transplant to rebuild his immune system (think of it as a system reboot after that nasty blue screen).

Unfortunately, we haven't been able to find a suitable donor yet. While there are millions of donors already registered, they have not found a "good enough" match. His doctors tell us that he's mostly likely to find a compatible donor in the area where we grew up, which happens to be in the Lincoln, NE area. Our family has set up a donor drive in conjunction with the the National Marrow Donor Program to get more on the donor registry and find a donor for him. Even if you suspect you are not a match for him, there are thousands of others who are also in his position (or will be).

We would appreciate very much if you would consider participating in the donor drive - or by passing the word along to someone who may be able to participate.
The drive will be Saturday, September 27, 2008 from 8am to noon
at the Southwood Community Center (5000 Tipperary Trail).
There is a small fee ($25) for the tissue typing. Minorities are currently free due to high demand and funding that is available.
We need an appx. head count in advance, so please send an email to donordrive @ kottwitz.org if you're planning to come. There are some businesses and people who are donating to the cause by offering prizes for registering as a donor.

You can get additional information about the drive, prizes, as well as all of those questions not already answered by visiting: http://donordrive.kottwitz.org By the way, the organization heading the donation drive is a 501(c)(3) just in case there was any worry about that. If you cannot attend, you can order a "typing kit" online for $52 - all costs are tax deductible either way and they also accept monetary donations for those who do not meet the health requirements.

Again, please share this information with others who may be willing to help. An official flier is on the home page of the website that was setup. If anyone is interested in posting the information at their office or church, they can reference that image or I can send you a copy.

Again, we are grateful to anyone willing to participate. Thank you and have a great day!

If you have questions you can also contact me via Grand Central - located on the right. (Please leave a message though)

Power of SQL: Indexing and sorting

2008-06-07T22:00:00.004-05:00

Coldfusion Muse recently made a couple of posts about clustered indexes on SQL Server. I was commenting off the grid to the Muse that the timely post helped me in solving a data import issue.

To protect the innocent, and make things less prone to narcoleptic episodes, I had the following scenario:
- An Access Database with Categories and Products.
- The First product listed in a category was a "special" product. Problem: There wasn't any information that would indicate that product was the first one.

Example Data:

category_id	product_id	product_name	...
1	2	Red Jumbo Ball	...
1	1	Green Jumbo Ball	...
1	3	Blue Jumbo Ball	...
2	4	Purple People Eater	...
2	6	Yellow Submarine	...
2	5	White Christmas	...

Once imported, I knew the data was at the whim of the database which meant that I needed to "preserve" the order prior to importing the data. I added another column "autonumber" on my products table in Access.

Once imported, I could set the clustered index to the "autoNum" column in SQL and know the sort order was correct! Of course, it never hurts to add an order by to your select statement. There are many factors that can affect sorting, clustered indexes are just one factor.

FFF CSS Menus Over Flex, Flash and Flash Forms

2008-05-07T11:06:00.010-05:00

Recently I came across the issue where I needed to change from an old menu design to a CSS drop-down menu. (Similar to the way File, Edit .... Help works).
The main application was Flash based content - which presented me with a problem. The issue is that the CSS content drops behind the flash content.

Fortunately, the solution is easier than I thought.

Add This to your Object tag:

<param name="wmode" value="transparent" />

If you use version detection, make sure you add it in there. It will look something like this:

AC_FL_RunContent(
...
"wmode","transparent",
...

If you still have Flash Forms for ColdFusion laying around, just add the following to your CFFORM tag:
wmode="transparent" (note that this goes in the tag, along with your format="flash" attribute.)

If others know of pitfalls or have a sexier solution, by all means - share. :)

Prism: Bridging the gap ...

2008-04-12T18:15:00.002-05:00

I've recently become aware of a desktop application called Prism. This application (6.66MB) in size installs as a desktop app, when you run it, it prompts you for the following information:

After you add the URL of the website, and give it a name you are good to go. From there, you have a shortcut icon that starts that site up every time - just like its a desktop application.

Why is this useful? Let's backup a bit ...
I have long been a fan of Meebo - as well as other sites like google documents. In fact, they are among the FIRST that I startup on any given day. Firfox Browser (with certain "startup" sites), Internet Exploer, Safari, Thunderbird, SQL Enterprise Manager, Query Analyzer, Eclipse ... but this also leads to an eventual problem. Firefox (my browser of choice) gets bloated and needs a restart every two or three days. While I love the fact that FF has so many developer tools, from the ability to add / edit cookies, to being able to inspect and modify the source code of a page using firebug - and even intercepting and tampering with page requests ... (we'll come back to this in a sec)

When I found meebo, I fell in love with it for a couple reasons:
1) I never have to update Trillian (or whatever software) again!
2) It supported everything I used.
3) I could use it at the library or on a friend's computer without hosing up his configuration.
Okay, for those that care, that was three and there are more reasons to like it, but I don't really think you care right now - let's tie this back into Prism.

What I didn't like about Meebo was the fact that when I closed my browser, it knocked me off the grid! It also hindered my development efforts when I was tampering data. (Read more about that when I finish writing the entry) Meebo continuously sends requests back and forth with the server - hence the concept of "instant" messaging. Google Mail or reader is just as bad, even if you don't have chat enabled on it. This is because of the way AJAX works - but this hindered the ability to get meaningful tampering / inspection done.

Moving these "web applications" off as Prism apps is the solution I've been looking for! I can close / reboot my browser (or even let it crash) without being annoyed that its dropping me from the grid, or interrupting that email to my Aunt. I can focus on developing applications without someone "pinging me" and interrupting me by having the tab flash their message (drawing valuable seconds away from my code). Speaking of rebooting the browser, at some point in the future ... I'm planning on featuring FFox and some tools I have found to be useful as a developer.

A few random thoughts about this:
- I don't like that there is only one icon for Prism, and it doesn't use the website icon (if there was one to begin with).
- I don't like the fact that ads pop up in a new prism window, but then again, if I wanted a site to pop up in a new window, this is helpful.
- There is a menu in the bottom-right corner to "Install to Desktop" - this puts the current page as an icon on your desktop, so if you were looking to "control" your kids browsing with this tool - forget it. They will just litter the desktop with thousands (or dozens) of shortcuts.
- This doesn't replace Adobe AIR, which allows for more feature-rich use of apps - like the ability to use the application offline. Prism is an on-the-grid-only application.
- If you setup the site as "https://" - it gives you the padlock in the bottom-right menu, but it doesn't allow you to inspect the security certificate. That's a bummer, but most people won't ever look at one, even though they should.
- If you setup an icon for Gmail on one account and setup an icon for Gmail for another account, they will NOT share the same session.
- Interestingly, if you setup an icon for Gmail and one for Google, they DO share the same session. If anyone knows how to make sure they do not share the same session on the same profile, fire away!

CFML to be Open Source (for Blue Dragon)

2008-03-11T09:00:00.004-05:00

Blue Dragon (A CFML server similar to Adobe's ColdFusion Server) has announced a dual-license where the basic CFML based Blue Dragon server will be open source. This is smilar to the way MySQL is! This means that for those who want to "pop the hood" and customize it (like Google did for their database) - they can.

I'll let you read details there, but the first code drop will be released "near" June 2008 - same time as CF United 2008!

There is no word yet as to whether this changes anything for Adobe, but in my mind, choices are a GOOD thing. Since the source code has not been released yet, right now this doesn't mean much (IMHO) other than a PR move for New Atlanta / Blue Dragon. Still, I will certainly be downloading and popping that hood myself when the option comes.

Updated: 03/14/2008 - Clarify what this means for the CF Community.

Power of SQL: Insert from Select

2008-03-07T16:14:00.005-06:00

Continuing the discussion of the Power of SQL. I came a situation where I wanted to move data from one table to another (without using DTS). How do you get the data moved over with just ONE SQL statement and no temp tables or complicated programming?

Consider the following example:

INSERT into foo2
(col, col2,col3)
select col,col2,col3
from foo1
where destination='foo2'

I hope that helps!

Excel file with CFcontent

2008-03-06T22:25:00.005-06:00

It is extremely easy to give your customers access to an Excel file containing their data.

The key is a nicely formatted HTML table, with correctly placed cfheader/cfcontent tags.
Consider the following example:

<cfsetting showdebugoutput="no">
<cfheader name="Content-Disposition" value="attachment;filename=""My File Name.xls""">
<cfcontent type="application/msexcel;charset=utf-16">
<table>
<tr>
<th>Color</th>
<th>Code</th>
</tr>
<tr>
<td>red</td>
<td>#ff0000</td>
</tr>
<tr>
<td>green</td>
<td>#00ff00</td>
</tr>
<tr>
<td>blue</td>
<td>#0000ff</td>
</tr>
</table>
<cfabort>

If you have any other tips or ideas feel free to share them!

Welcome Home Atlantis

2008-02-20T08:07:00.004-06:00

Just a quick note to welcome home the Atlantis crew!

STS-122 Quick Stats...
121st mission
24th flight for Atlantis
5.3Million miles

... priceless!

Power of SQL

2007-12-19T08:33:00.001-06:00

Update: This is a part of a series now called "Power of SQL".

As a former developer at an elite government organization, I had the opportunity to leverage some of this SQL functionality. I had the opportunity to work on a project where they had a text file dump of Driver's license information. They also had a file dump (Select * from all_employees_of_all_employers_in_the_state). These statistics could show that a certain age, sex, ethnicity or even height of a person could help determine the wage of a person for a certain industry. If there was a similar correlation that short people get paid less than tall people, in a similar way that women get paid less than men, that could be a huge discovery to the HGH Industry - but I digress.

This program was to tie some "statistical" driver's license information collected by the state DMV to the wage information already collected under existing labor laws by NWD-DOL. After the program ties this information together OIT was then supposed to make sure the program would remove the "identifying" information (the coveted SSN), and then pass it along to the appropriate department (LMI). The program was inefficient and needed to have it optimized.

I opened the code up and like many programs I worked on, I was not terribly shocked by what I saw. Here is the pseudo-code:

Select * from wage_info Into wages
Select * from dmv_info into dmv
BEGIN LOOP dmv
-- The dmv clearly shows that SSNs are not unique and are recycled.
IF dmv.deceased = 0
BEGIN LOOP wages
IF wages.ssn = dmv.ssn
insert into dmv_wage_match
/IF
END LOOP
/IF
END LOOP
Select * from dmv_wage_match into obfuscate
Set id = 0
BEGIN LOOP obfuscate
set id = id+1
update dmv_wage_match
set ssn = #id#
where ssn = #obfuscate.ssn#
END LOOP

So let me break this down for you. If there are 1,200,000 DMV records and 900,000 wage records, with a 750,000 of them that match ... well, you get the idea, the program was bloated.

Here's what I updated the code to:

INSERT INTO dmv_wage_match
select d.sex, d.height, d.weight, d.eye, d.hair, w.amount, w.industry_code ...
from dmv_info d
join wage_info w
on d.ssn = w.ssn
where d.deceased = 0

In this case, dmv_wage_match had an identity field that served to obfuscate the actual SSN. This also elegantly avoids the tedious insert / update that was being done in the previous version. This version ran in a few seconds as compared to the previous version that took almost an hour! Now that's an improvement.

This is a VERY simple example, but I continue to be amazed by the power of SQL. The key in my opinion though is knowing what SQL can do and then asking the right question, for example, "How can SQL do some of the heavy lifting for me?".

I hope this helps give some insight into those who are still using a spreadsheet or *tisk tisk* text files and then trying to do something meaningful with them. Yes, they hold data, but so does the phonebook. If you're not convinced yet, tell me, by glancing through the phonebook, what are all of your neighbor's names and phone numbers who share your street name? Please post back how long that took you, I'm sure the readers would be interested in knowing.

Barcodes in ColdFusion

2007-12-17T08:20:00.000-06:00

One of the brilliant minds that I have the pleasure to work with at CF WebTools, Ryan, has just released information on generating and reading 2D barcodes in ColdFusion. I won't steal his thunder by writing about it in too much detail here, read about it at Stillnet Studios.
I believe this example needs ColdFusion 8, but please post back if anyone has accomplished this with earlier versions, we'd love to see the CFC extended!

Thank you Ryan for sharing this with us!

Phishing - Part I

2007-11-15T22:01:00.000-06:00

This is Part I in what I hope will be a muli-part series on phishing.

Here is a brief quiz on phishing that I came across this evening.

I'm happy to report I got 5 out of 5 correct and have been labeled as a "PayPal Anti-Phishing Champion" ... now why do I not feel all warm and fuzzy inside? The reality is that we are all at risk in some way or another, from a simple typo in the URL to clicking a link in email, because - well, lets face it - its easier that way! There's a lot more to cover here, stay tuned!

ColdFusion 8 hotfix 2

2007-11-15T16:55:00.002-06:00

I just realized that ColdFusion 8 hotfix 2 is now available! Details there.

This is typical - I just did a new install of a download from Monday.

Now back to your regularly scheduled development.

Unintended Consequences

2007-11-09T08:03:00.000-06:00

I have found in my meager and short development career that even the easy things may not be so easy or obvious. Simple "user experience enhancements" that Microsoft has added are not really enhancements when you believe your site is saying "Please dick here", that is clearly an unintended consequence.

It all started when I turned on my machine and was alerted to the fact that Real Player has a security flaw. Okay, things happen and my hat tips to any company who fixes and alerts / nags the end-user to update.

The unfortunate thing here (as I saw it) were the following two things:

1) The patch tried to launch Real Player and initiate the update, but Real Player Froze. I personally was able to "patch" this flaw by uninstalling Real Player. HA! (Off-topic here, so I won't go any further with that).

2) The text "PLEASE CLICK HERE" immediately grabbed my attention and I thought I must have hit a phishing site or some other malware site. (more on phishing and malware another day). It appeared that the site was saying "Please dick here ..."!!

Here is what I saw in Internet Explorer:

I was dumbfounded when seeing this and out of curiosity, I looked in Firefox to see if there was difference on the font, there was:

It is subtle, but you can certainly see a space between the "c" and the "l" on "click" in Firefox. After looking into this, I determined this is because of Microsoft's anti-aliasing attempts to make text more readable. To tweak this you should and get the best "user experience" visit the Windows ClearType Tuner page.

Good luck and safe browsing.