First you need to have Greasemonkey for Firefox. After installing the script (a simple click on the link), you need to go to Tools / Greasemonkey / Manage scripts, find Google Reader Custom Search in the list and click Edit.
You'll see some instructions in the text file. Basically, you need to search for "test" using your custom search engine, copy the URL, remove "&q=test" from the address and put the result in SearchURL variable. You'll get something like:
Update 2: John Marshall wrote a better script that shows the search results inside Google Reader.