As I noted a few months ago, the Windows Installer is a declarative installation engine not a procedural installation engine. That means if you are going to extend the installation engine (which is what custom actions essentially do) then you should follow the same principles the engine follows. Namely, your custom action should read data out of a table and act based on those declarative instructions.
Let me give an example. Let's say you want to protect some user input (maybe a password) using DPAPI.* Your first thought should be, "What are the inputs to my custom action?" Those inputs are the columns for your custom action's table. In this case, one column would contain the value to be encrypted (Formatted so the value of a Property could be encrypted). Another column would provide the optional entropy value (that helps further randomize the protected output). A third column would specify whether the value will be protected for the current user or as the local machine account (limiting the set of users that can decrypt the data). A final column would specify the name of the Property to output the encrypted value to.
Then you would write your custom action to read each row out of the table with those for columns and call the ::CryptProtectData() API filling in the appropriate parameters. By acting on the data from the custom table, we call your custom action a "data driven custom action".
Those of you whose first instinct was to say, "Oh, I'll just write the code to get the value of a property called FOO and call the ::CryptProtectData() function with a bunch of parameters and write the value back out to a property called BAR." should go back an reevaluate all of the custom actions you've ever written. This type of custom action is what we call a "procedural custom action" and it is inferior to the data driven method. Unfortunately, to most developers procedural custom actions are also the most intuitive way to write code (because that's how most code is written).
Data driven custom actions are better than procedural custom actions for many reasons. as noted in the beginning, data driven custom actions follow the same declarative model that the rest of the Windows Installer follows. That means you fit into the rest of the installation system appropriately
One example is that data driven custom actions can be affected by transforms. You might think, "Oh, well I don't support administrative customizations so I don't care about transforms." That might be true but the most optimal form of patching uses transforms to affect the package. If your custom actions are not data driven, your transform must patch the custom action binary directly and that creates a much larger patch.
Data driven custom actions are also reusable. Another developer can use your custom action without modifying the code because the inputs all from from a table. The custom action library that comes with the WiX toolset works just like this. We write the actions to configure things like Users, FileShares, Web Sites, Virtual Directories, SQL Databases, etc and you can reuse those actions by providing your own data. Data driven custom actions paired with the extension model in WiX provides a very powerful platform for installation.
Data driven custom actions are also more transparent than procedural custom actions. Going back to the WiX toolset Web Site example. If you look at the tables in the MSI after adding a WebSite element it is very possible to predict what the installation is going to do. Procedural custom actions hide the data and the actions in the compiled code (you don't write script custom actions do you?). Admittedly, the data driven custom actions still have code in a black box but well written data driven custom actions can be quite transparent. Take a spin through the WiX custom action library if you don't believe me.
Now I wanted to say that *all* custom actions should be data driven from a table but there are some rare cases where it doesn't make sense. The custom action at the end of the install that launches your "readme.txt" through a ::ShellExecute() may just need a simple property passed to it (that's how the WixShellExec custom action works today).
However, any custom action that modifies system state (see #3 in my list of custom action classes) should definitely be a data driven custom action to participate in the installation transaction well. After I do an example of a data driven internal-only custom action (#1 from the list), I'll dive into system modification custom actions and further demonstrate why they should be data driven.
Anyway, that's a long blog entry that basically says, "If you have already failed and determined that you need a custom action, make sure you make it data driven."
* Yes, I am going to walk step by step through this example at some time in the future.