Make the Cream Puffs: Preheat the oven to 425 degrees F.
Combine the water, butter, salt, and sugar in a large saucepan and bring to a rolling boil over medium-high heat. When it boils, immediately take the pan off the heat. Stirring with a wooden spoon, add all the flour at once and stir hard until the flour is incorporated, about 30 to 60 seconds. Return the pan to the heat and cook, stirring, 2 minutes to evaporate some of the moisture.
Scrape the mixture into a mixer fitted with a paddle attachment (or use a hand mixer or mix by hand.) Mix at medium speed. With the mixer running, add 3 of the eggs, 1 at a time, stopping after each to scrape down the sides of the bowl. Mix until the dough is smooth and glossy and the eggs are completely incorporated. The dough should be thick, but should fall slowly and steadily from the beaters when you lift them out of the bowl. If the dough is still clinging to the beaters, add another egg and mix until completely incorporated.
Using a pastry bag, pipe the dough in bite-size "kisses" (about 1 1/2 inches across) onto 1 of the cookie sheets.
Dip your finger in flour and push down the points to smooth out any bumps. Bake 15 minutes, then reduce the heat to 375 degrees F and bake until puffed up and light golden brown, about 25 minutes more. Try not to open the oven door too often during the baking. Let cool on the baking sheet.
Make the Caramel: Pour the sugar into the center of a deep saucepan. Carefully pour the water around the sugar, trying not to splash any sugar onto the sides of the pan. Do not stir; gently draw your finger through the center of the sugar to moisten it. Over high heat, bring to a full boil and cook without stirring, swirling the pot occasionally to even out the color, until amber-caramel in color, 10 to 20 minutes. When the mixture is done, remove the pot from the heat and rest the bottom in the ice bath for 15 seconds to stop the cooking.
Dip the top of each cream puff in the caramel and place it caramel-side down, with the base sticking straight up, on the other greased cookie sheet. The caramel will pool slightly around the puff and will harden and develop a foot. (When you turn the puffs over later, this will be a smooth flat "lid" for each profiterole.) Let cool until set, at least 10 minutes.
When ready to serve, carefully turn the puffs over and cut each 1 horizontally in 1/2 using a long serrated knife. Fill each with a small scoop of ice cream or sweetened, flavored whipped cream. Place the lid on top and serve immediately.