Preheat the oven to 400 degrees F.
Combine the vinegar and anchovy paste in a small bowl and add salt and pepper, to taste. Whisk until well blended. Whisk in the olive oil. Stir in the olives, capers, shallot, garlic, and oregano.
Arrange the fillets skinned sides down in a baking dish large enough to hold the fish fillets in 1 flat layer. Season well with salt and pepper. Spoon a small amount of the vinegar and oil mixture over each fillet and drizzle the rest around the bottom of the dish. Cover with foil, and bake until opaque throughout and firm to the touch, 25 to 30 minutes. Transfer to warm serving plates and spoon on any juices in the bottom of the baking dish. Serve warm.
Recipe courtesy of Sara Moulton Cooks at Home, Broadway Books, 2002