Heat the oil in a large saucepan over medium heat. Add the leeks and garlic and cook, stirring, until softened, about 3 minutes. Add the tarragon and cook 1 minute, until fragrant. Add the potatoes and broth and bring to a simmer. Partially cover the pan and simmer until the potatoes are fork-tender, 8 to 10 minutes. Stir in the salmon and milk and simmer 2 minutes to heat through. Remove from the heat, season to taste with salt and pepper, and serve.