In a deep-fryer or heavy-bottomed pot, heat enough oil to come halfway up the sides of the pot, to 350 degrees F.
Add the potatoes to the hot oil and fry until golden and crisp, about 5 minutes. Remove from the fryer to a paper towel lined sheet tray to drain. Season with salt and pepper, to taste.
While the potatoes are frying, mix together the mustard, mayonnaise, dill, and capers in a small bowl.
Add the fried potatoes to a large serving bowl. Toss in the red onion and celery, then gently fold in the mayonnaise dressing. Season with more salt and pepper, if needed, and serve.