Bring a pot of water to a boil and season well with salt. Add the potatoes and boil until tender. Drain in a colander, then transfer to a mixing bowl with the oil and a good amount of salt and toss well. Lay the potatoes out on a sheet tray in one layer (reserve the bowl) and bake for 30 minutes.
After 30 minutes, transfer the potatoes back to the reserved bowl and toss well again, adding a little more olive oil if you're feeling generous. Pour them back onto the sheet tray (reserve the bowl) and bake until they are golden and crispy but not burnt, about another 30 minutes.
Add the mayonnaise, paprika and garlic to the same bowl, then pour in the potatoes and toss until the potatoes are well coated. Serve in that same bowl as finger food.
Using baby or fingerling potatoes for this recipe produces the best results, but really any potatoes will do (even sweet potatoes).