Melt 2 tablespoons of the butter in a cast-iron skillet over medium heat. Add the celery, carrot, shallot, chicken bouillon, celery seeds and 1/4 teaspoon salt and cook, stirring occasionally, until softened, about 7 minutes. Transfer to a large bowl and let cool to room temperature.
Meanwhile, melt the 2 remaining tablespoons butter in a small saucepan over medium heat. Add the flour and garlic powder and cook, whisking, 1 minute. Gradually add 1 cup milk and bring to a simmer, whisking constantly. Continue to simmer, whisking constantly, until thickened, about 2 minutes. Add the peas, 1/2 teaspoon salt and a generous amount of pepper. Take off the heat and keep warm, covered.
Mix the panko into the cooked vegetables and mix in the remaining 2 tablespoons milk; let stand 5 minutes. Add the ground chicken, 1 teaspoon salt and a few grinds of black pepper. Form into four 3/4-inch-thick patties. Spray the cast-iron skillet and patties with nonstick spray. Heat the skillet over medium-high heat, then place the patties in the skillet sprayed-side down and spray the tops of the patties. Reduce the heat to medium, then cover and cook for 2 minutes. Flip the patties and cook, uncovered, until cooked all the way through and the centers are firm, about 2 more minutes. Serve on buns topped with the pea gravy and with extra gravy on the side.