For the spinach and cheese mixture: Preheat the oven to 375 degrees F. Grease a 13-by-9-inch baking dish with cooking spray or butter.
In a large dry skillet over medium heat, lightly toast the pine nuts. Remove and reserve. In the same skillet, melt the butter with the olive oil. Add the onions and saute until translucent, about 10 minutes. Add the garlic and saute 2 to 3 minutes. Stir in the spinach. Cook until tender and any excess water evaporates, about 5 minutes. Remove from heat and let cool.
In a large bowl, mix the eggs, feta, ricotta, Parmesan, pine nuts, pepper, salt and nutmeg. Add the reserved spinach mixture and stir to combine. Transfer to the baking dish, pressing to make sure it is even.
For the phyllo dough crust: Layer sheets of phyllo onto the spinach mixture, brushing each sheet with melted butter before adding the next. Fold the edges to make a crust and brush with butter. Sprinkle with za'atar or sesame seeds. Bake until golden brown, 55 to 60 minutes.