Sacrificing to any other god is worthy of death. This is a common command throughout Scripture
Keep in mind that the demand from Pharaoh is to let the people serve Elohim. Elohim did not demand total release for Israel from Pharaoh at this time. The demand was only to let them go a three days journey to make sacrifices to Elohim.
They want to dwell in the land of Goshen. Joseph does not want them to stay in the palace of Pharaoh or even in the heart of Egypt. This appears to be an intentional act of Joseph. He knows that his father and brothers have a much bigger part of the bigger plan. They are not to become too “acquainted” with the idea of staying in Egypt.