Page-Refresh in a SPA with HTML5 Browser History API?

Instead of using hash URLs for my SinglePageApplication I would like to use the HTML5 Browser History API. This works fine so far (I'm using emberjs).

The pro of the hash URL is, I can refresh the page without worrying about the result. When I try to reload with the browser history API I'll get a 404 (since my server don't have a route for this).

So: Is there a common workaround to avoid this behaviour?

I could tell my server to redirect to the index page but this wouldn't show me the correct content.

Example:

GET host/#images
... shows the "images" page (even when it is the index.html)

GET host/images
... redirect to host/
... shows me the content of the index template