Django not recognizing my Token?

I'm using Django Rest Framework on the backend and an ember-cli app on the frontend. The authentication is working correctly, but there seems to be a hole somewhere in authorization.

# settings.py
REST_FRAMEWORK = {
    'DEFAULT_PERMISSION_CLASSES': (
        'rest_framework.permissions.AllowAny',
    ),
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.TokenAuthentication',
    ),
}

# views.py
class CurrentUserView(APIView):
    "get the data for the current authenticatd user"
    permission_classes = (IsAuthenticated,)

    def get_queryset(self, request):
        queryset = User.objects.filter(username=request.user.username)
        return queryset

    def get(self, request):
        serializer = UserSerializer(request.user)
        return Response(serializer.data)

When I make a request to this endpoint /v1/me/ it responds back with a 403. When I take off the permission class, I get back {"id":null,"username":"","is_active":false} because it doesn't know who I am.

Also, when I use the browsable API, the /v1/me URL works correctly.

On the Ember side of things, I login with my account and correctly get back my Token. In the request the Authorization: Token asdf1234asdf1234asdf1234 is being passed. I would think Django takes that token and knows who I am? Am I missing something?