This article on infoq talks about story point and its relation to complexity or time. The argument seems to be it should be only related to effort and not complexity
When team members say Extra large for a story, generally it can mean one of the following.
a) I don’t have all the information needed and hence I think it is large ( This is not complexity or effort. This is the ambiguity or uncertainity. In this case we cannot really base the size on Effort as effort can only come to play when the team member knows that he or she needs to do
b) I know all about the story but this is complex. This is a good case to say this is complexity or more so effort. Its going to take me 10 days to do it as an example.
Story sizing is all about relative measure.
The real effort only comes to play when you work on story. At times what is a small story becomes extra large , the effort grows. So there should be not be a correlation to time alone. If we did time alone then why do we have to do task breakdowns.
Estimate size and derive duration. This picture captures the intent.