How x[, 'colname1'] is implemented?

I'm not complaining that it is not documented.

Yes but you didn't answer my question. When you ask a question on
these (or any mailing lists) you should always say what efforts you've
made to answer the question. I first looked in the help for '[' but
didn't find anything there. If you had already done that and told us
then I wouldn't have wasted my time. If you'd gone on to say you'd
also looked in the source code, and in which files, then I wouldn't
have wasted my time looking in those files. Note that this is all for
your benefit as well because we'll be able to help you quicker!

?As Obi-wan Kenobi may have said in Star Wars: "Use the source, Luke!":

Could you explain what ns and nx represent?

No, you need to take Obi-wan's advice (or Seth Falcon's, who got to
this before me) and 'Use the source'. I would hope that since you have
an understanding of what a hashing algorithm is that you should also
have some understanding of programming, and even if not in C it's not
too hard to figure out.

 You could make the effort to look in subscript.c and work it out for yourself.

 If you are very interested in the low-level working of R then you
should compile R with debugging turned on, then run R in a debugger
and set a breakpoint in the array subscript function. Then you can
inspect the C variables when you run it.

Barry