fisher.test - can I use non-integer expected values?

Expected values are needed to test a null hypothesis against observed
counts, but if total observed counts are 20 for 3 categories, then a null
hypothesis of a random effect would use expected values = 6.67 in each of
the 3 categories (20/3).

Yes, fisher.test is for count data and so is chisq.test, but chisq.test
allows 6.67 to be input as expected values in each of 3 categories, while
fisher.test does not seem to allow this?

To the best of my knowledge (which may be limited) you never put
expected counts as input in Fisher Exact Test, you need to put actual
observed counts. Fisher test tests the independence of two different
random variables, each of which has a set of categorical outcomes.

From what you wrote it appears that you have only one random variable

that can take 3 different values, and you want a statistical test for
whether the frequencies are the same. You can use chisq.test for this
by specifying the probabilities (argument p) and running it as a
goodness-of-fit test. I am not aware of goodness-of-fit way of using
fisher.test.

If you actually have two different variables, one of which can take
two values and the other one can take 3 values, you need the actual
observed counts for each of the 6 combinations of the two variables.
You put these counts into a 2x3 table and supply that to fisher.test
or chisq.test.

I don't think it is inherent in Fisher's exact test itself that expected
values must be integers, but not sure.

I think it is inherent in Fisher's Exact test. The test makes certain
assumptions about the distribution of the numbers you put in. If you
put in non-integers, you necessarily  violate those assumptions and
the test is then not applicable.

Peter