Labelling a fortified GADM map plotted with ggplot and geom_map

Reffered here from the?R-Help mailing list:

I am having trouble getting data labels to display over the provinces in a GADM map of Canada.
Specifically, I need the variable "Number" from the data set "by_province", grouped by "region", to appear on the corresponding regions of the map.

The data set "by_province" looks like this:

long? ? ? lat? ? ? ? ? order? hole piece ? ? ? ? ? ? ? ? ? region ? ? ? ? ? ? ? ?group ? ? Number
-110.37074 60.00006 ? ? ? 1 FALSE ? ? 1 ? ? ? ? ? ?Alberta? ? ? ? ? ? Alberta.1? ? 132
-110.36250 60.00006 ? ? ? 2 FALSE ? ? 1 ? ? ? ? ? ?Alberta? ? ? ? ? ??Alberta.1? ? 132
-110.35103 60.00006 ? ? ? 3 FALSE ? ? 1 ? ? ? ? ? ?Alberta? ? ? ? ? ? Alberta.1? ? 132

and the data set "tract" is the map data without the "Number" variable.

I looked into working this out from by researching the geom_map function here: http://ggplot2.tidyverse.org/reference/geom_map.html

my code looks like this:
# get the raw map data
can_map <- getData('GADM', country = "CAN", level = 1) # download map level with provinces
tract <- fortify(can_map, region = "NAME_1") # transforms data from shapefiles into a dataframe that ggplot can understand, from http://www.kevjohnson.org/making-maps-in-r/

# create subsets of the kcj data if needed, for example by year
kids_data_2017 <- subset(kids_data, year == 2017) # data for the year 2017
kids_data_2018 <- subset(kids_data, year == 2018) # data for the year 2018

# extract the needed data
kids_province_data <- data.table::data.table(kids_data_2017$Province_Territory__c, kids_data_2017$Number_of_kids__c)
names(kids_province_data)[1] <- "Province"
names(kids_province_data)[2] <- "Number"

# sum the data by province
kids_province_sums <- aggregate(.~Province, data = kids_province_data, sum)

# join the data to the map
names(tract)[6] <- "region"
names(kids_province_sums)[1] <- "region"
by_province <- left_join(tract, kids_province_sums)

# create the data map
kids_map <- ggplot(by_province, aes(map_id = region)) +? ? ? ? ? ? ? #plots the map in by_province separating by region
?? ? ? ? ? geom_map(aes(fill = Number), ? ? ? ? ? ? ? ? ? ? ? ? ? ? #generates aestheticsa for the plot
? ? ? ? ? ? ? ? ? ? map = tract,? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? #takes the data from tract
? ? ? ? ? ? ? ? ? ? color = "#ffffff",? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? #makes the color of the borders between regions white
? ? ? ? ? ? ? ? ? ? size = 0.15) +? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? #sets the thickness of the boarder lines
?? ? ? ? ? coord_map("polyconic") + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? #sets the coordinates to polyconic (rounded lat and long)
?? ? ? ? ? scale_fill_gradient(name = "Children Reached", ? ? ? ? ? #sets the gradient of the value scale: names the scale
?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? low = grey_2,? ? ? ? ? ? ? ? ? ? ? ? ? ? #color of the low end
?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? high = orange_1) + ? ? ? ? ? ? ? ? ? ? ? #color of the high end
?? ? ? ? ? expand_limits(x = tract$long,? ? ? ? ? ? ? ? ? ? ? ? ? ? #ensure limits include all values for all plots
?? ? ? ? ? ? ? ? ? ? ? ? y = tract$lat) +
?? ? ? ? ? labs(x = NULL, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? #add labels, no x title
? ? ? ? ? ? ? ? y = NULL, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? #no y title
? ? ? ? ? ? ? ? title = "Number of Children Reached by Province", ? ? ? #map title
? ? ? ? ? ? ? ? subtitle = "2017") +? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? #map subtitle
?? ? ? ? ? geom_text(data = by_province,? ? ? ? ? ? ? ? ? ? ? ? ? ? #add a text layer
?? ? ? ? ? ? ? ? ? ? aes(long,? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? #aethetics for the text, x axis
?? ? ? ? ? ? ? ? ? ? ? ? lat, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? #y axis
?? ? ? ? ? ? ? ? ? ? ? ? label = Number,? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? #the value to display
?? ? ? ? ? ? ? ? ? ? ? ? size=3)) + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? #size of the text
?? ? ? ? ? theme(axis.ticks = element_blank(),? ? ? ? ? ? ? ? ? ? ? #theme of the graph, no axis ticks
?? ? ? ? ? ? ? ? axis.text = element_blank(), ? ? ? ? ? ? ? ? ? ? ? ? ? #no axis text
?? ? ? ? ? ? ? ? panel.background = element_blank(),? ? ? ? ? ? ? ? ? ? #blank background
?? ? ? ? ? ? ? ? panel.border = element_blank())? ? ? ? ? ? ? ? ? ? ? ? #blank border

# save as png
ggsave(kids_map, file = "kids_map.png", width = 6, height = 4.5, type = "cairo-png?)

I have asked this question on stack overflow, and was refered to this answer:
https://stackoverflow.com/questions/9441436/ggplot-centered-names-on-a-map

The solution there did not fix my problem, though it did get me closer.
The solution on that post is using a single vector of labels.

My post on stackoverflow has images of my output:
https://stackoverflow.com/questions/49118323/labelling-a-map-plotted-with-geom-map