Thanks for the request for feedback on this change. I share Josiah's view that "In the case of an isolated node, I think the best thing to do is to return NA rather than 0. For example consider administrative boundaries where the lagged variable is median household income. If we returned a 0 in that case, we'd likely be introducing quite the outlier!" Currently geostan's Moran scatter plot uses NA, so observations with no neighbors are neutral, i.e., they are dropped when calculating Moran's I or creating a Moran scatter plot. It also shows a kind of confusing error message about how this impacts a row-standardized spatial weights matrix---if you just drop one or more observations, the weights are no longer row-standardized relative to the values included in the calculation. Also, "creating nb neighbour objects should themselves report that an output object (graph) is not connected"---that's a good suggestion. Thanks, Connor
From: R-sig-Geo <r-sig-geo-bounces at r-project.org> on behalf of r-sig-geo-request at r-project.org <r-sig-geo-request at r-project.org>
Sent: Monday, November 6, 2023 5:00 AM
To: r-sig-geo at r-project.org <r-sig-geo at r-project.org>
Subject: R-sig-Geo Digest, Vol 243, Issue 3
Sent: Monday, November 6, 2023 5:00 AM
To: r-sig-geo at r-project.org <r-sig-geo at r-project.org>
Subject: R-sig-Geo Digest, Vol 243, Issue 3
Send R-sig-Geo mailing list submissions to
r-sig-geo at r-project.org
To subscribe or unsubscribe via the World Wide Web, visit
https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-geo&data=05%7C01%7Cconnor.donegan%40utdallas.edu%7C981c03aaa25a43db669d08dbdeb7b093%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C638348652754639883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Zv6cmTUDnTT3WBgX%2Bwg%2FNFc%2Bx8BK84YatH%2Fjalf6grY%3D&reserved=0<https://stat.ethz.ch/mailman/listinfo/r-sig-geo>
or, via email, send a message with subject or body 'help' to
r-sig-geo-request at r-project.org
You can reach the person managing the list at
r-sig-geo-owner at r-project.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of R-sig-Geo digest..."
Today's Topics:
1. Re: spdep: new zero.policy attribute (Roger Bivand)
2. Re: spdep: new zero.policy attribute (Josiah Parry)
----------------------------------------------------------------------
Message: 1
Date: Sun, 5 Nov 2023 15:31:16 +0000
From: Roger Bivand <Roger.Bivand at nhh.no>
To: "r-sig-geo at r-project.org" <r-sig-geo at r-project.org>
Subject: Re: [R-sig-Geo] spdep: new zero.policy attribute
Message-ID:
<SV0P279MB0475841DFE21EE66FBCBD252EEABA at SV0P279MB0475.NORP279.PROD.OUTLOOK.COM>
Content-Type: text/plain; charset="iso-8859-1"
And a question: in nb2listw() and similar functions creating spatial weights listw objects, would it be sensible to guess that the presence of no-neighbour observations in the input nb neighbour implies the choice of a spatially lagged value of zero (zero.policy=TRUE), lx = Wx, rather than NA (zero.policy=FALSE)?
That is, use by default zero.policy=any(card(nb) == 0L) rather than zero.policy=NULL and look in the spdep option set by default on package load to FALSE but settable by the user?
Would this be taking trying to be helpful too far, given that the analyst is creating the neighbour object and presumably should take responsibility for choices made?
Context: polygons not sharing boundaries with other polygons do exist legitimately in data sources, but setting spatially lagged values to zero for those polygons is quite an invasive imputation. It may be better to oblige the user to make the choice when the spatial weights listw object is created.
Little is known about the problem, for a recent treatment for CAR models see: https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Farxiv.org%2Fabs%2F1705.04854&data=05%7C01%7Cconnor.donegan%40utdallas.edu%7C981c03aaa25a43db669d08dbdeb7b093%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C638348652754639883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=G8%2B714lKy1JUjezV0C%2FlRmSGYsq8WCj0iPkwk%2FQjMQI%3D&reserved=0<https://arxiv.org/abs/1705.04854>, published as https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoi.org%2F10.1016%2Fj.sste.2018.04.002&data=05%7C01%7Cconnor.donegan%40utdallas.edu%7C981c03aaa25a43db669d08dbdeb7b093%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C638348652754639883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nUkXfGSH%2BX8Y0RJSL1MTch67EdzW861xbHZiWuQRJ9w%3D&reserved=0<https://doi.org/10.1016/j.sste.2018.04.002>, where: "The specification of a CAR model on a disconnected graph is undefined ... [t]here are essentially two types of disconnected graphs: first, a graph containing an island (a singleton node with no neighbours), second, a graph split in different sub-graphs (each of them being a connected graph)".
This question concerns the former, singleton, case, but adding sub-graph counts if greater than unity to summary.nb and print.nb address the second . Very possibly, functions creating nb neighbour objects should themselves report that an output object (graph) is not connected, bigDM CARBayes CARBayesST geostan spatialreg stampr do call spdep::n.comp.nb themselves to check the subgraph count.
Interested in feedback,
Roger
--
Roger Bivand
Emeritus Professor
Norwegian School of Economics
Postboks 3490 Ytre Sandviken, 5045 Bergen, Norway
Roger.Bivand at nhh.no
________________________________________
From: R-sig-Geo <r-sig-geo-bounces at r-project.org> on behalf of Roger Bivand <Roger.Bivand at nhh.no>
Sent: 04 November 2023 18:53
To: r-sig-geo at r-project.org
Subject: [R-sig-Geo] spdep: new zero.policy attribute
In forthcoming spdep 1.3-1, spatial weight listw objects get a new zero.policy attribute. The attribute is added as objects are created to record the status of the zero.policy argument in the function creating the object, see: https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fr-spatial%2Fspdep%2Fcommit%2Fe159de922c61713529a4075b0dfc2966eb8f9ad6&data=05%7C01%7Cconnor.donegan%40utdallas.edu%7C981c03aaa25a43db669d08dbdeb7b093%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C638348652754639883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=uy25DfhupOiu78MrTTjDn6nMPUDkHdYehSHimuMVoJk%3D&reserved=0<https://github.com/r-spatial/spdep/commit/e159de922c61713529a4075b0dfc2966eb8f9ad6>.
Reverse dependency checks only show problems from over-eager unit testing in SpatialFeatureExperiment, a Bioconductor package, but other workflows may be impacted. The new attribute is used in tests for spatial autocorrelation to set the zero.policy argument in those tests (the arguments were zero.policy=NULL, are now zero.policy=attr(listw, "zero.policy") where listw is the spatial weights object argument to the test function.
This will be extended to spatialreg and friends if nobody reports negative impacts here soon. I'll wait before releasing 1.3-1 for a few days to see if any feedback is forthcoming.
Hope this long-overdue change is helpful,
Roger
--
Roger Bivand
Emeritus Professor
Norwegian School of Economics
Postboks 3490 Ytre Sandviken, 5045 Bergen, Norway
Roger.Bivand at nhh.no
_______________________________________________
R-sig-Geo mailing list
R-sig-Geo at r-project.org
https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-geo&data=05%7C01%7Cconnor.donegan%40utdallas.edu%7C981c03aaa25a43db669d08dbdeb7b093%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C638348652754639883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Zv6cmTUDnTT3WBgX%2Bwg%2FNFc%2Bx8BK84YatH%2Fjalf6grY%3D&reserved=0<https://stat.ethz.ch/mailman/listinfo/r-sig-geo>
------------------------------
Message: 2
Date: Sun, 5 Nov 2023 12:01:38 -0500
From: Josiah Parry <josiah.parry at gmail.com>
To: Roger Bivand <Roger.Bivand at nhh.no>
Cc: "r-sig-geo at r-project.org" <r-sig-geo at r-project.org>
Subject: Re: [R-sig-Geo] spdep: new zero.policy attribute
Message-ID:
<CAL3ufULghGnSos1J1kMVGVVWdhiawoz+ec=QbraQRY-x2h=kgg at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
My take is generally that less is more. In the case of an isolated node, I
think the best thing to do is to return NA rather than 0. For example
consider administrative boundaries where the lagged variable is median
household income. If we returned a 0 in that case, we'd likely be
introducing quite the outlier!
My preference would be to _always_ default to NA lagged values when no
neighbors are present. Additionally, it would be quite nice to instruct
users on how to impute these values with other lagged variables. Say we
have an isolated node, but we don't want an NA value. We can impute that
missing value with the spatial lag of the variable using a different
neighborhood construction?e.g. using KNN with k = 3 to *ensure* that the
node always has values to lag.
Here is an example gist imputing k=3 lag
https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.github.com%2FJosiahParry%2Feb7878fc375fb931ddd6675a2c591a2b&data=05%7C01%7Cconnor.donegan%40utdallas.edu%7C981c03aaa25a43db669d08dbdeb7b093%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C638348652754639883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FcHiYgOC3o0dU1NqiP4SbHH1B4JEejGvfi1qyO62iWM%3D&reserved=0<https://gist.github.com/JosiahParry/eb7878fc375fb931ddd6675a2c591a2b>
Another option could be to use the focal feature's value *as* the lag
itself. If a location has no neighborhood could we argue that the location *is
*its own neighborhood?
I'm not too familiar with the models to comment on them, though. I do see
this more as a pre-processing / data imputation issue. I suspect that's
more of a machine learning paradigm though!
On Sun, Nov 5, 2023 at 10:31?AM Roger Bivand <Roger.Bivand at nhh.no> wrote:
> And a question: in nb2listw() and similar functions creating spatial
> weights listw objects, would it be sensible to guess that the presence of
> no-neighbour observations in the input nb neighbour implies the choice of a
> spatially lagged value of zero (zero.policy=TRUE), lx = Wx, rather than NA
> (zero.policy=FALSE)?
>
> That is, use by default zero.policy=any(card(nb) == 0L) rather than
> zero.policy=NULL and look in the spdep option set by default on package
> load to FALSE but settable by the user?
>
> Would this be taking trying to be helpful too far, given that the analyst
> is creating the neighbour object and presumably should take responsibility
> for choices made?
>
> Context: polygons not sharing boundaries with other polygons do exist
> legitimately in data sources, but setting spatially lagged values to zero
> for those polygons is quite an invasive imputation. It may be better to
> oblige the user to make the choice when the spatial weights listw object is
> created.
>
> Little is known about the problem, for a recent treatment for CAR models
> see: https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Farxiv.org%2Fabs%2F1705.04854&data=05%7C01%7Cconnor.donegan%40utdallas.edu%7C981c03aaa25a43db669d08dbdeb7b093%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C638348652754639883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=G8%2B714lKy1JUjezV0C%2FlRmSGYsq8WCj0iPkwk%2FQjMQI%3D&reserved=0<https://arxiv.org/abs/1705.04854>, published as
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoi.org%2F10.1016%2Fj.sste.2018.04.002&data=05%7C01%7Cconnor.donegan%40utdallas.edu%7C981c03aaa25a43db669d08dbdeb7b093%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C638348652754796114%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zsDOSppZjxGiSppcmXCSYb1lWlIh8RgettJucnm4RNg%3D&reserved=0<https://doi.org/10.1016/j.sste.2018.04.002>, where: "The specification of
> a CAR model on a disconnected graph is undefined ... [t]here are
> essentially two types of disconnected graphs: first, a graph containing an
> island (a singleton node with no neighbours), second, a graph split in
> different sub-graphs (each of them being a connected graph)".
>
> This question concerns the former, singleton, case, but adding sub-graph
> counts if greater than unity to summary.nb and print.nb address the second
> . Very possibly, functions creating nb neighbour objects should themselves
> report that an output object (graph) is not connected, bigDM CARBayes
> CARBayesST geostan spatialreg stampr do call spdep::n.comp.nb themselves to
> check the subgraph count.
>
> Interested in feedback,
>
> Roger
>
> --
> Roger Bivand
> Emeritus Professor
> Norwegian School of Economics
> Postboks 3490 Ytre Sandviken, 5045 Bergen, Norway
> Roger.Bivand at nhh.no
>
> ________________________________________
> From: R-sig-Geo <r-sig-geo-bounces at r-project.org> on behalf of Roger
> Bivand <Roger.Bivand at nhh.no>
> Sent: 04 November 2023 18:53
> To: r-sig-geo at r-project.org
> Subject: [R-sig-Geo] spdep: new zero.policy attribute
>
> In forthcoming spdep 1.3-1, spatial weight listw objects get a new
> zero.policy attribute. The attribute is added as objects are created to
> record the status of the zero.policy argument in the function creating the
> object, see:
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fr-spatial%2Fspdep%2Fcommit%2Fe159de922c61713529a4075b0dfc2966eb8f9ad6&data=05%7C01%7Cconnor.donegan%40utdallas.edu%7C981c03aaa25a43db669d08dbdeb7b093%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C638348652754796114%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=E3ixQ1qYnfd6iqsSSZzwmHbvhayd4S04EtYfh0wENTc%3D&reserved=0<https://github.com/r-spatial/spdep/commit/e159de922c61713529a4075b0dfc2966eb8f9ad6>
> .
>
> Reverse dependency checks only show problems from over-eager unit testing
> in SpatialFeatureExperiment, a Bioconductor package, but other workflows
> may be impacted. The new attribute is used in tests for spatial
> autocorrelation to set the zero.policy argument in those tests (the
> arguments were zero.policy=NULL, are now zero.policy=attr(listw,
> "zero.policy") where listw is the spatial weights object argument to the
> test function.
>
> This will be extended to spatialreg and friends if nobody reports negative
> impacts here soon. I'll wait before releasing 1.3-1 for a few days to see
> if any feedback is forthcoming.
>
> Hope this long-overdue change is helpful,
>
> Roger
>
> --
> Roger Bivand
> Emeritus Professor
> Norwegian School of Economics
> Postboks 3490 Ytre Sandviken, 5045 Bergen, Norway
> Roger.Bivand at nhh.no
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at r-project.org
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-geo&data=05%7C01%7Cconnor.donegan%40utdallas.edu%7C981c03aaa25a43db669d08dbdeb7b093%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C638348652754796114%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iebpGpErkLKR7eWsKVSU9znpLElbqqa1QUjjKCH9yZ8%3D&reserved=0<https://stat.ethz.ch/mailman/listinfo/r-sig-geo>
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at r-project.org
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-geo&data=05%7C01%7Cconnor.donegan%40utdallas.edu%7C981c03aaa25a43db669d08dbdeb7b093%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C638348652754796114%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iebpGpErkLKR7eWsKVSU9znpLElbqqa1QUjjKCH9yZ8%3D&reserved=0<https://stat.ethz.ch/mailman/listinfo/r-sig-geo>
>
[[alternative HTML version deleted]]
------------------------------
Subject: Digest Footer
_______________________________________________
R-sig-Geo mailing list
R-sig-Geo at r-project.org
https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-geo&data=05%7C01%7Cconnor.donegan%40utdallas.edu%7C981c03aaa25a43db669d08dbdeb7b093%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C638348652754796114%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iebpGpErkLKR7eWsKVSU9znpLElbqqa1QUjjKCH9yZ8%3D&reserved=0<https://stat.ethz.ch/mailman/listinfo/r-sig-geo>
------------------------------
End of R-sig-Geo Digest, Vol 243, Issue 3
*****************************************