Check for georeferencing errors in GDA2020 geopdfs

[Off-track]

Maybe this should be in the NSW subforum, but the warning is general. It seems necessary to check the for the correct georeferencing of new geopdf issues before trusting them. Images below are in oruxmaps (with tracks overlaid).

For example, the GDA2020 geopdf Potsville 1:25K seems to be incorrectly georeferenced (about 100 m N of actual location). This error is not present in the GDA94 version of the same map (although it was slightly west of true location). The full reference to the map with this problem is: 9641-3S Pottsville 4th Edn CollarOn_2022.

Potsville GDA2020

Screenshot_20240415_073627_OruxMaps.jpg (102.75 KiB) Viewed 6919 times

Potsville GDA94

Screenshot_20240415_073707_OruxMaps.jpg (118.99 KiB) Viewed 6919 times

I do not know if the error occurs in other maps. Spatial NSW will need to check (but they just say "Thank you for contacting DCS Spatial Services...We will pass this feedback on to our Experts...We consider the case resolved"). The Tyalgum map seems OK.
PS. The GDA2020 maps are generally much improved. For example, all contours are now shown in the Tyalgum map, where there were previously voids in steep areas (these voids are still in the SIX WMTS tiles). But the contours are now sometimes much harder to see - especially under similar colours such as the new Trees:dense colour (previously Closed forest in a much clearer colour), and under the widened National Park boundary lines.

Tyalgum GDA2020

Screenshot_20240415_074107_OruxMaps.jpg (72.71 KiB) Viewed 6919 times

The attachment Screenshot_20240415_073627_OruxMaps.jpg is no longer available

[tom_brennan]

Are you sure it's an issue with the GeoPDF?

If I load that PDF into QGIS, it lines up nicely with various other datasets (OSM raw, OSM tiles, NSW Spatial Service tiles) for the area.

How are you getting the map into Orux? Maybe it's an issue with either Orux, or the software/process you are using to get it into Orux?

[Off-track]

I think that Oruxmaps (OM) has a problem with some of the GDA2020 files (though I can not get an answer on the OM forum, or anything sensible from the NSW spatial ‘service’).

The reason I think this is that both the 2016 and the 2020 versions of the Pottsville map give the same (only slightly inaccurate) location using the Adobe Reader Geospatial Location Tool. For some reason, metadata shown by TerraGo shows that both versions use GDA94 (although the new ones are distributed as GDA2020). Probably much like Tom in QGIS, which I have not tested.

The CollarOff versions of the maps from the NSW.spatial website are in geotiff format, and AsTiffTagViewer shows that the 2020 maps specify GDA2020 in metadata. But they will not open in OM (version 7.4.28). Nor can OM open geopdf maps converted from the 2020 Pottsville geotiff using the (very slick) online converter at https://gdal3.js.org/; though Adobe Reader opens these converted files without problem.

One workaround is to use Map tweaks > Map calibrator in OM, though it is not entirely satisfactory (probably because it uses only a single point to calibrate).

What did work well for me (after I downloaded the geotiff file from NSW.spatial to C:\temp\pottsville.tif) was to use gdal programs: first gdalwarp -t_srs EPSG:4326 C:\temp\pottsville.tif C:\temp\pottsville1.tif (to get a WGS84 re-projection - though it will still not open in OM), then gdal_translate C:\temp\pottsville1.tif C:\temp\pottsville1.pdf to get it into geopdf. This aligned perfectly under my track in OM (better than the 2016 map direct from the NSW.spatial website, which has that slight E-W error). Whew, what a run-around.

GDA2020 geopdf re-projected to WGS84

Screenshot_20240424_150359_OruxMaps.jpg (94.64 KiB) Viewed 6429 times

The cached wms maps align perfectly under my track, so the errors seem to be associated with OM handling of the geopdfs.

2016 (cached) WMS

Screenshot_20240424_150610_OruxMaps.jpg (92.58 KiB) Viewed 6429 times

I think the take-home message is that OM (version 7.4.28) is happy with WGS84, but may be confused by at least some pdf maps in GDA94 or GDA2020 (these are not listed under OM Map tweaks > Map Datum, but at inception they were very close to WGS84).

[Off-track]

Here is some weird stuff:
(1) OM will not open a geotif made with gdal_translate from the NSW.spatial GDA2020 geopdf (java error); but it will open the geotif after gdalwarp to WGS84 (which also warns about an ignored/misplaced neatline) - now a moderate size file, but with poor resolution.
(2) OM will open correctly a pdf made with gdal_translate from that ‘warped’ tif - with smaller size and better (but still degraded) resolution. Starting with the geotif from NSW.spatial then warping and translating as described in previous posts is better (though still slightly lower resolution than the NSW.spatial geopdf).
(3) OM correctly opens the original GDA2020 NSW.spatial geopdf when re-loaded (into either mapfiles or the sub-folder geopdf). Now it does not give either variable georeference errors or variable problems with zoom (experienced previously). Likewise, a re-loaded (large) pottsville1.tif now works in OM. Maybe it was some subtle corruption on copying the files? Now I am nervous - you can see the evidence in the previous screenshots.
(4) The re-loaded files do not show at first in OM, even after refresh maps, but they do show in subsequent starts (probably a little bug in OM, not a problem after the second start).

[Off-track]

I eventually got a high-res geotif from spatial.nsw to load in OM. The method, starting with 9541-3N+TYALGUM.tif copied to C:\temp was:

1. Use Irfanview or gdalinfo to find that resolution of TYALGUM.tif is 11592 * 7421 pixels.
45,254 kb

2. gdalwarp -ts 11592 7421 -r average -t_srs EPSG:3857 C:\temp\9541-3N+TYALGUM.tif C:\temp\9541-3N+TYALGUM6.tif
336,077 kb (gdalwarp cannot compress; -r controls sampling which is bad in the default; must specify -ts output resolution)

3. gdal_translate -co COMPRESS=LZW C:\temp\9541-3N+TYALGUM6.tif C:\temp\9541-3N+TYALGUM7.tif
102,369 kb (not as efficient as whatever compression method spatial.nsw used, though both show LZW interleave=pixel in gdalinfo)

There was no advantage over the WMTS map (especially because contours are obscured in the current spatial.nsw geotif and geopdf).

My best interpretation is that OM cannot read GDA2020 / MGA zone 56 (EPSG:7844 / 7856) in geotif but fails to warn that is the problem.

[Off-track]

The interpretation that OM cannot read GDA2020 / MGA zone 56 (EPSG:7844 / 7856) in geotif is confirmed. Use on the nsw.spatial geotif files of gdalwarp with -t_srs EPSG:3857 (WGS84) or EPSG:28356 (GDA94) but not EPSG:7856 (GDA2020) results in geotif files that open without error in OM.

In gdalinfo, the GDA2020 geopdf files from nsw.spatial report EPSG:1168 / 17056 (conversion EPSG:9807 to Transverse Mercator). EPSG:1168 is the GDA2020 datum, not a coordinate system. EPSG:17056 cannot be found in epsg.io (or elsewhere) at 3 May 2024, but 17456 and 17356 were earlier MGA56 Transverse Mercator projections.

Note that gdalinfo checks only the first raster it finds (whatever that is - note the small reported size despite very high resolution of the map). The vector geopdf files have many layers with various raster images. Unfortunately,

Code: Select all

ogrinfo -al -so <source file path>

will read some of the 2016 geopdf files, but fails to read the 2023 geopdf files (unable to open). Using

Code: Select all

ogrinfo --config OGR_PDF_READ_NON_STRUCTURED YES -al <source file path>

gives some info (but no geometry). I have not found any other way to view vector geopdf metadata.

The GDA2020 geopdf files from nsw.spatial can be read by OM, but we cannot know what metadata OM is reading. Unfortunately, neither gdalwarp nor ogr2ogr can open the GDA2020 geopdfs that I tried from nsw.spatial (so I can not test it like the geotif files above). Most likely, these (multi-layer, vector) geopdf files are actually using GDA94 for the topo map frame (as reported by TerraGo). GDA94 maps do open in OM (though maps in the nsw.spatial 2016 series have slight georeferencing errors mentioned in previous posts).

So what is the practical implication? Always check georegistration of any new map on-site at the start of a walk, and have a back-up in case it is wrong. Personally I use downloaded WMTS maps in OM https://bushwalk.com/forum/viewtopic.php?f=21&t=41543, and carry a printed map https://bushwalk.com/forum/viewtopic.php?t=25619.

[ken333]

Thanks for the info. I was trying to use QGIS to combine 2 metre contours from the ELVIS NSW 2 metre grid DEMs with OSM and SIX maps. I also found the the QGIS geotiffs would not open in OM but the QGIS PDFs would. I think QGIS uses Gdal, including gdal_translate to convert files. I also changed one of the geotiff files using gdal and it worked in OM. I changed the srs to EPSG:3857. I thought it might be because I used the DEFLATE compression, not LZW, but you found LZW sometimes works. So its something else in the geotiffs OM doesnt like.

[Off-track]

I have still not understood why OM sometimes struggles with some GDA2020 files, but maybe this helps someone else:

New versions of GDAL and QGIS available from OSGeo4W as stable releases should soon handle the nsw.spatial 2022 geopdf series correctly. At present, this capability is in -dev versions.

GDAL before 3.9 (and QGIS versions using GDAL before 3.9) could not read vector content in the nsw.spatial 2022 geopdf series (or other maps produced using recent ArcGIS versions): they just showed the raster overview.
The problem was not the new Datum, rather it was a faulty pdf driver (https://github.com/OSGeo/gdal/issues/9870 now fixed).

But GDAL-dev 3.10 does not agree with TerraGo that the 2022 vector layers are GDA94; it gives GDA2020. I don't know how to tell for sure what OM is seeing.

[tom_brennan]

GDAL 3.9.0 was only released a week or so ago, so pretty much any QGIS version will be using GDAL 3.8.x (or earlier) out of the box.

[Off-track]

GDAL 3.9.0 was only released a week or so ago, so pretty much any QGIS version will be using GDAL 3.8.x (or earlier) out of the box.

True, unless you load a -dev version, available now as mentioned from OSGeo4W (already up to GDAL 3.10).
Update:
GDAL 3.9 seems to lack the fixed pdf driver, but GDAL-dev 3.10 has it. I don’t know what lies between.

[Off-track]

Ken, I guess we should stop trying to understand the things that OM cannot do, and just enjoy what it can do, but in case it helps:

Code: Select all

C:\OSGeo4W>gdal_translate -a_srs EPSG:28356 C:\temp\POTTSVILLE.tif C:\temp\POTTSVILLE2.tif

just changes the declared projection (to GDA94 UTM), without actually reprojecting. OM still fails to initialize (with the java null object reference error). So it is not as simple as just the declared projection.

Code: Select all

C:\OSGeo4W>gdal_translate -a_srs EPSG:7856 C:\temp\POTTSVILLE.tif C:\temp\POTTSVILLE5.tif

keeps the GDA2020 UTM projection (but blows out the file size) and gives the same error (as expected).

Whereas:

Code: Select all

C:\OSGeo4W>gdalwarp -t_srs EPSG:28356 C:\temp\POTTSVILLE.tif C:\temp\POTTSVILLE3.tif

which really reprojects (to GDA94 UTM) does work, but

Code: Select all

C:\OSGeo4W>gdalwarp -t_srs EPSG:7856 C:\temp\POTTSVILLE.tif C:\temp\POTTSVILLE4.tif

or

Code: Select all

C:\OSGeo4W>gdalwarp -t_srs EPSG:7844 C:\temp\POTTSVILLE.tif C:\temp\POTTSVILLE6.tif

(to GDA2020 UTM or Geographic 2D) does not work, though the OM error is different, just objecting to the EPSG code.

So OM seems not to like GDA2020 projections in geotiff; but it will load them somehow in geopdf (still a mystery to me).

Anyone who needs a GDA2020 geotiff in OM for now can reproject to GDA94 (or change format to geopdf if they are game) though the file size may blow out (from GDAL).

[Off-track]

In case the canetoads are feeling left out, the QTopo geopdf maps are straight raster (no vector content). I just explored the Tyalgum map. I do not know why the 2023 (GDA2020) series is so much larger in file size than the 2017 (GDA94) series as it is lower in resolution (check the raster size), but it does have some useful new information. The border track below Mt Durigan is still in the wrong place (as it is in Garmin Topo Aus v5), but Warin will be happy that OSM has it right.

Anyhow, OM does load the Qtopo GDA2020 geopdf correctly. Let's call the 2017 (GDA94) map TyalgumQ, and the 2023 (GDA2020) map TyalgumQn.
If you run

Code: Select all

C:\OSGeo4W>gdal_translate C:\temp\TyalgumQn.pdf C:\temp\TyalgumQn.tif

OM fails to load it (with the java null object reference error). OK, we expected that as OM does not like GDA2020 geotif.

But if you run

Code: Select all

C:\OSGeo4W>gdal_translate C:\temp\TyalgumQ.pdf C:\temp\TyalgumQ.tif

OM also fails to load that (with the same java null object reference error). But that is GDA94. Hmmm, maybe OM does not like something from gdal_translate?

So, we try

Code: Select all

C:\OSGeo4W>gdalwarp -t_srs EPSG:28356 C:\temp\TyalgumQ.pdf C:\temp\TyalgumQ2.tif

and

Code: Select all

C:\OSGeo4W>gdalwarp -t_srs EPSG:28356 C:\temp\TyalgumQn.pdf C:\temp\TyalgumQn2.tif

OM likes them both.So OM likes GDA94 from gdalwarp, but not from gdal_translate.

Just out of interest, we try:

Code: Select all

C:\OSGeo4W>gdalwarp -t_srs EPSG:7856 C:\temp\TyalgumQn.pdf C:\temp\TyalgumQn3.tif

Nope, OM gives an EPSG:7856 error.

All of these geotif files open just fine (as tif) in IrfanView. So OM is a bit finicky about both the Datum &/or projection (in geotif) and how the geotif file is made.

[Off-track]

Using gdalinfo on the GDA94 .tif files shows that conversion from .pdf via gdal_translate vs gdalwarp results in different metadata; including (i) different WKT under CONVERSION, (ii) differences in whether an EPSG code is specified in several places including DATUM and GEOGCRS, and (iii) whether they use “GeoTransform=” or “Origin=”. Neither introduces an EPSG code for PROJCRS ID, but warping and gdal_edit -a_srs adds a USAGE section with an EPSG code (which is the same as the PROJCRS).

From comparison of gdalinfo reports, and running gdal_edit -a_srs on selected files to test possible problems, we can conclude that the problem for OM is when geotif files have either EPSG:7856 for the USAGE ID or Transverse Mercator for CONVERSION (which comes with “GeoTransform=”) unless a CONVERSION ID (EPSG code) is given. Under GDA2020, OM gives an EPSG:16156 error, though gdalinfo reports no CONVERSION ID. There may be other incompatibilities with GDA2020.

The GDA94 and GDA2020 Qtotpo and nsw.spatial geopdf files have “GeoTransform=” (but they give EPSG:17056 for Transverse Mercator CONVERSION), no USAGE and no PROJCRS ID, just ESRI WKT. EPSG:17056 is not in epsg.io or elsewhere, and gets stripped out in gdal conversions. OM may be sufficiently puzzled by EPSG:17056 to take clues from WKT in the geopdf files and guess a PROJCRS ID (possibly GDA94, as in the TerraGo Toolbar). The difference between GDA2020 and GDA94 is about 1.8 m, indiscernible in a map at 1:25,000. If this is correct, OM opens GDA2020 geopdf files through an error, which is not ideal. This conclusion is supported by gdal_edit -a_srs EPSG:7856 (OM gives “Can not find GEO info!!” error) vs gdal_edit -a_srs EPSG:28356 (OM opens), and similarly by sequential gdalwarp and gdal_translate operations, on the GDA2020 TyalgumQ geopdf file. For key gdalinfo files, see Geospatial PDF Files. https://scithings.id.au/geopdf.pdf. See also: Map Datums. https://scithings.id.au/Datums.pdf

To confirm the absence of GDA2020 (and EPSG:17056) in OM, use apktool (https://apktool.org/docs/install) on the OM apk, then look in OruxMaps7.4.28\assets\proj4\nad\epsg where you will find lots of GDA94 but no GDA2020 codes. Until the current OM developer gets to GDA2020, you use this datum in OM at your own peril.

I still cannot explain the georeference error (of ~100m) shown for the GDA2020 geopdf in the original post. It is too hard to explore things that have changed. In any case, it was fixed by deleting and recopying the file - which may be hard to do in the field. I think there is a georeference error (of ~30m) in the GDA94 geopdf that I showed (the error is seen in Acrobat reader as well as in OM after file reload), but good luck with getting any sensible response, let alone an acknowledgement or fix of a 'past' error, from nsw.spatial.

[Off-track]

The GDA2020 EPSG codes are in OM 10.6.2beta3, according to apktool. The OM developer seems to be lately more occupied with versions released for payment on places like Google Playstore. So you might go that way if you want GDA2020 support any time soon in OM (and deal with the complications for established OM users of the new Android requirement for scoped=private storage by apps). Ironically, the version on Playstore (for which you must pay via Google, probably much less than you would otherwise donate via OM) is sometimes dubbed the "donate version" (go figure).

Just out of interest,OM uses null transformations from GDA94 to WGS84 (PROJ strings with towgs84) https://svn.osgeo.org/metacrs/proj/tags/proj_4_5_0/proj/html/gen_parms.html, eg:

Code: Select all

# GDA94 / MGA zone 56
<28356> +proj=utm +zone=56 +south +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs  <>

Note that the PROJ string itself does not specify a datum (other than towgs84), and the same string can apply to different EPSG codes (under different # PROJCS titles).
A null transformation to WGS84 is presumably also used for GDA2020 in OM 10.6.2 (and at least until epochs are considered in such transformations), though the towgs84 parameter is not used:

Code: Select all

# GDA2020 / MGA zone 56
<7856> +proj=utm +zone=56 +south +ellps=GRS80 +units=m +no_defs  <>

[Off-track]

Finally, someone from GDAL admitted that epsg: 17056 is a bug in GDAL (and PROJ) that should be fixed in future versions to EPSG:16156 (https://gis.stackexchange.com/questions/482037/what-is-epsg17056).