Government Plans to Privatise LPI (topo mapping)

[colinm]

My GIPA request for the topographic data covering NSW has failed, because GIPA says that if the government presents the data for sale, it doesn't have to provide any other access. Problem with that is that (a) the price is exorbitant and out of any relation to the marginal cost of production, (b) what data they have for sale is about to be sold wholesale to "a big international IT company," if the Minister has any say in it. This, I think, is why LPI and the Minister have been dragging the chain on replying to my proposals, trying to get as much done as possible done on the privatisation before the public has time to react.

I haven't heard a peep from the Minister in response to my proposal that LPI crowdsource their data for National Parks, so I guess it's time to go public about it, so I sent the following to the SMH:

NSW Land and Property Information [LPI] have map databases upon which we all rely, but they contain some dangerous errors. I detail (in https://docs.google.com/file/d/0B0PI7wN ... dBUkE/edit) a couple of cases where the maps are quite significantly out of alignment with reality, there are many more such cases.

This matters. In one case, NSW Parks and Wildlife (NPWS) advise that the only safe path through a field of unexploded ordnance (a former Defence firing range, you could almost say a 'mine field') is the marked path, but the path marked on our State's maps bears only a passing relationship to the actual path.

It matters, too, because peoples' lives depend upon it. You will, no doubt, remember the sad death of David Iredale in 2006. Navigational confusion played a large part in his misadventure. The Coroner recommended that NPWS improve its signs, but Iredale was only carrying a sketch map! Improved mapping, better access to maps, would have served the boys better.

Per http://www.smh.com.au/nsw/state-to-outs ... 28phx.html the state intends to privatise its mapping functions because it is 'not in a financial position to continually upgrade the services provided by the agency as technology changed.'

LPI is certainly unable to maintain accurate and up-to-date data for some areas in NSW, but the most cost-effective and effective solution is not privatisation, one part of the solution is to be found in crowdsourcing, as I outline in my proposal above.

LPI struggle with effective electronic distribution of geographic information, I would say that's because it is not in their DNA. They see Google maps and believe that if they could only find a sugar daddy to take all that useless data for money, everything would be better ... clearly, an ability to deal with commercial reality is also not in their DNA.

What should happen, now, is that LPI should forget about trying to run as a profit centre, in some mad 1980s Thatcheresque throwback, and they should take note of the Bureau of Meteorology - which runs the most accessed website in Australia by simply providing a good service needed by all. LPI should and could achieve this kind of prominence and public utility by simply giving up its (completely illusory) belief that it holds Crown Copyright in the State's topographic data, distributing it freely, and welcoming the willing collaboration of the population by crowd sourcing ... it's not like they can do it without us.

[photohiker]

That's a bit sad, colin.

At least you tried.

[wildwalks]

Shall we just build our own data set???
We could use OSM as a base. I have a pretty good 30m DEM that for contours.
I also have access too Google Earth Engine so we could generate some vegetation layers.
I have played with it a bit and done sections of NSW like this. Check out the maps on Wildwalks iphone app.
Maybe we do a Bushwalkers.com topo map set
Will take a fair bit of effort but could be fun
Matt

[colinm]

Shall we just build our own data set???

Yes, that's what will eventually happen. LPI aren't interested in (can't afford) high quality NP topo data.

My assessment is that LPI's data is pretty crap. Their paths bear little relation to the paths on the ground, their hydrography is good, but over-optimistic (I think they've been relying on models for a long time.)

The thing that's missing, mostly, is cliffs. LPI's cliffs are pretty poor (at least for the area I have.) I've been thinking about automating a scan over Bing to bring out sandstone. Coupled with DEM it might be good. Hydrography is going to be tough, really tough. OTOH, if I'm not mistaken, someone else is providing that data to LPI, and it may be possible to acquire it from them.

We could use OSM as a base. I have a pretty good 30m DEM that for contours.

DEM is easy enough, yeah. OSM lacks a lot, but it would be the best place to build it.

I also have access too Google Earth Engine so we could generate some vegetation layers.

Google Earth aren't going to like that. Bing, though, have some kind of deal with OSM.

I have played with it a bit and done sections of NSW like this. Check out the maps on Wildwalks iphone app.

I've seen your wildwalks stuff, and it is good. To the extent it's derived from LPI, though, you can't open it up ... they make most purchasers/recipients sign a license even though the LPI do not own copyright on much of the data.

Maybe we do a Bushwalkers.com topo map set
Will take a fair bit of effort but could be fun
Matt

To get the most out of it, the data has to be free, open and available to all. That will happen over time, but LPI are determined to swim against the tide.

Colin.

[wildwalks]

Hi Colin
Happy to do an open source licence of some kind -- probably run with the same as OSM to keep life easy.

"Google Earth Engine" is different from Google earth -- Googles naming convention is a bit confusing. It is a very powerful GIS platform that allows analysis of Landsat and other data. I have already done some training of NSW vegetation -- but still a fair bit of work to get it great -- it is good now. We can then export the regions in a shape file then make tweaks of the areas in a OSM type environment over time.

I am trying to get some 2.5m sat data onto to system that would open up some great options.
Cliff detection is hard based on DEM alone - the results are messy. Maybe useful to highlight areas that need tracing then trace with Bing of Australian Geographic Image data might be the best way to go.

Have a look at the hydro lines on the wildwalks app maps (different data set from the website). The hydro is derived from water drop analysis using the DEM. We can train this system a bit -- but generally it will give us creeks that we then need to improve based on field research.

The DEM is a 1arc sec from the SRTM. But I have also done some splicing of data from Japan's DTM project to fix up the huge errors in the Grose Valley. Then done some work tweak the overall DEM to be closer to some known points. I do have the NSW LPI DEM as well but the licence is a bit limited as you point out -- I will need to look at it a bit more to see how we might be able to us it if at all.

It is a huge amount of work -- but we could do one map sheet at a time -- working the handful of most popular areas first. We could take a snapshot of OSM and have our own trusted environment. Or work within OSM itself which could also get messy in it own way.

If there is a enough interest in doing this -- we could organise a get together in Syd to scope it out. I have a few other projects chewing up a lot of time at the moment so I will need delay the start of my time a bit.

Matt

[colinm]

Hi Colin
Happy to do an open source licence of some kind -- probably run with the same as OSM to keep life easy.

OSM are making the running in this area, it's as well to work in as close as possible with them, IMHO.

"Google Earth Engine" is different from Google earth -- Googles naming convention is a bit confusing. It is a very powerful GIS platform that allows analysis of Landsat and other data. I have already done some training of NSW vegetation -- but still a fair bit of work to get it great -- it is good now. We can then export the regions in a shape file then make tweaks of the areas in a OSM type environment over time.

I didn't know that. Excellent. If you're using the SRTM from CSIRO via Geospace Australia, they've done some adjustment to heights using vegetation maps (to correct for the tree heights, as the microwaves bounce off the trees, not the ground.) Better vegetation model would imply better DEM correction.

I am trying to get some 2.5m sat data onto to system that would open up some great options.
Cliff detection is hard based on DEM alone - the results are messy. Maybe useful to highlight areas that need tracing then trace with Bing of Australian Geographic Image data might be the best way to go.

Yes, cliffs are by definition discontinuities. I was thinking of Bing (or some other image source) inputs to merely detect bare rock (whatever its inclination) which, combined with DEM, would give you an indication of 'a cliff-like area', as it seems to me that (for hiking at least) you don't need to know the precise cliff line, but more that there is stuff here you'd have to think twice about descending. I've taken to pseudo-colouring terrain by slope, and I find it much more informative than LPI's vegetation cover conventions.

So, sounds like your vegetation mapping techniques, if done with sufficient precision, would give that 'here is bare rock', which could be outlined, then contours tell the rest of the story.

Have a look at the hydro lines on the wildwalks app maps (different data set from the website). The hydro is derived from water drop analysis using the DEM. We can train this system a bit -- but generally it will give us creeks that we then need to improve based on field research.

You took it that far? Very good. Indicating that a water source could exist in a valley is probably good enough ... do you know anyone who actually expects to find a creek where a topo map says there could be one?

The DEM is a 1arc sec from the SRTM. But I have also done some splicing of data from Japan's DTM project to fix up the huge errors in the Grose Valley. Then done some work tweak the overall DEM to be closer to some known points. I do have the NSW LPI DEM as well but the licence is a bit limited as you point out -- I will need to look at it a bit more to see how we might be able to us it if at all.

NASA have just done another overpass, I think. The more data the better. Over time, multiple samples will converge on something approaching accurate rendering.

It is a huge amount of work -- but we could do one map sheet at a time -- working the handful of most popular areas first. We could take a snapshot of OSM and have our own trusted environment. Or work within OSM itself which could also get messy in it own way.

If there is a enough interest in doing this -- we could organise a get together in Syd to scope it out. I have a few other projects chewing up a lot of time at the moment so I will need delay the start of my time a bit.

OSM doesn't really care much about hiking topos, they're mainly about streets. There are some groups operating at a tangent who are using OSM techniqies and schemas toward topo, whose work we could use.

I'm up for the conversation, Matt, and think it a worthwhile project. When you get a breather, let me know. BTW, I have quite a bit of LPI data we could use for comparison purposes - much of it legitimately acquired (most of the budawangs, e.g.)

Colin.

[wildwalks]

Sounding good Colin
Do you have info on the second NASA DEM - I have not heard of it. I have also been in touch with an arial mapping company that has a very high res DEM that might be willing to play - will chat more with them - we might be able to do a data share deal.

We can set up our own OSM environment fairly easily - I have one already built for a different project - it is a bit tricky but do-able. But once we set it up it is hard to re-merge datasets again in the future - I will have a chat with my developer and see if we can create an instance to manage data specific to our project (veg, creeks, contours, cliffs, tracks and trails etc) but keep the main OSM data separate - but this will get messy with some overlapping data sets.

The creek analysis is pretty cool. The size of the creek depends on the land area that feeds into each creek section. The creeks do end up with a few straight sections and funny bits - but that can be fixed visually.

I would like to come up with a method for improving a base DEM based on user contributed altitude points and from tweaked contour lines - if you know of any software or methods todo this it would be handy.

Lets grabs meal in a few weeks and scope this out more fully.

Maybe we could trial a small area and work out the effort involved - then organise workshops to focus on keys map areas.

Matt

[colinm]

Sounding good Colin
Do you have info on the second NASA DEM - I have not heard of it.

I apologise, was mistaken, a friend mentioned reading this http://www2.jpl.nasa.gov/srtm/, I just read the page 'last update 2009'.

Another possibility might be to hit up .nsw.gov for some of the LIDAR scanning they've been doing, but they tend to stick to populated areas.

I would like to come up with a method for improving a base DEM based on user contributed altitude points and from tweaked contour lines - if you know of any software or methods todo this it would be handy.

Well, I have the garmins with altimeter, and leave it on the whole time, so that gives quite a few tracks with presumably pretty accurate altitudes (not that I've ever calibrated it properly, but the error would be constant.) That would perhaps give an indication of the kinds of error sources and noise present in the DEM-derived spot heights. I'm pretty sure there's open source to turn the grid of DEM heights into a continuous-looking field, then project the track-points onto that and extract the interpolated altitude.

Something occurs to me, though ... trying to get precise and accurate altitude is perhaps more than is actually needed. I think (and correct me if I'm wrong) hikers mostly only care about relative heights, and more specifically about slope. Turning DEM into slope gives quite useful results, because the data is kind of ummm holographic ... each tile contains some information about the surrounding tiles too, because the tiles aren't completely independent samples, but are tied together by the physical terrain, and that constrains them. That's why you can get quite good high-res contours out of relatively low horizontal resolution tiles.

So I suspect that as long as the DEM is subject to systematic error which changes relatively slowly over a terrain the results are still useful. Places where that assumption breaks down would (I suspect) be transitions between different types of vegetation, as the largest error source (apart from reflections from cliffs) is considered (by CSIRO at least) to be reflection from the water in tree canopies. They estimate the heights of vegetation and simply subtract that from the data. So your vegetation mapping might be a really good source of correction.

Lets grabs meal in a few weeks and scope this out more fully.
Maybe we could trial a small area and work out the effort involved - then organise workshops to focus on keys map areas.

That could totally work. Bags the Budawangs!

Colin.

[wildwalks]

Sounds like fun.
I am meeting with Bushwalk.com admins next week. I will discuss this project with them.
Bused wants could be a good starting project.
Matt )

[tom_brennan]

It's disappointing we can't get the LPI DEM as a starting point. I've had a play with the SRTM DEM a couple of times, and it was pretty hopeless, at least for bushwalking. That said, it was the 3 arc second one - I haven't seen any 30m DEM available for Australia.

While the LPI contours/creeks are a bit rough in places, they are mostly good enough. That's the data that really can't be crowdsourced easily. Much of the other stuff is already in OSM. Cliffs can easily be added - I've done a lot of cliffs around Katoomba/Medlow Bath just off the aerials and knowledge.

I've looked at using GPS for altitude previously, and it's seemed pretty useless. I have a new GPS so I had a look at my log from the weekend. Some places look ok, other places it goes up 50m as I'm descending.

[Webguy]

I use this set currently on my GPS, works well for me...

http://www.lizarddrinking.net/index.html

[tom_brennan]

I had a look at that, and Shonkymaps. I just don't think they're good enough for bushwalking.

Here's a typical section of the Bluies. The wiggly depression across the middle is a creek, but sometimes the creek's going up, sometimes it's going down. Looks like a karst area, there's so many sinkholes!

[photohiker]

You're just looking at the contour map and nothing else there.

Try finding the same spot on the cyclemap at openstreetmap.org to get an idea what the OSM + Contours Australia will look like.

Like here

The Cyclemap does hillshading which isn't available AFAIK on Garmin with Contours Australia.

[tom_brennan]

Realistically the contours should be all I need. Shading, and even watercourses should just be byproducts of the contours. That's why the contours need to be accurate. And while it's well known that the LPI maps do have issues, by and large they're pretty good.

For comparison, here's a nearby area. The first image is just the LPI contours, the second is the Contours Australia contours. It's pretty clear there's a marked difference in quality between the two. Many of the small gullies are obvious from the LPI contours, and the creeks all flow downhill.

[colinm]

It's disappointing we can't get the LPI DEM as a starting point. I've had a play with the SRTM DEM a couple of times, and it was pretty hopeless, at least for bushwalking. That said, it was the 3 arc second one - I haven't seen any 30m DEM available for Australia.

See http://nedf.ga.gov.au/geoportal/catalog/main/home.page ... you can download reasonable slices, under creative commons license, at no cost (it starts getting pricey when you want Gb of data, I think.) 1 arc second is about 30m.

While the LPI contours/creeks are a bit rough in places, they are mostly good enough. That's the data that really can't be crowdsourced easily. Much of the other stuff is already in OSM. Cliffs can easily be added - I've done a lot of cliffs around Katoomba/Medlow Bath just off the aerials and knowledge.

LPI are pretty proud of their contours, which are mostly derived from 3d aerial photos (and, I have heard, recently, an automated process which somehow calculates the height stereoscopically.) This still only really gets you to the top of the trees, beyond that it's extrapolation.

LIDAR is the way to go for elevation data. Some local governments have undertaken LIDAR. I haven't tried to get any yet. USArmy estimates about $14/acre (no idea what that is in real money) as cost of LIDAR, so it could be a while before we see it for remote areas, but it will eventually come.

I think cliffs from aerial photos is the way to go. Bing have some kind of deal with OSM allowing one to derive data from their images. This is good, because using just any image to generate cliffs could conceivably taint the resultant data set with copyright claim potential. I have yet to actually undertake a detailed consideration, but my feeling is that just looking for cliff-coloured pixels, georeferencing and grouping them together, and projecting them onto a map with good contours will go a long way to matching what LPI have for maps (honestly, I don't even know how they got what they have ... in some cases it looks like Jackson Pollock was working there.)

I've looked at using GPS for altitude previously, and it's seemed pretty useless. I have a new GPS so I had a look at my log from the weekend. Some places look ok, other places it goes up 50m as I'm descending.

Yeah, is this elevation data derived from the satellite signals? That is hopeless, can't be fixed. My GPS devices have altimeters built in, so the accuracy is certainly high enough. Thing is, though, it's only ever going to be along a track, so at best it can really only be used to kind of calibrate DEM from another source.

[colinm]

Realistically the contours should be all I need. Shading, and even watercourses should just be byproducts of the contours. That's why the contours need to be accurate. And while it's well known that the LPI maps do have issues, by and large they're pretty good.

I agree that shading is just eye-candy (but I love my slope pseudo-colouring) and in any case it's a pretty trivial matter to generate it from a DEM with open source tools.

Can't agree that it's *all* you need, though ... cliffs are discontinuities, and will screw up any contour-bot.

BTW, and slightly aside ... I just love the way the slope pseudo-colouring (derived from the 1arc-second DEM) makes the route out to sentinal hill pop in the following (with 2m contours ... better than hill shading any day, because hill shading presupposes a light source, and a single angle, so hides a lot of information.)

[tom_brennan]

The other thing of course with topographic data is that once you get a good quality dataset, you're set for thousands of years! It doesn't generally change much (at least not in a geologically inactive country like Australia).

Whereas OSM mapping is always going to be a small ongoing effort as new roads are built etc

[photohiker]

For comparison, here's a nearby area. The first image is just the LPI contours, the second is the Contours Australia contours. It's pretty clear there's a marked difference in quality between the two. Many of the small gullies are obvious from the LPI contours, and the creeks all flow downhill.

The LPI looks better for sure. Your Contours Australia looks to be 10m contours, have you got the 5m version?

Have you got a waypoint for somewhere on that map?

[tom_brennan]

It's the 5m Contours Australlia, but at that zoom level it's only showing 10m. But the 5m contours don't improve the quality of the watercourses since they're basically interpolating. It just adds an extra contour.

No waypoint on the map, but the area in question is Waterfall Creek, Mt Wilson, NSW: approx -33.508, 150.399

[photohiker]

I think you must have the detail turned down. Zooming didn't make my contours look like yours, but reducing the detail got it pretty close.

This at max detail:

[tom_brennan]

Yes, detail was set to medium. However, what I was concerned about was not the level of detail, but the "sinkholes" in the creek - highlighted in yellow. The largely correct hydrology is the main area that the LPI data has over data derived from SRTM.

[colinm]

The largely correct hydrology is the main area that the LPI data has over data derived from SRTM.

Impossible to generalise, in my opinion. Some places the hydrography is good, some places it's not so good.

It may be that, for areas like BMNP the hydrography is nearly perfect (although ISTR there's a page about the Wollongambe right next door which has a few creeks wrong) but for the Budawangs it's really just a bit above guesswork. I put this down to how many eyes are on the maps, and how many people actually care, and how much it costs if they get it wrong.

IIUC, the hydrography was once done by stereography (a guy with funny binocular glasses and a strange drawing device) and it actually is just guesswork.

LPI are focused, I think, on places where maps represent contention, and where the contention costs money. So suburbs and roads and such, not wildernesses.

[tom_brennan]

LPI does have plenty of errors. See
http://www.bwrs.org.au/?q=MapErrors
for examples.

Most of my walking has been done in the (Northern) Blue Mountains. I've also spent a fair bit of time in the past down at the LPI office with the pairs of aerial photos and ths stereo viewer, and then I've walked many of those same areas, so I have a fair idea of how good it is.

For me the bottom line is that LPI as it currently stands is "good enough". SRTM is not, and I think there's too much work to try and get it to a similar standard to LPI. Could be wrong!

[colinm]

For me the bottom line is that LPI as it currently stands is "good enough". SRTM is not, and I think there's too much work to try and get it to a similar standard to LPI. Could be wrong!

I believe it's entirely dependent upon the area. If LPI originally drew the maps to as fine a scale as they're printing them, then that's a good thing, but they don't always do remake the map as they zoom in for publication, because they can't afford to.

I've seen enough internal disparities between cliffs, contours and trails to doubt the quality of their hydrography in my area of interest. Some areas, they do a bit of on-the-ground validation (BM would have to be popular enough for that) other areas they don't, because they simply can't afford to. Some areas they use up-to-date aerial photography, but I swear they're using 1949 stuff for some areas.

Map quality is therefore variable. Try the CSIRO-adjusted 1s SRTM before you dismiss its utility across the board.

[wildwalks]

Hi Tom
the public STRM is pretty crappy in places and some places it is great. EG grose valley it is out by about 300m
But using the 1sec srtm (recently released) and cleaning it up with some other DEM's you can get a pretty good contour set -- great is some areas. We can then look at using other scans to improve it.

Matt