VRT source filename paths #10
Comments
|
One thing you might want to look into is the > var ds2 = gdal.open('test/data/sample.vrt');
> ds.getFileList();
[ 'test/data/sample.vrt',
'/home/vagrant/repositories/node-gdal/test/data/sample.tif' ]
> var ds2 = gdal.open('test/data/sample_invalid.vrt');
> ds2.getFileList()
[ 'test/data/sample_invalid.vrt' ]I haven't used it before but it seems like it would help. From the simple test above it seems to turn all relative paths into absolute ones and if the file referenced in the VRT doesn't exist it doesn't return it. Unfortunately, this doesn't explicitly tell you which filenames are bad, which leaves you still needing to scan through the file to see what files are referenced. Maybe a generic error like "VRT file doesn't reference any existing files on the filesystem" would be fine instead though? If a VRT file references 10 raster files and 2 filenames are bad, I would imagine |
|
@GretaCB I'm going to focus on how we handle input files in TileMill -- let me know if your context is different. One way to change this would be to treat VRTs in a similar manner to multi-file vector sources, like a shapefile, where each of the component files is symlinked to the layers directory. For a VRT with relative paths, that might look like:
For a vrt with absolute paths, we'd only need to symlink the source vrt, since the absolute paths would remain valid. Without making any changes on tilemill/mapnik end, the easiest way to deal with the issue is to always use VRTs with absolute paths. I do this by making vrt with |
|
@hrwgc assuming there's no symlinking involved, have you run into any other issues with relative paths? |
|
If a VRT references GeoTIFFs with external overviews (which are $tiff.tif.ovr files produced in same directory as $tiff) , mapnik doesn't seem to know they exist or use them. To my knowledge, this behavior occurs regardless of absolute or relative paths in VRT. |
|
My guess is that its only tilemill symlinking/millstone that breaks vrt's and not Mapnik. But I'll change my mind if you can provide a testcase showing otherwise @hrwgc :) |
|
@springmeyer I'll defer to your expertise here :) |
|
Thanks @brandonreavis ,
So having just one invalid source in the VRT will cause the entire VRT to be invalid in my case.
Agreed, it would be ideal to list the specific sources that are invalid, but afraid there is no other way (that I can think of) to obtain these filenames except by parsing the xml for all SourceFilename elements and checking |
|
Thanks for your input @hrwgc ! I think regarding mapnik-omnivore, I want to be able to validate the VRT file properly, which would entail handling any errors in relation to mapnik/gdal and provide as much helpful information about the error as possible. And (best case) return metadata about the VRT file. |
|
Ran So, GDAL itself is only picking up the one existing source I then checked out the GDAL source for the |
|
@GretaCB . You're brave haha. I started digging into the source for the same thing just a while ago and ...yeah... it's dense. I haven't come up with anything productive yet either. As far as I can tell, the check for whether or not the file exists is done in vrtsources.cpp file here. But I am not sure how we could change node-gdal's |
|
@brandonreavis haha, yeah definitely a whirlwind going through the GDAL source code. Since |
|
Yeah I am totally for that. I talked it over briefly with @brianreavis and we agree that the best way to go about it would probably be to make another function like gdal.Dataset.prototype.getMissingFiles = function() {
var existing_filenames = this.getFileList();
var all_filenames = [];
if (this.driver.description === 'VRT') {
//scan through file for <SourceFilename>
//add filenames that dont exist in existing_filenames
}
return all_filenames;
} |
|
Awesome, would this be adding a |
|
So I changed my mind after looking into it more. It would be far from a simple fix and it would be prone to break with any changes to the GDAL source. I am just going to run down a list of complications
So there are just too many moving parts, and it seems like it would be beyond the scope of the node-gdal binding to support. It really should be something that the GDAL dudes are taking care of... it's not really the binding's job. Since god only knows when GDAL will support this functionality, if you decide you really need to know which filenames in a VRT are bad, I think writing a VRT parser purely with JS that lives outside of node-gdal is the way to go for now. Again, I wish there was a better solution. |
|
When you and @brianreavis turned verbose debugging on with ERROR 4: '/Users/brianreavis/Repositories/node-gdal/test/data/sample.tif' does not exist in the file system,
and is not recognised as a supported dataset name.The default behavior was to throw the last error, but with the commit above I (crudely) made the get/computeStatistics() methods throw any file errors instead of the other weird errors. |
|
Oofdah, yeah that's a lot of hurdles. I updated my local node-gdal master to try out your last commit, but not seeing a difference in the errors. I'm seeing that particular I'm not too familiar with C++ and this might be crazy talk, but would there be a way to integrate a separate javascript VRT parser module into the Also, would it be realistic to add this functionality in GDAL's source and do a pull request? |
|
Thinking it's probably best to chill on this issue for now, and maybe return to it later. I'm going to invalidate any VRTs that don't have ALL sources, and include the vague-ish error message that they're missing one or more sources. Thanks for falling down the dark tunnel with me @brandonreavis ! |
|
You should be able to have Code: var ds = gdal.open('test/data/sample_invalid.vrt');
var band = ds.bands.get(1)
var stats = band.getStatistics(false, true);Result: GDAL will only report the first file does not exist error that it notices, and thats the error that should be thrown in JS now. Is it throwing this error instead? |
|
After reinstalling |
|
About patching the GDAL source: Simply commenting out these 6 lines would make it work how you want, but it makes node-gdal function differently than everyone else's GDAL code which wouldn't be good. That function is called for each About a separate javascript VRT parser: If you really want to be able to call something like var parser = require('somexmlparser');
function getMissingVRTFiles(){
var driver_name = this.driver.description;
if(driver_name != 'VRT') return [];
var filename = this.description;
//verify its actually a filename
var missing_files = [];
//parse the xml
return missing_files;
}
module.exports.applyPatch = function(gdal){
gdal.Dataset.prototype.getMissingVRTFiles = getMissingVRTFiles;
}then in your mapnik-omnivore module do something like: var gdal = require('gdal');
require('gdal-patch-getmissingvrtfiles').applyPatch(gdal); |
|
@brandonreavis @GretaCB Late to the party, but how about on user input of a VRT we just gdal translate to a proper TIFF file and go from there? |
In reference to this issue I ran into Friday, I'm wondering how this potential filepath issue can best be managed/error handled. Right now, the path is relative to the VRT file:
<SourceFilename relativeToVRT="1">sample.tif</SourceFilename>@hrwgc , I spoke with @yhahn about this and he mentioned that in the past this was a pain, because you had to always input the absolute path. Any thoughts on a better way to control the source filename in a vrt?
@springmeyer Is there something that can be done on the mapnik/gdal side?
On the
mapnik-omnivoreside, a potential strategy to handle this possible file error would look something like...vrtfile<SourceFilename relativeToVRT="1">...and grab the filenamevrtfileThe text was updated successfully, but these errors were encountered: