CO2: 2018 data: /shared/anaconda2/bin/python2.7 /lustre/athorpe/ch4/extract_time_co2/compute_ime.py /lustre/athorpe/ch4/extract_time_co2/in_spreadsheet/FA18_CO2_Plumes_20190228.xlsx "Plume_List" -o /lustre/athorpe/ch4/extract_time_co2/output/FA18_CO2_15000ppmm_150fetch_10_20_30merge_v4 --calids v2t1 -p 22500 -m 10 20 30 -f 150 20000 15000 --clobber --randomize Generate IME: cat /lustre/athorpe/ch4/extract_time_co2/output/FA18_CO2_15000ppmm_150fetch_10_20_30merge_v4/ang2018*/*_ime.txt|sort|uniq >> /lustre/athorpe/ch4/extract_time_co2/output/FA18_CO2_15000ppmm_150fetch_10_20_30merge_v4/imevals_FA18_CO2_22500ppmm_150fetch_20190304.txt Aggregate .JSON, .PDF, .TIF: cp /lustre/athorpe/ch4/extract_time_co2/output/FA18_CO2_15000ppmm_150fetch_10_20_30merge_v4/ang2018*/mergedist10/*.tif /lustre/athorpe/ch4/extract_time_co2/output/FA18_CO2_15000ppmm_150fetch_10_20_30merge_v4/TIF_PDF_JSON/mergedist10 cp /lustre/athorpe/ch4/extract_time_co2/output/FA18_CO2_15000ppmm_150fetch_10_20_30merge_v4/ang2018*/mergedist10/*.pdf /lustre/athorpe/ch4/extract_time_co2/output/FA18_CO2_15000ppmm_150fetch_10_20_30merge_v4/TIF_PDF_JSON/mergedist10 cp /lustre/athorpe/ch4/extract_time_co2/output/FA18_CO2_15000ppmm_150fetch_10_20_30merge_v4/ang2018*/mergedist10/*.geojson /lustre/athorpe/ch4/extract_time_co2/output/FA18_CO2_15000ppmm_150fetch_10_20_30merge_v4/TIF_PDF_JSON/mergedist10 cp /lustre/athorpe/ch4/extract_time_co2/output/FA18_CO2_15000ppmm_150fetch_10_20_30merge_v4/ang2018*/mergedist20/*.tif /lustre/athorpe/ch4/extract_time_co2/output/FA18_CO2_15000ppmm_150fetch_10_20_30merge_v4/TIF_PDF_JSON/mergedist20 cp /lustre/athorpe/ch4/extract_time_co2/output/FA18_CO2_15000ppmm_150fetch_10_20_30merge_v4/ang2018*/mergedist20/*.pdf /lustre/athorpe/ch4/extract_time_co2/output/FA18_CO2_15000ppmm_150fetch_10_20_30merge_v4/TIF_PDF_JSON/mergedist20 cp /lustre/athorpe/ch4/extract_time_co2/output/FA18_CO2_15000ppmm_150fetch_10_20_30merge_v4/ang2018*/mergedist20/*.geojson /lustre/athorpe/ch4/extract_time_co2/output/FA18_CO2_15000ppmm_150fetch_10_20_30merge_v4/TIF_PDF_JSON/mergedist20 cp /lustre/athorpe/ch4/extract_time_co2/output/FA18_CO2_15000ppmm_150fetch_10_20_30merge_v4/ang2018*/mergedist30/*.tif /lustre/athorpe/ch4/extract_time_co2/output/FA18_CO2_15000ppmm_150fetch_10_20_30merge_v4/TIF_PDF_JSON/mergedist30 cp /lustre/athorpe/ch4/extract_time_co2/output/FA18_CO2_15000ppmm_150fetch_10_20_30merge_v4/ang2018*/mergedist30/*.pdf /lustre/athorpe/ch4/extract_time_co2/output/FA18_CO2_15000ppmm_150fetch_10_20_30merge_v4/TIF_PDF_JSON/mergedist30 cp /lustre/athorpe/ch4/extract_time_co2/output/FA18_CO2_15000ppmm_150fetch_10_20_30merge_v4/ang2018*/mergedist30/*.geojson /lustre/athorpe/ch4/extract_time_co2/output/FA18_CO2_15000ppmm_150fetch_10_20_30merge_v4/TIF_PDF_JSON/mergedist30 CH4: 2018 data: /shared/anaconda2/bin/python2.7 /lustre/athorpe/ch4/extract_time/compute_ime.py /lustre/athorpe/ch4/extract_time/in_spreadsheet/FA18_Plumes_20181024.xlsx "Plume_List" -o /lustre/athorpe/ch4/extract_time/output/FA18_1000ppmm_150fetch_10_20_30merge --calids v2t1 -p 1000 -m 10 20 30 -f 150 --clobber /shared/anaconda2/bin/python2.7 /lustre/athorpe/ch4/extract_time/compute_ime.py /lustre/athorpe/ch4/extract_time/in_spreadsheet/FA18_Plumes_20181024.xlsx "Plume_List" -o /lustre/athorpe/ch4/extract_time/output/FA18_1500ppmm_150fetch_10_20_30merge --calids v2t1 -p 1500 -m 10 20 30 -f 150 --clobber Generate IME: FA18 1000 ppmm, 150 fetch: cat /lustre/athorpe/ch4/extract_time/output/FA18_1000ppmm_150fetch_10_20_30merge/ang2018*/*_ime.txt|sort|uniq >> /lustre/athorpe/ch4/extract_time/output/FA18_1000ppmm_150fetch_10_20_30merge/imevals_FA18_1000ppmm_150fetch_20181024.txt FA18 1500 ppmm, 150 fetch: cat /lustre/athorpe/ch4/extract_time/output/FA18_1500ppmm_150fetch_10_20_30merge/ang2018*/*_ime.txt|sort|uniq >> /lustre/athorpe/ch4/extract_time/output/FA18_1500ppmm_150fetch_10_20_30merge/imevals_FA18_1500ppmm_150fetch_20181024.txt Aggregate .JSON, .PDF, .TIF: FA18 1000 ppmm, 150 fetch: cp /lustre/athorpe/ch4/extract_time/output/FA18_1000ppmm_150fetch_10_20_30merge/ang2018*/mergedist10/*.tif /lustre/athorpe/ch4/extract_time/output/FA18_1000ppmm_150fetch_10_20_30merge/TIF_PDF_JSON/mergedist10 cp /lustre/athorpe/ch4/extract_time/output/FA18_1000ppmm_150fetch_10_20_30merge/ang2018*/mergedist20/*.tif /lustre/athorpe/ch4/extract_time/output/FA18_1000ppmm_150fetch_10_20_30merge/TIF_PDF_JSON/mergedist20 cp /lustre/athorpe/ch4/extract_time/output/FA18_1000ppmm_150fetch_10_20_30merge/ang2018*/mergedist30/*.tif /lustre/athorpe/ch4/extract_time/output/FA18_1000ppmm_150fetch_10_20_30merge/TIF_PDF_JSON/mergedist30 cp /lustre/athorpe/ch4/extract_time/output/FA18_1000ppmm_150fetch_10_20_30merge/ang2018*/mergedist10/*.pdf /lustre/athorpe/ch4/extract_time/output/FA18_1000ppmm_150fetch_10_20_30merge/TIF_PDF_JSON/mergedist10 cp /lustre/athorpe/ch4/extract_time/output/FA18_1000ppmm_150fetch_10_20_30merge/ang2018*/mergedist20/*.pdf /lustre/athorpe/ch4/extract_time/output/FA18_1000ppmm_150fetch_10_20_30merge/TIF_PDF_JSON/mergedist20 cp /lustre/athorpe/ch4/extract_time/output/FA18_1000ppmm_150fetch_10_20_30merge/ang2018*/mergedist30/*.pdf /lustre/athorpe/ch4/extract_time/output/FA18_1000ppmm_150fetch_10_20_30merge/TIF_PDF_JSON/mergedist30 cp /lustre/athorpe/ch4/extract_time/output/FA18_1000ppmm_150fetch_10_20_30merge/ang2018*/mergedist10/*.geojson /lustre/athorpe/ch4/extract_time/output/FA18_1000ppmm_150fetch_10_20_30merge/TIF_PDF_JSON/mergedist10 cp /lustre/athorpe/ch4/extract_time/output/FA18_1000ppmm_150fetch_10_20_30merge/ang2018*/mergedist20/*.geojson /lustre/athorpe/ch4/extract_time/output/FA18_1000ppmm_150fetch_10_20_30merge/TIF_PDF_JSON/mergedist20 cp /lustre/athorpe/ch4/extract_time/output/FA18_1000ppmm_150fetch_10_20_30merge/ang2018*/mergedist30/*.geojson /lustre/athorpe/ch4/extract_time/output/FA18_1000ppmm_150fetch_10_20_30merge/TIF_PDF_JSON/mergedist30 FA18 1500 ppmm, 150 fetch: cp /lustre/athorpe/ch4/extract_time/output/FA18_1500ppmm_150fetch_10_20_30merge/ang2018*/mergedist10/*.tif /lustre/athorpe/ch4/extract_time/output/FA18_1500ppmm_150fetch_10_20_30merge/TIF_PDF_JSON/mergedist10 cp /lustre/athorpe/ch4/extract_time/output/FA18_1500ppmm_150fetch_10_20_30merge/ang2018*/mergedist20/*.tif /lustre/athorpe/ch4/extract_time/output/FA18_1500ppmm_150fetch_10_20_30merge/TIF_PDF_JSON/mergedist20 cp /lustre/athorpe/ch4/extract_time/output/FA18_1500ppmm_150fetch_10_20_30merge/ang2018*/mergedist30/*.tif /lustre/athorpe/ch4/extract_time/output/FA18_1500ppmm_150fetch_10_20_30merge/TIF_PDF_JSON/mergedist30 cp /lustre/athorpe/ch4/extract_time/output/FA18_1500ppmm_150fetch_10_20_30merge/ang2018*/mergedist10/*.pdf /lustre/athorpe/ch4/extract_time/output/FA18_1500ppmm_150fetch_10_20_30merge/TIF_PDF_JSON/mergedist10 cp /lustre/athorpe/ch4/extract_time/output/FA18_1500ppmm_150fetch_10_20_30merge/ang2018*/mergedist20/*.pdf /lustre/athorpe/ch4/extract_time/output/FA18_1500ppmm_150fetch_10_20_30merge/TIF_PDF_JSON/mergedist20 cp /lustre/athorpe/ch4/extract_time/output/FA18_1500ppmm_150fetch_10_20_30merge/ang2018*/mergedist30/*.pdf /lustre/athorpe/ch4/extract_time/output/FA18_1500ppmm_150fetch_10_20_30merge/TIF_PDF_JSON/mergedist30 cp /lustre/athorpe/ch4/extract_time/output/FA18_1500ppmm_150fetch_10_20_30merge/ang2018*/mergedist10/*.geojson /lustre/athorpe/ch4/extract_time/output/FA18_1500ppmm_150fetch_10_20_30merge/TIF_PDF_JSON/mergedist10 cp /lustre/athorpe/ch4/extract_time/output/FA18_1500ppmm_150fetch_10_20_30merge/ang2018*/mergedist20/*.geojson /lustre/athorpe/ch4/extract_time/output/FA18_1500ppmm_150fetch_10_20_30merge/TIF_PDF_JSON/mergedist20 cp /lustre/athorpe/ch4/extract_time/output/FA18_1500ppmm_150fetch_10_20_30merge/ang2018*/mergedist30/*.geojson /lustre/athorpe/ch4/extract_time/output/FA18_1500ppmm_150fetch_10_20_30merge/TIF_PDF_JSON/mergedist30 /shared/anaconda2/bin/python2.7 /lustre/athorpe/ch4/extract_time/compute_ime.py /lustre/athorpe/ch4/extract_time/in_spreadsheet/Extract_Time_2017.xlsx "Plume_List" -o /lustre/athorpe/ch4/extract_time/output --calids v2p4 v2p7 v2p9 v2p10 -p 1000 -m 10 20 30 -f 150 --clobber utchdrs outdict = dict(detid=NODATA,detdim=NODATA,detpos=(NODATA,NODATA), imevals=np.array([NODATA]),cmfnums=np.array([NODATA]), cmfvals=np.array([NODATA]),imerads=np.array([NODATA]), detrgb=[],detql=[],detmf=[],imergb=[],imelat=lat,imelon=lon, imemin=NODATA,imemax=NODATA,imesum=NODATA, fetchm=NODATA,radincm=radincm,detbounds=[], sumpix=NODATA,fetchpix=NODATA,plumasp=NODATA, #Thorpe addition pixres=NODATA,imeradsm=np.array([NODATA]),mf_detidsa=NODATA, imedivfetchfilt=np.array([NODATA]),meanimedivfetch=NODATA, stdimedivfetch=NODATA,utc_out=NODATA) #Thorpe addition utc_out=utc #Thorpe addition time imesumi,fetchmi,plumaspi,sumpixi,meanimedivfetchi,stdimedivfetchi,utci = \ [NODATA]*ndists,[NODATA]*ndists,[NODATA]*ndists,[NODATA]*ndists, \ [NODATA]*ndists,[NODATA]*ndists,[NODATA]*ndists #Thorpe addition time #imesumi,fetchmi = [NODATA]*ndists,[NODATA]*ndists #Thorpe commented out detidi = [int(NODATA)]*ndists mflab[r0:r1,c0:c1]=plab*np.int32(mfimg[r0:r1,c0:c1]>0) print('source',i,sid,cid,pid, '\nsource (lat,lon)=',(slat,slon), '\nplume (lat,lon)=',(plat,plon), cmfsuf_ang cmfsuf_ang = '_cmf_*_img' # replace '*' with calids_ang obssuf_ang = 'rdn_obs_ort' # Thorpe time cmffile obsfile openimg findhdr(img_file): mfdata = loadmaskedimage(mfinfile,load_bands=load_bands,masked_value=0, rgb_bands=rgb_bands) obsinfile def loadfiltdet(detfilt_imgf): print('loading filtered detection image: "%s"'%detfilt_imgf) detdir,detfile = pathsplit(detfilt_imgf) detimg = envi_open_file(detfilt_imgf+'.hdr',image=detfilt_imgf) ch4det = np.float32(detimg.load().squeeze()) nodata_value = float(detimg.metadata['data ignore value']) nodata_mask = ch4det==nodata_value ch4det[nodata_mask] = 0 return dict(ch4det=ch4det,mapinfo=mapinfo(detimg,astype=dict), nodata_mask=nodata_mask,nodata_value=nodata_value) Versions of results: 2016 v1n2 v2m4 2017 v2p4 v2p7 v2p9 v2p10 2016 and 2017 v2p4 v2p7 v2p9 v2p10 v1n2 v2m4 Instructions for IME parameters: Hi Andrew, Done, I actually had this in the script already but the option wasn't exposed. To set the ppmm threshold, use the -p/--ppmmthr command line argument (default=500.0 ppmm). Also, you can specify a list of mergedist values via the -m/--mergedists argument and set the maximum fetch value via -f/--fetchmax if you'd like to vary those parameters as well. See usage string below for other options which may or may not be useful. Let me know if you run into any issues. _B bbue@gibson:srcfinder$ python compute_ime/compute_ime.py --help usage: compute_ime.py [-h] [-m MERGEDISTS [MERGEDISTS ...]] [-c] [-o OUTDIR] [-f FETCHMAX] [-p PPMMTHR] [--nodetfilt] [-i inputfile] [-l listfile] [--randomize] [--filter filter] xlsfile sheetname CMF IME Computation + Product Generation positional arguments: xlsfile Source master list spreadsheet sheetname Source master list spreadsheet sheet name optional arguments: -h, --help show this help message and exit -m MERGEDISTS [MERGEDISTS ...], --mergedists MERGEDISTS [MERGEDISTS ...] List of one or more distances (in meters) to merge neighboring blobs (default=[10, 20, 50]) -c, --clobber Clobber (overwrite) existing outputs -o OUTDIR, --outdir OUTDIR Save filtered detection image to OUTDIR -f FETCHMAX, --fetchmax FETCHMAX Maximum fetch value (default=unlimited) -p PPMMTHR, --ppmmthr PPMMTHR Minimum ppmm threshold (default=500.00) --nodetfilt Do not filter out small or weakly connected detections -i inputfile, --inputfile inputfile CMF image (assumed listed in spreadsheet) -l listfile, --listfile listfile List of CMF images to process (subset of images in provided spreadsheet) --randomize Randomize spreadsheet processing order (by source id) --filter filter Compute IME for spreadsheet rows matching filters of the form "spreadsheet_column:filter_string (e.g., "flightline:ang201605")" Hi Andrew, Sorry for the confusion; the [] in the usage string means the argument is optional, and assumes a default value if unspecified. So, to set the ppmm threshold to 350 you'd use "-p 350" and to use mergedists=10,25 you'd use "-m 10 25". So, for the params you specified below it'd be: /shared/anaconda2/bin/python2.7 /lustre/bbue/ch4/compute_ime/compute_ime.py lustre/athorpe/ch4/gen_ime_geotif/input/SP17_Master_Source_List_IME_20170822.xlsx "SP17 plumes 2" -o /lustre/athorpe/ch4/gen_ime_geotif/output/ime_gtif_1000ppmmm_20170817 -p 1000 -m 10 25 -f 200 Note that mergedists is the only param that can take multiple values (hence the multivalued spec in the usage string: [-m MERGEDISTS [MERGEDISTS ...]]), the other params -p/-f are floating point scalars. _B