Kyleigh Kowalski

Meeting:  7/ 22:

• Remove 7x7 narrative, redo k-means w/o 7x7, compare clustering results, fix figures 6 and 7

  • remake cluster files for (Fig 6. All: in k=7, move hum 3 to HR cluster), (Fig 7 k=5: move rub 3 to rub2,  move rub 1 and 5 to 4 and 6, move LR 2 and all rubinf (except 1) to diff cluster) (Fig 7 k=6, move rub 1 to 3 and 5) (Fig 7 k =7: move rub 1 to 3 and 5)

• Improve conclusion

• Remake study site map

• Update figure folder & captions

Meeting:  6/19 :

• Do direct comparisons between skew work and non skew data and then summarize results for JVGR publication

Meeting:  6/5 :

• Skew Normal mayhem: try removing shape parameter and keeping the other terms

  • • Remove shape: https://drive.google.com/drive/folders/1-G3uxdhjZr4SzjkeyhUMoVkEOm0wrcet 

    • Remove shape and kurtosis: https://drive.google.com/drive/folders/1Q-zQYEvPPQQbQdq-Gv909gyzp1KjJHp2 

    • Remove shape, kurtosis, and scale (just location): https://drive.google.com/drive/folders/15LiUZ0TZGzrAK1N71qzjyM1DcLc5D9eL 

    • Shape and kurtosis seem best? But is it "better?" Maybe shape and kurtosis added on to original input data?

• Chat with Dali about how to make normalized frequency distributions, Slope values at 0-90, bins of 0.5 degrees, count/countsum normalization type, put everything in one chart, normalized freq all at once, color by type, plot by site 

• Finish JVGR draft and tag to SKN: 

Meeting:  5/29:

• Get Skew Norm for RMS Slope: Try workaround method for the frequency curve

  • • fit the distirbution using the weight for each bin center based on normalized freq. scipy.stats.skewnorm.fit does not support weight values, but used scipy.optimize.curve_fit to do a weighted fit to the skew-normal PDF

• Try kmeans using Slope & RMS height & RMS Slope (10cm, 50cm, 1m, 2m, all)

  • 10cm, 50cm, 1m, 2m - bad cluster results, no consistent improvement at any resolution except maybe 50cm?

  • All resolutions at once - bad cluster results (tried two methods, one leaving high values (4) as is, one using proxy values (4) from same morphology

  • incorporate into manuscript as appropriate

• Try kmeans on individual parameters

  • Slope:  https://drive.google.com/drive/folders/1JohT-Pkfw-0rYf_nvHvqEXJYiZNJIW3_ 

  • RMS Height: https://drive.google.com/drive/folders/1zKZJR0-gAVfJwreRQ8umx3zXViCiEOva 

  • RMS Slope: https://drive.google.com/drive/folders/1V26oU8fBsGlIeQdy2Gg8_2BY8bWzal0- 

• Be available to assist Dali with the high-res roughness and associated stuff

• JVGR guidelines, files

  • • Abstract

    • 9/9 figures downloaded & caption document w/ figure title as first sentence in caption

    • add citation

    • highlights file

    • equations in text, not images

Meeting:  5/22:

• Get Skew Norm for RMS Slope: https://drive.google.com/drive/folders/1iIT5wCfh6lJIQkYY3StHmngAFO2vRXcf had to reconstruct pseudo numbers for fitting? multiplied norm freq values (scaled to 100) by 100 to get a suitable number of samples since skew norm fit can't be applied directly to a normalized histogram, needs the raw data? Other possible workaround, using different fit method to get skew coefficients, but more complicated. 

• Try kmeans using Slope & RMS height & RMS Slope - No license for Microsoft products? :( can't make the kmeans files  

  • incorporate into manuscript as appropriate

• Be available to assist Dali with the high-res roughness and associated stuff

• Look up author guidelines for JVGR, BullVolc, pick a preference: JVGR

Meeting:  5/8

• Normalize the histograms using count / countsum instead of Density = True

  • • New normalized results: https://drive.google.com/drive/u/0/folders/1xQF2OF0ejGG_gpDw0HK4mzDTB1NwBIPN 

    • RMS Height & Slope are good, but not AR. Also still need to test RMS Slope

• Invert AR then calc skew normal coefficients

  • • Inverting AR requires all 0s to be removed... got odd results when printing how many values were removed, and shape values are still large

• Add graticule

• Back up all data to drive 

• Dali's Slope Curves?

• Move data to new computer (check with CB about best machine option)

Meeting: 4/17 & 5/1

• {SKN look at skew normal code}

• Check for bad shape parameter values associated with distributions that have vertical cut at 0

  • • Yes, only 2 plots have vertical cut at 0 which corresponds with the 2 bad shape parameters 

• Build table of skew normal coefficients (shape parameter plus others) for each training patch

  • run kmeans on that file

• Edits to thesis: highlight new text so SKN can review faster

• Calculate skew normal coefficients for all available raster data (Slope, AR, & RMS Height, ... do not have rasters for RMS Slope, attempt to determine skew from the text file format again for RMS Slope)

  • • Lots of large values for AR :( possibly for RMS Height too?

• Add graticule to map using Powerpoint / Arc 

• Density = True  (area under curve = 1) vs counts / counts.sum (sum of data = 1), https://medium.com/@gawainchin/understanding-density-in-histograms-why-values-can-exceed-1-10d8ce8721cf 

Meeting: 4/10

• Send code and input file to SKN; both of us tinker

• Finish slides, practice defense 4/7

Meeting: 4/3

• Instead of moving window prediction of morphology using means of complete distribution, write code to find skew normal coefficients for each of the inputs. 

  • put the moving window prediction content into the "future work" section

• Thesis: Final Draft due to committee (3/30)

• Thesis Presentation Slideshow: due 4/7 for practice defense

  • • Questions: 

      • • AR and RMS Height figures from Hester 

        • Acknowledgments slide

        • Reimbursement?

        • Hours?

• Send poster copy to SKN

  • {send copy to COTM}

Meeting: 3/20

• Statistics model

  • Possibly drop, too much overlap?

• Modeling using k-means

  • success! kind of? able to generate a classification map now using k-means on just 1 parameter (slope, 1m res & 30 pixel window size, 50% overlap, 7 clusters), but difficult to interpret which class is which aside from smooth and HR

  • now how to do it using multiple rasters at once?

• Thesis: Final Draft due to committee 3/28

• Travel Reimbursement

• Thesis Presentation Slideshow: due 4/7 for practice defense

• Future Work

  • Skew normal coefficients

  • Update poster & share with COTM

  • Publication draft

Meeting: 2/25 - 3/6

• Statistics:

  • Visualize as mean w/ std dev error bars, and mode (integerized)

  • Include AR, RMS Height, RMS Slope, statistics as well

• Modeling:

  • DEMs have not been filtered for vegetation, use NDVI to mask it out > features do not look like vegetation, no vegetation appears to be captured by the DEMs

  • Make decision tree using Slope, AR, RMS Height, and RMS Slope stats results

  • Make one that utilizes all resolutions to identify which morphology a patch is

  • Make one that utilizes only 1-2 m resolutions to identify which morphology a patch is

  • *AND Model based on running k-mean on each patch as new input into a k-means model, starting with the training data. As the new k-means for each patch shifts, use a majority match method to align it with the new grouping schema (additional column) - have option for an intermediate type is majority cannot be determined

  • Apply to COTM (1m & 2m), HHA (1m & 2m), and North Robbers?? (this dataset is only 30m resolution)

• Thesis:

  • ~10 pages writing and all sections drafted by Feb 24th

  • 2 figures complete by Feb 28th: 0/2

• LPSC

  • Register

  • Book Flight

  • Poster! (send draft to SKN Monday 3/3)

    • • Headings, with evidence

      • Problem Statement and background in upper left - morphology is important indicator of processes and conditions, but qualitative field observations. We aim to...

      • Keep it pretty general, emphasize the reasons for what we've done and why its valuable and then talk a little about methods. Value of using different scales and roughness. Citations box at the bottom [ abbreviated style], Acknowledgement section. Place figures first. Height max: 42", width max: 44", min font size: 24

  • Cash Advance Amount is now slightly under expected costs if Dali is not coming?

Meeting 2/6

• Statistics

  • Determine statistics based on the actual slope values (not binned or normalized), to try and determine stat logic gates for the morphologies

  • Verify the statistics functions are working right

  • Testing different site group combos: https://drive.google.com/drive/u/0/folders/12gpue8q9HobSZpL8Kr8lBb_Cj7Rc0F-k 

    • • New Groups v1: based on 3x3 10cm slope K means cluster result, 7 clusters

      • New Groups v2: 4 groups based on Mean statistics of the slope data using New Groups v1

      • New Groups v3: 5 groups based on Mean statistics of the slope data using Apriori Labels ( I like this one the best! Many apriori labels maintained, except rubinf & lr, rub & lb, and slb & smth are combined respectively.)

      • Probably worth generating a chart that plots the mean with the std dev around it, std dev still seems high/follows trends in mean very closely?

• New DEMs

  • start testing area for 1m and 2m identification (no PCA)

  • Walking window (probably at least 30 or 40 pixels based on slope sample visual results) that calculates surface characteristics that best identify morphologies

    • Overlapping moving window (~50% overlap?)

    • Options: kmeans walking window or statistics based if the statistic are different enough form each other

• Thesis

Meeting 1/30

• Statistics: 

  • Calculated median, st dev, and kurtosis for: 3x3 & 7x7 Normalized Slope and Normalized Slope Samples grouped by a priori morphologies

  • Visualize the statistics somehow for easy comparison: https://drive.google.com/drive/folders/1cjX0KsCCLeiu2I4TJb5ZBFgeQqr8KB3R 

  • Determine new groupings and then run this again using those as groupings instead of a priori labels

• Acquire Southern COTM, HHA, and Cerro Grande DEMs from Dali

  • start testing area for 1m and 2m identification (no PCA)

  • Walking window (probably at least 30 or 40 pixels based on slope sample visual results) that calculates surface characteristics that best identify morphologies

• Work on thesis draft

• Pick up Travel Advance and begin booking for LPSC

Meeting 1/23

• Thesis defense

  • Email thesis committee 3rd member

    • email committee with WhenIsGood or When2Meet to schedule defense

• Where to pick up travel advance?

• PCA

  • Mosaic PCA  

    • • rerun mosaic PCA

      • clip mosaic PCA back into sites and export rasters

      • generate structure in MATLAB for input to kmeans

      • Run Mosaic PCA kmeans  https://drive.google.com/drive/u/0/folders/1kMjE-6fKYNXPT3T5z-AT4c3EzdkFmee6 

  • Site PCA kmeans: didn't yield great results previously. Restored site PCA 10cm rasters using PCA_site.py just in case.

  • extract values from PCA transformation

  • Spot-check the small-patch eigenvectors relative to the bulk analysis

• Rerun... more?

  • Statistics: median, st dev, kurtosis for getting the average of the slope dataset to compare samples to. Do by site and compare to new named groups.

• Acquire Southern COTM DEM from USGS Explorer

  • Issue w/ coverage: https://docs.google.com/presentation/d/1Pe7L-ZQdwJGMXs64ovJfjb4rCKJqn44knhURb1WHCNk/edit#slide=id.g31f09a191fb_0_104 

    • start testing area for 1m and 2m identification (PCA and non-PCA)

    • Walking window (probably at least 30 or 40 pixels based on slope sample visual results) that calculates surface characteristics that best identify morphologies

• Shared raster symbology: https://community.esri.com/t5/arcgis-pro-questions/setting-min-and-max-values-across-different/td-p/382868

• Writing: 

  • Initialize rough draft

  • Compile all relevant writing so far

Meeting 1/16

• LPSC Logistics

  • Travel form

  • Poster: look up dimensions, make layout boxes, start with images

• Defense & Thesis Deadlines / Expectations Logistics

  • Committee (Donna or Carrie)

  • Defense Date & Final Draft Deadline (2 weeks before defense)

  • Thesis length? Content? Figures that support findings comprehensively??

• Wrapping up Research

  • Re-run slope processing on new machine

  • Rerun... more?

    • Statistics: median, st dev, kurtosis for getting the average of the slope dataset to compare samples to. Do by site and compare to new named groups.

  • Acquire Southern COTM DEM from USGS Explorer

    • • start testing area for 1m and 2m identification (PCA and non-PCA)

      • Walking window (probably at least 30 or 40 pixels based on slope sample visual results) that calculates surface characteristics that best identify morphologies

  • PCA:

    • Run Kmeans with site PCA > not good: more mixing than anything else so far

    • Run Kmeans with mosaic PCA > clip mosaic PCA back into sites, export rasters, generate structure in MATLAB for input to means 

    • extract values from PCA transformation

    • Spot-check the small-patch eigenvectors relative to the bulk analysis

Weekly targets: week of 1/13 (date TBD)

• PCA: run it now that the glitch has been caught

  • Run Kmeans with site PCA > not good: more mixing than anything else so far

  • Run Kmeans with mosaic PCA > clip mosaic PCA back into sites, export rasters, generate structure in MATLAB for input to means 

  • extract values from PCA transformation

  • Spot-check the small-patch eigenvectors relative to the bulk analysis

• Acquire Southern COTM DEM from USGS Explorer

  • start testing area for 1m and 2m identification (PCA and non-PCA)

  • Walking window (probably at least 30 or 40 pixels based on slope sample visual results) that calculates surface characteristics that best identify morphologies

• Slope samples: finish re-run

  • • Statistics: median, st dev, kurtosis for getting the average of the slope dataset to compare samples to

• LPSC abstract: send drafts back and forth via email for revision no later than 1/3

Weekly targets: 12/12

• PCA: Use PCA results (individual and comprehensive) in kmeans

  • • Export multiband raster to csv

    • Run Kmeans with site PCA > generate structure in MATLAB for input to kmeans

    • Run Kmeans with mosaic PCA > generate structure in MATLAB for input to kmeans

    • Generate visual based on morphology for site specific PCA trends?

    • Eigenvalues: can inform us about differences between components?  https://drive.google.com/drive/u/0/folders/1lXlwL6qJozbnLmxkkB2j_Ilokf-rgPoh 

• Ask Dali about DEM source: her DEM only covers the southern part of CoTM

• Slope Samples: Try to implement Chi-Squared distance in Python

  • • Rerun MATLAB code now that its fixed

    • not expected to finish, but run a few and then compare to check for rational output

    • Discovered that the normalized whole slope data had actually been including an extra bin (16 total instead of 15), fixed, regenerate MATLAB figures & back-up the .m scripts, redo k-means (all and slope?), did not need to redo PCA because the slope raster was used, not the binned normalized slope data, the sampled slope data was also binned correctly

    • Generated Trial 1 statistic comparing each sampled file to its matching slope file, but what about averaging the slope data first?

• LPSC Abstract due Jan 7th

  • Two page mini paper, 2-columns, and abbreviated citations, 1-2 figures > tiling pictures of PCA of different morphologies w/ labels, upload progress so far

  • I want to generate a figure that shows all the results of kmeans clusters at once like Hester's for the Poster (Illustrator)

Weekly targets: 12/6

• PCA

  • Run version of PCA on all sites combined into a single raster

  • Compare to the independent runs

  • https://docs.google.com/presentation/d/1Pe7L-ZQdwJGMXs64ovJfjb4rCKJqn44knhURb1WHCNk/edit#slide=id.p 

• Look into LiDAR resources to make 1m and 2m DTMs for whole area processing

• Try to implement Chi-Squared distance in Python

  • not expected to finish, but run a few and then compare to check for rational output

Weekly targets: 11/22

• Unsubmitted Expense Report from fieldwork?

• PCA

  • Resample all 2m rasters for RMS Height and RMS Slope to 1m

  • Fix inconsistent and duplicated file names

  • Run missing PCAs

  • Compare PCA results to kmeans > Not similar groupings :(

  • Run version of PCA on all sites combined into a single raster

    • • Mosaicked AR for all resolutions, still need to mosaic RMS Height and Slope

• Look into LiDAR resources to make 1m and 2m DTMs for whole area processing

• Try to implement Chi-Squared distance in Python

  • not expected to finish, but run a few and then compare to check for rational output

Weekly targets: 11/15

• Figure out logical site groupings based on kmeans for each resolution, cons/pros of each resolution

• Figure out how to do r squared or other related metric relative to existing data, goodness of fit between two lines with one being "correct". Average the standards for each morphology/resolution combination

  • • Earth Mover's Distance or Chi-Squared distance using Python?

• {SKN: upload the AR and RMS directories}

• Start on PCA

  • • Rasterized all AR and RMS Height data

    • I only have 2m, 50cm, and 10cm, but not 1m data?

    • for AR there's a lot of 0s that make a thick border (7 pixels) around the 7x7 and a thin border (1 pixel) around the 3x3. Good news, that means that all data for each site are overlapping and the same size... bad news, 0s? The RMS data doesn't have a border. In the slope rasters I still have -9999 values which means that there are holes in the PCA, they get removed from the exported slope textfiles, should I find a way to fill them?

    • First run: 3x3 10cm filled & 3x3 2m filled

      • • https://docs.google.com/presentation/d/1Pe7L-ZQdwJGMXs64ovJfjb4rCKJqn44knhURb1WHCNk/edit#slide=id.g314e87479d7_0_88 

        • https://docs.google.com/document/d/1dIUCQhUVohr1WEEA7oXgrZwfvxAotR_8y7Tie2c8yso/edit?tab=t.0 > compare to 2m kmeans patterns

Weekly targets: 10/25 - 11/1

• Slope sampling:

  • • Fix script, re-process files, re-generate charts in MATLAB, continue comparison

• Kmeans:

  • • Slope and RMS Height > RMS Height is actually identical to RMS Height and Slope, what does this say about Slope? Does it only apply to this specific model, or does this tell us that Slope is not highly influential overall? Because Slope on it's own still does fairly well... but independently, Slope and RMS Height generate somewhat different results from each other.

    • Run with all resolutions put together > kind of summarizes a lot of common patterns, but not necessarily great at picking things out

    • Interpret things as orbital vs sub-orbital capabilities, investment worth, (10cm and 50cm, how beneficial is the ground data? Think in terms of recommendations for next missions)

• PCA:

  • • Pick 1 data resolution, make rasters of RMS, AR, and Slope (ex, 10cm 3x3) > I only have the cluster results for AR, not X Y data... work around?

• If time: 

  • • Back-up most recent scripts, data, and charts from DML, Laptop, Python, and MATLAB to Google Drive & Thumbdrive

    • Read/summarize Gawronska et al. (2020)

Weekly targets: 10/18 - 10/25

• Send CV copy to SKN

• Slope sampling:

  • Check on summation/normalization question: Normalization incorrectly being applied to sample files... Will figure out how to fix and re-process sampling files. Then will continue putting results in slideshow for comparison.

• Kmeans plans: 

  • (1st) Remove 7x7 and assess difference. 

    • • Removing 7x7 improved ability to group slb, smth, and rub together, especially at 10cm and 1m, obviously no difference for 2m, overall I think 7x7 is not improving the model's interpretation of the data, and if anything is actually causing more mixing

  • (2nd) Run with just 3x3 Slope (3rd) just 3x3 AR (4th) just 3x3 RMS Height and (5th) just 3x3 RMS slope

    • • Slope: appears to be beneficial and highly influential to the initial model, but performs better independently for 10cm k=7 for grouping rub, smth, hr, rubinf, lb, and slb 

      • AR: appears to be beneficial to the model. Independently it is better at grouping rub, smth, lb, and rubinf at 10cm, though it is weaker for slb, lr, and hum. Performs well at k = 6 and 7  

      • RMS Height: appears to be beneficial at 10cm but definitley "harmful" at 2m. Not objectively worse/better at 1m and 50 cm but different which is possibly leading to confusion (except 5 cluster 1m which was nearly identical). Performs well at k = 6 and 7 

      • RMS Slope: consistently deviates slightly from results using all data but doesn’t appear to be objectively worse or better, possibly responsible for confusion, possibly “harmful” to the model 

  • (6th) Run 3x3 Slope & AR (7th) 3x3 Slope & RMS Height (8th) 3x3 Slope and RMS Slope

    • • Slope and AR: appears to be improved for rubinf and rub at 10cm k=6 and 7, but not objectively better/worse for other resolutions. Still struggles with grouping the two slb together. Even though Slope and AR appear to be independently beneficial to the model, this indicates that there is atleast one other important dataset

      • Slope and RMS Height: **need to re-run, accidentally uploaded same data as RMS Height

      • Slope and RMS Slope: appears to be best at 10cm k=5, comparable to initial kmeans for 2m, 1m, and 50cm 

  • (9th) Run with all resolutions put together

• PCA?

  • Pick 1 data resolution, make rasters of RMS, AR, and Slope (ex, 1m 3x3)

• If time: 

  • • Back-up most recent scripts, data, and charts from DML, Laptop, Python, and MATLAB to Google Drive & Thumbdrive

Weekly targets: 10/4 - 10/18

• Kmeans: https://docs.google.com/presentation/d/1XmxUgEIPTAGKY5XIHZR-6grsqiLHepiTLqZOFa5R_2s/edit#slide=id.g307d4c1b22a_0_29  

  • Cross-check how individual sites group/split across the numbers of groups and resolutions

  • Arrange graphs in sets by data resolution

  • Can graphs automatically show site names?

  • Write draft text: which combinations are best? What things are weak? Does this still match from Hester's findings? How would you characterize the groups in a way that would provide meaning for a user?: 

  • nc1_1_slb, ncw3_smooth look different from other sites? Only weird splitting at some resolutions, what is it about the rubinf, rubbly, lr, and lobate combinations that makes them grouped together so often (rubbly most commonly kicked out)? Look at site data. 

  • Characterize best resolution/cluster combinations for each morphology. Identify how it can perform at different resolutions for applications to other datasets. Define names for grouped morphologies for given resolutions.

  • Do clusters for situation that combines the different data resolutions and reassess results

• Sampling: https://docs.google.com/presentation/d/1qGZwUFJoZ60AP-o0hUnwyZJ35UFCfaNxBq7jfZ_Qpbo/edit#slide=id.p 

  • argue with Matlab about renormalized graphs > one file had nan for one sample? just deleted, probably weird sample clip placement, only 1 out of many thousands of sample files had the issue

  • Push Trial 3 data to laptop and finish generating graphs of limited normalization

  • Finish importing charts and compare limited to non-limited normalization results: 

  • Is the 10 cm wilder for 3x3 or 7x7 relative to the 2 m?

  • ** Ensure normalization is consistent across slope and slope samples. Consider using statistics such as median/mode/st dev to connect slope sample signatures to slope signatures due to downward shift?

  • Which sample size resolution - morphology combos are best?

• If time: 

  • • Update original Slope Analysis scripts to be more efficient now that I'm improving in Python (1 out of 2 scripts revamped)

      • • Validate and then back-up new scripts in drive & thumbdrive

mini check-in 9/18

• Grad school app stuff

• LPSC March 10-14, 2-page mini paper abstract due by Jan?

Weekly targets: 9/20

• Random Samples: [Wednesday in DML]

  • • Normalize within window of slope angles considered (the tail is cut off)

    • Is the 10 cm wilder for 3x3 or 7x7 relative to the 2 m?

    • Which resolution - morphology combos are best?

• K-means clustering:

  • • Completed & visualized results for 8, 7, 6, and 5 clusters: https://docs.google.com/presentation/d/1XmxUgEIPTAGKY5XIHZR-6grsqiLHepiTLqZOFa5R_2s/edit#slide=id.g2f8b4b78e8c_0_19 

• If time: 

  • • Update original Slope Analysis scripts to be more efficient now that I'm improving in Python (1 out of 2 scripts revamped)

      • • Validate and then back-up new scripts in drive & thumbdrive

Weekly targets: 9/13

• Random Samples:

  • • Generated a figure for each morphology and window size combination to compare sampling trial slope data against the respective standard slope data (Trial1 = 15px15p sample area, Trial2 = 20p, Trial 3 = 30p, Trial4 = 40p, Trial5= 50p https://drive.google.com/drive/folders/1qo8hpTp219paBvBdb1M5UKNQxsHosQfy 

• K-means clustering:

  • • Try 6 clusters instead of 8 and evaluate results: https://docs.google.com/spreadsheets/d/1vLl18oMBpuv2Ik1WJgn5NUIJVgCoPZWIGW41flI31fc/edit?gid=0#gid=0 

• If time: 

  • • Update original Slope Analysis scripts to be more efficient now that I'm improving in Python (1 out of 2 scripts revamped)

      • • Validate and then back-up new scripts in drive & thumbdrive

Weekly targets: 8/30

• Random Samples:

  • • Break down the visualizations for improved clarity

• K-means clustering:

  • • Check Hester's thesis about elbow plot; create: https://docs.google.com/spreadsheets/d/1gSeo4asyEDIA9-P0_Xh-TMSv1kq7LloYZCkzfJBSBC0/edit?gid=0#gid=0 

    • Color code and make sure only 7 groups initially > There are 8 "initial" morphologies, so it is working as intended: https://docs.google.com/spreadsheets/d/1JJqTbsAF456KKQLR4GenqaFCwKPJ6qH1lSei_CFAFQg/edit?gid=0#gid=0 

• If time: 

  • • Update original Slope Analysis scripts to be more efficient now that I'm improving in Python (1 out of 2 scripts revamped)

      • • Validate and then back-up new scripts in drive & thumbdrive

Weekly targets: 7/23 (and beyond):

• Random Samples:

  • • Update script to run all files at once for a specific trial, allow for overlap of sample sites, and 10 samples taked from each file. 

    • Process all slope files through 5 trials and upload to drive, then download to laptop

    • In Progress: Visualize results in MATLAB

      • Create figure with a subplot for each morphology for Trial 1, Trail 2, Trial 3, Trail 4, Trial 5

        • • Standard slope not appearing in all subplots??? Current visualization not ideal anyway, very chaotic

      • Script for generating individual figures for each morphology for each trial is in progress?

• RMS Slope:

  • • Complete RMS Slope calculation for all files

    • Normalize RMS files

    • Visualize normalized RMS Slope in MATLAB & verify that color coding is as intended for all morphologies

• K-means clustering:

  • • Append all data into 4 files for kmeans clustering > need to export RMS_s_3x3 MATLAB table for clustering

    • Run kmeans for each resolution

    • Ran k-means using python, clusters were assigned, but sns.pairplot visualization couldn't handle the amount of data, other visualization options?

• If time: 

  • • Update original Slope Analysis scripts to be more efficient now that I'm improving in Python (1 out of 2 scripts revamped)

      • • Validate and then back-up new scripts in drive & thumbdrive

    • Next week: field work help w/ IDEAS campaign?

Weekly targets: 7/11

• Random Samples:

  • • Plot random samples against standards for each resolution (not using 5x5 or 10x10 pixel window, test 40x40)

    • Did not have time to test 40x40, but samples vs standard are here: https://drive.google.com/drive/u/0/folders/1uFBEJtNvd4838hWkyf1OSGaVa0NQQNEo 

• RMS Slope:

  • • Multiply my normalized freq by 100 to represent percent, also do not have to match the RMS Height bins, maybe do 1 degree bins from 0-15

    • Look further into RMS files compared against the freq chart, make charts in excel to compare

      • • Looked more closely at the file data and it was a site issue, the first sites I had preliminary looked at weren't contributing to the first frequency values, but by looking at all the different sites for rubinf it lined up.

    • RMS Slope is a different measurement than RMS height, need to verify if RMS Slope is helpful in distinguishing between similar slope morphologies such as rubinf and lobate 

      • • It's particularly the coarser resolutions which are more similar to each other, best case scenario is to see significantly different trends between the 2m and 1m resolutions between morphologies

• K-means clustering:

  • • Using MATLAB, make slope clustering files in compatible format

    • Validate Slope, RMS, and AR then append all into files for kmeans clustering

• If time: 

  • • Update original Slope Analysis scripts to be more efficient now that I'm improving in Python (1 out of 2 scripts revamped)

      • • Validate and then back-up new scripts in drive & thumbdrive

    • Practice describing the field samples & acquire copy of mineral % guide from SKN

      • Basalt w/ Olivine, plag, mayyybe clino pyroxene (black, 90 cleavage) no hornblende, weathering of olivines may look like garnets

• Field work assistant logistics

Weekly targets: 7/2 (zoom)

• RMS Slope

  • • Download Hester's clustering files and pull what is helpful / spot check to verify

    • To do: Update RMS normalize script to have bins from 0-5 with .005 degree intervals (100 bins) in order to compare values against Hester's. If RMS results are consistent, just copy over Hester's RMS values rather than re-running RMS for each file

      • • RMS values are not similar. Hester's add to 100, mine add to 1. She is seeing values in bins much earlier than I am. Why did she use such tiny bin intervals? Is something wrong with my RMS?

    • Ultimately want to look at similar morphologies (such as Lobate and Rub Inf) to see if RMS slope can help distinguish between them since the slope patterns are so similar

    • Grab images of all sites for reference

• Slope Sample Analysis

  • • To do: Repeat with different parameters until trend arises, then try for other sites/morphologies, record differences in morphologies, but find what would work for all

      • • • Completed 5 trials w/ a rubbly inflated site and 5 trials with a smooth site

    •  Ultimately want to determine how many pixels are necessary to reasonably interpret the morphology from the normalized slope frequency

• K-means clustering:

  • • Using MATLAB, make slope clustering files in compatible format

    • Validate Slope, RMS, and AR then append all into files for kmeans clustering

• If time: 

  • • Update original Slope Analysis scripts to be more efficient now that I'm improving in Python (1 out of 2 scripts revamped)

      • • Validate and then back-up new scripts in drive & thumbdrive

    • Practice describing the field samples & acquire copy of mineral % guide from SKN

      • Basalt w/ Olivine, plag, mayyybe clino pyroxene (black, 90 cleavage) no hornblende, weathering of olivines may look like garnets

Weekly targets: 6/4

• RMS Slope

  • • Fill holes in slope analysis (fill with -99999) at the end of script before running export script, then re-run script

    • Upload new slope data for RMS script processing to drive and back-up everything else on thumbdrive (computers being wiped soon)

    • Redo RMS processing in VB > -99999 values not present in Slope .txt files?? > Was accessing old files, download correct ones to laptop and run the failed ones

    • Process the 3x3 lobate sites in RMS & upload to drive

    • Re-generate MATLAB Slope figures to ensure things are working correctly w/ slope data

    • Convert to normalized frequency first, then visualize RMS Slope rubinf  and lobate data in MATLAB 

    • To do: Compare RMS slope signature to normal slope signature for the similar morphologies

    • Ultimately want to look at similar morphologies (such as Lobate and Rub Inf) to see if RMS slope can help distinguish between them since the slope patterns are so similar

• Slope Sample Analysis: 5 pixel area, 5-10 random non-overlapping samples *2m doesn't have enough space for 10 samples

  • • Check the 5x5 m sub-samples (incorrect size from imperfect overlap w/ pixel boundaries) 

    • Run on point files of slope instead 

    • Complete 3x3_bc1_rubinf 1m, 2m, 10cm, and 50cm at 5 pixel sample area then upload to drive and put on laptop

    • To do: Visualize sample areas for 3x3_bc1_rubinf in MATLAB

    • To do: Interpret/improve visualization of 3x3_bc1_rubinf and then repeat process for other sites??

    • Ultimately want to determine how many pixels are necessary to reasonably interpret the morphology from the normalized slope frequency

• If time: 

  • • Extract from MATLAB structure array to combine all frequency distribution sets as training data for kmeans (Wednesday)

    • Update original Slope Analysis scripts to be more efficient now that I'm improving in Python (1 out of 2 scripts revamped)

    • Practice describing the field samples & acquire copy of mineral % guide from SKN

Weekly targets: 5/28

• RMS Slope

  • • Re-generate MATLAB Slope figures to ensure things are working correctly w/ slope data

    • Redo RMS processing in VB > -9999 values not present in Slope .txt files?? Need to investigate

    • Ultimately want to look at similar morphologies (such as Lobate and Rub Inf) to see if RMS slope can help distinguish between them since the slope patterns are so similar

• Slope Sample Analysis

  • • Visualize in MATLAB

• If time: combine all frequency distribution sets as training data for kmeans

  • • Extract from MATLAB structure array

    • Update MATLAB script to ignore -99999 values if needed?

Weekly targets: 5/21

• RMS slope: 

  • Grid misfit error: import XY data and look for a bonus pixel on one of the sides. If there, cut it and re-export

  • Ultimately want to look at similar morphologies (such as Lobate and Rub Inf) to see if RMS slope can help distinguish between them since the slope patterns are so similar

• How many pixels are needed to lead to reasonable interpretation of match to morphology normalize slope frequency

  • • Use indexing of UTM coords to randomly [random # gen] sample 10 5x5m windows, need to make sure it can't exceed the site extent

    • Start w/ one 3x3 site at 1m resolution, then play with size of sample window and # of samples

    • Random # gen, x and y axis on 0-100, random num gen is normally on scale form 0-1, tell it desired range ex rand(0,95). Use rand num to define two random numbers (N, E) of the origin (limit to avoid exceeding boundary), then add 5m for example for 5m window

    • Then output histogram for each window, visualize on same plot as whole patch (one color) then the smaller patch results (diff color)

• If time: combine all frequency distribution sets as training data for kmeans

  • • Extract from MATLAB structure array

Weekly targets: 5/9

• RMS Slope

  • • Export 'simple' slope to .txt containing x, y, and 'z' = slope, columns

    • Edit script to match order: OID_, pointid, grid_code, POINT_X, POINT_Y

    • Re-run in VB >> some are working, others are not?

    • Ultimately want to look at similar morphologies (such as Lobate and Rub Inf) to see if RMS slope can help distinguish between them since the slope patterns are so similar

• How many pixels are needed to lead to reasonable interpretation of match to morphology normalize slope frequency

  • • Use indexing of UTM coords to randomly [random # gen] sample 10 5x5m windows, need to make sure it can't exceed the site extent

    • Start w/ one 3x3 site at 1m resolution, then play with size of sample window and # of samples

    • Random # gen, x and y axis on 0-100, random num gen is normally on scale form 0-1, tell it desired range ex rand(0,95). Use rand num to define two random numbers (N, E) of the origin (limit to avoid exceeding boundary), then add 5m for example for 5m window

    • Then output histogram for each window, visualize on same plot as whole patch (one color) then the smaller patch results (diff color)

  • hold: combine all frequency distribution sets as training data for kmeans

    • • Extract from MATLAB structure array

• Still need to register for 1 indep research honors credit

Weekly targets 5/2:

• Do test runs of RMS Slope for key files (lobate and rub inf)

  • Export simple slope to text file for adaptation of Hester's code

    • • what's the res of a simple slope for a given resolution without defining a window size...acquire resolution of non-windowed (simple) slope for a given site/resolution combo >> "The default value is the input raster cell size, resulting in a 3 by 3 neighborhood." >> The current 3x3 cell slope rasters are as "simple" as we can get (I think?)

        • • • • Lobate site: bc1_4_lb | Simple Slope Resolutions: 2m: 2x2m, 1m: 1x1m, 50cm: 0.5x0.5, 10cm: 0.1x0.1

              • Rubbly Inflated site: bc1_1_rubinf | Simple Slope Resolutions:  2m: 2x2m, 1m: 1x1m, 50cm: 0.5x0.5, 10cm: 0.1x0.1

      • Exported "simple/3x3" slope to Drive, See if the f90 code will run with these? >> Might need a different compiler? I don't know how to run in Visual Basics because I don't know how to set up debugging, but installed extensions in VB that imply I should be able to run fortran code? >> Need to run 'debugging' to create executable, then run executable file

        • • PS C:\Users\kylei\Downloads\College\SUBSEA\RMS_Slope> FTN95 RMSHeight.f90 -o rms

          • PS C:\Users\kylei\Downloads\College\SUBSEA\RMS_Slope> ./rms.exe

      • Install f90 compiler for .f90 script, usually bundled with C compiler, look for script editor, or in command window: f90 sciptname.f90 -o outfilename

• If time: pull random 3x3 and 7x7 patches of data from the defined morphologies and compare their distributions to the standard curve groups

  • • Figure out how to make sure sample AOI is exactly 3x3 or 7x7, 1 sample of each per site, in similar location within the site

      • • • Place points (roughly) in center of each site, then for 3x3_1m, buffer by 1.5m, then feature envelope to polygon = 3x3 clip boundary? But not perfectly placed. Should I continue in this fashion? >> New plan

• If time: combine all frequency distribution sets as training data for kmeans

  • • Extract from MATLAB structure array

Weekly targets 4/25:

• MATLAB: Create plots of each morphology, separated into different plots by input resolution

  • • Update morphology labels and list and structure array

    • Update i = 6 to i = 8 for subplot generation

• Take hwy 'smooth' and 'rubbly' out: Update naming conventions in DML, replace data in google drive & thumbdrive. hwys_smooth --> hwys_lr,  hwyr_rub --> hwyr_hr, ...maybe 'low relief' and 'high relief'?

  • • Look at imagery of hwy sites and determine if hwys is a good candidate for hummocky. hwyr doesn't appear to line up with the other normalized slope freq curves

• <Hold: pull random 3x3 and 7x7 patches of data from the defined morphologies and compare their distributions to the standard curve groups>

• <Hold: combine all frequency distribution sets as training data for means>

  • • Extract from MATLAB structure array

Weekly targets 4/11:

• Smooth: review files to see if there is a pattern for the within-resolution heterogeneities

• Rubbly: check heterogeneity within group

• When feeding values into spreadsheet for means cut off at 30 deg (don't go all the way to 50)

Weekly targets 3/28:

• Process the missing sites, add to existing set

• remove the totally missing site(s) from Hester's set

• Update scripts to account for new morphologies "hum" & "slb"

• Clear data and re-run scripts

• Back-up essential data & scripts (thumb drive and google drive)

• Upload new visual results to google drive

• Read-me for data

• Read-me for scripts

• Work on MATLAB aggregation and plot generation project

Weekly targets 3/14:

• Symposium slides

  • rearrange frequency curve plots

  • snag example images from GE

• Email Sonia about slide upload

• Keep aggregating data for clustering (matlab class)

Weekly targets 3/7:

• Dig through pile of links/files for missing bc and nc files

  • build what you can *OR* email SKN that everything sucks

• Start symposium slides

Weekly targets 2/29:

• Create text files for missing bc2 and nc1 sets *check for 10 and 50cm resolution files?

  • then: re-run dataset with the missing files

• Write abstract for ISU Research Symposium (due noon 2/21, https://www.isu.edu/researchsymposium/ )

Weekly targets 2/15:

• Check on data between 7x7 and 3x3 freq dist for 2m/pixel set > solved

• Dig through clustering files and instructions

  • before adding own data, try to recreate what's there

Weekly targets 2/8:

• set curves to same x axes

• testing 2 degree bin sizes

• 7x7 curves

• Look for excel sheet from Hester for rms and areaRatio

Weekly targets 2/1:

• Make combing frequency curves for slope distributions

  • propose locations to trim datasets (if appropriate)

• Meet with Dali

Weekly targets 1/25:

• Check the data type for the grid_code field in the point data, then check what it's incorrectly converting to, just for curiosity.

• Try Feature-to-Raster alternative

• Put together a crash course guide for Dali, schedule a meeting together

• Contact Brooks about working through the issue with me

Weekly targets 1/18:

• In Arc, open one of the XYtoPoint outputs to check on data status; go from there searching for the integer switch (Find and fix the integer conversion)

• After fixing integer problem, re-run slope analyses

Weekly targets: winter break

• Break down step by step comparison between manual and scripted slope analyses, organized into slides. Target: why is the slop coming out so oddly in the scripted, and can we actually turn that lemon into lemonade?

  • • • I have identified what is wrong, but not why :( I think this goes beyond my understanding and is an issue with the data type that each process is defaulting to storing in the rasters...maybe Carrie or Keith could provide some insight on this? [https://docs.google.com/presentation/d/1T_pTQ2bHnnHFPx-GGVoWzMYhIgzWpz6P62Ef68b7ChY/edit#slide=id.g28e468475f7_0_61] 

  • for the scripted version: is there a useful difference in output for a rough vs smooth patch?

• Finish normalizing histograms

• If time: pick and read/summarize one of the papers from the 11/23 list

Weekly Targets: 11/23 

• Read & briefly summarize Kennish & Lutz (1998), Gooding (1978), Krinsley et al. (1979), Beal et al. (2016)  <-- haven't actually read these but have either cited in rationale or plan to use

Weekly Targets: 11/09 (email update; SKN away)

• Wrap up slope analyses > Finish Slope processing > Write Export code for .csv > Finish Code > Transfer all DTMs and run ALL data through the script & upload results to Google Drive

  • • (Finished slope processing! But fighting with the final export code to correctly bin and make normalized histograms of the slope data. Histograms are being made but don't actually match the data)

• Prospectus for Review: https://docs.google.com/document/d/1zydESUBTVBuSBlJLvd8x7rjNq6zDoGXigoSWDpTVB0s/edit 

• Prospectus Presentation: https://docs.google.com/presentation/d/1f4a-eRWeogmZCuxPJVvhaiZ0QvoXViaLi7bX2kiFLgk/edit 

• Fight with Point-to-Raster bug  <-- Solved

• Download and review bathymetry and backscatter(?)

• Read/summarize 2 Clague papers 

Weekly Targets: 11/02

• Update top block of website

• Keep up with scripting/debugging (if not fixed, bring quick-guide to meeting)

• Read/summarize Clague et al. (2020)

• Explore NG bathymetry data (email SKN if she sent the wrong thing)

Weekly Targets: 10/26

• Prep problem statement slides

• Start scripting

• Read/summarize Cao and Cai (2018)

Weekly Targets: 10/19

• Revise problem statement worksheet

• Send methods worksheet

• Study for midterm

Weekly Targets: 9/28 (via email)

• Redo ~7m window analyses

• Summarize Hughes et al. (2020) include project relevance

Weekly targets: 9/21

• Check Katz and Cashman (2003) and de Lima et al. (2012) about xls in lava types

• Additional check on window size via comparison with re-run on a selected patch

  • if 3x3 correct: move on to spreadsheet

  • if incorrect: redo

• Read/summarize Gregg and Fink (1995)

• Have fun on Seminar!!

Weekly targets: 9/7

• Double-check that 3x3 is in pixels, not meters 

• Work on 7x7 (and 3x3 if needed)

• Read/summarize Gregg et al. (2017), add sentence/paragraph about relevance to project

Weekly targets: 8/31

• Populate upper part of website (ex., images, overall project idea, personal goals, etc.)

• Reorient to slope data and processes

• Read Hester's thesis & paper on different types of lava flows (use paper Hester cited); write abstracts of each