How should we store external data sets that are on order 100 GB?
Is git-lfs
still the way forward?
From the deployment side, do you have concerns or suggestions for having 100 GB data sets available to build, test, and validation slaves?
The particular context motivating this question is expanding the validation data sets. In particular, @price is kindly helping to make available curated (public) data from HSC for use in testing. To fully test co-addition and get large enough areas to have enough objects to perform the variety of tests we plan, we would like to store a reference dataset of 138 GB.