Structure of the autonomous-phase-sensitive-radar github organization

[jkingslake]

In a zoom meeting of 18-20 (A)pRES-interested people on Dec 4th 2024 several ideas were put forward for useful resources/initiatives.
From my perspective, the four main ones were:

1. discussion - a forum like this one where people can ask and answer questions and discuss what resources we want to produce etc.
2. documentation - any and all material on the theory, operation, etc. of ApPES for new to experienced users
3. recipes - a place to put code that you have used for some specific purpose related to ApRES data processing, or data quality control etc. This code would not be maintained or benchmarked.
4. benchmarked code for processing the data - a code base that the community broadly agrees is the most up-to-date, benchmarked way of doing some core ApRES-processing tasks.

One way we could move forward is to have a repo for each of the these resources in this organization.

This thread can be a place for the group to discuss this suggested structure.

A few points that come to mind straight away:

• these don't need to be the only repos in the organization, but judging from the discussion on Dec 4th, they might be the most useful and widely used.
• as @Elizabethcase pointed out to me recently, the four idea above are probably very different from each other in terms of how difficult they will be to coordinate. I listed them in what I judge to be increasing order of difficulty. For example, setting up a discussion forum involves only enabling the discussions feature on this organization and pointing the community to where these discussion can happen (right here!), whereas number 4 will require some discussion building on the Dec 4th meeting as to what it actually entails and how to benchmark the code when there might be no 'truth' per se (as @irenavankova mention in the call). This could all be discussed in a different thread in this discussion forum, and/or in issues in an appropriate repo (to be created).
• Keith Nicholls mentioned not wanting to restrict us to one language in the call on Dec 4th. Maybe we can achieve this by having the recipes be unrestricted in that regard, but make things easier for ourselves when trying to come up with a benchmarked code by sticking to one language. Alternatively, maybe we do the core things (e.g. load chirps and the fft) twice, once in python and once in matlab. Again, let's discuss this in the appropriate repo (to be created).

For now, it would be good to hear people's thoughts on using the organization for everything, with 4 core repos being our starting point.

[mmannello-12]

Thanks so much for summarizing! As someone without much ApRES processing experience, this all seems super useful to me. I think I heard a slack channel was also on the table which seems like a nice companion option to #1.

[jkingslake]

Thanks mmannello-12!

We can definitely good this! My only reservation about slack in this context is that it is not a very open way of sharing information.

If you have a question answered on slack it is not available to people who don't happen to know about the slack workspace (or who know about it but don't want to join). It also deletes messages older than 90 days old unless we pay, which means all information is eventually lost, either after 90 days or after whoever is paying doesn't have funding anymore. Information stored on github is at least in theory going to be around for the foreseeable future.

Happy to hear the counter arguments about slack's advantages, though! :-)

[irenavankova]

I have seen a few attempts with similar broader community efforts before that started a slack channel, but they all seemed to have failed. One reason is the space not being open as Jonny mentioned. I think another one is the social media aspect of slack, which can put off some people. It also comes with the extra burden of having to have yet another app and check it regularly. I personally would not be very responsive, if at all, on a potential ApRES slack channel.

[benhills]

Yes, the searchability of discussions on github is useful in my opinion.

[rdrews-dev]

Another practical option to explore is to compile a list of where ApRES instruments are available and could (potentially) be borrowed. In Tübingen we have 2 Mimo ApRES with eight Antennas and we are usually open to lending them out.

[jkingslake]

Good idea. I'll start another thread to notes ideas about how this could work: #5

[benhills]

Thanks @jkingslake!! This is great.
I agree with the four main ideas you highlighted. In particular, I think this open discussion is a very good idea.

My main question/comment is something that Andrew Hoffman brought up during the call; that is, there are other instruments that have a lot of the same features/capabilities as the BAS ApRES. Do we have strong thoughts on where the scope of this group lies within the broader radioglaciology community?
For instance, I could see the documentation being written as a new version of the Radioglaciology textbook (which is long overdue) to explain pulse compression, interferometry, polarimetry, and even SAR focusing (which some folks are now using in their ApRES profiling experiments). All of that will naturally overlap with what folks are developing on Open Polar Radar. It is my opinion this will be most effective if both communities are working together to make the projects synergistic rather than developing two versions of everything dependent on the specific instrument.

For the code recipes I think it is easiest to expect that people self publish (can use zenodo) and then we link doi's from this repository in a readme or something. Could link relevant papers that way too.

For the benchmarked code, I think it is fine to continue parallel development. If Keith feels most comfortable in Matlab it makes sense to post a version of his scripts somewhere public (assuming he is comfortable with that and able to do it, or someone could post it for him). Then, the Python version would be xApRES since you are going to keep developing that. Posting some benchmark tests that both can rune would be useful too (as we discussed).

[KeithNicholls]

Definitely very happy to supply the code I use. This is essentially the same as used by Irena, for example, and I'm sure several others too.

[jkingslake]

Great, thanks @KeithNicholls! Do you think this version of the code should act as the 'official' most up-to-date version that we could (1) point people towards using if they are new to ApRES, and (2) build upon to add more features as a group?

[KeithNicholls]

Well, I think I would need to tidy it up a bit. This code is designed specifically for time series - ApRES (not pRES). The code takes the raw data, and generates the range domain profiles using the Brennan methods. There is then another function that creates the stack of time series (using cross-correlation) that provides the basis for purpose-built scripts tailored to the particular site. It also offers the option to carry out a tidal analysis of each of the time series using u-tide. There are a load of options to do with leap-frogging, interval depth etc.

For the revisit (is pRES) sites, I tend to use Craig Stewart's code.

[jkingslake]

terminology question: So, do you use ApRES when talking about unattended measurements and pRES when talking about attended measurements? For me pRES has always just meant the old system (the network analyzer).

[KeithNicholls]

I've found it convenient to refer to ApRES, pRES and "old pRES"! Not very elegant, I know, and am happy to be corrected by the community...

[jkingslake]

@benhills thanks for this comment! Sorry for the delay in replying.

To answer your question about the scope: I don't have a strong view of where the scope of this group lies in the broader community. I had initially thought that the uses of ApRES have been a little separate from other radars systems and maybe therefore ApRES can form the basis of a slightly separate community from the whole radioglaciology community. This seems a little more manageable at the moment - just given how big the radioglaciology community is, it seems like a challenge to produce resources that would be broadly useful, at least in terms of software. Having said that, as you know I am excited about the jupyterbook model of a collaborative online textbook, which could be a way forward on the radioglaciology textbook you mentioned.

I guess what I would advocate for is that we try to make some progress here in the ApRES community and then try to connect this to broader groups later. We obviously have good contacts/cross-over with the Open Polar Radar project, so I hope that will be doable. What do you think?

To discuss the point about the recipes you make, I will move the conversation to the other thread on ideas about recipes.

Benchmarked code: I agree we can have matlab and python versions. For others' reference, xapres is a python package I have been working on which does some of the core processing steps using numpy and xarray. It is somewhat benchmarked in the sense that it produces the same profiles as the version of fmcw plot I had access to.

I would be very happy for this package to be something that we worked on together! But we should discuss this in a separate thread to keep things tractable.

[KeithNicholls]

I think the main difference in the case of ApRES is that it's designed specifically to explore the time dimension, which means that most of the datasets it generates will have methods of analysis distinct from those used for a more conventional (mobile) radar. (I suppose one could configure a space dimension into a time dimension, and then use similar tools, but that sounds like an unwelcome complication...) On the whole, like Jonny, I think a separate code base might be the place to start.

[jkingslake]

Thanks everyone for your input on this!

It sounds from this discussion and our call in December that there is some agreement that there is value in this community adopting/developing and hosting code for loading and processing ApRES data that is broadly agreed upon by this community as producing the 'correct' outputs.

ApRES users can then be pointed to this code when looking for a place to start in analysis of data. I think this is a key part of making the community more open.

There is support for two sets of code, one written in MATLAB, one in python.

Benefits of the matlab code:

• there is already a large group of users who process ApREs data in MATLAB
• most people who use ApRES will have access to MATLAB in some form
• there is already code that is generrally thought of as the ApRES processing code that users are pointed to when they first get into this (albeit there are a few versions)

Benefits of a python-based code:

• python is being used by an increasing number of geoscientists
• python is free and open source
• such a code can integrate nicely with other open source projects that allow fun things like publishing reproducible workflows (binder, jupyterbook, etc.) and processing data in the cloud (cyrocloud, pangeo etc.)

I would love to hear more thoughts here. And I will in parallel start two new discussions to gather ideas for establishing each set of code.

[jkingslake]

here's the discussion about the matlab code: #6