Hero-scale data analysis with VisIt on Andes

[BenWibking]

Hi,

I am trying to analyze a 100k domain, >100 billion cell Cartesian AMR mesh with VisIt 3.5.0 on Andes at OLCF.

This causes frequent OOM kills, but I am having a hard time diagnosing exactly where things are going wrong. In some cases, adding more nodes to the analysis job does not appear to help or change when the OOM happens.

I am using the custom AMReX plugin I wrote here: https://github.com/BenWibking/amrex-visit-reader. Is there a potential load-balance issue that I need to work around in the plugin? Is there some domain metadata that's replicated over all MPI ranks? I tried to avoid doing the latter in the plugin. I have also tried to use the Box operator to try to reduce the total number of domains it loads at once.

If there's anything that might help when running at this scale, please let me know.

[markcmiller86]

Can you confirm that it is indeed the engine_par process that is OOM'ing and not the mdserver? Reason I ask is that portions of a database plugin code execute on the mdserver and if one is not careful, its easy to do unnecessary work and allocations there.

How many MPI ranks per node are you running with?

You don't have scalable rendering set to never do you?

Your note mentions 100k domains. So, I just wanna make sure that when PopulateDatabaseMetaData(...) executes in your plugin, the mesh metadata for the mesh your are talking about here gets a numBlocks value that is that 100k number you are talking about?

Does it OOM only soon after changing timesteps? If so, this could suggest leaks are contributing.

I noticed there is no call to FreeUpResources from avtamrexFileFormat::~avtamrexFileFormat(). But, because this is an MTMD database, that is probably not a big deal because it should instantiate only one plugin instance for all domains and time on each MPI rank and hold onto that instance until the database is closed. If this were an STMD or STSD plugin, I would worry because VisIt currently instantiates all plugin constructors over all times (and domains) before it uses any one of them with the expectation that in the plugin constructor there is very little work done and very little memory allocated.

It looks like your plugin source code builds a hierarchy object with a lot of the details of the AMR mesh structure which appears to include a number of O(#domains)-sized members and I don't see anything that suggests its construction is restricted to just one MPI rank. That is getting built on every MPI rank isn't it. And then you also need to build the StruturedDomainNesting version of (some of) that for VisIt too.

You mention using Box operator to reduce domains loaded. Is your plugin plummed for that? I think that involves some advanced plugin (and database) design. For example, the domain spatial extents need to be provided to VisIt when it makes the auxiliary data request for key AUXILIARY_DATA_SPATIAL_EXTENTS and I don't see any hits for that string in the plugin repo. It returns a O(#domains)-sized array of 6 floats for x/y/z min/max values for each domain.

[markcmiller86]

Didn't think to ask but is the plugin built with -DAMREX_DISABLE_STRUCTURED_BOUNDARY_CACHE?

[markcmiller86]

It looks like Andes has 256G per node. Lets assume each MPI rank will need 32 GB of memory to run (hierarchy metadata + some number of patch/domain bulk data instances. Then, you probably should run with <=8 MPI ranks per node or less. With 100K domains, if you ran with 100 nodes, thats 125 domains per MPI rank. Thats a little too much. I'd want to cut that to maybe a 3rd of that. So, maybe half the machine at 8 mpi ranks per node if that is something possible for you to do. Have you already tried something on that order?

[markcmiller86]

BTW...I neglected to mention but something I typicaly do is use the Domain subset selection to reduce the number of domains to load down to just one, or a handful and see if it makes it through to actually plotting them. You should be able to do this on just a few MPI ranks. So, try maybe 2, 4, 8, 16 and/or 32 MPI ranks one just one node. Open the database, add a mesh plot but before hitting Draw button, go to Subset controlls and turn off all domains. Then selectively turn on only a handful of domains, like maybe 10. See what happens. If it plots fine on 2, 4 and 8 but OOMs on 16 or 32, I'd be inclined to think its the 16 or 32 copies of the hiearchy data structure that is potentially getting in the way.

You can also run valgrind with VisIt on a modest number of MPI ranks

[BenWibking]

Thanks, these suggestions have been helpful. I've optimized the plugin along the lines you suggest and I'll test it as soon as I can get a large enough interactive job on Andes.