Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial ADIOS2 implementation #338

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

Initial ADIOS2 implementation #338

wants to merge 3 commits into from

Conversation

mdzik
Copy link
Member

@mdzik mdzik commented Nov 15, 2020

Reference:

ADIOS2

dataReader

Digital rock data

Idea (TCLB d2q9 karman example:

		<Wall mask="ALL" dx="33" dy="10" nx="734" ny="734">
			<ADIOS2 Stream="/home/mdzik/projekty/TCLB_tools/ADIOS2/globalArray" Variable="GlobalArray" 
		</Wall>

TCLB output

[  ]    ---- : loading geometry ...
Region: 734x344x1 + 33,400,0
Region: 734x390x1 + 33,10,0
Reader 0 (0x55728b5f5420): Looking for writer contact in file /home/mdzik/projekty/TCLB_tools/ADIOS2/globalArray.sst, with timeout 60 secs
Reader 0 (0x55728b5f5420): Waiting for writer response message in SstReadOpen("/home/mdzik/projekty/TCLB_tools/ADIOS2/globalArray")
Reader 0 (0x55728b5f5420): finished wait writer response message in read_open
Reader 1 (0x561c4af5afa0): Sending Reader Activate messages to writer
Reader 1 (0x561c4af5afa0): Finish opening Stream "/home/mdzik/projekty/TCLB_tools/ADIOS2/globalArray", starting with Step number 0
Reader 0 (0x55728b5f5420): Opening Reader Stream.
Writer stream params are:
Param -   RegistrationMethod=File
Param -   RendezvousReaderCount=1
Param -   QueueLimit=0 (unlimited)
Param -   QueueFullPolicy=Block
Param -   DataTransport=evpath
Param -   ControlTransport=sockets
Param -   NetworkInterface=(default)
Param -   ControlInterface=(default to NetworkInterface if applicable)
Param -   DataInterface=(default to NetworkInterface if applicable)
Param -   CompressionMethod=None
Param -   CPCommPattern=Min
Param -   MarshalMethod=BP
Param -   FirstTimestepPrecious=False
Param -   IsRowMajor=1  (not user settable) 
Param -   OpenTimeoutSecs=60 (seconds)
Param -   SpeculativePreloadMode=Auto
Param -   SpecAutoNodeThreshold=1
Param -   ControlModule=select
Reader 0 (0x55728b5f5420): Reader stream params are:
Param -   RegistrationMethod=File
Param -   DataTransport=evpath
Param -   ControlTransport=sockets
Param -   NetworkInterface=(default)
Param -   ControlInterface=(default to NetworkInterface if applicable)
Param -   DataInterface=(default to NetworkInterface if applicable)
Param -   AlwaysProvideLatestTimestep=False
Param -   OpenTimeoutSecs=60 (seconds)
Param -   SpeculativePreloadMode=Auto
Param -   SpecAutoNodeThreshold=1
Param -   ControlModule=select
Reader 0 (0x55728b5f5420): Writer is doing BP-based marshalling
Reader 0 (0x55728b5f5420): Writer is using Minimum Connection Communication pattern (min)
Reader 0 (0x55728b5f5420): Sending Reader Activate messages to writer
10 R 734
400 C 344
I need Y: 390 COUNT 344
10 R 734
10 C 390
I need Y: 0 COUNT 390
Reader 0 (0x55728b5f5420): Finish opening Stream "/home/mdzik/projekty/TCLB_tools/ADIOS2/globalArray", starting with Step number 0
Reader 0 (0x55728b5f5420): MaxQueued Timestep returning -1
Reader 0 (0x55728b5f5420): Wait for metadata with timeout 60 secs starting at time 1605473894.823900 
Reader 0 (0x55728b5f5420): timercmp, now is 1605473894.823924    end is 1605473954.823900 
Reader 1 (0x561c4af5afa0): SstAdvanceStep returning Success on timestep 0
Reader 0 (0x55728b5f5420): Received a Timestep metadata message for timestep 0, signaling condition
Reader 0 (0x55728b5f5420): Returning from wait with timeout, NO TIMEOUT
Reader 0 (0x55728b5f5420): MaxQueued Timestep returning 0
Reader 0 (0x55728b5f5420): Wait for next metadata after last timestep -1
Reader 0 (0x55728b5f5420): Examining metadata for Timestep 0
Reader 0 (0x55728b5f5420): Returning metadata for Timestep 0
Reader 0 (0x55728b5f5420): Setting TSmsg to Rootentry value
Reader 0 (0x55728b5f5420): SstAdvanceStep returning Success on timestep 0
Reader 1 (0x561c4af5afa0): Registering reader close handler for peer 0 CONNECTION 0x561c4af689b0
Reader 0 (0x55728b5f5420): Received a Timestep metadata message for timestep 1, signaling condition
Reader 1 (0x561c4af5afa0): Sending ReleaseTimestep message for timestep 0, one to each writer
Reader 0 (0x55728b5f5420): Sending ReleaseTimestep message for timestep 0, one to each writer
I've read: MARKED: 12  ALL: 252496
I've read: MARKED: 847  ALL: 286260
Reader 0 (0x55728b5f5420): 
Stream "/home/mdzik/projekty/TCLB_tools/ADIOS2/globalArray" (0x55728b5f5420) summary info:
Reader 0 (0x55728b5f5420): 	Duration (secs) = 0.087084
Reader 0 (0x55728b5f5420): 	Timestep Metadata Received = 2
Reader 0 (0x55728b5f5420): 	Timesteps Consumed = 1
Reader 0 (0x55728b5f5420): 	MetadataBytesReceived = 662 (662 bytes)
Reader 0 (0x55728b5f5420): 	DataBytesReceived = 1076778 (1.0 MB)
Reader 0 (0x55728b5f5420): 	PreloadBytesReceived = 0 (0 bytes)
Reader 0 (0x55728b5f5420): 	PreloadTimestepsReceived = 0
Reader 0 (0x55728b5f5420): 	AverageReadRankFanIn = 1.0
Reader 0 (0x55728b5f5420): 
Reader 1 (0x561c4af5afa0): Destroying stream 0x561c4af5afa0, name /home/mdzik/projekty/TCLB_tools/ADIOS2/globalArray
Reader 0 (0x55728b5f5420): Destroying stream 0x55728b5f5420, name /home/mdzik/projekty/TCLB_tools/ADIOS2/globalArray
Reader 1 (0x561c4af5afa0): Reference count now zero, Destroying process SST info cache
Reader 0 (0x55728b5f5420): Reference count now zero, Destroying process SST info cache
Reader 0 (0x55728b5f5420): Freeing LastCallList
Reader 0 (0x7ffe63cb9eb0): SstStreamDestroy successful, returning
Reader 1 (0x561c4af5afa0): Freeing LastCallList
Reader 1 (0x7ffd9fa60c10): SstStreamDestroy successful, returning
[  ]    ---- : Setting number of zones to 1
[  ]    ---- : Setting VelocityX in zone  (-1) to 0.01 (0.010000)
[ 1] WARNING ! Unknown setting Smag
[ 1] WARNING ! Unknown setting PressDiffInObj
[ 1] WARNING ! Unknown setting EOSScale
[ 1] WARNING ! Unknown setting Tension
[ 1] WARNING ! Unknown setting Coriolis
[ 1] WARNING ! Unknown setting SolidAlfa
[ 1] WARNING ! Unknown setting FluidAlfa
[ 1] WARNING ! Unknown setting InitTemperature
[ 1] WARNING ! Unknown setting InletTemperature
[  ]    ---- : Setting Viscosity to 0.02 (0.020000)
[  ]    ---- : [0] Settings [viscosity] to 0.020000
[  ]    ---- : [0] Settings [one over relaxation time] to 1.785714
[  ]    ---- : [0] Settings [MRT Sx] to -0.785714
[ 0] WARNING ! Unknown setting Smag
[ 0] WARNING ! Unknown setting PressDiffInObj
[ 0] WARNING ! Unknown setting EOSScale
[ 0] WARNING ! Unknown setting Tension
[ 0] WARNING ! Unknown setting Coriolis
[ 0] WARNING ! Unknown setting SolidAlfa
[ 0] WARNING ! Unknown setting FluidAlfa
[ 0] WARNING ! Unknown setting InitTemperature
[ 0] WARNING ! Unknown setting InletTemperature
[  ]    ---- : Initializing Lattice ...
[  ]    ---- : Callback VTK with no Iterations attribute=]  0s
[  ]    ---- :        0 it writing vtk
[  ]    ---- : Total duration: 0.284218 s = 0.004737 min = 0.000079 h

Reader output

[mdzik@icm-pc ADIOS2]$ ./ReadTomographyRAW /home/mdzik/Pobrane/05_CC_K2_dry_vx_6x6x6_dim_734x734x425.raw /home/mdzik/projekty/TCLB/example/adios/karman.adios.xml 734 734 5 32 1 0.500003
THIS IS EXPERIMENTAL/UNTESTED, DONT USE MPIRUN on IT!!
Datatype: 32 bytes signed
Will read RAW file: /home/mdzik/Pobrane/05_CC_K2_dry_vx_6x6x6_dim_734x734x425.raw
Will read XML file: /home/mdzik/projekty/TCLB/example/adios/karman.adios.xml
Expected RAW shape: (734, 734, 5)
Byte size: 4
Expected filesize: (ls -la) 10775120
all characters read successfully.Set threshold: 0.500003-> VALUE < 12800
Reading one Z slice
All: 538756 V < THRESHOLD (return 1) 537890
MIN: 630 MAX:13039
Writer 0 (0x55f75fc87a80): Opening Stream "globalArray"
Writer 0 (0x55f75fc87a80): Writer stream params are:
Param -   RegistrationMethod=File
Param -   RendezvousReaderCount=1
Param -   QueueLimit=0 (unlimited)
Param -   QueueFullPolicy=Block
Param -   DataTransport=evpath
Param -   ControlTransport=sockets
Param -   NetworkInterface=(default)
Param -   ControlInterface=(default to NetworkInterface if applicable)
Param -   DataInterface=(default to NetworkInterface if applicable)
Param -   CompressionMethod=None
Param -   CPCommPattern=Min
Param -   MarshalMethod=BP
Param -   FirstTimestepPrecious=False
Param -   IsRowMajor=1  (not user settable) 
Param -   OpenTimeoutSecs=60 (seconds)
Param -   SpeculativePreloadMode=Auto
Param -   SpecAutoNodeThreshold=1
Param -   ControlModule=select
Writer 0 (0x55f75fc87a80): Stream "globalArray" waiting for 1 readers
Writer 0 (0x55f75fc87a80): Beginning writer-side reader open protocol
Writer 0 (0x55f75fc87a80): My oldest timestep was 0, global oldest timestep was 0
Writer 0 (0x55f75fc87a80): Finish writer-side reader open protocol for reader 0x55f75fc86620, reader ready response pending
Writer 0 (0x55f75fc87a80): (PID bf2f, TID 7fd1e43dce00) Waiting for Reader ready on WSR 0x55f75fc86620.
Writer 0 (0x55f75fc87a80): Reader Activate message received for Stream 0x55f75fc86620.  Setting state to Established.
Writer 0 (0x55f75fc87a80): Parent stream reader count is now 1.
Writer 0 (0x55f75fc87a80): Reader ready on WSR 0x55f75fc86620, Stream established, Starting 0 LastProvided 0.
Writer 0 (0x55f75fc87a80): Finish opening Stream "globalArray"
Reading Z-slice no:0
Writer 0 (0x55f75fc87a80): Reader 0 status Established has last released 4294967295, last sent 0
Writer 0 (0x55f75fc87a80): QueueMaintenance, smallest last released = -1, count = 1
Writer 0 (0x55f75fc87a80): Removing dead entries
Writer 0 (0x55f75fc87a80): QueueMaintenance complete
Writer 0 (0x55f75fc87a80): Sending TimestepMetadata for timestep 0 (ref count 1), one to each reader
Writer 0 (0x55f75fc87a80): Sent timestep 0 to reader cohort 0
Writer 0 (0x55f75fc87a80): ADDING timestep 0 to sent list for reader cohort 0, READER 0x55f75fc86620, reference count is now 2
Writer 0 (0x55f75fc87a80): Sending a message to reader 0 (0x55728b5f5420)
Writer 0 (0x55f75fc87a80): SubRef : Writer-side Timestep 0 now has reference count 1, expired 0, precious 0
Writer 0 (0x55f75fc87a80): Reader 0 status Established has last released 4294967295, last sent 0
Writer 0 (0x55f75fc87a80): QueueMaintenance, smallest last released = -1, count = 1
Writer 0 (0x55f75fc87a80): Removing dead entries
Writer 0 (0x55f75fc87a80): QueueMaintenance complete
Reading Z-slice no:1
Writer 0 (0x55f75fc87a80): Reader 0 status Established has last released 4294967295, last sent 0
Writer 0 (0x55f75fc87a80): QueueMaintenance, smallest last released = -1, count = 2
Writer 0 (0x55f75fc87a80): Removing dead entries
Writer 0 (0x55f75fc87a80): QueueMaintenance complete
Writer 0 (0x55f75fc87a80): Sending TimestepMetadata for timestep 1 (ref count 1), one to each reader
Writer 0 (0x55f75fc87a80): Sent timestep 1 to reader cohort 0
Writer 0 (0x55f75fc87a80): ADDING timestep 1 to sent list for reader cohort 0, READER 0x55f75fc86620, reference count is now 2
Writer 0 (0x55f75fc87a80): Sending a message to reader 0 (0x55728b5f5420)
Writer 0 (0x55f75fc87a80): SubRef : Writer-side Timestep 1 now has reference count 1, expired 0, precious 0
Writer 0 (0x55f75fc87a80): Reader 0 status Established has last released 4294967295, last sent 1
Writer 0 (0x55f75fc87a80): QueueMaintenance, smallest last released = -1, count = 2
Writer 0 (0x55f75fc87a80): Removing dead entries
Writer 0 (0x55f75fc87a80): QueueMaintenance complete
Reading Z-slice no:2
Writer 0 (0x55f75fc87a80): Received a release timestep message for timestep 0 from reader cohort 0
Writer 0 (0x55f75fc87a80): Got the lock in release timestep
Writer 0 (0x55f75fc87a80): Doing dereference sent
Writer 0 (0x55f75fc87a80): Reader sent timestep list 0x55f75fcabd90, trying to release 0
Writer 0 (0x55f75fc87a80): Reader considering sent timestep 0,trying to release 0
Writer 0 (0x55f75fc87a80): SubRef : Writer-side Timestep 0 now has reference count 0, expired 0, precious 0
Writer 0 (0x55f75fc87a80): Doing QueueMaint
Writer 0 (0x55f75fc87a80): Reader 0 status Established has last released 0, last sent 1
Writer 0 (0x55f75fc87a80): QueueMaintenance, smallest last released = 0, count = 2
Writer 0 (0x55f75fc87a80): Writer tagging timestep 0 as expired
Writer 0 (0x55f75fc87a80): Removing dead entries
Writer 0 (0x55f75fc87a80): Remove queue Entries removing Timestep 0 (exp 1, Prec 0, Ref 0), Count now 1
Writer 0 (0x55f75fc87a80): QueueMaintenance complete
Writer 0 (0x55f75fc87a80): Releasing the lock in release timestep
Writer 0 (0x55f75fc87a80): Reader Close message received for stream 0x55f75fc86620.  Setting state to PeerClosed and releasing timesteps.
Writer 0 (0x55f75fc87a80): In PeerFailCloseWSReader, releasing sent timesteps
Writer 0 (0x55f75fc87a80): Dereferencing all timesteps sent to reader 0x55f75fc86620
Writer 0 (0x55f75fc87a80): Checking on timestep 1
Writer 0 (0x55f75fc87a80): Reader sent timestep list 0x55f75fcae280, trying to release 1
Writer 0 (0x55f75fc87a80): Reader considering sent timestep 1,trying to release 1
Writer 0 (0x55f75fc87a80): SubRef : Writer-side Timestep 1 now has reference count 0, expired 0, precious 0
Writer 0 (0x55f75fc87a80): DONE DEREFERENCING
Writer 0 (0x55f75fc87a80): Moving Reader stream 0x55f75fc86620 to status PeerClosed
Writer 0 (0x55f75fc87a80): Reader 0 status PeerClosed has last released 0, last sent 1
Writer 0 (0x55f75fc87a80): QueueMaintenance, smallest last released = LONG_MAX, count = 1
Writer 0 (0x55f75fc87a80): Writer tagging timestep 1 as expired
Writer 0 (0x55f75fc87a80): Removing dead entries
Writer 0 (0x55f75fc87a80): Remove queue Entries removing Timestep 1 (exp 1, Prec 0, Ref 0), Count now 0
Writer 0 (0x55f75fc87a80): QueueMaintenance complete
Writer 0 (0x55f75fc87a80): Reader 0 status PeerClosed has last released 0, last sent 1
Writer 0 (0x55f75fc87a80): QueueMaintenance, smallest last released = LONG_MAX, count = 1
Writer 0 (0x55f75fc87a80): Writer tagging timestep 2 as expired
Writer 0 (0x55f75fc87a80): Removing dead entries
Writer 0 (0x55f75fc87a80): QueueMaintenance complete
Writer 0 (0x55f75fc87a80): Sending TimestepMetadata for timestep 2 (ref count 1), one to each reader
Writer 0 (0x55f75fc87a80): SubRef : Writer-side Timestep 2 now has reference count 0, expired 1, precious 0
Writer 0 (0x55f75fc87a80): Reader 0 status PeerClosed has last released 0, last sent 1
Writer 0 (0x55f75fc87a80): QueueMaintenance, smallest last released = LONG_MAX, count = 1
Writer 0 (0x55f75fc87a80): Removing dead entries
Writer 0 (0x55f75fc87a80): Remove queue Entries removing Timestep 2 (exp 1, Prec 0, Ref 0), Count now 0
Writer 0 (0x55f75fc87a80): QueueMaintenance complete
Reading Z-slice no:3
Writer 0 (0x55f75fc87a80): Reader 0 status PeerClosed has last released 0, last sent 1
Writer 0 (0x55f75fc87a80): QueueMaintenance, smallest last released = LONG_MAX, count = 1
Writer 0 (0x55f75fc87a80): Writer tagging timestep 3 as expired
Writer 0 (0x55f75fc87a80): Removing dead entries
Writer 0 (0x55f75fc87a80): QueueMaintenance complete
Writer 0 (0x55f75fc87a80): Sending TimestepMetadata for timestep 3 (ref count 1), one to each reader
Writer 0 (0x55f75fc87a80): SubRef : Writer-side Timestep 3 now has reference count 0, expired 1, precious 0
Writer 0 (0x55f75fc87a80): Reader 0 status PeerClosed has last released 0, last sent 1
Writer 0 (0x55f75fc87a80): QueueMaintenance, smallest last released = LONG_MAX, count = 1
Writer 0 (0x55f75fc87a80): Removing dead entries
Writer 0 (0x55f75fc87a80): Remove queue Entries removing Timestep 3 (exp 1, Prec 0, Ref 0), Count now 0
Writer 0 (0x55f75fc87a80): QueueMaintenance complete
Reading Z-slice no:4
Writer 0 (0x55f75fc87a80): Writer-side Rank received a connection-close event after close, not unexpected
Writer 0 (0x55f75fc87a80): Reader 0 status PeerClosed has last released 0, last sent 1
Writer 0 (0x55f75fc87a80): QueueMaintenance, smallest last released = LONG_MAX, count = 0
Writer 0 (0x55f75fc87a80): Removing dead entries
Writer 0 (0x55f75fc87a80): QueueMaintenance complete
Writer 0 (0x55f75fc87a80): Writer-side Rank received a connection-close event after close, not unexpected
Writer 0 (0x55f75fc87a80): Reader 0 status PeerClosed has last released 0, last sent 1
Writer 0 (0x55f75fc87a80): QueueMaintenance, smallest last released = LONG_MAX, count = 0
Writer 0 (0x55f75fc87a80): Removing dead entries
Writer 0 (0x55f75fc87a80): QueueMaintenance complete
Writer 0 (0x55f75fc87a80): Reader 0 status PeerClosed has last released 0, last sent 1
Writer 0 (0x55f75fc87a80): QueueMaintenance, smallest last released = LONG_MAX, count = 1
Writer 0 (0x55f75fc87a80): Writer tagging timestep 4 as expired
Writer 0 (0x55f75fc87a80): Removing dead entries
Writer 0 (0x55f75fc87a80): QueueMaintenance complete
Writer 0 (0x55f75fc87a80): Sending TimestepMetadata for timestep 4 (ref count 1), one to each reader
Writer 0 (0x55f75fc87a80): SubRef : Writer-side Timestep 4 now has reference count 0, expired 1, precious 0
Writer 0 (0x55f75fc87a80): Reader 0 status PeerClosed has last released 0, last sent 1
Writer 0 (0x55f75fc87a80): QueueMaintenance, smallest last released = LONG_MAX, count = 1
Writer 0 (0x55f75fc87a80): Removing dead entries
Writer 0 (0x55f75fc87a80): Remove queue Entries removing Timestep 4 (exp 1, Prec 0, Ref 0), Count now 0
Writer 0 (0x55f75fc87a80): QueueMaintenance complete
Writer 0 (0x55f75fc87a80): SstWriterClose, Sending Close at Timestep 4, one to each reader
Writer 0 (0x55f75fc87a80): Skipping reader cohort 0
Writer 0 (0x55f75fc87a80): Reader 0 status PeerClosed has last released 0, last sent 1
Writer 0 (0x55f75fc87a80): QueueMaintenance, smallest last released = LONG_MAX, count = 0
Writer 0 (0x55f75fc87a80): Removing dead entries
Writer 0 (0x55f75fc87a80): QueueMaintenance complete
Writer 0 (0x55f75fc87a80): 
Stream "globalArray" (0x55f75fc87a80) summary info:
Writer 0 (0x55f75fc87a80):      Duration (secs) = 0.308753
Writer 0 (0x55f75fc87a80):      Timesteps Created = 5
Writer 0 (0x55f75fc87a80):      Timesteps Delivered = 2
Writer 0 (0x55f75fc87a80): 
Writer 0 (0x55f75fc87a80): All timesteps are released in WriterClose
Writer 0 (0x55f75fc87a80): Destroying stream 0x55f75fc87a80, name globalArray
Writer 0 (0x55f75fc87a80): Reference count now zero, Destroying process SST info cache
Writer 0 (0x55f75fc87a80): Freeing LastCallList
Writer 0 (0x7ffe7ef58240): SstStreamDestroy successful, returning
[mdzik@icm-pc ADIOS2]$ 

@mdzik mdzik self-assigned this Nov 15, 2020
@@ -1038,6 +1044,137 @@ int Geometry::Draw(pugi::xml_node & node)
Dot(x, y, z);
}
fclose(f);
} else if (strcmp(n.name(), "ADIOS2") == 0) {
lbRegion crop = getRegion(n.parent());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also has to be in #ifdef

@@ -379,6 +389,24 @@ if test "x${with_hdf5}" != "xno"; then
fi
fi

if test "x${with_adios}" != "xno"; then
adios2_config="${ADIOS_DIR}/bin/adios2-config"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see where ADIOS2_USE_MPI is set to true.

It has to be:

  1. Add ADIOS2_USE_MPI=1 in the if in configure.ac
  2. Add AC_SUBST(ADIOS2_USE_MPI) at the bottom of configure.ac
  3. Add #undef ADIOS2_USE_MPI to in src/config.h.in

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ADIOS2_USE_MPI is an ADIOS config flag (the adios2-config returns it in --flags string) - I'm abusing it a little now

This is to be changed I think:

  • Option 1: If TCLB is MPI then ADIOS must be MPI and vice versa, nice but it doesn't make sense even for testing imho?
  • Option 2: Enable adios in MPI version of TCLB and require ADIOS2_USE_MPI.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need our own flag for enabling ADIOS then. Remember that if an flag is in the compilation (like -DFLAG) and not in the source (like #define FLAG 1) then make cannot see the change and will not recompile properly. Please make a flag for the ADIOS support as described above (maybe WITH_ADIOS)

TCLB is always MPI, we don't have an option for that.

@mdzik
Copy link
Member Author

mdzik commented Nov 17, 2020

@llaniewski TCLB is only X-decomposed? or X-Z?

@llaniewski
Copy link
Member

@mdzik TCLB is MPI decomposed in Y and Z. It is not divided in X (it's aligned with warps of 32 threads in X).

@llaniewski
Copy link
Member

@mdzik Please do the #ifdefs, as currently the tests don't go through and this cannot be compiled without ADIOS.

@llaniewski llaniewski added the enhancement New features/updates label Dec 12, 2020
@mdzik
Copy link
Member Author

mdzik commented Dec 14, 2020 via email

@llaniewski
Copy link
Member

@mdzik thanks for the update

@llaniewski
Copy link
Member

@mdzik Any update on this? Are you using ADIOS in your setups? (or TCLB in general now?)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New features/updates
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants