24/09/10a
24/09/10 の続き。
GADGET4-Osakaのコードの準備
$ git clone git@github.com:YuriOku/Osaka-gadget4.git
Cloning into 'Osaka-gadget4'...
The authenticity of host 'github.com (20.27.177.113)' can't be established.
ED25519 key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
ERROR: Repository not found.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
アクセス権が無かった…奥さんに連絡を取る。
$ git clone git@github.com:YuriOku/Osaka-gadget4.git
Cloning into 'Osaka-gadget4'...
remote: Enumerating objects: 16948, done.
remote: Counting objects: 100% (2767/2767), done.
remote: Compressing objects: 100% (420/420), done.
remote: Total 16948 (delta 2494), reused 2414 (delta 2347), pack-reused 14181 (from 1)
Receiving objects: 100% (16948/16948), 57.02 MiB | 9.77 MiB/s, done.
Resolving deltas: 100% (13037/13037), done.
buildsystemの準備
$ cd buildsystem/
$ ls
check.py Makefile.gen.libs
config.py Makefile.lib
git_version.sh Makefile.path.bwforcluster
Makefile.comp.cobra Makefile.path.cobra
Makefile.comp.cobraopenmpi Makefile.path.default
Makefile.comp.flatiron-rusty Makefile.path.flatiron-rusty
Makefile.comp.freya Makefile.path.freya
Makefile.comp.freyaopenmpi Makefile.path.gcc-debug
Makefile.comp.gcc Makefile.path.haswell
Makefile.comp.gcc-debug Makefile.path.hydra
Makefile.comp.gcc-paranoia Makefile.path.libs
Makefile.comp.myintel Makefile.path.macports
Makefile.comp.mymac Makefile.path.magny
Makefile.comp.octopus Makefile.path.mpa_desktop
Makefile.comp.ravenopenmpi Makefile.path.myintel
Makefile.comp.squid Makefile.path.mymac
Makefile.comp.squid-gnu Makefile.path.octopus
Makefile.comp.supermuc-ng Makefile.path.squid
Makefile.comp.supermuc-ng-gcc Makefile.path.squid-gnu
Makefile.comp.supermuc-ng-openmpi Makefile.path.supermuc-ng
Makefile.comp.virgo Makefile.path.supermuc-ng-gcc
Makefile.comp.xc50-gnu Makefile.path.virgo
Makefile.config Makefile.path.xc50-gnu
Makefile.path を自分用に編集する。編集する上でどういったことが書かれているかを確認する。
$ cat Makefile.path.mymac
GSL_INCL = -I/opt/local/include
GSL_LIBS = -L/opt/local/lib -L/opt/local/lib/libgcc
FFTW_INCL = -I/opt/local/include
FFTW_LIBS = -L/opt/local/lib
HDF5_INCL = -I/opt/local/include
HDF5_LIBS = -L/opt/local/lib
HWLOC_INCL = -I/opt/local/include
HWLOC_LIBS = -L/opt/local/lib
MAPS_INCL = -I/opt/local/include/healpix_cxx
MAPS_LIBS = -L/opt/local/lib
GRACKLE_INCL = -I/Users/yuri/local/include
GRACKLE_LIBS = -L/Users/yuri/local/lib -lgrackle
GRACKLE_OPT = -DLINUX -fPIC -DLARGE_INTS -DCONFIG_BFLOAT_8
$ cat Makefile.path.virgo
LIB_DIR = $(HOME)/local
GSL_INCL = -I$(LIB_DIR)/include
GSL_LIBS = -L$(LIB_DIR)/lib
FFTW_INCL = -I$(LIB_DIR)/include
FFTW_LIBS = -L$(LIB_DIR)/lib
HDF5_INCL = -I$(LIB_DIR)/include
HDF5_LIBS = -L$(LIB_DIR)/lib
HWLOC_INCL = -I$(LIB_DIR)/include
HWLOC_LIBS = -L$(LIB_DIR)/lib
GRACKLE_INCL = -I$(LIB_DIR)/include
GRACKLE_LIBS = -L$(LIB_DIR)/lib -lgrackle
GRACKLE_OPT = -DLINUX -fPIC -DLARGE_INTS -DCONFIG_BFLOAT_8
見る感じ Makefile.path.virgo をそのまま使えそう。
$ pwd
/home/daisuke/Osaka-gadget4
$ cp Template-Makefile.systype Makefile.systype
$ ls
buildsystem documentation make-examples.sh src
data Doxyfile Makefile Template-Config.sh
defines_extra examples Makefile.systype Template-Makefile.systype
DEVELOPERS LICENSE README.md
でMakefile.systype の中の使うやつのコメントアウトを外す。
- #SYSTYPE="virgo"
+ SYSTYPE="virgo"
実行
ディレクトリの準備をする
$ pwd
/home/daisuke
$ ls
6248d133ab20.zip fftw-3.3.10.tar.gz LINE.py res2023-tex
arepo grackle-3.3.0.tar.gz local smartctl.txt
arepo-public grackle-grackle-3.3.0 mach50 zlib-1.3.1
badsectors.txt gsl-2.8 nohup.out zlib-1.3.1.tar.gz
check.txt gsl-latest.tar.gz ohahn-music-6248d133ab20
erosita hdf5-1.13.0 OpenTsiolkovsky
fftw-3.3.10 hdf5-1.13.0.tar.gz Osaka-gadget4
$ mkdir -p sim/example
$ tree sim
sim
└── example
1 directory, 0 files
compile
$ make -j CONFIG=./examples/DM-L50-N128/Config.sh DIR=/home/daisuke/sim/example
エラーが無く、完了したら確認。
$ ls /home/daisuke/sim/example/
build Gadget4
Gadget4 という実行ファイルができている。
run
outputs.txt と param.txt を移動する。
$ ls examples/DM-L50-N128/
Config.sh job.sh outputs.txt param.txt PlotPowerSpec.pro PlotPowerSpec.py
$ cp examples/DM-L50-N128/*.txt /home/daisuke/sim/example
$ ls /home/daisuke/sim/example/
build Gadget4 outputs.txt param.txt
次のコマンドを投げたい。複数あるのでrun.sh とかでも作成して書けばいい。
G4_DIR=/home/daisuke/Osaka-gadget4
WORK_DIR=/home/daisuke/sim/example
cd $WORK_DIR
mpirun -np 2 ./Gadget4 param.txt > logfile_gadget_0
ただし2のとこは注意。
$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 39 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Vendor ID: GenuineIntel
Model name: Intel(R) Core(TM) i5-5287U CPU @ 2.90GHz
CPU family: 6
Model: 61
Thread(s) per core: 2
Core(s) per socket: 2
Socket(s): 1
Stepping: 4
CPU max MHz: 3300.0000
CPU min MHz: 500.0000
BogoMIPS: 5799.52
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pa
t pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall
nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good
nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes6
4 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid
sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave a
vx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid
_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ep
t vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpc
id rdseed adx smap intel_pt xsaveopt dtherm ida arat pln pts md_
clear flush_l1d
Virtualization features:
Virtualization: VT-x
Caches (sum of all):
L1d: 64 KiB (2 instances)
L1i: 64 KiB (2 instances)
L2: 512 KiB (2 instances)
L3: 3 MiB (1 instance)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-3
Vulnerabilities:
Gather data sampling: Not affected
Itlb multihit: KVM: Mitigation: VMX disabled
L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vu
lnerable
Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Meltdown: Mitigation; PTI
Mmio stale data: Unknown: No mitigations
Reg file data sampling: Not affected
Retbleed: Not affected
Spec rstack overflow: Not affected
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and secc
omp
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitiza
tion
Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP conditi
onal; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Srbds: Mitigation; Microcode
Tsx async abort: Not affected
2コアぐらいで進めることにする。
$ sh run.sh
(略)
./Gadget4: error while loading shared libraries: libhdf5.so.300: cannot open shared object file: No such file or directory
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
./Gadget4: error while loading shared libraries: libhdf5.so.300: cannot open shared object file: No such file or directory
となっている。インストールし忘れたか？？
$ locate libhdf5.so
/home/daisuke/hdf5-1.13.0/src/.libs/libhdf5.so
/home/daisuke/hdf5-1.13.0/src/.libs/libhdf5.so.300
/home/daisuke/hdf5-1.13.0/src/.libs/libhdf5.so.300.0.0
/home/daisuke/local/lib/libhdf5.so
/home/daisuke/local/lib/libhdf5.so.300
/home/daisuke/local/lib/libhdf5.so.300.0.0
/mnt/hdd/STORAGE_Backup/local/lib/libhdf5.so
/mnt/hdd/STORAGE_Backup/local/lib/libhdf5.so.300
/mnt/hdd/STORAGE_Backup/local/lib/libhdf5.so.300.0.0
/opt/anaconda3/lib/libhdf5.so
/opt/anaconda3/lib/libhdf5.so.200
/opt/anaconda3/lib/libhdf5.so.200.1.0
/opt/anaconda3/pkgs/hdf5-1.12.1-h2b7332f_3/lib/libhdf5.so
/opt/anaconda3/pkgs/hdf5-1.12.1-h2b7332f_3/lib/libhdf5.so.200
/opt/anaconda3/pkgs/hdf5-1.12.1-h2b7332f_3/lib/libhdf5.so.200.1.0
しっかりあるが開けないのは、pathを通してないからだろう。.bashrcに次のことを記述する。
+ export LD_LIBRARY_PATH=/home/daisuke/local/lib:$LD_LIBRARY_PATH
+ export C_INCLUDE_PATH=/home/daisuke/local/include:$C_INCLUDE_PATH
これで実行すると
$ sh run.sh
だいぶ時間がかかりそう。（= うまくいってそう！）
これにて完了。
あくまでもtest runなので中断させて、途中経過をみる。
$ ls
build logfile_gadget_0 param.txt
ewald_table_1-1-1_64-64-64_precision8-order3.dat output param.txt-usedvalues
Gadget4 outputs.txt run.sh
とりあえずしっかり動いてそう。