A collection of Python and Bash scripts to enable easy testing of Fortran compilers using BATS.
$ ./run_test.sh gfortran$ ./run_test.sh bleedingedgecompiler$ ./compare.py gfortran bleedingedgecompiler
This is a framework built to help test bleeding-edge Fortran compilers. By testing the compilation of a wide variety of Fortran programs, and cross-checking results with stable compilers like gfortran, one can find where a compiler might be going wrong.
To add a directory containing Fortran programs to use for testing compilation, cd into /main/, and use ./addNew.sh <path-to-directory>.
Then, use ./run_test.sh (compiler) to compile all files found in the subdirectories of /source/. If no compiler is specified, it will use the current value of $FC (which is set to f77 by default within the Makefile). Each time ./run_test.sh is executed, the results will be saved to /logs/ as <compiler>.log, where you can view all compilation errors encountered during runtime.
Additionally, running ./compare.py <compiler1> <compiler2> (-b / -g / -v <col>) from the base directory outputs differences in errors between two given compilers, where:
-b outputs a very basic summary,-g outputs only the first line in each error to a two-column table, and-v <col> outputs errors in full to a two-column table with <col> columns on each side, up to 256../addMakefile.sh <directory>: creates a generic Makefile. Runs automatically during ./run_test.sh if no Makefile is present../verify.sh: checks all subdirectories of /source/ for possible race conditions../addEnv.sh <directory>: adds comm.env to a subdirectory, allowing BATS to use necessary environmental commands before compiling, e.g. module load../cleanup.sh <directory>: helps delete and/or backup your .bats files. This is important for when you add or delete files from a source directory../resetAll.sh: deletes all instances of tests.bats across all directories — that is, the BATS file generated during ./run_test.sh.Scrapers can be immensely helpful with gathering huge swathes of data at once. Within the /scraping/ directory, one can find scraper.py, a Python program that can help pull multiple Fortran files from a single website. It will then save these files to a directory named from the website, at which point the user can use ./addNew.sh ../scraper/<directory> in /main/ to add the files. To use, type:
./scraper.py <website url> <path to files> <extension>
<website url>: the page on which links to each Fortran file are found.<path to files>: URL path preceding each file location. Example below.<extension>: the desired extension to search for and download, e.g. .f90.Using the example of Michel Olagnon’s ORDERPACK 2.0, one might execute:
./scraper.py http://www.fortran-2000.com/rank/index.html http://www.fortran-2000.com/rank/ .f90
In this example, http://www.fortran-2000.com/rank/ is used as the second argument because every Fortran file on the page is located at http://www.fortran-2000.com/rank/<FILE.f90>.
Though not all of these need a scraper, other websites one might pull Fortran files from include:
Use ./verify.py from /main/ to check for possible race conditions!
For the files being compiled, it’s helpful that all files involved would already make perfectly with the -j flag. If there’s a problem with the code itself, both compilers will encounter it, which should be alright; but sometimes, race conditions present within directories can cause sporadic errors that do not occur every runtime, thereby rendering the testing inconsistent and therefore invalid.
For example, if a directory has multiple declarations of the same module across different files, multiple instances of make might try to write to <modulename>.mod at once during make, thereby allowing for a race condition. This mod will subsequently be corrupted when referenced later on.
In short, race conditions are bad.
$ ./compare.py gfortran flang////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////DIR: home/agustin/Projects/ftframework/source/buDIFF_COUNT: gfortran: 0 flang: 1TOTAL_ERRORS: gfortran: 0 flang: 1================================================================COMPILER: flangERROR: real.o# flang -c real.f90# F90-S-0081-Illegal selector - KIND parameter has unknown value for data type (real.f90: 5)# F90-S-0034-Syntax error at or near identifier q0 (real.f90: 14)# 0 inform, 0 warnings, 2 severes, 0 fatal for MAIN# make: *** [Makefile:151: real.o] Error 1////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////DIR: home/agustin/Projects/ftframework/source/nasa_fDIFF_COUNT: gfortran: 0 flang: 10TOTAL_ERRORS: gfortran: 0 flang: 10================================================================COMPILER: flangERROR: prcset.o# flang -c prcset.f# flang-7: error: unable to execute command: Segmentation fault (core dumped)# flang-7: error: Fortran frontend to LLVM command failed due to signal (use -v to see invocation)# clang version 7.0.1# Target: x86_64-pc-linux-gnu# Thread model: posix# InstalledDir: /usr/bin# flang-7: note: diagnostic msg: PLEASE submit a bug report to and include the crash backtrace, preprocessed source, and associated run script.# flang-7: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.# make: *** [Makefile:433: prcset.o] Error 254ERROR: pool.o# flang -c pool.f# flang-7: error: unable to execute command: Segmentation fault (core dumped)# flang-7: error: Fortran frontend to LLVM command failed due to signal (use -v to see invocation)# clang version 7.0.1# Target: x86_64-pc-linux-gnu# Thread model: posix# InstalledDir: /usr/bin# flang-7: note: diagnostic msg: PLEASE submit a bug report to and include the crash backtrace, preprocessed source, and associated run script.# flang-7: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.# make: *** [Makefile:2179: pool.o] Error 254ERROR: zzdskbsr.o# flang -c zzdskbsr.f# /tmp/zzdskbsr-0fa576.ll:2936:12: error: '@_master___zzdskbsr__' defined with type 'void (i32, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64, i64*)*'# call void @_master___zzdskbsr__(i32 0, i64* %fname, i64* %bodyid, i64* %handle, i64* %cmpfun, i64* %usrctr, i64* %update, i64* %dladsc, i64* %dskdsc, i64* %found, i64* %cmpfun$sd, i64%.U0001.addr)# ^# 1 error generated.# make: *** [Makefile:3667: zzdskbsr.o] Error 1ERROR: zzgfdiu.o# flang -c zzgfdiu.f# /tmp/zzgfdiu-739e1b.ll:313:12: error: '@_master___zzgfdiu__' defined with type 'void (i32, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64, i64, i64, i64*)*'# call void @_master___zzgfdiu__(i32 0, i64* %target, i64* %abcorr, i64* %obsrvr, i64* %udfunc, i64* %et, i64* %decres, i64* %dist, i64* %udfunc$sd, i64%.U0003.addr, i64%.U0004.addr, i64%.U0005.addr)# ^# 1 error generated.# make: *** [Makefile:4297: zzgfdiu.o] Error 1ERROR: zzgfcou.o# flang -c zzgfcou.f# /tmp/zzgfcou-02d33d.ll:2110:12: error: '@_master___zzgfcou__' defined with type 'void (i32, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64, i64, i64, i64, i64, i64, i64, i64, i64, i64*)*'# call void @_master___zzgfcou__(i32 0, i64* %vecdef, i64* %method, i64* %target, i64* %et, i64* %ref, i64* %abcorr, i64* %obsrvr, i64* %dref, i64* %dvec, i64* %crdsys, i64* %crdnam, i64* %decres, i64* %crdval, i64* %crdfnd, i64* %udfunc, i64* %udfunc$sd, i64%.U0009.addr, i64%.U0010.addr, i64%.U0011.addr, i64%.U0012.addr, i64%.U0013.addr, i64%.U0014.addr, i64%.U0015.addr, i64%.U0016.addr, i64%.U0017.addr)# ^# 1 error generated.# make: *** [Makefile:4285: zzgfcou.o] Error 1ERROR: zzgfilu.o# flang -c zzgfilu.f# /tmp/zzgfilu-0c1f3b.ll:788:12: error: '@_master___zzgfilu__' defined with type 'void (i32, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64, i64, i64, i64, i64, i64, i64, i64*)*'# call void @_master___zzgfilu__(i32 0, i64* %method, i64* %angtyp, i64* %target, i64* %illum, i64* %fixref, i64* %abcorr, i64* %obsrvr, i64* %spoint, i64* %et, i64* %udfunc, i64* %decres, i64* %angle, i64* %udfunc$sd, i64%.U0007.addr, i64%.U0008.addr, i64%.U0009.addr, i64%.U0010.addr, i64%.U0011.addr, i64%.U0012.addr, i64%.U0013.addr)# ^# 1 error generated.# make: *** [Makefile:4306: zzgfilu.o] Error 1ERROR: zzgfpau.o# flang -c zzgfpau.f# /tmp/zzgfpau-22f585.ll:552:12: error: '@_master___zzgfpau__' defined with type 'void (i32, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64, i64, i64, i64, i64, i64*)*'# call void @_master___zzgfpau__(i32 0, i64* %target, i64* %illmn, i64* %abcorr, i64* %obsrvr, i64* %udfunc, i64* %et, i64* %decres, i64* %rvl, i64* %xtarg, i64* %xillmn, i64* %xabcor, i64* %xobs, i64* %xablk, i64* %udfunc$sd, i64%.U0005.addr, i64%.U0006.addr, i64%.U0007.addr, i64%.U0008.addr, i64%.U0009.addr)# ^# 1 error generated.# make: *** [Makefile:4318: zzgfpau.o] Error 1ERROR: zzgfrru.o# flang -c zzgfrru.f# /tmp/zzgfrru-a1f258.ll:447:12: error: '@_master___zzgfrru__' defined with type 'void (i32, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64, i64, i64, i64, i64*)*'# call void @_master___zzgfrru__(i32 0, i64* %target, i64* %abcorr, i64* %obsrvr, i64* %dt, i64* %udfunc, i64* %et, i64* %decres, i64* %rvl, i64* %xtarg, i64* %xabcor, i64* %xobs, i64* %xdt, i64* %udfunc$sd, i64%.U0004.addr, i64%.U0005.addr, i64%.U0006.addr, i64%.U0007.addr)# ^# 1 error generated.# make: *** [Makefile:4336: zzgfrru.o] Error 1ERROR: zzgfspu.o# flang -c zzgfspu.f# /tmp/zzgfspu-f6f7b3.ll:1041:12: error: '@_master___zzgfspu__' defined with type 'void (i32, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64, i64, i64, i64, i64, i64, i64, i64, i64*)*'# call void @_master___zzgfspu__(i32 0, i64* %of, i64* %from, i64* %shape, i64* %frame, i64* %et, i64* %udfunc, i64* %abcorr, i64* %decres, i64* %sep, i64* %xabcr, i64* %xbod, i64* %yref, i64* %xref, i64* %xobs, i64* %xrad, i64* %xshp, i64* %udfunc$sd, i64%.U0008.addr, i64%.U0009.addr, i64%.U0010.addr, i64%.U0011.addr, i64%.U0012.addr, i64%.U0013.addr, i64%.U0014.addr, i64%.U0015.addr)# ^# 1 error generated.# make: *** [Makefile:4348: zzgfspu.o] Error 1ERROR: tabrpt.o# flang -c tabrpt.f# /tmp/tabrpt-ca1837.ll:1687:12: error: '@_master___tabrpt__' defined with type 'void (i32, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64*, i64, i64*)*'# call void @_master___tabrpt__(i32 0, i64* %nitems, i64* %item, i64* %size, i64* %width, i64* %justr, i64* %presrv, i64* %spcial, i64* %lmarge, i64* %space, i64* %fetch, i64* %fetch$sd, i64%.U0001.addr)# ^# 1 error generated.# make: *** [Makefile:5314: tabrpt.o] Error 1
$ ./compare.py gfortran flang -b================================================================DIR: home/agustin/Projects/ftframework/source/buDIFF_COUNT: gfortran: 0 flang: 1TOTAL_ERRORS: gfortran: 0 flang: 1COMPILER: flangreal.o================================================================DIR: home/agustin/Projects/ftframework/source/nasa_fDIFF_COUNT: gfortran: 0 flang: 10TOTAL_ERRORS: gfortran: 0 flang: 10COMPILER: flangprcset.opool.ozzdskbsr.ozzgfdiu.ozzgfcou.ozzgfilu.ozzgfpau.ozzgfrru.ozzgfspu.otabrpt.o
$ ./compare.py gfortran flang -ggfortran | flang0 -------------- /source/bu ------------- 1| real.o|0 ----------- /source/nasa_f ----------- 10| prcset.o| pool.o| zzdskbsr.o| zzgfdiu.o| zzgfcou.o| zzgfilu.o| zzgfpau.o| zzgfrru.o| zzgfspu.o| tabrpt.o|
$ ./compare.py gfortran flang -v 48gfortran | flang0 ------------------------------------------ /source/bu ----------------------------------------- 1| real.o|| # flang -c real.f90| # F90-S-0081-Illegal selector - KIND paramete...| # F90-S-0034-Syntax error at or near identifi...| # 0 inform, 0 warnings, 2 severes, 0 fa...| # make: *** [Makefile:151: real.o] Error 1|| ------------------------------------------------|0 --------------------------------------- /source/nasa_f --------------------------------------- 10| prcset.o|| # flang -c prcset.f| # flang-7: error: unable to execute command: ...| # flang-7: error: Fortran frontend to LLVM co...| # clang version 7.0.1| # Target: x86_64-pc-linux-gnu| # Thread model: posix| # InstalledDir: /usr/bin| # flang-7: note: diagnostic msg: PLEASE submi...| # flang-7: note: diagnostic msg: Error genera...| # make: *** [Makefile:433: prcset.o] Error 254|| ------------------------------------------------| pool.o|| # flang -c pool.f| # flang-7: error: unable to execute command: ...| # flang-7: error: Fortran frontend to LLVM co...| # clang version 7.0.1| # Target: x86_64-pc-linux-gnu| # Thread model: posix| # InstalledDir: /usr/bin| # flang-7: note: diagnostic msg: PLEASE submi...| # flang-7: note: diagnostic msg: Error genera...| # make: *** [Makefile:2179: pool.o] Error 254|| ------------------------------------------------| zzdskbsr.o|| # flang -c zzdskbsr.f| # /tmp/zzdskbsr-0fa576.ll:2936:12: error: '@_...| # call void @_master___zzdskbsr__(i32...| # ^| # 1 error generated.| # make: *** [Makefile:3667: zzdskbsr.o] Error 1|| ------------------------------------------------| zzgfdiu.o|| # flang -c zzgfdiu.f| # /tmp/zzgfdiu-739e1b.ll:313:12: error: '@_ma...| # call void @_master___zzgfdiu__(i32 ...| # ^| # 1 error generated.| # make: *** [Makefile:4297: zzgfdiu.o] Error 1|| ------------------------------------------------| zzgfcou.o|| # flang -c zzgfcou.f| # /tmp/zzgfcou-02d33d.ll:2110:12: error: '@_m...| # call void @_master___zzgfcou__(i32 ...| # ^| # 1 error generated.| # make: *** [Makefile:4285: zzgfcou.o] Error 1|| ------------------------------------------------| zzgfilu.o|| # flang -c zzgfilu.f| # /tmp/zzgfilu-0c1f3b.ll:788:12: error: '@_ma...| # call void @_master___zzgfilu__(i32 ...| # ^| # 1 error generated.| # make: *** [Makefile:4306: zzgfilu.o] Error 1|| ------------------------------------------------| zzgfpau.o|| # flang -c zzgfpau.f| # /tmp/zzgfpau-22f585.ll:552:12: error: '@_ma...| # call void @_master___zzgfpau__(i32 ...| # ^| # 1 error generated.| # make: *** [Makefile:4318: zzgfpau.o] Error 1|| ------------------------------------------------| zzgfrru.o|| # flang -c zzgfrru.f| # /tmp/zzgfrru-a1f258.ll:447:12: error: '@_ma...| # call void @_master___zzgfrru__(i32 ...| # ^| # 1 error generated.| # make: *** [Makefile:4336: zzgfrru.o] Error 1|| ------------------------------------------------| zzgfspu.o|| # flang -c zzgfspu.f| # /tmp/zzgfspu-f6f7b3.ll:1041:12: error: '@_m...| # call void @_master___zzgfspu__(i32 ...| # ^| # 1 error generated.| # make: *** [Makefile:4348: zzgfspu.o] Error 1|| ------------------------------------------------| tabrpt.o|| # flang -c tabrpt.f| # /tmp/tabrpt-ca1837.ll:1687:12: error: '@_ma...| # call void @_master___tabrpt__(i32 0...| # ^| # 1 error generated.| # make: *** [Makefile:5314: tabrpt.o] Error 1|| ------------------------------------------------|