Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compiling library and Python binding on Apple M1 #276

Open
1 task done
nadnerbity opened this issue Mar 17, 2022 · 12 comments
Open
1 task done

Compiling library and Python binding on Apple M1 #276

nadnerbity opened this issue Mar 17, 2022 · 12 comments
Labels
bug Something isn't working

Comments

@nadnerbity
Copy link

Solution to issue cannot be found in the documentation.

  • I checked the documentation.

Issue

I am trying to compile a piece of niche software called SRW using miniforge3 for arm64. The software installs just fine using the system python. When using conda/miniforge, the issue is that the compiler can't seem to find the correct libraries so I get an error like (complete error is below):

Undefined symbols for architecture arm64:
  "_PyArg_ParseTuple", referenced from:
      srwlpy_CalcMagnField(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcPartTraj(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcPartTrajFromKickMatr(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcElecFieldSR(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcElecFieldGaussian(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcElecFieldPointSrc(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcStokesUR(_object*, _object*) in srwlpy-87a854.o

ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [pylib] Error 1

The steps to reproduce this output are:

  1. Install miniforge3

  2. Clone SRW
    git clone https://github.com/ochubar/SRW.git

  3. Compile fftw3, instructions are in section III.1.2.1 here:
    https://github.com/ochubar/SRW
    You'll need to compile both versions of 3.3.8, but no 2.1.5

  4. change to directory SRW/cpp/gcc

  5. Update the makefile (copy of file I use below)
    changes are:
    A) CC = gcc, CXX = g+=
    B) CFLAGS+= -D__MAC__
    C) Get rid of -DLINUX flag hard coded into SRW_SRC_DEF
    D) Update PYFLAGS

  6. make lib

  7. make pylib (this generates the error)


Complete error:

g++ -shared -O3 -fPIC -I..//src/core -I..//src/lib -I..//src/ext/auxparse -I..//src/ext/genmath -D_GNU_SOURCE -D__USE_XOPEN2K8 -DFFTW_ENABLE_FLOAT -D_GM_WITHOUT_BASE -DSRWLIB_STATIC -DNO_TIMER -DANSI_DECLARATORS -DTRILIBRARY  -D_FFTW3 -D__MAC__ -w -I/Users/brendan/miniforge3/include/python3.9 -I /Users/brendan/miniforge3/include/python3.9 -L/Users/brendan/miniforge3/lib/python3.9/config-3.9-darwin -ldl -o srwlpy.so ../src/clients/python/srwlpy.cpp libsrw.a -L..//../ext_lib -lm  -lfftw3f -lfftw3 
Undefined symbols for architecture arm64:
  "_PyArg_ParseTuple", referenced from:
      srwlpy_CalcMagnField(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcPartTraj(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcPartTrajFromKickMatr(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcElecFieldSR(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcElecFieldGaussian(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcElecFieldPointSrc(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcStokesUR(_object*, _object*) in srwlpy-87a854.o
      ...
  "_PyBuffer_Release", referenced from:
      ReleasePyBuffers(std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >&) in srwlpy-87a854.o
      srwlpy_CalcMagnField(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcPartTraj(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcPartTrajFromKickMatr(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcElecFieldSR(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcElecFieldGaussian(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcElecFieldPointSrc(_object*, _object*) in srwlpy-87a854.o
      ...
  "_PyBytes_AsString", referenced from:
      CopyPyStringToC(_object*, char*, int) in srwlpy-87a854.o
  "_PyBytes_AsStringAndSize", referenced from:
      CopyPyStringToC(_object*, char*, int) in srwlpy-87a854.o
  "_PyBytes_Size", referenced from:
      CopyPyStringToC(_object*, char*, int) in srwlpy-87a854.o
  "_PyCallable_Check", referenced from:
      ModifySRWLWfr(int, SRWLStructWaveFront*, char) in srwlpy-87a854.o
      AllocPyArrayGetBuf(char, long long) in srwlpy-87a854.o
  "_PyErr_PrintEx", referenced from:
      ProcRes(int) in srwlpy-87a854.o
      srwlpy_CalcMagnField(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcPartTraj(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcPartTrajFromKickMatr(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcElecFieldSR(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcElecFieldGaussian(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcElecFieldPointSrc(_object*, _object*) in srwlpy-87a854.o
      ...
  "_PyErr_SetString", referenced from:
      ProcRes(int) in srwlpy-87a854.o
      srwlpy_CalcMagnField(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcPartTraj(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcPartTrajFromKickMatr(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcElecFieldSR(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcElecFieldGaussian(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcElecFieldPointSrc(_object*, _object*) in srwlpy-87a854.o
      ...
  "_PyExc_RuntimeError", referenced from:
      srwlpy_CalcMagnField(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcPartTraj(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcPartTrajFromKickMatr(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcElecFieldSR(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcElecFieldGaussian(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcElecFieldPointSrc(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcStokesUR(_object*, _object*) in srwlpy-87a854.o
      ...
  "_PyExc_Warning", referenced from:
      ProcRes(int) in srwlpy-87a854.o
      srwlpy_CalcMagnField(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcPartTraj(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcPartTrajFromKickMatr(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcElecFieldSR(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcElecFieldGaussian(_object*, _object*) in srwlpy-87a854.o
      srwlpy_CalcElecFieldPointSrc(_object*, _object*) in srwlpy-87a854.o
      ...
  "_PyFloat_AsDouble", referenced from:
      ParseSructSRWLParticle(SRWLStructParticle*, _object*) in srwlpy-87a854.o
      ParseSructSRWLPartBeam(SRWLStructParticleBeam*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >&) in srwlpy-87a854.o
      char CopyPyListElemsToNumArray<double>(_object*, char, double*&, int&) in srwlpy-87a854.o
      ParseSructSRWLPrtTrj(SRWLStructParticleTrajectory*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*) in srwlpy-87a854.o
      ParseSructSRWLKickM(SRWLStructKickMatrix*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*) in srwlpy-87a854.o
      ParseSructSRWLMagFld3D(SRWLStructMagneticField3D*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*) in srwlpy-87a854.o
      ParseSructSRWLMagFldM(SRWLStructMagneticFieldMultipole*, _object*) in srwlpy-87a854.o
      ...
  "_PyFloat_Type", referenced from:
      char CPyParse::CopyNumOrPyListElemsToNumArray<double, int>(_object*, char, double*&, int&, bool&) in srwlpy-87a854.o
      void UpdatePyListNum<double>(_object*, double const*, int) in srwlpy-87a854.o
      void UpdatePyListNum<int>(_object*, int const*, int) in srwlpy-87a854.o
  "_PyImport_AddModule", referenced from:
      AllocPyArrayGetBuf(char, long long) in srwlpy-87a854.o
  "_PyList_Append", referenced from:
      UpdatePyPropInt(_object*, SRWLStructRadMesh*, char**, int) in srwlpy-87a854.o
      void UpdatePyListNum<double>(_object*, double const*, int) in srwlpy-87a854.o
      void UpdatePyListNum<int>(_object*, int const*, int) in srwlpy-87a854.o
  "_PyList_GetItem", referenced from:
      char CopyPyListElemsToNumArray<double>(_object*, char, double*&, int&) in srwlpy-87a854.o
      ParseSructSRWLMagFldU(SRWLStructMagneticFieldUndulator*, _object*) in srwlpy-87a854.o
      ParseSructSRWLMagFldC(SRWLStructMagneticFieldContainer*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*) in srwlpy-87a854.o
      ParseSructSRWLOptC(SRWLStructOpticsContainer*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*) in srwlpy-87a854.o
      ParseSructSRWLPropIntDef(char**, SRWLStructRadMesh*&, _object*) in srwlpy-87a854.o
      ParseSructSmpObj3D(double**&, int&, _object*) in srwlpy-87a854.o
      UpdatePyMagFldU(_object*, SRWLStructMagneticFieldUndulator*) in srwlpy-87a854.o
      ...
  "_PyList_New", referenced from:
      UpdatePyPropInt(_object*, SRWLStructRadMesh*, char**, int) in srwlpy-87a854.o
      srwlpy_UtiIntInf(_object*, _object*) in srwlpy-87a854.o
  "_PyList_SetItem", referenced from:
      UpdatePyPropInt(_object*, SRWLStructRadMesh*, char**, int) in srwlpy-87a854.o
      srwlpy_UtiIntInf(_object*, _object*) in srwlpy-87a854.o
      void UpdatePyListNum<double>(_object*, double const*, int) in srwlpy-87a854.o
      void UpdatePyListNum<int>(_object*, int const*, int) in srwlpy-87a854.o
  "_PyList_Size", referenced from:
      char CopyPyListElemsToNumArray<double>(_object*, char, double*&, int&) in srwlpy-87a854.o
      ParseSructSRWLMagFldU(SRWLStructMagneticFieldUndulator*, _object*) in srwlpy-87a854.o
      ParseSructSRWLMagFldC(SRWLStructMagneticFieldContainer*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*) in srwlpy-87a854.o
      ParseSructSRWLOptC(SRWLStructOpticsContainer*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*) in srwlpy-87a854.o
      ParseSructSRWLPropIntDef(char**, SRWLStructRadMesh*&, _object*) in srwlpy-87a854.o
      ParseSructSmpObj3D(double**&, int&, _object*) in srwlpy-87a854.o
      UpdatePyMagFldU(_object*, SRWLStructMagneticFieldUndulator*) in srwlpy-87a854.o
      ...
  "_PyLong_AsLong", referenced from:
      ParseSructSRWLParticle(SRWLStructParticle*, _object*) in srwlpy-87a854.o
      char CopyPyListElemsToNumArray<double>(_object*, char, double*&, int&) in srwlpy-87a854.o
      ParseSructSRWLPrtTrj(SRWLStructParticleTrajectory*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*) in srwlpy-87a854.o
      ParseSructSRWLKickM(SRWLStructKickMatrix*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*) in srwlpy-87a854.o
      ParseSructSRWLMagFld3D(SRWLStructMagneticField3D*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*) in srwlpy-87a854.o
      ParseSructSRWLMagFldM(SRWLStructMagneticFieldMultipole*, _object*) in srwlpy-87a854.o
      ParseSructSRWLMagFldH(SRWLStructMagneticFieldHarmonic*, _object*) in srwlpy-87a854.o
      ...
  "_PyModule_Create2", referenced from:
      _PyInit_srwlpy in srwlpy-87a854.o
  "_PyNumber_Check", referenced from:
      ParseSructSRWLParticle(SRWLStructParticle*, _object*) in srwlpy-87a854.o
      ParseSructSRWLPartBeam(SRWLStructParticleBeam*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >&) in srwlpy-87a854.o
      char CopyPyListElemsToNumArray<double>(_object*, char, double*&, int&) in srwlpy-87a854.o
      ParseSructSRWLPrtTrj(SRWLStructParticleTrajectory*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*) in srwlpy-87a854.o
      ParseSructSRWLKickM(SRWLStructKickMatrix*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*) in srwlpy-87a854.o
      ParseSructSRWLMagFld3D(SRWLStructMagneticField3D*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*) in srwlpy-87a854.o
      ParseSructSRWLMagFldM(SRWLStructMagneticFieldMultipole*, _object*) in srwlpy-87a854.o
      ...
  "_PyObject_CallObject", referenced from:
      UpdatePyPropInt(_object*, SRWLStructRadMesh*, char**, int) in srwlpy-87a854.o
      ModifySRWLWfr(int, SRWLStructWaveFront*, char) in srwlpy-87a854.o
      AllocPyArrayGetBuf(char, long long) in srwlpy-87a854.o
  "_PyObject_CheckBuffer", referenced from:
      GetPyArrayBuf(_object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*, long*) in srwlpy-87a854.o
      char CopyPyListElemsToNumArray<double>(_object*, char, double*&, int&) in srwlpy-87a854.o
      AllocPyArrayGetBuf(char, long long) in srwlpy-87a854.o
      char CPyParse::CopyPyListElemsToNumArray<double, int>(_object*, char, double*&, int&, bool&) in srwlpy-87a854.o
      char CPyParse::CopyPyNestedListElemsToNumVect<double>(_object*, char, std::__1::vector<double, std::__1::allocator<double> >*) in srwlpy-87a854.o
      char CopyPyListElemsToNumArray<int>(_object*, char, int*&, int&) in srwlpy-87a854.o
  "_PyObject_GetAttrString", referenced from:
      ParseSructSRWLParticle(SRWLStructParticle*, _object*) in srwlpy-87a854.o
      ParseSructSRWLPartBeam(SRWLStructParticleBeam*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >&) in srwlpy-87a854.o
      ParseSructSRWLPrtTrj(SRWLStructParticleTrajectory*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*) in srwlpy-87a854.o
      ParseSructSRWLKickM(SRWLStructKickMatrix*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*) in srwlpy-87a854.o
      ParseSructSRWLMagFld3D(SRWLStructMagneticField3D*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*) in srwlpy-87a854.o
      ParseSructSRWLMagFldM(SRWLStructMagneticFieldMultipole*, _object*) in srwlpy-87a854.o
      ParseSructSRWLMagFldS(SRWLStructMagneticFieldSolenoid*, _object*) in srwlpy-87a854.o
      ...
  "_PyObject_GetBuffer", referenced from:
      GetPyArrayBuf(_object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*, long*) in srwlpy-87a854.o
      char CopyPyListElemsToNumArray<double>(_object*, char, double*&, int&) in srwlpy-87a854.o
      AllocPyArrayGetBuf(char, long long) in srwlpy-87a854.o
      char CPyParse::CopyPyListElemsToNumArray<double, int>(_object*, char, double*&, int&, bool&) in srwlpy-87a854.o
      char CPyParse::CopyPyNestedListElemsToNumVect<double>(_object*, char, std::__1::vector<double, std::__1::allocator<double> >*) in srwlpy-87a854.o
      char CopyPyListElemsToNumArray<int>(_object*, char, int*&, int&) in srwlpy-87a854.o
  "_PyObject_HasAttrString", referenced from:
      ParseSructSRWLPrtTrj(SRWLStructParticleTrajectory*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*) in srwlpy-87a854.o
      ParseSructSRWLMagFldC(SRWLStructMagneticFieldContainer*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*) in srwlpy-87a854.o
      ParseSructSRWLWfr(SRWLStructWaveFront*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*, std::__1::map<SRWLStructWaveFront*, AuxStructPyObjectPtrs, std::__1::less<SRWLStructWaveFront*>, std::__1::allocator<std::__1::pair<SRWLStructWaveFront* const, AuxStructPyObjectPtrs> > >&) in srwlpy-87a854.o
      ModifySRWLWfr(int, SRWLStructWaveFront*, char) in srwlpy-87a854.o
  "_PyObject_SetAttrString", referenced from:
      UpdatePyRadMesh(_object*, SRWLStructRadMesh*) in srwlpy-87a854.o
      UpdatePyWfr(_object*, SRWLStructWaveFront*) in srwlpy-87a854.o
      UpdatePyStokes(_object*, SRWLStructStokes*) in srwlpy-87a854.o
      UpdatePyMagFldH(_object*, SRWLStructMagneticFieldHarmonic*) in srwlpy-87a854.o
      UpdatePyMagFldU(_object*, SRWLStructMagneticFieldUndulator*) in srwlpy-87a854.o
  "_PyType_IsSubtype", referenced from:
      char CPyParse::CopyNumOrPyListElemsToNumArray<double, int>(_object*, char, double*&, int&, bool&) in srwlpy-87a854.o
      void UpdatePyListNum<double>(_object*, double const*, int) in srwlpy-87a854.o
      void UpdatePyListNum<int>(_object*, int const*, int) in srwlpy-87a854.o
  "_PyUnicode_AsUTF8String", referenced from:
      CopyPyStringToC(_object*, char*, int) in srwlpy-87a854.o
  "_Py_BuildValue", referenced from:
      Py_BuildValueChar(char) in srwlpy-87a854.o
      UpdatePyRadMesh(_object*, SRWLStructRadMesh*) in srwlpy-87a854.o
      UpdatePyWfr(_object*, SRWLStructWaveFront*) in srwlpy-87a854.o
      UpdatePyStokes(_object*, SRWLStructStokes*) in srwlpy-87a854.o
      UpdatePyMagFldH(_object*, SRWLStructMagneticFieldHarmonic*) in srwlpy-87a854.o
      UpdatePyMagFldU(_object*, SRWLStructMagneticFieldUndulator*) in srwlpy-87a854.o
      UpdatePyPropInt(_object*, SRWLStructRadMesh*, char**, int) in srwlpy-87a854.o
      ...
  "__Py_Dealloc", referenced from:
      CopyPyStringToC(_object*, char*, int) in srwlpy-87a854.o
      _Py_DECREF(_object*) in srwlpy-87a854.o
      ParseSructSRWLParticle(SRWLStructParticle*, _object*) in srwlpy-87a854.o
      ParseSructSRWLPartBeam(SRWLStructParticleBeam*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >&) in srwlpy-87a854.o
      ParseSructSRWLPrtTrj(SRWLStructParticleTrajectory*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*) in srwlpy-87a854.o
      ParseSructSRWLKickM(SRWLStructKickMatrix*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*) in srwlpy-87a854.o
      ParseSructSRWLMagFld3D(SRWLStructMagneticField3D*, _object*, std::__1::vector<bufferinfo, std::__1::allocator<bufferinfo> >*) in srwlpy-87a854.o
      ...
  "__Py_NoneStruct", referenced from:
      srwlpy_PropagElecField(_object*, _object*) in srwlpy-87a854.o
      srwlpy_UtiIntProc(_object*, _object*) in srwlpy-87a854.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [pylib] Error 1

Makefile

SOFT_DEV_DIR=	../
SRW_SRC_DIR=	$(SOFT_DEV_DIR)/src
SRW_SRC_GEN_DIR=	$(SRW_SRC_DIR)/core
SRW_SRC_LIB_DIR=	$(SRW_SRC_DIR)/lib
SRW_SRC_GENESIS_DIR=	$(SRW_SRC_DIR)/ext/genesis/genesis_july08
SH_SRC_PARSE_DIR=	$(SRW_SRC_DIR)/ext/auxparse
SH_SRC_GEN_MATH_DIR=	$(SRW_SRC_DIR)/ext/genmath
LIB_DIR=	$(SOFT_DEV_DIR)/../ext_lib

MODE ?= 0

CC  = gcc
CXX = g++

SRW_SRC_DEF=	-D_GNU_SOURCE -D__USE_XOPEN2K8 -DFFTW_ENABLE_FLOAT -D_GM_WITHOUT_BASE -DSRWLIB_STATIC -DNO_TIMER -DANSI_DECLARATORS -DTRILIBRARY
CFLAGS=	-O3 -fPIC -I$(SRW_SRC_GEN_DIR) -I$(SRW_SRC_LIB_DIR) -I$(SH_SRC_PARSE_DIR) -I$(SH_SRC_GEN_MATH_DIR) $(SRW_SRC_DEF) 
LDFLAGS=-L$(LIB_DIR) -lm 

ifeq ($(MODE), omp)
CFLAGS+= -D_WITH_OMP -fopenmp -Wno-write-strings 
LDFLAGS+= -lfftw 
else
ifeq ($(MODE), 0)
CFLAGS+= -D_FFTW3 -D__MAC__ -w
LDFLAGS+= -lfftw3f -lfftw3 
else
$(error Unknown SRW compilation option)
endif
endif

# For miniforge env
PYPATH=/Users/brendan/miniforge3
PYFLAGS=-I$(PYPATH)/include/python3.9 -I $(PYPATH)/include/python3.9 -L$(PYPATH)/lib/python3.9/config-3.9-darwin -ldl

# For system python
#PYPATH=/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8
#PYFLAGS=-I$(PYPATH)/include/python3.8 -I$(PYPATH)/include/python3.8 -L$(PYPATH)/lib/python3.8/config-3.8-darwin -lpython3.8 -ldl

OBJ=	auxparse.o gmfft.o gmfit.o gminterp.o gmmeth.o gmtrans.o srclcuti.o srcradint.o srctrjdt.o sremitpr.o srgsnbm.o srgtrjdt.o srisosrc.o srmagcnt.o srmagfld.o srmatsta.o sroptapt.o sroptcnt.o sroptdrf.o sroptel2.o sroptel3.o sroptelm.o sroptfoc.o sroptgrat.o sroptgtr.o sropthck.o sroptcryst.o sroptmat.o sroptpsh.o sroptshp.o sroptsmr.o sroptwgr.o sroptzp.o sroptzps.o srpersto.o srpowden.o srprdint.o srprgind.o srpropme.o srptrjdt.o srradinc.o srradint.o srradmnp.o srradstr.o srremflp.o srsase.o srsend.o srstowig.o srsysuti.o srthckbm.o srthckbm2.o srtrjaux.o srtrjdat.o srtrjdat3d.o all_com.o check.o diagno.o esource.o field.o incoherent.o initrun.o input.o loadbeam.o loadrad.o magfield.o main.o math.o mpi.o output.o partsim.o pushp.o rpos.o scan.o source.o stepz.o string.o tdepend.o timerec.o track.o srerror.o srwlib.o 

PRG=	libsrw.a

lib:	$(OBJ)
	ar -cvq $(PRG) *.o
	#cp $(PRG) $(LIB_DIR)/
	rm -f *.o

%.o: $(SRW_SRC_LIB_DIR)/%.cpp
	$(CXX) $(CFLAGS) -c $<  

%.o: $(SH_SRC_PARSE_DIR)/%.cpp
	$(CXX) $(CFLAGS) -c $<  

%.o: $(SH_SRC_GEN_MATH_DIR)/%.cpp
	$(CXX) $(CFLAGS) -c $<  

%.o: $(SRW_SRC_GEN_DIR)/%.cpp
	$(CXX) $(CFLAGS) -c $<  

%.o: $(SRW_SRC_GENESIS_DIR)/%.c
	$(CC) $(CFLAGS) -c $<  

pylib:
	$(CXX) -shared $(CFLAGS) $(PYFLAGS) -o srwlpy.so ../src/clients/python/srwlpy.cpp libsrw.a $(LDFLAGS)

srwlclient: $(SRW_SRC_DIR)/clients/c/srwlclient.cpp
	$(CXX) $(CFLAGS) -O3 -o srwlclient ../src/clients/c/srwlclient.cpp libsrw.a $(LDFLAGS)

clean:
	rm -f *.o *.so *.a srwlclient

all: lib pylib srwlclient

Another library error

I recognize that the miniforge env PYPATH doesn't have the -lpython3.9, that is because when I include that I get a different error.

# For miniforge env
PYPATH=/Users/brendan/miniforge3
PYFLAGS=-I$(PYPATH)/include/python3.9 -I $(PYPATH)/include/python3.9 -L$(PYPATH)/lib/python3.9/config-3.9-darwin -lpython3.9 -ldl
g++ -shared -O3 -fPIC -I..//src/core -I..//src/lib -I..//src/ext/auxparse -I..//src/ext/genmath -D_GNU_SOURCE -D__USE_XOPEN2K8 -DFFTW_ENABLE_FLOAT -D_GM_WITHOUT_BASE -DSRWLIB_STATIC -DNO_TIMER -DANSI_DECLARATORS -DTRILIBRARY  -D_FFTW3 -D__MAC__ -w -I/Users/brendan/miniforge3/include/python3.9 -I /Users/brendan/miniforge3/include/python3.9 -L/Users/brendan/miniforge3/lib/python3.9/config-3.9-darwin -lpython3.9 -ldl -o srwlpy.so ../src/clients/python/srwlpy.cpp libsrw.a -L..//../ext_lib -lm  -lfftw3f -lfftw3 
ld: library not found for -lpython3.9
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [pylib] Error 1

Installed packages

# packages in environment at /Users/brendan/miniforge3/envs/SRW:
#
# Name                    Version                   Build  Channel
brotlipy                  0.7.0           py39h5161555_1003    conda-forge
bzip2                     1.0.8                h3422bc3_4    conda-forge
ca-certificates           2021.10.8            h4653dfc_0    conda-forge
certifi                   2021.10.8        py39h2804cbe_1    conda-forge
cffi                      1.15.0           py39h52b1de0_0    conda-forge
charset-normalizer        2.0.12             pyhd8ed1ab_0    conda-forge
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
conda-package-handling    1.7.3            py39h5161555_1    conda-forge
cryptography              36.0.1           py39hfb8cd70_0    conda-forge
idna                      3.3                pyhd8ed1ab_0    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libzlib                   1.2.11            hee7b306_1013    conda-forge
ncurses                   6.3                  hc470f4d_0    conda-forge
openssl                   1.1.1l               h3422bc3_0    conda-forge
pip                       22.0.3             pyhd8ed1ab_0    conda-forge
pycosat                   0.6.3           py39h5161555_1009    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyopenssl                 22.0.0             pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1            py39h2804cbe_4    conda-forge
python                    3.9.10          hd16f9c5_2_cpython    conda-forge
python_abi                3.9                      2_cp39    conda-forge
readline                  8.1                  hedafd6a_0    conda-forge
requests                  2.27.1             pyhd8ed1ab_0    conda-forge
ruamel_yaml               0.15.80         py39h5161555_1006    conda-forge
setuptools                60.9.3           py39h2804cbe_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sqlite                    3.37.0               h72a2b83_0    conda-forge
tk                        8.6.12               he1e0b03_0    conda-forge
tqdm                      4.62.3             pyhd8ed1ab_0    conda-forge
tzdata                    2021e                he74cb21_0    conda-forge
urllib3                   1.26.8             pyhd8ed1ab_1    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h642e427_1    conda-forge
yaml                      0.2.5                h3422bc3_2    conda-forge
zlib                      1.2.11            hee7b306_1013    conda-forge

Environment info

active environment : SRW
    active env location : /Users/brendan/miniforge3/envs/SRW
            shell level : 2
       user config file : /Users/brendan/.condarc
 populated config files : /Users/brendan/miniforge3/.condarc
                          /Users/brendan/.condarc
                          /Users/brendan/miniforge3/envs/SRW/.condarc
          conda version : 4.11.0
    conda-build version : not installed
         python version : 3.9.10.final.0
       virtual packages : __osx=12.3=0
                          __unix=0=0
                          __archspec=1=arm64
       base environment : /Users/brendan/miniforge3  (writable)
      conda av data dir : /Users/brendan/miniforge3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /Users/brendan/miniforge3/pkgs
                          /Users/brendan/.conda/pkgs
       envs directories : /Users/brendan/miniforge3/envs
                          /Users/brendan/.conda/envs
               platform : osx-arm64
             user-agent : conda/4.11.0 requests/2.27.1 CPython/3.9.10 Darwin/21.4.0 OSX/12.3
                UID:GID : 504:20
             netrc file : None
           offline mode : False
@nadnerbity nadnerbity added the bug Something isn't working label Mar 17, 2022
@nadnerbity
Copy link
Author

I managed to get the program to compile but now python can't find the dynamic libraries. I manually tracked down the libraries with the correct functions in them and had to I use these libraries:

PYPATH=/Users/brendan/miniforge3
PYFLAGS=-I$(PYPATH)/include/python3.9 -I $(PYPATH)/include/python3.9 -L$(PYPATH)/lib -lpython3.9 -ldl

Why doesn't 'python3-config --includes --ldflags' return the correct location for the libraries?

But now when I try to run the program I get the following error:

ImportError: dlopen(/Users/brendan/Documents/SRW/SRW/env/work/srw_python/srwlpy.so, 0x0002): Library not loaded: @rpath/libpython3.9.dylib
  Referenced from: /Users/brendan/Documents/SRW/SRW/env/work/srw_python/srwlpy.so
  Reason: tried: '/usr/local/lib/libpython3.9.dylib' (no such file), '/usr/lib/libpython3.9.dylib' (no such file)

If I start python in the correct conda/miniforge environment and load sys I get the following:

In [2]: import sys

In [3]: sys.path
Out[3]: 
['/usr/local/bin',
 '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip',
 '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8',
 '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload',
 '',
 '/Users/brendan/Library/Python/3.8/lib/python/site-packages',
 '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages',
 '/Library/Python/3.8/site-packages']

Why does the conda/miniforge3 python version not look in the conda/miniforge libraries? Also, python says "'/usr/local/lib/libpython3.9.dylib' (no such file)" but /usr/local/lib/ isn't on the list provided by sys.path. I think I have some variable setup incorrectly, or miniforge isn't setting things up properly, but I'm not sure.

@isuruf
Copy link
Member

isuruf commented Mar 28, 2022

If I start python in the correct conda/miniforge environment and load sys I get the following:

This is using system ipython and not miniforge's python. You need to do python -m pip install ipython in a miniforge env to get an ipython that works on miniforge env.

@nadnerbity
Copy link
Author

nadnerbity commented Mar 28, 2022

D'oh, yup! It is right there when it starts up :/

Any reason I shouldn't use conda install ipython?

If I just run python (instead of ipython) I get the following error:

(SRW) brendan@stuff srw_python % python -i SRWLIB_Example07.py 
zsh: segmentation fault  python -i SRWLIB_Example07.py

Something isn't stirring the Kool-Aid. I discovered I should have used /Users/brendan/miniforge3/envs/SRW as the base directory for libraries, and tried compiling against that, but I get the same segmentation fault.

@isuruf
Copy link
Member

isuruf commented Mar 28, 2022

Any reason I shouldn't use conda install ipython?

Nope. conda install ipython is preferred.

Can't know what the segfault is about. Do you have a backtrace?

@nadnerbity
Copy link
Author

I do not have a backtrace. From quick googling it looks like if python isn't giving me one by default I'd have to edit the source code. I'm out of my depth here though.

This is how I produce the segfault:

(SRW) brendan@stuff srw_python % ipython
Python 3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:27:43) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.2.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: run SRWLIB_Example07.py
zsh: segmentation fault  ipython

@nadnerbity
Copy link
Author

Does this help:

(SRW) brendan@stuff srw_python % lldb
(lldb) file python
Current executable set to 'python' (arm64).
(lldb) run SRWLIB_Example06.py
Process 2617 launched: '/Users/brendan/miniforge3/envs/SRW/bin/python' (arm64)
Process 2617 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x10)
    frame #0: 0x000000010513d7bc libpython3.9.dylib`PyModule_Create2 + 24
libpython3.9.dylib`PyModule_Create2:
->  0x10513d7bc <+24>: ldr    x9, [x9, #0x10]
    0x10513d7c0 <+28>: ldr    x9, [x9, #0x358]
    0x10513d7c4 <+32>: cbz    x9, 0x10513d7d8           ; <+52>
    0x10513d7c8 <+36>: ldp    x29, x30, [sp, #0x20]
Target 0: (python) stopped.
(lldb) 

I think it means there is something wrong with libpython3.9.dylib PyModule_Create2?

@isuruf
Copy link
Member

isuruf commented Mar 28, 2022

can you send the makefile?

@nadnerbity
Copy link
Author

Sure!


SOFT_DEV_DIR=	../
SRW_SRC_DIR=	$(SOFT_DEV_DIR)/src
SRW_SRC_GEN_DIR=	$(SRW_SRC_DIR)/core
SRW_SRC_LIB_DIR=	$(SRW_SRC_DIR)/lib
SRW_SRC_GENESIS_DIR=	$(SRW_SRC_DIR)/ext/genesis/genesis_july08
SH_SRC_PARSE_DIR=	$(SRW_SRC_DIR)/ext/auxparse
SH_SRC_GEN_MATH_DIR=	$(SRW_SRC_DIR)/ext/genmath
LIB_DIR=	$(SOFT_DEV_DIR)/../ext_lib

MODE ?= 0

CC  = gcc
CXX = g++
#CC  = cc
#CXX = c++

SRW_SRC_DEF=	-D_GNU_SOURCE -D__USE_XOPEN2K8 -DFFTW_ENABLE_FLOAT -D_GM_WITHOUT_BASE -DSRWLIB_STATIC -DNO_TIMER -DANSI_DECLARATORS -DTRILIBRARY
CFLAGS=	-O3 -fPIC -I$(SRW_SRC_GEN_DIR) -I$(SRW_SRC_LIB_DIR) -I$(SH_SRC_PARSE_DIR) -I$(SH_SRC_GEN_MATH_DIR) $(SRW_SRC_DEF) 
LDFLAGS=-L$(LIB_DIR) -lm 

ifeq ($(MODE), omp)
CFLAGS+= -D_WITH_OMP -fopenmp -Wno-write-strings 
LDFLAGS+= -lfftw 
else
ifeq ($(MODE), 0)
CFLAGS+= -D_FFTW3 -D__MAC__ -w
LDFLAGS+= -lfftw3f -lfftw3 
else
$(error Unknown SRW compilation option)
endif
endif

PYPATH=/Users/brendan/miniforge3/envs/SRW/
PYFLAGS=-I$(PYPATH)/include/python3.9 -I $(PYPATH)/include/python3.9 -L$(PYPATH)/lib -lpython3.9 -ldl


OBJ=	auxparse.o gmfft.o gmfit.o gminterp.o gmmeth.o gmtrans.o srclcuti.o srcradint.o srctrjdt.o sremitpr.o srgsnbm.o srgtrjdt.o srisosrc.o srmagcnt.o srmagfld.o srmatsta.o sroptapt.o sroptcnt.o sroptdrf.o sroptel2.o sroptel3.o sroptelm.o sroptfoc.o sroptgrat.o sroptgtr.o sropthck.o sroptcryst.o sroptmat.o sroptpsh.o sroptshp.o sroptsmr.o sroptwgr.o sroptzp.o sroptzps.o srpersto.o srpowden.o srprdint.o srprgind.o srpropme.o srptrjdt.o srradinc.o srradint.o srradmnp.o srradstr.o srremflp.o srsase.o srsend.o srstowig.o srsysuti.o srthckbm.o srthckbm2.o srtrjaux.o srtrjdat.o srtrjdat3d.o all_com.o check.o diagno.o esource.o field.o incoherent.o initrun.o input.o loadbeam.o loadrad.o magfield.o main.o math.o mpi.o output.o partsim.o pushp.o rpos.o scan.o source.o stepz.o string.o tdepend.o timerec.o track.o srerror.o srwlib.o 

PRG=	libsrw.a

lib:	$(OBJ)
	ar -cvq $(PRG) *.o
	#cp $(PRG) $(LIB_DIR)/
	rm -f *.o

%.o: $(SRW_SRC_LIB_DIR)/%.cpp
	$(CXX) $(CFLAGS) -c $<  

%.o: $(SH_SRC_PARSE_DIR)/%.cpp
	$(CXX) $(CFLAGS) -c $<  

%.o: $(SH_SRC_GEN_MATH_DIR)/%.cpp
	$(CXX) $(CFLAGS) -c $<  

%.o: $(SRW_SRC_GEN_DIR)/%.cpp
	$(CXX) $(CFLAGS) -c $<  

%.o: $(SRW_SRC_GENESIS_DIR)/%.c
	$(CC) $(CFLAGS) -c $<  

pylib:
	$(CXX) -shared $(CFLAGS) $(PYFLAGS) -o srwlpy.so ../src/clients/python/srwlpy.cpp libsrw.a $(LDFLAGS)

srwlclient: $(SRW_SRC_DIR)/clients/c/srwlclient.cpp
	$(CXX) $(CFLAGS) -O3 -o srwlclient ../src/clients/c/srwlclient.cpp libsrw.a $(LDFLAGS)

clean:
	rm -f *.o *.so *.a srwlclient

all: lib pylib srwlclient

@isuruf
Copy link
Member

isuruf commented Mar 28, 2022

Change PYFLAGS=-I$(PYPATH)/include/python3.9 -I $(PYPATH)/include/python3.9 -L$(PYPATH)/lib -lpython3.9 -ldl
to
PYFLAGS=-I$(PYPATH)/include/python3.9 -I $(PYPATH)/include/python3.9 -L$(PYPATH)/lib -undefined dynamic_lookup -ldl

@nadnerbity
Copy link
Author

I think that made things worse?

(SRW) brendan@stuff srw_python % lldb
(lldb) file python
Current executable set to 'python' (arm64).
(lldb) run SRWLIB_Example06.py
Process 3613 launched: '/Users/brendan/miniforge3/envs/SRW/bin/python' (arm64)
zsh: killed     lldb
(SRW) brendan@stuff srw_python % 

@nadnerbity
Copy link
Author

A similar problem can be found here, but I can't make heads or tails of it.

It seems like the library conda python is using and the library SRW is using are different. It would seem to stem from conda python being linked statically.

@nadnerbity
Copy link
Author

nadnerbity commented Apr 26, 2022

I tried to compile everything in osx-64 to no avail.

  1. I used the advice here to get conda to install x64 stuff

  2. Then I followed the advice here to get an x64 environment for compiling

I get the following error when running a SRW example:
`(lldb) file python
Current executable set to 'python' (x86_64).
(lldb) run SRWLIB_Example07.py
Process 46946 launched: '/Users/brendan/miniforge3/envs/rosetta/bin/python' (x86_64)
Fatal Python error: _PyInterpreterState_Get(): no current thread state
Python runtime state: unknown

Process 46946 stopped

  • thread Create a first version of the builder #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
    frame #0: 0x00007ff81da5300e libsystem_kernel.dylib__pthread_kill + 10 libsystem_kernel.dylib__pthread_kill:
    -> 0x7ff81da5300e <+10>: jae 0x7ff81da53018 ; <+20>
    0x7ff81da53010 <+12>: movq %rax, %rdi
    0x7ff81da53013 <+15>: jmp 0x7ff81da4d1c5 ; cerror_nocancel
    0x7ff81da53018 <+20>: retq
    Target 0: (python) stopped.`

Now that I am typing this out I wonder if I should have set architecture, then installed python using conda. i.e. step 2 then step 1 above.

Edit: The order of steps 1 and 2 doesn't seem to matter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Development

No branches or pull requests

2 participants