https://delta.io logo
b

bharat chaudhury

06/16/2023, 9:49 AM
hi All , I have spark 3.3.0 trying to use delta lake 2.3.0 (as its compatible with spark 3.3) Because of not having python >=3.7 I created a virtual env of python 3.9 But when trying to install delta-spark== 2.3.0 I get the below error :
/home/bchaudhu/python3_9_env/bin/python3.9  -m pip install delta-spark==2.3.0
Copy code
Collecting delta-spark==2.3.0
  Using cached delta_spark-2.3.0-py3-none-any.whl (20 kB)
Collecting pyspark<3.4.0,>=3.3.0 (from delta-spark==2.3.0)
  Using cached pyspark-3.3.2.tar.gz (281.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  Ã Preparing metadata (pyproject.toml) did not run successfully.
  â exit code: 1
  â°â> [80 lines of output]
      /tmp/pip-build-env-d4kbutzi/overlay/lib/python3.9/site-packages/setuptools/dist.py:755: SetuptoolsDeprecationWarning: Invalid dash-separated options
      !!

              ********************************************************************************
              Usage of dash-separated 'description-file' will not be supported in future
              versions. Please use the underscore name 'description_file' instead.

              By 2023-Sep-26, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.

              See <https://setuptools.pypa.io/en/latest/userguide/declarative_config.html> for details.
              ********************************************************************************

      !!
        opt = self.warn_dash_deprecation(opt, section)
      running dist_info
      creating /tmp/pip-modern-metadata-axjhbnal/pyspark.egg-info
      writing /tmp/pip-modern-metadata-axjhbnal/pyspark.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-modern-metadata-axjhbnal/pyspark.egg-info/dependency_links.txt
      writing requirements to /tmp/pip-modern-metadata-axjhbnal/pyspark.egg-info/requires.txt
      writing top-level names to /tmp/pip-modern-metadata-axjhbnal/pyspark.egg-info/top_level.txt
      writing manifest file '/tmp/pip-modern-metadata-axjhbnal/pyspark.egg-info/SOURCES.txt'
      reading manifest file '/tmp/pip-modern-metadata-axjhbnal/pyspark.egg-info/SOURCES.txt'
      reading manifest template '<http://MANIFEST.in|MANIFEST.in>'
      warning: no previously-included files matching '*.py[cod]' found anywhere in distribution
      warning: no previously-included files matching '__pycache__' found anywhere in distribution
      warning: no previously-included files matching '.DS_Store' found anywhere in distribution
      writing manifest file '/tmp/pip-modern-metadata-axjhbnal/pyspark.egg-info/SOURCES.txt'
      creating '/tmp/pip-modern-metadata-axjhbnal/pyspark-3.3.2.dist-info'
      Traceback (most recent call last):
        File "/home/bchaudhu/python3_9_env/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/bchaudhu/python3_9_env/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/bchaudhu/python3_9_env/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
        File "/tmp/pip-build-env-d4kbutzi/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 380, in prepare_metadata_for_build_wheel
          self.run_setup()
        File "/tmp/pip-build-env-d4kbutzi/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 487, in run_setup
          super(_BuildMetaLegacyBackend,
        File "/tmp/pip-build-env-d4kbutzi/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 338, in run_setup
          exec(code, locals())
        File "<string>", line 199, in <module>
        File "/tmp/pip-build-env-d4kbutzi/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 107, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-d4kbutzi/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-d4kbutzi/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-d4kbutzi/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-d4kbutzi/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1244, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-d4kbutzi/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-d4kbutzi/overlay/lib/python3.9/site-packages/setuptools/command/dist_info.py", line 104, in run
          bdist_wheel = self.get_finalized_command('bdist_wheel')
        File "/tmp/pip-build-env-d4kbutzi/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 304, in get_finalized_command
          cmd_obj = self.distribution.get_command_obj(command, create)
        File "/tmp/pip-build-env-d4kbutzi/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 860, in get_command_obj
          klass = self.get_command_class(command)
        File "/tmp/pip-build-env-d4kbutzi/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 989, in get_command_class
          self.cmdclass[command] = cmdclass = ep.load()
        File "/tmp/pip-build-env-d4kbutzi/overlay/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 208, in load
          module = import_module(match.group('module'))
        File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
        File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
        File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
        File "<frozen importlib._bootstrap_external>", line 850, in exec_module
        File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
        File "/tmp/pip-build-env-d4kbutzi/overlay/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 28, in <module>
          from .macosx_libfile import calculate_macosx_platform_tag
        File "/tmp/pip-build-env-d4kbutzi/overlay/lib/python3.9/site-packages/wheel/macosx_libfile.py", line 43, in <module>
          import ctypes
        File "/usr/local/lib/python3.9/ctypes/__init__.py", line 8, in <module>
          from _ctypes import Union, Structure, Array
      ModuleNotFoundError: No module named '_ctypes'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

à Encountered error while generating package metadata.
â°â> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
2 Views