I get SyntaxError: invalid syntax Show
while trying to install biopython using the following command.
Could you please help me install biopython in anaconda (3) ?
Nic3500 7,19410 gold badges30 silver badges38 bronze badges asked Aug 31, 2018 at 14:33
4 Package maintainers recommend using (in the terminal):
answered May 13, 2019 at 6:23
DmitriDmitri 7171 gold badge6 silver badges15 bronze badges Open the terminal and export your path to anaconda:
Then type:
answered Feb 24, 2019 at 16:46
Install Anaconda Navigator, go to " answered Aug 19, 2019 at 22:58
M__M__ 6012 gold badges11 silver badges24 bronze badges Not the answer you're looking for? Browse other questions tagged python syntax anaconda biopython or ask your own question.In software, it's said that all abstractions are leaky, and this is true for the Jupyter notebook as it is for any other software. I most often see this manifest itself with the following issue:
This issue is a perrennial source of StackOverflow questions (e.g. this, that, here, there, another, this one, that one, and this... etc.). Fundamentally the problem is usually rooted in the fact that the Jupyter kernels are disconnected from Jupyter's shell; in other words, the installer points to a different Python version than is being used in the notebook. In the simplest contexts this issue does not arise, but when it does, debugging the problem requires knowledge of the intricacies of the operating system, the intricacies of Python package installation, and the intricacies of Jupyter itself. In other words, the Jupyter notebook, like all abstractions, is leaky. In the wake of several discussions on this topic with colleagues, some online (exhibit A, exhibit B) and some off, I decided to treat this issue in depth here. This post will address a couple things:
This post will focus on two approaches to installing Python packages: pip and conda. Other package managers exist (including platform-specific tools like yum, apt, homebrew, etc., as well as cross-platform tools like enstaller), but I'm less familiar with them and won't be remarking on them further. Quick Fix: How To Install Packages from the Jupyter Notebook¶If you're just looking for a quick answer to the question, how do I install packages so they work with the notebook, then look no further. pip vs. conda¶First, a few words on
If you already have a Python installation that you're using, then the choice of which to use is easy:
Finally, because it often comes up, I should mention that you should never use NEVER. It will always lead to problems in the long term, even if it seems
to solve them in the short-term. For example, if How to use Conda from the Jupyter Notebook¶If you're in the jupyter notebook and you want to install a package with conda, you might be tempted to use the In [1]: # DON'T DO THIS! !conda install --yes numpy Fetching package metadata ........... Solving package specifications: . # All requested packages already installed. # packages in environment at /Users/jakevdp/anaconda/envs/python3.6: # numpy 1.13.3 py36h2cdce51_0 (Note that we use For various reasons that I'll outline more fully below, this will not generally work if you want to use these installed packages from the current notebook, though it may work in the simplest cases. Here is a short snippet that should work in general: In [2]: # Install a conda package in the current Jupyter kernel import sys !conda install --yes --prefix {sys.prefix} numpy Fetching package metadata ........... Solving package specifications: . # All requested packages already installed. # packages in environment at /Users/jakevdp/anaconda: # numpy 1.13.3 py36h2cdce51_0 That bit of extra boiler-plate makes certain that conda installs the package in the currently-running Jupyter kernel (thanks to Min Ragan-Kelley for suggesting this approach). I'll discuss why this is needed momentarily. How to use Pip from the Jupyter Notebook¶If you're using the Jupyter notebook and want to install a package with In [3]: # DON'T DO THIS !pip install numpy Requirement already satisfied: numpy in /Users/jakevdp/anaconda/envs/python3.6/lib/python3.6/site-packages For various reasons that I'll outline more fully below, this will not generally work if you want to use these installed packages from the current notebook, though it may work in the simplest cases. Here is a short snippet that should generally work: In [4]: # Install a pip package in the current Jupyter kernel import sys !{sys.executable} -m pip install numpy Requirement already satisfied: numpy in /Users/jakevdp/anaconda/lib/python3.6/site-packages That bit of extra boiler-plate makes certain that you are running the
rather than
because the former is more explicit about where the package will be installed (more on this below). The Details: Why is Installation from Jupyter so Messy?¶Those above solutions should work in all cases... but why is that additional boilerplate necessary? In short, it's because in Jupyter, the shell environment and the Python executable are disconnected. Understanding why that matters depends on a basic understanding of a few different concepts:
For completeness, I'm going to delve briefly into each of these topics (this discussion is partly drawn from This StackOverflow answer that I wrote last year). Note: the following discussion assumes Linux, Unix, MacOSX and similar operating systems. Windows has a slightly different architecture, and so some details will differ. How your operating system locates executables¶When you're using the terminal and type a command like On Linux & Mac systems, the system will first check for an
alias matching the command; if this fails it references the /Users/jakevdp/anaconda/envs/python3.6/bin:/Users/jakevdp/anaconda/envs/python3.6/bin:/Users/jakevdp/anaconda/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
(Parenthetical note: why is the first entry
of If you want to know what is actually executed when you type python is /Users/jakevdp/anaconda/envs/python3.6/bin/python Note that this is true of any command you use from the terminal: Even built-in commands like You can optionally add the python is /Users/jakevdp/anaconda/envs/python3.6/bin/python python is /Users/jakevdp/anaconda/envs/python3.6/bin/python python is /Users/jakevdp/anaconda/bin/python python is /usr/bin/python conda is /Users/jakevdp/anaconda/envs/python3.6/bin/conda conda is /Users/jakevdp/anaconda/envs/python3.6/bin/conda conda is /Users/jakevdp/anaconda/bin/conda pip is /Users/jakevdp/anaconda/envs/python3.6/bin/pip pip is /Users/jakevdp/anaconda/envs/python3.6/bin/pip pip is /Users/jakevdp/anaconda/bin/pip When you have multiple available versions of any command, it is important to keep in mind the role of How Python locates packages¶Python uses a similar mechanism to locate imported packages. The list of paths searched by Python on import is found in Out[12]: ['', '/Users/jakevdp/anaconda/lib/python36.zip', '/Users/jakevdp/anaconda/lib/python3.6', '/Users/jakevdp/anaconda/lib/python3.6/lib-dynload', '/Users/jakevdp/anaconda/lib/python3.6/site-packages', '/Users/jakevdp/anaconda/lib/python3.6/site-packages/schemapi-0.3.0.dev0+791c7f6-py3.6.egg', '/Users/jakevdp/anaconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg', '/Users/jakevdp/anaconda/lib/python3.6/site-packages/IPython/extensions', '/Users/jakevdp/.ipython'] By default, the first place Python looks for a module is an empty path, meaning the
current working directory. If the module is not found there, it goes down the list of locations until the module is found. You can find out which location has been used using the In [13]: import numpy numpy.__path__ Out[13]: ['/Users/jakevdp/anaconda/lib/python3.6/site-packages/numpy'] In most cases, a Python package you install with We can see this by printing the In [14]: paths = !type -a python for path in set(paths): path = path.split()[-1] print(path) !{path} -c "import sys; print(sys.path)" print() /Users/jakevdp/anaconda/envs/python3.6/bin/python ['', '/Users/jakevdp/anaconda/envs/python3.6/lib/python36.zip', '/Users/jakevdp/anaconda/envs/python3.6/lib/python3.6', '/Users/jakevdp/anaconda/envs/python3.6/lib/python3.6/lib-dynload', '/Users/jakevdp/anaconda/envs/python3.6/lib/python3.6/site-packages'] /usr/bin/python ['', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/Library/Python/2.7/site-packages', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC'] /Users/jakevdp/anaconda/bin/python ['', '/Users/jakevdp/anaconda/lib/python36.zip', '/Users/jakevdp/anaconda/lib/python3.6', '/Users/jakevdp/anaconda/lib/python3.6/lib-dynload', '/Users/jakevdp/anaconda/lib/python3.6/site-packages', '/Users/jakevdp/anaconda/lib/python3.6/site-packages/schemapi-0.3.0.dev0+791c7f6-py3.6.egg', '/Users/jakevdp/anaconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg'] The full details here are not particularly important, but it is important to emphasize that each Python executable has its own distinct paths, and unless you modify When you run
So, for example we see that pip is /Users/jakevdp/anaconda/envs/python3.6/bin/pip And # conda environments: # python2.7 /Users/jakevdp/anaconda/envs/python2.7 python3.5 /Users/jakevdp/anaconda/envs/python3.5 python3.6 * /Users/jakevdp/anaconda/envs/python3.6 rstats /Users/jakevdp/anaconda/envs/rstats root /Users/jakevdp/anaconda The reason both I'll say this again for emphasis: the shell environment in Jupyter notebook matches the Python version used to launch the notebook. How Jupyter executes code: Jupyter Kernels¶The next relevant question is how Jupyter chooses to execute Python code, and this brings us to the concept of a Jupyter Kernel. A Jupyter kernel is a set of files that point Jupyter to some means of executing code within the notebook. For Python kernels, this will point to a particular Python version, but Jupyter is designed to be much more general than this: Jupyter has dozens of available kernels for languages including Python 2, Python 3, Julia, R, Ruby, Haskell, and even C++ and Fortran! If you're using the Jupyter notebook, you can change your kernel at any time using the Kernel → Choose Kernel menu item. To see the kernels you have available on your system, you can run the following command in the shell: Available kernels: python3 /Users/jakevdp/anaconda/envs/python3.6/lib/python3.6/site-packages/ipykernel/resources conda-root /Users/jakevdp/Library/Jupyter/kernels/conda-root python2.7 /Users/jakevdp/Library/Jupyter/kernels/python2.7 python3.5 /Users/jakevdp/Library/Jupyter/kernels/python3.5 python3.6 /Users/jakevdp/Library/Jupyter/kernels/python3.6 Each of these listed kernels is a directory that contains a file called In [18]: !cat /Users/jakevdp/Library/Jupyter/kernels/conda-root/kernel.json
{ "argv": [ "/Users/jakevdp/anaconda/bin/python", "-m", "ipykernel_launcher", "-f", "{connection_file}" ], "display_name": "python (conda-root)", "language": "python" } If you'd like to create a new kernel, you can do so using the jupyter ipykernel command; for example, I created the above kernels for my primary conda environments using the following as a template:
The Root of the Issue¶Now we have the full background to answer our question: Why don't The root of the issue is this: the shell environment is determined when the Jupyter notebook is launched, while the Python executable
is determined by the kernel, and the two do not necessarily match. In other words, there is no guarantee that the Recall that the python is /Users/jakevdp/anaconda/envs/python3.6/bin/python The Python executable being used in the notebook can be determined using Out[20]: '/Users/jakevdp/anaconda/bin/python' In my current notebook
environment, the two differ. This is why a simple As noted above, we can get around this by explicitly identifying where we want packages to be installed. For conda, you can set the prefix manually in the shell command:
or, to automatically use the correct prefix (using syntax available in the notebook)
For pip, you can specify the Python executable explicitly:
or, to automatically use the correct executable (again using notebook shell syntax)
Remember: you need your installation command to match the current python kernel if you want installed packages to be available in the notebook. Some Modest Proposals¶So, in summary, the reason that installation of packages in the Jupyter notebook is fraught with difficulty is fundamentally that Jupyter's shell environment and Python kernel are mismatched, and that means that you have to do more than simply But that leaves us in an undesireable place, as it increases the learning curve for novice users who may want to do something they (rightly) presume should be simple: install a package and then use it. So what can we as a community do to smooth-out this issue? I have a few ideas, some of which might even be useful:
Potential Changes to Jupyter¶As I mentioned, the fundamental issue is a mismatch between Jupyter's shell environment and compute kernel. So, could we massage kernel specifications such that they force the two to match? Perhaps: for example, this github issue shows an approach to modifying shell variables as part of kernel startup. Basically, in your kernel directory, you can add a script
Then in your
Once you do this, switching to the There is one tricky issue here: this approach will fail if your Potential Changes to pip¶One source of installation confusion, even outside of
Jupyter, is the fact that, depending on the nature of your system's aliases and This is one reason that
Changes to Conda¶I can think of a couple modifications to conda's API that may be helpful to users Explicit invocation¶For symmetry with
I suspect that allowing A pip channel for conda?¶Another useful change conda could make would be to add a channel that essentially mirrors the Python Package Index, so that when you do I don't have a deep enough knowledge of conda's architecture to know how easy such a feature would be to implement, but I do have loads of experiences helping newcomers to Python and/or conda: I can say with certainty that such a feature would go a long way toward softening their learning curve. New Jupyter Magic Functions¶Even if the above changes to
the stack are not possible or desirable, we could simplify the user experience somewhat by introducing pip magic¶For example, here's how you can define a In [21]: from IPython.core.magic import register_line_magic @register_line_magic def pip(args): """Use pip from the current kernel""" from pip import main main(args.split()) Running it as follows will install packages in the expected location Requirement already satisfied: numpy in /Users/jakevdp/anaconda/lib/python3.6/site-packages Note that Jupyter developer Matthias Bussonnier has published essentially this in his pip_magic repository, so you can do
and use this right now (that is, assuming you install conda magic¶Similarly, we can define a conda magic that will do the right thing if you type In [23]: from IPython.core.magic import register_line_magic import sys import os from subprocess import Popen, PIPE def is_conda_environment(): """Return True if the current Python executable is in a conda env""" # TODO: make this work with Conda.exe in Windows conda_exec = os.path.join(os.path.dirname(sys.executable), 'conda') conda_history = os.path.join(sys.prefix, 'conda-meta', 'history') return os.path.exists(conda_exec) and os.path.exists(conda_history) @register_line_magic def conda(args): """Use conda from the current kernel""" # TODO: make this work with Conda.exe in Windows # TODO: fix string encoding to work with Python 2 if not is_conda_environment(): raise ValueError("The python kernel does not appear to be a conda environment. " "Please use ``%pip install`` instead.") conda_executable = os.path.join(os.path.dirname(sys.executable), 'conda') args = [conda_executable] + args.split() # Add --prefix to point conda installation to the current environment if args[1] in ['install', 'update', 'upgrade', 'remove', 'uninstall', 'list']: if '-p' not in args and '--prefix' not in args: args.insert(2, '--prefix') args.insert(3, sys.prefix) # Because the notebook does not allow us to respond "yes" during the # installation, we need to insert --yes in the argument list for some commands if args[1] in ['install', 'update', 'upgrade', 'remove', 'uninstall', 'create']: if '-y' not in args and '--yes' not in args: args.insert(2, '--yes') # Call conda from command line with subprocess & send results to stdout & stderr with Popen(args, stdout=PIPE, stderr=PIPE) as process: # Read stdout character by character, as it includes real-time progress updates for c in iter(lambda: process.stdout.read(1), b''): sys.stdout.write(c.decode(sys.stdout.encoding)) # Read stderr line by line, because real-time does not matter for line in iter(process.stderr.readline, b''): sys.stderr.write(line.decode(sys.stderr.encoding)) You can now use Fetching package metadata ........... Solving package specifications: . # All requested packages already installed. # packages in environment at /Users/jakevdp/anaconda: # numpy 1.13.3 py36h2cdce51_0 This conda magic still needs some work to be a general solution (cf. the TODO comments in the code), but I think this is a useful start. If a pip magic and conda magic similar to the above were added to Jupyter's default set of magic commands, I think it could go a long way toward solving the common problems that users have when trying to install Python packages for use with Jupyter notebooks. This approach is not without its own dangers, though: these magics are yet another layer of abstraction that, like all abstractions, will inevitably leak. But if they are implemented carefully, I think it would lead to a much nicer overall user experience. Summary¶In this post, I tried to answer once and for all the perennial question, how do I install Python packages in the Jupyter notebook. After proposing some simple solutions that can be used today, I went into a detailed explanation of why these solutions are necessary: it comes down to the fact that in Jupyter, the kernel is disconnected from the shell. The kernel environment can be changed at runtime, while the shell environment is determined when the notebook is launched. The fact that a full explanation took so many words and touched so many concepts, I think, indicates a real usability issue for the Jupyter ecosystem, and so I proposed a few possible avenues that the community might adopt to try to streamline the experience for users. One final addendum: I have a huge amount of respect and appreciation for the developers of Jupyter, conda, pip, and related tools that form the foundations of the Python data science ecosystem. I'm fairly certain those developers have already considered these issues and weighed some of these potential fixes – if any of you are reading this, please feel free to comment and set me straight on anything I've overlooked! And, finally, thanks for all that you do for the open source community. Thanks to Andy Mueller, Craig Citro, and Matthias Bussonnier for helpful comments on an early draft of this post. This post was written within a Jupyter notebook; you can view a static version here or download the full notebook here. |