Jupyter[Lab] Language Server Protocol¶
This is the documentation for:
Both are managed on GitHub, where you can find the issue tracker.
Installation¶
Please Read This First¶
Delivering LSP features to your JupyterLab requires three pieces:
jupyter-lsp
¶
runs in your
notebook
web application on your server to handle requests from the browser to language serversto run, you need:
python >=3.5
notebook >=4.3
jupyterlab-lsp
¶
runs in your browser, as an extension to JupyterLab
to install it, you need:
nodejs >8
jupyterlab >=1.1,<2
Language Servers¶
run on your server
probably in another language runtime than python
some can be automatically detected if installed
others also need to be configured
Fast Paths¶
Here are two approches based on Jupyter documentation. If these do not meet your needs, try The Harder Way.
conda (minimal python)¶
conda create -c conda-forge -n lsp 'python >=3.7,<3.8' 'jupyterlab=1.2' 'nodejs>8' python-language-server
# Also consider: r-languageserver [*]
source activate lsp
python -m pip install 'jupyter-lsp=0.8.0' --no-deps
jupyter labextension install '@krassowski/jupyterlab-lsp@0.8.0'
Then run
jupyter lab
Your browser should open to your local server.
docker (data science)¶
This approach is based roughly on the Jupyter docker-stacks documentation, which should be consulted for more about connecting volumes, passwords, and other advanced features:
Dockerfile
¶
# This already contains the python, r, julia, and nodejs runtimes as well as jupyterlab 1.25
FROM jupyter/datascience-notebook@sha256:73a577b006b496e1a1c02f5be432f4aab969c456881c4789e0df77c89a0a60c2
RUN conda install --quiet --yes --freeze-installed \
'python-language-server' \
'r-languageserver' \
&& python3 -m pip install --no-cache-dir --no-deps \
'jupyter-lsp=0.8.0' \
&& jupyter labextension install --no-build \
'@krassowski/jupyterlab-lsp@0.8.0' \
&& jupyter lab build --dev-build=False --minimize=True \
&& conda clean --all -f -y \
&& rm -rf \
$CONDA_DIR/share/jupyter/lab/staging \
/home/$NB_USER/.cache/yarn \
&& fix-permissions $CONDA_DIR \
&& fix-permissions /home/$NB_USER
docker-compose.yml
¶
version: '2'
services:
lsp-lab:
build: .
ports:
- '18888:8888'
Build and Start¶
docker-compose up
You should now be able to access http://localhost:18888/lab
, using the token
provided in the log.
The Harder Way¶
Get A Working JupyterLab environment¶
Refer to the official JupyterLab Installation Documentation for your installation approach.
pip |
conda |
pipenv |
poetry |
|
---|---|---|---|---|
|
|
|
*
PRs welcome!
Verify your lab works:
jupyter lab --version
jupyter lab
Get a Working NodeJS¶
The JupyterLab Development Environment Documentation shows some approaches for getting NodeJS.
conda |
|
---|---|
|
Verify your node works and is findable from python.
jlpm versions
Install Jupyter[Lab] LSP¶
pip install jupyter-lsp=0.8.0
jupyter labextension install @krassowski/jupyterlab-lsp@0.8.0
Next Step: Language Servers¶
Now that you have jupyterlab-lsp
, jupyter-lsp
and all of their dependencies, you’ll need some language servers. See:
Language Servers that will be found automatically once installed
configuring
jupyter-lsp
for more control over which servers to load
Language Servers¶
jupyter-lsp
does not come with any Language Servers! However, we will try to use them if they are installed and we know about them. For the language servers in the tables below, use one of the suggested package managers to install them: these implementations are tested to work with jupyter-lsp
.
You can disable this feature by configuring autodetect
If you do not see a language you would like, but can find it one of these lists:
the official list of language servers
a community-curated list of language servers
…you might be able to add it via configuration or build your own spec for the server in question.
Notebook-Optimized Language Servers¶
These servers have well-tested support for notebooks and file editors.
[5]:
Languages | Implementation | Installation |
---|---|---|
python |
pyls |
|
r |
r-languageserver |
|
NodeJS-based Language Servers¶
These servers have mostly been tested with file editors.
[6]:
Languages | Implementation | Installation |
---|---|---|
bash sh |
bash-language-server |
|
dockerfile |
dockerfile-language-server-nodejs |
|
javascript jsx typescript typescript-jsx typescriptreact javascriptreact |
javascript-typescript-langserver |
|
markdown ipythongfm gfm |
unified-language-server |
|
css less scss |
vscode-css-languageserver-bin |
|
html |
vscode-html-languageserver-bin |
|
json |
vscode-json-languageserver-bin |
|
yaml |
yaml-language-server |
|
Example: Getting All the NodeJS-based Language Servers¶
A number of language servers are built on the reference implementation, powered by NodeJS. The most reliable place to install these is in a node_modules
in the directory where you launch jupyter lab
.
For example, to install all the servers which are tested as part of jupyterlab-lsp
:
jlpm add --dev \
bash-language-server \
vscode-css-languageserver-bin \
dockerfile-language-server-nodejs \
vscode-html-languageserver-bin \
javascript-typescript-langserver \
vscode-json-languageserver-bin \
yaml-language-server
This will create create (or add to):
package.json
(check this in!)yarn.lock
(check this in!)node_modules/
(add to your VCS ignore file)
If you wish to install these someplace else, you may need to specify where you install them with extra_node_roots.
Configuring¶
Configuration Files¶
Like the Jupyter Notebook server, JupyterHub, and other Jupyter interactive computing tools, jupyter-lsp
can be configured via Python or JSON files in well-known locations. You can find out where to put them on your system with:
jupyter --paths
They will be merged from bottom to top, and the directory where you launch your notebook
server wins, making it easy to check in to version control.
Configuration Options¶
language_servers¶
jupyter-lsp
does not come with any Language Servers! However, we will try to use known language servers if they are installed and we know about them: you can disable this behavior by configuring autodetect.
If you don’t see an implementation for the language server you need, continue reading!
Please consider contributing your language server spec to
jupyter-lsp
!
The absolute minimum language server spec requires:
argv
, a list of shell tokens to launch the server instdio
mode (as opposed totcp
),the
languages
which the server will respond to, andthe schema
version
of the spec (currently only1
)
# ./jupyter_notebook_config.json ---------- unique! -----------
# | |
# or e.g. V V
# $PREFIX/etc/jupyter/jupyter_notebook_config.d/a-language-server-implementation.json
{
"LanguageServerManager": {
"language_servers": {
"a-language-server-implementation": {
"version": 1,
"argv": ["/absolute/path/to/a-language-server", "--stdio"],
"languages": ["a-language"]
}
}
}
}
A number of other options we hope to use to enrich the user experience are available in the schema.
More complex configurations that can’t be hard-coded may benefit from the python approach:
# jupyter_notebook_config.py
import shutil
# c is a magic, lazy variable
c.LanguageServerManager.language_servers = {
"a-language-server-implementation": {
# if installed as a binary
"argv": [shutil.which("a-language-server")],
"languages": ["a-language"]
},
"another-language-implementation": {
# if run like a script
"argv": [shutil.which("another-language-interpreter"), "another-language-server"],
"languages": ["another-language"]
}
}
nodejs¶
default:
None
An absolute path to your nodejs
executable. If None
, nodejs
will be detected in a number of well-known places.
autodetect¶
default:
True
If True
, jupyter-lsp
will look for all known language servers. User-configured language_servers
of the same implementation will be preferred over autodetect
ed ones.
node_roots¶
default:
[]
Absolute paths to search for directories named node_modules
, such as nodejs
-backed language servers. The order is, roughly:
the folder where
notebook
orlab
was launchedthe JupyterLab
staging
folderwherever
conda
puts global node moduleswherever some other conventions put it
extra_node_roots¶
default:
[]
Additional places jupyter-lsp
will look for node_modules
. These will be checked before node_roots
, and should not contain the trailing node_modules
.
Python entry_points
¶
pip
-installable packages in the same environment as the Jupyter notebook
server can be automatically detected as providing language_servers. These are a little more involved, but also more powerful: see more in Contributing. Servers configured this way are loaded before those defined in configuration files, so that a user can fine-tune their available servers.
Contributing¶
[1]:
jupyter-lsp
and jupyterlab-lsp
are open source software, and all contributions conforming to good sense, good taste, and the Jupyter Code of Conduct are welcome, and will be reviewed by the contributors, time-permitting.
You can contribute to the project through:
creating language server specs
you can publish them yourself (it might be a single file)…
or advocate for adding your spec to the github repository and its various distributions
these are great first issues, as you might not need to know any python or javascript
proposing parts of the architecture that can be extended
improving documentation
tackling Big Issues from the future roadmap
improving testing
reviewing pull requests
Set up the environment¶
Development requires:
nodejs
8 or laterpython
3.5+jupyterlab
1.1
It is recommended to use a virtual environment (e.g. virtualenv
or conda env
) for development.
conda env update -n jupyterlab-lsp # create a conda env
source activate jupyterlab-lsp # activate it
# or...
pip install -r requirements/dev.txt # in a virtualenv, probably
# ... and install nodejs, somehow
The Easy Way¶
Once your environment is created and activated, on Linux/OSX you can run:
bash postBuild
This performs all of the basic setup steps, and is used for the binder demo.
The Hard Way¶
Install jupyter-lsp
from source in your virtual environment:
python -m pip install -e .
Enable the server extension:
jupyter serverextension enable --sys-prefix --py jupyter_lsp
Install npm
dependencies, build TypeScript packages, and link to JupyterLab for development:
jlpm
jlpm build
jlpm lab:link
Frontend Development¶
To rebuild the schemas, packages, and the JupyterLab app:
jlpm build
jupyter lab build
To watch the files and build continuously:
jlpm watch # leave this running...
jupyter lab --watch # ...in another terminal
Note: the backend schema is not included in ``watch``, and is only refreshed by ``build``
To check and fix code style:
jlpm lint
To run test the suite (after running jlpm build
or watch
):
jlpm test
To run tests matching specific phrase, forward -t
argument over yarn and lerna to the test runners with two --
:
jlpm test -- -- -t match_phrase
Documentation¶
To build the documentation:
python scripts/docs.py
To watch documentation sources and build continuously:
python scripts/docs.py --watch
To check internal links in the docs after building:
python scripts/docs.py --check --local-only
To check internal and external links in the docs after building:
python scripts/docs.py --check
Note: you may get spurious failures due to rate limiting, especially in CI, but it's good to test locally
Browser-based Acceptance Tests¶
The browser tests will launch JupyterLab on a random port and exercise the Language Server features with Robot Framework and SeleniumLibrary. It is recommended to peruse the Robot Framework User’s Guide (and the existing .robot
files in atest
) before working on tests in anger.
First, ensure you’ve prepared JupyterLab for jupyterlab-lsp
frontend and server development.
Prepare the enviroment:
conda env update -n jupyterlab-lsp --file requirements/atest.yml
# or
pip install -r requirements/atest.txt # ... and install geckodriver, somehow
apt-get install firefox-geckodriver # ... e.g. on debian/ubuntu
Run the tests:
python scripts/atest.py
The Robot Framework reports and screenshots will be in atest/output
, with <operating system>_<python version>_<attempt>.<log|report>.html
and subsequent screenshots
being the most interesting artifact, e.g.
atest/
output/
linux_37_1.log.html
linux_37_1.report.html
linux_37_1/
screenshots/
Customizing the Acceptance Test Run¶
By default, all of the tests will be run, once.
The underlying robot
command supports a vast number of options and many support wildcards (*
and ?
) and boolean operators (NOT
, OR
). For more, start with simple patterns.
Run a suite¶
python scripts/atest.py --suite "05_Features.Completion"
Run a single test¶
python scripts/atest.py --test "Works With Kernel Running"
Run test with a tag¶
Tags are preferrable to file names and test name matching in many settings, as they are aggregated nicely between runs.
python scripts/atest.py --include feature:completion
… or only Python completion
python scripts/atest.py --include feature:completionANDlanguage:python
Just Keep Testing with ATEST_RETRIES
¶
Run tests, and rerun only failed tests up to two times:
ATEST_RETRIES=2 python scripts/atest.py --include feature:completion
After running a bunch of tests, it may be helpful to combine them back together into a single log.html
and report.html
with rebot. Like atest.py
, combine.py
also passes through extra arguments
python scripts/combine.py
Troubleshooting¶
If you see the following error message:
python Parent suite setup failed: TypeError: expected str, bytes or os.PathLike object, not NoneType
it may indicate that you have no firefox
, or geckodriver
installed (or discoverable in the search path).
If a test suite for a specific language fails it may indicate that you have no appropriate server language installed (see LANGUAGESERVERS)
If you are seeing errors like
Element is blocked by .jp-Dialog
, caused by the JupyterLab Build suggested dialog, (likely if you have been usingjlpm watch
) ensure you have a “clean” lab (with production assets) with:
bash jupyter lab clean jlpm build jlpm lab:link jupyter lab build --dev-build=False --minimize=True
and re-run the tests.
To display logs on the screenshots, write logs with
virtual_editor.console.log
method, and changecreate_console('browser')
tocreate_console('floating')
inVirtualEditor
constructor (please feel free to add a config option for this).
Formatting¶
Minimal code style is enforced with pytest-flake8
during unit testing. If installed, pytest-black
and pytest-isort
can help find potential problems, and lead to cleaner commits, but are not enforced during CI tests (but are checked during lint).
You can clean up your code, and check for using the project’s style guide with:
python scripts/lint.py
Specs¶
It is convenient to collect common patterns for connecting to installed language servers as pip
-installable packages that Just Work with jupyter-lsp
.
If an advanced user installs, locates, and configures, their own language server it will always win vs an auto-configured one.
Writing a spec¶
A spec is a python function that accepts a single argument, the LanguageServerManager
, and returns a dictionary of the form:
{
"python-language-server": { # the name of the implementation
"version": 1, # the version of the spec schema
"argv": ["python", "-m", "pyls"], # a list of command line arguments
"languages": ["python"] # a list of languages it supports
}
}
The absolute minimum listing requires argv
(a list of shell tokens to launch the server) and languages
(which languages to respond to), but many number of other options to enrich the user experience are available in the schema and are exercised by the current entry_points
-based specs.
The spec should only be advertised if the command could actually be run:
its runtime (e.g.
julia
,nodejs
,python
,r
,ruby
) is installedthe language server itself is installed (e.g.
python-language-server
)
Common Concerns¶
some language servers need to have their connection mode specified
the
stdio
interface is the only one supported byjupyter_lsp
PRs welcome to support other modes!
because of its VSCode heritage, many language servers use
nodejs
LanguageServerManager.nodejs
will provide the location of our best guess at where a user’snodejs
might be foundsome language servers are hard to start purely from the command line
use a helper script to encapsulate some complexity.
See the r spec for an example
Example: making a pip-installable cool-language-server
spec¶
Consider the following (absolutely minimal) directory structure:
- setup.py
- jupyter_lsp_my_cool_language_server.py
You should consider adding a LICENSE, some documentation, etc.
Define your spec:
# jupyter_lsp_my_cool_language_server.py
from shutil import which
def cool(app):
cool_language_server = shutil.which("cool-language-server")
if not cool_language_server:
return {}
return {
"cool-language-server": {
"version": 1,
"argv": [cool_language_server],
"languages": ["cool"]
}
}
Tell pip
how to package your spec:
# setup.py
import setuptools
setuptools.setup(
name="jupyter-lsp-my-cool-language-server",
py_modules=["jupyter_lsp_my_cool_language_server"],
entry_points={
"jupyter_lsp_spec_v1": [
"cool-language-server":
"jupyter_lsp_my_cool_language_server:cool"
]
}
)
Test it!
python -m pip install -e .
Build it!
python setup.py sdist bdist_wheel
Extend jupyterlab-lsp and jupyter-lsp¶
jupyterlab-lsp¶
At present,
jupyterlab-lsp
is still in very early development, and does not expose any runtime extension points. The roadmap lists several potential points of extension, but will require some refactoring to achieve.
jupyter-lsp¶
Language Server Specs¶
Language Server Specs can be configured by Jupyter users, or distributed by third parties as python or JSON files. Since we’d like to see as many Language Servers work out of the box as possible, consider contributing a spec, if it works well for you!
Message Listeners¶
Message listeners may choose to receive LSP messages immediately after being received from the client (e.g. jupyterlab-lsp
) or a language server. All listeners of a message are scheduled concurrently, and the message is passed along once all listeners return (or fail). This allows listeners to, for example, modify files on disk before the language server reads them.
If a listener is going to perform an expensive activity that shouldn’t block delivery of a message, a non-blocking technique like IOLoop.add_callback and/or a queue should be used.
Add a Listener with entry_points
¶
Listeners can be added via entry_points by a package installed in the same environment as notebook
:
## setup.cfg
[options.entry_points]
jupyter_lsp_listener_all_v1 =
some-unique-name = some.module:some_function
jupyter_lsp_listener_client_v1 =
some-other-unique-name = some.module:some_other_function
jupyter_lsp_listener_server_v1 =
yet-another-unique-name = some.module:yet_another_function
At present, the entry point names generally have no impact on functionality aside from logging in the event of an error on import.
Add a Listener with Jupyter Configuration¶
Listeners can be added via traitlets
configuration, e.g.
## jupyter_notebook_config.jsons
{
'LanguageServerManager':
{
'all_listeners': ['some.module.some_function'],
'client_listeners': ['some.module.some_other_function'],
'server_listeners': ['some.module.yet_another_function'],
},
}
Add a listener with the Python API¶
lsp_message_listener
can be used as a decorator, accessed as part of a serverextension
.
This listener receives all messages from the client and server, and prints them out.
from jupyter_lsp import lsp_message_listener
def load_jupyter_server_extension(nbapp):
@lsp_message_listener("all")
async def my_listener(scope, message, language_server, manager):
print("received a {} {} message from {}".format(
scope, message["method"], language_server
))
scope
is one of client
, server
or all
, and is required.
Listener options¶
Fine-grained controls are available as part of the Python API. Pass these as named arguments to lsp_message_listener
.
language_server
: a regular expression of language serversmethod
: a regular expression of LSP JSON-RPC method names
Releasing¶
jupyterlab-lsp
and jupyter-lsp
releases may require building both the python package and nodejs packages.
Updating Version Strings¶
Check the version strings across the various files:
python scripts/integrity.py
TODO: create a
release.py
script #88
The PyPI version must be updated in the following places:
py_src/jupyter_lsp/_version.py
(canonical)azure-pipelines.yml
CHANGELOG.md
The npm version must be updated in the following places
packages/jupyterlab-lsp/package.json
(canonical)azure-pipelines.yml
packages/metapackage/package.json
CHANGELOG.md
CHANGELOG¶
[1]:
@krassowski/jupyterlab-lsp 0.8.0
(2020-03-12)¶
features
opens a maximum of one WebSocket per language server (#165, #199)
lazy-loads language server protocol machinery (#165)
waits much longer for slow-starting language servers (#165)
cleans up documents, handlers, events, and signals more aggressively (#165)
ignores malformed diagnostic ranges, enabling markdown support (#165)
passes tests on Python 3.8 on Windows (#165)
improves support for rpy2 magic cells with parameters ( #206 )
bug fixes
reports files are open only after installing all handlers to avoid missing messages (#201)
lsp-ws-connection 0.4.0
(2020-03-12)¶
jupyter-lsp 0.8.0
(2020-03-12)¶
@krassowski/jupyterlab-lsp 0.7.1
¶
features
users can now choose which columns to display in the diagnostic panel, using a context menu action ( #159 )
start the diagnostics panel docked at the bottom and improve the re-spawning of the diagnostics panel ( #166 )
bugfixes
fixed various small bugs in the completer ( #162 )
fix documentation display in signature for LSP servers which return MarkupContent ( #164 )
lsp-ws-connection 0.3.1
¶
@krassowski/jupyterlab-lsp 0.7.0
¶
features
reduced space taken up by the statusbar indicator ( #106 )
implemented statusbar popover with connections statuses ( #106 )
generates types for server data responses from JSON schema ( #110 )
added ‘rename’ function for notebooks, using shadow filesystem ( #115 )
added a UX workaround for rope rename issues when there is a SyntaxError in the Python code ( #127 )
added a widget panel with diagnostics (inspections), allowing to sort and explore diagnostics, and to go to the respective location in code (with a click); accessible from the context menu ( #129 )
all commands are now accessible from the command palette ( #142 )
bash LSP now also covers
%%bash
magic cell in addition to%%sh
( #144 )rpy2 magics received enhanced support for argument parsing in both parent Python document (re-written overrides) and exctracted R documents (improved foreign code extractor) ( #148, #153 )
console logs can now easily be redirected to a floating console windows for debugging of the browser tests (see CONTRIBUTING.md)
bugfixes
diagnostics in foreign documents are now correctly updated ( 133fd3d )
diagnostics are now always correctly displayed in the document they were intended for
the workaround for relative root path is now also applied on Mac ( #139 )
fixed LSP of R in Python (
%%R
magic cell from rpy2) ( #144 )completion now work properly when the kernel is shut down ( #146 )
a lowercase completion option selected from an uppercase token will now correctly substitute the incomplete token ( #143 )
didSave()
is emitted on file save, enabling the workaround used by R language server to lazily loadlibrary(tidyverse)
( #95, #147, )signature feature is now correctly working in notebooks ( #140 )
lsp-ws-connection 0.3.0
¶
jupyter-lsp 0.7.0b0
¶
@krassowski/jupyterlab-lsp 0.6.1
¶
features
adds an indicator to the statusbar
and many other improvements, see the release notes
dependencies
removes unused npm dependencies
@krassowski/jupyterlab-lsp 0.6.0
¶
features
allows “rename” action in file editor
bugfixes
handles some non-standard diagnostic responses
testing
adds browser-based testing for file editor
dependencies
requires
jupyter-lsp
jupyter-lsp 0.6.0b0
¶
features
starts language servers on demand
accepts configuration via Jupyter config system (traitlets) and python
entry_point
sautodetects language servers for bash, CSS, LESS, SASS, Dockerfile, YAML, JS, TypeScript, JSX, TSX, JSON, YAML
Roadmap¶
If a feature you would like is not on the lists above, please feel free to suggest it by opening a new issue.
Front End¶
improved code navigation when there are multiple jump targets
autocompleter with documentation and sorting based on LSP suggestions
system of settings, including options:
to enable aggressive autocompletion (like in hinterland)
to change the verbosity of signature hints (whether to show documentation, number of lines to be shown)
custom foreign extractors allowing to customize behaviour for magics
code actions (allowing to “quick fix” a typo, etc.)
gutter with linter results
use the kernel session for autocompletion in FileEditor if available (PR welcome)