Codecov Report

Merging #868 into master will increase coverage by 0.11%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #868      +/-   ##
==========================================
+ Coverage   83.31%   83.42%   +0.11%     
==========================================
  Files          27       27              
  Lines        3595     3614      +19     
==========================================
+ Hits         2995     3015      +20     
+ Misses        600      599       -1     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a8a5f81...ef837b3. Read the comment docs.

Codecov Report

Merging #869 into master will increase coverage by 0.23%.
The diff coverage is 90.62%.

@@            Coverage Diff             @@
##           master     #869      +/-   ##
==========================================
+ Coverage   82.99%   83.23%   +0.23%     
==========================================
  Files          26       27       +1     
  Lines        3547     3609      +62     
==========================================
+ Hits         2944     3004      +60     
- Misses        603      605       +2     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1bb33da...9e693a7. Read the comment docs.

Codecov Report

Merging #866 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #866   +/-   ##
=======================================
  Coverage   82.99%   82.99%           
=======================================
  Files          26       26           
  Lines        3547     3547           
=======================================
  Hits         2944     2944           
  Misses        603      603           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1a5b7d3...fb64dda. Read the comment docs.

But also want to say thanks for a great tool

+1!

This seems to be a duplicate of: https://github.com/simonw/datasette/issues/838

@simonw

Did you find the reason? I had similar situation and I check this on millions ways. I am sure app doesn't consume such memory.

I was trying the app with:
docker run --rm -it -p 80:80 -m 128M foo

I was watching app with docker stats. Even limited memory by CMD ["java", "-Xms60M", "-Xmx60M", "-jar", "api.jar"].
Checked memory usage by app in code and print bash commands. The app definitely doesn't use this memory. Also doesn't write files.

Only one solution is to change memory to 512M.

It is definitely something wrong with cloud run.

I even did special app for testing this. It looks like when I cross very small amount of code / memory / app size in random when, then memory needs grow +hundreds. Nothing make sense here. Especially it works everywhere expect cloud run.

Please let me know if you discovered something more.

I also am seeing the same issue with an Apache setup (same even w/o ProxyPassReverse, though I typically use it as @tsibley stated).

But also want to say thanks for a great tool (this issue not withstanding)!

It's the one with python3.7::

>>> sqlite3.sqlite_version
'3.11.0'

On Wed, Jun 17, 2020 at 10:24 -0700, Simon Willison wrote:

That means your version of SQLite is old enough that it doesn't support the FTS5 extension.

Could you share what operating system you're running, and what the output is that you get from running this?

python -c 'import sqlite3; print(sqlite3.connect(":memory:").execute("select sqlite_version()").fetchone()[0])'

I can teach this tool to fall back on FTS4 if FTS5 isn't available.

--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645512127

Codecov Report

Merging #848 into master will decrease coverage by 0.04%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master     #848      +/-   ##
==========================================
- Coverage   82.87%   82.82%   -0.05%     
==========================================
  Files          26       26              
  Lines        3538     3540       +2     
==========================================
  Hits         2932     2932              
- Misses        606      608       +2     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a4ad5a5...4a222a1. Read the comment docs.

The server should reload in the config_dir mode.

Ref: #848

Hmm, I haven't tried removing ProxyPassReverse, but it doesn't touch the HTML, which is the issue I'm seeing. You can read the documentation here. ProxyPassReverse is a standard directive when proxying with Apache. I've used it dozens of times with other applications.

Looking a little more at the code, I think the issue here is that the behaviour of base_url makes sense when Datasette is mounted at a path within a larger application, but not when HTTP requests are being proxied to it.

In a mount situation, it is perfectly fine to construct URLs reusing the domain and path from the request. In a proxy situation, it never is, as the domain and path in the request are not the domain and path that the non-proxy client actually needs to use. That is, links which include the Apache → Datasette request origin, localhost:8001, instead of the browser → Apache request origin, example.com, will be broken.

The tests you pointed to also reflect this in two ways:

1. They strip a leading http://localhost, allowing such URLs in the facet links to pass, but inclusion of that in a proxy situation would mean the URL is broken.

2. The test client emits direct ASGI events instead of actual proxied HTTP requests. The headers of these ASGI events don't reflect the way an HTTP proxy works; instead they pass through the original request path which contains base_url. This works because Datasette responds to requests equivalently at either /… or /{base_url}/…, which makes some sense in a mount situation but is unconventional (albeit workable) for a proxied app.

Apps that support being proxied automatically support being mounted, but apps that only support being mounted don't automatically support being proxied.

Hi @wragge,

This looks great, thanks for the share! I refactored it into a self-contained function, binding on a random available TCP port (multi-user context). I am using subprocess API directly since the %run magic was leaving defunct process behind :/

import socket

from signal import SIGINT
from subprocess import Popen, PIPE

from IPython.display import display, HTML
from notebook.notebookapp import list_running_servers


def get_free_tcp_port():
    """
    Get a free TCP port.
    """
    tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    tcp.bind(('', 0))
    _, port = tcp.getsockname()
    tcp.close()
    return port


def datasette(database):
    """
    Run datasette on an SQLite database.
    """
    # Get current running servers
    servers = list_running_servers()

    # Get the current base url
    base_url = next(servers)['base_url']

    # Get a free port
    port = get_free_tcp_port()

    # Create a base url for Datasette suing the proxy path
    proxy_url = f'{base_url}proxy/absolute/{port}/'

    # Display a link to Datasette
    display(HTML(f'<p><a href="{proxy_url}">View Datasette</a> (Click on the stop button to close the Datasette server)</p>'))

    # Launch Datasette
    with Popen(
        [
            'python', '-m', 'datasette', '--',
            database,
            '--port', str(port),
            '--config', f'base_url:{proxy_url}'
        ],
        stdout=PIPE,
        stderr=PIPE,
        bufsize=1,
        universal_newlines=True
    ) as p:
        print(p.stdout.readline(), end='')
        while True:
            try:
                line = p.stderr.readline()
                if not line:
                    break
                print(line, end='')
                exit_code = p.poll()
            except KeyboardInterrupt:
                p.send_signal(SIGINT)

Ideally, I'd like some extra magic to notify users when they are leaving the closing the notebook tab and make them terminate the running datasette processes. I'll be looking for it.

There's a working demo here: https://github.com/wragge/datasette-test

And if you want something that's more than just proof-of-concept, here's a notebook which does some harvesting from web archives and then displays the results using Datasette: https://nbviewer.jupyter.org/github/GLAM-Workbench/web-archives/blob/master/explore_presentations.ipynb

Hi,

I came across this issue while looking for a way to spawn Datasette as a SQLite files viewer in JupyterLab. I found https://github.com/simonw/jupyterserverproxy-datasette-demo which seems to be the most up to date proof of concept, but it seems to be failing to list the available db (at least in the Binder demo, https://hub.gke.mybinder.org/user/simonw-jupyters--datasette-demo-uw4dmlnn/datasette/, I only have :memory).

Does anyone tried to improve on this proof of concept to have a Datasette visualization for SQLite files?

Thanks!

think, because the given URL of the CSS file doesn't have any complete parameters after query
Try to complete the parameter
<link rel="stylesheet" href="-/static/app.css?(whatever you want that shows a css file)">

Wow, I'm in some way very proud!

@simonw I would prefer just the 7 character hash. No need to make the urls any longer than they need to be :)

Dear @simonw thank you for your time, now IT WORKS!!!

I hope that this edit to datasette code is not for an exceptional case (my PC configuration) and that it will be useful to other users.

Thank you again!!

Oh no! It looks like dirs_exist_ok is Python 3.8 only. This is a bad fix, it needs to work on older Python's too. Re-opening.

Thank you very much

As that seems to be closed, can you give a hint on how to make this work?

@simonw now I have

Traceback (most recent call last):
  File "/home/aborruso/.local/bin/datasette", line 8, in <module>
    sys.exit(cli())
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py", line 103, in heroku
    extra_metadata,
  File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py", line 191, in temporary_heroku_directory
    os.path.join(tmp.name, "templates"),
  File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/__init__.py", line 605, in link_or_copy_directory
    shutil.copytree(src, dst, copy_function=os.link, dirs_exist_ok=True)
TypeError: copytree() got an unexpected keyword argument 'dirs_exist_ok'

Do I must open a new issue?

Thank you

Thank you very much!!

I will try and I write you here

@simonw can I test it know? What I must do to update it?

Thank you

I've added support for the above exif data to v0.28.17 of osxphotos. PhotoInfo.exif_info will return an ExifInfo dataclass object with the following properties:

    flash_fired: bool
    iso: int
    metering_mode: int
    sample_rate: int
    track_format: int
    white_balance: int
    aperture: float
    bit_rate: float
    duration: float
    exposure_bias: float
    focal_length: float
    fps: float
    latitude: float
    longitude: float
    shutter_speed: float
    camera_make: str
    camera_model: str
    codec: str
    lens_model: str

It's not all the EXIF data available in most files but is the data Photos deems important to save. Of course, you can get all the exif_data

Note: this only works in Photos 5. As best as I can tell, EXIF data is not stored in the database for earlier versions.

I'm not familiar with ExifReader. I wrote my own wrapper around exiftool because I wanted a simple way to write EXIF data when exporting photos (e.g. writing out to PersonInImage and keywords to IPTC:Keywords) and the existing python packages like pyexiftool didn't do quite what I wanted. If all you're after is the camera and shot info, that's available in ZEXTENDEDATTRIBUTES table. I've got an open issue #11 to add this to osxphotos but it hasn't bubbled to the top of my backlog yet.

osxphotos will give you the location info: PhotoInfo.location returns a tuple of (lat, lon) though this info is in ZEXTENDEDATTRIBUTES too (though it might not be correct as I believe Photos creates this table at import and the user might have changed the location of a photo, e.g. if camera didn't have GPS).

CREATE TABLE ZEXTENDEDATTRIBUTES (
  Z_PK INTEGER PRIMARY KEY, Z_ENT INTEGER, 
  Z_OPT INTEGER, ZFLASHFIRED INTEGER, 
  ZISO INTEGER, ZMETERINGMODE INTEGER, 
  ZSAMPLERATE INTEGER, ZTRACKFORMAT INTEGER, 
  ZWHITEBALANCE INTEGER, ZASSET INTEGER, 
  ZAPERTURE FLOAT, ZBITRATE FLOAT, ZDURATION FLOAT, 
  ZEXPOSUREBIAS FLOAT, ZFOCALLENGTH FLOAT, 
  ZFPS FLOAT, ZLATITUDE FLOAT, ZLONGITUDE FLOAT, 
  ZSHUTTERSPEED FLOAT, ZCAMERAMAKE VARCHAR, 
  ZCAMERAMODEL VARCHAR, ZCODEC VARCHAR, 
  ZLENSMODEL VARCHAR
);

Very nice! Thank you for sharing that :+1: :) Will try it out!

Authorization: bearer xxx auth for API keys is a plus plus for me. Looked into just adding this into your Flask logic but learned this project doesn't use flask. Interesting 🤔

@simonw FYI, osxphotos includes a built in ExifTool class that uses exiftool to read and write exif data. It's not exposed yet in the docs because I really only use it right now in the osphotos command line interface to write tags when exporting. In v0.28.16 (just pushed) I added an ExifTool.as_dict() method which will give you a dict with all the exif tags in a file. For example:

import osxphotos
photos = osxphotos.PhotosDB().photos()
exiftool = osxphotos.exiftool.ExifTool(photos[0].path)
exifdata = exiftool.as_dict()
tags = exifdata["IPTC:Keywords"]

Not as elegant perhaps as a python only implementation because ExifTool has to make subprocess calls to an external tool but exiftool is by far the best tool available for reading and writing EXIF data and it does support HEIC.

As for implementation, ExifTool uses a singleton pattern so the first time you instantiate it, it spawns an IPC to exiftool but then keeps it open and uses the same process for any subsequent calls (even on different files).

Frustrates me when package authors create a "drop in" replacement with the same import name...this kind of thing has bitten me more than once! Would've been nicer I think for bpylist2 to do "import bpylist2 as bpylist"

Did removing old bpylist solve the original problem or do you still have a photo that throws circular reference?

@simonw does Photos show valid reverse geolocation info? Are you sure you're using bpylist2 and not bpylist? They're both unfortunately imported as "bpylist" so if you somehow got the wrong (original bpylist) version installed, it could be the issue.

Ugh....Yeah, I think easiest is to catch the exception and return no place as you suggest. This particular bit of code involves un-archiving a serialized NSKeyedArchiver which uses an object table and it is certainly possible to create a circular reference that way. Because this is happening in the decode, the circular reference must be in the original data. Does Photos show valid reverse geolocation info for the photo in question? If so, Photos may be doing something beyond a simple decode of the binary plist. For now, I'll push a patch to catch the exception.

This has now been resolved and is not present in current version of datasette.

Sample query @simonw mentioned now returns as expected.

https://aggreg8streams.tinyviking.ie/simplestreams?sql=select+*+from+cloudimage+where+%22content_id%22+%3D+%22com.ubuntu.cloud%3Areleased%3Adownload%22+order+by+id+limit+10

just trying out datasette and quite like it, thanks! i found this issue annoying enough to have a go at a fix. have you any thoughts on a good approach? (i'm happy to dig in myself if you haven't thought about it yet, but wanted to check if you had an idea for how to fix when you raised the issue)

It seems that heroku wasn't updating to 0.41 on deployment.

Had to add --branch 0.41 and that solved it! Heroku caches dependencies

and (i think) because the requirements.txt doesn't specify the datasette version, it didn't update from 0.40 to 0.41 on heroku even though it was specified on my local requirements file as datasette >= 0.41

These are the lines that gave me an idea on how to solve it:
https://github.com/simonw/datasette/blob/182e5c8745c94576718315f7596ccc81e5e2417b/datasette/publish/heroku.py#L164-L186

Hi! I'm using datasette on this repository: https://github.com/chekos/RIPA-2018-datasette

and on my local machine i can see an /about page i created but when i deploy to heroku i get a 404 (http://ripa-2018-db.herokuapp.com)

I bumped datasette in my requirements file to 0.41 so I'm 100% what the issue is 🤔

Do you have any idea what could be the problem? 👀

EDIT: for context, I have a templates directory with a pages/about.html file in https://github.com/chekos/RIPA-2018-datasette/tree/master/datasette/templates

I have installed heroku plugins:install heroku-builds, but I have the same error.

Then I have removed from datasette\publish\heroku.py

        # Check for heroku-builds plugin
        plugins = [
            line.split()[0] for line in check_output(["heroku", "plugins"]).splitlines()
        ]
        if b"heroku-builds" not in plugins:
            click.echo(
                "Publishing to Heroku requires the heroku-builds plugin to be installed."
            )
            click.confirm(
                "Install it? (this will run `heroku plugins:install heroku-builds`)",
                abort=True,
            )
            call(["heroku", "plugins:install", "heroku-builds"])

And now I have

Traceback (most recent call last):
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py", line 210, in temporary_heroku_directory
    yield
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py", line 96, in heroku
    list_output = check_output(["heroku", "apps:list", "--json"]).decode(
  File "c:\python37\lib\subprocess.py", line 395, in check_output
    **kwargs).stdout
  File "c:\python37\lib\subprocess.py", line 472, in run
    with Popen(*popenargs, **kwargs) as process:
  File "c:\python37\lib\subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "c:\python37\lib\subprocess.py", line 1178, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The specified file could not be found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\Scripts\datasette.exe\__main__.py", line 9, in <module>
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py", line 120, in heroku
    call(["heroku", "builds:create", "-a", app_name, "--include-vcs-ignore"])
  File "c:\python37\lib\contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py", line 213, in temporary_heroku_directory
    tmp.cleanup()
  File "c:\python37\lib\tempfile.py", line 809, in cleanup
    _shutil.rmtree(self.name)
  File "c:\python37\lib\shutil.py", line 513, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "c:\python37\lib\shutil.py", line 401, in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
  File "c:\python37\lib\shutil.py", line 399, in _rmtree_unsafe
    os.rmdir(path)
PermissionError: [WinError 32] Unable to access file. The file is being used by another process: 'C:\\Users\\aborr\\AppData\\Local\\Temp\\tmpkcxy8i_q'

In Windows I'm not very strong. I use debian (inside WSL).

However these are the possible steps:

• I have installed Python 3 for win (I have 3.7.3);
• I have installed heroku cli for win64 and logged in;
• I have installed datasette running python -m pip install --upgrade --user datasette.

It's a very basic Python env that I do not use. This time only to reach my goal: try to publish using custom template

@simonw another error starting from Windows.

I run

datasette  publish heroku -n comunepa --template-dir template commissioniComunePalermo.db

And I have

Traceback (most recent call last):
  File "c:\python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\Scripts\datasette.exe\__main__.py", line 9, in <module>
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py", line 53, in heroku
    line.split()[0] for line in check_output(["heroku", "plugins"]).splitlines()
  File "c:\python37\lib\subprocess.py", line 395, in check_output
    **kwargs).stdout
  File "c:\python37\lib\subprocess.py", line 472, in run
    with Popen(*popenargs, **kwargs) as process:
  File "c:\python37\lib\subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "c:\python37\lib\subprocess.py", line 1178, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The specified file could not be found

files.zip

Hi @simonw probably I could try to do it in Python for windows. I do not like to do these things in win enviroment.

Because probably WSL Linux env (in which I do a lot of great things) is not an environment that will be tested for datasette.

In win I shouldn't have any problems. Am I right?

Thank you. Now it's ok with the url

http://localhost:8001/index/summary?_search=language%3Aeng&_sort=title&_searchmode=raw

But I'm not able to manage it in the metadata file. Here is mine (note that the sort column is taken into account)
Here it is:

{
    "databases": {
      "index": {
        "tables": {
            "summary": {
                "sort": "title",
                "searchmode": "raw"
            }
        }
      }
    }
  }

Any idea ?

FYI, I've got an issue to make osxphotos cross-platform but it's low on my priority list. About 90% of the functionality could be done cross-platform but right now the MacOS specific stuff is embedded throughout and would take some work. Though I try to minimize it, there's sprinklings of ObjC & Applescript throughout osxphotos.

I'm traveling w/o access to my Mac so can't help with any code right now. I suspected ZSCENEIDENTIFIER was a foreign key into one of these psi.sqlite tables. But looks like you're on to something connecting groups to assets. As for the UUID, I think there's two ints because each is 64-bits but UUIDs are 128-bits. Thus they need to be combined to get the 128 bit UUID. You might be able to use Apple's NSUUID, for example, by wrapping with pyObjC. Here's one example of using this in PyObjC's test suite. Interesting it's stored this way instead of a UUIDString as in Photos.sqlite. Perhaps it for faster indexing.

thanks a lot, Simon

1. More than glad to share feedback from the sidelines as a starrer.

-- Motivation:
--  Datasette is a data hammer and I'm looking for nails
--  e.g. Find which repos a user has starred => trigger a TBD downstream action
select
  starred_at,
  starred_by,
  full_name as repo_name
from
  repos_starred
where
  starred_by = "zzeleznick"
order by
  starred_at desc

1. In my day job, I use airflow, and that's the mental model I'm bringing to datasette.

2. I see your project like twitter-to-sqlite akin to Operators in Airflow world.

Thanks, @simonw!

I feel a little foolish in hindsight, but I'm on the same page now and am glad to have discovered first-hand a motivation for this repos_starred use case.

I also go it working with:

run: echo ${{ secrets.github_token }} | github-to-sqlite auth

Hi @simonw it's debian as Windows Subsystem for Linux

PRETTY_NAME="Pengwin"
NAME="Pengwin"
VERSION_ID="10"
VERSION="10 (buster)"
ID=debian
ID_LIKE=debian
HOME_URL="https://github.com/whitewaterfoundry/Pengwin"
SUPPORT_URL="https://github.com/whitewaterfoundry/Pengwin"
BUG_REPORT_URL="https://github.com/whitewaterfoundry/Pengwin"
VERSION_CODENAME=buster

A stupid note: I have no tmpcqv_1i5d folder in in /tmp.

It seems to me that it does not create any /tmp/tmpcqv_1i5d/templates folder (or other name folder, inside /tmp)

Hi @simonw , I have installed it and I have the below errors.

Is it possible that your /tmp directory is on a different volume from the template folder? That could cause a problem with the symlinks.

No, /tmp folder is in the same volume.

Thank you

Traceback (most recent call last):
  File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/__init__.py", line 607, in link_or_copy_directory
    shutil.copytree(src, dst, copy_function=os.link)
  File "/usr/lib/python3.7/shutil.py", line 365, in copytree
    raise Error(errors)
shutil.Error: [('/var/youtubeComunePalermo/processing/./template/base.html', '/tmp/tmpcqv_1i5d/templates/base.html', "[Errno 18] Invalid cross-device link: '/var/youtubeComunePalermo/processing/./template/base.html' -> '/tmp/tmpcqv_1i5d/templates/base.html'"), ('/var/youtubeComunePalermo/processing/./template/index.html', '/tmp/tmpcqv_1i5d/templates/index.html', "[Errno 18] Invalid cross-device link: '/var/youtubeComunePalermo/processing/./template/index.html' -> '/tmp/tmpcqv_1i5d/templates/index.html'")]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aborruso/.local/bin/datasette", line 8, in <module>
    sys.exit(cli())
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 610, in invoke    return callback(*args, **kwargs)
  File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py", line 103, in heroku
    extra_metadata,
  File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py", line 191, in temporary_heroku_directory
    os.path.join(tmp.name, "templates"),
  File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/__init__.py", line 609, in link_or_copy_directory
    shutil.copytree(src, dst)
  File "/usr/lib/python3.7/shutil.py", line 321, in copytree
    os.makedirs(dst)
  File "/usr/lib/python3.7/os.py", line 221, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/tmp/tmpcqv_1i5d/templates'

Using the Heroku web interface, you can set the WEB_CONCURRENCY = 1

It works in heroku, than might be a bug with datasette-publish-now.

Thank you

Would you mind trying publishing your database using one of the other options - Heroku, Cloud Run or https://fly.io/ - and see if you have the same bug there?

It works in heroku, than might be a bug with datasette-publish-now.

Thank you

as a complementary remark: the versioning of datasette dependancies will become a problem when the new pip "dependancy resolver" will be activated. for now, it's just warnings via pip checks, later it will be a "no":

datasette 0.40 has requirement aiofiles~=0.4.0, but you have aiofiles 0.5.0.
datasette 0.40 has requirement Jinja2~=2.10.3, but you have jinja2 2.11.2.

Mounting ./static at /static seems the simplest way. Saves you the trouble of deciding what else (img for example) gets special treatment.

I've almost suggested this same thing a couple times. I tend to have Makefile (because I'm doing other make stuff anyway to get data prepped), and I end up putting all those CLI options in something like make run. But it would be way easier to just have all those typical options -- plugins, templates, metadata -- be defaults.

@eads I'm interested in helping, if there's still a need...

Thanks for your hard work!

Woah! Thanks a lot. Next time I will add a more obvious/explicit "if you like this idea let me know I'd love to work on it to get my feet wet here" :D

is there any specific reason to stick to Jinja2~=2.10.3 when there is Jinja-2.11.1 ?

That looks great, thanks!

Following on from @betatim's suggestion on Twitter, I've changed the proxy url to include 'absolute'.

proxy_url = f'{base_url}proxy/absolute/8001/'

This works both on Binder and locally, without using the path_from_header option. I've updated the demo repository. Sorry @simonw if I've led you down the wrong path!

Great! Yes, can confirm that this works on Binder. However, when I try to run the same code locally, I get an Internal Server Error when I try to access Datasette.

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/Volumes/Workspace/mycode/datasette-test/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 385, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/Volumes/Workspace/mycode/datasette-test/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/Volumes/Workspace/mycode/datasette-test/lib/python3.7/site-packages/datasette_debug_asgi.py", line 24, in wrapped_app
    await app(scope, recieve, send)
  File "/Volumes/Workspace/mycode/datasette-test/lib/python3.7/site-packages/datasette/utils/asgi.py", line 174, in __call__
    await self.app(scope, receive, send)
  File "/Volumes/Workspace/mycode/datasette-test/lib/python3.7/site-packages/datasette/tracer.py", line 75, in __call__
    await self.app(scope, receive, send)
  File "/Volumes/Workspace/mycode/datasette-test/lib/python3.7/site-packages/datasette/app.py", line 746, in __call__
    raw_path = dict(scope["headers"])[path_from_header.encode("utf8")].split(b"?")[0]
KeyError: b'x-original-uri'
INFO:     127.0.0.1:49320 - "GET / HTTP/1.1" 500 Internal Server Error

Thanks! I'm trying to launch Datasette from within a notebook using the jupyter-server-proxy and the new base_url parameter. While the assets load ok, and the breadcrumb navigation works, the facet links don't seem to use the base_url. Or have I missed something?

My test repository is here: https://github.com/wragge/datasette-test

Great - thanks again.

Hi Simon - I'm just (trying, at least) to follow along in the above. I can't try it out now, but I will if no one else gets to it. Sorry I didn't write any tests in the original bit of code I pushed - I was just trying to see if it could work & whether you'd want to maybe head in that direction. Anyway, thank you, I will certainly use this. Comment back here if no one tried it out & I'll make time.

@simonw You're welcome - I was just trying it out back in December as I thought it should work. Now there's a pandemic to work on though.... so no time at all for more at the moment. BTW, I have datasette running on several protein and full (virus) genome databases I build, and it's great - thank you! Hi and best regards to you & Nat :-)

I just updated #652 to remove a merge conflict. I think it's an easy way to add this functionality. I don't have time to do more though, sorry!

This would also be useful for running Datasette in Jupyter notebooks on Binder. While you can use Jupyter-server-proxy to access Datasette on Binder, the links are broken.

Why run Datasette on Binder? I'm developing a range of Jupyter notebooks that are aimed at getting humanities researchers to explore data from libraries, archives, and museums. Many of them are aimed at researchers with limited digital skills, so being able to run examples in Binder without them installing anything is fantastic.

For example, there are a series of notebooks that help researchers harvest digitised historical newspaper articles from Trove. The metadata from this harvest is saved as a CSV file that users can download. I've also provided some extra notebooks that use Pandas etc to demonstrate ways of analysing and visualising the harvested data.

But it would be really nice if, after completing a harvest, the user could spin up Datasette for some initial exploration of their harvested data without ever leaving their browser.

On mybinder.org we allow access to arbitrary processes listening on a port inside the container via a reverse proxy.

This means we need support for a proxy prefix as the proxy ends up running at a URL like /something/random/proxy/datasette/...

An example that shows the problem is https://github.com/psychemedia/jupyterserverproxy-datasette-demo. Launch directly into a datasette instance on mybinder.org with https://mybinder.org/v2/gh/psychemedia/jupyterserverproxy-datasette-demo/master?urlpath=datasette then try to follow links inside the UI.

I'd love to be using your library as a better cached gh layer for a new library I have built, replacing large parts of the very ugly https://github.com/jayvdb/pypidb/blob/master/pypidb/_github.py , and then probably being able to rebuild the setuppy chunk as a feature here at a later stage.

I would also need tokenless and netrc support, but I would be happy to add those bits.

If you're happy with the proposed implementation, I have code & tests written that I'll get ready for a PR.

Did you manage to find an answer to this? I've got a notebook to help people generate datasets on the fly from an API, so it would be cool if they flick it to Datasette for initial exploration.

@simonw any thoughts on allow extracts to specify the lookup column name? If I'm understanding the documentation right, .lookup() allows you to define the "value" column (the documentation uses name), but when you use extracts keyword as part of .insert(), .upsert() etc. the lookup must be done against a column named "value". I have an existing lookup table that I've populated with columns "id" and "name" as opposed to "id" and "value", and seems I can't use extracts=, unless I'm missing something...

Initial thought on how to do this would be to allow the dictionary value to be a tuple of table name column pair... so:

table = db.table("trees", extracts={"species_id": ("Species", "name"})

I haven't dug too much into the existing code yet, but does this make sense? Worth doing?

Design looks great to me.

Excellent, thanks!

I'm not keen on two letter short versions (-cp) - I'd rather either have a single character or no short form at all.

Hmm, well, anyone running datasette package is probably at least somewhat familiar with UNIX CLIs… so how about --cp as a middle ground?

$ datasette package --cp /the/source/path /the/target/path data.db

I think I like it. Easy to remember!

Excellent. I'll implement the simple plugin-solution now. And will have a go at a more mature plugin later. Thanks!

So yeah - if you're happy to design this I think it would be worth us adding.

Great! I’ll give it a go.

Small design suggestion: allow --copy to be applied multiple times…

Makes a ton of sense, will do.

Also since Click arguments can take multiple options I don't think you need to have the : in there - although if it better matches Docker's own UI it might be more consistent to have it.

Great point. I double checked the docs for docker cp and in that context the colon is used to delimit a container and a path, while spaces are used to separate the source and target.

The usage string is:

docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH

so in fact it’ll be more consistent to use a space to delimit the source and destination paths, like so:

$ datasette package --copy /the/source/path /the/target/path data.db

and I suppose the short-form version of the option should be cp like so:

$ datasette package -cp /the/source/path /the/target/path data.db

Awesome, thank you so much. I’ll try it out and let you know.

On Sun, Feb 23, 2020 at 1:44 PM Simon Willison notifications@github.com
wrote:

You can try this right now like so:

pip install https://github.com/simonw/datasette/archive/search-raw.zip

Then use the following:

?_search=foo*&_searchmode=raw`

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/simonw/datasette/issues/676?email_source=notifications&email_token=AN3FXEFS6B22U2NOT6M5FULRELNY7A5CNFSM4KYIOIB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMWGYTI#issuecomment-590113869,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AN3FXEANDJ6AIHGU4ADK4D3RELNY7ANCNFSM4KYIOIBQ
.

Thanks Simon,
My use case is using Datasette for full text search type ahead. That was working pretty well. The _search_wildcard= option will be awesome. Thanks

Probably the best option to just throw the error.
Is there any active dev chan where we could post the issue to python sqlite3?

@simonw fwiw, here's the plugin I implemented to support CLDF datasets: https://github.com/cldf/datasette-cldf/blob/master/README.md
It's a bit of a hybrid in that it does both, building the SQLite database and extending datasette by exploting what we know about the data format - so it may not be worth listing it with the other plugins.

Having tools like datasette available definitely helps selling people on package formats like CLDF (or CSVW), many thanks for this!

I think I found a better way to implement my use case: I wrap the datasette serve call into my own cli, which
- creates the SQLite from CSV data
- writes metadata.json for datasette
- determines suitable config like max_page_size
- then calls datasette serve.

See https://github.com/simonw/datasette/issues/657 and my changes that allow datasette to load parquet files

I don't want to re-open this issue, but I'm wondering whether it would be possible to include the full row for which a specific cell is to be rendered in the hook signature. My use case are rows where custom rendering would need access to multiple values (specifically, rows containing the constituents of interlinear glossed text (IGT) in separate columns, see https://github.com/cldf/cldf/tree/master/components/examples).

I could probably cobble this together with custom SQL and the sql-to-html plugin. But having a full row within a render_cell implementation seems a lot simpler.

Simon,

Yes, there is an "app.css" on static folder, however, anyone modification I do on this .css, doesn't apply on the datasette.

I'm using this command: datasette publish heroku "databases folder" -n "herokuapp name" --extra-options="--config sql_time_limit_ms:60000 --config max_returned_rows:10000 --config force_https_urls:1" --template-dir "templates folder" -m "metadata.json folder"

So the conundrum continues.. The simple test case above now runs, but if I upsert a large number of new records (successfully) and then try to upsert a fewer number of new records to a different table, I get the same error.

If I run the same upserts again (which in the first case means there are no new records to add, because they were already added), the second upsert works correctly.

It feels as if the number of items added via an upsert >> the number of items I try to add in an upsert immediately after, I get the error.

Thanks for the elegant solution to the problem as stated. I'm packaging right now :-)

This is excellent news. I'll wait until version 0.34. It would be tiresome to rewrite all standard-queries into custom queries. Thank you!

Hi Simon

Thankt you for a quick reply. Here are a few examples of urls, where I search the 'cases_fts'-virtual table for tokens in the title-column. It returns the same results, wether the other query-params are present or not.

Searching for sky
http://localhost:8001/db-7596a4e/cases?_search_title=sky&year__gte=1997&year__lte=2017&_sort_desc=last_deliberation_date
Returns searchresults

Searching for sky
http://localhost:8001/db-7596a4e/cases?_search_title=sky&year__gte=1997&year__lte=2017&_sort_desc=last_deliberation_date
Returns searchresults

Searching for sky-tog
http://localhost:8001/db-7596a4e/cases?_search_title=sky-tog&year__gte=1997&year__lte=2017&_sort_desc=last_deliberation_date
Throws: No such column: tog

searching for sky+
http://localhost:8001/db-7596a4e/cases?_search_title=sky%2B&year__gte=1997&year__lte=2017&_sort_desc=last_deliberation_date
Throws: Invalid SQL: fts5: syntax error near ""

Searching for "madpakke" (including double quotes)
http://localhost:8001/db-7596a4e/cases?_search_title=%22madpakke%22&year__gte=1997&year__lte=2017&_sort_desc=last_deliberation_date
Returns searchresults even though 'madpakke' only appears in the fulltextindex without quotes

As I said, my other plugins work just fine, and I just copied your sql_functions.py from the datasette-repo.

Thanks!

Hi Simon

Thank you for adding the escape_function, but it does not work on my datasette-installation (0.33). I've added the following file to my datasette-dir: /plugins/sql_functions.py:

`from datasette import hookimpl

def escape_fts_query(query):
bits = query.split()
return ' '.join('"{}"'.format(bit.replace('"', '')) for bit in bits)

@hookimpl
def prepare_connection(conn):
conn.create_function("escape_fts_query", 1, escape_fts_query)`

It has no effect on the standard queries to the tables though, as they still produce errors when including any characters like '-', '/', '+' or '?'

Does the function only work when using costum queries, where I can include the escape_fts-function explicitly in the sql-query?

PS. I'm calling datasette with --plugins=plugins, and my other plugins work just fine.
PPS. The fts5 virtual table is created with 'sqlite3' like so:

CREATE VIRTUAL TABLE "cases_fts" USING FTS5( title, subtitle, resume, suggestion, presentation, detail = full, content_rowid = 'id', content = 'cases', tokenize='unicode61', 'remove_diacritics 2', 'tokenchars "-_"' );

Thanks!

Hi,

I've completed some changes to my fork of datasette that allows it to automatically create the parquet virtual table when you supply it with a filename that has the ".parquet" extension.

I had to figure out how to make the "CREATE VIRTUAL TABLE" statement only be applied to the fake in memory parquet database and not to any others that were also being loaded. Thus it supports mixed mode databases e.g

datasette my_test.parquet normal_sqlite_file.db  --load-extension=libparquet.so --load-extensio
n=mod_spatialite.so

Please see my changes here:
https://github.com/dazzag24/datasette/commit/8e18394353114f17291fd1857073b1e0485a1faf

Thanks

Seems that headers and definitions has simply to be filled as an HTML table in the description field of matadata.json.

Omg sorry I took so long to reply!

On SQL we can say how the foreign key behaves when it is deleted or updated on the parent table (see https://www.sqlitetutorial.net/sqlite-foreign-key/ for more details).

I did not see clearly how to create tables with this feature on sqlite-utils library.

I have successfully tested datasette using a parquet VIRTUAL TABLE. In the first terminal:

datasette airports.db --load-extension=libparquet

In another terminal I load the same sqlite db file using the sqlite3 cli client.

$ sqlite3 airports.db

and then load the parquet extension and create the virtual table.

sqlite> .load /home/darreng/metars/libparquet
sqlite> CREATE VIRTUAL TABLE mytable USING parquet('/home/xx/data.parquet');

Now the parquet virtual table is usable by the datasette web UI.

Its not an ideal solution but is a proof that datasette works the parquet extension.

Hmmm... just tried with installs from pip and the repo (v2.0.0 and v2.0.1) and I get the error each time (start of second run through the second loop).

Could it be sqlite3? I'm on 3.30.1.

UPDATE: just tried it on jupyter.org/try and I get the error there, too.

I re-tried after doing auth, and I get the same result.

I think I actually had several issues in play...

The missing key was one, but I think there is also an issue as per below.

For example, in the following:

def init_testdb(dbname='test.db'):

    if os.path.exists(dbname):
        os.remove(dbname)

    conn = sqlite3.connect(dbname)
    db = Database(conn)

    return conn, db

conn, db = init_testdb()

c = conn.cursor()
c.executescript('CREATE TABLE "test1" ("Col1" TEXT, "Col2" TEXT, PRIMARY KEY ("Col1"));')
c.executescript('CREATE TABLE "test2" ("Col1" TEXT, "Col2" TEXT, PRIMARY KEY ("Col1"));')

print('Test 1...')
for i in range(3):
    db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1'))
    db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1'))

print('Test 2...')
for i in range(3):
    db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1'))
    db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'},
                            {'Col1':'c','Col2':'x'}], pk=('Col1'))
print('Done...')

---------------------------------------------------------------------------
Test 1...
Test 2...
 IndexError: list index out of range 
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-763-444132ca189f> in <module>
     22 print('Test 2...')
     23 for i in range(3):
---> 24     db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1'))
     25     db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'},
     26                             {'Col1':'c','Col2':'x'}], pk=('Col1'))

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, extracts)
   1157             alter=alter,
   1158             extracts=extracts,
-> 1159             upsert=True,
   1160         )
   1161 

/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, upsert)
   1097                 # self.last_rowid will be 0 if a "INSERT OR IGNORE" happened
   1098                 if (hash_id or pk) and self.last_rowid:
-> 1099                     row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0]
   1100                     if hash_id:
   1101                         self.last_pk = row[hash_id]

IndexError: list index out of range

the first test works but the second fails. Is the length of the list of items being upserted leaking somewhere?

Meant to add that of course it would be better not to reinvent CSS (one time was already enough). But one option would be to provide a mechanism to specify a CSS class for a column (a cell, a row...) and let the user give a URL path to a CSS file on the command line.

I initially went looking for a way to hide a column completely. Today I found the setting to truncate cells, but it applies to all cells. In my case I have text columns that can have many thousands of characters. I was wondering whether the metadata JSON would be an appropriate place to indicate how columns are displayed (on a col-by-col basis). E.g., I'd like to be able to specify that only 20 chars of a given column be shown, and the font be monospace. But maybe I can do that in some other way - I barely know anything about datasette yet, sorry!

@simonw What about allowing a base url. The <base>....</base> tag has been around forever. Then just use all relative URLs, which I guess is likely what you already do. See https://www.w3schools.com/TAGs/tag_base.asp