issue_comments

Table actions

7,162 rows sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

issue >1000

user 263

author_association 4

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
1017016553 https://github.com/simonw/datasette/issues/1356#issuecomment-1017016553 https://api.github.com/repos/simonw/datasette/issues/1356 IC_kwDOBm6k_c48nnDp simonw 9599 2022-01-20T01:06:37Z 2022-01-20T01:06:37Z OWNER

A problem with this is that if you're using --query you likely want ALL of the results - at the moment the only Datasette output type that can stream everything is .csv and plugin formats can't handle full streams, see #1062 and #1177.

I figured out a neat pattern for streaming JSON arrays in this TIL: https://til.simonwillison.net/python/output-json-array-streaming

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Research: syntactic sugar for using --get with SQL queries, maybe "datasette query" 910092577  
1016994329 https://github.com/simonw/datasette/issues/1605#issuecomment-1016994329 https://api.github.com/repos/simonw/datasette/issues/1605 IC_kwDOBm6k_c48nhoZ eyeseast 25778 2022-01-20T00:27:17Z 2022-01-20T00:27:17Z CONTRIBUTOR

Right now, I usually have a line in a Makefile like this:

combined.geojson: project.db
    pipenv run datasette project.db --get /project/combined.geojson \
     --load-extension spatialite \
     --setting sql_time_limit_ms 5000 \
     --setting max_returned_rows 20000 \
     -m metadata.yml > $@

That all assumes I've loaded whatever I need into project.db and created a canned query called combined (and then uses datasette-geojson for geojson output).

It works, but as you can see, it's a lot to manage, a lot of boilerplate, and it wasn't obvious how to get there. If there's an error in the canned query, I get an HTML error page, so that's hard to debug. And it's only one query, so each output needs a line like this. Make isn't ideal, either, for that reason.

The thing I really liked with datafreeze was doing templated filenames. I have a project now where I need to export a bunch of litttle geojson files, based on queries, and it would be awesome to be able to do something like this:

databases:
  project:
    queries:
      boundaries:
        sql: "SELECT * FROM boundaries"
        filename: "boundaries/{id}.geojson"
        mode: "item"
        format: geojson

And then do:

datasette freeze -m metadata.yml project.db

For HTML export, maybe there's a template argument, or format: template or something. And that gets you a static site generator, kinda for free.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Scripted exports 1108671952  
1016977725 https://github.com/simonw/datasette/issues/1605#issuecomment-1016977725 https://api.github.com/repos/simonw/datasette/issues/1605 IC_kwDOBm6k_c48ndk9 simonw 9599 2022-01-19T23:55:08Z 2022-01-19T23:55:08Z OWNER

Oh that's interesting. I was thinking about this from a slightly different angle recently - pondering what a static site generator built on top of Datasette might look like.

Just a sketch at the moment, but I was imagining a YAML configuration file with a SQL query that returns a list of paths - then a tool that runs that query and uses the equivalent of datasette --get to create a static copy of each of those paths.

I think these two ideas can probably be merged. I'd love to know more about how you are solving this right now!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Scripted exports 1108671952  
1016651485 https://github.com/simonw/datasette/issues/1601#issuecomment-1016651485 https://api.github.com/repos/simonw/datasette/issues/1601 IC_kwDOBm6k_c48mN7d eyeseast 25778 2022-01-19T16:39:03Z 2022-01-19T16:39:03Z CONTRIBUTOR

I think both of these are Spatialite specific. They get generated when you first initialize the extension. KNN is actually deprecated in favor of KNN2, as I understand it.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add KNN and data_licenses to hidden tables list 1105916061  
1016639294 https://github.com/simonw/datasette/issues/1601#issuecomment-1016639294 https://api.github.com/repos/simonw/datasette/issues/1601 IC_kwDOBm6k_c48mK8- simonw 9599 2022-01-19T16:26:44Z 2022-01-19T16:26:56Z OWNER

I need to add sqlite_stat1 to the hidden tables too, see:
- https://github.com/simonw/sqlite-utils/issues/366

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add KNN and data_licenses to hidden tables list 1105916061  
1016637722 https://github.com/simonw/datasette/issues/1601#issuecomment-1016637722 https://api.github.com/repos/simonw/datasette/issues/1601 IC_kwDOBm6k_c48mKka simonw 9599 2022-01-19T16:25:07Z 2022-01-19T16:25:07Z OWNER

Good idea - though I'm nervous about accidentally hiding a data_licenses table outside of the SpatiaLite case. I'll only hide that one if SpatiaLite is loaded.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add KNN and data_licenses to hidden tables list 1105916061  
1016636561 https://github.com/simonw/datasette/issues/1604#issuecomment-1016636561 https://api.github.com/repos/simonw/datasette/issues/1604 IC_kwDOBm6k_c48mKSR simonw 9599 2022-01-19T16:23:54Z 2022-01-19T16:23:54Z OWNER

Potential design:

datasette publish cloudrun data.db \
  --service my-service \
  --domain demo.datasette.io

I think I'm OK with calling this --domain even when it is being used with a subdomain.

This will require datasette.io to already have been verified with Google.

Not sure how best to handle the DNS part - maybe print out instructions for the necessary CNAME?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Option to assign a domain/subdomain using `datasette publish cloudrun` 1108300685  
1016589519 https://github.com/simonw/datasette/issues/1603#issuecomment-1016589519 https://api.github.com/repos/simonw/datasette/issues/1603 IC_kwDOBm6k_c48l-zP simonw 9599 2022-01-19T15:36:38Z 2022-01-19T15:36:38Z OWNER

Also people can use a custom base template and link to a custom favicon if they want to.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 A proper favicon 1108235694  
1016589140 https://github.com/simonw/datasette/issues/1603#issuecomment-1016589140 https://api.github.com/repos/simonw/datasette/issues/1603 IC_kwDOBm6k_c48l-tU simonw 9599 2022-01-19T15:36:16Z 2022-01-19T15:36:16Z OWNER

For the moment then I will hard-code a new favicon and leave it to ASGI plugins if people want to define their own.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 A proper favicon 1108235694  
1016588326 https://github.com/simonw/datasette/issues/1603#issuecomment-1016588326 https://api.github.com/repos/simonw/datasette/issues/1603 IC_kwDOBm6k_c48l-gm simonw 9599 2022-01-19T15:35:33Z 2022-01-19T15:35:33Z OWNER

I don't think abusing the template loader mechanism for this will work: Jinja provides an API for loading text templates, but I don't think it can be sensibly abused to open binary image files instead. Loaded code is here: https://github.com/pallets/jinja/blob/main/src/jinja2/loaders.py

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 A proper favicon 1108235694  
1016579661 https://github.com/simonw/datasette/issues/1603#issuecomment-1016579661 https://api.github.com/repos/simonw/datasette/issues/1603 IC_kwDOBm6k_c48l8ZN simonw 9599 2022-01-19T15:27:05Z 2022-01-19T15:27:05Z OWNER

I'd like this to be customizable. I'm going to load it from the template loading system such that a custom favicon for a specific instance can be dropped in using a file in templates/.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 A proper favicon 1108235694  
1016456784 https://github.com/simonw/datasette/pull/1602#issuecomment-1016456784 https://api.github.com/repos/simonw/datasette/issues/1602 IC_kwDOBm6k_c48leZQ codecov[bot] 22429695 2022-01-19T13:17:24Z 2022-01-19T13:17:24Z NONE

Codecov Report

Merging #1602 (29f5772) into main (cb29119) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1602   +/-   ##
=======================================
  Coverage   92.09%   92.09%           
=======================================
  Files          34       34           
  Lines        4516     4516           
=======================================
  Hits         4159     4159           
  Misses        357      357

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cb29119...29f5772. Read the comment docs.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Update pytest-timeout requirement from <2.1,>=1.4.2 to >=1.4.2,<2.2 1108084641  
1013698557 https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1013698557 https://api.github.com/repos/simonw/sqlite-utils/issues/79 IC_kwDOCGYnMM48a8_9 eyeseast 25778 2022-01-15T15:15:22Z 2022-01-15T15:15:22Z NONE

@simonw I have a PR here https://github.com/simonw/sqlite-utils/pull/385 that adds Spatialite helpers on the Python side. Please let me know how it looks.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Helper methods for working with SpatiaLite 557842245  
1013669543 https://github.com/simonw/datasette/issues/1591#issuecomment-1013669543 https://api.github.com/repos/simonw/datasette/issues/1591 IC_kwDOBm6k_c48a16n simonw 9599 2022-01-15T11:56:59Z 2022-01-15T11:56:59Z OWNER

There's actually already a way to move regular Datasette --setting options to a settings.json file thanks to configuration directory mode: https://docs.datasette.io/en/stable/settings.html

Maybe extending that mechanism to handle plugins would be a neat path forward here.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Maybe let plugins define custom serve options? 1100015398  
1013668967 https://github.com/simonw/datasette/issues/1591#issuecomment-1013668967 https://api.github.com/repos/simonw/datasette/issues/1591 IC_kwDOBm6k_c48a1xn simonw 9599 2022-01-15T11:53:21Z 2022-01-15T11:53:21Z OWNER

The datasette publish --plugin-secret name setting value option already implements something a bit like this. https://docs.datasette.io/en/stable/plugins.html#secret-configuration-values

It's a bit of a messy hack to compensate for metadata being visible. Maybe I could replace that mechanism with the proposed plugin configuration rethink from this issue.

I still like the debug benefits of making plugin settings public - perhaps add a rule that if a plugin setting has a secret: prefix it gets redacted on a new /-/plugin-settings page.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Maybe let plugins define custom serve options? 1100015398  
1012680228 https://github.com/simonw/datasette/issues/1566#issuecomment-1012680228 https://api.github.com/repos/simonw/datasette/issues/1566 IC_kwDOBm6k_c48XEYk simonw 9599 2022-01-14T01:59:54Z 2022-01-14T01:59:54Z OWNER

This is now shipped!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Release Datasette 0.60 1083669410  
1012664607 https://github.com/simonw/datasette/issues/1595#issuecomment-1012664607 https://api.github.com/repos/simonw/datasette/issues/1595 IC_kwDOBm6k_c48XAkf simonw 9599 2022-01-14T01:22:58Z 2022-01-14T01:22:58Z OWNER
  • Upgraded Pluggy dependency to 1.0. #1575
  • Now using Plausible analytics for the Datasette documentation.
  • The db.execute_write() internals method now defaults to blocking until the write operation has completed. Previously it defaulted to queuing the write and then continuing to run code while the write was in the queue. #1579
  • explain query plan is now allowed with varying amounts of white space in the query. #1588
  • New CLI reference page showing the output of --help for each of the datasette sub-commands. This lead to several small improvements to the help copy. #1594
  • Fixed bug where columns with a underscore prefix could result in unnecessary hidden form fields. #1527
 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Release notes for 0.60 1102484126  
1012626410 https://github.com/simonw/datasette/issues/1595#issuecomment-1012626410 https://api.github.com/repos/simonw/datasette/issues/1595 IC_kwDOBm6k_c48W3Pq simonw 9599 2022-01-14T00:00:56Z 2022-01-14T01:17:47Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Release notes for 0.60 1102484126  
1012661522 https://github.com/simonw/datasette/issues/1590#issuecomment-1012661522 https://api.github.com/repos/simonw/datasette/issues/1590 IC_kwDOBm6k_c48W_0S simonw 9599 2022-01-14T01:16:08Z 2022-01-14T01:16:34Z OWNER

OK, I'm going to recommend a workaround for this instead. Here's asgi.py updated to strip the prefix before passing the request on to Datasette:

import pathlib
from asgi_cors import asgi_cors
from channels.routing import URLRouter
from django.urls import re_path
from datasette.app import Datasette


def rewrite_path(app, prefix_to_strip):
    async def rewrite_path_app(scope, receive, send):
        if (
            scope["type"] == "http"
            and "path" in scope
            and scope["path"].startswith(prefix_to_strip)
        ):
            scope["path"] = scope["path"][len(prefix_to_strip) :]
            if "raw_path" in scope:
                scope["raw_path"] = scope["raw_path"][len(prefix_to_strip) :]
        await app(scope, receive, send)

    return rewrite_path_app


datasette_ = Datasette(
    files=["fixtures.db"],
    settings={"base_url": "/datasettes/", "plugins": {}},
)
application = URLRouter(
    [
        re_path(
            r"^datasettes/.*",
            asgi_cors(rewrite_path(datasette_.app(), "/datasettes"), allow_all=True),
        ),
    ]
)

This works on my laptop - please re-open the ticket if it doesn't work for you!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Table+query JSON and CSV links broken when using `base_url` setting 1099723916  
1012656790 https://github.com/simonw/datasette/issues/1590#issuecomment-1012656790 https://api.github.com/repos/simonw/datasette/issues/1590 IC_kwDOBm6k_c48W-qW simonw 9599 2022-01-14T01:05:34Z 2022-01-14T01:05:34Z OWNER

I think this prefixed string mechanism is supposed to prevent the base_url prefix from being applied twice: https://github.com/simonw/datasette/blob/3664ddd400062123e99500d28b160c7944408c1a/datasette/url_builder.py#L9-L16

But with a bit of extra logging all of the inputs to that are NOT prefixed strings:

Urls.path called with: /datasettes/fixtures/compound_three_primary_keys?_sort=content (PrefixedUrlString = False)
  returning /datasettes/datasettes/fixtures/compound_three_primary_keys?_sort=content

So it looks like urls.path(...) is indeed the code responsible for doubling up that /datasettes/ prefix.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Table+query JSON and CSV links broken when using `base_url` setting 1099723916  
1012653966 https://github.com/simonw/datasette/issues/1590#issuecomment-1012653966 https://api.github.com/repos/simonw/datasette/issues/1590 IC_kwDOBm6k_c48W9-O simonw 9599 2022-01-14T00:59:07Z 2022-01-14T00:59:07Z OWNER

Since this is a special case bug for when using Datasette as a library I wonder if a good fix here would be to support something like this:

application = URLRouter([
  re_path(r"^datasettes/.*", asgi_cors(datasette_.app(remove_path_prefix="datasettes/"), allow_all=True)),
])

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Table+query JSON and CSV links broken when using `base_url` setting 1099723916  
1012653109 https://github.com/simonw/datasette/issues/1527#issuecomment-1012653109 https://api.github.com/repos/simonw/datasette/issues/1527 IC_kwDOBm6k_c48W9w1 simonw 9599 2022-01-14T00:57:08Z 2022-01-14T00:57:08Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Columns starting with an underscore behave poorly in filters 1059555791  
1012643882 https://github.com/simonw/datasette/issues/1598#issuecomment-1012643882 https://api.github.com/repos/simonw/datasette/issues/1598 IC_kwDOBm6k_c48W7gq simonw 9599 2022-01-14T00:34:49Z 2022-01-14T00:34:49Z OWNER

There are four places in the documentation that use .. literalinclude:: at the moment which I can replace - I can actually just link directly to the new https://docs.datasette.io/en/latest/cli-reference.html page instead of embedding the help directly.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Replace update-docs-help.py script with cog 1102637351  
1012635696 https://github.com/simonw/datasette/issues/1527#issuecomment-1012635696 https://api.github.com/repos/simonw/datasette/issues/1527 IC_kwDOBm6k_c48W5gw simonw 9599 2022-01-14T00:19:10Z 2022-01-14T00:20:36Z OWNER

Oh! This is because _city_id has a leading underscore (for testing purposes). I think I need to filter out any keys that contain __ in that case.

What happens to columns that contain a __? They shouldn't be reflected in the hidden arguments either - this code is really only supposed to catch things like _where and _m2m_through= and _col - basically most of the list on https://docs.datasette.io/en/stable/json_api.html#special-table-arguments

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Columns starting with an underscore behave poorly in filters 1059555791  
1012634659 https://github.com/simonw/datasette/issues/1527#issuecomment-1012634659 https://api.github.com/repos/simonw/datasette/issues/1527 IC_kwDOBm6k_c48W5Qj simonw 9599 2022-01-14T00:17:00Z 2022-01-14T00:18:11Z OWNER

That's because that page has this unnecessary hidden form field:

<input type="hidden" name="_city_id__gt" value="1">

That field is added by this bit in the template: https://github.com/simonw/datasette/blob/515f8d38ebae203efc15ca79a8b42848276b35e5/datasette/templates/table.html#L119-L122

Which is populated here: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/views/table.py#L813-L821

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Columns starting with an underscore behave poorly in filters 1059555791  
1012629825 https://github.com/simonw/datasette/issues/1597#issuecomment-1012629825 https://api.github.com/repos/simonw/datasette/issues/1597 IC_kwDOBm6k_c48W4FB simonw 9599 2022-01-14T00:07:36Z 2022-01-14T00:07:36Z OWNER

Fixed:

% datasette --help        
Usage: datasette [OPTIONS] COMMAND [ARGS]...

  Datasette is an open source multi-tool for exploring and publishing data

  About Datasette: https://datasette.io/
  Full documentation: https://docs.datasette.io/

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  serve*     Serve up specified SQLite database files with a web UI
  inspect    Generate JSON summary of provided database files
...

% datasette inspect --help
Usage: datasette inspect [OPTIONS] [FILES]...

  Generate JSON summary of provided database files

  This can then be passed to "datasette --inspect-file" to speed up count
  operations against immutable database files.

Options:
  --inspect-file TEXT
  --load-extension TEXT  Path to a SQLite extension to load
  --help                 Show this message and exit.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 "datasette inspect" has no help summary 1102612922  
1012626243 https://github.com/simonw/datasette/issues/1595#issuecomment-1012626243 https://api.github.com/repos/simonw/datasette/issues/1595 IC_kwDOBm6k_c48W3ND simonw 9599 2022-01-14T00:00:33Z 2022-01-14T00:00:33Z OWNER

Copying those in here:

  • New plugin hook: filters_from_request(request, database, table, datasette), which runs on the table page and can be used to support new custom query string parameters that modify the SQL query. (#473)
  • The number of unique values in a facet is now always displayed. Previously it was only displayed if the user specified ?_facet_size=max. (#1556)
  • Fixed bug where ?_facet_array=tags&_facet=tags would only display one of the two selected facets. (#625)
  • Facets of type date or array can now be configured in metadata.json, see Facets in metadata.json. Thanks, David Larlet. (#1552)
  • New ?_nosuggest=1 parameter for table views, which disables facet suggestion. (#1557)
  • Label columns detected for foreign keys are now case-insensitive, so Name or TITLE will be detected in the same way as name or title. (#1544)
  • The query string variables exposed by request.args will now include blank strings for arguments such as foo in ?foo=&bar=1 rather than ignoring those parameters entirely. (#1551)
  • Database write connections now execute the prepare_connection(conn, database, datasette) plugin hook. (#1564)
  • The Datasette() constructor no longer requires the files= argument, and is now documented at Datasette class. (#1563)
  • The tracing feature now traces write queries, not just read queries. (#1568)
  • Added two methods for writing to the database: await db.execute_write_script(sql, block=False) and await db.execute_write_many(sql, params_seq, block=False). (#1570)
  • Made several performance improvements to the database schema introspection code that runs when Datasette first starts up. (#1555)
  • Fixed bug where writable canned queries could not be used with custom templates. (#1547)
 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Release notes for 0.60 1102484126  
1012583091 https://github.com/simonw/datasette/issues/1590#issuecomment-1012583091 https://api.github.com/repos/simonw/datasette/issues/1590 IC_kwDOBm6k_c48Wsqz simonw 9599 2022-01-13T22:41:15Z 2022-01-13T22:41:15Z OWNER

Seeing as this area of the code has produced so many bugs in the past, I think part of the fix may be to write comprehensive documentation about how routing works for the internals documentation. Doing so might help me figure this bug out!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Table+query JSON and CSV links broken when using `base_url` setting 1099723916  
1012575013 https://github.com/simonw/datasette/issues/1595#issuecomment-1012575013 https://api.github.com/repos/simonw/datasette/issues/1595 IC_kwDOBm6k_c48Wqsl simonw 9599 2022-01-13T22:29:22Z 2022-01-13T22:29:22Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Release notes for 0.60 1102484126  
1012554208 https://github.com/simonw/datasette/issues/1588#issuecomment-1012554208 https://api.github.com/repos/simonw/datasette/issues/1588 IC_kwDOBm6k_c48Wlng simonw 9599 2022-01-13T22:07:15Z 2022-01-13T22:07:15Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `explain query plan select` is too strict about whitespace 1097101917  
1012552760 https://github.com/simonw/datasette/issues/650#issuecomment-1012552760 https://api.github.com/repos/simonw/datasette/issues/650 IC_kwDOBm6k_c48WlQ4 simonw 9599 2022-01-13T22:04:56Z 2022-01-13T22:04:56Z OWNER

Challenge: explain the difference between view as in SQL view, and view as in the code that serves TableView / DatabaseView etc.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add a glossary to the documentation 534629631  
1012546924 https://github.com/simonw/datasette/issues/1336#issuecomment-1012546924 https://api.github.com/repos/simonw/datasette/issues/1336 IC_kwDOBm6k_c48Wj1s simonw 9599 2022-01-13T21:55:58Z 2022-01-13T21:55:58Z OWNER

See also:
- #1412

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Document turning on WAL for live served SQLite databases 895686039  
1012546223 https://github.com/simonw/datasette/issues/1466#issuecomment-1012546223 https://api.github.com/repos/simonw/datasette/issues/1466 IC_kwDOBm6k_c48Wjqv simonw 9599 2022-01-13T21:54:51Z 2022-01-13T21:54:51Z OWNER

Going with this for the copy:

Datasette Desktop is a packaged Mac application which bundles Datasette together with Python and allows you to install and run Datasette directly on your laptop. This is the best option for local installation if you are not comfortable using the command line.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add Datasette Desktop to installation documentation 991467558  
1012536257 https://github.com/simonw/datasette/issues/1594#issuecomment-1012536257 https://api.github.com/repos/simonw/datasette/issues/1594 IC_kwDOBm6k_c48WhPB simonw 9599 2022-01-13T21:38:48Z 2022-01-13T21:38:48Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add a CLI reference page to the docs, inspired by sqlite-utils 1102359726  
1012535024 https://github.com/simonw/datasette/issues/1594#issuecomment-1012535024 https://api.github.com/repos/simonw/datasette/issues/1594 IC_kwDOBm6k_c48Wg7w simonw 9599 2022-01-13T21:36:53Z 2022-01-13T21:36:53Z OWNER

I went with a simpler pattern that sqlite-utils because Datasette has a lot less commands: https://github.com/simonw/datasette/blob/4b23f01f3e668c8f2a2f1a294be49f49b4073969/docs/cli-reference.rst#L9-L35

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add a CLI reference page to the docs, inspired by sqlite-utils 1102359726  
1012508787 https://github.com/simonw/datasette/issues/1594#issuecomment-1012508787 https://api.github.com/repos/simonw/datasette/issues/1594 IC_kwDOBm6k_c48Wahz simonw 9599 2022-01-13T20:56:14Z 2022-01-13T20:56:34Z OWNER

The implementation can be almost exactly the same as this: https://github.com/simonw/sqlite-utils/blame/74586d3cb26fa3cc3412721985ecdc1864c2a31d/docs/cli-reference.rst#L11-L76

I need to do something extra to ensure the output of datasette --help-settings is shown too.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add a CLI reference page to the docs, inspired by sqlite-utils 1102359726  
1012506595 https://github.com/simonw/datasette/issues/1591#issuecomment-1012506595 https://api.github.com/repos/simonw/datasette/issues/1591 IC_kwDOBm6k_c48WZ_j simonw 9599 2022-01-13T20:52:56Z 2022-01-13T20:52:56Z OWNER

You can already run datasette --help-settings to see detailed help on available settings. Maybe datasette --help-plugin-settings could do the same thing for plugin settings?

Or I could even have available plugin settings show up as a list at the bottom of the datasette --help-settings output - which currently looks like this:

% datasette --help-settings
Settings:
  default_page_size            Default page size for the table view
                               (default=100)
  max_returned_rows            Maximum rows that can be returned from a table
                               or custom query (default=1000)
  num_sql_threads              Number of threads in the thread pool for
                               executing SQLite queries (default=3)
  sql_time_limit_ms            Time limit for a SQL query in milliseconds
                               (default=1000)
  default_facet_size           Number of values to return for requested facets
                               (default=30)
  facet_time_limit_ms          Time limit for calculating a requested facet
                               (default=200)
  facet_suggest_time_limit_ms  Time limit for calculating a suggested facet
                               (default=50)
  hash_urls                    Include DB file contents hash in URLs, for far-
                               future caching (default=False)
  allow_facet                  Allow users to specify columns to facet using
                               ?_facet= parameter (default=True)
  allow_download               Allow users to download the original SQLite
                               database files (default=True)
  suggest_facets               Calculate and display suggested facets
                               (default=True)
  default_cache_ttl            Default HTTP cache TTL (used in Cache-Control:
                               max-age= header) (default=5)
  default_cache_ttl_hashed     Default HTTP cache TTL for hashed URL pages
                               (default=31536000)
  cache_size_kb                SQLite cache size in KB (0 == use SQLite
                               default) (default=0)
  allow_csv_stream             Allow .csv?_stream=1 to download all rows
                               (ignoring max_returned_rows) (default=True)
  max_csv_mb                   Maximum size allowed for CSV export in MB - set
                               0 to disable this limit (default=100)
  truncate_cells_html          Truncate cells longer than this in HTML table
                               view - set 0 to disable (default=2048)
  force_https_urls             Force URLs in API output to always use https://
                               protocol (default=False)
  template_debug               Allow display of template debug information
                               with ?_context=1 (default=False)
  trace_debug                  Allow display of SQL trace debug information
                               with ?_trace=1 (default=False)
  base_url                     Datasette URLs should use this base path
                               (default=/)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Maybe let plugins define custom serve options? 1100015398  
1012505706 https://github.com/simonw/datasette/issues/1591#issuecomment-1012505706 https://api.github.com/repos/simonw/datasette/issues/1591 IC_kwDOBm6k_c48WZxq simonw 9599 2022-01-13T20:51:30Z 2022-01-13T20:51:30Z OWNER

Another option: if I make plugin settings a higher level concept in Datasette than they are at the moment, I could allow them to be set either using --options OR using the existing metadata.yml/json mechanism. https://docs.datasette.io/en/stable/plugins.html#plugin-configuration

I want to make changes to that anyway, because I'm increasingly uncomfortable with plugin settings ending up in the "metadata" mechanism.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Maybe let plugins define custom serve options? 1100015398  
1012504251 https://github.com/simonw/datasette/issues/1591#issuecomment-1012504251 https://api.github.com/repos/simonw/datasette/issues/1591 IC_kwDOBm6k_c48WZa7 simonw 9599 2022-01-13T20:49:19Z 2022-01-13T20:49:59Z OWNER

I try to stick pretty closely to what Click supports, and Click likes you to define options explicitly so that it can display them in the output of --help.

But... that makes me think that actually showing these options in --help is likely a better idea. My --plugin-setting concept would have help that looks something like this:

% datasette serve --help
  ...
  --plugin-setting <TEXT TEXT TEXT>...     Setting for a specified plugin.

That's not great help!

The alternative would be to allow plugins to register their extra options with the command - which would mean the help output could look like this instead:

% datasette serve --help
  ...
  --tiddlywiki-db <TEXT>    Name of database to use for datasette-tiddlywiki

This feels like a good argument to me for plugins to explicitly register their settings. I'm not sure if I should enforce the name-of-plugin- prefix, or if I should allow plugins to define any setting they like. I'm still nervous about plugins over-riding existing or future default options to that command.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Maybe let plugins define custom serve options? 1100015398  
1012413729 https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012413729 https://api.github.com/repos/simonw/sqlite-utils/issues/79 IC_kwDOCGYnMM48WDUh eyeseast 25778 2022-01-13T18:50:00Z 2022-01-13T18:50:00Z NONE

One more thing I'm going to add: A method to add a geometry column, which I'll need to do to create a spatial index on a table.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Helper methods for working with SpatiaLite 557842245  
1012253198 https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012253198 https://api.github.com/repos/simonw/sqlite-utils/issues/79 IC_kwDOCGYnMM48VcIO eyeseast 25778 2022-01-13T15:39:14Z 2022-01-13T15:39:14Z NONE

Other thing: If there get to be enough utils, I think it's worth moving all the spatialite stuff into its own file (gis.py or something) just so it's easier to find later.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Helper methods for working with SpatiaLite 557842245  
1012230212 https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012230212 https://api.github.com/repos/simonw/sqlite-utils/issues/79 IC_kwDOCGYnMM48VWhE eyeseast 25778 2022-01-13T15:15:13Z 2022-01-13T15:15:13Z NONE

Some proposals I'd add to sqlite-utils:

Some version of this, from geojson-to-sqlite:

def init_spatialite(db, lib):
    db.conn.enable_load_extension(True)
    db.conn.load_extension(lib)
    # Initialize SpatiaLite if not yet initialized
    if "spatial_ref_sys" in db.table_names():
        return
    db.conn.execute("select InitSpatialMetadata(1)")

Also a function for creating a spatial index:

db.conn.execute("select CreateSpatialIndex(?, ?)", [table, "geometry"])

I don't know the nuances of updating a spatial index, or checking if one already exists. This could be a CLI method like:

sqlite-utils spatial-index spatial.db table-name column-name

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Helper methods for working with SpatiaLite 557842245  
1012158895 https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012158895 https://api.github.com/repos/simonw/sqlite-utils/issues/79 IC_kwDOCGYnMM48VFGv eyeseast 25778 2022-01-13T13:55:59Z 2022-01-13T13:55:59Z NONE

Came here to add this. I might pick it up.

Would also add a utility to create (and update and delete?) a spatial index. It's not much code but I have to look it up every time.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Helper methods for working with SpatiaLite 557842245  
1012128696 https://github.com/simonw/datasette/pull/1593#issuecomment-1012128696 https://api.github.com/repos/simonw/datasette/issues/1593 IC_kwDOBm6k_c48U9u4 codecov[bot] 22429695 2022-01-13T13:18:35Z 2022-01-13T13:18:35Z NONE

Codecov Report

Merging #1593 (df73ebb) into main (8c401ee) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1593   +/-   ##
=======================================
  Coverage   92.09%   92.09%           
=======================================
  Files          34       34           
  Lines        4516     4516           
=======================================
  Hits         4159     4159           
  Misses        357      357

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8c401ee...df73ebb. Read the comment docs.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Update pytest-asyncio requirement from <0.17,>=0.10 to >=0.10,<0.18 1101705012  
1011855133 https://github.com/simonw/sqlite-utils/issues/348#issuecomment-1011855133 https://api.github.com/repos/simonw/sqlite-utils/issues/348 IC_kwDOCGYnMM48T68d simonw 9599 2022-01-13T07:06:59Z 2022-01-13T07:06:59Z OWNER

Wrote a lot more about this feature here: https://simonwillison.net/2022/Jan/12/how-i-build-a-feature/

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Command for creating an empty database 1067771698  
1011185061 https://github.com/simonw/datasette/issues/1592#issuecomment-1011185061 https://api.github.com/repos/simonw/datasette/issues/1592 IC_kwDOBm6k_c48RXWl simonw 9599 2022-01-12T15:50:41Z 2022-01-12T15:50:41Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Row pages should show links to foreign keys 1100499619  
1010947634 https://github.com/simonw/datasette/issues/1591#issuecomment-1010947634 https://api.github.com/repos/simonw/datasette/issues/1591 IC_kwDOBm6k_c48QdYy psychemedia 82988 2022-01-12T11:32:17Z 2022-01-12T11:32:17Z CONTRIBUTOR

Is it possible to parse things like --ext-{plugin}-{arg} VALUE ?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Maybe let plugins define custom serve options? 1100015398  
1010764036 https://github.com/simonw/datasette/issues/1591#issuecomment-1010764036 https://api.github.com/repos/simonw/datasette/issues/1591 IC_kwDOBm6k_c48PwkE simonw 9599 2022-01-12T08:22:16Z 2022-01-12T08:22:32Z OWNER

The challenge here is avoiding clashes. What if a plugin adds an option that I later want to use for a new Datasette core feature? Or what if two plugins define the same option?

Maybe the solution is to make them use namespaces defined by the plugin name.

How about this:

datasette t.db --plugin-setting datasette-tiddlywiki db t.db

It's a bit verbose having an option that itself then takes THREE strings: plugin name, setting name, setting value - but it would work.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Maybe let plugins define custom serve options? 1100015398  
1010559681 https://github.com/simonw/datasette/issues/1590#issuecomment-1010559681 https://api.github.com/repos/simonw/datasette/issues/1590 IC_kwDOBm6k_c48O-rB eelkevdbos 1001306 2022-01-12T02:10:20Z 2022-01-12T02:10:20Z NONE

In my example, path matching happens at the application layer (being the Django channels URLRouter). That might be a somewhat exotic solution that would normally be solved by a proxy like Apache or Nginx. However, in my specific use case, this is a "feature" enabling me to do simple management of databases and metadata from within a Django admin app instance mapped in that same router.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Table+query JSON and CSV links broken when using `base_url` setting 1099723916  
1010556333 https://github.com/simonw/datasette/issues/1590#issuecomment-1010556333 https://api.github.com/repos/simonw/datasette/issues/1590 IC_kwDOBm6k_c48O92t eelkevdbos 1001306 2022-01-12T02:03:59Z 2022-01-12T02:03:59Z NONE

Thank you for the quick reply! Just a quick observation, I am running this locally without a proxy, whereas your fly example seems to be running behind an apache proxy (if the name is accurate). Can it be that the apache proxy strips the prefix before it passes on the request to the daphne backend?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Table+query JSON and CSV links broken when using `base_url` setting 1099723916  
1010540923 https://github.com/simonw/datasette/issues/1590#issuecomment-1010540923 https://api.github.com/repos/simonw/datasette/issues/1590 IC_kwDOBm6k_c48O6F7 simonw 9599 2022-01-12T01:33:49Z 2022-01-12T01:33:49Z OWNER

Looking closer at the code quoted above, it doesn't modify path or raw_path at all - ALL it does is add the route_path to the scope.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Table+query JSON and CSV links broken when using `base_url` setting 1099723916  
1010538188 https://github.com/simonw/datasette/issues/1590#issuecomment-1010538188 https://api.github.com/repos/simonw/datasette/issues/1590 IC_kwDOBm6k_c48O5bM simonw 9599 2022-01-12T01:28:41Z 2022-01-12T01:30:43Z OWNER

Oh wait! It looks like route_path is something I invented there.

Yup, I added it in https://github.com/simonw/datasette/commit/a63412152518581c6a3d4e142b937e27dabdbfdb - commit message says:

  • new route_path key in request.scope storing the path that was used for routing with the base_url prefix stripped

So actually part of the mystery here is: why does the Fly hosted one NOT have that key?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Table+query JSON and CSV links broken when using `base_url` setting 1099723916  
1010538016 https://github.com/simonw/datasette/issues/1590#issuecomment-1010538016 https://api.github.com/repos/simonw/datasette/issues/1590 IC_kwDOBm6k_c48O5Yg simonw 9599 2022-01-12T01:28:19Z 2022-01-12T01:28:19Z OWNER

The Daphne one has this key: 'route_path': '/-/asgi-scope',

Maybe Datasette's routing code needs to look out for that, if it's available, and use it to reconstruct the requested path?

The code in question is here: https://github.com/simonw/datasette/blob/8c401ee0f054de2f568c3a8302c9223555146407/datasette/app.py#L1143-L1149

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Table+query JSON and CSV links broken when using `base_url` setting 1099723916  
1010537058 https://github.com/simonw/datasette/issues/1590#issuecomment-1010537058 https://api.github.com/repos/simonw/datasette/issues/1590 IC_kwDOBm6k_c48O5Ji simonw 9599 2022-01-12T01:26:34Z 2022-01-12T01:26:34Z OWNER

I'm using the https://datasette.io/plugins/datasette-debug-asgi plugin to investigate.

On my laptop using Daphne I get this: http://127.0.0.1:8032/datasettes/-/asgi-scope

{'actor': None,
 'asgi': {'version': '3.0'},
 'client': ['127.0.0.1', 53767],
 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x1122aeef0>,
 'headers': [(b'host', b'127.0.0.1:8032'),
             (b'user-agent',
              b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:95.0) Gecko'
              b'/20100101 Firefox/95.0'),
             (b'accept',
              b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
              b'avif,image/webp,*/*;q=0.8'),
             (b'accept-language', b'en-US,en;q=0.5'),
             (b'accept-encoding', b'gzip, deflate'),
             (b'dnt', b'1'),
             (b'connection', b'keep-alive'),
             (b'cookie', b'_ga=GA1.1.742283954.1628542653'),
             (b'upgrade-insecure-requests', b'1'),
             (b'sec-fetch-dest', b'document'),
             (b'sec-fetch-mode', b'navigate'),
             (b'sec-fetch-site', b'none'),
             (b'sec-fetch-user', b'?1')],
 'http_version': '1.1',
 'method': 'GET',
 'path': '/datasettes/-/asgi-scope',
 'path_remaining': '',
 'query_string': b'',
 'raw_path': b'/datasettes/-/asgi-scope',
 'root_path': '',
 'route_path': '/-/asgi-scope',
 'scheme': 'http',
 'server': ['127.0.0.1', 8032],
 'type': 'http',
 'url_route': {'kwargs': {}}}

On the demo running on Fly (which I just redeployed with that plugin) I get this: https://datasette-apache-proxy-demo.fly.dev/prefix/-/asgi-scope

{'actor': None,
 'asgi': {'spec_version': '2.1', 'version': '3.0'},
 'client': ('86.109.12.167', 0),
 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x7f4c0413bca0>,
 'headers': [(b'host', b'datasette-apache-proxy-demo.fly.dev'),
             (b'user-agent',
              b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:95.0) Gecko'
              b'/20100101 Firefox/95.0'),
             (b'accept',
              b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
              b'avif,image/webp,*/*;q=0.8'),
             (b'accept-language', b'en-US,en;q=0.5'),
             (b'accept-encoding', b'gzip, deflate, br'),
             (b'dnt', b'1'),
             (b'x-request-start', b't=1641950740651658'),
             (b'sec-fetch-dest', b'document'),
             (b'sec-fetch-mode', b'navigate'),
             (b'sec-fetch-site', b'none'),
             (b'sec-fetch-user', b'?1'),
             (b'fly-client-ip', b'24.5.172.176'),
             (b'x-forwarded-for',
              b'24.5.172.176, 213.188.193.173, 86.109.12.167'),
             (b'fly-forwarded-proto', b'https'),
             (b'x-forwarded-proto', b'https'),
             (b'fly-forwarded-ssl', b'on'),
             (b'x-forwarded-ssl', b'on'),
             (b'fly-forwarded-port', b'443'),
             (b'x-forwarded-port', b'443'),
             (b'fly-region', b'sjc'),
             (b'fly-request-id', b'01FS5Y805BX43HM94T8XW610KG'),
             (b'via', b'2 fly.io'),
             (b'fly-dispatch-start', b't=1641950740683198;instance=87f188a2'),
             (b'x-forwarded-host', b'datasette-apache-proxy-demo.fly.dev'),
             (b'x-forwarded-server', b'localhost'),
             (b'connection', b'Keep-Alive')],
 'http_version': '1.1',
 'method': 'GET',
 'path': '/-/asgi-scope',
 'query_string': b'',
 'raw_path': b'/-/asgi-scope',
 'root_path': '',
 'scheme': 'https',
 'server': ('127.0.0.1', 8001),
 'type': 'http',
 'url_route': {'kwargs': {}}}

The version that works as 'raw_path': b'/-/asgi-scope' - the version that fails has 'raw_path': b'/datasettes/-/asgi-scope'.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Table+query JSON and CSV links broken when using `base_url` setting 1099723916  
1010533133 https://github.com/simonw/datasette/issues/1590#issuecomment-1010533133 https://api.github.com/repos/simonw/datasette/issues/1590 IC_kwDOBm6k_c48O4MN simonw 9599 2022-01-12T01:19:19Z 2022-01-12T01:19:19Z OWNER

Thanks for the steps to reproduce - I have your bug running on my laptop now.

I've been mostly testing this stuff using the hosted copy of Datasette here, which doesn't exhibit the bug: https://datasette-apache-proxy-demo.fly.dev/prefix/fixtures?sql=select+sqlite_version%28%29

Something interesting definitely going on here!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Table+query JSON and CSV links broken when using `base_url` setting 1099723916  
1010462035 https://github.com/simonw/sqlite-utils/issues/381#issuecomment-1010462035 https://api.github.com/repos/simonw/sqlite-utils/issues/381 IC_kwDOCGYnMM48Om1T simonw 9599 2022-01-11T23:33:37Z 2022-01-11T23:33:37Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `sqlite-utils rows` options `--limit` and `--offset` 1099584685  
1010461844 https://github.com/simonw/sqlite-utils/issues/382#issuecomment-1010461844 https://api.github.com/repos/simonw/sqlite-utils/issues/382 IC_kwDOCGYnMM48OmyU simonw 9599 2022-01-11T23:33:14Z 2022-01-11T23:33:14Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `--where` option for `sqlite-rows` 1099585611  
1010441118 https://github.com/simonw/sqlite-utils/issues/381#issuecomment-1010441118 https://api.github.com/repos/simonw/sqlite-utils/issues/381 IC_kwDOCGYnMM48Ohue simonw 9599 2022-01-11T22:56:53Z 2022-01-11T22:57:09Z OWNER

sqlite-utils search has --limit already: https://sqlite-utils.datasette.io/en/latest/cli-reference.html#search

  --limit INTEGER        Number of rows to return - defaults to everything

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `sqlite-utils rows` options `--limit` and `--offset` 1099584685  
1010440166 https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010440166 https://api.github.com/repos/simonw/sqlite-utils/issues/383 IC_kwDOCGYnMM48Ohfm simonw 9599 2022-01-11T22:55:05Z 2022-01-11T22:55:05Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add documentation page with the output of `--help` 1099586786  
1010387223 https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010387223 https://api.github.com/repos/simonw/sqlite-utils/issues/383 IC_kwDOCGYnMM48OUkX simonw 9599 2022-01-11T21:45:32Z 2022-01-11T21:45:32Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add documentation page with the output of `--help` 1099586786  
1010386802 https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010386802 https://api.github.com/repos/simonw/sqlite-utils/issues/383 IC_kwDOCGYnMM48OUdy simonw 9599 2022-01-11T21:44:53Z 2022-01-11T21:44:53Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add documentation page with the output of `--help` 1099586786  
1010333511 https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010333511 https://api.github.com/repos/simonw/sqlite-utils/issues/383 IC_kwDOCGYnMM48OHdH simonw 9599 2022-01-11T20:27:08Z 2022-01-11T20:27:08Z OWNER

I'll call the new page "CLI reference", for consistency with the API reference page here: https://sqlite-utils.datasette.io/en/stable/reference.html

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Add documentation page with the output of `--help` 1099586786  
1009548580 https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1009548580 https://api.github.com/repos/simonw/sqlite-utils/issues/365 IC_kwDOCGYnMM48LH0k fgregg 536941 2022-01-11T02:43:34Z 2022-01-11T02:43:34Z NONE

thanks so much! always a pleasure to see how you work through these things

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 create-index should run analyze after creating index 1096558279  
1009544785 https://github.com/simonw/sqlite-utils/issues/380#issuecomment-1009544785 https://api.github.com/repos/simonw/sqlite-utils/issues/380 IC_kwDOCGYnMM48LG5R simonw 9599 2022-01-11T02:32:56Z 2022-01-11T02:32:56Z OWNER

CLI and Python library improvements to help run ANALYZE after creating indexes or inserting rows, to gain better performance from the SQLite query planner when it runs against indexes.

Three new CLI commands: create-database, analyze and bulk.

  • New sqlite-utils create-database command for creating new empty database files. (#348)
  • New Python methods for running ANALYZE against a database, table or index: db.analyze() and table.analyze(), see Optimizing index usage with ANALYZE. (#366)
  • New sqlite-utils analyze command for running ANALYZE using the CLI. (#379)
  • The create-index, insert and update commands now have a new --analyze option for running ANALYZE after the command has completed. (#379)
  • New sqlite-utils bulk command which can import records in the same way as sqlite-utils insert (from JSON, CSV or TSV) and use them to bulk execute a parametrized SQL query. (#375)
  • The CLI tool can now also be run using python -m sqlite_utils. (#368)
  • Using --fmt now implies --table, so you don't need to pass both options. (#374)
  • The --convert function applied to rows can now modify the row in place. (#371)
  • The insert-files command supports two new columns: stem and suffix. (#372)
  • The --nl import option now ignores blank lines in the input. (#376)
  • Fixed bug where streaming input to the insert command with --batch-size 1 would appear to only commit after several rows had been ingested, due to unnecessary input buffering. (#364)
 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Release notes for 3.21 1098574572  
1009536276 https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1009536276 https://api.github.com/repos/simonw/sqlite-utils/issues/375 IC_kwDOCGYnMM48LE0U simonw 9599 2022-01-11T02:12:58Z 2022-01-11T02:12:58Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `sqlite-utils bulk` command 1097251014  
1009534817 https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009534817 https://api.github.com/repos/simonw/sqlite-utils/issues/377 IC_kwDOCGYnMM48LEdh simonw 9599 2022-01-11T02:09:38Z 2022-01-11T02:09:38Z OWNER

I tested this like so:

% wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv'
% sqlite-utils create-database test.db
% sqlite-utils create-table test.db power_plants url text owner text
% sqlite-utils schema test.db 
CREATE TABLE [power_plants] (
   [url] TEXT,
   [owner] TEXT
);
% sqlite-utils bulk test.db 'insert into power_plants (url, owner) values (:url, :owner)' global_power_plant_database.csv --csv
  [------------------------------------]    0%
  [###################################-]   99%
% sqlite-utils tables --counts test.db -t
table           count
------------  -------
power_plants    33643

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `sqlite-utils bulk` command 1097477582  
1009532125 https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009532125 https://api.github.com/repos/simonw/sqlite-utils/issues/377 IC_kwDOCGYnMM48LDzd simonw 9599 2022-01-11T02:03:35Z 2022-01-11T02:03:35Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `sqlite-utils bulk` command 1097477582  
1009531863 https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009531863 https://api.github.com/repos/simonw/sqlite-utils/issues/377 IC_kwDOCGYnMM48LDvX codecov[bot] 22429695 2022-01-11T02:03:00Z 2022-01-11T02:03:00Z NONE

Codecov Report

Merging #377 (f4ea0d3) into main (b6dad08) will decrease coverage by 0.01%.
The diff coverage is 96.00%.

@@            Coverage Diff             @@
##             main     #377      +/-   ##
==========================================
- Coverage   96.52%   96.50%   -0.02%     
==========================================
  Files           6        6              
  Lines        2330     2378      +48     
==========================================
+ Hits         2249     2295      +46     
- Misses         81       83       +2
<table> <thead> <tr> <th>Impacted Files</th> <th>Coverage Δ</th> <th></th> </tr> </thead> <tbody> <tr> <td>sqlite_utils/cli.py</td> <td>95.49% <94.28%> (-0.06%)</td> <td>:arrow_down:</td> </tr> <tr> <td>sqlite_utils/db.py</td> <td>97.68% <100.00%> (+0.03%)</td> <td>:arrow_up:</td> </tr> </tbody> </table>

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b6dad08...f4ea0d3. Read the comment docs.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `sqlite-utils bulk` command 1097477582  
1009521921 https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1009521921 https://api.github.com/repos/simonw/sqlite-utils/issues/365 IC_kwDOCGYnMM48LBUB simonw 9599 2022-01-11T01:37:53Z 2022-01-11T01:37:53Z OWNER

I decided to go with making this opt-in, mainly for consistency with the other places where I added this feature - see:
- #379
- #366

You can now run the following:

sqlite-utils create-index mydb.db mytable mycolumn --analyze

And ANALYZE will be run on the index once it has been created.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 create-index should run analyze after creating index 1096558279  
1009508865 https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009508865 https://api.github.com/repos/simonw/sqlite-utils/issues/366 IC_kwDOCGYnMM48K-IB simonw 9599 2022-01-11T01:08:51Z 2022-01-11T01:08:51Z OWNER

The Python methods are all done now, next step is the CLI options. I'll do those in a separate issue.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Python library methods for calling ANALYZE 1096563265  
1009288898 https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009288898 https://api.github.com/repos/simonw/sqlite-utils/issues/366 IC_kwDOCGYnMM48KIbC simonw 9599 2022-01-10T19:54:04Z 2022-01-10T19:54:04Z OWNER

Having browsed the API reference I think the methods that would benefit from an analyze=True parameter are:

  • db.create_index
  • table.insert_all
  • table.upsert_all
  • table.delete_where
 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Python library methods for calling ANALYZE 1096563265  
1009285627 https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009285627 https://api.github.com/repos/simonw/sqlite-utils/issues/366 IC_kwDOCGYnMM48KHn7 simonw 9599 2022-01-10T19:49:19Z 2022-01-10T19:51:25Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Python library methods for calling ANALYZE 1096563265  
1009286373 https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009286373 https://api.github.com/repos/simonw/sqlite-utils/issues/366 IC_kwDOCGYnMM48KHzl simonw 9599 2022-01-10T19:50:22Z 2022-01-10T19:50:22Z OWNER

With respect to #365, I'm now thinking that having the ability to say "... and then run ANALYZE" could be useful for a bunch of Python methods. For example:

db["dogs"].insert_all(list_of_dogs, analyze=True)
db["dogs"].create_index(["name"], analyze=True)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Python library methods for calling ANALYZE 1096563265  
1009273525 https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009273525 https://api.github.com/repos/simonw/sqlite-utils/issues/366 IC_kwDOCGYnMM48KEq1 simonw 9599 2022-01-10T19:32:39Z 2022-01-10T19:32:39Z OWNER

I'm going to implement the Python library methods based on the prototype:

commit 650f97a08f29a688c530e5f6c9eedc9269ed7bdc
Author: Simon Willison <swillison@gmail.com>
Date:   Sat Jan 8 13:34:01 2022 -0800

    Initial prototype of .analyze(), refs #366

diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
index dfc4723..1348b4a 100644
--- a/sqlite_utils/db.py
+++ b/sqlite_utils/db.py
@@ -923,6 +923,13 @@ class Database:
         "Run a SQLite ``VACUUM`` against the database."
         self.execute("VACUUM;")

+    def analyze(self, name=None):
+        "Run ``ANALYZE`` against the entire database or a named table or index."
+        sql = "ANALYZE"
+        if name is not None:
+            sql += " [{}]".format(name)
+        self.execute(sql)
+

 class Queryable:
     def exists(self) -> bool:
@@ -2902,6 +2909,10 @@ class Table(Queryable):
             )
         return self

+    def analyze(self):
+        "Run ANALYZE against this table"
+        self.db.analyze(self.name)
+
     def analyze_column(
         self, column: str, common_limit: int = 10, value_truncate=None, total_rows=None
     ) -> "ColumnDetails":

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Python library methods for calling ANALYZE 1096563265  
1009272446 https://github.com/simonw/sqlite-utils/pull/367#issuecomment-1009272446 https://api.github.com/repos/simonw/sqlite-utils/issues/367 IC_kwDOCGYnMM48KEZ- simonw 9599 2022-01-10T19:31:08Z 2022-01-10T19:31:08Z OWNER

I'm going to implement this in a separate commit from this PR.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Initial prototype of .analyze() methods 1097041471  
1008557414 https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008557414 https://api.github.com/repos/simonw/sqlite-utils/issues/364 IC_kwDOCGYnMM48HV1m simonw 9599 2022-01-10T05:36:19Z 2022-01-10T05:36:19Z OWNER

That did the trick.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `--batch-size 1` doesn't seem to commit for every item 1095570074  
1008556706 https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1008556706 https://api.github.com/repos/simonw/sqlite-utils/issues/375 IC_kwDOCGYnMM48HVqi simonw 9599 2022-01-10T05:33:41Z 2022-01-10T05:33:41Z OWNER

I tested the prototype like this:

sqlite-utils blah.db 'create table blah (id integer primary key, name text)' 
echo 'id,name
1,Cleo
2,Chicken' > blah.csv
sqlite-utils bulk blah.db 'insert into blah (id, name) values (:id, :name)' blah.csv --csv

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `sqlite-utils bulk` command 1097251014  
1008546573 https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008546573 https://api.github.com/repos/simonw/sqlite-utils/issues/364 IC_kwDOCGYnMM48HTMN simonw 9599 2022-01-10T05:05:15Z 2022-01-10T05:05:15Z OWNER

Bit nasty but it might work:

    def try_until(expected):
        tries = 0
        while True:
            rows = list(Database(db_path)["rows"].rows)
            if rows == expected:
                return
            tries += 1
            if tries > 10:
                assert False, "Expected {}, got {}".format(expected, rows)
            time.sleep(tries * 0.1)

    try_until([{"name": "Azi"}])
    proc.stdin.write(b'{"name": "Suna"}\n')
    proc.stdin.flush()
    try_until([{"name": "Azi"}, {"name": "Suna"}])

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `--batch-size 1` doesn't seem to commit for every item 1095570074  
1008545140 https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008545140 https://api.github.com/repos/simonw/sqlite-utils/issues/364 IC_kwDOCGYnMM48HS10 simonw 9599 2022-01-10T05:01:34Z 2022-01-10T05:01:34Z OWNER

Urgh, tests are still failing intermittently - for example:

        time.sleep(0.4)
>       assert list(Database(db_path)["rows"].rows) == [{"name": "Azi"}]
E       AssertionError: assert [] == [{'name': 'Azi'}]
E         Right contains one more item: {'name': 'Azi'}
E         Full diff:
E         - [{'name': 'Azi'}]
E         + []

I'm going to change this code to keep on trying up to 10 seconds - that should get the tests to pass faster on most machines.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `--batch-size 1` doesn't seem to commit for every item 1095570074  
1008537194 https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008537194 https://api.github.com/repos/simonw/sqlite-utils/issues/364 IC_kwDOCGYnMM48HQ5q simonw 9599 2022-01-10T04:29:53Z 2022-01-10T04:31:29Z OWNER

After a bunch of debugging with print() statements it's clear that the problem isn't with when things are committed or the size of the batches - it's that the data sent to standard input is all being processed in one go, not a line at a time.

I think that's because it is being buffered by this: https://github.com/simonw/sqlite-utils/blob/d2a79d200f9071a86027365fa2a576865b71064f/sqlite_utils/cli.py#L759-L770

The buffering is there so that we can sniff the first few bytes to detect if it's a CSV file - added in 99ff0a288c08ec2071139c6031eb880fa9c95310 for #230. So maybe for non-CSV inputs we should disable buffering?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `--batch-size 1` doesn't seem to commit for every item 1095570074  
1008526736 https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008526736 https://api.github.com/repos/simonw/sqlite-utils/issues/364 IC_kwDOCGYnMM48HOWQ simonw 9599 2022-01-10T04:07:29Z 2022-01-10T04:07:29Z OWNER

I think this test is right:

def test_insert_streaming_batch_size_1(db_path):
    # https://github.com/simonw/sqlite-utils/issues/364
    # Streaming with --batch-size 1 should commit on each record
    # Can't use CliRunner().invoke() here bacuse we need to
    # run assertions in between writing to process stdin
    proc = subprocess.Popen(
        [
            sys.executable,
            "-m",
            "sqlite_utils",
            "insert",
            db_path,
            "rows",
            "-",
            "--nl",
            "--batch-size",
            "1",
        ],
        stdin=subprocess.PIPE,
    )
    proc.stdin.write(b'{"name": "Azi"}')
    proc.stdin.flush()
    assert list(Database(db_path)["rows"].rows) == [{"name": "Azi"}]
    proc.stdin.write(b'{"name": "Suna"}')
    proc.stdin.flush()
    assert list(Database(db_path)["rows"].rows) == [{"name": "Azi"}, {"name": "Suna"}]
    proc.stdin.close()
    proc.wait()

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `--batch-size 1` doesn't seem to commit for every item 1095570074  
1008383293 https://github.com/simonw/sqlite-utils/issues/348#issuecomment-1008383293 https://api.github.com/repos/simonw/sqlite-utils/issues/348 IC_kwDOCGYnMM48GrU9 simonw 9599 2022-01-09T20:38:17Z 2022-01-09T20:38:17Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Command for creating an empty database 1067771698  
1008367607 https://github.com/simonw/sqlite-utils/issues/348#issuecomment-1008367607 https://api.github.com/repos/simonw/sqlite-utils/issues/348 IC_kwDOCGYnMM48Gnf3 simonw 9599 2022-01-09T20:22:43Z 2022-01-09T20:22:43Z OWNER

I'm not going to implement --page-size unless someone specifically requests it - I don't like having features that I've never needed to use myself.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Command for creating an empty database 1067771698  
1008364701 https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008364701 https://api.github.com/repos/simonw/sqlite-utils/issues/371 IC_kwDOCGYnMM48Gmyd simonw 9599 2022-01-09T20:04:35Z 2022-01-09T20:04:35Z OWNER

The previous code for highlighting errors in syntax (which was already a bit confused thanks to the added return, see https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991393684 - isn't compatible with this approach at all. I'm going to ditch it and just show a generic Error: Could not compile code message.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Support mutating row in `--convert` without returning it 1097128334  
1008354207 https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008354207 https://api.github.com/repos/simonw/sqlite-utils/issues/371 IC_kwDOCGYnMM48GkOf simonw 9599 2022-01-09T18:54:54Z 2022-01-09T18:54:54Z OWNER

This seems to work:

def _compile_code(code, imports, variable="value"):
    locals = {}
    globals = {"r": recipes, "recipes": recipes}
    # If user defined a convert() function, return that
    try:
        exec(code, globals, locals)
        return locals["convert"]
    except (AttributeError, SyntaxError, NameError, KeyError, TypeError):
        pass

    # Try compiling their code as a function instead
    body_variants = [code]
    # If single line and no 'return', try adding the return
    if "\n" not in code and not code.strip().startswith("return "):
        body_variants.insert(0, "return {}".format(code))

    for variant in body_variants:
        new_code = ["def fn({}):".format(variable)]
        for line in variant.split("\n"):
            new_code.append("    {}".format(line))
        try:
            code_o = compile("\n".join(new_code), "<string>", "exec")
            break
        except SyntaxError:
            # Try another variant, e.g. for 'return row["column"] = 1'
            continue

    for import_ in imports:
        globals[import_.split(".")[0]] = __import__(import_)
    exec(code_o, globals, locals)
    return locals["fn"]

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Support mutating row in `--convert` without returning it 1097128334  
1008348032 https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008348032 https://api.github.com/repos/simonw/sqlite-utils/issues/371 IC_kwDOCGYnMM48GiuA simonw 9599 2022-01-09T18:14:02Z 2022-01-09T18:14:02Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Support mutating row in `--convert` without returning it 1097128334  
1008347768 https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008347768 https://api.github.com/repos/simonw/sqlite-utils/issues/371 IC_kwDOCGYnMM48Gip4 simonw 9599 2022-01-09T18:12:30Z 2022-01-09T18:12:30Z OWNER

Tried this test:

    result = CliRunner().invoke(
        cli.cli,
        [
            "insert",
            db_path,
            "rows",
            "-",
            "--convert",
            'row["is_chicken"] = True',
        ],
        input='{"name": "Azi"}',
    )

And got this error:

E + where 1 = <Result SyntaxError('invalid syntax', ('<string>', 2, 30, ' return row["is_chicken"] = True\n'))>.exit_code

The code snippet compilation isn't currently compatible with this.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Support mutating row in `--convert` without returning it 1097128334  
1008346841 https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008346841 https://api.github.com/repos/simonw/sqlite-utils/issues/374 IC_kwDOCGYnMM48GibZ simonw 9599 2022-01-09T18:06:50Z 2022-01-09T18:06:50Z OWNER

In addition to a unit test I manually tested all of the above, e.g.

% sqlite-utils indexes global-power-plants.db sqlite_master --fmt rst
=======  ============  =======  =====  ======  ======  ======  =====
table    index_name    seqno    cid    name    desc    coll    key
=======  ============  =======  =====  ======  ======  ======  =====
=======  ============  =======  =====  ======  ======  ======  =====

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `--fmt` should imply `-t` 1097135860  
1008346338 https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008346338 https://api.github.com/repos/simonw/sqlite-utils/issues/374 IC_kwDOCGYnMM48GiTi simonw 9599 2022-01-09T18:03:22Z 2022-01-09T18:03:22Z OWNER

Commands that support --fmt (via the @output_options decorator) are:

  • tables
  • views
  • query
  • memory
  • search
  • rows
  • triggers
  • indexes
 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `--fmt` should imply `-t` 1097135860  
1008345267 https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008345267 https://api.github.com/repos/simonw/sqlite-utils/issues/374 IC_kwDOCGYnMM48GiCz simonw 9599 2022-01-09T17:56:37Z 2022-01-09T17:56:37Z OWNER

Better:

if fmt:
    table = True

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `--fmt` should imply `-t` 1097135860  
1008344980 https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008344980 https://api.github.com/repos/simonw/sqlite-utils/issues/373 IC_kwDOCGYnMM48Gh-U simonw 9599 2022-01-09T17:54:53Z 2022-01-09T17:54:53Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 List `--fmt` options in the docs  1097135732  
1008344525 https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008344525 https://api.github.com/repos/simonw/sqlite-utils/issues/373 IC_kwDOCGYnMM48Gh3N simonw 9599 2022-01-09T17:52:22Z 2022-01-09T17:52:22Z OWNER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 List `--fmt` options in the docs  1097135732  
1008341078 https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008341078 https://api.github.com/repos/simonw/sqlite-utils/issues/373 IC_kwDOCGYnMM48GhBW simonw 9599 2022-01-09T17:31:12Z 2022-01-09T17:31:12Z OWNER

Found an example of using cog in a rST file here: https://github.com/nedbat/coveragepy/blob/f3238eea7e403d13a217b30579b1a1c2cbff62e3/doc/dbschema.rst#L21

.. [[[cog
    from coverage.sqldata import SCHEMA_VERSION
    print(".. code::")
    print()
    print(f"    SCHEMA_VERSION = {SCHEMA_VERSION}")
    print()
.. ]]]
.. code::

    SCHEMA_VERSION = 7

.. [[[end]]]

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 List `--fmt` options in the docs  1097135732  
1008338186 https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1008338186 https://api.github.com/repos/simonw/sqlite-utils/issues/375 IC_kwDOCGYnMM48GgUK simonw 9599 2022-01-09T17:13:33Z 2022-01-09T17:13:54Z OWNER 
cat blah.csv | sqlite-utils bulk blah.db - \
   "insert into blah (:foo, :bar)" --csv

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `sqlite-utils bulk` command 1097251014  
1008275546 https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008275546 https://api.github.com/repos/simonw/sqlite-utils/issues/365 IC_kwDOCGYnMM48GRBa fgregg 536941 2022-01-09T11:01:15Z 2022-01-09T13:37:51Z NONE

i don’t want to be such a partisan for analyze, but the query planner deciding not to use an index based on information collected by analyze is not necessarily a bug, but could be the correct choice.

<s>the original poster in that stack overflow doesn’t say there’s a performance regression </s>

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 create-index should run analyze after creating index 1096558279  
1008279307 https://github.com/simonw/datasette/pull/1574#issuecomment-1008279307 https://api.github.com/repos/simonw/datasette/issues/1574 IC_kwDOBm6k_c48GR8L fs111 33631 2022-01-09T11:26:06Z 2022-01-09T11:26:06Z NONE

@fgregg my thinking was backwards compatibility. I don't know what people do to their builds, I just wanted a smaller image for my use case.

@simonw any chance to take a look at this? If there is no interest, feel free to close the PR

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 introduce new option for datasette package to use a slim base image 1084193403  
1008252732 https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008252732 https://api.github.com/repos/simonw/sqlite-utils/issues/374 IC_kwDOCGYnMM48GLc8 simonw 9599 2022-01-09T08:25:30Z 2022-01-09T08:25:30Z OWNER

Need to change if table: to if table or fmt: in a few places.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 `--fmt` should imply `-t` 1097135860  
1008247370 https://github.com/simonw/sqlite-utils/issues/372#issuecomment-1008247370 https://api.github.com/repos/simonw/sqlite-utils/issues/372 IC_kwDOCGYnMM48GKJK simonw 9599 2022-01-09T07:51:18Z 2022-01-09T07:51:18Z OWNER

Pathlib says the stem of that would be dogs.and.cats.jpg - best stick with that for consistency. https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.suffix

It calls the last bit suffix - maybe I should use that instead of ext.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Idea: `suffix` and `stem` file columns 1097129710  
1008246366 https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008246366 https://api.github.com/repos/simonw/sqlite-utils/issues/371 IC_kwDOCGYnMM48GJ5e simonw 9599 2022-01-09T07:42:14Z 2022-01-09T07:42:14Z OWNER

Also need to update relevant docs for that example.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Support mutating row in `--convert` without returning it 1097128334  
1008246239 https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008246239 https://api.github.com/repos/simonw/sqlite-utils/issues/371 IC_kwDOCGYnMM48GJ3f simonw 9599 2022-01-09T07:41:24Z 2022-01-09T07:41:24Z OWNER

Might be a case of modifying this line: https://github.com/simonw/sqlite-utils/blob/e0c476bc380744680c8b7675c24fb0e9f5ec6dcd/sqlite_utils/cli.py#L828

To:

docs = (fn(doc) or doc for doc in docs)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Support mutating row in `--convert` without returning it 1097128334  

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);