4,497 rows sorted by updated_at descending

View and edit SQL

Suggested facets: reactions, created_at (date), updated_at (date)

author_association

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
733288841 https://github.com/simonw/datasette/issues/860#issuecomment-733288841 https://api.github.com/repos/simonw/datasette/issues/860 MDEyOklzc3VlQ29tbWVudDczMzI4ODg0MQ== simonw 9599 2020-11-24T23:19:47Z 2020-11-24T23:20:24Z OWNER

Here's what I have today - it's an undocumented datasette.metadata() method that returns a full JSON dictionary of values OR a single value if the optional key= argument is provided: https://github.com/simonw/datasette/blob/f2e2bfcdd9ad4891f3f66c9104c09943d943ffe4/datasette/app.py#L357-L388

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Plugin hook for database/table metadata 642651572  
733288522 https://github.com/simonw/datasette/issues/860#issuecomment-733288522 https://api.github.com/repos/simonw/datasette/issues/860 MDEyOklzc3VlQ29tbWVudDczMzI4ODUyMg== simonw 9599 2020-11-24T23:18:47Z 2020-11-24T23:18:47Z OWNER

In #942 I want to add support for per-column metadata - which means this new lookup mechanism will need to be able to answer the question "what description is available for this column".

So what should the .metadata() method look like? A couple of options:

  • datasette.metadata("description", table=x, database=y) - can take optional column= too.
  • datasette.table_metadata("description", table=x, database=y) and datasette.database_metadata("description", database=y) and so on - multiple methods for the different types of metadata.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Plugin hook for database/table metadata 642651572  
733287619 https://github.com/simonw/datasette/issues/860#issuecomment-733287619 https://api.github.com/repos/simonw/datasette/issues/860 MDEyOklzc3VlQ29tbWVudDczMzI4NzYxOQ== simonw 9599 2020-11-24T23:16:21Z 2020-11-24T23:16:21Z OWNER

I'll also allow any key to be looked up - so if users want to invent their own metadata keys other than the default license_url etc they can do so.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Plugin hook for database/table metadata 642651572  
733287416 https://github.com/simonw/datasette/issues/860#issuecomment-733287416 https://api.github.com/repos/simonw/datasette/issues/860 MDEyOklzc3VlQ29tbWVudDczMzI4NzQxNg== simonw 9599 2020-11-24T23:15:44Z 2020-11-24T23:15:44Z OWNER

I'm going to go with a plugin hook (and Datasette method) that returns individual values - so you ask it for e.g. the license_url for a specific table and it returns a string or None.

The default plugin hook implementation that ships with Datasette will then implement cascading lookups against metadata.json - but other plugins will be able to provide their own implementations, which should make it easy to build a plugin that lets you keep metadata in a database file and edit it interactively.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Plugin hook for database/table metadata 642651572  
733261501 https://github.com/simonw/datasette/issues/1107#issuecomment-733261501 https://api.github.com/repos/simonw/datasette/issues/1107 MDEyOklzc3VlQ29tbWVudDczMzI2MTUwMQ== simonw 9599 2020-11-24T22:09:11Z 2020-11-24T22:09:11Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename datasette.config() method to datasette.setting() 750079085  
733257071 https://github.com/simonw/datasette/issues/1107#issuecomment-733257071 https://api.github.com/repos/simonw/datasette/issues/1107 MDEyOklzc3VlQ29tbWVudDczMzI1NzA3MQ== simonw 9599 2020-11-24T21:59:32Z 2020-11-24T21:59:32Z OWNER

I'm going to make this a documented method in https://docs.datasette.io/en/latest/internals.html#datasette-class

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename datasette.config() method to datasette.setting() 750079085  
733249176 https://github.com/simonw/datasette/issues/1105#issuecomment-733249176 https://api.github.com/repos/simonw/datasette/issues/1105 MDEyOklzc3VlQ29tbWVudDczMzI0OTE3Ng== simonw 9599 2020-11-24T21:40:28Z 2020-11-24T21:40:28Z OWNER

This rebranding is complete - #1107 is a follow-up internal refactor.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rebrand config as settings 749982022  
733248437 https://github.com/simonw/datasette/issues/1106#issuecomment-733248437 https://api.github.com/repos/simonw/datasette/issues/1106 MDEyOklzc3VlQ29tbWVudDczMzI0ODQzNw== simonw 9599 2020-11-24T21:38:50Z 2020-11-24T21:38:50Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rebrand and redirect config.rst as settings.rst 749983857  
733247101 https://github.com/simonw/datasette/issues/1106#issuecomment-733247101 https://api.github.com/repos/simonw/datasette/issues/1106 MDEyOklzc3VlQ29tbWVudDczMzI0NzEwMQ== simonw 9599 2020-11-24T21:35:29Z 2020-11-24T21:36:04Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rebrand and redirect config.rst as settings.rst 749983857  
733245596 https://github.com/simonw/datasette/issues/1106#issuecomment-733245596 https://api.github.com/repos/simonw/datasette/issues/1106 MDEyOklzc3VlQ29tbWVudDczMzI0NTU5Ng== simonw 9599 2020-11-24T21:32:11Z 2020-11-24T21:32:11Z OWNER

https://docs.datasette.io/en/latest/settings.html is now live - need to redirect https://docs.datasette.io/en/latest/config.html to it using the ReadTheDocs redirects interface.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rebrand and redirect config.rst as settings.rst 749983857  
733245097 https://github.com/simonw/datasette/issues/1107#issuecomment-733245097 https://api.github.com/repos/simonw/datasette/issues/1107 MDEyOklzc3VlQ29tbWVudDczMzI0NTA5Nw== simonw 9599 2020-11-24T21:31:10Z 2020-11-24T21:31:10Z OWNER

Most of these use plugin_config which is unaffected. It looks like the only code I need to worry about is this trick in datasette-graphl: https://github.com/simonw/datasette-graphql/blob/483c9a9e203bb90365def3df8b8f01dda1e75865/datasette_graphql/utils.py#L456-L460

class DatasetteSpecialConfig(wrapt.ObjectProxy):
    def config(self, key):
        if key == "suggest_facets":
            return False
        return self.__wrapped__.config(key)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename datasette.config() method to datasette.setting() 750079085  
733244471 https://github.com/simonw/datasette/issues/1107#issuecomment-733244471 https://api.github.com/repos/simonw/datasette/issues/1107 MDEyOklzc3VlQ29tbWVudDczMzI0NDQ3MQ== simonw 9599 2020-11-24T21:29:59Z 2020-11-24T21:29:59Z OWNER

I ran rg '.config\(' datasette-*/ in my top-level directory:

datasette-sentry/test_datasette_sentry.py:    def plugin_config(self, name):
datasette-sentry/datasette_sentry.py:    config = datasette.plugin_config("datasette-sentry") or {}
datasette-render-markdown/datasette_render_markdown/__init__.py:        datasette.plugin_config(
datasette-render-images/datasette_render_images.py:        plugin_config = datasette.plugin_config("datasette-render-images") or {}
datasette-render-html/datasette_render_html.py:    config = datasette.plugin_config(
datasette-render-timestamps/datasette_render_timestamps/__init__.py:        datasette.plugin_config(
datasette-permissions-sql/datasette_permissions_sql/__init__.py:        for rule in datasette.plugin_config("datasette-permissions-sql") or []:
datasette-mask-columns/datasette_mask_columns/__init__.py:            datasette.plugin_config("datasette-mask-columns", database=database) or {}
datasette-mask-columns/datasette_mask_columns/__init__.py:    masks = datasette.plugin_config("datasette-mask-columns", database=database) or {}
datasette-media/datasette_media/__init__.py:    plugin_config = datasette.plugin_config("datasette-media") or {}
datasette-insert/datasette_insert/__init__.py:    plugin_config = datasette.plugin_config("datasette-insert") or {}
datasette-indieauth/datasette_indieauth/__init__.py:    plugin_config = datasette.plugin_config("datasette-indieauth") or {}
datasette-graphql/datasette_graphql/__init__.py:    config = datasette.plugin_config("datasette-graphql") or {}
datasette-graphql/datasette_graphql/__init__.py:    config = datasette.plugin_config("datasette-graphql") or {}
datasette-graphql/datasette_graphql/utils.py:            auto_camelcase=(datasette.plugin_config("datasette-graphql") or {}).get(
datasette-graphql/datasette_graphql/utils.py:    table_plugin_config = datasette.plugin_config(
datasette-graphql/datasette_graphql/utils.py:    def config(self, key):
datasette-graphql/datasette_graphql/utils.py:        return self.__wrapped__.config(key)
datasette-init/datasette_init/__init__.py:        config = datasette.plugin_config("datasette-init")
datasette-edit-templates/datasette_edit_templates/__init__.py:    plugin_config = datasette.plugin_config("datasette-edit-templates") or {}
datasette-cors/datasette_cors.py:    config = datasette.plugin_config("datasette-cors") or {}
datasette-cluster-map-old/build/lib/datasette_cluster_map/__init__.py:        datasette.plugin_config("datasette-cluster-map", database=database, table=table)
datasette-cluster-map/datasette_cluster_map/__init__.py:        datasette.plugin_config("datasette-cluster-map", database=database, table=table)
datasette-cluster-map/datasette_cluster_map/__init__.py:        datasette.plugin_config("datasette-cluster-map", database=database, table=table)
datasette-cluster-map/tests/test_cluster_map.py:async def test_plugin_config(db_path, config, table, expected_fragments):
datasette-configure-asgi/datasette_configure_asgi.py:        configs = datasette.plugin_config("datasette-configure-asgi") or []
datasette-configure-asgi/test_datasette_configure_asgi.py:    def plugin_config(self, name):
datasette-cluster-map-old/datasette_cluster_map/__init__.py:        datasette.plugin_config("datasette-cluster-map", database=database, table=table)
datasette-auth-passwords/datasette_auth_passwords/__init__.py:    config = datasette.plugin_config("datasette-auth-passwords") or {}
datasette-auth-github/datasette_auth_github/views.py:def verify_config(config):
datasette-auth-github/datasette_auth_github/views.py:    config = datasette.plugin_config("datasette-auth-github")
datasette-auth-github/datasette_auth_github/views.py:    verify_config(config)
datasette-auth-github/datasette_auth_github/views.py:    config = datasette.plugin_config("datasette-auth-github")
datasette-auth-github/datasette_auth_github/views.py:    verify_config(config)
datasette-atom/datasette_atom/__init__.py:        plugin_config = datasette.plugin_config("datasette-atom")
datasette-auth-google/datasette_auth_google/__init__.py:    config = datasette.plugin_config("datasette-auth-github") or {}
datasette-auth-existing-cookies/test_datasette_auth_existing_cookies.py:    def plugin_config(self, name):
datasette-auth-passwords/build/lib/datasette_auth_passwords/__init__.py:    config = datasette.plugin_config("datasette-auth-passwords") or {}
datasette-annotate/datasette_annotate/utils.py:    plugin_config = datasette.plugin_config("datasette-annotate") or {}
datasette-auth-existing-cookies/datasette_auth_existing_cookies/__init__.py:    config = datasette.plugin_config("datasette-auth-existing-cookies") or {}
datasette-auth-simple/datasette_auth_simple/__init__.py:datasette.plugin_config("datasette-auth-simple")
datasette-auth-tokens/datasette_auth_tokens/__init__.py:        config = datasette.plugin_config("datasette-auth-tokens") or {}
datasette-block-robots/datasette_block_robots/__init__.py:    config = datasette.plugin_config("datasette-block-robots") or {}
datasette-block-robots/datasette_block_robots/__init__.py:    config = datasette.plugin_config("datasette-block-robots") or {}
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename datasette.config() method to datasette.setting() 750079085  
733241949 https://github.com/simonw/datasette/issues/1107#issuecomment-733241949 https://api.github.com/repos/simonw/datasette/issues/1107 MDEyOklzc3VlQ29tbWVudDczMzI0MTk0OQ== simonw 9599 2020-11-24T21:24:26Z 2020-11-24T21:24:26Z OWNER

Are there any plugins that use this API even though it isn't documented?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename datasette.config() method to datasette.setting() 750079085  
733221359 https://github.com/simonw/datasette/issues/1106#issuecomment-733221359 https://api.github.com/repos/simonw/datasette/issues/1106 MDEyOklzc3VlQ29tbWVudDczMzIyMTM1OQ== simonw 9599 2020-11-24T20:40:21Z 2020-11-24T20:40:21Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rebrand and redirect config.rst as settings.rst 749983857  
733212084 https://github.com/simonw/datasette/issues/1104#issuecomment-733212084 https://api.github.com/repos/simonw/datasette/issues/1104 MDEyOklzc3VlQ29tbWVudDczMzIxMjA4NA== simonw 9599 2020-11-24T20:20:33Z 2020-11-24T20:20:33Z OWNER

I'll throw an error if a config.json file is detected.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
config.json in directory config mode should be settings.json 749981663  
733198051 https://github.com/simonw/datasette/issues/226#issuecomment-733198051 https://api.github.com/repos/simonw/datasette/issues/226 MDEyOklzc3VlQ29tbWVudDczMzE5ODA1MQ== simonw 9599 2020-11-24T19:52:46Z 2020-11-24T19:52:46Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Unit tests for installable plugins 315738696  
733190827 https://github.com/simonw/datasette/issues/1105#issuecomment-733190827 https://api.github.com/repos/simonw/datasette/issues/1105 MDEyOklzc3VlQ29tbWVudDczMzE5MDgyNw== simonw 9599 2020-11-24T19:38:02Z 2020-11-24T19:38:02Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rebrand config as settings 749982022  
733189737 https://github.com/simonw/datasette/issues/1103#issuecomment-733189737 https://api.github.com/repos/simonw/datasette/issues/1103 MDEyOklzc3VlQ29tbWVudDczMzE4OTczNw== simonw 9599 2020-11-24T19:35:45Z 2020-11-24T19:35:45Z OWNER

Part of #1105

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename /-/config to /-/settings 749979454  
733189693 https://github.com/simonw/datasette/issues/992#issuecomment-733189693 https://api.github.com/repos/simonw/datasette/issues/992 MDEyOklzc3VlQ29tbWVudDczMzE4OTY5Mw== simonw 9599 2020-11-24T19:35:38Z 2020-11-24T19:35:38Z OWNER

Part of #1105

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Change "--config foo:bar" to "--setting foo bar" 714449879  
733189620 https://github.com/simonw/datasette/issues/1104#issuecomment-733189620 https://api.github.com/repos/simonw/datasette/issues/1104 MDEyOklzc3VlQ29tbWVudDczMzE4OTYyMA== simonw 9599 2020-11-24T19:35:30Z 2020-11-24T19:35:30Z OWNER

Part of #1105

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
config.json in directory config mode should be settings.json 749981663  
733187586 https://github.com/simonw/datasette/issues/1103#issuecomment-733187586 https://api.github.com/repos/simonw/datasette/issues/1103 MDEyOklzc3VlQ29tbWVudDczMzE4NzU4Ng== simonw 9599 2020-11-24T19:31:23Z 2020-11-24T19:31:23Z OWNER

I'll set up a redirect from /-/config to /-/settings.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename /-/config to /-/settings 749979454  
733180289 https://github.com/simonw/datasette/issues/992#issuecomment-733180289 https://api.github.com/repos/simonw/datasette/issues/992 MDEyOklzc3VlQ29tbWVudDczMzE4MDI4OQ== simonw 9599 2020-11-24T19:16:30Z 2020-11-24T19:16:30Z OWNER

Need to figure out the --setting foo bar alternative for this --config foo:bar logic: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/cli.py#L31-L63

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Change "--config foo:bar" to "--setting foo bar" 714449879  
733176252 https://github.com/simonw/datasette/issues/992#issuecomment-733176252 https://api.github.com/repos/simonw/datasette/issues/992 MDEyOklzc3VlQ29tbWVudDczMzE3NjI1Mg== simonw 9599 2020-11-24T19:07:49Z 2020-11-24T19:07:49Z OWNER

I'm going to keep --config for the moment but show a deprecation warning that it will be gone in Datasette 1.0.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Change "--config foo:bar" to "--setting foo bar" 714449879  
733175965 https://github.com/simonw/datasette/issues/992#issuecomment-733175965 https://api.github.com/repos/simonw/datasette/issues/992 MDEyOklzc3VlQ29tbWVudDczMzE3NTk2NQ== simonw 9599 2020-11-24T19:07:13Z 2020-11-24T19:07:13Z OWNER

This is blocking progress on other metadata tickets like #860 because I want to split the concept of concrete metadata (source, license, etc) from configuration that currently lives in metadata (default sort order, default facets).

I'm going to solve this next to unblock that stuff.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Change "--config foo:bar" to "--setting foo bar" 714449879  
733175454 https://github.com/simonw/datasette/issues/860#issuecomment-733175454 https://api.github.com/repos/simonw/datasette/issues/860 MDEyOklzc3VlQ29tbWVudDczMzE3NTQ1NA== simonw 9599 2020-11-24T19:06:07Z 2020-11-24T19:06:07Z OWNER

I see two ways this plugin hook could work. It could be asked about a specific instance, database or table and return the full metadata for that object. OR it could ask for a specific metadata field - e.g. source_url for table X, and return that.

The more finely grained one would allow plugins to implement their own cascading rules pretty easily. Is there a reason it would be better for the hook to return an entire block of JSON for a specific table or database?

I also need to decide if this hook is just going to be about source/license/about displayed metadata, or if it will include the functionality that has been sneaking into metadata.json over time - stuff like page size, default sort order or default facets.

Perhaps I should split those out into a "configuration" concept first, after renaming --config to --setting in #992.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Plugin hook for database/table metadata 642651572  
732544590 https://github.com/simonw/datasette/issues/1101#issuecomment-732544590 https://api.github.com/repos/simonw/datasette/issues/1101 MDEyOklzc3VlQ29tbWVudDczMjU0NDU5MA== simonw 9599 2020-11-24T02:22:55Z 2020-11-24T02:22:55Z OWNER

The trick I'm using here is to follow the next_url in order to paginate through all of the matching results. The loop calls the data() method multiple times, once for each page of results: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/views/base.py#L304-L307

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
register_output_renderer() should support streaming data 749283032  
732543700 https://github.com/simonw/datasette/issues/1101#issuecomment-732543700 https://api.github.com/repos/simonw/datasette/issues/1101 MDEyOklzc3VlQ29tbWVudDczMjU0MzcwMA== simonw 9599 2020-11-24T02:20:30Z 2020-11-24T02:20:30Z OWNER

Current design: https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette

@hookimpl
def register_output_renderer(datasette):
    return {
        "extension": "test",
        "render": render_demo,
        "can_render": can_render_demo,  # Optional
    }

Where render_demo looks something like this:

async def render_demo(datasette, columns, rows):
    db = datasette.get_database()
    result = await db.execute("select sqlite_version()")
    first_row = " | ".join(columns)
    lines = [first_row]
    lines.append("=" * len(first_row))
    for row in rows:
        lines.append(" | ".join(row))
    return Response(
        "\n".join(lines),
        content_type="text/plain; charset=utf-8",
        headers={"x-sqlite-version": result.first()[0]}
    )

Meanwhile here's where the CSV streaming mode is implemented: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/views/base.py#L297-L380

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
register_output_renderer() should support streaming data 749283032  
732542285 https://github.com/simonw/datasette/issues/1096#issuecomment-732542285 https://api.github.com/repos/simonw/datasette/issues/1096 MDEyOklzc3VlQ29tbWVudDczMjU0MjI4NQ== simonw 9599 2020-11-24T02:16:22Z 2020-11-24T02:16:22Z OWNER

I'd like to implement this by first extending the register_output_renderer() hook to support streaming huge responses, then switching CSV to use the plugin hook in addition to TSV using it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
TSV should be a default export option 743359646  
731658059 https://github.com/simonw/datasette/issues/860#issuecomment-731658059 https://api.github.com/repos/simonw/datasette/issues/860 MDEyOklzc3VlQ29tbWVudDczMTY1ODA1OQ== simonw 9599 2020-11-22T00:31:47Z 2020-11-22T00:33:48Z OWNER

Documented behaviour right now, for metadata set at the instance level, is: https://docs.datasette.io/en/stable/metadata.html

The above metadata will be displayed on the index page of your Datasette-powered site. The source and license information will also be included in the footer of every page served by Datasette.

...

Metadata at the top level of the JSON will be shown on the index page and in the footer on every page of the site. The license and source is expected to apply to all of your data.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Plugin hook for database/table metadata 642651572  
731657660 https://github.com/simonw/datasette/issues/860#issuecomment-731657660 https://api.github.com/repos/simonw/datasette/issues/860 MDEyOklzc3VlQ29tbWVudDczMTY1NzY2MA== simonw 9599 2020-11-22T00:27:32Z 2020-11-22T00:31:54Z OWNER

Open question: how should cascading work? If a table is missing a field but the database or instance has it, should that value cascade down to the table?

It feels like license should definitely cascade: if an instance lists a certain license that should absolutely filter through to all databases and tables.

But... should the other fields cascade? Cascading description doesn't feel right at all, and neither does title.

What about about and about_url and source and source_url? I'm a bit torn on whether they should cascade or not. I'm leaning towards cascading them.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Plugin hook for database/table metadata 642651572  
731657486 https://github.com/simonw/datasette/issues/860#issuecomment-731657486 https://api.github.com/repos/simonw/datasette/issues/860 MDEyOklzc3VlQ29tbWVudDczMTY1NzQ4Ng== simonw 9599 2020-11-22T00:25:34Z 2020-11-22T00:25:34Z OWNER

There are three layers of metadata: table, database and instance.

Currently the metadata fields are (ignoring not-quite-metadata like sort and sort_desc):

  • title
  • description (or description_html)
  • about / about_url
  • source / source_url
  • license / license_url
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Plugin hook for database/table metadata 642651572  
731654132 https://github.com/simonw/datasette/issues/1084#issuecomment-731654132 https://api.github.com/repos/simonw/datasette/issues/1084 MDEyOklzc3VlQ29tbWVudDczMTY1NDEzMg== simonw 9599 2020-11-21T23:45:59Z 2020-11-21T23:45:59Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Table/database action menu cut off if too short  737394470  
731653083 https://github.com/simonw/datasette/issues/1084#issuecomment-731653083 https://api.github.com/repos/simonw/datasette/issues/1084 MDEyOklzc3VlQ29tbWVudDczMTY1MzA4Mw== simonw 9599 2020-11-21T23:35:07Z 2020-11-21T23:35:07Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Table/database action menu cut off if too short  737394470  
731652991 https://github.com/simonw/datasette/issues/1084#issuecomment-731652991 https://api.github.com/repos/simonw/datasette/issues/1084 MDEyOklzc3VlQ29tbWVudDczMTY1Mjk5MQ== simonw 9599 2020-11-21T23:34:22Z 2020-11-21T23:34:22Z OWNER

Fixed by positioning the menu relative to the <div class="page-header"> element rather than the cog icon:

https://user-images.githubusercontent.com/9599/99889950-02eab580-2c0f-11eb-85dc-dd4bc840a92e.png">

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Table/database action menu cut off if too short  737394470  
731644064 https://github.com/simonw/datasette/issues/1084#issuecomment-731644064 https://api.github.com/repos/simonw/datasette/issues/1084 MDEyOklzc3VlQ29tbWVudDczMTY0NDA2NA== simonw 9599 2020-11-21T22:10:15Z 2020-11-21T22:10:15Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Table/database action menu cut off if too short  737394470  
731260091 https://github.com/simonw/datasette/issues/1094#issuecomment-731260091 https://api.github.com/repos/simonw/datasette/issues/1094 MDEyOklzc3VlQ29tbWVudDczMTI2MDA5MQ== bapowell 4808085 2020-11-20T16:11:29Z 2020-11-20T16:11:29Z NONE

I can confirm this issue, running version 0.51.1 under Windows.

Fixed by commenting out the following line near the top of datasette\utils\asgi.py :

#from os import EX_CANTCREAT

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
import EX_CANTCREAT means datasette fails to work on Windows 743011397  
730893729 https://github.com/simonw/datasette/issues/511#issuecomment-730893729 https://api.github.com/repos/simonw/datasette/issues/511 MDEyOklzc3VlQ29tbWVudDczMDg5MzcyOQ== Carib0u 4060506 2020-11-20T06:35:13Z 2020-11-20T06:35:13Z NONE

Trying to run on Windows today, I get an error from the utils/asgi.py module.

It's trying from os import EX_CANTCREAT which is Unix-only. I commented this line out, and (so far) it's working.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Get Datasette working on Windows, including CI 456578474  
729484478 https://github.com/dogsheep/twitter-to-sqlite/issues/52#issuecomment-729484478 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/52 MDEyOklzc3VlQ29tbWVudDcyOTQ4NDQ3OA== fatihky 4169772 2020-11-18T07:12:45Z 2020-11-18T07:12:45Z NONE

I'm so sorry that you already have --since_id option and that's enough for the case I've mentioned. Thank you for this excellent tool!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Discussion: Adding support for fetching only fresh tweets 745393298  
729045320 https://github.com/simonw/datasette/issues/1091#issuecomment-729045320 https://api.github.com/repos/simonw/datasette/issues/1091 MDEyOklzc3VlQ29tbWVudDcyOTA0NTMyMA== tballison 6739646 2020-11-17T16:31:00Z 2020-11-17T16:31:00Z NONE

We're using mod_proxy.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.json and .csv exports fail to apply base_url 742011049  
729018386 https://github.com/simonw/datasette/issues/1091#issuecomment-729018386 https://api.github.com/repos/simonw/datasette/issues/1091 MDEyOklzc3VlQ29tbWVudDcyOTAxODM4Ng== tballison 6739646 2020-11-17T15:48:58Z 2020-11-17T15:48:58Z NONE

I don't think we are, but I'll check with Maruan.

I think this is the relevant part of our config?

  Alias "/base/" "/usr/share/corpora/"
  <Directory "/usr/share/corpora/">
    Options +Indexes -Multiviews
    AllowOverride None
  </Directory>

  ProxyPreserveHost On

  ProxyPass /datasette http://0.0.0.0:8001
  ProxyPassReverse /datasette http://0.0.0.0:8001

</VirtualHost>
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.json and .csv exports fail to apply base_url 742011049  
728262974 https://github.com/simonw/datasette/issues/1091#issuecomment-728262974 https://api.github.com/repos/simonw/datasette/issues/1091 MDEyOklzc3VlQ29tbWVudDcyODI2Mjk3NA== simonw 9599 2020-11-16T19:05:08Z 2020-11-16T19:05:08Z OWNER

I have a hunch that there may be some extra configuration in play here - could Apache itself be rewriting some of the links using mod_proxy_html?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.json and .csv exports fail to apply base_url 742011049  
727692413 https://github.com/dogsheep/swarm-to-sqlite/issues/11#issuecomment-727692413 https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/11 MDEyOklzc3VlQ29tbWVudDcyNzY5MjQxMw== simonw 9599 2020-11-16T02:15:22Z 2020-11-16T02:15:22Z MEMBER

Thanks, I'll look into this.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Error thrown: sqlite3.OperationalError: table users has no column named lastName 743400216  
727656208 https://github.com/simonw/datasette/issues/1098#issuecomment-727656208 https://api.github.com/repos/simonw/datasette/issues/1098 MDEyOklzc3VlQ29tbWVudDcyNzY1NjIwOA== simonw 9599 2020-11-15T23:26:14Z 2020-11-15T23:26:14Z OWNER

Schema for that broken example:

CREATE TABLE generators_eia860 (
    id INTEGER NOT NULL, 
    plant_id_eia INTEGER, 
    generator_id TEXT, 
    report_date DATE, 
    operational_status_code TEXT, 
    operational_status TEXT, 
    ownership_code TEXT, 
    utility_id_eia INTEGER, 
    capacity_mw FLOAT, 
    summer_capacity_mw FLOAT, 
    winter_capacity_mw FLOAT, 
    energy_source_code_1 TEXT, 
    energy_source_code_2 TEXT, 
    energy_source_code_3 TEXT, 
    energy_source_code_4 TEXT, 
    energy_source_code_5 TEXT, 
    energy_source_code_6 TEXT, 
    fuel_type_code_pudl TEXT, 
    multiple_fuels BOOLEAN, 
    deliver_power_transgrid BOOLEAN, 
    syncronized_transmission_grid BOOLEAN, 
    turbines_num INTEGER, 
    planned_modifications BOOLEAN, 
    planned_net_summer_capacity_uprate_mw FLOAT, 
    planned_net_winter_capacity_uprate_mw FLOAT, 
    planned_uprate_date DATE, 
    planned_net_summer_capacity_derate_mw FLOAT, 
    planned_net_winter_capacity_derate_mw FLOAT, 
    planned_derate_date DATE, 
    planned_new_prime_mover_code TEXT, 
    planned_energy_source_code_1 TEXT, 
    planned_repower_date DATE, 
    other_planned_modifications BOOLEAN, 
    other_modifications_date DATE, 
    planned_retirement_date DATE, 
    carbon_capture BOOLEAN, 
    startup_source_code_1 TEXT, 
    startup_source_code_2 TEXT, 
    startup_source_code_3 TEXT, 
    startup_source_code_4 TEXT, 
    technology_description TEXT, 
    turbines_inverters_hydrokinetics TEXT, 
    time_cold_shutdown_full_load_code TEXT, 
    planned_new_capacity_mw FLOAT, 
    cofire_fuels BOOLEAN, 
    switch_oil_gas BOOLEAN, 
    nameplate_power_factor FLOAT, 
    minimum_load_mw FLOAT, 
    uprate_derate_during_year BOOLEAN, 
    uprate_derate_completed_date DATE, 
    current_planned_operating_date DATE, 
    summer_estimated_capability_mw FLOAT, 
    winter_estimated_capability_mw FLOAT, 
    retirement_date DATE, 
    PRIMARY KEY (id), 
    FOREIGN KEY(plant_id_eia, generator_id) REFERENCES generators_entity_eia (plant_id_eia, generator_id), 
    FOREIGN KEY(utility_id_eia) REFERENCES utilities_entity_eia (utility_id_eia), 
    CHECK (multiple_fuels IN (0, 1)), 
    CHECK (deliver_power_transgrid IN (0, 1)), 
    CHECK (syncronized_transmission_grid IN (0, 1)), 
    CHECK (planned_modifications IN (0, 1)), 
    CHECK (other_planned_modifications IN (0, 1)), 
    CHECK (carbon_capture IN (0, 1)), 
    CHECK (cofire_fuels IN (0, 1)), 
    CHECK (switch_oil_gas IN (0, 1)), 
    CHECK (uprate_derate_during_year IN (0, 1))
);

https://pudl-datasette-xl7xwcpe2a-uc.a.run.app/pudl/generators_entity_eia is:

CREATE TABLE generators_entity_eia (
    plant_id_eia INTEGER NOT NULL, 
    generator_id TEXT NOT NULL, 
    prime_mover_code TEXT, 
    duct_burners BOOLEAN, 
    operating_date DATE, 
    topping_bottoming_code TEXT, 
    solid_fuel_gasification BOOLEAN, 
    pulverized_coal_tech BOOLEAN, 
    fluidized_bed_tech BOOLEAN, 
    subcritical_tech BOOLEAN, 
    supercritical_tech BOOLEAN, 
    ultrasupercritical_tech BOOLEAN, 
    stoker_tech BOOLEAN, 
    other_combustion_tech BOOLEAN, 
    bypass_heat_recovery BOOLEAN, 
    rto_iso_lmp_node_id TEXT, 
    rto_iso_location_wholesale_reporting_id TEXT, 
    associated_combined_heat_power BOOLEAN, 
    original_planned_operating_date DATE, 
    operating_switch TEXT, 
    previously_canceled BOOLEAN, 
    PRIMARY KEY (plant_id_eia, generator_id), 
    FOREIGN KEY(plant_id_eia) REFERENCES plants_entity_eia (plant_id_eia), 
    CHECK (duct_burners IN (0, 1)), 
    CHECK (solid_fuel_gasification IN (0, 1)), 
    CHECK (pulverized_coal_tech IN (0, 1)), 
    CHECK (fluidized_bed_tech IN (0, 1)), 
    CHECK (subcritical_tech IN (0, 1)), 
    CHECK (supercritical_tech IN (0, 1)), 
    CHECK (ultrasupercritical_tech IN (0, 1)), 
    CHECK (stoker_tech IN (0, 1)), 
    CHECK (other_combustion_tech IN (0, 1)), 
    CHECK (bypass_heat_recovery IN (0, 1)), 
    CHECK (associated_combined_heat_power IN (0, 1)), 
    CHECK (previously_canceled IN (0, 1))
);
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Foreign key links break for compound foreign keys 743370900  
727655636 https://github.com/simonw/datasette/issues/1098#issuecomment-727655636 https://api.github.com/repos/simonw/datasette/issues/1098 MDEyOklzc3VlQ29tbWVudDcyNzY1NTYzNg== simonw 9599 2020-11-15T23:22:27Z 2020-11-15T23:22:27Z OWNER

Need to replicate this in the fixtures, then fix it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Foreign key links break for compound foreign keys 743370900  
727655018 https://github.com/simonw/datasette/pull/1097#issuecomment-727655018 https://api.github.com/repos/simonw/datasette/issues/1097 MDEyOklzc3VlQ29tbWVudDcyNzY1NTAxOA== codecov[bot] 22429695 2020-11-15T23:18:18Z 2020-11-15T23:18:18Z NONE

Codecov Report

Merging #1097 (e89211d) into main (5eb8e9b) will not change coverage.
The diff coverage is 84.61%.

@@           Coverage Diff           @@
##             main    #1097   +/-   ##
=======================================
  Coverage   91.38%   91.38%           
=======================================
  Files          30       30           
  Lines        3785     3785           
=======================================
  Hits         3459     3459           
  Misses        326      326           
<table> <thead> <tr> <th>Impacted Files</th> <th>Coverage Δ</th> <th></th> </tr> </thead> <tbody> <tr> <td>datasette/cli.py</td> <td>73.63% <0.00%> (ø)</td> <td></td> </tr> <tr> <td>datasette/inspect.py</td> <td>36.11% <ø> (ø)</td> <td></td> </tr> <tr> <td>datasette/publish/common.py</td> <td>94.73% <ø> (ø)</td> <td></td> </tr> <tr> <td>datasette/tracer.py</td> <td>81.60% <0.00%> (ø)</td> <td></td> </tr> <tr> <td>datasette/utils/testing.py</td> <td>95.16% <ø> (ø)</td> <td></td> </tr> <tr> <td>datasette/publish/heroku.py</td> <td>87.12% <50.00%> (ø)</td> <td></td> </tr> <tr> <td>datasette/app.py</td> <td>96.46% <66.66%> (ø)</td> <td></td> </tr> <tr> <td>datasette/filters.py</td> <td>94.35% <77.77%> (ø)</td> <td></td> </tr> <tr> <td>datasette/utils/__init__.py</td> <td>94.01% <86.20%> (ø)</td> <td></td> </tr> <tr> <td>datasette/views/table.py</td> <td>95.92% <92.30%> (ø)</td> <td></td> </tr> <tr> <td>... and 9 more</td> <td></td> <td></td> </tr> </tbody> </table>

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5eb8e9b...e89211d. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use f-strings 743369188  
727626657 https://github.com/simonw/datasette/issues/942#issuecomment-727626657 https://api.github.com/repos/simonw/datasette/issues/942 MDEyOklzc3VlQ29tbWVudDcyNzYyNjY1Nw== simonw 9599 2020-11-15T19:54:44Z 2020-11-15T19:54:44Z OWNER

This will also benefit from the metadata plugin hook: #860

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support column descriptions in metadata.json 681334912  
727233553 https://github.com/simonw/datasette/issues/1091#issuecomment-727233553 https://api.github.com/repos/simonw/datasette/issues/1091 MDEyOklzc3VlQ29tbWVudDcyNzIzMzU1Mw== simonw 9599 2020-11-14T16:46:52Z 2020-11-14T16:46:52Z OWNER

@tballison could I see the section of your Apache config that configures the proxying to /datasette/?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.json and .csv exports fail to apply base_url 742011049  
726801731 https://github.com/simonw/datasette/issues/1091#issuecomment-726801731 https://api.github.com/repos/simonw/datasette/issues/1091 MDEyOklzc3VlQ29tbWVudDcyNjgwMTczMQ== tballison 6739646 2020-11-13T14:40:56Z 2020-11-13T14:40:56Z NONE

My headers aren't clickable/sortable with custom sql, but I think that's by design.

In the default view, https://corpora.tika.apache.org/datasette/file_profiles/file_profiles, ah, y, now I see that the headers should be sortable, but you're right the base_url is not applied.

base_url works with "View and Edit SQL" and with "(advanced)"

As you point out, does not work with the export csv, json, other or with the "Next page" navigational button at the bottom.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.json and .csv exports fail to apply base_url 742011049  
726798745 https://github.com/simonw/datasette/issues/1091#issuecomment-726798745 https://api.github.com/repos/simonw/datasette/issues/1091 MDEyOklzc3VlQ29tbWVudDcyNjc5ODc0NQ== tballison 6739646 2020-11-13T14:35:22Z 2020-11-13T14:35:22Z NONE

I'm starting this with docker like so:

docker run --name datasette -d -p 8001:8001 -vpwd:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/file_profiles.db --config sql_time_limit_ms:120000 --config max_returned_rows:100000 --config base_url:/datasette/ --config cache_size_kb:50000

I'm not doing any templating or anything else custom.

Apropos of nothing, I swapped out a simpler db, so this query should now work:

https://corpora.tika.apache.org/datasette/file_profiles?sql=select%0D%0A++*%0D%0Afrom%0D%0A++file_profiles+fp%0D%0Alimit%0D%0A++10

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.json and .csv exports fail to apply base_url 742011049  
726419027 https://github.com/simonw/datasette/issues/268#issuecomment-726419027 https://api.github.com/repos/simonw/datasette/issues/268 MDEyOklzc3VlQ29tbWVudDcyNjQxOTAyNw== simonw 9599 2020-11-13T00:09:04Z 2020-11-13T00:09:04Z OWNER

Part of the challenge here is that this is the first time the TableView will have had a complete rewrite of the SQL it is going to execute. That SQL is currently constructed here: https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/views/table.py#L628-L636

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Mechanism for ranking results from SQLite full-text search 323718842  
726417847 https://github.com/simonw/datasette/issues/749#issuecomment-726417847 https://api.github.com/repos/simonw/datasette/issues/749 MDEyOklzc3VlQ29tbWVudDcyNjQxNzg0Nw== simonw 9599 2020-11-13T00:05:14Z 2020-11-13T00:05:14Z OWNER

https://cloud.google.com/blog/products/serverless/cloud-run-now-supports-http-grpc-server-streaming indicates this limit should no longer apply:

With this addition, Cloud Run can now ... Send responses larger than the previous 32 MB limit

But I'm still getting errors from Cloud Run attempting to download .db files larger than 32 MB.

I filed a question in their issue tracker about that here: https://issuetracker.google.com/issues/173038375

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Respect Cloud Run max response size of 32MB 610829227  
726416330 https://github.com/simonw/datasette/issues/1091#issuecomment-726416330 https://api.github.com/repos/simonw/datasette/issues/1091 MDEyOklzc3VlQ29tbWVudDcyNjQxNjMzMA== simonw 9599 2020-11-13T00:00:43Z 2020-11-13T00:00:43Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.json and .csv exports fail to apply base_url 742011049  
726415991 https://github.com/simonw/datasette/issues/1091#issuecomment-726415991 https://api.github.com/repos/simonw/datasette/issues/1091 MDEyOklzc3VlQ29tbWVudDcyNjQxNTk5MQ== simonw 9599 2020-11-12T23:59:34Z 2020-11-12T23:59:34Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.json and .csv exports fail to apply base_url 742011049  
726415019 https://github.com/simonw/datasette/issues/1091#issuecomment-726415019 https://api.github.com/repos/simonw/datasette/issues/1091 MDEyOklzc3VlQ29tbWVudDcyNjQxNTAxOQ== simonw 9599 2020-11-12T23:56:23Z 2020-11-12T23:56:23Z OWNER

@tballison is there any chance you're running any custom templates in that installation? I'm really confused as to why I can't replicate the bug.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.json and .csv exports fail to apply base_url 742011049  
726413829 https://github.com/simonw/datasette/issues/1091#issuecomment-726413829 https://api.github.com/repos/simonw/datasette/issues/1091 MDEyOklzc3VlQ29tbWVudDcyNjQxMzgyOQ== simonw 9599 2020-11-12T23:52:50Z 2020-11-12T23:54:16Z OWNER

Hmm... it's not just the .csv and .json export links - it's the column headings (which can be clicked to change the sort order) as well. Here's an extract of the HTML from that page:

<p class="export-links">This data as 
  <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json">json</a>, 
  <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.csv?_size=max">CSV</a> (
  <a href="#export">advanced</a>)
</p>
<div class="table-wrapper">
  <table class="rows-and-columns">
    <thead>
      <tr>
        <th class="col-Link" scope="col" data-column="Link" data-column-type="" data-column-not-null="0" data-is-pk="0">
          Link
        </th>
        <th class="col-rowid" scope="col" data-column="rowid" data-column-type="integer" data-column-not-null="0" data-is-pk="1">
          <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES?_sort_desc=rowid" rel="nofollow">rowid&nbsp;▼</a>
        </th>
        <th class="col-PARSE_EXCEPTION_ID" scope="col" data-column="PARSE_EXCEPTION_ID" data-column-type="INTEGER" data-column-not-null="0" data-is-pk="0">
          <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES?_sort=PARSE_EXCEPTION_ID" rel="nofollow">PARSE_EXCEPTION_ID</a>
        </th>
        <th class="col-PARSE_EXCEPTION_DESCRIPTION" scope="col" data-column="PARSE_EXCEPTION_DESCRIPTION" data-column-type="VARCHAR(128)" data-column-not-null="0" data-is-pk="0">
          <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES?_sort=PARSE_EXCEPTION_DESCRIPTION" rel="nofollow">PARSE_EXCEPTION_DESCRIPTION</a>
        </th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td class="col-Link type-pk">
          <a href="/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/1">1</a>
        </td>
        <td class="col-rowid type-int">1</td>
        <td class="col-PARSE_EXCEPTION_ID type-int">0</td>
        <td class="col-PARSE_EXCEPTION_DESCRIPTION type-str">RUNTIME</td>
      </tr>
      <tr>
        <td class="col-Link type-pk">
          <a href="/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/2">2</a>
        </td>
        <td class="col-rowid type-int">2</td>
        <td class="col-PARSE_EXCEPTION_ID type-int">1</td>
        <td class="col-PARSE_EXCEPTION_DESCRIPTION type-str">ENCRYPTION</td>
      </tr>
      <tr>
        <td class="col-Link type-pk">
          <a href="/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/3">3</a>
        </td>
        <td class="col-rowid type-int">3</td>
        <td class="col-PARSE_EXCEPTION_ID type-int">2</td>
        <td class="col-PARSE_EXCEPTION_DESCRIPTION type-str">ACCESS_PERMISSION</td>
      </tr>
      <tr>
        <td class="col-Link type-pk">
          <a href="/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/4">4</a>
        </td>
        <td class="col-rowid type-int">4</td>
        <td class="col-PARSE_EXCEPTION_ID type-int">3</td>
        <td class="col-PARSE_EXCEPTION_DESCRIPTION type-str">UNSUPPORTED_VERSION</td>
      </tr>
    </tbody>
  </table>
</div>
<div id="export" class="advanced-export">
  <h3>Advanced export</h3>
  <p>JSON shape:
    <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json">default</a>,
    <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json?_shape=array">array</a>,
    <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json?_shape=array&amp;_nl=on">newline-delimited</a>
  </p>
  <form action="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.csv" method="get">
    <p>
      CSV options:            
      <label>
        <input type="checkbox" name="_dl"> download file
        </label>
        <input type="submit" value="Export CSV">
          <input type="hidden" name="_size" value="max">
          </p>
        </form>
      </div>

But here's something really weird - the links to the individual rows DO include the /datasette/ prefix:

<td class="col-Link type-pk">
    <a href="/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/2">2</a>
</td>

The navigation bar on that page is correct too:

<p class="crumbs">
        <a href="/datasette/">home</a> /
        <a href="/datasette/corpora-metadata">corpora-metadata</a>
</p>

I've also been unable to replicate this in my own local environment, running datasette fixtures.db --config base_url:/datasette/.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.json and .csv exports fail to apply base_url 742011049  
726412057 https://github.com/simonw/datasette/issues/865#issuecomment-726412057 https://api.github.com/repos/simonw/datasette/issues/865 MDEyOklzc3VlQ29tbWVudDcyNjQxMjA1Nw== simonw 9599 2020-11-12T23:49:23Z 2020-11-12T23:49:23Z OWNER

@tballison thanks, I've split that out into a new issue #1091

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
base_url doesn't seem to work when adding criteria and clicking "apply" 644582921  
726385782 https://github.com/simonw/datasette/issues/865#issuecomment-726385782 https://api.github.com/repos/simonw/datasette/issues/865 MDEyOklzc3VlQ29tbWVudDcyNjM4NTc4Mg== tballison 6739646 2020-11-12T22:41:06Z 2020-11-12T22:41:06Z NONE

The same is true if I select advanced export and hit the 'export csv' at the bottom of the page.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
base_url doesn't seem to work when adding criteria and clicking "apply" 644582921  
726385422 https://github.com/simonw/datasette/issues/865#issuecomment-726385422 https://api.github.com/repos/simonw/datasette/issues/865 MDEyOklzc3VlQ29tbWVudDcyNjM4NTQyMg== tballison 6739646 2020-11-12T22:40:14Z 2020-11-12T22:40:14Z NONE

Just tested with the latest Docker image, and it works pretty much everywhere! THANK YOU!

I did notice that if I try to export json or csv, the base is not applied. Not sure if I should reopen this issue or open a new one.

To see this, go here: https://corpora.tika.apache.org/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES

Click/hover over json or CSV and you'll see that the 'datasette' base is not included.

Again, many thanks!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
base_url doesn't seem to work when adding criteria and clicking "apply" 644582921  
726127465 https://github.com/simonw/datasette/issues/1089#issuecomment-726127465 https://api.github.com/repos/simonw/datasette/issues/1089 MDEyOklzc3VlQ29tbWVudDcyNjEyNzQ2NQ== simonw 9599 2020-11-12T14:54:11Z 2020-11-12T14:54:11Z OWNER

Suggested list to look out for from that PR:

  • simply/simple
  • easy/easier/easiest
  • obvious/obviously
  • just
  • merely
  • straightforward
  • ridiculous
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Sweep documentation for words that minimize involved difficulty 741665726  
725830716 https://github.com/simonw/datasette/issues/1088#issuecomment-725830716 https://api.github.com/repos/simonw/datasette/issues/1088 MDEyOklzc3VlQ29tbWVudDcyNTgzMDcxNg== simonw 9599 2020-11-12T04:35:38Z 2020-11-12T04:35:38Z OWNER

I'm going to fix this without a test, because writing a test for this is a bit fiddly and it's a very minor bug. If it comes back again I'll do the work to test for it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
OperationalError('interrupted') can 500 on row page 741268956  
725830533 https://github.com/simonw/datasette/issues/1088#issuecomment-725830533 https://api.github.com/repos/simonw/datasette/issues/1088 MDEyOklzc3VlQ29tbWVudDcyNTgzMDUzMw== simonw 9599 2020-11-12T04:35:08Z 2020-11-12T04:35:08Z OWNER

Yup, swapping QueryInterrupted fixed this against my giant database.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
OperationalError('interrupted') can 500 on row page 741268956  
725829903 https://github.com/simonw/datasette/issues/1088#issuecomment-725829903 https://api.github.com/repos/simonw/datasette/issues/1088 MDEyOklzc3VlQ29tbWVudDcyNTgyOTkwMw== simonw 9599 2020-11-12T04:33:14Z 2020-11-12T04:33:14Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
OperationalError('interrupted') can 500 on row page 741268956  
725731685 https://github.com/simonw/datasette/pull/1085#issuecomment-725731685 https://api.github.com/repos/simonw/datasette/issues/1085 MDEyOklzc3VlQ29tbWVudDcyNTczMTY4NQ== codecov[bot] 22429695 2020-11-12T00:01:18Z 2020-11-12T00:01:18Z NONE

Codecov Report

Merging #1085 (51e7651) into main (2a981e2) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1085   +/-   ##
=======================================
  Coverage   91.38%   91.38%           
=======================================
  Files          30       30           
  Lines        3785     3785           
=======================================
  Hits         3459     3459           
  Misses        326      326           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2a981e2...51e7651. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use FTS4 in fixtures 740512882  
725729857 https://github.com/simonw/datasette/issues/1086#issuecomment-725729857 https://api.github.com/repos/simonw/datasette/issues/1086 MDEyOklzc3VlQ29tbWVudDcyNTcyOTg1Nw== simonw 9599 2020-11-11T23:55:39Z 2020-11-11T23:55:39Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Foreign keys with blank titles result in non-clickable links 741021342  
725714908 https://github.com/simonw/datasette/issues/1086#issuecomment-725714908 https://api.github.com/repos/simonw/datasette/issues/1086 MDEyOklzc3VlQ29tbWVudDcyNTcxNDkwOA== simonw 9599 2020-11-11T23:17:26Z 2020-11-11T23:17:26Z OWNER

I'm just going to use a regular coloured hyphen.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Foreign keys with blank titles result in non-clickable links 741021342  
725622784 https://github.com/simonw/datasette/issues/1086#issuecomment-725622784 https://api.github.com/repos/simonw/datasette/issues/1086 MDEyOklzc3VlQ29tbWVudDcyNTYyMjc4NA== simonw 9599 2020-11-11T19:41:45Z 2020-11-11T19:41:45Z OWNER

Maybe use a grey hyphen here?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Foreign keys with blank titles result in non-clickable links 741021342  
723829147 https://github.com/simonw/sqlite-utils/issues/168#issuecomment-723829147 https://api.github.com/repos/simonw/sqlite-utils/issues/168 MDEyOklzc3VlQ29tbWVudDcyMzgyOTE0Nw== simonw 9599 2020-11-09T07:43:30Z 2020-11-09T07:43:30Z OWNER

Yeah whatever process the have in place is working great without any extra intervention: they upgraded to 3.0 four hours ago.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Automate (as much as possible) updates published to Homebrew 706167456  
723740546 https://github.com/simonw/datasette/issues/268#issuecomment-723740546 https://api.github.com/repos/simonw/datasette/issues/268 MDEyOklzc3VlQ29tbWVudDcyMzc0MDU0Ng== simonw 9599 2020-11-09T04:01:50Z 2020-11-09T04:01:50Z OWNER

I should depend on sqlite-fts4 - I'm doing that in sqlite-utils now and it works great: https://github.com/simonw/sqlite-utils/issues/198

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Mechanism for ranking results from SQLite full-text search 323718842  
723637930 https://github.com/simonw/sqlite-utils/issues/192#issuecomment-723637930 https://api.github.com/repos/simonw/sqlite-utils/issues/192 MDEyOklzc3VlQ29tbWVudDcyMzYzNzkzMA== simonw 9599 2020-11-08T17:06:56Z 2020-11-08T17:06:56Z OWNER

This looks pretty good now!

% sqlite-utils search 24ways.db articles simon -c title -c author -t 
title                                                                          author
-----------------------------------------------------------------------------  ------------------
Don't be eval()                                                                Simon Willison
DOM Scripting Your Way to Better Blockquotes                                   Jeremy Keith
Swooshy Curly Quotes Without Images                                            Simon Collison
The Articulate Web Designer of Tomorrow                                        Simon Collison
Writing Responsible JavaScript                                                 Drew McLellan
Going Nuts with CSS Transitions                                                Natalie Downe
Managing a Mind                                                                Christopher Murphy
Design Systems                                                                 Laura Kalbag
Bringing Your Code to the Streets                                              Ruth John
Taming Complexity                                                              Simon Collison
Unobtrusively Mapping Microformats with jQuery                                 Simon Willison
Crafting the Front-end                                                         Ben Bodien
Develop Your Naturalist Superpowers with Observable Notebooks and iNaturalist  Natalie Downe
Fast Autocomplete Search for Your Website                                      Simon Willison
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
sqlite-utils search command 735532751  
723369033 https://github.com/simonw/sqlite-utils/issues/201#issuecomment-723369033 https://api.github.com/repos/simonw/sqlite-utils/issues/201 MDEyOklzc3VlQ29tbWVudDcyMzM2OTAzMw== simonw 9599 2020-11-07T01:28:11Z 2020-11-07T01:28:11Z OWNER

Need to fix this to close #192 and #197.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.search(columns=) and sqlite-utils search -c ... bug 738128913  
723368528 https://github.com/simonw/sqlite-utils/issues/194#issuecomment-723368528 https://api.github.com/repos/simonw/sqlite-utils/issues/194 MDEyOklzc3VlQ29tbWVudDcyMzM2ODUyOA== simonw 9599 2020-11-07T01:24:55Z 2020-11-07T01:24:55Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
3.0 release with some minor breaking changes 735650864  
723365651 https://github.com/simonw/sqlite-utils/issues/197#issuecomment-723365651 https://api.github.com/repos/simonw/sqlite-utils/issues/197 MDEyOklzc3VlQ29tbWVudDcyMzM2NTY1MQ== simonw 9599 2020-11-07T01:06:32Z 2020-11-07T01:06:32Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rethink how table.search() method works 737153927  
723360842 https://github.com/simonw/sqlite-utils/issues/197#issuecomment-723360842 https://api.github.com/repos/simonw/sqlite-utils/issues/197 MDEyOklzc3VlQ29tbWVudDcyMzM2MDg0Mg== simonw 9599 2020-11-07T00:40:55Z 2020-11-07T00:40:55Z OWNER

The order= parameter should be called order_by for consistency with this:

for row in db["dogs"].rows_where("age > 1", order_by="age"):
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rethink how table.search() method works 737153927  
723357855 https://github.com/simonw/sqlite-utils/issues/200#issuecomment-723357855 https://api.github.com/repos/simonw/sqlite-utils/issues/200 MDEyOklzc3VlQ29tbWVudDcyMzM1Nzg1NQ== simonw 9599 2020-11-07T00:24:37Z 2020-11-07T00:24:37Z OWNER
(sqlite-utils) sqlite-utils % sqlite-utils rows 24ways-fts4.db articles -c title -c author | head -n 3
[{"title": "Why Bother with Accessibility?", "author": "Laura Kalbag"},
 {"title": "Levelling Up", "author": "Ashley Baxter"},
 {"title": "Project Hubs: A Home Base for Design Projects", "author": "Brad Frost"},
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
sqlite-utils rows -c option 738115165  
723357117 https://github.com/simonw/sqlite-utils/issues/192#issuecomment-723357117 https://api.github.com/repos/simonw/sqlite-utils/issues/192 MDEyOklzc3VlQ29tbWVudDcyMzM1NzExNw== simonw 9599 2020-11-07T00:21:05Z 2020-11-07T00:21:05Z OWNER

I'm going to have it only output the exact -c columns you requested (if you requested any). Add --rank to specify the rank column, since you may not know what its name is.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
sqlite-utils search command 735532751  
723356020 https://github.com/simonw/sqlite-utils/issues/194#issuecomment-723356020 https://api.github.com/repos/simonw/sqlite-utils/issues/194 MDEyOklzc3VlQ29tbWVudDcyMzM1NjAyMA== simonw 9599 2020-11-07T00:16:06Z 2020-11-07T00:16:06Z OWNER

I'm going to release this as an alpha first and sit on it for a few days, since I don't want to ship any mistakes that would result in having to bump straight to 4.0!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
3.0 release with some minor breaking changes 735650864  
723350956 https://github.com/simonw/sqlite-utils/issues/91#issuecomment-723350956 https://api.github.com/repos/simonw/sqlite-utils/issues/91 MDEyOklzc3VlQ29tbWVudDcyMzM1MDk1Ng== simonw 9599 2020-11-06T23:53:25Z 2020-11-06T23:53:25Z OWNER

This is now possible, for both FTS4 and FTS5 - see #197.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Enable ordering FTS results by rank 577302229  
723348722 https://github.com/simonw/sqlite-utils/issues/192#issuecomment-723348722 https://api.github.com/repos/simonw/sqlite-utils/issues/192 MDEyOklzc3VlQ29tbWVudDcyMzM0ODcyMg== simonw 9599 2020-11-06T23:43:09Z 2020-11-06T23:43:09Z OWNER

Also that order looks incorrect. It looks like most relevant came back last, not first.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
sqlite-utils search command 735532751  
723348614 https://github.com/simonw/sqlite-utils/issues/192#issuecomment-723348614 https://api.github.com/repos/simonw/sqlite-utils/issues/192 MDEyOklzc3VlQ29tbWVudDcyMzM0ODYxNA== simonw 9599 2020-11-06T23:42:38Z 2020-11-06T23:42:38Z OWNER

This is a bit surprising:

(sqlite-utils) sqlite-utils % sqlite-utils search 24ways-fts4.db articles maps -c title      
[{"rowid": 41, "title": "What Is Vagrant and Why Should I Care?", "rank": -1.9252039178908076},
 {"rowid": 298, "title": "First Steps in VR", "rank": -1.9945466378736434},
 {"rowid": 43, "title": "Content Production Planning", "rank": -2.1928058363046143},
 {"rowid": 100, "title": "Moo'y Christmas", "rank": -2.2698482999851675},
 {"rowid": 91, "title": "Infinite Canvas: Moving Beyond the Page", "rank": -2.290928999035195},
 {"rowid": 175, "title": "Front-End Code Reusability with CSS and JavaScript", "rank": -2.498731782924352},
 {"rowid": 209, "title": "Feeding the Audio Graph", "rank": -2.619968930100356},
 {"rowid": 296, "title": "Animation in Design Systems", "rank": -2.62060151817201},
 {"rowid": 118, "title": "Ghosts On The Internet", "rank": -2.7224894534521087},
 {"rowid": 77, "title": "Colour Accessibility", "rank": -2.7389782859427343},
 {"rowid": 245, "title": "Web Content Accessibility Guidelines 2.1\u2014for People Who Haven\u2019t Read the Update", "rank": -2.9750992611162888},
 {"rowid": 56, "title": "Helping VIPs Care About Performance", "rank": -3.0819662908932535},
 {"rowid": 109, "title": "Geotag Everywhere with Fire Eagle", "rank": -3.1371975973877277},
 {"rowid": 203, "title": "Jobs-to-Be-Done in Your UX Toolbox", "rank": -3.2416719461682733},
 {"rowid": 276, "title": "Your jQuery: Now With 67% Less Suck", "rank": -3.4947916564653028},
 {"rowid": 58, "title": "Beyond the Style Guide", "rank": -3.7508321464447905},
 {"rowid": 225, "title": "Good Ideas Grow on Paper", "rank": -4.120077674716844},
 {"rowid": 168, "title": "Unobtrusively Mapping Microformats with jQuery", "rank": -4.662224207228984},
 {"rowid": 27, "title": "Putting Design on the Map", "rank": -5.667327088267961},
 {"rowid": 220, "title": "Finding Your Way with Static Maps", "rank": -9.952534352591737}]

I requested just -c title but also got back rowid and rank.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
sqlite-utils search command 735532751  
723234493 https://github.com/simonw/sqlite-utils/issues/194#issuecomment-723234493 https://api.github.com/repos/simonw/sqlite-utils/issues/194 MDEyOklzc3VlQ29tbWVudDcyMzIzNDQ5Mw== simonw 9599 2020-11-06T18:32:34Z 2020-11-06T18:32:34Z OWNER

The breaking changes will be:

  • table.search() now returns a generator that produces dictionaries, similar to table.rows_where()
  • The -c shortcut no longer works, use --csv instead.
  • The -f shortcut no longer works, use --fmt instead.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
3.0 release with some minor breaking changes 735650864  
723230732 https://github.com/simonw/sqlite-utils/issues/197#issuecomment-723230732 https://api.github.com/repos/simonw/sqlite-utils/issues/197 MDEyOklzc3VlQ29tbWVudDcyMzIzMDczMg== simonw 9599 2020-11-06T18:24:29Z 2020-11-06T18:24:29Z OWNER

Still need to update docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rethink how table.search() method works 737153927  
723148906 https://github.com/simonw/sqlite-utils/pull/195#issuecomment-723148906 https://api.github.com/repos/simonw/sqlite-utils/issues/195 MDEyOklzc3VlQ29tbWVudDcyMzE0ODkwNg== simonw 9599 2020-11-06T15:43:51Z 2020-11-06T15:43:51Z OWNER

Thanks to #198 (introducing a rank_bm25() custom function for FTS4) this feature will be able to offer relevance search for both FTS5 AND FTS4 tables.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.search() improvements plus sqlite-utils search command 735663855  
723148310 https://github.com/simonw/sqlite-utils/issues/197#issuecomment-723148310 https://api.github.com/repos/simonw/sqlite-utils/issues/197 MDEyOklzc3VlQ29tbWVudDcyMzE0ODMxMA== simonw 9599 2020-11-06T15:42:43Z 2020-11-06T15:42:43Z OWNER

Having .search() return tuples when .rows_where() returns dictionaries just feels like bad API design to me - it's inconsistent.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rethink how table.search() method works 737153927  
723147463 https://github.com/simonw/sqlite-utils/issues/199#issuecomment-723147463 https://api.github.com/repos/simonw/sqlite-utils/issues/199 MDEyOklzc3VlQ29tbWVudDcyMzE0NzQ2Mw== simonw 9599 2020-11-06T15:41:00Z 2020-11-06T15:41:00Z OWNER

Something like this:

@db.register_function(replace=True)
def my_function(a):
    return a.upper()

If replace=True then this function will be registered even if a my_function of arity 1 has already been registered previously.

It defaults to False though which means the Database object tracks what functions and arities have been registered in the past and silently ignores any new attempts to register the same name/arity.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
@db.register_function(..., replace=False) to avoid double-registering custom functions 737855731  
723145383 https://github.com/simonw/sqlite-utils/issues/198#issuecomment-723145383 https://api.github.com/repos/simonw/sqlite-utils/issues/198 MDEyOklzc3VlQ29tbWVudDcyMzE0NTM4Mw== simonw 9599 2020-11-06T15:36:47Z 2020-11-06T15:36:47Z OWNER

Should I register the custom rank_bm25 SQLite function for every connection, or should I register it against the connection just the first time the user attempts an FTS4 search? I think I'd rather register it only if it is needed.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support order by relevance against FTS4 737476423  
723144893 https://github.com/simonw/sqlite-utils/issues/198#issuecomment-723144893 https://api.github.com/repos/simonw/sqlite-utils/issues/198 MDEyOklzc3VlQ29tbWVudDcyMzE0NDg5Mw== simonw 9599 2020-11-06T15:35:45Z 2020-11-06T15:35:45Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support order by relevance against FTS4 737476423  
723143633 https://github.com/simonw/sqlite-utils/issues/198#issuecomment-723143633 https://api.github.com/repos/simonw/sqlite-utils/issues/198 MDEyOklzc3VlQ29tbWVudDcyMzE0MzYzMw== simonw 9599 2020-11-06T15:33:12Z 2020-11-06T15:33:12Z OWNER

Here's the FTS5 query:

with original as (
    select
        rowid,
        *
    from [global-power-plants]
)
select
    original.*,
    [global-power-plants_fts].rank as rank
from
    [original]
    join [global-power-plants_fts] on [original].rowid = [global-power-plants_fts].rowid
where
    [global-power-plants_fts] match :query
order by
    rank desc
limit 20

The equivalent using rank_bm25() for FTS4 would be:

with original as (
    select
        rowid,
        *
    from [global-power-plants]
)
select
    original.*,
    rank_bm25(matchinfo([global-power-plants_fts], 'pcnalx')) as rank
from
    [original]
    join [global-power-plants_fts] on [original].rowid = [global-power-plants_fts].rowid
where
    [global-power-plants_fts] match :query
order by
    rank desc
limit 20
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support order by relevance against FTS4 737476423  
722895825 https://github.com/simonw/sqlite-utils/issues/198#issuecomment-722895825 https://api.github.com/repos/simonw/sqlite-utils/issues/198 MDEyOklzc3VlQ29tbWVudDcyMjg5NTgyNQ== simonw 9599 2020-11-06T06:29:17Z 2020-11-06T06:29:17Z OWNER

I released a 1.0 (and 1.0.1) version of https://github.com/simonw/sqlite-fts4

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support order by relevance against FTS4 737476423  
722852262 https://github.com/simonw/sqlite-utils/issues/198#issuecomment-722852262 https://api.github.com/repos/simonw/sqlite-utils/issues/198 MDEyOklzc3VlQ29tbWVudDcyMjg1MjI2Mg== simonw 9599 2020-11-06T05:41:58Z 2020-11-06T05:41:58Z OWNER

Example query (from the tests):

select c0, c1, rank_bm25(matchinfo(search, 'pcnalx')) as bm25
        from search where search match ?
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support order by relevance against FTS4 737476423  
722849539 https://github.com/simonw/sqlite-utils/issues/198#issuecomment-722849539 https://api.github.com/repos/simonw/sqlite-utils/issues/198 MDEyOklzc3VlQ29tbWVudDcyMjg0OTUzOQ== simonw 9599 2020-11-06T05:39:17Z 2020-11-06T05:39:17Z OWNER

I'd have to copy almost all of the code in https://github.com/simonw/sqlite-fts4/blob/master/sqlite_fts4/__init__.py so I think I will add it as a dependency instead.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support order by relevance against FTS4 737476423  
722545442 https://github.com/simonw/sqlite-utils/issues/197#issuecomment-722545442 https://api.github.com/repos/simonw/sqlite-utils/issues/197 MDEyOklzc3VlQ29tbWVudDcyMjU0NTQ0Mg== simonw 9599 2020-11-05T18:05:33Z 2020-11-05T18:05:33Z OWNER

This is likely to result in a 3.0 release due to a backwards-incompatible change to the current .search() method - #194

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rethink how table.search() method works 737153927  
722542895 https://github.com/simonw/sqlite-utils/pull/195#issuecomment-722542895 https://api.github.com/repos/simonw/sqlite-utils/issues/195 MDEyOklzc3VlQ29tbWVudDcyMjU0Mjg5NQ== simonw 9599 2020-11-05T18:01:33Z 2020-11-05T18:01:33Z OWNER

Latest test failure:

114  ->         assert [("racoons are biting trash pandas", "USA", "bar")] == table.search(
115                 "bite", order="rowid"
116             )
117     
118     
119     def test_optimize_fts(fresh_db):
(Pdb) table.search("bite")
[(2, 'racoons are biting trash pandas', 'USA', 'bar', -9.641434262948206e-07)]

The problem here is that the table.search() method now behaves differently for FTS4 v.s. FTS5 tables.

With FTS4 you get back just the table columns.

With FTS5 you also get back the rowid as the first column and the rank score as the last column.

This is weird. It also makes me question whether having .search() return a list of tuples is the right API design.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
table.search() improvements plus sqlite-utils search command 735663855  
722086105 https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722086105 https://api.github.com/repos/simonw/sqlite-utils/issues/196 MDEyOklzc3VlQ29tbWVudDcyMjA4NjEwNQ== simonw 9599 2020-11-05T02:29:50Z 2020-11-05T03:39:58Z OWNER

The finished monster:

_virtual_table_using_re = re.compile(r"""
^ # Start of string
\s*CREATE\s+VIRTUAL\s+TABLE\s+ # CREATE VIRTUAL TABLE
(
    '(?P<squoted_table>[^']*(?:''[^']*)*)' | # single quoted name
    "(?P<dquoted_table>[^"]*(?:""[^"]*)*)" | # double quoted name
    `(?P<backtick_table>[^`]+)`            | # `backtick` quoted name
    \[(?P<squarequoted_table>[^\]]+)\]     | # [...] quoted name
    (?P<identifier>                          # SQLite non-quoted identifier
        [A-Za-z_\u0080-\uffff]  # \u0080-\uffff = "any character larger than u007f"
        [A-Za-z_\u0080-\uffff0-9\$]* # zero-or-more alphanemuric or $
    )
)
\s+(IF\s+NOT\s+EXISTS\s+)?      # IF NOT EXISTS (optional)
USING\s+(?P<using>\w+)          # e.g. USING FTS5
""", re.VERBOSE | re.IGNORECASE)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Introspect if table is FTS4 or FTS5 736520310  
722084593 https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722084593 https://api.github.com/repos/simonw/sqlite-utils/issues/196 MDEyOklzc3VlQ29tbWVudDcyMjA4NDU5Mw== simonw 9599 2020-11-05T02:24:47Z 2020-11-05T02:24:47Z OWNER

And an identifier is "ALPHABETIC character and continue with zero or more ALPHANUMERIC characters and/or "$" (u0024) characters"

So...

[\u0041-\u005a\u0061-\u0071\u007f-\uffff\u005f][\u0041-\u005a\u0061-\u0071\u007f-\uffff\u005f\u0030-\u0039\u0024]+
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Introspect if table is FTS4 or FTS5 736520310  
722084213 https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722084213 https://api.github.com/repos/simonw/sqlite-utils/issues/196 MDEyOklzc3VlQ29tbWVudDcyMjA4NDIxMw== simonw 9599 2020-11-05T02:23:37Z 2020-11-05T02:23:37Z OWNER

So...

ALPHABETIC: [\u0041-\u005a\u0061-\u0071\u007f-\uffff\u005f]

NUMERIC: [\u0030-\u0039]

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Introspect if table is FTS4 or FTS5 736520310  
722083527 https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722083527 https://api.github.com/repos/simonw/sqlite-utils/issues/196 MDEyOklzc3VlQ29tbWVudDcyMjA4MzUyNw== simonw 9599 2020-11-05T02:21:26Z 2020-11-05T02:21:26Z OWNER

I think that's \u007F-\uFFFF in regex range speak.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Introspect if table is FTS4 or FTS5 736520310  
722082874 https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722082874 https://api.github.com/repos/simonw/sqlite-utils/issues/196 MDEyOklzc3VlQ29tbWVudDcyMjA4Mjg3NA== simonw 9599 2020-11-05T02:19:18Z 2020-11-05T02:19:18Z OWNER

"any other character larger than u007f." Need to figure that out!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Introspect if table is FTS4 or FTS5 736520310  
722082759 https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722082759 https://api.github.com/repos/simonw/sqlite-utils/issues/196 MDEyOklzc3VlQ29tbWVudDcyMjA4Mjc1OQ== simonw 9599 2020-11-05T02:18:58Z 2020-11-05T02:18:58Z OWNER

More from that document, describing ALPHANUMERIC:

ALPHABETIC

Any of the characters in the range u0041 through u005a (letters "A" through "Z") or in the range u0061 through u007a (letters "a" through "z") or the character u005f ("_") or any other character larger than u007f.

NUMERIC

Any of the characters in the range u0030 through u0039 (digits "0" through "9")

ALPHANUMERIC

Any character which is either ALPHABETIC or NUMERIC

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Introspect if table is FTS4 or FTS5 736520310  
722082497 https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722082497 https://api.github.com/repos/simonw/sqlite-utils/issues/196 MDEyOklzc3VlQ29tbWVudDcyMjA4MjQ5Nw== simonw 9599 2020-11-05T02:18:08Z 2020-11-05T02:18:08Z OWNER

I'm missing the case where a table has no quotes around it at all - create virtual table foo using fts5

So I need to know how to create a regex for a SQLite identifier. https://www.sqlite.org/draft/tokenreq.html seems to be the only available documentation for that.

Identifier tokens

Identifiers follow the usual rules with the exception that SQLite allows the dollar-sign symbol in the interior of an identifier. The dollar-sign is for compatibility with Microsoft SQL-Server and is not part of the SQL standard.

H41130: SQLite shall recognize as an ID token any sequence of characters that begins with an ALPHABETIC character and continue with zero or more ALPHANUMERIC characters and/or "$" (u0024) characters and which is not a keyword token.

Identifiers can be arbitrary character strings within square brackets. This feature is also for compatibility with Microsoft SQL-Server and not a part of the SQL standard.

H41140: SQLite shall recognize as an ID token any sequence of non-zero characters that begins with "[" (u005b) and continuing through the first "]" (u005d) character.

The standard way of quoting SQL identifiers is to use double-quotes.

H41150: SQLite shall recognize as an ID token any sequence of characters that begins with a double-quote (u0022), is followed by zero or more non-zero characters and/or pairs of double-quotes (u0022) and terminates with a double-quote (u0022) that is not part of a pair.

MySQL allows identifiers to be quoted using the grave accent character. SQLite supports this for interoperability.

H41160: SQLite shall recognize as an ID token any sequence of characters that begins with a grave accent (u0060), is followed by zero or more non-zero characters and/or pairs ofgrave accents (u0060) and terminates with a grave accent (u0022) that is not part of a pair.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Introspect if table is FTS4 or FTS5 736520310  
722078361 https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722078361 https://api.github.com/repos/simonw/sqlite-utils/issues/196 MDEyOklzc3VlQ29tbWVudDcyMjA3ODM2MQ== simonw 9599 2020-11-05T02:04:33Z 2020-11-05T02:04:33Z OWNER

Next step: lots of unit tests.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Introspect if table is FTS4 or FTS5 736520310  

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);