5,002 rows sorted by updated_at descending

View and edit SQL

Suggested facets: reactions, created_at (date), updated_at (date)

issue

author_association

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
761705076 https://github.com/simonw/datasette/issues/1191#issuecomment-761705076 https://api.github.com/repos/simonw/datasette/issues/1191 MDEyOklzc3VlQ29tbWVudDc2MTcwNTA3Ng== simonw 9599 2021-01-17T00:35:13Z 2021-01-17T00:37:51Z OWNER

I'm going to try using Jinja macros to implement this: https://jinja.palletsprojects.com/en/2.11.x/templates/#macros

Maybe using one of these tricks to auto-load the macro? http://codyaray.com/2015/05/auto-load-jinja2-macros

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability for plugins to collaborate when adding extra HTML to blocks in default templates 787098345  
761703555 https://github.com/simonw/datasette/issues/1181#issuecomment-761703555 https://api.github.com/repos/simonw/datasette/issues/1181 MDEyOklzc3VlQ29tbWVudDc2MTcwMzU1NQ== simonw 9599 2021-01-17T00:24:20Z 2021-01-17T00:24:40Z OWNER

Here's the incomplete sketch of a test - to go at the bottom of test_cli.py.

@pytest.mark.parametrize(
    "filename", ["test-database (1).sqlite", "database (1).sqlite"]
)
def test_weird_database_names(ensure_eventloop, tmpdir, filename):
    # https://github.com/simonw/datasette/issues/1181
    runner = CliRunner()
    db_path = str(tmpdir / filename)
    sqlite3.connect(db_path).execute("vacuum")
    result1 = runner.invoke(cli, [db_path, "--get", "/"])
    assert result1.exit_code == 0, result1.output
    homepage_html = result1.output
    assert False
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Certain database names results in 404: "Database not found: None" 781262510  
761703368 https://github.com/simonw/datasette/issues/1191#issuecomment-761703368 https://api.github.com/repos/simonw/datasette/issues/1191 MDEyOklzc3VlQ29tbWVudDc2MTcwMzM2OA== simonw 9599 2021-01-17T00:22:46Z 2021-01-17T00:22:46Z OWNER

I'm going to prototype this in a branch.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability for plugins to collaborate when adding extra HTML to blocks in default templates 787098345  
761703232 https://github.com/simonw/datasette/issues/1191#issuecomment-761703232 https://api.github.com/repos/simonw/datasette/issues/1191 MDEyOklzc3VlQ29tbWVudDc2MTcwMzIzMg== simonw 9599 2021-01-17T00:21:31Z 2021-01-17T00:21:54Z OWNER

I think this ends up being a whole collection of new plugin hooks, something like:

  • include_table_top
  • include_table_bottom
  • include_row_top
  • include_row_bottom
  • include_database_top
  • include_database_bottom
  • include_query_bottom
  • include_query_bottom
  • include_index_bottom
  • include_index_bottom
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability for plugins to collaborate when adding extra HTML to blocks in default templates 787098345  
761703022 https://github.com/simonw/datasette/issues/1191#issuecomment-761703022 https://api.github.com/repos/simonw/datasette/issues/1191 MDEyOklzc3VlQ29tbWVudDc2MTcwMzAyMg== simonw 9599 2021-01-17T00:20:00Z 2021-01-17T00:20:00Z OWNER

Plugins that want to provide extra context to the template can already do so using the extra_template_vars() plugin hook.

So this hook could work by returning a list of template filenames to be included. Those templates can be bundled with the plugin. Since they are included they will have access to the template context and to any extra_template_vars() values.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability for plugins to collaborate when adding extra HTML to blocks in default templates 787098345  
761179229 https://github.com/simonw/datasette/issues/657#issuecomment-761179229 https://api.github.com/repos/simonw/datasette/issues/657 MDEyOklzc3VlQ29tbWVudDc2MTE3OTIyOQ== simonw 9599 2021-01-15T20:24:35Z 2021-01-15T20:24:35Z OWNER

I'm not sure how I missed this issue but it's almost a year later and I'm finally taking a look at your Parquet work.

This is yet more evidence that allowing plugins to provide their own custom Database objects would be a good idea.

I started exploring what Datasette would like on PostgreSQL in #670 - my concern was that I would need to add a large amount of database abstraction code which would dramatically increase the complexity of the core project, but my thinking now is that it might be tractable - Datasette doesn't actually construct SQL in complex ways anywhere outside of the TableView class so abstracting away just that bit should be feasible.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Allow creation of virtual tables at startup 548591089  
761104933 https://github.com/simonw/datasette/issues/1191#issuecomment-761104933 https://api.github.com/repos/simonw/datasette/issues/1191 MDEyOklzc3VlQ29tbWVudDc2MTEwNDkzMw== simonw 9599 2021-01-15T18:21:26Z 2021-01-15T18:21:26Z OWNER

Also related: #857 (comprehensive documentation of variables available to templates) - since then the plugin hook could be fed the full template context and use that to do its thing.

Or maybe the plugin hooks get to return the name of a template that should be {% include %} into the page at that point? But the plugin may want to add extra context that is available to that template include.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability for plugins to collaborate when adding extra HTML to blocks in default templates 787098345  
761103910 https://github.com/simonw/datasette/issues/1191#issuecomment-761103910 https://api.github.com/repos/simonw/datasette/issues/1191 MDEyOklzc3VlQ29tbWVudDc2MTEwMzkxMA== simonw 9599 2021-01-15T18:19:29Z 2021-01-15T18:19:29Z OWNER

This relates to #987 (documented HTML hooks for JavaScript plugins) but is not quite the same thing.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability for plugins to collaborate when adding extra HTML to blocks in default templates 787098345  
761101878 https://github.com/simonw/datasette/issues/657#issuecomment-761101878 https://api.github.com/repos/simonw/datasette/issues/657 MDEyOklzc3VlQ29tbWVudDc2MTEwMTg3OA== simonw 9599 2021-01-15T18:16:01Z 2021-01-15T18:16:01Z OWNER

I think the startup() plugin hook at https://docs.datasette.io/en/stable/plugin_hooks.html#startup-datasette should be able to fit this. You can write a plugin which uses that hook to execute CREATE VIRTUAL TABLE against one or more databases when Datasette first starts running.

Would that work here?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Allow creation of virtual tables at startup 548591089  
761015218 https://github.com/simonw/sqlite-utils/issues/220#issuecomment-761015218 https://api.github.com/repos/simonw/sqlite-utils/issues/220 MDEyOklzc3VlQ29tbWVudDc2MTAxNTIxOA== mhalle 649467 2021-01-15T15:40:08Z 2021-01-15T15:40:08Z NONE

Make sense. If you're coming from the sqlite3 side of things, rather than the datasette side, wanting the fts methods to work for views makes more sense. sqlite3 allows fts5 tables on views, so I was looking for CLI functionality to build the fts virtual tables. Ultimately, though, sharing fts virtual tables across tables and derivative views is likely more efficient.

Maybe an explicit error message like, "fts is not supported for views" rather than just throwing an exception that the method doesn't exist" might be helpful. Not critical though.

Thanks.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
*_fts methods don't work on views, only tables 783778672  
760950128 https://github.com/dogsheep/twitter-to-sqlite/pull/55#issuecomment-760950128 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/55 MDEyOklzc3VlQ29tbWVudDc2MDk1MDEyOA== jacobian 21148 2021-01-15T13:44:52Z 2021-01-15T13:44:52Z NONE

I found and fixed another bug, this one around importing the tweets table. @simonw let me know if you'd prefer this broken out into multiple PRs, happy to do that if it makes review/merging easier.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fix archive imports 779211940  
759875239 https://github.com/simonw/datasette/issues/1187#issuecomment-759875239 https://api.github.com/repos/simonw/datasette/issues/1187 MDEyOklzc3VlQ29tbWVudDc1OTg3NTIzOQ== simonw 9599 2021-01-14T02:02:24Z 2021-01-14T02:02:31Z OWNER

This plugin hook currently returns a string of JavaScript. It could optionally return this instead of a string:

{
  "script": "string of JavaScript goes here",
  "module": true
}
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
extra_body_script() support for script type="module" 785588942  
759874332 https://github.com/simonw/datasette/issues/1186#issuecomment-759874332 https://api.github.com/repos/simonw/datasette/issues/1186 MDEyOklzc3VlQ29tbWVudDc1OTg3NDMzMg== simonw 9599 2021-01-14T01:59:35Z 2021-01-14T01:59:35Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
script type="module" support 785573793  
759306228 https://github.com/simonw/datasette/pull/1159#issuecomment-759306228 https://api.github.com/repos/simonw/datasette/issues/1159 MDEyOklzc3VlQ29tbWVudDc1OTMwNjIyOA== lovasoa 552629 2021-01-13T08:59:31Z 2021-01-13T08:59:31Z NONE

@simonw : Did you have the time to take a look at this ?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Improve the display of facets information 774332247  
759098964 https://github.com/simonw/sqlite-utils/issues/220#issuecomment-759098964 https://api.github.com/repos/simonw/sqlite-utils/issues/220 MDEyOklzc3VlQ29tbWVudDc1OTA5ODk2NA== simonw 9599 2021-01-12T23:19:55Z 2021-01-12T23:19:55Z OWNER

I don't think it makes sense to call .enable_fts() on a view does it? When I'm working with views and FTS I tend to write my queries against a FTS table for one of the tables that is used by the view - https://docs.datasette.io/en/stable/full_text_search.html#configuring-full-text-search-for-a-table-or-view describes how I do that in Datasette for example.

Can you expand on your use-case for FTS and views? I'm ready to be convinced otherwise, but I don't see how it would work right now.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
*_fts methods don't work on views, only tables 783778672  
759069342 https://github.com/simonw/datasette/issues/1185#issuecomment-759069342 https://api.github.com/repos/simonw/datasette/issues/1185 MDEyOklzc3VlQ29tbWVudDc1OTA2OTM0Mg== simonw 9599 2021-01-12T22:13:18Z 2021-01-12T22:13:18Z OWNER

I'm going to change the error message to list the allowed pragmas.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"Statement may not contain PRAGMA" error is not strictly true 784628163  
759067427 https://github.com/simonw/datasette/issues/1185#issuecomment-759067427 https://api.github.com/repos/simonw/datasette/issues/1185 MDEyOklzc3VlQ29tbWVudDc1OTA2NzQyNw== simonw 9599 2021-01-12T22:09:21Z 2021-01-12T22:09:21Z OWNER

That allow-list was added in #761 but is not currently documented. It's here in the code:

https://github.com/simonw/datasette/blob/8e8fc5cee5c78da8334495c6d6257d5612c40792/datasette/utils/__init__.py#L173-L186

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"Statement may not contain PRAGMA" error is not strictly true 784628163  
759066777 https://github.com/simonw/datasette/issues/1185#issuecomment-759066777 https://api.github.com/repos/simonw/datasette/issues/1185 MDEyOklzc3VlQ29tbWVudDc1OTA2Njc3Nw== simonw 9599 2021-01-12T22:07:58Z 2021-01-12T22:07:58Z OWNER

https://docs.datasette.io/en/stable/sql_queries.html?highlight=pragma#named-parameters documentation is out-of-date as well:

Datasette disallows custom SQL containing the string PRAGMA, as SQLite pragma statements can be used to change database settings at runtime. If you need to include the string "pragma" in a query you can do so safely using a named parameter.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"Statement may not contain PRAGMA" error is not strictly true 784628163  
758668359 https://github.com/simonw/datasette/issues/1091#issuecomment-758668359 https://api.github.com/repos/simonw/datasette/issues/1091 MDEyOklzc3VlQ29tbWVudDc1ODY2ODM1OQ== tballison 6739646 2021-01-12T13:52:29Z 2021-01-12T13:52:29Z NONE

Y, thank you to both of you!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.json and .csv exports fail to apply base_url 742011049  
758448525 https://github.com/simonw/datasette/issues/1091#issuecomment-758448525 https://api.github.com/repos/simonw/datasette/issues/1091 MDEyOklzc3VlQ29tbWVudDc1ODQ0ODUyNQ== henry501 19328961 2021-01-12T06:55:08Z 2021-01-12T06:55:08Z NONE

Great, really happy I could help! Reverse proxies get tricky.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.json and .csv exports fail to apply base_url 742011049  
758356640 https://github.com/simonw/datasette/issues/1183#issuecomment-758356640 https://api.github.com/repos/simonw/datasette/issues/1183 MDEyOklzc3VlQ29tbWVudDc1ODM1NjY0MA== simonw 9599 2021-01-12T02:42:08Z 2021-01-12T02:42:08Z OWNER

Should Datasette have subcommands for this? datasette enable-counts data.db and datasette disable-counts data.db and datasette reset-counts data.db commands?

Maybe. The sqlite-utils CLI tool could be used here instead, but that won't be easily available if Datasette was installed as a standalone binary or using brew install datasette or pipx install datasette.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Take advantage of sqlite-utils cached table counts, if available 782708469  
758356097 https://github.com/simonw/datasette/issues/1183#issuecomment-758356097 https://api.github.com/repos/simonw/datasette/issues/1183 MDEyOklzc3VlQ29tbWVudDc1ODM1NjA5Nw== simonw 9599 2021-01-12T02:40:30Z 2021-01-12T02:40:30Z OWNER

So how would this work?

I think I'm going to automatically use these values if the _counts table exists, unless a ignore_counts_table boolean setting has been set. I won't bother looking to see if the triggers have been created.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Take advantage of sqlite-utils cached table counts, if available 782708469  
758283074 https://github.com/simonw/datasette/issues/1091#issuecomment-758283074 https://api.github.com/repos/simonw/datasette/issues/1091 MDEyOklzc3VlQ29tbWVudDc1ODI4MzA3NA== simonw 9599 2021-01-11T23:12:46Z 2021-01-11T23:12:46Z OWNER

Fantastic!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.json and .csv exports fail to apply base_url 742011049  
758280611 https://github.com/simonw/datasette/issues/1091#issuecomment-758280611 https://api.github.com/repos/simonw/datasette/issues/1091 MDEyOklzc3VlQ29tbWVudDc1ODI4MDYxMQ== tballison 6739646 2021-01-11T23:06:10Z 2021-01-11T23:06:10Z NONE

+1

Yep! Fixes it. If I navigate to https://corpora.tika.apache.org/datasette, I get a 404 (database not found: datasette), but if I navigate to https://corpora.tika.apache.org/datasette/file_profiles/, everything WORKS!

Thank you!

{
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 1,
    "eyes": 0
}
.json and .csv exports fail to apply base_url 742011049  
757375858 https://github.com/simonw/datasette/issues/1182#issuecomment-757375858 https://api.github.com/repos/simonw/datasette/issues/1182 MDEyOklzc3VlQ29tbWVudDc1NzM3NTg1OA== simonw 9599 2021-01-09T22:18:47Z 2021-01-09T22:18:47Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Retire "Ecosystem" page in favour of datasette.io/plugins and /tools 782692159  
757373741 https://github.com/simonw/datasette/issues/1182#issuecomment-757373741 https://api.github.com/repos/simonw/datasette/issues/1182 MDEyOklzc3VlQ29tbWVudDc1NzM3Mzc0MQ== simonw 9599 2021-01-09T22:01:41Z 2021-01-09T22:01:41Z OWNER

It can talk about Dogsheep too.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Retire "Ecosystem" page in favour of datasette.io/plugins and /tools 782692159  
757373082 https://github.com/simonw/datasette/issues/1182#issuecomment-757373082 https://api.github.com/repos/simonw/datasette/issues/1182 MDEyOklzc3VlQ29tbWVudDc1NzM3MzA4Mg== simonw 9599 2021-01-09T21:55:33Z 2021-01-09T21:55:33Z OWNER

I'll leave the page there but change it into more of a blurb about the existence of the plugins and tools directories.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Retire "Ecosystem" page in favour of datasette.io/plugins and /tools 782692159  
756487966 https://github.com/simonw/datasette/issues/1181#issuecomment-756487966 https://api.github.com/repos/simonw/datasette/issues/1181 MDEyOklzc3VlQ29tbWVudDc1NjQ4Nzk2Ng== simonw 9599 2021-01-08T01:25:42Z 2021-01-08T01:25:42Z OWNER

I'm going to add a unit test that tries a variety of weird database names.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Certain database names results in 404: "Database not found: None" 781262510  
756482163 https://github.com/simonw/datasette/issues/1181#issuecomment-756482163 https://api.github.com/repos/simonw/datasette/issues/1181 MDEyOklzc3VlQ29tbWVudDc1NjQ4MjE2Mw== simonw 9599 2021-01-08T01:06:23Z 2021-01-08T01:06:54Z OWNER

Yes, that logic is definitely at fault. It looks like it applies urllib.parse.unquote_plus() AFTER it's tried to do the - hash splitting thing, which is why it's failing here:

https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/views/base.py#L184-L198

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Certain database names results in 404: "Database not found: None" 781262510  
756453945 https://github.com/simonw/datasette/issues/1091#issuecomment-756453945 https://api.github.com/repos/simonw/datasette/issues/1091 MDEyOklzc3VlQ29tbWVudDc1NjQ1Mzk0NQ== simonw 9599 2021-01-07T23:42:50Z 2021-01-07T23:42:50Z OWNER

@henry501 it looks like you spotted a bug in the documentation - I just addressed that, the fix is now live here: https://docs.datasette.io/en/latest/deploying.html#running-datasette-behind-a-proxy

{
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 1,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.json and .csv exports fail to apply base_url 742011049  
756453010 https://github.com/simonw/datasette/issues/1091#issuecomment-756453010 https://api.github.com/repos/simonw/datasette/issues/1091 MDEyOklzc3VlQ29tbWVudDc1NjQ1MzAxMA== simonw 9599 2021-01-07T23:39:58Z 2021-01-07T23:40:25Z OWNER

@tballison I think that's the solution! It looks like you need to use this in your config:

ProxyPass /datasette http://127.0.0.1:8001/datasette

Instead of this:

ProxyPass /datasette http://127.0.0.1:8001/

Give that a go and let me know if it fixes it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.json and .csv exports fail to apply base_url 742011049  
756425587 https://github.com/simonw/datasette/issues/1091#issuecomment-756425587 https://api.github.com/repos/simonw/datasette/issues/1091 MDEyOklzc3VlQ29tbWVudDc1NjQyNTU4Nw== henry501 19328961 2021-01-07T22:27:19Z 2021-01-07T22:27:19Z NONE

I found this issue while troubleshooting the same behavior with an nginx reverse proxy. The solution was to make sure I set:

proxy_pass http://server:8001/baseurl/
instead of just:

proxy_pass http://server:8001
The custom SQL query and header links are now correct.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
.json and .csv exports fail to apply base_url 742011049  
756335394 https://github.com/simonw/datasette/issues/1171#issuecomment-756335394 https://api.github.com/repos/simonw/datasette/issues/1171 MDEyOklzc3VlQ29tbWVudDc1NjMzNTM5NA== simonw 9599 2021-01-07T19:35:59Z 2021-01-07T19:35:59Z OWNER

I requested a D-U-N-S number as a first step in getting a developer certificate: https://developer.apple.com/support/D-U-N-S/

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
GitHub Actions workflow to build and sign macOS binary executables 778450486  
756312213 https://github.com/simonw/datasette/issues/1180#issuecomment-756312213 https://api.github.com/repos/simonw/datasette/issues/1180 MDEyOklzc3VlQ29tbWVudDc1NjMxMjIxMw== simonw 9599 2021-01-07T18:56:24Z 2021-01-07T18:56:24Z OWNER

The async_call_with_supported_arguments version should be able to await any of the lazy arguments that are awaitable - can use await_me_maybe for that.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Lazily evaluated arguments for call_with_supported_arguments 780767542  
755500475 https://github.com/simonw/datasette/issues/1180#issuecomment-755500475 https://api.github.com/repos/simonw/datasette/issues/1180 MDEyOklzc3VlQ29tbWVudDc1NTUwMDQ3NQ== simonw 9599 2021-01-06T18:43:41Z 2021-01-06T18:43:41Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Lazily evaluated arguments for call_with_supported_arguments 780767542  
755495387 https://github.com/simonw/datasette/issues/1179#issuecomment-755495387 https://api.github.com/repos/simonw/datasette/issues/1179 MDEyOklzc3VlQ29tbWVudDc1NTQ5NTM4Nw== simonw 9599 2021-01-06T18:39:23Z 2021-01-06T18:39:23Z OWNER

In that case maybe there are three new arguments: path, full_path and url.

I'll also add request.full_path for consistency with these: https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/utils/asgi.py#L77-L90

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Make original path available to render hooks 780278550  
755492945 https://github.com/simonw/datasette/issues/1179#issuecomment-755492945 https://api.github.com/repos/simonw/datasette/issues/1179 MDEyOklzc3VlQ29tbWVudDc1NTQ5Mjk0NQ== simonw 9599 2021-01-06T18:37:39Z 2021-01-06T18:37:39Z OWNER

I think I'll call this full_path for consistency with Django.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Make original path available to render hooks 780278550  
755489974 https://github.com/simonw/datasette/issues/1179#issuecomment-755489974 https://api.github.com/repos/simonw/datasette/issues/1179 MDEyOklzc3VlQ29tbWVudDc1NTQ4OTk3NA== simonw 9599 2021-01-06T18:35:24Z 2021-01-06T18:35:24Z OWNER

Django calls this HttpRequest.get_full_path() - for the path plus the querystring.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Make original path available to render hooks 780278550  
755486103 https://github.com/simonw/datasette/issues/1179#issuecomment-755486103 https://api.github.com/repos/simonw/datasette/issues/1179 MDEyOklzc3VlQ29tbWVudDc1NTQ4NjEwMw== simonw 9599 2021-01-06T18:32:41Z 2021-01-06T18:34:11Z OWNER

This parameter will return the URL path, with querystring arguments, to the HTML version of the page - e.g. /github/issue_comments or /github/issue_comments?_sort_desc=created_at

Open questions:

  • What should it be called? path could be misleading since it also includes the querystring.
  • Should I provide a url or full_url version which includes https://blah.com/...?
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Make original path available to render hooks 780278550  
755484384 https://github.com/simonw/datasette/issues/782#issuecomment-755484384 https://api.github.com/repos/simonw/datasette/issues/782 MDEyOklzc3VlQ29tbWVudDc1NTQ4NDM4NA== simonw 9599 2021-01-06T18:31:14Z 2021-01-06T18:31:57Z OWNER

In building https://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_labels=on I discovered the following patterns for importing data into both Pandas and Observable/d3:

import pandas
df = pandas.read_json(
  "https://latest-with-plugins.datasette.io/github/issue_comments.json?_shape=array"
)

And:

d3 = require("d3@5")
rows = d3.json(
    "https://latest-with-plugins.datasette.io/github/issue_comments.json?_shape=array"
)

Once again I find myself torn on the best possible default. A list of JSON objects is instantly compatible with both pandas.read_json() and d3.json() - but it leaves nowhere to put the extra information like pagination and suchlike!

Even given this I still think the correct default is an object with "rows", "total" and "next_url" keys. I should commit to that and implement it - this thought exercise has been running for far too long.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Redesign default JSON format in preparation for Datasette 1.0 627794879  
755476820 https://github.com/simonw/datasette/issues/1178#issuecomment-755476820 https://api.github.com/repos/simonw/datasette/issues/1178 MDEyOklzc3VlQ29tbWVudDc1NTQ3NjgyMA== simonw 9599 2021-01-06T18:24:47Z 2021-01-06T18:24:47Z OWNER

Issue fixed - https://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_labels=on displays the correct schemes now.

I can't think of a reason anyone on Cloud Run would ever NOT want the force_https_urls option, but just in case I've made it so if you pass --extra-options --setting force_https_urls off to publish cloudrun your setting will be respected.

https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/publish/cloudrun.py#L105-L110

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use force_https_urls on when deploying with Cloud Run 780267857  
755468795 https://github.com/simonw/datasette/issues/1178#issuecomment-755468795 https://api.github.com/repos/simonw/datasette/issues/1178 MDEyOklzc3VlQ29tbWVudDc1NTQ2ODc5NQ== simonw 9599 2021-01-06T18:14:35Z 2021-01-06T18:14:35Z OWNER

Deploying that change now to test it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use force_https_urls on when deploying with Cloud Run 780267857  
755163886 https://github.com/simonw/datasette/issues/1178#issuecomment-755163886 https://api.github.com/repos/simonw/datasette/issues/1178 MDEyOklzc3VlQ29tbWVudDc1NTE2Mzg4Ng== simonw 9599 2021-01-06T08:37:51Z 2021-01-06T08:37:51Z OWNER

Easiest fix would be for publish cloudrun to set force_https_urls:

datasette publish now used to do this: https://github.com/simonw/datasette/blob/07e208cc6d9e901b87552c1be2854c220b3f9b6d/datasette/publish/now.py#L59-L63

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use force_https_urls on when deploying with Cloud Run 780267857  
755161574 https://github.com/simonw/datasette/issues/1179#issuecomment-755161574 https://api.github.com/repos/simonw/datasette/issues/1179 MDEyOklzc3VlQ29tbWVudDc1NTE2MTU3NA== simonw 9599 2021-01-06T08:32:31Z 2021-01-06T08:32:31Z OWNER

An optional path argument to https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette which shows the path WITHOUT the .Notebook extension would be useful here.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Make original path available to render hooks 780278550  
755160187 https://github.com/simonw/datasette/issues/1178#issuecomment-755160187 https://api.github.com/repos/simonw/datasette/issues/1178 MDEyOklzc3VlQ29tbWVudDc1NTE2MDE4Nw== simonw 9599 2021-01-06T08:29:35Z 2021-01-06T08:29:35Z OWNER

https://latest-with-plugins.datasette.io/-/asgi-scope

{'asgi': {'spec_version': '2.1', 'version': '3.0'},
 'client': ('169.254.8.129', 54971),
 'headers': [(b'host', b'latest-with-plugins.datasette.io'),
             (b'user-agent',
              b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko'
              b'/20100101 Firefox/84.0'),
             (b'accept',
              b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
              b'webp,*/*;q=0.8'),
             (b'accept-language', b'en-US,en;q=0.5'),
             (b'dnt', b'1'),
             (b'cookie',
              b'_ga_LL6M7BK6D4=GS1.1.1609886546.49.1.1609886923.0; _ga=GA1.1'
              b'.894633707.1607575712'),
             (b'upgrade-insecure-requests', b'1'),
             (b'x-client-data', b'CgSL6ZsV'),
             (b'x-cloud-trace-context',
              b'e776af843c657d2a3da28a73b726e6fe/14187666787557102189;o=1'),
             (b'x-forwarded-for', b'148.64.98.14'),
             (b'x-forwarded-proto', b'https'),
             (b'forwarded', b'for="148.64.98.14";proto=https'),
             (b'accept-encoding', b'gzip, deflate, br'),
             (b'content-length', b'0')],
 'http_version': '1.1',
 'method': 'GET',
 'path': '/-/asgi-scope',
 'query_string': b'',
 'raw_path': b'/-/asgi-scope',
 'root_path': '',
 'scheme': 'http',
 'server': ('169.254.8.130', 8080),
 'type': 'http'}

Note the 'scheme': 'http' but also the (b'x-forwarded-proto', b'https').

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use force_https_urls on when deploying with Cloud Run 780267857  
755159583 https://github.com/simonw/datasette/issues/1176#issuecomment-755159583 https://api.github.com/repos/simonw/datasette/issues/1176 MDEyOklzc3VlQ29tbWVudDc1NTE1OTU4Mw== simonw 9599 2021-01-06T08:28:20Z 2021-01-06T08:28:20Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Policy on documenting "public" datasette.utils functions 779691739  
755158310 https://github.com/simonw/datasette/issues/1178#issuecomment-755158310 https://api.github.com/repos/simonw/datasette/issues/1178 MDEyOklzc3VlQ29tbWVudDc1NTE1ODMxMA== simonw 9599 2021-01-06T08:25:31Z 2021-01-06T08:25:31Z OWNER

Moving this to the Datasette repo.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use force_https_urls on when deploying with Cloud Run 780267857  
755157732 https://github.com/simonw/datasette/issues/1178#issuecomment-755157732 https://api.github.com/repos/simonw/datasette/issues/1178 MDEyOklzc3VlQ29tbWVudDc1NTE1NzczMg== simonw 9599 2021-01-06T08:24:12Z 2021-01-06T08:24:12Z OWNER

https://latest-with-plugins.datasette.io/fixtures/sortable.json has the bug too - the next_url is http:// when it should be https://.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use force_https_urls on when deploying with Cloud Run 780267857  
755157281 https://github.com/simonw/datasette/issues/1178#issuecomment-755157281 https://api.github.com/repos/simonw/datasette/issues/1178 MDEyOklzc3VlQ29tbWVudDc1NTE1NzI4MQ== simonw 9599 2021-01-06T08:23:14Z 2021-01-06T08:23:14Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use force_https_urls on when deploying with Cloud Run 780267857  
755157066 https://github.com/simonw/datasette/issues/1178#issuecomment-755157066 https://api.github.com/repos/simonw/datasette/issues/1178 MDEyOklzc3VlQ29tbWVudDc1NTE1NzA2Ng== simonw 9599 2021-01-06T08:22:47Z 2021-01-06T08:22:47Z OWNER

Weird... https://github.com/simonw/datasette/blob/a882d679626438ba0d809944f06f239bcba8ee96/datasette/app.py#L609-L613

    def absolute_url(self, request, path):
        url = urllib.parse.urljoin(request.url, path)
        if url.startswith("http://") and self.setting("force_https_urls"):
            url = "https://" + url[len("http://") :]
        return url

That looks like it should work. Needs more digging.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use force_https_urls on when deploying with Cloud Run 780267857  
755156606 https://github.com/simonw/datasette/issues/1178#issuecomment-755156606 https://api.github.com/repos/simonw/datasette/issues/1178 MDEyOklzc3VlQ29tbWVudDc1NTE1NjYwNg== simonw 9599 2021-01-06T08:21:49Z 2021-01-06T08:21:49Z OWNER

https://github.com/simonw/datasette-export-notebook/blob/aec398eab4f34791d240d7bc47b6eec575b357be/datasette_export_notebook/__init__.py#L18-L23

Maybe this is a bug in datasette.absolute_url? Perhaps it doesn't take the scheme into account.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use force_https_urls on when deploying with Cloud Run 780267857  
755134771 https://github.com/simonw/datasette/issues/1101#issuecomment-755134771 https://api.github.com/repos/simonw/datasette/issues/1101 MDEyOklzc3VlQ29tbWVudDc1NTEzNDc3MQ== simonw 9599 2021-01-06T07:28:01Z 2021-01-06T07:28:01Z OWNER

With this structure it will become possible to stream non-newline-delimited JSON array-of-objects too - the stream_rows() method could output [ first, then each row followed by a comma, then ] after the very last row.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
register_output_renderer() should support streaming data 749283032  
755133937 https://github.com/simonw/datasette/issues/1101#issuecomment-755133937 https://api.github.com/repos/simonw/datasette/issues/1101 MDEyOklzc3VlQ29tbWVudDc1NTEzMzkzNw== simonw 9599 2021-01-06T07:25:48Z 2021-01-06T07:26:43Z OWNER

Idea: instead of returning a dictionary, register_output_renderer could return an object. The object could have the following properties:

  • .extension - the extension to use
  • .can_render(...) - says if it can render this
  • .can_stream(...) - says if streaming is supported
  • async .stream_rows(rows_iterator, send) - method that loops through all rows and uses send to send them to the response in the correct format

I can then deprecate the existing dict return type for 1.0.

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
register_output_renderer() should support streaming data 749283032  
755128038 https://github.com/simonw/datasette/issues/1101#issuecomment-755128038 https://api.github.com/repos/simonw/datasette/issues/1101 MDEyOklzc3VlQ29tbWVudDc1NTEyODAzOA== simonw 9599 2021-01-06T07:10:22Z 2021-01-06T07:10:22Z OWNER

Yet another use-case for this: I want to be able to stream newline-delimited JSON in order to better import into Pandas:

pandas.read_json("https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_nl=on", lines=True)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
register_output_renderer() should support streaming data 749283032  
754958998 https://github.com/simonw/datasette/issues/1171#issuecomment-754958998 https://api.github.com/repos/simonw/datasette/issues/1171 MDEyOklzc3VlQ29tbWVudDc1NDk1ODk5OA== simonw 9599 2021-01-05T23:16:33Z 2021-01-05T23:16:33Z OWNER

That's really useful, thanks @rcoup

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
GitHub Actions workflow to build and sign macOS binary executables 778450486  
754958610 https://github.com/simonw/datasette/issues/782#issuecomment-754958610 https://api.github.com/repos/simonw/datasette/issues/782 MDEyOklzc3VlQ29tbWVudDc1NDk1ODYxMA== simonw 9599 2021-01-05T23:15:24Z 2021-01-05T23:15:24Z OWNER

https://latest-with-plugins.datasette.io/fixtures/roadside_attraction_characteristics/1.json-preview returns a 500 error at the moment - a KeyError on 'filtered_table_rows_count'.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Redesign default JSON format in preparation for Datasette 1.0 627794879  
754957658 https://github.com/simonw/datasette/issues/576#issuecomment-754957658 https://api.github.com/repos/simonw/datasette/issues/576 MDEyOklzc3VlQ29tbWVudDc1NDk1NzY1OA== simonw 9599 2021-01-05T23:12:50Z 2021-01-05T23:12:50Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Documented internals API for use in plugins 497170355  
754957563 https://github.com/simonw/datasette/issues/576#issuecomment-754957563 https://api.github.com/repos/simonw/datasette/issues/576 MDEyOklzc3VlQ29tbWVudDc1NDk1NzU2Mw== simonw 9599 2021-01-05T23:12:37Z 2021-01-05T23:12:37Z OWNER

I'm happy with how this has evolved, so I'm closing the issue.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Documented internals API for use in plugins 497170355  
754957378 https://github.com/simonw/datasette/issues/1176#issuecomment-754957378 https://api.github.com/repos/simonw/datasette/issues/1176 MDEyOklzc3VlQ29tbWVudDc1NDk1NzM3OA== simonw 9599 2021-01-05T23:12:03Z 2021-01-05T23:12:03Z OWNER

This needs to be done for Datasette 1.0. At the very least I need to ensure it's clear that datasette.utils is not part of the public API unless explicitly marked as such.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Policy on documenting "public" datasette.utils functions 779691739  
754952146 https://github.com/simonw/datasette/issues/1176#issuecomment-754952146 https://api.github.com/repos/simonw/datasette/issues/1176 MDEyOklzc3VlQ29tbWVudDc1NDk1MjE0Ng== simonw 9599 2021-01-05T22:57:26Z 2021-01-05T22:57:26Z OWNER

Known public APIs might be worth adding type annotations to as well.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Policy on documenting "public" datasette.utils functions 779691739  
754952040 https://github.com/simonw/datasette/issues/1176#issuecomment-754952040 https://api.github.com/repos/simonw/datasette/issues/1176 MDEyOklzc3VlQ29tbWVudDc1NDk1MjA0MA== simonw 9599 2021-01-05T22:57:09Z 2021-01-05T22:57:09Z OWNER

It might be neater to move all of the non-public functions into a separate module - datasette.utils.internal perhaps.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Policy on documenting "public" datasette.utils functions 779691739  
754951786 https://github.com/simonw/datasette/issues/1176#issuecomment-754951786 https://api.github.com/repos/simonw/datasette/issues/1176 MDEyOklzc3VlQ29tbWVudDc1NDk1MTc4Ng== simonw 9599 2021-01-05T22:56:27Z 2021-01-05T22:56:43Z OWNER

Idea: introduce a @documented decorator which marks specific functions as part of the public, documented API. The unit tests can then confirm that anything with that decorator is both documented and tested.

@documented
def escape_css_string(s):
    return _css_re.sub(
        lambda m: "\\" + (f"{ord(m.group()):X}".zfill(6)),
        s.replace("\r\n", "\n"),
    )

Or maybe @public?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Policy on documenting "public" datasette.utils functions 779691739  
754911290 https://github.com/simonw/datasette/issues/1171#issuecomment-754911290 https://api.github.com/repos/simonw/datasette/issues/1171 MDEyOklzc3VlQ29tbWVudDc1NDkxMTI5MA== rcoup 59874 2021-01-05T21:31:15Z 2021-01-05T21:31:15Z NONE

We did this for Sno under macOS — it's a PyInstaller binary/setup which uses Packages for packaging.

FYI (if you ever get to it) for Windows you need to get a code signing certificate. And if you want automated CI, you'll want to get an "EV CodeSigning for HSM" certificate from GlobalSign, which then lets you put the certificate into Azure Key Vault. Which you can use with azuresigntool to sign your code & installer. (Non-EV certificates are a waste of time, the user still gets big warnings at install time).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
GitHub Actions workflow to build and sign macOS binary executables 778450486  
754729035 https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-754729035 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54 MDEyOklzc3VlQ29tbWVudDc1NDcyOTAzNQ== jacobian 21148 2021-01-05T16:03:29Z 2021-01-05T16:03:29Z NONE

I was able to fix this, at least enough to get my archive to import. Not sure if there's more work to be done here or not.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Archive import appears to be broken on recent exports 779088071  
754728696 https://github.com/dogsheep/twitter-to-sqlite/pull/55#issuecomment-754728696 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/55 MDEyOklzc3VlQ29tbWVudDc1NDcyODY5Ng== jacobian 21148 2021-01-05T16:02:55Z 2021-01-05T16:02:55Z NONE

This now works for me, though I'm entirely ensure if it's a just-my-export thing or a wider issue. Also, this doesn't contain any tests. So I'm not sure if there's more work to be done here, or if this is good enough.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fix archive imports 779211940  
754721153 https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-754721153 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54 MDEyOklzc3VlQ29tbWVudDc1NDcyMTE1Mw== jacobian 21148 2021-01-05T15:51:09Z 2021-01-05T15:51:09Z NONE

Correction: the failure is on lists-member.js (I was thrown by the block variable name, but that's just a coincidence)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Archive import appears to be broken on recent exports 779088071  
754696725 https://github.com/simonw/datasette/issues/1175#issuecomment-754696725 https://api.github.com/repos/simonw/datasette/issues/1175 MDEyOklzc3VlQ29tbWVudDc1NDY5NjcyNQ== simonw 9599 2021-01-05T15:12:30Z 2021-01-05T15:12:30Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use structlog for logging 779156520  
754619930 https://github.com/simonw/datasette/issues/1167#issuecomment-754619930 https://api.github.com/repos/simonw/datasette/issues/1167 MDEyOklzc3VlQ29tbWVudDc1NDYxOTkzMA== benpickles 3637 2021-01-05T12:57:57Z 2021-01-05T12:57:57Z CONTRIBUTOR

Not sure where exactly to put the actual docs (presumably somewhere in docs/contributing.rst) but I've made a slight change to make it easier to run locally (copying the approach in excalidraw): https://github.com/simonw/datasette/compare/main...benpickles:prettier-docs

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add Prettier to contributing documentation 777145954  
647922203 https://github.com/simonw/datasette/issues/859#issuecomment-647922203 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0NzkyMjIwMw== abdusco 3243482 2020-06-23T05:44:58Z 2021-01-05T08:22:43Z CONTRIBUTOR

I'm seeing the problem on database page. Index page and table page runs quite fast.

  • Tables have <10 columns (id, url, title, body_html, date, author, meta (for keeping unstructured json)). I've added index on date columns (using sqlite-utils) in addition to the index present on id columns.
  • All tables have FTS enabled on text and varchar columns (title, body_html etc) to speed up searching.
  • There are couple of tables related with foreign keys (think a thread in a forum and posts in that thread, related with thread_id)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841  
754463845 https://github.com/simonw/datasette/issues/1173#issuecomment-754463845 https://api.github.com/repos/simonw/datasette/issues/1173 MDEyOklzc3VlQ29tbWVudDc1NDQ2Mzg0NQ== simonw 9599 2021-01-05T07:41:43Z 2021-01-05T07:41:43Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
GitHub Actions workflow to build manylinux binary 778682317  
754296761 https://github.com/simonw/datasette/issues/1171#issuecomment-754296761 https://api.github.com/repos/simonw/datasette/issues/1171 MDEyOklzc3VlQ29tbWVudDc1NDI5Njc2MQ== simonw 9599 2021-01-04T23:55:44Z 2021-01-04T23:55:44Z OWNER

Bit uncomfortable that it looks like you need to include your Apple ID username and password in the CI configuration to do this. I'll use GitHub Secrets for this but I don't like it - I'll definitely setup a dedicated code signing account that's not my access-to-everything AppleID for this.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
GitHub Actions workflow to build and sign macOS binary executables 778450486  
754295380 https://github.com/simonw/datasette/issues/1171#issuecomment-754295380 https://api.github.com/repos/simonw/datasette/issues/1171 MDEyOklzc3VlQ29tbWVudDc1NDI5NTM4MA== simonw 9599 2021-01-04T23:54:32Z 2021-01-04T23:54:32Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
GitHub Actions workflow to build and sign macOS binary executables 778450486  
754287882 https://github.com/simonw/datasette/issues/1171#issuecomment-754287882 https://api.github.com/repos/simonw/datasette/issues/1171 MDEyOklzc3VlQ29tbWVudDc1NDI4Nzg4Mg== simonw 9599 2021-01-04T23:40:10Z 2021-01-04T23:42:32Z OWNER

This looks VERY useful: https://github.com/mitchellh/gon - " Sign, notarize, and package macOS CLI tools and applications written in any language. Available as both a CLI and a Go library."

And it installs like this:

brew install mitchellh/gon/gon
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
GitHub Actions workflow to build and sign macOS binary executables 778450486  
754286783 https://github.com/simonw/datasette/issues/1171#issuecomment-754286783 https://api.github.com/repos/simonw/datasette/issues/1171 MDEyOklzc3VlQ29tbWVudDc1NDI4Njc4Mw== simonw 9599 2021-01-04T23:38:18Z 2021-01-04T23:38:18Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
GitHub Actions workflow to build and sign macOS binary executables 778450486  
754286618 https://github.com/simonw/datasette/issues/1171#issuecomment-754286618 https://api.github.com/repos/simonw/datasette/issues/1171 MDEyOklzc3VlQ29tbWVudDc1NDI4NjYxOA== simonw 9599 2021-01-04T23:37:45Z 2021-01-04T23:37:45Z OWNER

https://github.com/actions/virtual-environments/issues/1820#issuecomment-719549887 looks useful - not sure if those notes are for iOS or macOS though.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
GitHub Actions workflow to build and sign macOS binary executables 778450486  
754285795 https://github.com/simonw/datasette/issues/93#issuecomment-754285795 https://api.github.com/repos/simonw/datasette/issues/93 MDEyOklzc3VlQ29tbWVudDc1NDI4NTc5NQ== simonw 9599 2021-01-04T23:35:13Z 2021-01-04T23:35:13Z OWNER

Next step is to automate this all!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Package as standalone binary 273944952  
754285588 https://github.com/simonw/datasette/issues/1152#issuecomment-754285588 https://api.github.com/repos/simonw/datasette/issues/1152 MDEyOklzc3VlQ29tbWVudDc1NDI4NTU4OA== simonw 9599 2021-01-04T23:34:30Z 2021-01-04T23:34:30Z OWNER

I think the way to do this is to have a new plugin hook that returns two SQL where clauses: one returning a list of resources that the user should be able to access (the allow-list) and one returning a list of resources they are explicitly forbidden from accessing (the deny-list). Either of these can be blank.

Datasette can then combine those into a full SQL query and use it to answer the question "show me a list of resources that the user is allowed to perform action X on". It can also answer the existing question, "is user X allowed to perform action Y on resource Z"?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Efficiently calculate list of databases/tables a user can view 770598024  
754233960 https://github.com/simonw/datasette/issues/93#issuecomment-754233960 https://api.github.com/repos/simonw/datasette/issues/93 MDEyOklzc3VlQ29tbWVudDc1NDIzMzk2MA== simonw 9599 2021-01-04T21:35:37Z 2021-01-04T21:35:37Z OWNER

I tested it by running a tmate session on the GitHub macOS machines, and it worked!

Mac-1609795972770:tmp runner$ wget 'https://github.com/simonw/datasette/releases/download/0.53/datasette-0.53-macos-binary.zip'
--2021-01-04 21:34:10--  https://github.com/simonw/datasette/releases/download/0.53/datasette-0.53-macos-binary.zip
Resolving github.com (github.com)... 140.82.114.4
Connecting to github.com (github.com)|140.82.114.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/107914493/74658700-4e90-11eb-8f3b-ee77e6dfad90?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210104T213414Z&X-Amz-Expires=300&X-Amz-Signature=6f3c54211077092553590b33a7c36cd052895c9d4619607ad1df094782f64acf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=107914493&response-content-disposition=attachment%3B%20filename%3Ddatasette-0.53-macos-binary.zip&response-content-type=application%2Foctet-stream [following]
--2021-01-04 21:34:14--  https://github-production-release-asset-2e65be.s3.amazonaws.com/107914493/74658700-4e90-11eb-8f3b-ee77e6dfad90?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210104T213414Z&X-Amz-Expires=300&X-Amz-Signature=6f3c54211077092553590b33a7c36cd052895c9d4619607ad1df094782f64acf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=107914493&response-content-disposition=attachment%3B%20filename%3Ddatasette-0.53-macos-binary.zip&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.217.43.164
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.217.43.164|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8297283 (7.9M) [application/octet-stream]
Saving to: ‘datasette-0.53-macos-binary.zip’

datasette-0.53-maco 100%[===================>]   7.91M  --.-KB/s    in 0.1s    

2021-01-04 21:34:14 (73.4 MB/s) - ‘datasette-0.53-macos-binary.zip’ saved [8297283/8297283]

Mac-1609795972770:tmp runner$ unzip datasette-0.53-macos-binary.zip 
Archive:  datasette-0.53-macos-binary.zip
   creating: datasette-0.53-macos-binary/
  inflating: datasette-0.53-macos-binary/datasette  
Mac-1609795972770:tmp runner$ datasette-0.53-macos-binary/datasette --help
Usage: datasette [OPTIONS] COMMAND [ARGS]...

  Datasette!

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  serve*     Serve up specified SQLite database files with a web UI
  inspect
  install    Install Python packages - e.g.
  package    Package specified SQLite files into a new datasette Docker...
  plugins    List currently available plugins
  publish    Publish specified SQLite database files to the internet along...
  uninstall  Uninstall Python packages (e.g.
Mac-1609795972770:tmp runner$ datasette-0.53-macos-binary/datasette --get /-/versions.json
{"python": {"version": "3.9.1", "full": "3.9.1 (default, Dec 10 2020, 10:36:35) \n[Clang 12.0.0 (clang-1200.0.32.27)]"}, "datasette": {"version": "0.53"}, "asgi": "3.0", "uvicorn": "0.13.3", "sqlite": {"version": "3.34.0", "fts_versions": ["FTS5", "FTS4", "FTS3"], "extensions": {"json1": null}, "compile_options": ["COMPILER=clang-12.0.0", "ENABLE_COLUMN_METADATA", "ENABLE_FTS3", "ENABLE_FTS3_PARENTHESIS", "ENABLE_FTS4", "ENABLE_FTS5", "ENABLE_GEOPOLY", "ENABLE_JSON1", "ENABLE_PREUPDATE_HOOK", "ENABLE_RTREE", "ENABLE_SESSION", "MAX_VARIABLE_NUMBER=250000", "THREADSAFE=1"]}}
Mac-1609795972770:tmp runner$ 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Package as standalone binary 273944952  
754229977 https://github.com/simonw/datasette/issues/93#issuecomment-754229977 https://api.github.com/repos/simonw/datasette/issues/93 MDEyOklzc3VlQ29tbWVudDc1NDIyOTk3Nw== simonw 9599 2021-01-04T21:28:01Z 2021-01-04T21:28:01Z OWNER

As an experiment, I put the macOS one in a zip file and attached it to the latest release:

mkdir datasette-0.53-macos-binary
cp dist/datasette datasette-0.53-macos-binary
zip -r datasette-0.53-macos-binary.zip datasette-0.53-macos-binary

It's available here: https://github.com/simonw/datasette/releases/tag/0.53 - download URL is https://github.com/simonw/datasette/releases/download/0.53/datasette-0.53-macos-binary.zip

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Package as standalone binary 273944952  
754227543 https://github.com/simonw/datasette/issues/93#issuecomment-754227543 https://api.github.com/repos/simonw/datasette/issues/93 MDEyOklzc3VlQ29tbWVudDc1NDIyNzU0Mw== simonw 9599 2021-01-04T21:23:13Z 2021-01-04T21:23:13Z OWNER
(pyinstaller-venv) root@dogsheep:/tmp/pyinstaller-venv# dist/datasette --get /-/databases.json
[{"name": ":memory:", "path": null, "size": 0, "is_mutable": true, "is_memory": true, "hash": null}]
(pyinstaller-venv) root@dogsheep:/tmp/pyinstaller-venv# ls -lah dist/datasette 
-rwxr-xr-x 1 root root 8.9M Jan  4 21:05 dist/datasette
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Package as standalone binary 273944952  
754219002 https://github.com/simonw/datasette/issues/93#issuecomment-754219002 https://api.github.com/repos/simonw/datasette/issues/93 MDEyOklzc3VlQ29tbWVudDc1NDIxOTAwMg== simonw 9599 2021-01-04T21:06:49Z 2021-01-04T21:22:27Z OWNER

Works on Linux/Ubuntu too, except I had to do export BASE= on a separate line. I also did this:

apt-get install python3 python3-venv
python3 -m venv pyinstaller-venv
source pyinstaller-venv/bin/activate
pip install wheel
pip install datasette pyinstaller

export DATASETTE_BASE=$(python -c 'import os; print(os.path.dirname(__import__("datasette").__file__))')

pyinstaller -F \
    --add-data "$DATASETTE_BASE/templates:datasette/templates" \
    --add-data "$DATASETTE_BASE/static:datasette/static" \
    --hidden-import datasette.publish \
    --hidden-import datasette.publish.heroku \
    --hidden-import datasette.publish.cloudrun \
    --hidden-import datasette.facets \
    --hidden-import datasette.sql_functions \
    --hidden-import datasette.actor_auth_cookie \
    --hidden-import datasette.default_permissions \
    --hidden-import datasette.default_magic_parameters \
    --hidden-import datasette.blob_renderer \
    --hidden-import datasette.default_menu_links \
    --hidden-import uvicorn \
    --hidden-import uvicorn.logging \
    --hidden-import uvicorn.loops \
    --hidden-import uvicorn.loops.auto \
    --hidden-import uvicorn.protocols \
    --hidden-import uvicorn.protocols.http \
    --hidden-import uvicorn.protocols.http.auto \
    --hidden-import uvicorn.protocols.websockets \
    --hidden-import uvicorn.protocols.websockets.auto \
    --hidden-import uvicorn.lifespan \
    --hidden-import uvicorn.lifespan.on \
    $(which datasette)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Package as standalone binary 273944952  
754218545 https://github.com/simonw/datasette/issues/93#issuecomment-754218545 https://api.github.com/repos/simonw/datasette/issues/93 MDEyOklzc3VlQ29tbWVudDc1NDIxODU0NQ== simonw 9599 2021-01-04T21:05:57Z 2021-01-04T21:05:57Z OWNER

That BASE= trick seems to work with zsh but not with bash.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Package as standalone binary 273944952  
754215392 https://github.com/simonw/datasette/issues/93#issuecomment-754215392 https://api.github.com/repos/simonw/datasette/issues/93 MDEyOklzc3VlQ29tbWVudDc1NDIxNTM5Mg== simonw 9599 2021-01-04T20:59:20Z 2021-01-04T21:03:14Z OWNER

Updated pyinstaller recipe - lots of hidden imports needed now:

pip install wheel
pip install datasette pyinstaller

BASE=$(python -c 'import os; print(os.path.dirname(__import__("datasette").__file__))') \
    pyinstaller -F \
        --add-data "$BASE/templates:datasette/templates" \
        --add-data "$BASE/static:datasette/static" \
        --hidden-import datasette.publish \
        --hidden-import datasette.publish.heroku \
        --hidden-import datasette.publish.cloudrun \
        --hidden-import datasette.facets \
        --hidden-import datasette.sql_functions \
        --hidden-import datasette.actor_auth_cookie \
        --hidden-import datasette.default_permissions \
        --hidden-import datasette.default_magic_parameters \
        --hidden-import datasette.blob_renderer \
        --hidden-import datasette.default_menu_links \
        --hidden-import uvicorn \
        --hidden-import uvicorn.logging \
        --hidden-import uvicorn.loops \
        --hidden-import uvicorn.loops.auto \
        --hidden-import uvicorn.protocols \
        --hidden-import uvicorn.protocols.http \
        --hidden-import uvicorn.protocols.http.auto \
        --hidden-import uvicorn.protocols.websockets \
        --hidden-import uvicorn.protocols.websockets.auto \
        --hidden-import uvicorn.lifespan \
        --hidden-import uvicorn.lifespan.on \
        $(which datasette)
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Package as standalone binary 273944952  
754215793 https://github.com/simonw/datasette/issues/93#issuecomment-754215793 https://api.github.com/repos/simonw/datasette/issues/93 MDEyOklzc3VlQ29tbWVudDc1NDIxNTc5Mw== simonw 9599 2021-01-04T21:00:14Z 2021-01-04T21:00:14Z OWNER
(pyinstaller-datasette) pyinstaller-datasette % file dist/datasette
dist/datasette: Mach-O 64-bit executable x86_64
(pyinstaller-datasette) pyinstaller-datasette % ls -lah dist/datasette
-rwxr-xr-x  1 simon  wheel   8.0M Jan  4 12:58 dist/datasette

I'm surprised it's only 8MB!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Package as standalone binary 273944952  
754210356 https://github.com/simonw/datasette/issues/983#issuecomment-754210356 https://api.github.com/repos/simonw/datasette/issues/983 MDEyOklzc3VlQ29tbWVudDc1NDIxMDM1Ng== carlmjohnson 222245 2021-01-04T20:49:05Z 2021-01-04T20:49:05Z NONE

For reasons I've written about elsewhere, I'm in favor of modules. It has several beneficial effects. One, old browsers just ignore it all together. Two, if you include the same plain script on the page more than once, it will be executed twice, but if you include the same module script on a page twice, it will only execute once. Three, you get a module local namespace, instead of having to use the global window namespace or a function private namespace.

OTOH, if you are going to use an old style script, the code from before isn't ideal, because you wipe out your registry if the script it included more than once. Also you may as well use object methods and splat arguments.

The event based architecture probably makes more sense though. Just make up some event names prefixed with datasette: and listen for them on the root. The only concern with that approach is it can sometimes be tricky to make sure your plugins are run after datasette has run. Maybe

function mycallback(){
  // whatever
}

if (window.datasette) {
  window.datasette.init(mycallback);
} else {
  document.addEventListener('datasette:init', mycallback);
}
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
JavaScript plugin hooks mechanism similar to pluggy 712260429  
754194996 https://github.com/simonw/datasette/issues/668#issuecomment-754194996 https://api.github.com/repos/simonw/datasette/issues/668 MDEyOklzc3VlQ29tbWVudDc1NDE5NDk5Ng== simonw 9599 2021-01-04T20:18:39Z 2021-01-04T20:18:39Z OWNER

I fixed this in #1115 - you can run --load-extension=spatialite now and it will look for the extension in common places.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Make it easier to load SpatiaLite 563347679  
754193501 https://github.com/simonw/datasette/issues/436#issuecomment-754193501 https://api.github.com/repos/simonw/datasette/issues/436 MDEyOklzc3VlQ29tbWVudDc1NDE5MzUwMQ== simonw 9599 2021-01-04T20:15:41Z 2021-01-04T20:15:41Z OWNER

Sadly publish.datasettes.com was broken by changes to Zeit, and I don't think I'll be bringing it back.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
400 Error when trying to register new user via https://publish.datasettes.com/ 435819321  
754192873 https://github.com/simonw/datasette/issues/371#issuecomment-754192873 https://api.github.com/repos/simonw/datasette/issues/371 MDEyOklzc3VlQ29tbWVudDc1NDE5Mjg3Mw== simonw 9599 2021-01-04T20:14:28Z 2021-01-04T20:14:28Z OWNER

Now that Digital Ocean has App Platform this is less necessary, especially since the documentation covers how to use App Platform here: https://docs.datasette.io/en/stable/deploying.html#deploying-using-buildpacks

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette publish digitalocean plugin 377156339  
754192267 https://github.com/simonw/datasette/issues/102#issuecomment-754192267 https://api.github.com/repos/simonw/datasette/issues/102 MDEyOklzc3VlQ29tbWVudDc1NDE5MjI2Nw== simonw 9599 2021-01-04T20:13:19Z 2021-01-04T20:13:19Z OWNER

I'm more likely to do Lambda than Elastic Beanstalk, especially now the size limit for Lambdas has been increased as part of their support for Docker.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette publish elasticbeanstalk 274264175  
754191699 https://github.com/simonw/datasette/issues/221#issuecomment-754191699 https://api.github.com/repos/simonw/datasette/issues/221 MDEyOklzc3VlQ29tbWVudDc1NDE5MTY5OQ== simonw 9599 2021-01-04T20:12:14Z 2021-01-04T20:12:14Z OWNER

I'm going to close this. Plugins can register their own CLI tools (see https://github.com/simonw/click-app) if they need to.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Allow plugins to add new cli sub commands  315142414  
754190952 https://github.com/simonw/datasette/issues/221#issuecomment-754190952 https://api.github.com/repos/simonw/datasette/issues/221 MDEyOklzc3VlQ29tbWVudDc1NDE5MDk1Mg== simonw 9599 2021-01-04T20:10:51Z 2021-01-04T20:10:51Z OWNER

Is this still a good idea? I don't have any pressing need for it at the moment.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Allow plugins to add new cli sub commands  315142414  
754190814 https://github.com/simonw/datasette/issues/221#issuecomment-754190814 https://api.github.com/repos/simonw/datasette/issues/221 MDEyOklzc3VlQ29tbWVudDc1NDE5MDgxNA== simonw 9599 2021-01-04T20:10:34Z 2021-01-04T20:10:34Z OWNER

For the csvs-to-sqlite case I'm going with datasette insert instead, see #1160.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Allow plugins to add new cli sub commands  315142414  
754188383 https://github.com/simonw/datasette/issues/18#issuecomment-754188383 https://api.github.com/repos/simonw/datasette/issues/18 MDEyOklzc3VlQ29tbWVudDc1NDE4ODM4Mw== simonw 9599 2021-01-04T20:05:48Z 2021-01-04T20:05:48Z OWNER

I'm not using Sanic any more, but this is still very feasible. If I ever do it I'll write a plugin.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
See if I can get a websockets interface working 267739593  
754188099 https://github.com/simonw/datasette/issues/103#issuecomment-754188099 https://api.github.com/repos/simonw/datasette/issues/103 MDEyOklzc3VlQ29tbWVudDc1NDE4ODA5OQ== simonw 9599 2021-01-04T20:05:14Z 2021-01-04T20:05:14Z OWNER

Wontfix, Cloud Run is already implemented and is a better fit for Datasette.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette publish appengine 274265878  
754187520 https://github.com/simonw/datasette/issues/913#issuecomment-754187520 https://api.github.com/repos/simonw/datasette/issues/913 MDEyOklzc3VlQ29tbWVudDc1NDE4NzUyMA== simonw 9599 2021-01-04T20:04:10Z 2021-01-04T20:04:10Z OWNER

That's pretty elegant: each plugin gets its own namespace and can register new settings.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Mechanism for passing additional options to `datasette my.db` that affect plugins 670209331  
754187326 https://github.com/simonw/datasette/issues/913#issuecomment-754187326 https://api.github.com/repos/simonw/datasette/issues/913 MDEyOklzc3VlQ29tbWVudDc1NDE4NzMyNg== simonw 9599 2021-01-04T20:03:50Z 2021-01-04T20:03:50Z OWNER

I renamed --config to --setting and changed it to work like this:

datasette --setting sql_time_limit_ms 1000

Note the lack of colons.

This actually makes colons cleaner to use for plugins - I could support this:

datasette --setting datasette-insert:unsafe 1
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Mechanism for passing additional options to `datasette my.db` that affect plugins 670209331  
754184287 https://github.com/simonw/datasette/issues/1111#issuecomment-754184287 https://api.github.com/repos/simonw/datasette/issues/1111 MDEyOklzc3VlQ29tbWVudDc1NDE4NDI4Nw== simonw 9599 2021-01-04T19:57:53Z 2021-01-04T19:57:53Z OWNER

Relevant new feature in sqlite-utils: the ability to use triggers to maintain fast counts. This optimization could help a lot here. https://github.com/simonw/sqlite-utils/issues/212

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Accessing a database's `.json` is slow for very large SQLite files 751195017  
754182058 https://github.com/simonw/datasette/issues/1164#issuecomment-754182058 https://api.github.com/repos/simonw/datasette/issues/1164 MDEyOklzc3VlQ29tbWVudDc1NDE4MjA1OA== simonw 9599 2021-01-04T19:53:31Z 2021-01-04T19:53:31Z OWNER

This will be helped by the new package.json added in #1170.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Mechanism for minifying JavaScript that ships with Datasette 776634318  
754181646 https://github.com/simonw/datasette/pull/1170#issuecomment-754181646 https://api.github.com/repos/simonw/datasette/issues/1170 MDEyOklzc3VlQ29tbWVudDc1NDE4MTY0Ng== simonw 9599 2021-01-04T19:52:40Z 2021-01-04T19:52:40Z OWNER

Thank you very much!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Install Prettier via package.json 778126516  
754181647 https://github.com/simonw/datasette/issues/983#issuecomment-754181647 https://api.github.com/repos/simonw/datasette/issues/983 MDEyOklzc3VlQ29tbWVudDc1NDE4MTY0Nw== jussiarpalahti 11941245 2021-01-04T19:52:40Z 2021-01-04T19:52:40Z NONE

I was thinking JavaScript plugins going with server side template extensions custom HTML. Attach my own widgets on there and listen for Datasette events to refresh when user interacts with main UI. Like a map view or table that updates according to selected column. There's certainly other ways to look at this. Perhaps you could list possible hooks or high level design doc on what would be possible with the plugin system?

Re: modules. I would like to see modules supported at least in development. The developer experience is so much better than what JavaScript coding has been in the past. With large parts of NPM at your disposal I’d imagine even less experienced coder can whisk a custom plugin in no time. Proper production build system (like one you get with Pika or Parcel) could package everything up into bundles that older browsers can understand. Though that does come with performance and size penalties alongside the added complexity.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
JavaScript plugin hooks mechanism similar to pluggy 712260429  

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);