# issue_comments

html_url: https://github.com/simonw/datasette/issues/1719#issuecomment-1108907238
issue_url: https://api.github.com/repos/simonw/datasette/issues/1719
node_id: IC_kwDOBm6k_c5CGJTm
created_at: 2022-04-25T18:34:21Z
updated_at: 2022-04-25T18:34:21Z
author_association: OWNER
body:
  Well this refactor turned out to be pretty quick and really does greatly simplify both the `RowView` and `TableView` classes. Very happy with this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1720#issuecomment-1109158903
issue_url: https://api.github.com/repos/simonw/datasette/issues/1720
node_id: IC_kwDOBm6k_c5CHGv3
created_at: 2022-04-26T00:11:42Z
updated_at: 2022-04-26T00:11:42Z
author_association: OWNER
body:
  Places this plugin hook (or hooks?) should be able to affect:

  - JSON for a table/view
  - JSON for a row
  - JSON for a canned query
  - JSON for a custom arbitrary query

  I'm going to combine those last two, which means there are three places. But maybe I can combine the table one and the row one as well?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1720#issuecomment-1109159307
issue_url: https://api.github.com/repos/simonw/datasette/issues/1720
node_id: IC_kwDOBm6k_c5CHG2L
created_at: 2022-04-26T00:12:28Z
updated_at: 2022-04-26T00:12:28Z
author_association: OWNER
body:
  I'm going to keep table and row separate. So I think I need to add three new plugin hooks:

  - `table_extras()`
  - `row_extras()`
  - `query_extras()`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1720#issuecomment-1109160226
issue_url: https://api.github.com/repos/simonw/datasette/issues/1720
node_id: IC_kwDOBm6k_c5CHHEi
created_at: 2022-04-26T00:14:11Z
updated_at: 2022-04-26T00:14:11Z
author_association: OWNER
body:
  There are four existing plugin hooks that include the word "extra" but use it to mean something else - to mean additional CSS/JS/variables to be injected into the page:

  - `def extra_css_urls(...)`
  - `def extra_js_urls(...)`
  - `def extra_body_script(...)`
  - `def extra_template_vars(...)`

  I think `extra_*` and `*_extras` are different enough that they won't be confused with each other.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1720#issuecomment-1109162123
issue_url: https://api.github.com/repos/simonw/datasette/issues/1720
node_id: IC_kwDOBm6k_c5CHHiL
created_at: 2022-04-26T00:16:42Z
updated_at: 2022-04-26T00:16:51Z
author_association: OWNER
body:
  Actually I'm going to imitate the existing `register_*` hooks:

  - `def register_output_renderer(datasette)`
  - `def register_facet_classes()`
  - `def register_routes(datasette)`
  - `def register_commands(cli)`
  - `def register_magic_parameters(datasette)`

  So I'm going to call the new hooks:

  - `register_table_extras(datasette)`
  - `register_row_extras(datasette)`
  - `register_query_extras(datasette)`

  They'll return a list of `async def` functions. The names of those functions will become the names of the extras.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1720#issuecomment-1109164803
issue_url: https://api.github.com/repos/simonw/datasette/issues/1720
node_id: IC_kwDOBm6k_c5CHIMD
created_at: 2022-04-26T00:21:40Z
updated_at: 2022-04-26T00:21:40Z
author_association: OWNER
body:
  What would the existing https://latest.datasette.io/fixtures/simple_primary_key/1.json?_extras=foreign_key_tables feature look like if it was re-imagined as a `register_row_extras()` plugin?

  Rough sketch, copying most of the code from https://github.com/simonw/datasette/blob/579f59dcec43a91dd7d404e00b87a00afd8515f2/datasette/views/row.py#L98

  ```python
  from datasette import hookimpl

  @hookimpl
  def register_row_extras(datasette):
      return [foreign_key_tables]

  async def foreign_key_tables(datasette, database, table, pk_values):
      if len(pk_values) != 1:
          return []
      db = datasette.get_database(database)
      all_foreign_keys = await db.get_all_foreign_keys()
      foreign_keys = all_foreign_keys[table]["incoming"]
      if len(foreign_keys) == 0:
          return []

      sql = "select " + ", ".join(
          [
              "(select count(*) from {table} where {column}=:id)".format(
                  table=escape_sqlite(fk["other_table"]),
                  column=escape_sqlite(fk["other_column"]),
              )
              for fk in foreign_keys
          ]
      )
      try:
          rows = list(await db.execute(sql, {"id": pk_values[0]}))
      except QueryInterrupted:
          # Almost certainly hit the timeout
          return []

      foreign_table_counts = dict(
          zip(
              [(fk["other_table"], fk["other_column"]) for fk in foreign_keys],
              list(rows[0]),
          )
      )
      foreign_key_tables = []
      for fk in foreign_keys:
          count = (
              foreign_table_counts.get((fk["other_table"], fk["other_column"])) or 0
          )
          key = fk["other_column"]
          if key.startswith("_"):
              key += "__exact"
          link = "{}?{}={}".format(
              self.ds.urls.table(database, fk["other_table"]),
              key,
              ",".join(pk_values),
          )
          foreign_key_tables.append({**fk, **{"count": count, "link": link}})
      return foreign_key_tables
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1720#issuecomment-1109165411
issue_url: https://api.github.com/repos/simonw/datasette/issues/1720
node_id: IC_kwDOBm6k_c5CHIVj
created_at: 2022-04-26T00:22:42Z
updated_at: 2022-04-26T00:22:42Z
author_association: OWNER
body:
  Passing `pk_values` to the plugin hook feels odd. I think I'd pass a `row` object instead and let the code look up the primary key values on that row (by introspecting the primary keys for the table).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1720#issuecomment-1109171871
issue_url: https://api.github.com/repos/simonw/datasette/issues/1720
node_id: IC_kwDOBm6k_c5CHJ6f
created_at: 2022-04-26T00:34:48Z
updated_at: 2022-04-26T00:34:48Z
author_association: OWNER
body:
  Let's try sketching out a `register_table_extras` plugin for something new.

  The first idea I came up with suggests adding new fields to the individual row records that come back - my mental model for extras so far has been that they add new keys to the root object.

  So if a table result looked like this:

  ```json
  {
    "rows": [
      {"id": 1, "name": "Cleo"},
      {"id": 2, "name": "Suna"}
    ],
    "next_url": null
  }
  ```
  I was initially thinking that `?_extra=facets` would add a `"facets": {...}` key to that root object.

  Here's a plugin idea I came up with that would probably justify adding to the individual row objects instead:

  - `?_extra=check404s` - does an async `HEAD` request against every column value that looks like a URL and checks if it returns a 404

  This could also work by adding a `"check404s": {"url-here": 200}` key to the root object though.

  I think I need some better plugin concepts before committing to this new hook. There's overlap between this and how I want the enrichments mechanism ([see here](https://simonwillison.net/2021/Jan/17/weeknotes-still-pretty-distracted/)) to work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1720#issuecomment-1109174715
issue_url: https://api.github.com/repos/simonw/datasette/issues/1720
node_id: IC_kwDOBm6k_c5CHKm7
created_at: 2022-04-26T00:40:13Z
updated_at: 2022-04-26T00:43:33Z
author_association: OWNER
body:
  Some of the things I'd like to use `?_extra=` for, that may or not make sense as plugins:

  - Performance breakdown information, maybe including explain output for a query/table
  - Information about the tables that were consulted in a query - imagine pulling in additional table metadata
  - Statistical aggregates against the full set of results. This may well be a Datasette core feature at some point in the future, but being able to provide it early as a plugin would be really cool.
  - For tables, what are the other tables they can join against?
  - Suggested facets
  - Facet results themselves
  - New custom facets I haven't thought of - though the `register_facet_classes` hook covers that already
  - Table schema
  - Table metadata
  - Analytics - how many times has this table been queried? Would be a plugin thing
  - For geospatial data, how about a GeoJSON polygon that represents the bounding box for all returned results? Effectively this is an extra aggregation.

  Looking at https://github-to-sqlite.dogsheep.net/github/commits.json?_labels=on&_shape=objects for inspiration.

  I think there's a separate potential mechanism in the future that lets you add custom columns to a table. This would affect `.csv` and the HTML presentation too, which makes it a different concept from the `?_extra=` hook that affects the JSON export (and the context that is fed to the HTML templates).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/428#issuecomment-1109190401
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/428
node_id: IC_kwDOCGYnMM5CHOcB
created_at: 2022-04-26T01:05:29Z
updated_at: 2022-04-26T01:05:29Z
author_association: OWNER
body:
  Django makes extensive use of savepoints for nested transactions: https://docs.djangoproject.com/en/4.0/topics/db/transactions/#savepoints

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1720#issuecomment-1109200335
issue_url: https://api.github.com/repos/simonw/datasette/issues/1720
node_id: IC_kwDOBm6k_c5CHQ3P
created_at: 2022-04-26T01:24:47Z
updated_at: 2022-04-26T01:24:47Z
author_association: OWNER
body:
  Sketching out a `?_extra=statistics` table plugin:

  ```python
  from datasette import hookimpl

  @hookimpl
  def register_table_extras(datasette):
      return [statistics]

  async def statistics(datasette, query, columns, sql):
      # ... need to figure out which columns are integer/floats
      # then build and execute a SQL query that calculates sum/avg/etc for each column
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1720#issuecomment-1109200774
issue_url: https://api.github.com/repos/simonw/datasette/issues/1720
node_id: IC_kwDOBm6k_c5CHQ-G
created_at: 2022-04-26T01:25:43Z
updated_at: 2022-04-26T01:26:15Z
author_association: OWNER
body:
  Had a thought: if a custom HTML template is going to make use of stuff generated using these extras, it will need a way to tell Datasette to execute those extras even in the absence of the `?_extra=...` URL parameters.

  Is that necessary? Or should those kinds of plugins use the existing `extra_template_vars` hook instead?

  Or maybe the `extra_template_vars` hook gets redesigned so it can depend on other `extras` in some way?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1720#issuecomment-1109305184
issue_url: https://api.github.com/repos/simonw/datasette/issues/1720
node_id: IC_kwDOBm6k_c5CHqdg
created_at: 2022-04-26T04:03:35Z
updated_at: 2022-04-26T04:03:35Z
author_association: OWNER
body:
  I bet there's all kinds of interesting potential extras that could be calculated by loading the results of the query into a Pandas DataFrame.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1720#issuecomment-1109306070
issue_url: https://api.github.com/repos/simonw/datasette/issues/1720
node_id: IC_kwDOBm6k_c5CHqrW
created_at: 2022-04-26T04:05:20Z
updated_at: 2022-04-26T04:05:20Z
author_association: OWNER
body:
  The proposed plugin for annotations - allowing users to attach comments to database tables, columns and rows - would be a great application for all three of those `?_extra=` plugin hooks.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1720#issuecomment-1109309683
issue_url: https://api.github.com/repos/simonw/datasette/issues/1720
node_id: IC_kwDOBm6k_c5CHrjz
created_at: 2022-04-26T04:12:39Z
updated_at: 2022-04-26T04:12:39Z
author_association: OWNER
body:
  I think the rough shape of the three plugin hooks is right. The detailed decisions that are needed concern what the parameters should be, which I think will mainly happen as part of:

  - #1715

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1720#issuecomment-1110212021
issue_url: https://api.github.com/repos/simonw/datasette/issues/1720
node_id: IC_kwDOBm6k_c5CLH21
created_at: 2022-04-26T20:20:27Z
updated_at: 2022-04-26T20:20:27Z
author_association: OWNER
body:
  Closing this because I have a good enough idea of the design for now - the details of the parameters can be figured out when I implement this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1715#issuecomment-1110219185
issue_url: https://api.github.com/repos/simonw/datasette/issues/1715
node_id: IC_kwDOBm6k_c5CLJmx
created_at: 2022-04-26T20:28:40Z
updated_at: 2022-04-26T20:56:48Z
author_association: OWNER
body:
  The refactor I did in #1719 pretty much clashes with all of the changes in https://github.com/simonw/datasette/commit/5053f1ea83194ecb0a5693ad5dada5b25bf0f7e6 so I'll probably need to start my `api-extras` branch again from scratch.

  Using a new `tableview-asyncinject` branch.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1715#issuecomment-1110229319
issue_url: https://api.github.com/repos/simonw/datasette/issues/1715
node_id: IC_kwDOBm6k_c5CLMFH
created_at: 2022-04-26T20:41:32Z
updated_at: 2022-04-26T20:44:38Z
author_association: OWNER
body:
  This time I'm not going to bother with the `filter_args` thing - I'm going to just try to use `asyncinject` to execute some big high level things in parallel - facets, suggested facets, counts, the query - and then combine it with the `extras` mechanism I'm trying to introduce too.

  Most importantly: I want that `extra_template()` function that adds more template context for the HTML to be executed as part of an `asyncinject` flow!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1715#issuecomment-1110238896
issue_url: https://api.github.com/repos/simonw/datasette/issues/1715
node_id: IC_kwDOBm6k_c5CLOaw
created_at: 2022-04-26T20:53:59Z
updated_at: 2022-04-26T20:53:59Z
author_association: OWNER
body:
  I'm going to rename `database` to `database_name` and `table` to `table_name` to avoid confusion with the `Database` object as opposed to the string name for the database.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1715#issuecomment-1110246593
issue_url: https://api.github.com/repos/simonw/datasette/issues/1715
node_id: IC_kwDOBm6k_c5CLQTB
created_at: 2022-04-26T21:03:56Z
updated_at: 2022-04-26T21:03:56Z
author_association: OWNER
body:
  Well this is fun... I applied this change:

  ```diff
  diff --git a/datasette/views/table.py b/datasette/views/table.py
  index d66adb8..85f9e44 100644
  --- a/datasette/views/table.py
  +++ b/datasette/views/table.py
  @@ -1,3 +1,4 @@
  +import asyncio
   import itertools
   import json
   
  @@ -5,6 +6,7 @@ import markupsafe
   
   from datasette.plugins import pm
   from datasette.database import QueryInterrupted
  +from datasette import tracer
   from datasette.utils import (
       await_me_maybe,
       CustomRow,
  @@ -174,8 +176,11 @@ class TableView(DataView):
                   write=bool(canned_query.get("write")),
               )
   
  -        is_view = bool(await db.get_view_definition(table_name))
  -        table_exists = bool(await db.table_exists(table_name))
  +        with tracer.trace_child_tasks():
  +            is_view, table_exists = map(bool, await asyncio.gather(
  +                db.get_view_definition(table_name),
  +                db.table_exists(table_name)
  +            ))
   
           # If table or view not found, return 404
           if not is_view and not table_exists:
  ```
  And now using https://datasette.io/plugins/datasette-pretty-traces I get this:

  ![CleanShot 2022-04-26 at 14 03 33@2x](https://user-images.githubusercontent.com/9599/165392009-84c4399d-3e94-46d4-ba7b-a64a116cac5c.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1715#issuecomment-1110265087
issue_url: https://api.github.com/repos/simonw/datasette/issues/1715
node_id: IC_kwDOBm6k_c5CLUz_
created_at: 2022-04-26T21:26:17Z
updated_at: 2022-04-26T21:26:17Z
author_association: OWNER
body:
  Running facets and facet suggestions in parallel using `asyncio.gather()` turns out to be a lot less hassle than I had thought - maybe I don't need `asyncinject` for this at all?

  ```diff
           if not nofacet:
  -            for facet in facet_instances:
  -                (
  -                    instance_facet_results,
  -                    instance_facets_timed_out,
  -                ) = await facet.facet_results()
  +            # Run them in parallel
  +            facet_awaitables = [facet.facet_results() for facet in facet_instances]
  +            facet_awaitable_results = await asyncio.gather(*facet_awaitables)
  +            for (
  +                instance_facet_results,
  +                instance_facets_timed_out,
  +            ) in facet_awaitable_results:
                   for facet_info in instance_facet_results:
                       base_key = facet_info["name"]
                       key = base_key
  @@ -522,8 +540,10 @@ class TableView(DataView):
               and not nofacet
               and not nosuggest
           ):
  -            for facet in facet_instances:
  -                suggested_facets.extend(await facet.suggest())
  +            # Run them in parallel
  +            facet_suggest_awaitables = [facet.suggest() for facet in facet_instances]
  +            for suggest_result in await asyncio.gather(*facet_suggest_awaitables):
  +                suggested_facets.extend(suggest_result)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1723#issuecomment-1110278182
issue_url: https://api.github.com/repos/simonw/datasette/issues/1723
node_id: IC_kwDOBm6k_c5CLYAm
created_at: 2022-04-26T21:43:34Z
updated_at: 2022-04-26T21:43:34Z
author_association: OWNER
body:
  Here's the diff I'm using:
  ```diff
  diff --git a/datasette/views/table.py b/datasette/views/table.py
  index d66adb8..f15ef1e 100644
  --- a/datasette/views/table.py
  +++ b/datasette/views/table.py
  @@ -1,3 +1,4 @@
  +import asyncio
   import itertools
   import json
   
  @@ -5,6 +6,7 @@ import markupsafe
   
   from datasette.plugins import pm
   from datasette.database import QueryInterrupted
  +from datasette import tracer
   from datasette.utils import (
       await_me_maybe,
       CustomRow,
  @@ -150,6 +152,16 @@ class TableView(DataView):
           default_labels=False,
           _next=None,
           _size=None,
  +    ):
  +        with tracer.trace_child_tasks():
  +            return await self._data_traced(request, default_labels, _next, _size)
  +
  +    async def _data_traced(
  +        self,
  +        request,
  +        default_labels=False,
  +        _next=None,
  +        _size=None,
       ):
           database_route = tilde_decode(request.url_vars["database"])
           table_name = tilde_decode(request.url_vars["table"])
  @@ -159,6 +171,20 @@ class TableView(DataView):
               raise NotFound("Database not found: {}".format(database_route))
           database_name = db.name
   
  +        # For performance profiling purposes, ?_parallel=1 turns on asyncio.gather
  +        async def _gather_parallel(*args):
  +            return await asyncio.gather(*args)
  +
  +        async def _gather_sequential(*args):
  +            results = []
  +            for fn in args:
  +                results.append(await fn)
  +            return results
  +
  +        gather = (
  +            _gather_parallel if request.args.get("_parallel") else _gather_sequential
  +        )
  +
           # If this is a canned query, not a table, then dispatch to QueryView instead
           canned_query = await self.ds.get_canned_query(
               database_name, table_name, request.actor
  @@ -174,8 +200,12 @@ class TableView(DataView):
                   write=bool(canned_query.get("write")),
               )
   
  -        is_view = bool(await db.get_view_definition(table_name))
  -        table_exists = bool(await db.table_exists(table_name))
  +        is_view, table_exists = map(
  +            bool,
  +            await gather(
  +                db.get_view_definition(table_name), db.table_exists(table_name)
  +            ),
  +        )
   
           # If table or view not found, return 404
           if not is_view and not table_exists:
  @@ -497,33 +527,44 @@ class TableView(DataView):
                   )
               )
   
  -        if not nofacet:
  -            for facet in facet_instances:
  -                (
  +        async def execute_facets():
  +            if not nofacet:
  +                # Run them in parallel
  +                facet_awaitables = [facet.facet_results() for facet in facet_instances]
  +                facet_awaitable_results = await gather(*facet_awaitables)
  +                for (
                       instance_facet_results,
                       instance_facets_timed_out,
  -                ) = await facet.facet_results()
  -                for facet_info in instance_facet_results:
  -                    base_key = facet_info["name"]
  -                    key = base_key
  -                    i = 1
  -                    while key in facet_results:
  -                        i += 1
  -                        key = f"{base_key}_{i}"
  -                    facet_results[key] = facet_info
  -                facets_timed_out.extend(instance_facets_timed_out)
  -
  -        # Calculate suggested facets
  +                ) in facet_awaitable_results:
  +                    for facet_info in instance_facet_results:
  +                        base_key = facet_info["name"]
  +                        key = base_key
  +                        i = 1
  +                        while key in facet_results:
  +                            i += 1
  +                            key = f"{base_key}_{i}"
  +                        facet_results[key] = facet_info
  +                    facets_timed_out.extend(instance_facets_timed_out)
  +
           suggested_facets = []
  -        if (
  -            self.ds.setting("suggest_facets")
  -            and self.ds.setting("allow_facet")
  -            and not _next
  -            and not nofacet
  -            and not nosuggest
  -        ):
  -            for facet in facet_instances:
  -                suggested_facets.extend(await facet.suggest())
  +
  +        async def execute_suggested_facets():
  +            # Calculate suggested facets
  +            if (
  +                self.ds.setting("suggest_facets")
  +                and self.ds.setting("allow_facet")
  +                and not _next
  +                and not nofacet
  +                and not nosuggest
  +            ):
  +                # Run them in parallel
  +                facet_suggest_awaitables = [
  +                    facet.suggest() for facet in facet_instances
  +                ]
  +                for suggest_result in await gather(*facet_suggest_awaitables):
  +                    suggested_facets.extend(suggest_result)
  +
  +        await gather(execute_facets(), execute_suggested_facets())
   
           # Figure out columns and rows for the query
           columns = [r[0] for r in results.description]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1723#issuecomment-1110278577
issue_url: https://api.github.com/repos/simonw/datasette/issues/1723
node_id: IC_kwDOBm6k_c5CLYGx
created_at: 2022-04-26T21:44:04Z
updated_at: 2022-04-26T21:44:04Z
author_association: OWNER
body:
  And some simple benchmarks with `ab` - using the `?_parallel=1` hack to try it with and without a parallel `asyncio.gather()`:

  ```
  ~ % ab -n 100 'http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2'                    
  This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
  Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
  Licensed to The Apache Software Foundation, http://www.apache.org/

  Benchmarking 127.0.0.1 (be patient).....done


  Server Software:        uvicorn
  Server Hostname:        127.0.0.1
  Server Port:            8001

  Document Path:          /global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2
  Document Length:        314187 bytes

  Concurrency Level:      1
  Time taken for tests:   68.279 seconds
  Complete requests:      100
  Failed requests:        13
     (Connect: 0, Receive: 0, Length: 13, Exceptions: 0)
  Total transferred:      31454937 bytes
  HTML transferred:       31418437 bytes
  Requests per second:    1.46 [#/sec] (mean)
  Time per request:       682.787 [ms] (mean)
  Time per request:       682.787 [ms] (mean, across all concurrent requests)
  Transfer rate:          449.89 [Kbytes/sec] received

  Connection Times (ms)
                min  mean[+/-sd] median   max
  Connect:        0    0   0.0      0       0
  Processing:   621  683  68.0    658     993
  Waiting:      620  682  68.0    657     992
  Total:        621  683  68.0    658     993

  Percentage of the requests served within a certain time (ms)
    50%    658
    66%    678
    75%    687
    80%    711
    90%    763
    95%    879
    98%    926
    99%    993
   100%    993 (longest request)


  ----

  In parallel:

  ~ % ab -n 100 'http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel=1'
  This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
  Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
  Licensed to The Apache Software Foundation, http://www.apache.org/

  Benchmarking 127.0.0.1 (be patient).....done


  Server Software:        uvicorn
  Server Hostname:        127.0.0.1
  Server Port:            8001

  Document Path:          /global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel=1
  Document Length:        315703 bytes

  Concurrency Level:      1
  Time taken for tests:   34.763 seconds
  Complete requests:      100
  Failed requests:        11
     (Connect: 0, Receive: 0, Length: 11, Exceptions: 0)
  Total transferred:      31607988 bytes
  HTML transferred:       31570288 bytes
  Requests per second:    2.88 [#/sec] (mean)
  Time per request:       347.632 [ms] (mean)
  Time per request:       347.632 [ms] (mean, across all concurrent requests)
  Transfer rate:          887.93 [Kbytes/sec] received

  Connection Times (ms)
                min  mean[+/-sd] median   max
  Connect:        0    0   0.0      0       0
  Processing:   311  347  28.0    338     450
  Waiting:      311  347  28.0    338     450
  Total:        312  348  28.0    338     451

  Percentage of the requests served within a certain time (ms)
    50%    338
    66%    348
    75%    361
    80%    367
    90%    396
    95%    408
    98%    436
    99%    451
   100%    451 (longest request)

  ----

  With concurrency 10, not parallel:

  ~ % ab -c 10 -n 100 'http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel=' 
  This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
  Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
  Licensed to The Apache Software Foundation, http://www.apache.org/

  Benchmarking 127.0.0.1 (be patient).....done


  Server Software:        uvicorn
  Server Hostname:        127.0.0.1
  Server Port:            8001

  Document Path:          /global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel=
  Document Length:        314346 bytes

  Concurrency Level:      10
  Time taken for tests:   38.408 seconds
  Complete requests:      100
  Failed requests:        93
     (Connect: 0, Receive: 0, Length: 93, Exceptions: 0)
  Total transferred:      31471333 bytes
  HTML transferred:       31433733 bytes
  Requests per second:    2.60 [#/sec] (mean)
  Time per request:       3840.829 [ms] (mean)
  Time per request:       384.083 [ms] (mean, across all concurrent requests)
  Transfer rate:          800.18 [Kbytes/sec] received

  Connection Times (ms)
                min  mean[+/-sd] median   max
  Connect:        0    0   0.1      0       1
  Processing:   685 3719 354.0   3774    4096
  Waiting:      684 3707 353.7   3750    4095
  Total:        685 3719 354.0   3774    4096

  Percentage of the requests served within a certain time (ms)
    50%   3774
    66%   3832
    75%   3855
    80%   3878
    90%   3944
    95%   4006
    98%   4057
    99%   4096
   100%   4096 (longest request)


  ----

  Concurrency 10 parallel:

  ~ % ab -c 10 -n 100 'http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel=1'
  This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
  Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
  Licensed to The Apache Software Foundation, http://www.apache.org/

  Benchmarking 127.0.0.1 (be patient).....done


  Server Software:        uvicorn
  Server Hostname:        127.0.0.1
  Server Port:            8001

  Document Path:          /global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel=1
  Document Length:        315703 bytes

  Concurrency Level:      10
  Time taken for tests:   36.762 seconds
  Complete requests:      100
  Failed requests:        89
     (Connect: 0, Receive: 0, Length: 89, Exceptions: 0)
  Total transferred:      31606516 bytes
  HTML transferred:       31568816 bytes
  Requests per second:    2.72 [#/sec] (mean)
  Time per request:       3676.182 [ms] (mean)
  Time per request:       367.618 [ms] (mean, across all concurrent requests)
  Transfer rate:          839.61 [Kbytes/sec] received

  Connection Times (ms)
                min  mean[+/-sd] median   max
  Connect:        0    0   0.1      0       0
  Processing:   381 3602 419.6   3609    4458
  Waiting:      381 3586 418.7   3607    4457
  Total:        381 3603 419.6   3609    4458

  Percentage of the requests served within a certain time (ms)
    50%   3609
    66%   3741
    75%   3791
    80%   3821
    90%   3972
    95%   4074
    98%   4386
    99%   4458
   100%   4458 (longest request)


  Trying -c 3 instead. Non parallel:

  ~ % ab -c 3 -n 100 'http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel='
  This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
  Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
  Licensed to The Apache Software Foundation, http://www.apache.org/

  Benchmarking 127.0.0.1 (be patient).....done


  Server Software:        uvicorn
  Server Hostname:        127.0.0.1
  Server Port:            8001

  Document Path:          /global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel=
  Document Length:        314346 bytes

  Concurrency Level:      3
  Time taken for tests:   39.365 seconds
  Complete requests:      100
  Failed requests:        83
     (Connect: 0, Receive: 0, Length: 83, Exceptions: 0)
  Total transferred:      31470808 bytes
  HTML transferred:       31433208 bytes
  Requests per second:    2.54 [#/sec] (mean)
  Time per request:       1180.955 [ms] (mean)
  Time per request:       393.652 [ms] (mean, across all concurrent requests)
  Transfer rate:          780.72 [Kbytes/sec] received

  Connection Times (ms)
                min  mean[+/-sd] median   max
  Connect:        0    0   0.0      0       0
  Processing:   731 1153 126.2   1189    1359
  Waiting:      730 1151 125.9   1188    1358
  Total:        731 1153 126.2   1189    1359

  Percentage of the requests served within a certain time (ms)
    50%   1189
    66%   1221
    75%   1234
    80%   1247
    90%   1296
    95%   1309
    98%   1343
    99%   1359
   100%   1359 (longest request)

  ----

  Parallel:

  ~ % ab -c 3 -n 100 'http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel=1'
  This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
  Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
  Licensed to The Apache Software Foundation, http://www.apache.org/

  Benchmarking 127.0.0.1 (be patient).....done


  Server Software:        uvicorn
  Server Hostname:        127.0.0.1
  Server Port:            8001

  Document Path:          /global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel=1
  Document Length:        315703 bytes

  Concurrency Level:      3
  Time taken for tests:   34.530 seconds
  Complete requests:      100
  Failed requests:        18
     (Connect: 0, Receive: 0, Length: 18, Exceptions: 0)
  Total transferred:      31606179 bytes
  HTML transferred:       31568479 bytes
  Requests per second:    2.90 [#/sec] (mean)
  Time per request:       1035.902 [ms] (mean)
  Time per request:       345.301 [ms] (mean, across all concurrent requests)
  Transfer rate:          893.87 [Kbytes/sec] received

  Connection Times (ms)
                min  mean[+/-sd] median   max
  Connect:        0    0   0.0      0       0
  Processing:   412 1020 104.4   1018    1280
  Waiting:      411 1018 104.1   1014    1275
  Total:        412 1021 104.4   1018    1280

  Percentage of the requests served within a certain time (ms)
    50%   1018
    66%   1041
    75%   1061
    80%   1079
    90%   1136
    95%   1176
    98%   1251
    99%   1280
   100%   1280 (longest request)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1723#issuecomment-1110279869
issue_url: https://api.github.com/repos/simonw/datasette/issues/1723
node_id: IC_kwDOBm6k_c5CLYa9
created_at: 2022-04-26T21:45:39Z
updated_at: 2022-04-26T21:45:39Z
author_association: OWNER
body:
  Getting some nice traces out of this:

  <img width="1384" alt="CleanShot 2022-04-26 at 14 45 21@2x" src="https://user-images.githubusercontent.com/9599/165397745-e8bfbe0a-306f-45bd-81f1-f5f6fc6422b9.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1723#issuecomment-1110305790
issue_url: https://api.github.com/repos/simonw/datasette/issues/1723
node_id: IC_kwDOBm6k_c5CLev-
created_at: 2022-04-26T22:19:04Z
updated_at: 2022-04-26T22:19:04Z
author_association: OWNER
body:
  I realized that seeing the total time in queries wasn't enough to understand this, because if the queries were executed in serial or parallel it should still sum up to the same amount of SQL time (roughly).

  Instead I need to know how long the page took to render. But that's hard to display on the page since you can't measure it until rendering has finished!

  So I built an ASGI plugin to handle that measurement: https://github.com/simonw/datasette-total-page-time

  And with that plugin installed, `http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel2&_facet=other_fuel1&_parallel=1` (the parallel version) takes 377ms:

  <img width="543" alt="CleanShot 2022-04-26 at 15 17 38@2x" src="https://user-images.githubusercontent.com/9599/165401856-d592ed7a-0240-4514-b9d8-fb9e7d8c9629.png">

  While `http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel2&_facet=other_fuel1` (the serial version) takes 762ms:

  <img width="543" alt="image" src="https://user-images.githubusercontent.com/9599/165401933-6d647014-4cab-4fbd-b9aa-958fc24ff435.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1723#issuecomment-1110330554
issue_url: https://api.github.com/repos/simonw/datasette/issues/1723
node_id: IC_kwDOBm6k_c5CLky6
created_at: 2022-04-26T23:06:20Z
updated_at: 2022-04-26T23:06:20Z
author_association: OWNER
body:
  Deployed here: https://latest-with-plugins.datasette.io/github/commits?_facet=repo&_trace=1&_facet=committer

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1724#issuecomment-1110369004
issue_url: https://api.github.com/repos/simonw/datasette/issues/1724
node_id: IC_kwDOBm6k_c5CLuLs
created_at: 2022-04-27T00:16:35Z
updated_at: 2022-04-27T00:17:04Z
author_association: OWNER
body:
  I bet this is because it's exceeding the size limit: https://github.com/simonw/datasette/blob/da53e0360da4771ffb56a8e3eb3f7476f3168299/datasette/tracer.py#L80-L88

  https://github.com/simonw/datasette/blob/da53e0360da4771ffb56a8e3eb3f7476f3168299/datasette/tracer.py#L102-L113

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111380282
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CPlE6
created_at: 2022-04-27T19:10:27Z
updated_at: 2022-04-27T19:10:27Z
author_association: OWNER
body:
  Wrote more about that here: https://simonwillison.net/2022/Apr/27/parallel-queries/

  Compare https://latest-with-plugins.datasette.io/github/commits?_facet=repo&_facet=committer&_trace=1

  ![image](https://user-images.githubusercontent.com/9599/165601503-2083c5d2-d740-405c-b34d-85570744ca82.png)

  With the same thing but with parallel execution disabled:

  https://latest-with-plugins.datasette.io/github/commits?_facet=repo&_facet=committer&_trace=1&_noparallel=1

  ![image](https://user-images.githubusercontent.com/9599/165601525-98abbfb1-5631-4040-b6bd-700948d1db6e.png)

  Those total page load time numbers are very similar. Is this parallel optimization worthwhile?

  Maybe it's only worth it on larger databases? Or maybe larger databases perform worse with this?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111385875
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CPmcT
created_at: 2022-04-27T19:16:57Z
updated_at: 2022-04-27T19:16:57Z
author_association: OWNER
body:
  I just remembered the `--setting num_sql_threads` option... which defaults to 3! https://github.com/simonw/datasette/blob/942411ef946e9a34a2094944d3423cddad27efd3/datasette/app.py#L109-L113

  Would explain why the first trace never seems to show more than three SQL queries executing at once.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111390433
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CPnjh
created_at: 2022-04-27T19:21:02Z
updated_at: 2022-04-27T19:21:02Z
author_association: OWNER
body:
  One weird thing: I noticed that in the parallel trace above the SQL query bars are wider. Mousover shows duration in ms, and I got 13ms for this query:

      select message as value, count(*) as n from (

  But in the `?_noparallel=1` version that some query took 2.97ms.

  Given those numbers though I would expect the overall page time to be MUCH worse for the parallel version - but the page load times are instead very close to each other, with parallel often winning.

  This is super-weird.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111408273
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CPr6R
created_at: 2022-04-27T19:40:51Z
updated_at: 2022-04-27T19:42:17Z
author_association: OWNER
body:
  Relevant: here's the code that sets up a Datasette SQLite connection: https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L73-L96

  It's using `check_same_thread=False` - here's [the Python docs on that](https://docs.python.org/3/library/sqlite3.html#sqlite3.connect):

  > By default, *check_same_thread* is [`True`](https://docs.python.org/3/library/constants.html#True "True") and only the creating thread may use the connection. If set [`False`](https://docs.python.org/3/library/constants.html#False "False"), the returned connection may be shared across multiple threads. When using multiple threads with the same connection writing operations should be serialized by the user to avoid data corruption.

  This is why Datasette reserves a single connection for write queries and queues them up in memory, [as described here](https://simonwillison.net/2020/Feb/26/weeknotes-datasette-writes/).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111431785
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CPxpp
created_at: 2022-04-27T20:07:16Z
updated_at: 2022-04-27T20:07:16Z
author_association: OWNER
body:
  I think I need some much more in-depth tracing tricks for this.

  https://www.maartenbreddels.com/perf/jupyter/python/tracing/gil/2021/01/14/Tracing-the-Python-GIL.html looks relevant - uses the `perf` tool on Linux.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111432375
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CPxy3
created_at: 2022-04-27T20:07:57Z
updated_at: 2022-04-27T20:07:57Z
author_association: OWNER
body:
  Also useful: https://avi.im/blag/2021/fast-sqlite-inserts/ - from a tip on Twitter: https://twitter.com/ricardoanderegg/status/1519402047556235264

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111442012
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CP0Jc
created_at: 2022-04-27T20:19:00Z
updated_at: 2022-04-27T20:19:00Z
author_association: OWNER
body:
  Something worth digging into: are these parallel queries running against the same SQLite connection or are they each rubbing against a separate SQLite connection?

  Just realized I know the answer: they're running against separate SQLite connections, because that's how the time limit mechanism works: it installs a progress handler for each connection which terminates it after a set time.

  This means that if SQLite benefits from multiple threads using the same connection (due to shared caches or similar) then Datasette will not be seeing those benefits.

  It also means that if there's some mechanism within SQLite that penalizes you for having multiple parallel connections to a single file (just guessing here, maybe there's some kind of locking going on?) then Datasette will suffer those penalties.

  I should try seeing what happens with WAL mode enabled.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111448928
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CP11g
created_at: 2022-04-27T20:27:05Z
updated_at: 2022-04-27T20:27:05Z
author_association: NONE
body:
  You don't want to re-use an SQLite connection from multiple threads anyway: https://www.sqlite.org/threadsafe.html

  Multiple connections can operate on the file in parallel, but a single connection can't:

  > Multi-thread. In this mode, SQLite can be safely used by multiple threads **provided that no single database connection is used simultaneously in two or more threads**.

  (emphasis mine)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111451790
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CP2iO
created_at: 2022-04-27T20:30:33Z
updated_at: 2022-04-27T20:30:33Z
author_association: NONE
body:
  > I should try seeing what happens with WAL mode enabled.

  I've only skimmed above but it looks like you're doing mainly read-only queries?  WAL mode is about better interactions between writers & readers, primarily.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111456500
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CP3r0
created_at: 2022-04-27T20:36:01Z
updated_at: 2022-04-27T20:36:01Z
author_association: OWNER
body:
  Yeah all of this is pretty much assuming read-only connections. Datasette has a separate mechanism for ensuring that writes are executed one at a time against a dedicated connection from an in-memory queue:
  - https://github.com/simonw/datasette/issues/682

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111460068
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CP4jk
created_at: 2022-04-27T20:38:32Z
updated_at: 2022-04-27T20:38:32Z
author_association: OWNER
body:
  WAL mode didn't seem to make a difference. I thought there was a chance it might help multiple read connections operate at the same time but it looks like it really does only matter for when writes are going on.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111462442
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CP5Iq
created_at: 2022-04-27T20:40:59Z
updated_at: 2022-04-27T20:42:49Z
author_association: OWNER
body:
  This looks VERY relevant: [SQLite Shared-Cache Mode](https://www.sqlite.org/sharedcache.html):

  > SQLite includes a special "shared-cache" mode (disabled by default) intended for use in embedded servers. If shared-cache mode is enabled and a thread establishes multiple connections to the same database, the connections share a single data and schema cache. This can significantly reduce the quantity of memory and IO required by the system.

  Enabled as part of the URI filename:

      ATTACH 'file:aux.db?cache=shared' AS aux;

  Turns out I'm already using this for in-memory databases that have `.memory_name` set, but not (yet) for regular file-backed databases:

  https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L73-L75

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111485722
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CP-0a
created_at: 2022-04-27T21:08:20Z
updated_at: 2022-04-27T21:08:20Z
author_association: OWNER
body:
  Tried that and it didn't seem to make a difference either.

  I really need a much deeper view of what's going on here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111535818
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CQLDK
created_at: 2022-04-27T22:18:45Z
updated_at: 2022-04-27T22:18:45Z
author_association: OWNER
body:
  Another avenue: https://twitter.com/weargoggles/status/1519426289920270337

  > SQLite has its own mutexes to provide thread safety, which as another poster noted are out of play in multi process setups. Perhaps downgrading from the “serializable” to “multi-threaded” safety would be okay for Datasette? https://sqlite.org/c3ref/c_config_covering_index_scan.html#sqliteconfigmultithread

  Doesn't look like there's an obvious way to access that from Python via the `sqlite3` module though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111551076
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CQOxk
created_at: 2022-04-27T22:44:51Z
updated_at: 2022-04-27T22:45:04Z
author_association: OWNER
body:
  Really wild idea: what if I created three copies of the SQLite database file - as three separate file names - and then balanced the parallel queries across all these? Any chance that could avoid any mysterious locking issues?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111558204
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CQQg8
created_at: 2022-04-27T22:58:39Z
updated_at: 2022-04-27T22:58:39Z
author_association: OWNER
body:
  I should check my timing mechanism. Am I capturing the time taken just in SQLite or does it include time spent in Python crossing between async and threaded world and waiting for a thread pool worker to become available?

  That could explain the longer query times.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111595319
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CQZk3
created_at: 2022-04-28T00:09:45Z
updated_at: 2022-04-28T00:11:01Z
author_association: OWNER
body:
  Here's where read queries are instrumented: https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L241-L242

  So the instrumentation is actually capturing quite a bit of Python activity before it gets to SQLite:

  https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L179-L190

  And then:

  https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L204-L233

  Ideally I'd like that `trace()` block to wrap just the `cursor.execute()` and `cursor.fetchmany(...)` or `cursor.fetchall()` calls.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111597176
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CQaB4
created_at: 2022-04-28T00:11:44Z
updated_at: 2022-04-28T00:11:44Z
author_association: OWNER
body:
  Though it would be interesting to also have the trace reveal how much time is spent in the functions that wrap that core SQL - the stuff that is being measured at the moment.

  I have a hunch that this could help solve the over-arching performance mystery.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111602802
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CQbZy
created_at: 2022-04-28T00:21:35Z
updated_at: 2022-04-28T00:21:35Z
author_association: OWNER
body:
  Tried this but I'm getting back an empty JSON array of traces at the bottom of the page most of the time (intermittently it works correctly):

  ```diff
  diff --git a/datasette/database.py b/datasette/database.py
  index ba594a8..d7f9172 100644
  --- a/datasette/database.py
  +++ b/datasette/database.py
  @@ -7,7 +7,7 @@ import sys
   import threading
   import uuid
   
  -from .tracer import trace
  +from .tracer import trace, trace_child_tasks
   from .utils import (
       detect_fts,
       detect_primary_keys,
  @@ -207,30 +207,31 @@ class Database:
                   time_limit_ms = custom_time_limit
   
               with sqlite_timelimit(conn, time_limit_ms):
  -                try:
  -                    cursor = conn.cursor()
  -                    cursor.execute(sql, params if params is not None else {})
  -                    max_returned_rows = self.ds.max_returned_rows
  -                    if max_returned_rows == page_size:
  -                        max_returned_rows += 1
  -                    if max_returned_rows and truncate:
  -                        rows = cursor.fetchmany(max_returned_rows + 1)
  -                        truncated = len(rows) > max_returned_rows
  -                        rows = rows[:max_returned_rows]
  -                    else:
  -                        rows = cursor.fetchall()
  -                        truncated = False
  -                except (sqlite3.OperationalError, sqlite3.DatabaseError) as e:
  -                    if e.args == ("interrupted",):
  -                        raise QueryInterrupted(e, sql, params)
  -                    if log_sql_errors:
  -                        sys.stderr.write(
  -                            "ERROR: conn={}, sql = {}, params = {}: {}\n".format(
  -                                conn, repr(sql), params, e
  +                with trace("sql", database=self.name, sql=sql.strip(), params=params):
  +                    try:
  +                        cursor = conn.cursor()
  +                        cursor.execute(sql, params if params is not None else {})
  +                        max_returned_rows = self.ds.max_returned_rows
  +                        if max_returned_rows == page_size:
  +                            max_returned_rows += 1
  +                        if max_returned_rows and truncate:
  +                            rows = cursor.fetchmany(max_returned_rows + 1)
  +                            truncated = len(rows) > max_returned_rows
  +                            rows = rows[:max_returned_rows]
  +                        else:
  +                            rows = cursor.fetchall()
  +                            truncated = False
  +                    except (sqlite3.OperationalError, sqlite3.DatabaseError) as e:
  +                        if e.args == ("interrupted",):
  +                            raise QueryInterrupted(e, sql, params)
  +                        if log_sql_errors:
  +                            sys.stderr.write(
  +                                "ERROR: conn={}, sql = {}, params = {}: {}\n".format(
  +                                    conn, repr(sql), params, e
  +                                )
                               )
  -                        )
  -                        sys.stderr.flush()
  -                    raise
  +                            sys.stderr.flush()
  +                        raise
   
               if truncate:
                   return Results(rows, truncated, cursor.description)
  @@ -238,9 +239,8 @@ class Database:
               else:
                   return Results(rows, False, cursor.description)
   
  -        with trace("sql", database=self.name, sql=sql.strip(), params=params):
  -            results = await self.execute_fn(sql_operation_in_thread)
  -        return results
  +        with trace_child_tasks():
  +            return await self.execute_fn(sql_operation_in_thread)
   
       @property
       def size(self):
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111681513
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CQunp
created_at: 2022-04-28T02:44:26Z
updated_at: 2022-04-28T02:44:26Z
author_association: OWNER
body:
  I could try `py-spy top`, which I previously used here:
  - https://github.com/simonw/datasette/issues/1673

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111683539
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CQvHT
created_at: 2022-04-28T02:47:57Z
updated_at: 2022-04-28T02:47:57Z
author_association: OWNER
body:
  Maybe this is the Python GIL after all?

  I've been hoping that the GIL won't be an issue because the `sqlite3` module releases the GIL for the duration of the execution of a SQL query - see https://github.com/python/cpython/blob/f348154c8f8a9c254503306c59d6779d4d09b3a9/Modules/_sqlite/cursor.c#L749-L759

  So I've been hoping this means that SQLite code itself can run concurrently on multiple cores even when Python threads cannot.

  But maybe I'm misunderstanding how that works?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1728#issuecomment-1111697985
issue_url: https://api.github.com/repos/simonw/datasette/issues/1728
node_id: IC_kwDOBm6k_c5CQypB
created_at: 2022-04-28T03:17:20Z
updated_at: 2022-04-28T03:17:20Z
author_association: OWNER
body:
  How did you deploy to Cloud Run?

  `datasette publish cloudrun` defaults to running databases there in `-i` immutable mode, because if you managed to change a file on disk on Cloud Run those changes would be lost the next time your container restarted there.

  That's why I upgraded `datasette-publish-fly` to provide a way of working with their volumes support - they're the best option I know of right now for running Datasette in a container with a persistent volume that can accept writes: https://simonwillison.net/2022/Feb/15/fly-volumes/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1728#issuecomment-1111698307
issue_url: https://api.github.com/repos/simonw/datasette/issues/1728
node_id: IC_kwDOBm6k_c5CQyuD
created_at: 2022-04-28T03:18:02Z
updated_at: 2022-04-28T03:18:02Z
author_association: OWNER
body:
  If the behaviour you are seeing is because the database is running in immutable mode then that's a bug - you should get a useful error message instead!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111699175
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CQy7n
created_at: 2022-04-28T03:19:48Z
updated_at: 2022-04-28T03:20:08Z
author_association: OWNER
body:
  I ran `py-spy` and then hammered refresh a bunch of times on the `http://127.0.0.1:8856/github/commits?_facet=repo&_facet=committer&_trace=1&_noparallel=` page - it generated this SVG profile for me.

  The area on the right is the threads running the DB queries:

  ![profile](https://user-images.githubusercontent.com/9599/165669677-5461ede5-3dc4-4b49-8319-bfe5fd8a723d.svg)

  Interactive version here: https://static.simonwillison.net/static/2022/datasette-parallel-profile.svg

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1728#issuecomment-1111705069
issue_url: https://api.github.com/repos/simonw/datasette/issues/1728
node_id: IC_kwDOBm6k_c5CQ0Xt
created_at: 2022-04-28T03:31:33Z
updated_at: 2022-04-28T03:31:33Z
author_association: OWNER
body:
  Confirmed - this is a bug where immutable databases fail to show a useful error if you write to them with a canned query.

  Steps to reproduce:
  ```
  echo '
  databases:
    writable:
      queries:
        add_name:
          sql: insert into names(name) values (:name)
          write: true
  ' > write-metadata.yml
  echo '{"name": "Simon"}' | sqlite-utils insert writable.db names -
  datasette writable.db -m write-metadata.yml
  ```
  Then visit http://127.0.0.1:8001/writable/add_name - adding names works.

  Now do this instead:

  ```
  datasette -i writable.db -m write-metadata.yml
  ```

  And I'm getting a broken error:

  ![error](https://user-images.githubusercontent.com/9599/165670823-6604dd69-9905-475c-8098-5da22ab026a1.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1728#issuecomment-1111705323
issue_url: https://api.github.com/repos/simonw/datasette/issues/1728
node_id: IC_kwDOBm6k_c5CQ0br
created_at: 2022-04-28T03:32:06Z
updated_at: 2022-04-28T03:32:06Z
author_association: CONTRIBUTOR
body:
  Ah, that would be it! I have a core set of data which doesn't change to which I want authorised users to be able to submit corrections. I was going to deal with the persistence issue by just grabbing the user corrections at regular intervals and saving to GitHub. I might need to rethink. Thanks!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1728#issuecomment-1111706519
issue_url: https://api.github.com/repos/simonw/datasette/issues/1728
node_id: IC_kwDOBm6k_c5CQ0uX
created_at: 2022-04-28T03:34:49Z
updated_at: 2022-04-28T03:34:49Z
author_association: OWNER
body:
  I've wanted to do stuff like that on Cloud Run too. So far I've assumed that it's not feasible, but recently I've been wondering how hard it would be to have a small (like less than 100KB or so) Datasette instance which persists data to a backing GitHub repository such that when it starts up it can pull the latest copy and any time someone edits it can push their changes.

  I'm still not sure it would work well on Cloud Run due to the uncertainty at what would happen if Cloud Run decided to boot up a second instance - but it's still an interesting thought exercise.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1728#issuecomment-1111707384
issue_url: https://api.github.com/repos/simonw/datasette/issues/1728
node_id: IC_kwDOBm6k_c5CQ074
created_at: 2022-04-28T03:36:46Z
updated_at: 2022-04-28T03:36:56Z
author_association: OWNER
body:
  A more realistic solution (which I've been using on several of my own projects) is to keep the data itself in GitHub and encourage users to edit it there - using the GitHub web interface to edit YAML files or similar.

  Needs your users to be comfortable hand-editing YAML though! You can at least guard against critical errors by having CI run tests against their YAML before deploying.

  I have a dream of building a more friendly web forms interface which edits the YAML back on GitHub for the user, but that's just a concept at the moment.

  Even more fun would be if a user-friendly form could submit PRs for review without the user having to know what a PR is!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1728#issuecomment-1111708206
issue_url: https://api.github.com/repos/simonw/datasette/issues/1728
node_id: IC_kwDOBm6k_c5CQ1Iu
created_at: 2022-04-28T03:38:56Z
updated_at: 2022-04-28T03:38:56Z
author_association: OWNER
body:
  In terms of this bug, there are a few potential fixes:

  1. Detect the write to a immutable database and show the user a proper, meaningful error message in the red error box at the top of the page
  2. Don't allow the user to even submit the form - show a message saying that this canned query is unavailable because the database cannot be written to
  3. Don't even allow Datasette to start running at all - if there's a canned query configured in `metadata.yml` and the database it refers to is in `-i` immutable mode throw an error on startup

  I'm not keen on that last one because it would be frustrating if you couldn't launch Datasette just because you had an old canned query lying around in your metadata file.

  So I'm leaning towards option 2.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1728#issuecomment-1111712953
issue_url: https://api.github.com/repos/simonw/datasette/issues/1728
node_id: IC_kwDOBm6k_c5CQ2S5
created_at: 2022-04-28T03:48:36Z
updated_at: 2022-04-28T03:48:36Z
author_association: CONTRIBUTOR
body:
  I don't think that'd work for this project. The db is very big, and my aim was to have an environment where researchers could be making use of the data, but be easily able to add corrections to the HTR/OCR extracted data when they came across problems. It's in its immutable (!) form here: https://sydney-stock-exchange-xqtkxtd5za-ts.a.run.app/stock_exchange/stocks

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1728#issuecomment-1111714665
issue_url: https://api.github.com/repos/simonw/datasette/issues/1728
node_id: IC_kwDOBm6k_c5CQ2tp
created_at: 2022-04-28T03:52:47Z
updated_at: 2022-04-28T03:52:58Z
author_association: OWNER
body:
  Nice custom template/theme!

  Yeah, for that I'd recommend hosting elsewhere - on a regular VPS (I use `systemd` like this: https://docs.datasette.io/en/stable/deploying.html#running-datasette-using-systemd ) or using Fly if you want to tub containers without managing a full server.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111725638
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CQ5ZG
created_at: 2022-04-28T04:15:15Z
updated_at: 2022-04-28T04:15:15Z
author_association: OWNER
body:
  Useful theory from Keith Medcalf https://sqlite.org/forum/forumpost/e363c69d3441172e

  > This is true, but the concurrency is limited to the execution which occurs with the GIL released (that is, in the native C sqlite3 library itself). Each row (for example) can be retrieved in parallel but "constructing the python return objects for each row" will be serialized (by the GIL).
  > 
  > That is to say that if your have two python threads each with their own connection, and each one is performing a select that returns 1,000,000 rows (lets say that is 25% of the candidates for each select) then the difference in execution time between executing two python threads in parallel vs a single serial thead will not be much different (if even detectable at all). In fact it is possible that the multiple-threaded version takes longer to run both queries to completion because of the increased contention over a shared resource (the GIL).

  So maybe this is a GIL thing.

  I should test with some expensive SQL queries (maybe big aggregations against large tables) and see if I can spot an improvement there.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1111726586
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CQ5n6
created_at: 2022-04-28T04:17:16Z
updated_at: 2022-04-28T04:19:31Z
author_association: OWNER
body:
  I could experiment with the `await asyncio.run_in_executor(processpool_executor, fn)` mechanism described in https://stackoverflow.com/a/29147750

  Code examples: https://cs.github.com/?scopeName=All+repos&scope=&q=run_in_executor+ProcessPoolExecutor

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1633#issuecomment-1111955628
issue_url: https://api.github.com/repos/simonw/datasette/issues/1633
node_id: IC_kwDOBm6k_c5CRxis
created_at: 2022-04-28T09:12:56Z
updated_at: 2022-04-28T09:12:56Z
author_association: NONE
body:
  I have verified that the problem with base_url still exists in the latest version 0.61.1. I would need some guidance if my code change suggestion is correct or if base_url should be included in some other code?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1112668411
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CUfj7
created_at: 2022-04-28T21:25:34Z
updated_at: 2022-04-28T21:25:44Z
author_association: OWNER
body:
  The two most promising theories at the moment, from here and Twitter and the SQLite forum, are:

  - SQLite is I/O bound - it generally only goes as fast as it can load data from disk. Multiple connections all competing for the same file on disk are going to end up blocked at the file system layer. But maybe this means in-memory databases will perform better?
  - It's the GIL. The sqlite3 C code may release the GIL, but the bits that do things like assembling `Row` objects to return still happen in Python, and that Python can only run on a single core.

  A couple of ways to research the in-memory theory:

  - Use a RAM disk on macOS (or Linux). https://stackoverflow.com/a/2033417/6083 has instructions - short version:

        hdiutil attach -nomount ram://$((2 * 1024 * 100))
        diskutil eraseVolume HFS+ RAMDisk name-returned-by-previous-command (was `/dev/disk2` when I tried it)
        cd /Volumes/RAMDisk
        cp ~/fixtures.db .

  - Copy Datasette databases into an in-memory database on startup. I built a new plugin to do that here: https://github.com/simonw/datasette-copy-to-memory

  I need to do some more, better benchmarks using these different approaches.

  https://twitter.com/laurencerowe/status/1519780174560169987 also suggests:

  > Maybe try:
  > 1. Copy the sqlite file to /dev/shm and rerun (all in ram.)
  > 2. Create a CTE which calculates Fibonacci or similar so you can test something completely cpu bound (only return max value or something to avoid crossing between sqlite/Python.)

  I like that second idea a lot - I could use the mandelbrot example from https://www.sqlite.org/lang_with.html#outlandish_recursive_query_examples

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1715#issuecomment-1112711115
issue_url: https://api.github.com/repos/simonw/datasette/issues/1715
node_id: IC_kwDOBm6k_c5CUp_L
created_at: 2022-04-28T22:26:56Z
updated_at: 2022-04-28T22:26:56Z
author_association: OWNER
body:
  I'm not going to use `asyncinject` in this refactor - at least not until I really need it. My research in these issues has put me off the idea ( in favour of `asyncio.gather()` or even not trying for parallel execution at all):

  - #1727

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1729#issuecomment-1112713581
issue_url: https://api.github.com/repos/simonw/datasette/issues/1729
node_id: IC_kwDOBm6k_c5CUqlt
created_at: 2022-04-28T22:31:11Z
updated_at: 2022-04-28T22:31:11Z
author_association: OWNER
body:
  I'm going to change the default API response to look like this: 
  ```json
  {
    "rows": [
      {
        "pk": 1,
        "created": "2019-01-14 08:00:00",
        "planet_int": 1,
        "on_earth": 1,
        "state": "CA",
        "_city_id": 1,
        "_neighborhood": "Mission",
        "tags": "[\"tag1\", \"tag2\"]",
        "complex_array": "[{\"foo\": \"bar\"}]",
        "distinct_some_null": "one",
        "n": "n1"
      },
      {
        "pk": 2,
        "created": "2019-01-14 08:00:00",
        "planet_int": 1,
        "on_earth": 1,
        "state": "CA",
        "_city_id": 1,
        "_neighborhood": "Dogpatch",
        "tags": "[\"tag1\", \"tag3\"]",
        "complex_array": "[]",
        "distinct_some_null": "two",
        "n": "n2"
      }
    ],
    "next": null,
    "next_url": null
  }
  ```
  Basically https://latest.datasette.io/fixtures/facetable.json?_shape=objects but with just the `rows`, `next` and `next_url` fields returned by default.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1729#issuecomment-1112716611
issue_url: https://api.github.com/repos/simonw/datasette/issues/1729
node_id: IC_kwDOBm6k_c5CUrVD
created_at: 2022-04-28T22:36:24Z
updated_at: 2022-04-28T22:36:24Z
author_association: OWNER
body:
  Then I'm going to implement the following `?_extra=` options:

  - `?_extra=facet_results` - to see facet results
  - `?_extra=suggested_facets` - for suggested facets
  - `?_extra=count` - for the count of total rows
  - `?_extra=columns` - for a list of column names
  - `?_extra=primary_keys` - for a list of primary keys
  - `?_extra=query` - a `{"sql" "select ...", "params": {}}` object

  I thought about having `?_extra=facet_results` returned automatically if the user specifies at least one `?_facet` - but that doesn't work for default facets configured in `metadata.json` - how can the user opt out of those being returned? So I'm going to say you don't see facets at all if you don't include `?_extra=facet_results`.

  I'm tempted to add `?_extra=_all` to return everything, but I can decide if that's a good idea later.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1729#issuecomment-1112717210
issue_url: https://api.github.com/repos/simonw/datasette/issues/1729
node_id: IC_kwDOBm6k_c5CUrea
created_at: 2022-04-28T22:37:37Z
updated_at: 2022-04-28T22:37:37Z
author_association: OWNER
body:
  This means `filtered_table_rows_count` is going to become `count`. I had originally picked that terrible name to avoid confusion between the count of all rows in the table and the count of rows that were filtered.

  I'll add `?_extra=table_count` for getting back the full table count instead. I think `count` is clear enough!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1729#issuecomment-1112717745
issue_url: https://api.github.com/repos/simonw/datasette/issues/1729
node_id: IC_kwDOBm6k_c5CUrmx
created_at: 2022-04-28T22:38:39Z
updated_at: 2022-04-28T22:39:05Z
author_association: OWNER
body:
  (I remain keen on the idea of shipping a plugin that restores the old default API shape to people who have written pre-Datasette-1.0 code against it, but I'll tackle that much later. I really like how jQuery has a culture of doing this.)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1729#issuecomment-1112721321
issue_url: https://api.github.com/repos/simonw/datasette/issues/1729
node_id: IC_kwDOBm6k_c5CUsep
created_at: 2022-04-28T22:44:05Z
updated_at: 2022-04-28T22:44:14Z
author_association: OWNER
body:
  I may be able to implement this mostly in the `json_renderer()` function: https://github.com/simonw/datasette/blob/94a3171b01fde5c52697aeeff052e3ad4bab5391/datasette/renderer.py#L29-L34

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1729#issuecomment-1112730416
issue_url: https://api.github.com/repos/simonw/datasette/issues/1729
node_id: IC_kwDOBm6k_c5CUusw
created_at: 2022-04-28T23:01:21Z
updated_at: 2022-04-28T23:01:21Z
author_association: OWNER
body:
  I'm not sure what to do about the `"truncated": true/false` key.

  It's not really relevant to table results, since they are paginated whether or not you ask for them to be.

  It plays a role in query results, where you might run `select * from table` and get back 1000 results because Datasette truncates at that point rather than returning everything.

  Adding it to every table result and always setting it to `"truncated": false` feels confusing.

  I think I'm going to keep it exclusively in the default representation for the `/db?sql=...` query endpoint, and not return it at all for tables.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1729#issuecomment-1112732563
issue_url: https://api.github.com/repos/simonw/datasette/issues/1729
node_id: IC_kwDOBm6k_c5CUvOT
created_at: 2022-04-28T23:05:03Z
updated_at: 2022-04-28T23:05:03Z
author_association: OWNER
body:
  OK, the prototype of this is looking really good - it's very pleasant to use.

  `http://127.0.0.1:8001/github_memory/issue_comments.json?_search=simon&_sort=id&_size=5&_extra=query_ms&_extra=count&_col=body` returns this:

  ```json
  {
    "rows": [
      {
        "id": 338854988,
        "body": "    /database-name/table-name?name__contains=simon&sort=id+desc\r\n\r\nNote that if there's a column called \"sort\" you can still do sort__exact=blah\r\n\r\n"
      },
      {
        "id": 346427794,
        "body": "Thanks. There is a way to use pip to grab apsw, which also let's you configure it (flags to build extensions, use an internal sqlite, etc). Don't know how that works as a dependency for another package, though.\n\nOn November 22, 2017 11:38:06 AM EST, Simon Willison <notifications@github.com> wrote:\n>I have a solution for FTS already, but I'm interested in apsw as a\n>mechanism for allowing custom virtual tables to be written in Python\n>(pysqlite only lets you write custom functions)\n>\n>Not having PyPI support is pretty tough though. I'm planning a\n>plugin/extension system which would be ideal for things like an\n>optional apsw mode, but that's a lot harder if apsw isn't in PyPI.\n>\n>-- \n>You are receiving this because you authored the thread.\n>Reply to this email directly or view it on GitHub:\n>https://github.com/simonw/datasette/issues/144#issuecomment-346405660\n"
      },
      {
        "id": 348252037,
        "body": "WOW!\n\n--\nPaul Ford // (646) 369-7128 // @ftrain\n\nOn Thu, Nov 30, 2017 at 11:47 AM, Simon Willison <notifications@github.com>\nwrote:\n\n> Remaining work on this now lives in a milestone:\n> https://github.com/simonw/datasette/milestone/6\n>\n> —\n> You are receiving this because you were mentioned.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/simonw/datasette/issues/153#issuecomment-348248406>,\n> or mute the thread\n> <https://github.com/notifications/unsubscribe-auth/AABPKHzaVPKwTOoHouK2aMUnM-mPnPk6ks5s7twzgaJpZM4Qq2zW>\n> .\n>\n"
      },
      {
        "id": 391141391,
        "body": "I'm going to clean this up for consistency tomorrow morning so hold off\nmerging until then please\n\nOn Tue, May 22, 2018 at 6:34 PM, Simon Willison <notifications@github.com>\nwrote:\n\n> Yeah let's try this without pysqlite3 and see if we still get the correct\n> version.\n>\n> —\n> You are receiving this because you authored the thread.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/simonw/datasette/pull/280#issuecomment-391076458>, or mute\n> the thread\n> <https://github.com/notifications/unsubscribe-auth/AAihfMI-H6CBt-Py0xdBbH2xDK0KsjT2ks5t1EwYgaJpZM4UI_2m>\n> .\n>\n"
      },
      {
        "id": 391355030,
        "body": "No objections;\r\nIt's good to go @simonw\r\n\r\nOn Wed, 23 May 2018, 14:51 Simon Willison, <notifications@github.com> wrote:\r\n\r\n> @r4vi <https://github.com/r4vi> any objections to me merging this?\r\n>\r\n> —\r\n> You are receiving this because you were mentioned.\r\n> Reply to this email directly, view it on GitHub\r\n> <https://github.com/simonw/datasette/pull/280#issuecomment-391354237>, or mute\r\n> the thread\r\n> <https://github.com/notifications/unsubscribe-auth/AAihfM_2DN5WR2mkO-VK6ozDmkUQ4IMjks5t1WlcgaJpZM4UI_2m>\r\n> .\r\n>\r\n"
      }
    ],
    "next": "391355030,391355030",
    "next_url": "http://127.0.0.1:8001/github_memory/issue_comments.json?_search=simon&_size=5&_extra=query_ms&_extra=count&_col=body&_next=391355030%2C391355030&_sort=id",
    "count": 57,
    "query_ms": 21.780223003588617
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1729#issuecomment-1112734577
issue_url: https://api.github.com/repos/simonw/datasette/issues/1729
node_id: IC_kwDOBm6k_c5CUvtx
created_at: 2022-04-28T23:08:42Z
updated_at: 2022-04-28T23:08:42Z
author_association: OWNER
body:
  That prototype is a very small amount of code so far:
  ```diff
  diff --git a/datasette/renderer.py b/datasette/renderer.py
  index 4508949..b600e1b 100644
  --- a/datasette/renderer.py
  +++ b/datasette/renderer.py
  @@ -28,6 +28,10 @@ def convert_specific_columns_to_json(rows, columns, json_cols):
   
   def json_renderer(args, data, view_name):
       """Render a response as JSON"""
  +    from pprint import pprint
  +
  +    pprint(data)
  +
       status_code = 200
   
       # Handle the _json= parameter which may modify data["rows"]
  @@ -43,6 +47,41 @@ def json_renderer(args, data, view_name):
       if "rows" in data and not value_as_boolean(args.get("_json_infinity", "0")):
           data["rows"] = [remove_infinites(row) for row in data["rows"]]
   
  +    # Start building the default JSON here
  +    columns = data["columns"]
  +    next_url = data.get("next_url")
  +    output = {
  +        "rows": [dict(zip(columns, row)) for row in data["rows"]],
  +        "next": data["next"],
  +        "next_url": next_url,
  +    }
  +
  +    extras = set(args.getlist("_extra"))
  +
  +    extras_map = {
  +        # _extra=   :    data[field]
  +        "count": "filtered_table_rows_count",
  +        "facet_results": "facet_results",
  +        "suggested_facets": "suggested_facets",
  +        "columns": "columns",
  +        "primary_keys": "primary_keys",
  +        "query_ms": "query_ms",
  +        "query": "query",
  +    }
  +    for extra_key, data_key in extras_map.items():
  +        if extra_key in extras:
  +            output[extra_key] = data[data_key]
  +
  +    body = json.dumps(output, cls=CustomJSONEncoder)
  +    content_type = "application/json; charset=utf-8"
  +    headers = {}
  +    if next_url:
  +        headers["link"] = f'<{next_url}>; rel="next"'
  +    return Response(
  +        body, status=status_code, headers=headers, content_type=content_type
  +    )
  +
  +
       # Deal with the _shape option
       shape = args.get("_shape", "arrays")
       # if there's an error, ignore the shape entirely
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1112878955
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CVS9r
created_at: 2022-04-29T05:02:40Z
updated_at: 2022-04-29T05:02:40Z
author_association: OWNER
body:
  Here's a very useful (recent) article about how the GIL works and how to think about it: https://pythonspeed.com/articles/python-gil/ - via https://lobste.rs/s/9hj80j/when_python_can_t_thread_deep_dive_into_gil

  From that article:

  > For example, let's consider an extension module written in C or Rust that lets you talk to a PostgreSQL database server.
  > 
  > Conceptually, handling a SQL query with this library will go through three steps:
  > 
  > 1.  Deserialize from Python to the internal library representation. Since this will be reading Python objects, it needs to hold the GIL.
  > 2.  Send the query to the database server, and wait for a response. This doesn't need the GIL.
  > 3.  Convert the response into Python objects. This needs the GIL again.
  > 
  > As you can see, how much parallelism you can get depends on how much time is spent in each step. If the bulk of time is spent in step 2, you'll get parallelism there. But if, for example, you run a `SELECT` and get a large number of rows back, the library will need to create many Python objects, and step 3 will have to hold GIL for a while.

  That explains what I'm seeing here. I'm pretty convinced now that the reason I'm not getting a performance boost from parallel queries is that there's more time spent in Python code assembling the results than in SQLite C code executing the query.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1112879463
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CVTFn
created_at: 2022-04-29T05:03:58Z
updated_at: 2022-04-29T05:03:58Z
author_association: OWNER
body:
  It would be _really_ fun to try running this with the in-development `nogil` Python from https://github.com/colesbury/nogil

  There's a Docker container for it: https://hub.docker.com/r/nogil/python

  It suggests you can run something like this:

      docker run -it --rm --name my-running-script -v "$PWD":/usr/src/myapp \
        -w /usr/src/myapp nogil/python python your-daemon-or-script.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1112889800
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CVVnI
created_at: 2022-04-29T05:29:38Z
updated_at: 2022-04-29T05:29:38Z
author_association: OWNER
body:
  OK, I just got the most incredible result with that!

  I started up a container running `bash` like this, from my `datasette` checkout. I'm mapping port 8005 on my laptop to port 8001 inside the container because laptop port 8001 was already doing something else:
  ```
  docker run -it --rm --name my-running-script -p 8005:8001 -v "$PWD":/usr/src/myapp \
    -w /usr/src/myapp nogil/python bash
  ```
  Then in `bash` I ran the following commands to install Datasette and its dependencies:
  ```
  pip install -e '.[test]'
  pip install datasette-pretty-traces # For debug tracing
  ```
  Then I started Datasette against my `github.db` database (from github-to-sqlite.dogsheep.net/github.db) like this:

  ```
  datasette github.db -h 0.0.0.0 --setting trace_debug 1
  ```
  I hit the following two URLs to compare the parallel v.s. not parallel implementations:

  - `http://127.0.0.1:8005/github/issues?_facet=milestone&_facet=repo&_trace=1&_size=10`
  - `http://127.0.0.1:8005/github/issues?_facet=milestone&_facet=repo&_trace=1&_size=10&_noparallel=1`

  And... the parallel one beat the non-parallel one decisively, on multiple page refreshes!

  Not parallel: 77ms

  Parallel: 47ms

  <img width="1213" alt="CleanShot 2022-04-28 at 22 10 54@2x" src="https://user-images.githubusercontent.com/9599/165889437-60d4200d-698a-4175-af23-7c03bb456e66.png">

  <img width="1213" alt="CleanShot 2022-04-28 at 22 10 21@2x" src="https://user-images.githubusercontent.com/9599/165889445-2dfb8676-d823-405e-aecb-ad28ec3043da.png">

  So yeah, I'm very confident this is a problem with the GIL. And I am absolutely **stunned** that @colesbury's fork ran Datasette (which has some reasonably tricky threading and async stuff going on) out of the box!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1729#issuecomment-1114013757
issue_url: https://api.github.com/repos/simonw/datasette/issues/1729
node_id: IC_kwDOBm6k_c5CZoA9
created_at: 2022-04-30T16:15:51Z
updated_at: 2022-04-30T18:54:39Z
author_association: OWNER
body:
  Deployed a preview of this here: https://latest-1-0-alpha.datasette.io/

  Examples:

  - https://latest-1-0-alpha.datasette.io/fixtures/facetable.json
  - https://latest-1-0-alpha.datasette.io/fixtures/facetable.json?_facet=state&_size=0&_extra=facet_results&_extra=count

  Second example produces:

  ```json
  {
    "rows": [],
    "next": null,
    "next_url": null,
    "count": 15,
    "facet_results": {
      "state": {
        "name": "state",
        "type": "column",
        "hideable": true,
        "toggle_url": "/fixtures/facetable.json?_size=0&_extra=facet_results&_extra=count",
        "results": [
          {
            "value": "CA",
            "label": "CA",
            "count": 10,
            "toggle_url": "https://latest-1-0-alpha.datasette.io/fixtures/facetable.json?_facet=state&_size=0&_extra=facet_results&_extra=count&state=CA",
            "selected": false
          },
          {
            "value": "MI",
            "label": "MI",
            "count": 4,
            "toggle_url": "https://latest-1-0-alpha.datasette.io/fixtures/facetable.json?_facet=state&_size=0&_extra=facet_results&_extra=count&state=MI",
            "selected": false
          },
          {
            "value": "MC",
            "label": "MC",
            "count": 1,
            "toggle_url": "https://latest-1-0-alpha.datasette.io/fixtures/facetable.json?_facet=state&_size=0&_extra=facet_results&_extra=count&state=MC",
            "selected": false
          }
        ],
        "truncated": false
      }
    }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1729#issuecomment-1114036946
issue_url: https://api.github.com/repos/simonw/datasette/issues/1729
node_id: IC_kwDOBm6k_c5CZtrS
created_at: 2022-04-30T18:56:25Z
updated_at: 2022-04-30T19:04:03Z
author_association: OWNER
body:
  Related:
  - #1558

  Which talks about how there was confusion in this example: https://latest.datasette.io/fixtures/facetable.json?_facet=created&_facet_date=created&_facet=tags&_facet_array=tags&_nosuggest=1&_size=0

  Which I fixed in #625 by introducing `tags` and `tags_2` keys, but actually the facet results would be better if they were a list rather than a dictionary.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1729#issuecomment-1114037521
issue_url: https://api.github.com/repos/simonw/datasette/issues/1729
node_id: IC_kwDOBm6k_c5CZt0R
created_at: 2022-04-30T19:01:07Z
updated_at: 2022-04-30T19:01:07Z
author_association: OWNER
body:
  I had to look up what `hideable` means - it means that you can't hide the current facet because it was defined in metadata, not as a `?_facet=` parameter:

  https://github.com/simonw/datasette/blob/4e47a2d894b96854348343374c8e97c9d7055cf6/datasette/facets.py#L228

  That's a bit of a weird thing to expose in the API. Maybe change that to `source` so it can be `metadata` or `request`? That's very slightly less coupled to how the UI works.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1729#issuecomment-1114038259
issue_url: https://api.github.com/repos/simonw/datasette/issues/1729
node_id: IC_kwDOBm6k_c5CZt_z
created_at: 2022-04-30T19:06:03Z
updated_at: 2022-04-30T19:06:03Z
author_association: OWNER
body:
  > but actually the facet results would be better if they were a list rather than a dictionary

  I think `facet_results` in the JSON should match this (used by the HTML) instead:

  https://github.com/simonw/datasette/blob/942411ef946e9a34a2094944d3423cddad27efd3/datasette/views/table.py#L737-L741

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1727#issuecomment-1114058210
issue_url: https://api.github.com/repos/simonw/datasette/issues/1727
node_id: IC_kwDOBm6k_c5CZy3i
created_at: 2022-04-30T21:39:34Z
updated_at: 2022-04-30T21:39:34Z
author_association: OWNER
body:
  Something to consider if I look into subprocesses for parallel query execution:

  https://sqlite.org/howtocorrupt.html#_carrying_an_open_database_connection_across_a_fork_

  >  Do not open an SQLite database connection, then fork(), then try to use that database connection in the child process. All kinds of locking problems will result and you can easily end up with a corrupt database. SQLite is not designed to support that kind of behavior. Any database connection that is used in a child process must be opened in the child process, not inherited from the parent. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1479#issuecomment-1114601882
issue_url: https://api.github.com/repos/simonw/datasette/issues/1479
node_id: IC_kwDOBm6k_c5Cb3ma
created_at: 2022-05-02T08:10:27Z
updated_at: 2022-05-02T11:54:49Z
author_association: NONE
body:
  Also ran into this issue today using `datasette package`. The stack trace takes up my whole PowerShell history, though (recursionerror), but it also concerns the temporary directory.
  Our development machines have a very zealous scanner that appears to insert itself between every call to the filesystem. I suspected that was causing some racing, but this turned out not to be the case: inserting `time.sleep(3)` on line 451 of `datasette/datasette/utils/__init__.py` does not make the problem go away. Commenting out the `tmp.cleanup()` line does.

  The next error I get is docker-specific, so that probably does resolve the Datasette error here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/429#issuecomment-1115196863
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/429
node_id: IC_kwDOCGYnMM5CeI2_
created_at: 2022-05-02T18:03:47Z
updated_at: 2022-05-02T18:52:42Z
author_association: OWNER
body:
  I made a build of this branch and tested it like this: https://pyodide.org/en/stable/console.html

  ```pycon
  >>> import micropip
  >>> await micropip.install("https://s3.amazonaws.com/simonwillison-cors-allowed-public/sqlite_utils-3.26-py3-none-any.whl")
  >>> import sqlite_utils
  >>> db = sqlite_utils.Database(memory=True)
  >>> list(db.query("select 32443 + 55"))
  [{'32443 + 55': 32498}]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1733#issuecomment-1115256318
issue_url: https://api.github.com/repos/simonw/datasette/issues/1733
node_id: IC_kwDOBm6k_c5CeXX-
created_at: 2022-05-02T19:05:55Z
updated_at: 2022-05-02T19:05:55Z
author_association: OWNER
body:
  I released a `click-default-group-wheel` package to solve that dependency issue. I've already upgraded `sqlite-utils` to that, so now you can use that in Pyodide:

  - https://github.com/simonw/sqlite-utils/pull/429

  `python-baseconv` is only used for actor cookie expiration times:

  https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/actor_auth_cookie.py#L16-L20

  Datasette never actually sets that cookie itself - it instead encourages plugins to set it in the authentication documentation here: https://docs.datasette.io/en/0.61.1/authentication.html#including-an-expiry-time

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1733#issuecomment-1115258737
issue_url: https://api.github.com/repos/simonw/datasette/issues/1733
node_id: IC_kwDOBm6k_c5CeX9x
created_at: 2022-05-02T19:08:17Z
updated_at: 2022-05-02T19:08:17Z
author_association: OWNER
body:
  I was going to vendor `baseconv.py`, but then I reconsidered - what if there are plugins out there that expect `import baseconv` to work because they have dependend on Datasette?

  I used https://cs.github.com/ and as far as I can tell there aren't any!

  So I'm going to remove that dependency and work out a smarter way to do this - probably by providing a utility function within Datasette itself.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1734#issuecomment-1115260999
issue_url: https://api.github.com/repos/simonw/datasette/issues/1734
node_id: IC_kwDOBm6k_c5CeYhH
created_at: 2022-05-02T19:10:34Z
updated_at: 2022-05-02T19:10:34Z
author_association: OWNER
body:
  This is actually mostly a documentation thing: here: https://docs.datasette.io/en/0.61.1/authentication.html#including-an-expiry-time

  In the code it's only used in these two places:

  https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/actor_auth_cookie.py#L16-L20

  https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/tests/test_auth.py#L56-L60

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1733#issuecomment-1115262218
issue_url: https://api.github.com/repos/simonw/datasette/issues/1733
node_id: IC_kwDOBm6k_c5CeY0K
created_at: 2022-05-02T19:11:51Z
updated_at: 2022-05-02T19:14:01Z
author_association: OWNER
body:
  Here's the full diff I applied to Datasette to get it fully working in Pyodide:

  https://github.com/simonw/datasette/compare/94a3171b01fde5c52697aeeff052e3ad4bab5391...8af32bc5b03c30b1f7a4a8cc4bd80eb7e2ee7b81

  And as a visible diff:

  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index d269372..6c0c5fc 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -15,7 +15,6 @@ import pkg_resources
   import re
   import secrets
   import sys
  -import threading
   import traceback
   import urllib.parse
   from concurrent import futures
  @@ -26,7 +25,6 @@ from itsdangerous import URLSafeSerializer
   from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader
   from jinja2.environment import Template
   from jinja2.exceptions import TemplateNotFound
  -import uvicorn
   
   from .views.base import DatasetteError, ureg
   from .views.database import DatabaseDownload, DatabaseView
  @@ -813,7 +811,6 @@ class Datasette:
               },
               "datasette": datasette_version,
               "asgi": "3.0",
  -            "uvicorn": uvicorn.__version__,
               "sqlite": {
                   "version": sqlite_version,
                   "fts_versions": fts_versions,
  @@ -854,23 +851,7 @@ class Datasette:
           ]
   
       def _threads(self):
  -        threads = list(threading.enumerate())
  -        d = {
  -            "num_threads": len(threads),
  -            "threads": [
  -                {"name": t.name, "ident": t.ident, "daemon": t.daemon} for t in threads
  -            ],
  -        }
  -        # Only available in Python 3.7+
  -        if hasattr(asyncio, "all_tasks"):
  -            tasks = asyncio.all_tasks()
  -            d.update(
  -                {
  -                    "num_tasks": len(tasks),
  -                    "tasks": [_cleaner_task_str(t) for t in tasks],
  -                }
  -            )
  -        return d
  +        return {"num_threads": 0, "threads": []}
   
       def _actor(self, request):
           return {"actor": request.actor}
  diff --git a/datasette/database.py b/datasette/database.py
  index ba594a8..b50142d 100644
  --- a/datasette/database.py
  +++ b/datasette/database.py
  @@ -4,7 +4,6 @@ from pathlib import Path
   import janus
   import queue
   import sys
  -import threading
   import uuid
   
   from .tracer import trace
  @@ -21,8 +20,6 @@ from .utils import (
   )
   from .inspect import inspect_hash
   
  -connections = threading.local()
  -
   AttachedDatabase = namedtuple("AttachedDatabase", ("seq", "name", "file"))
   
   
  @@ -43,12 +40,12 @@ class Database:
           self.hash = None
           self.cached_size = None
           self._cached_table_counts = None
  -        self._write_thread = None
  -        self._write_queue = None
           if not self.is_mutable and not self.is_memory:
               p = Path(path)
               self.hash = inspect_hash(p)
               self.cached_size = p.stat().st_size
  +        self._read_connection = None
  +        self._write_connection = None
   
       @property
       def cached_table_counts(self):
  @@ -134,60 +131,17 @@ class Database:
           return results
   
       async def execute_write_fn(self, fn, block=True):
  -        task_id = uuid.uuid5(uuid.NAMESPACE_DNS, "datasette.io")
  -        if self._write_queue is None:
  -            self._write_queue = queue.Queue()
  -        if self._write_thread is None:
  -            self._write_thread = threading.Thread(
  -                target=self._execute_writes, daemon=True
  -            )
  -            self._write_thread.start()
  -        reply_queue = janus.Queue()
  -        self._write_queue.put(WriteTask(fn, task_id, reply_queue))
  -        if block:
  -            result = await reply_queue.async_q.get()
  -            if isinstance(result, Exception):
  -                raise result
  -            else:
  -                return result
  -        else:
  -            return task_id
  -
  -    def _execute_writes(self):
  -        # Infinite looping thread that protects the single write connection
  -        # to this database
  -        conn_exception = None
  -        conn = None
  -        try:
  -            conn = self.connect(write=True)
  -            self.ds._prepare_connection(conn, self.name)
  -        except Exception as e:
  -            conn_exception = e
  -        while True:
  -            task = self._write_queue.get()
  -            if conn_exception is not None:
  -                result = conn_exception
  -            else:
  -                try:
  -                    result = task.fn(conn)
  -                except Exception as e:
  -                    sys.stderr.write("{}\n".format(e))
  -                    sys.stderr.flush()
  -                    result = e
  -            task.reply_queue.sync_q.put(result)
  +        # We always treat it as if block=True now
  +        if self._write_connection is None:
  +            self._write_connection = self.connect(write=True)
  +            self.ds._prepare_connection(self._write_connection, self.name)
  +        return fn(self._write_connection)
   
       async def execute_fn(self, fn):
  -        def in_thread():
  -            conn = getattr(connections, self.name, None)
  -            if not conn:
  -                conn = self.connect()
  -                self.ds._prepare_connection(conn, self.name)
  -                setattr(connections, self.name, conn)
  -            return fn(conn)
  -
  -        return await asyncio.get_event_loop().run_in_executor(
  -            self.ds.executor, in_thread
  -        )
  +        if self._read_connection is None:
  +            self._read_connection = self.connect()
  +            self.ds._prepare_connection(self._read_connection, self.name)
  +        return fn(self._read_connection)
   
       async def execute(
           self,
  diff --git a/setup.py b/setup.py
  index 7f0562f..c41669c 100644
  --- a/setup.py
  +++ b/setup.py
  @@ -44,20 +44,20 @@ setup(
       install_requires=[
           "asgiref>=3.2.10,<3.6.0",
           "click>=7.1.1,<8.2.0",
  -        "click-default-group~=1.2.2",
  +        # "click-default-group~=1.2.2",
           "Jinja2>=2.10.3,<3.1.0",
           "hupper~=1.9",
           "httpx>=0.20",
           "pint~=0.9",
           "pluggy>=1.0,<1.1",
  -        "uvicorn~=0.11",
  +        # "uvicorn~=0.11",
           "aiofiles>=0.4,<0.9",
           "janus>=0.6.2,<1.1",
           "asgi-csrf>=0.9",
           "PyYAML>=5.3,<7.0",
           "mergedeep>=1.1.1,<1.4.0",
           "itsdangerous>=1.1,<3.0",
  -        "python-baseconv==1.2.2",
  +        # "python-baseconv==1.2.2",
       ],
       entry_points="""
           [console_scripts]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1733#issuecomment-1115268245
issue_url: https://api.github.com/repos/simonw/datasette/issues/1733
node_id: IC_kwDOBm6k_c5CeaSV
created_at: 2022-05-02T19:18:11Z
updated_at: 2022-05-02T19:18:11Z
author_association: OWNER
body:
  Maybe I can leave `uvicorn` as a dependency? Installing it works OK, it only generates errors when you try to import it:

  ```pycon
  Welcome to the Pyodide terminal emulator 🐍
  Python 3.10.2 (main, Apr  9 2022 20:52:01) on WebAssembly VM
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import micropip
  >>> await micropip.install("uvicorn")
  >>> import uvicorn
  Traceback (most recent call last):
    File "<console>", line 1, in <module>
    File "/lib/python3.10/site-packages/uvicorn/__init__.py", line 1, in <module>
      from uvicorn.config import Config
    File "/lib/python3.10/site-packages/uvicorn/config.py", line 8, in <module>
      import ssl
    File "/lib/python3.10/ssl.py", line 98, in <module>
      import _ssl             # if we can't import it, let the error propagate
  ModuleNotFoundError: No module named '_ssl'
  >>> import ssl
  >>> import uvicorn
  Traceback (most recent call last):
    File "<console>", line 1, in <module>
    File "/lib/python3.10/site-packages/uvicorn/__init__.py", line 2, in <module>
      from uvicorn.main import Server, main, run
    File "/lib/python3.10/site-packages/uvicorn/main.py", line 24, in <module>
      from uvicorn.supervisors import ChangeReload, Multiprocess
    File "/lib/python3.10/site-packages/uvicorn/supervisors/__init__.py", line 3, in <module>
      from uvicorn.supervisors.basereload import BaseReload
    File "/lib/python3.10/site-packages/uvicorn/supervisors/basereload.py", line 12, in <module>
      from uvicorn.subprocess import get_subprocess
    File "/lib/python3.10/site-packages/uvicorn/subprocess.py", line 14, in <module>
      multiprocessing.allow_connection_pickling()
    File "/lib/python3.10/multiprocessing/context.py", line 170, in allow_connection_pickling
      from . import connection
    File "/lib/python3.10/multiprocessing/connection.py", line 21, in <module>
      import _multiprocessing
  ModuleNotFoundError: No module named '_multiprocessing'
  >>> import multiprocessing
  >>> import uvicorn
  Traceback (most recent call last):
    File "<console>", line 1, in <module>
    File "/lib/python3.10/site-packages/uvicorn/__init__.py", line 2, in <module>
      from uvicorn.main import Server, main, run
    File "/lib/python3.10/site-packages/uvicorn/main.py", line 24, in <module>
      from uvicorn.supervisors import ChangeReload, Multiprocess
    File "/lib/python3.10/site-packages/uvicorn/supervisors/__init__.py", line 3, in <module>
      from uvicorn.supervisors.basereload import BaseReload
    File "/lib/python3.10/site-packages/uvicorn/supervisors/basereload.py", line 12, in <module>
      from uvicorn.subprocess import get_subprocess
    File "/lib/python3.10/site-packages/uvicorn/subprocess.py", line 14, in <module>
      multiprocessing.allow_connection_pickling()
    File "/lib/python3.10/multiprocessing/context.py", line 170, in allow_connection_pickling
      from . import connection
    File "/lib/python3.10/multiprocessing/connection.py", line 21, in <module>
      import _multiprocessing
  ModuleNotFoundError: No module named '_multiprocessing'
  >>> 
  ```
  Since the `import ssl` trick fixed the `_ssl` error I was hopeful that `import multiprocessing` could fix the `_multiprocessing` one, but sadly it did not.

  But it looks like i can address this issue just by making `import uvicorn` in `app.py` an optional import.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1733#issuecomment-1115278325
issue_url: https://api.github.com/repos/simonw/datasette/issues/1733
node_id: IC_kwDOBm6k_c5Cecv1
created_at: 2022-05-02T19:29:05Z
updated_at: 2022-05-02T19:29:05Z
author_association: OWNER
body:
  I'm going to add a Datasette setting to disable threading entirely, designed for usage in this particular case.

  I thought about adding a new setting, then I noticed this:

      datasette mydatabase.db --setting num_sql_threads 10

  I'm going to let users set that to `0` to disable threaded execution of SQL queries.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1733#issuecomment-1115288284
issue_url: https://api.github.com/repos/simonw/datasette/issues/1733
node_id: IC_kwDOBm6k_c5CefLc
created_at: 2022-05-02T19:40:33Z
updated_at: 2022-05-02T19:40:33Z
author_association: OWNER
body:
  I'll release this as a `0.62a0` as soon as it's ready, so I can start testing it out in Pyodide for real.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1735#issuecomment-1115301733
issue_url: https://api.github.com/repos/simonw/datasette/issues/1735
node_id: IC_kwDOBm6k_c5Ceidl
created_at: 2022-05-02T19:57:19Z
updated_at: 2022-05-02T19:59:03Z
author_association: OWNER
body:
  This code breaks if that setting is 0:

  https://github.com/simonw/datasette/blob/a29c1277896b6a7905ef5441c42a37bc15f67599/datasette/app.py#L291-L293

  It's used here:

  https://github.com/simonw/datasette/blob/a29c1277896b6a7905ef5441c42a37bc15f67599/datasette/database.py#L188-L190

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1733#issuecomment-1115318303
issue_url: https://api.github.com/repos/simonw/datasette/issues/1733
node_id: IC_kwDOBm6k_c5Cemgf
created_at: 2022-05-02T20:13:36Z
updated_at: 2022-05-02T20:13:36Z
author_association: OWNER
body:
  I got a build from the `pyodide` branch to work!

  ```
  Welcome to the Pyodide terminal emulator 🐍
  Python 3.10.2 (main, Apr  9 2022 20:52:01) on WebAssembly VM
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import micropip
  >>> await micropip.install("https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.62a0-py3-none-any.whl")
  Traceback (most recent call last):
    File "<console>", line 1, in <module>
    File "/lib/python3.10/asyncio/futures.py", line 284, in __await__
      yield self  # This tells Task to wait for completion.
    File "/lib/python3.10/asyncio/tasks.py", line 304, in __wakeup
      future.result()
    File "/lib/python3.10/asyncio/futures.py", line 201, in result
      raise self._exception
    File "/lib/python3.10/asyncio/tasks.py", line 234, in __step
      result = coro.throw(exc)
    File "/lib/python3.10/site-packages/micropip/_micropip.py", line 183, in install
      transaction = await self.gather_requirements(requirements, ctx, keep_going)
    File "/lib/python3.10/site-packages/micropip/_micropip.py", line 173, in gather_requirements
      await gather(*requirement_promises)
    File "/lib/python3.10/asyncio/futures.py", line 284, in __await__
      yield self  # This tells Task to wait for completion.
    File "/lib/python3.10/asyncio/tasks.py", line 304, in __wakeup
      future.result()
    File "/lib/python3.10/asyncio/futures.py", line 201, in result
      raise self._exception
    File "/lib/python3.10/asyncio/tasks.py", line 232, in __step
      result = coro.send(None)
    File "/lib/python3.10/site-packages/micropip/_micropip.py", line 245, in add_requirement
      await self.add_wheel(name, wheel, version, (), ctx, transaction)
    File "/lib/python3.10/site-packages/micropip/_micropip.py", line 316, in add_wheel
      await self.add_requirement(recurs_req, ctx, transaction)
    File "/lib/python3.10/site-packages/micropip/_micropip.py", line 291, in add_requirement
      await self.add_wheel(
    File "/lib/python3.10/site-packages/micropip/_micropip.py", line 316, in add_wheel
      await self.add_requirement(recurs_req, ctx, transaction)
    File "/lib/python3.10/site-packages/micropip/_micropip.py", line 291, in add_requirement
      await self.add_wheel(
    File "/lib/python3.10/site-packages/micropip/_micropip.py", line 316, in add_wheel
      await self.add_requirement(recurs_req, ctx, transaction)
    File "/lib/python3.10/site-packages/micropip/_micropip.py", line 276, in add_requirement
      raise ValueError(
  ValueError: Requested 'h11<0.13,>=0.11', but h11==0.13.0 is already installed
  >>> await micropip.install("https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.62a0-py3-none-any.whl")
  Traceback (most recent call last):
    File "<console>", line 1, in <module>
    File "/lib/python3.10/asyncio/futures.py", line 284, in __await__
      yield self  # This tells Task to wait for completion.
    File "/lib/python3.10/asyncio/tasks.py", line 304, in __wakeup
      future.result()
    File "/lib/python3.10/asyncio/futures.py", line 201, in result
      raise self._exception
    File "/lib/python3.10/asyncio/tasks.py", line 234, in __step
      result = coro.throw(exc)
    File "/lib/python3.10/site-packages/micropip/_micropip.py", line 183, in install
      transaction = await self.gather_requirements(requirements, ctx, keep_going)
    File "/lib/python3.10/site-packages/micropip/_micropip.py", line 173, in gather_requirements
      await gather(*requirement_promises)
    File "/lib/python3.10/asyncio/futures.py", line 284, in __await__
      yield self  # This tells Task to wait for completion.
    File "/lib/python3.10/asyncio/tasks.py", line 304, in __wakeup
      future.result()
    File "/lib/python3.10/asyncio/futures.py", line 201, in result
      raise self._exception
    File "/lib/python3.10/asyncio/tasks.py", line 232, in __step
      result = coro.send(None)
    File "/lib/python3.10/site-packages/micropip/_micropip.py", line 245, in add_requirement
      await self.add_wheel(name, wheel, version, (), ctx, transaction)
    File "/lib/python3.10/site-packages/micropip/_micropip.py", line 316, in add_wheel
      await self.add_requirement(recurs_req, ctx, transaction)
    File "/lib/python3.10/site-packages/micropip/_micropip.py", line 291, in add_requirement
      await self.add_wheel(
    File "/lib/python3.10/site-packages/micropip/_micropip.py", line 316, in add_wheel
      await self.add_requirement(recurs_req, ctx, transaction)
    File "/lib/python3.10/site-packages/micropip/_micropip.py", line 291, in add_requirement
      await self.add_wheel(
    File "/lib/python3.10/site-packages/micropip/_micropip.py", line 316, in add_wheel
      await self.add_requirement(recurs_req, ctx, transaction)
    File "/lib/python3.10/site-packages/micropip/_micropip.py", line 276, in add_requirement
      raise ValueError(
  ValueError: Requested 'h11<0.13,>=0.11', but h11==0.13.0 is already installed
  >>> await micropip.install("h11==0.12")
  >>> await micropip.install("https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.62a0-py3-none-any.whl")
  >>> import datasette
  >>> from datasette.app import Datasette
  Traceback (most recent call last):
    File "<console>", line 1, in <module>
    File "/lib/python3.10/site-packages/datasette/app.py", line 9, in <module>
      import httpx
    File "/lib/python3.10/site-packages/httpx/__init__.py", line 2, in <module>
      from ._api import delete, get, head, options, patch, post, put, request, stream
    File "/lib/python3.10/site-packages/httpx/_api.py", line 4, in <module>
      from ._client import Client
    File "/lib/python3.10/site-packages/httpx/_client.py", line 9, in <module>
      from ._auth import Auth, BasicAuth, FunctionAuth
    File "/lib/python3.10/site-packages/httpx/_auth.py", line 10, in <module>
      from ._models import Request, Response
    File "/lib/python3.10/site-packages/httpx/_models.py", line 16, in <module>
      from ._content import ByteStream, UnattachedStream, encode_request, encode_response
    File "/lib/python3.10/site-packages/httpx/_content.py", line 17, in <module>
      from ._multipart import MultipartStream
    File "/lib/python3.10/site-packages/httpx/_multipart.py", line 7, in <module>
      from ._types import (
    File "/lib/python3.10/site-packages/httpx/_types.py", line 5, in <module>
      import ssl
    File "/lib/python3.10/ssl.py", line 98, in <module>
      import _ssl             # if we can't import it, let the error propagate
  ModuleNotFoundError: No module named '_ssl'
  >>> import ssl
  >>> from datasette.app import Datasette
  Traceback (most recent call last):
    File "<console>", line 1, in <module>
    File "/lib/python3.10/site-packages/datasette/app.py", line 14, in <module>
      import pkg_resources
  ModuleNotFoundError: No module named 'pkg_resources'
  >>> import setuptools
  >>> from datasette.app import Datasette
  >>> ds = Datasette(memory=True)
  >>> ds
  <datasette.app.Datasette object at 0x1cc4fb8>
  >>> await ds.client.get("/")
  Traceback (most recent call last):
    File "/lib/python3.10/site-packages/datasette/app.py", line 1268, in route_path
      response = await view(request, send)
    File "/lib/python3.10/site-packages/datasette/views/base.py", line 134, in view
      return await self.dispatch_request(request)
    File "/lib/python3.10/site-packages/datasette/views/base.py", line 89, in dispatch_request
      await self.ds.refresh_schemas()
    File "/lib/python3.10/site-packages/datasette/app.py", line 353, in refresh_schemas
      await self._refresh_schemas()
    File "/lib/python3.10/site-packages/datasette/app.py", line 358, in _refresh_schemas
      await init_internal_db(internal_db)
    File "/lib/python3.10/site-packages/datasette/utils/internal_db.py", line 65, in init_internal_db
      await db.execute_write_script(create_tables_sql)
    File "/lib/python3.10/site-packages/datasette/database.py", line 116, in execute_write_script
      results = await self.execute_write_fn(_inner, block=block)
    File "/lib/python3.10/site-packages/datasette/database.py", line 155, in execute_write_fn
      self._write_thread.start()
    File "/lib/python3.10/threading.py", line 928, in start
      _start_new_thread(self._bootstrap, ())
  RuntimeError: can't start new thread
  <Response [500 Internal Server Error]>
  >>> ds = Datasette(memory=True, settings={"num_sql_threads": 0})
  >>> await ds.client.get("/")
  <Response [200 OK]>
  >>> (await ds.client.get("/")).text
  '<!DOCTYPE html>\n<html>\n<head>\n    <title>Datasette: _memory</title>\n    <link rel="stylesheet" href="/-/static/app.css
  ?cead5a">\n    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">\n\n<link rel="alterna
  te" type="application/json+datasette" href="http://localhost/.json"></head>\n<body class="index">\n<div class="not-footer">
  \n<header><nav>\n    \n    \n</nav></header>\n\n\n\n    \n\n\n\n<section class="content">\n\n<h1>Datasette</h1>\n\n\n\n\n\n
      <h2 
  <long output truncated>
  r detailsClickedWithin = null;\n    while (target && target.tagName != \'DETAILS\') {\n        target = target.parentNode;\
  n    }\n    if (target && target.tagName == \'DETAILS\') {\n        detailsClickedWithin = target;\n    }\n    Array.from(d
  ocument.getElementsByTagName(\'details\')).filter(\n        (details) => details.open && details != detailsClickedWithin\n 
     ).forEach(details => details.open = false);\n});\n</script>\n\n\n\n<!-- Templates considered: *index.html -->\n</body>\n
  </html>'
  >>> 
  ```

  That `ValueError: Requested 'h11<0.13,>=0.11', but h11==0.13.0 is already installed` error is annoying. I assume it's a `uvicorn` dependency clash of some sort, because I wasn't getting that when I removed `uvicorn` as a dependency.

  I can avoid it by running this first though:

      await micropip.install("h11==0.12")

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1733#issuecomment-1115404729
issue_url: https://api.github.com/repos/simonw/datasette/issues/1733
node_id: IC_kwDOBm6k_c5Ce7m5
created_at: 2022-05-02T21:49:01Z
updated_at: 2022-05-02T21:49:38Z
author_association: OWNER
body:
  That alpha release works!

  https://pyodide.org/en/stable/console.html

  ```pycon
  Welcome to the Pyodide terminal emulator 🐍
  Python 3.10.2 (main, Apr  9 2022 20:52:01) on WebAssembly VM
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import micropip
  >>> await micropip.install("datasette==0.62a0")
  >>> import ssl
  >>> import setuptools
  >>> from datasette.app import Datasette
  >>> ds = Datasette(memory=True, settings={"num_sql_threads": 0})
  >>> await ds.client.get("/.json")
  <Response [200 OK]>
  >>> (await ds.client.get("/.json")).json()
  {'_memory': {'name': '_memory', 'hash': None, 'color': 'a6c7b9', 'path': '/_memory', 'tables_and_views_truncated': [], 'tab
  les_and_views_more': False, 'tables_count': 0, 'table_rows_sum': 0, 'show_table_row_counts': False, 'hidden_table_rows_sum'
  : 0, 'hidden_tables_count': 0, 'views_count': 0, 'private': False}}
  >>> 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1737#issuecomment-1115462720
issue_url: https://api.github.com/repos/simonw/datasette/issues/1737
node_id: IC_kwDOBm6k_c5CfJxA
created_at: 2022-05-02T23:25:03Z
updated_at: 2022-05-02T23:25:03Z
author_association: OWNER
body:
  Here's a script that seems to work. It builds the wheel, starts a Python web server that serves the wheel, runs a test with `shot-scraper` and then shuts down the server again.

  ```bash
  #!/bin/bash

  # Build the wheel
  python3 -m build

  # Find name of wheel
  wheel=$(basename $(ls dist/*.whl))
  # strip off the dist/


  # Create a blank index page
  echo '
  <script src="https://cdn.jsdelivr.net/pyodide/v0.20.0/full/pyodide.js"></script>
  ' > dist/index.html

  # Run a server for that dist/ folder
  cd dist
  python3 -m http.server 8529 &
  cd ..

  shot-scraper javascript http://localhost:8529/ "
  async () => {
    let pyodide = await loadPyodide();
    await pyodide.loadPackage(['micropip', 'ssl', 'setuptools']);
    let output = await pyodide.runPythonAsync(\`
      import micropip
      await micropip.install('h11==0.12.0')
      await micropip.install('http://localhost:8529/$wheel')
      import ssl
      import setuptools
      from datasette.app import Datasette
      ds = Datasette(memory=True, settings={'num_sql_threads': 0})
      (await ds.client.get('/_memory.json?sql=select+55+as+itworks&_shape=array')).text
    \`);
    if (JSON.parse(output)[0].itworks != 55) {
      throw 'Got ' + output + ', expected itworks: 55';
    }
    return 'Test passed!';
  }
  "

  # Shut down the server
  pkill -f 'http.server 8529'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1737#issuecomment-1115464097
issue_url: https://api.github.com/repos/simonw/datasette/issues/1737
node_id: IC_kwDOBm6k_c5CfKGh
created_at: 2022-05-02T23:27:40Z
updated_at: 2022-05-02T23:27:40Z
author_association: OWNER
body:
  I'm going to start off by running this manually - I may run it on every commit once this is all a little bit more stable.

  I can base the workflow on https://github.com/simonw/scrape-hacker-news-by-domain/blob/main/.github/workflows/scrape.yml

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1737#issuecomment-1115468193
issue_url: https://api.github.com/repos/simonw/datasette/issues/1737
node_id: IC_kwDOBm6k_c5CfLGh
created_at: 2022-05-02T23:35:26Z
updated_at: 2022-05-02T23:35:26Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/runs/6265915080?check_suite_focus=true failed but looks like it passed because I forgot to use `set -e` at the start of the bash script.

  It failed because it didn't have `build` available.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1737#issuecomment-1115470180
issue_url: https://api.github.com/repos/simonw/datasette/issues/1737
node_id: IC_kwDOBm6k_c5CfLlk
created_at: 2022-05-02T23:39:29Z
updated_at: 2022-05-02T23:39:29Z
author_association: OWNER
body:
  Test ran in 38 seconds and passed! https://github.com/simonw/datasette/runs/6265954274?check_suite_focus=true

  I'm going to have it run on every commit and PR.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1732#issuecomment-1115542067
issue_url: https://api.github.com/repos/simonw/datasette/issues/1732
node_id: IC_kwDOBm6k_c5CfdIz
created_at: 2022-05-03T01:50:44Z
updated_at: 2022-05-03T01:50:44Z
author_association: NONE
body:
  I haven’t set one up unfortunately. My time is very limited because we just had a baby. 

  On Mon, May 2, 2022, at 6:42 PM, Simon Willison wrote:
  > 
  > 
  > Thanks, this definitely sounds like a bug. Do you have simple steps to reproduce this?
  > 
  > 
  > —
  > Reply to this email directly, view it on GitHub <https://github.com/simonw/datasette/issues/1732#issuecomment-1115533820>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAM3KIY5L6FENZ22XANTHDVICAAXANCNFSM5UYOTKQA>.
  > You are receiving this because you authored the thread.Message ID: ***@***.***>
  > 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1739#issuecomment-1115760104
issue_url: https://api.github.com/repos/simonw/datasette/issues/1739
node_id: IC_kwDOBm6k_c5CgSXo
created_at: 2022-05-03T05:50:19Z
updated_at: 2022-05-03T05:50:19Z
author_association: OWNER
body:
  Here's how Starlette does it: https://github.com/encode/starlette/blob/830f3486537916bae6b46948ff922adc14a22b7c/starlette/staticfiles.py#L213

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1739#issuecomment-1116178727
issue_url: https://api.github.com/repos/simonw/datasette/issues/1739
node_id: IC_kwDOBm6k_c5Ch4kn
created_at: 2022-05-03T14:38:46Z
updated_at: 2022-05-03T14:38:46Z
author_association: OWNER
body:
  Reminded myself how this works by reviewing `conditional-get`: https://github.com/simonw/conditional-get/blob/db6dfec0a296080aaf68fcd80e55fb3f0714e738/conditional_get/cli.py#L33-L52

  Simply add a `If-None-Match: last-known-etag` header to the request and check that the response is a status 304 with an empty body.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1739#issuecomment-1116180599
issue_url: https://api.github.com/repos/simonw/datasette/issues/1739
node_id: IC_kwDOBm6k_c5Ch5B3
created_at: 2022-05-03T14:40:32Z
updated_at: 2022-05-03T14:40:32Z
author_association: OWNER
body:
  Database downloads are served here: https://github.com/simonw/datasette/blob/d60f163528f466b1127b2935c3b6869c34fd6545/datasette/views/database.py#L186-L192

  Here's `AsgiFileDownload`: https://github.com/simonw/datasette/blob/d60f163528f466b1127b2935c3b6869c34fd6545/datasette/utils/asgi.py#L410-L430

  I can add an `etag=` parameter to that and populate it with `db.hash`, if it is populated (which it always should be for immutable databases that can be downloaded).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1739#issuecomment-1116183369
issue_url: https://api.github.com/repos/simonw/datasette/issues/1739
node_id: IC_kwDOBm6k_c5Ch5tJ
created_at: 2022-05-03T14:43:14Z
updated_at: 2022-05-03T14:43:14Z
author_association: OWNER
body:
  Relevant tests start here: https://github.com/simonw/datasette/blob/d60f163528f466b1127b2935c3b6869c34fd6545/tests/test_html.py#L395

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1739#issuecomment-1116215371
issue_url: https://api.github.com/repos/simonw/datasette/issues/1739
node_id: IC_kwDOBm6k_c5CiBhL
created_at: 2022-05-03T15:12:16Z
updated_at: 2022-05-03T15:12:16Z
author_association: OWNER
body:
  That worked - both DBs are 304 for me now on a subsequent load of the page:

  <img width="587" alt="CleanShot 2022-05-03 at 08 11 05@2x" src="https://user-images.githubusercontent.com/9599/166481669-9570d225-78d4-461b-88c8-044f02b68a64.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/430#issuecomment-1116336340
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/430
node_id: IC_kwDOCGYnMM5CifDU
created_at: 2022-05-03T17:03:31Z
updated_at: 2022-05-03T17:03:31Z
author_association: NONE
body:
  So, the good news is that it appears that setting one of those PRAGMA statements fixed the issue of `table.extract()` method call on this large database completing (that I described above.) The bad news is that I'm not sure which one!

  I wonder if it's something system / environment specific about SQLite, or maybe something else going on.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1116684581
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/416
node_id: IC_kwDOCGYnMM5Cj0El
created_at: 2022-05-03T21:36:49Z
updated_at: 2022-05-03T21:36:49Z
author_association: NONE
body:
  Thanks for addressing this @simonw! However, I just reinstalled sqlite-utils 3.26.1 and get an `ParserError: Unknown string format: None`:
  ```
  sqlite-utils --version
  sqlite-utils, version 3.26.1
  ```
  ```
  sqlite-utils convert idfpr.db license "Original Issue Date" "r.parsedate(value)"
  Traceback (most recent call last):
    File "/home/matt/.local/lib/python3.9/site-packages/sqlite_utils/db.py", line 2514, in convert_value
      return fn(v)
    File "<string>", line 2, in fn
    File "/home/matt/.local/lib/python3.9/site-packages/sqlite_utils/recipes.py", line 19, in parsedate
      parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst)
    File "/usr/lib/python3/dist-packages/dateutil/parser/_parser.py", line 1374, in parse
      return DEFAULTPARSER.parse(timestr, **kwargs)
    File "/usr/lib/python3/dist-packages/dateutil/parser/_parser.py", line 649, in parse
      raise ParserError("Unknown string format: %s", timestr)
  dateutil.parser._parser.ParserError: Unknown string format: None
  Traceback (most recent call last):
    File "/home/matt/.local/bin/sqlite-utils", line 8, in <module>
      sys.exit(cli())
    File "/usr/lib/python3/dist-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/usr/lib/python3/dist-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/usr/lib/python3/dist-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/lib/python3/dist-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/usr/lib/python3/dist-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/home/matt/.local/lib/python3.9/site-packages/sqlite_utils/cli.py", line 2707, in convert
      db[table].convert(
    File "/home/matt/.local/lib/python3.9/site-packages/sqlite_utils/db.py", line 2530, in convert
      self.db.execute(sql, where_args or [])
    File "/home/matt/.local/lib/python3.9/site-packages/sqlite_utils/db.py", line 463, in execute
      return self.conn.execute(sql, parameters)
  sqlite3.OperationalError: user-defined function raised exception
  ```
  I definitely have some invalid data in the db. Happy to send a copy if it's helpful.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1298#issuecomment-1125083348
issue_url: https://api.github.com/repos/simonw/datasette/issues/1298
node_id: IC_kwDOBm6k_c5DD2jU
created_at: 2022-05-12T14:43:51Z
updated_at: 2022-05-12T14:43:51Z
author_association: NONE
body:
  user report: I found this issue because the first time I tried to use datasette for real, I displayed a large table, and thought there was no horizontal scroll bar at all. I didn't even consider that I had to scroll all the way to the end of the page to find it.

  Just chipping in to say that this confused me, and I didn't even find the scroll bar until after I saw this issue. I don't know what the right answer is, but IMO the UI should suggest to the user that there is a way to view the data that's hidden to the right.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/741#issuecomment-1125342229
issue_url: https://api.github.com/repos/simonw/datasette/issues/741
node_id: IC_kwDOBm6k_c5DE1wV
created_at: 2022-05-12T19:21:16Z
updated_at: 2022-05-12T19:21:16Z
author_association: CONTRIBUTOR
body:
  Came here to check if this had been flagged already. Was helping a colleague get something on Cloud Run and had to dig to find `--extra-options="--setting sql_time_limit_ms 2500"`.

  If I get some time next week, maybe I'll try to tackle it. Would definitely make things easier to be able to do something like this:

  ```sh
  datasette publish cloudrun something.db --setting sql_time_limit_ms 2500
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1742#issuecomment-1128033018
issue_url: https://api.github.com/repos/simonw/datasette/issues/1742
node_id: IC_kwDOBm6k_c5DPGr6
created_at: 2022-05-16T19:06:38Z
updated_at: 2022-05-16T19:06:38Z
author_association: OWNER
body:
  The same URL with `.json` instead works fine: https://calands.datasettes.com/calands/CPAD_2020a_SuperUnits.json?_sort=id&id__exact=4&_labels=on&_trace=1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1742#issuecomment-1128049716
issue_url: https://api.github.com/repos/simonw/datasette/issues/1742
node_id: IC_kwDOBm6k_c5DPKw0
created_at: 2022-05-16T19:24:44Z
updated_at: 2022-05-16T19:24:44Z
author_association: CONTRIBUTOR
body:
  Where is `_trace` getting injected? And is it something a plugin should be able to handle? (If it is, I guess I should handle it in this case.)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1742#issuecomment-1128052948
issue_url: https://api.github.com/repos/simonw/datasette/issues/1742
node_id: IC_kwDOBm6k_c5DPLjU
created_at: 2022-05-16T19:28:31Z
updated_at: 2022-05-16T19:28:31Z
author_association: OWNER
body:
  The trace mechanism is a bit gnarly - it's actually done by some ASGI middleware I wrote, so I'm pretty sure the bug is in there somewhere: https://github.com/simonw/datasette/blob/280ff372ab30df244f6c54f6f3002da57334b3d7/datasette/tracer.py#L73

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1742#issuecomment-1128064864
issue_url: https://api.github.com/repos/simonw/datasette/issues/1742
node_id: IC_kwDOBm6k_c5DPOdg
created_at: 2022-05-16T19:42:13Z
updated_at: 2022-05-16T19:42:13Z
author_association: CONTRIBUTOR
body:
  Just to add a wrinkle here, this loads fine: https://alltheplaces-datasette.fly.dev/alltheplaces/places.geojson?_trace=1

  But also, this doesn't add any trace data: https://alltheplaces-datasette.fly.dev/alltheplaces/places.json?_trace=1

  What am I missing?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1744#issuecomment-1129184908
issue_url: https://api.github.com/repos/simonw/datasette/issues/1744
node_id: IC_kwDOBm6k_c5DTf6M
created_at: 2022-05-17T18:25:57Z
updated_at: 2022-05-17T18:25:57Z
author_association: OWNER
body:
  I knocked out a quick prototype of this and it worked!

      datasette ~/Library/Application\ Support/Google/Chrome/Default/History --nolock

  Here's the prototype diff:

  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index b7b8437..f43700d 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -213,6 +213,7 @@ class Datasette:
           config_dir=None,
           pdb=False,
           crossdb=False,
  +        nolock=False,
       ):
           assert config_dir is None or isinstance(
               config_dir, Path
  @@ -238,6 +239,7 @@ class Datasette:
           self.databases = collections.OrderedDict()
           self._refresh_schemas_lock = asyncio.Lock()
           self.crossdb = crossdb
  +        self.nolock = nolock
           if memory or crossdb or not self.files:
               self.add_database(Database(self, is_memory=True), name="_memory")
           # memory_name is a random string so that each Datasette instance gets its own
  diff --git a/datasette/cli.py b/datasette/cli.py
  index 3c6e1b2..7e44665 100644
  --- a/datasette/cli.py
  +++ b/datasette/cli.py
  @@ -452,6 +452,11 @@ def uninstall(packages, yes):
       is_flag=True,
       help="Enable cross-database joins using the /_memory database",
   )
  +@click.option(
  +    "--nolock",
  +    is_flag=True,
  +    help="Ignore locking and open locked files in read-only mode",
  +)
   @click.option(
       "--ssl-keyfile",
       help="SSL key file",
  @@ -486,6 +491,7 @@ def serve(
       open_browser,
       create,
       crossdb,
  +    nolock,
       ssl_keyfile,
       ssl_certfile,
       return_instance=False,
  @@ -545,6 +551,7 @@ def serve(
           version_note=version_note,
           pdb=pdb,
           crossdb=crossdb,
  +        nolock=nolock,
       )
   
       # if files is a single directory, use that as config_dir=
  diff --git a/datasette/database.py b/datasette/database.py
  index 44d3266..fa55804 100644
  --- a/datasette/database.py
  +++ b/datasette/database.py
  @@ -89,6 +89,8 @@ class Database:
           # mode=ro or immutable=1?
           if self.is_mutable:
               qs = "?mode=ro"
  +            if self.ds.nolock:
  +                qs += "&nolock=1"
           else:
               qs = "?immutable=1"
           assert not (write and not self.is_mutable)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1744#issuecomment-1129241283
issue_url: https://api.github.com/repos/simonw/datasette/issues/1744
node_id: IC_kwDOBm6k_c5DTtrD
created_at: 2022-05-17T19:32:35Z
updated_at: 2022-05-17T19:32:35Z
author_association: OWNER
body:
  I tried writing a test like this:

  ```python
  @pytest.mark.parametrize("locked", (True, False))
  def test_locked_sqlite_db(tmp_path_factory, locked):
      dir = tmp_path_factory.mktemp("test_locked_sqlite_db")
      test_db = str(dir / "test.db")
      sqlite3.connect(test_db).execute("create table t (id integer primary key)")
      if locked:
          fp = open(test_db, "w")
          fcntl.lockf(fp.fileno(), fcntl.LOCK_EX)
      runner = CliRunner()
      result = runner.invoke(
          cli,
          [
              "serve",
              "--memory",
              "--get",
              "/test",
          ],
          catch_exceptions=False,
      )
  ```
  But it didn't work, because the test runs in the same process - so taking an exclusive lock on that file didn't cause an error when the test later tried to access it via Datasette!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1744#issuecomment-1129241873
issue_url: https://api.github.com/repos/simonw/datasette/issues/1744
node_id: IC_kwDOBm6k_c5DTt0R
created_at: 2022-05-17T19:33:16Z
updated_at: 2022-05-17T19:33:16Z
author_association: OWNER
body:
  I'm going to skip adding a test for this - the test logic would have to be pretty convoluted to exercise it properly, and it's a pretty minor and low-risk feature in the scheme of things.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1744#issuecomment-1129243427
issue_url: https://api.github.com/repos/simonw/datasette/issues/1744
node_id: IC_kwDOBm6k_c5DTuMj
created_at: 2022-05-17T19:35:02Z
updated_at: 2022-05-17T19:35:02Z
author_association: OWNER
body:
  One thing to note is that the `datasette-copy-to-memory` plugin broke with a locked file, because it does this: https://github.com/simonw/datasette-copy-to-memory/blob/d541c18a78ae6f707a8f9b1e7fc4c020a9f68f2e/datasette_copy_to_memory/__init__.py#L27
  ```python
  tmp.execute("ATTACH DATABASE ? AS _copy_from", [db.path])
  ```
  That would need to use a URI filename too for it to work with locked files.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1744#issuecomment-1129251699
issue_url: https://api.github.com/repos/simonw/datasette/issues/1744
node_id: IC_kwDOBm6k_c5DTwNz
created_at: 2022-05-17T19:44:47Z
updated_at: 2022-05-17T19:46:38Z
author_association: OWNER
body:
  Updated docs: https://docs.datasette.io/en/latest/getting_started.html#using-datasette-on-your-own-computer and https://docs.datasette.io/en/latest/cli-reference.html#datasette-serve-help

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/425#issuecomment-1129332959
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/425
node_id: IC_kwDOCGYnMM5DUEDf
created_at: 2022-05-17T21:27:02Z
updated_at: 2022-05-17T21:27:02Z
author_association: NONE
body:
  Hi, I'm trying to deploy my site using elasticbeanstalk and I keep getting this same error :
  deterministic=True requires SQLite 3.8.3 or higher

  I saw your previous solution that involves editing sqlite-utils/sqlite_utils/db.py file, but I'm curious as to how that will work in production.

reactions:
  {"total_count": 5, "+1": 5, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/1746#issuecomment-1133210032
issue_url: https://api.github.com/repos/simonw/datasette/issues/1746
node_id: IC_kwDOBm6k_c5Di2mw
created_at: 2022-05-20T18:48:17Z
updated_at: 2022-05-20T18:48:17Z
author_association: OWNER
body:
  A couple of changes I want to make. First, I don't really like the way Furo keeps the in-page titles in a separate menu on the right rather than expanding them on the left.

  I like this:

  ![CleanShot 2022-05-20 at 11 43 33@2x](https://user-images.githubusercontent.com/9599/169592611-ac0f9bd2-ff99-49b6-88d3-92dace9d85a6.png)

  Furo wants to do this instead:

  <img width="1100" alt="image" src="https://user-images.githubusercontent.com/9599/169592991-979865af-511b-43de-bb6c-13bc24e2c88e.png">

  I also still want to include those inline tables of contents on the two pages that have them:

  - https://docs.datasette.io/en/stable/installation.html
  - https://docs.datasette.io/en/stable/plugin_hooks.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1746#issuecomment-1133210651
issue_url: https://api.github.com/repos/simonw/datasette/issues/1746
node_id: IC_kwDOBm6k_c5Di2wb
created_at: 2022-05-20T18:49:11Z
updated_at: 2022-05-20T18:49:11Z
author_association: OWNER
body:
  I found a workaround for the no-longer-nested left hand navigation: drop this into `_templates/sidebar/navigation.html`:
  ```html+jinja
  <div class="sidebar-tree">
    {{ toctree(
      collapse=True,
      titles_only=False,
      maxdepth=3,
      includehidden=True,
  ) }}
  </div>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1746#issuecomment-1133210942
issue_url: https://api.github.com/repos/simonw/datasette/issues/1746
node_id: IC_kwDOBm6k_c5Di20-
created_at: 2022-05-20T18:49:40Z
updated_at: 2022-05-20T18:49:40Z
author_association: OWNER
body:
  And for those local table of contents, do this:

  ```rst
  .. contents::
     :local:
     :class: this-will-duplicate-information-and-it-is-still-useful-here
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1746#issuecomment-1133215684
issue_url: https://api.github.com/repos/simonw/datasette/issues/1746
node_id: IC_kwDOBm6k_c5Di3_E
created_at: 2022-05-20T18:56:29Z
updated_at: 2022-05-20T18:56:29Z
author_association: OWNER
body:
  One other problem: in dark mode the Datasette logo looks bad:

  <img width="698" alt="image" src="https://user-images.githubusercontent.com/9599/169594178-6a779d9b-1388-4ab9-837f-4b8bfbfc6011.png">

  This helps a bit:

  ```css
  .sidebar-logo-container {
    background-color: white;
    padding: 5px;
    opacity: 0.6;
  }
  ```

  <img width="789" alt="image" src="https://user-images.githubusercontent.com/9599/169594107-9c153bee-e2e5-4ba1-90de-ecc0a853e608.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1746#issuecomment-1133217219
issue_url: https://api.github.com/repos/simonw/datasette/issues/1746
node_id: IC_kwDOBm6k_c5Di4XD
created_at: 2022-05-20T18:58:54Z
updated_at: 2022-05-20T18:58:54Z
author_association: OWNER
body:
  Need to address other customizations I've made in https://github.com/simonw/datasette/blob/0.62a0/docs/_templates/layout.html - such as Plausible analytics and some custom JavaScript.

  https://github.com/simonw/datasette/blob/943aa2e1f7341cb51e60332cde46bde650c64217/docs/_templates/layout.html#L1-L61

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1153#issuecomment-1133222848
issue_url: https://api.github.com/repos/simonw/datasette/issues/1153
node_id: IC_kwDOBm6k_c5Di5vA
created_at: 2022-05-20T19:07:10Z
updated_at: 2022-05-20T19:07:10Z
author_association: OWNER
body:
  I could use https://github.com/pradyunsg/sphinx-inline-tabs for this - recommended by https://pradyunsg.me/furo/recommendations/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1748#issuecomment-1133232301
issue_url: https://api.github.com/repos/simonw/datasette/issues/1748
node_id: IC_kwDOBm6k_c5Di8Ct
created_at: 2022-05-20T19:15:00Z
updated_at: 2022-05-20T19:15:00Z
author_association: OWNER
body:
  Now live on https://docs.datasette.io/en/latest/testing_plugins.html

  ![copy](https://user-images.githubusercontent.com/9599/169596586-396eb6c7-ef5a-405a-bb21-348499478d9b.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1746#issuecomment-1133242063
issue_url: https://api.github.com/repos/simonw/datasette/issues/1746
node_id: IC_kwDOBm6k_c5Di-bP
created_at: 2022-05-20T19:22:49Z
updated_at: 2022-05-20T19:22:49Z
author_association: OWNER
body:
  I have some custom CSS in this file:

  https://github.com/simonw/datasette/blob/1465fea4798599eccfe7e8f012bd8d9adfac3039/docs/_static/css/custom.css#L1-L7

  I tested and the `overflow-wrap: anywhere` is still needed for this fix:
  - #828

  The `.wy-side-nav-search` bit is no longer needed with the new theme.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1746#issuecomment-1133246791
issue_url: https://api.github.com/repos/simonw/datasette/issues/1746
node_id: IC_kwDOBm6k_c5Di_lH
created_at: 2022-05-20T19:26:49Z
updated_at: 2022-05-20T19:26:49Z
author_association: OWNER
body:
  Putting this in the `css/custom.css` file seems to work for fixing that logo problem:

  ```css
  body[data-theme="dark"] .sidebar-logo-container {
      background-color: white;
      padding: 5px;
      opacity: 0.6;
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1746#issuecomment-1133250151
issue_url: https://api.github.com/repos/simonw/datasette/issues/1746
node_id: IC_kwDOBm6k_c5DjAZn
created_at: 2022-05-20T19:29:37Z
updated_at: 2022-05-20T19:29:37Z
author_association: OWNER
body:
  I want the Datasette logo in the sidebar to link to https://datasette.io/

  Looks like I can do that by dropping in my own `sidebar/brand.html` template based on this:

  https://github.com/pradyunsg/furo/blob/0c2acbbd23f8146dd0ae50a2ba57258c1f63ea9f/src/furo/theme/furo/sidebar/brand.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1746#issuecomment-1133252598
issue_url: https://api.github.com/repos/simonw/datasette/issues/1746
node_id: IC_kwDOBm6k_c5DjA_2
created_at: 2022-05-20T19:31:30Z
updated_at: 2022-05-20T19:31:30Z
author_association: OWNER
body:
  I'd also like to bring back this stable / latest / version indicator:

  ![CleanShot 2022-05-20 at 12 30 49@2x](https://user-images.githubusercontent.com/9599/169598732-e2093ec1-7eaf-40dd-acfa-1a7c31091ff1.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1746#issuecomment-1133254599
issue_url: https://api.github.com/repos/simonw/datasette/issues/1746
node_id: IC_kwDOBm6k_c5DjBfH
created_at: 2022-05-20T19:33:08Z
updated_at: 2022-05-20T19:33:08Z
author_association: OWNER
body:
  Actually maybe I don't? I just noticed that on other pages on https://docs.datasette.io/en/stable/installation.html the only way to get back to that useful table of context / index page at https://docs.datasette.io/en/stable/index.html is by clicking the tiny house icon. Can I do better or should I have the logo do that?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1153#issuecomment-1133267290
issue_url: https://api.github.com/repos/simonw/datasette/issues/1153
node_id: IC_kwDOBm6k_c5DjEla
created_at: 2022-05-20T19:44:05Z
updated_at: 2022-05-20T19:50:58Z
author_association: OWNER
body:
  Undocumented Sphinx feature: you can add extra classes to a code example like this:

  ```
  .. code-block:: json
     :class: metadata-json

      {
          "databases": {
              "russian-ads": {
                  "tables": {
                      "display_ads": {
                          "fts_table": "ads_fts",
                          "fts_pk": "id",
                          "searchmode": "raw"
                      }
                  }
              }
          }
      }
  ```
  https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#directive-code-block doesn't mention this.

  Filed an issue about the lack of documentation here:
  - https://github.com/sphinx-doc/sphinx/issues/10461

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1746#issuecomment-1133288501
issue_url: https://api.github.com/repos/simonw/datasette/issues/1746
node_id: IC_kwDOBm6k_c5DjJw1
created_at: 2022-05-20T20:00:17Z
updated_at: 2022-05-20T20:00:17Z
author_association: OWNER
body:
  Here's a TIL from when I first customized the `layout.html` template: https://til.simonwillison.net/readthedocs/custom-sphinx-templates

  Note that Furo doesn't use `layout.html` so I need to completely change how I did that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1746#issuecomment-1133299417
issue_url: https://api.github.com/repos/simonw/datasette/issues/1746
node_id: IC_kwDOBm6k_c5DjMbZ
created_at: 2022-05-20T20:05:34Z
updated_at: 2022-05-20T20:05:34Z
author_association: OWNER
body:
  I can't get that thing that displays the version working.

  https://github.com/readthedocs/sphinx_rtd_theme/blob/9264091087620d421b0804c00937b00980ac3916/sphinx_rtd_theme/layout.html#L154 is where`sphinx_rtd_theme` implements it - `{%- if theme_display_version %}` - but I can't find where that variable is first set, from searching both that theme and Sphinx itself!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1746#issuecomment-1133309452
issue_url: https://api.github.com/repos/simonw/datasette/issues/1746
node_id: IC_kwDOBm6k_c5DjO4M
created_at: 2022-05-20T20:10:36Z
updated_at: 2022-05-20T20:10:36Z
author_association: OWNER
body:
  Weird, I cannot figure out this `theme_display_version` thing - I even tried this:
  ```bash
  cd /tmp
  mkdir s
  cd s
  python3 -m venv venv
  source venv/bin/activate
  pip install sphinx_rtd_theme
  cd venv
  rg theme_display_version
  ```
  And got just that one reference:
  ```
  lib/python3.10/site-packages/sphinx_rtd_theme/layout.html
  154:          {%- if theme_display_version %}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1746#issuecomment-1133310253
issue_url: https://api.github.com/repos/simonw/datasette/issues/1746
node_id: IC_kwDOBm6k_c5DjPEt
created_at: 2022-05-20T20:11:00Z
updated_at: 2022-05-20T20:11:00Z
author_association: OWNER
body:
  Oh but `rg display_version` is a lot more interesting:
  ```
  lib/python3.10/site-packages/sphinx/builders/html/__init__.py:from sphinx import __display_version__, package_dir
  lib/python3.10/site-packages/sphinx/builders/html/__init__.py:            'sphinx_version': __display_version__,
  lib/python3.10/site-packages/sphinx/application.py:        logger.info(bold(__('Running Sphinx v%s') % sphinx.__display_version__))
  lib/python3.10/site-packages/sphinx/application.py:        if self.config.needs_sphinx and self.config.needs_sphinx > sphinx.__display_version__:
  lib/python3.10/site-packages/sphinx/application.py:        if version > sphinx.__display_version__[:3]:
  lib/python3.10/site-packages/sphinx/cmd/build.py:from sphinx import __display_version__, package_dir
  lib/python3.10/site-packages/sphinx/cmd/build.py:                        version='%%(prog)s %s' % __display_version__)
  lib/python3.10/site-packages/sphinx/cmd/make_mode.py:        print(bold("Sphinx v%s" % sphinx.__display_version__))
  lib/python3.10/site-packages/sphinx/__init__.py:__display_version__ = __version__  # used for command line version
  lib/python3.10/site-packages/sphinx/__init__.py:    __display_version__ = __version__
  lib/python3.10/site-packages/sphinx/__init__.py:            __display_version__ += '/' + ret.stdout.strip()
  lib/python3.10/site-packages/sphinx/ext/githubpages.py:    return {'version': sphinx.__display_version__, 'parallel_read_safe': True}
  lib/python3.10/site-packages/sphinx/ext/intersphinx.py:        'version': sphinx.__display_version__,
  lib/python3.10/site-packages/sphinx/cmd/quickstart.py:from sphinx import __display_version__, package_dir
  lib/python3.10/site-packages/sphinx/cmd/quickstart.py:    print(bold(__('Welcome to the Sphinx %s quickstart utility.')) % __display_version__)
  lib/python3.10/site-packages/sphinx/cmd/quickstart.py:                        version='%%(prog)s %s' % __display_version__)
  lib/python3.10/site-packages/sphinx/ext/viewcode.py:        'version': sphinx.__display_version__,
  lib/python3.10/site-packages/sphinx/util/__init__.py:                  (sphinx.__display_version__,
  lib/python3.10/site-packages/sphinx/ext/ifconfig.py:    return {'version': sphinx.__display_version__, 'parallel_read_safe': True}
  lib/python3.10/site-packages/sphinx/ext/todo.py:        'version': sphinx.__display_version__,
  lib/python3.10/site-packages/sphinx/ext/doctest.py:    return {'version': sphinx.__display_version__, 'parallel_read_safe': True}
  lib/python3.10/site-packages/sphinx/ext/autosummary/__init__.py:    return {'version': sphinx.__display_version__, 'parallel_read_safe': True}
  lib/python3.10/site-packages/sphinx/ext/napoleon/__init__.py:from sphinx import __display_version__ as __version__
  lib/python3.10/site-packages/sphinx/ext/autosummary/generate.py:from sphinx import __display_version__, package_dir
  lib/python3.10/site-packages/sphinx/ext/autosummary/generate.py:                        version='%%(prog)s %s' % __display_version__)
  lib/python3.10/site-packages/sphinx/ext/inheritance_diagram.py:    return {'version': sphinx.__display_version__, 'parallel_read_safe': True}
  lib/python3.10/site-packages/sphinx/ext/imgmath.py:    return {'version': sphinx.__display_version__, 'parallel_read_safe': True}
  lib/python3.10/site-packages/sphinx/ext/linkcode.py:    return {'version': sphinx.__display_version__, 'parallel_read_safe': True}
  lib/python3.10/site-packages/sphinx/ext/coverage.py:    return {'version': sphinx.__display_version__, 'parallel_read_safe': True}
  lib/python3.10/site-packages/sphinx/writers/texinfo.py:from sphinx import __display_version__, addnodes
  lib/python3.10/site-packages/sphinx/writers/texinfo.py:@*Generated by Sphinx """ + __display_version__ + """.@*
  lib/python3.10/site-packages/sphinx/ext/graphviz.py:    return {'version': sphinx.__display_version__, 'parallel_read_safe': True}
  lib/python3.10/site-packages/sphinx/ext/mathjax.py:    return {'version': sphinx.__display_version__, 'parallel_read_safe': True}
  lib/python3.10/site-packages/sphinx/ext/extlinks.py:    return {'version': sphinx.__display_version__, 'parallel_read_safe': True}
  lib/python3.10/site-packages/sphinx/ext/apidoc.py:from sphinx import __display_version__, package_dir
  lib/python3.10/site-packages/sphinx/ext/apidoc.py:                        version='%%(prog)s %s' % __display_version__)
  lib/python3.10/site-packages/sphinx/ext/autodoc/type_comment.py:    return {'version': sphinx.__display_version__, 'parallel_read_safe': True}
  lib/python3.10/site-packages/sphinx/ext/autodoc/__init__.py:    return {'version': sphinx.__display_version__, 'parallel_read_safe': True}
  lib/python3.10/site-packages/pip/_internal/models/target_python.py:        display_version = None
  lib/python3.10/site-packages/pip/_internal/models/target_python.py:            display_version = '.'.join(
  lib/python3.10/site-packages/pip/_internal/models/target_python.py:            ('version_info', display_version),
  lib/python3.10/site-packages/sphinx_rtd_theme/theme.conf:display_version = True
  lib/python3.10/site-packages/sphinx_rtd_theme/layout.html:          {%- if theme_display_version %}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1746#issuecomment-1133331564
issue_url: https://api.github.com/repos/simonw/datasette/issues/1746
node_id: IC_kwDOBm6k_c5DjURs
created_at: 2022-05-20T20:27:12Z
updated_at: 2022-05-20T20:27:12Z
author_association: OWNER
body:
  This seems to work for `brand.html`:

  ```html+jinja
  <div class="sidebar-brand centered">
    {% block brand_content %}
    <div class="sidebar-logo-container">
      <a href="https://datasette.io/"><img class="sidebar-logo" src="{{ logo_url }}" alt="Datasette"></a>
    </div>
    {%- set nav_version = version %}
    {% if READTHEDOCS and current_version %}
      {%- set nav_version = current_version %}
    {% endif %}
    {% if nav_version %}
      <div class="version">
        {{ nav_version }}
      </div>
    {% endif %}
    {% endblock brand_content %}
  </div>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1746#issuecomment-1133331997
issue_url: https://api.github.com/repos/simonw/datasette/issues/1746
node_id: IC_kwDOBm6k_c5DjUYd
created_at: 2022-05-20T20:27:31Z
updated_at: 2022-05-20T20:27:31Z
author_association: OWNER
body:
  I'm going to move my custom JavaScript from `layout.html` into `js/custom.js`, similar to how the custom CSS works.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1746#issuecomment-1133333144
issue_url: https://api.github.com/repos/simonw/datasette/issues/1746
node_id: IC_kwDOBm6k_c5DjUqY
created_at: 2022-05-20T20:28:25Z
updated_at: 2022-05-20T20:28:25Z
author_association: OWNER
body:
  One last question: how to include the Plausible analytics?

  Furo doesn't have any specific tools for this:

  - https://github.com/pradyunsg/furo/discussions/243

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1746#issuecomment-1133335940
issue_url: https://api.github.com/repos/simonw/datasette/issues/1746
node_id: IC_kwDOBm6k_c5DjVWE
created_at: 2022-05-20T20:30:29Z
updated_at: 2022-05-20T20:30:29Z
author_association: OWNER
body:
  I think the trick will be to extend the `base.html` template from Furo using the same trick I used in https://til.simonwillison.net/readthedocs/custom-sphinx-templates

  https://github.com/pradyunsg/furo/blob/2022.04.07/src/furo/theme/furo/base.html - the `site_meta` block looks good.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1746#issuecomment-1133347051
issue_url: https://api.github.com/repos/simonw/datasette/issues/1746
node_id: IC_kwDOBm6k_c5DjYDr
created_at: 2022-05-20T20:39:17Z
updated_at: 2022-05-20T20:39:17Z
author_association: OWNER
body:
  Now live at https://docs.datasette.io/en/latest/ - the JavaScript that adds the banner about that not being the stable version doesn't seem to work though.

  Before:

  <img width="1096" alt="image" src="https://user-images.githubusercontent.com/9599/169607254-99bc0358-4a08-43bf-9aac-a24cc2121979.png">

  After:

  <img width="1096" alt="image" src="https://user-images.githubusercontent.com/9599/169607296-bcec1ed2-517c-4acc-a9a9-d1119d0cc589.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1746#issuecomment-1133348094
issue_url: https://api.github.com/repos/simonw/datasette/issues/1746
node_id: IC_kwDOBm6k_c5DjYT-
created_at: 2022-05-20T20:40:09Z
updated_at: 2022-05-20T20:40:09Z
author_association: OWNER
body:
  Relevant JavaScript: https://github.com/simonw/datasette/blob/1d33fd03b3c211e0f48a8f3bde83880af89e4e69/docs/_static/js/custom.js#L20-L24

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1746#issuecomment-1133396285
issue_url: https://api.github.com/repos/simonw/datasette/issues/1746
node_id: IC_kwDOBm6k_c5DjkE9
created_at: 2022-05-20T21:28:29Z
updated_at: 2022-05-20T21:28:29Z
author_association: OWNER
body:
  That fixed it:

  <img width="1106" alt="image" src="https://user-images.githubusercontent.com/9599/169614893-ec81fe0e-6043-4d7d-b429-7f087dbeaf61.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/435#issuecomment-1133417432
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/435
node_id: IC_kwDOCGYnMM5DjpPY
created_at: 2022-05-20T21:56:10Z
updated_at: 2022-05-20T21:56:10Z
author_association: OWNER
body:
  Before:

  ![sqlite-utils-datasette-io-en-stable-reference-html](https://user-images.githubusercontent.com/9599/169617623-457b4c01-3713-4e5c-b3c3-8574a2214238.png)

  After:

  ![sqlite-utils-datasette-io-en-latest-reference-html](https://user-images.githubusercontent.com/9599/169617666-ba388167-36b8-4a3e-be35-931ee2ab6b3b.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/438#issuecomment-1139379923
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/438
node_id: IC_kwDOCGYnMM5D6Y7T
created_at: 2022-05-27T08:05:01Z
updated_at: 2022-05-27T08:05:01Z
author_association: NONE
body:
  I tried to debug it using `pdb`,  but it looks `sqlite-utils` catches the exception, so it is not quick to figure out where the failure is happening.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1139426398
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/439
node_id: IC_kwDOCGYnMM5D6kRe
created_at: 2022-05-27T09:04:05Z
updated_at: 2022-05-27T10:44:54Z
author_association: NONE
body:
  This code works:

  ```python
  import csv
  import sqlite_utils
  db = sqlite_utils.Database("test.db")
  reader = csv.DictReader(open("csv", encoding="utf-16-le").read().split("\r\n"), delimiter=";")
  db["test"].insert_all(reader, pk="Id")
  ```

  I used `iconv` to change the encoding; sqlite-utils can import the resulting file, even if it stops at 98 %:

  ```
  sqlite-utils insert --csv test test.db clean 
    [------------------------------------]    0%
    [###################################-]   98%  00:00:00
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/26#issuecomment-1141711418
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/26
node_id: IC_kwDOCGYnMM5EDSI6
created_at: 2022-05-31T06:21:15Z
updated_at: 2022-05-31T06:21:15Z
author_association: NONE
body:
  I ran into this. My use case has a JSON file with array of `book` objects with a key called `reviews` which is also an array of objects. My JSON is human-edited and does not specify IDs for either books or reviews. Because sqlite-utils does not support inserting nested objects, I instead have to maintain two separate CSV files with `id` column in `books.csv` and `book_id` column in reviews.csv.

  I think the right way to declare the relationship while inserting a JSON might be to describe the relationship:

  `sqlite-utils insert data.db books mydata.json --hasmany reviews --hasone author --manytomany tags`

  This is relying on the assumption that foreign keys can point to `rowid` primary key.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1753#issuecomment-1147435032
issue_url: https://api.github.com/repos/simonw/datasette/issues/1753
node_id: IC_kwDOBm6k_c5EZHgY
created_at: 2022-06-06T13:15:11Z
updated_at: 2022-06-06T13:15:11Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1753?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1753](https://codecov.io/gh/simonw/datasette/pull/1753?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (23a8515) into [main](https://codecov.io/gh/simonw/datasette/commit/2e9751672d4fe329b3c359d5b7b1992283185820?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2e97516) will **not change** coverage.
  > The diff coverage is `n/a`.

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1753   +/-   ##
  =======================================
    Coverage   91.67%   91.67%           
  =======================================
    Files          36       36           
    Lines        4658     4658           
  =======================================
    Hits         4270     4270           
    Misses        388      388           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1753?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1753?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [2e97516...23a8515](https://codecov.io/gh/simonw/datasette/pull/1753?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1528#issuecomment-1151887842
issue_url: https://api.github.com/repos/simonw/datasette/issues/1528
node_id: IC_kwDOBm6k_c5EqGni
created_at: 2022-06-10T03:23:08Z
updated_at: 2022-06-10T03:23:08Z
author_association: CONTRIBUTOR
body:
  I just put together a version of this in a plugin: https://github.com/eyeseast/datasette-query-files. Happy to have any feedback.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/441#issuecomment-1154373361
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/441
node_id: IC_kwDOCGYnMM5Ezlbx
created_at: 2022-06-13T20:01:25Z
updated_at: 2022-06-13T20:01:25Z
author_association: OWNER
body:
  Yeah, at the moment the best way to do this is with `search_sql()`, but you're right it really isn't very intuitive.

  Here's how I would do this, using a CTE trick to combine the queries:
  ```python
  search_sql = db["articles"].search_sql(columns=["title", "author"]))
  sql = f"""
  with search_results as ({search_sql})
  select * from search_results where owner = :owner
  """
  results = db.query(sql, {"query": "my search query", "owner": "my owner"})
  ```
  I'm not sure if `sqlite-utils` should ever evolve to provide a better way of doing this kind of thing to be honest - if it did, it would turn into more of an ORM. Something like [PeeWee](http://docs.peewee-orm.com/en/latest/) may be a better option here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154386795
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/440
node_id: IC_kwDOCGYnMM5Ezotr
created_at: 2022-06-13T20:16:53Z
updated_at: 2022-06-13T20:16:53Z
author_association: OWNER
body:
  Steps to demonstrate that `sqlite-utils insert` is not affected:

  ```bash
  curl -o artsdatabanken.csv https://artsdatabanken.no/Fab2018/api/export/csv
  sqlite-utils insert arts.db artsdatabanken artsdatabanken.csv --sniff --csv --encoding utf-16le
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154396400
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/440
node_id: IC_kwDOCGYnMM5EzrDw
created_at: 2022-06-13T20:28:25Z
updated_at: 2022-06-13T20:28:25Z
author_association: OWNER
body:
  Fixing that `key` thing (to ignore any key that is `None`) revealed a new bug:

  ```
  File ~/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py:376, in hash_record(record, keys)
      373 if keys is not None:
      374     to_hash = {key: record[key] for key in keys}
      375 return hashlib.sha1(
  --> 376     json.dumps(to_hash, separators=(",", ":"), sort_keys=True, default=repr).encode(
      377         "utf8"
      378     )
      379 ).hexdigest()

  File ~/.pyenv/versions/3.8.2/lib/python3.8/json/__init__.py:234, in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
      232 if cls is None:
      233     cls = JSONEncoder
  --> 234 return cls(
      235     skipkeys=skipkeys, ensure_ascii=ensure_ascii,
      236     check_circular=check_circular, allow_nan=allow_nan, indent=indent,
      237     separators=separators, default=default, sort_keys=sort_keys,
      238     **kw).encode(obj)

  File ~/.pyenv/versions/3.8.2/lib/python3.8/json/encoder.py:199, in JSONEncoder.encode(self, o)
      195         return encode_basestring(o)
      196 # This doesn't pass the iterator directly to ''.join() because the
      197 # exceptions aren't as detailed.  The list call should be roughly
      198 # equivalent to the PySequence_Fast that ''.join() would do.
  --> 199 chunks = self.iterencode(o, _one_shot=True)
      200 if not isinstance(chunks, (list, tuple)):
      201     chunks = list(chunks)

  File ~/.pyenv/versions/3.8.2/lib/python3.8/json/encoder.py:257, in JSONEncoder.iterencode(self, o, _one_shot)
      252 else:
      253     _iterencode = _make_iterencode(
      254         markers, self.default, _encoder, self.indent, floatstr,
      255         self.key_separator, self.item_separator, self.sort_keys,
      256         self.skipkeys, _one_shot)
  --> 257 return _iterencode(o, 0)

  TypeError: '<' not supported between instances of 'NoneType' and 'str'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154449442
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/440
node_id: IC_kwDOCGYnMM5Ez4Ai
created_at: 2022-06-13T21:18:26Z
updated_at: 2022-06-13T21:20:12Z
author_association: OWNER
body:
  Here are full steps to replicate the bug:
  ```python
  from urllib.request import urlopen
  import sqlite_utils
  db = sqlite_utils.Database(memory=True)
  with urlopen("https://artsdatabanken.no/Fab2018/api/export/csv") as fab:
      reader, other = sqlite_utils.utils.rows_from_file(fab, encoding="utf-16le")
      db["fab2018"].insert_all(reader, pk="Id")
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154453319
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/440
node_id: IC_kwDOCGYnMM5Ez49H
created_at: 2022-06-13T21:23:16Z
updated_at: 2022-06-13T21:23:16Z
author_association: OWNER
body:
  Aha! I think I see what's happening here. Here's what `DictReader` does if one of the lines has too many items in it:

  ```pycon
  >>> import csv, io
  >>> list(csv.DictReader(io.StringIO("id,name\n1,Cleo,nohead\n2,Barry")))
  [{'id': '1', 'name': 'Cleo', None: ['nohead']}, {'id': '2', 'name': 'Barry'}]
  ```
  See how that row with too many items gets this:
  `[{'id': '1', 'name': 'Cleo', None: ['nohead']}`

  That's a `None` for the key and (weirdly) a list containing the single item for the value!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154454127
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/440
node_id: IC_kwDOCGYnMM5Ez5Jv
created_at: 2022-06-13T21:24:18Z
updated_at: 2022-06-13T21:24:18Z
author_association: OWNER
body:
  That weird behaviour is documented here: https://docs.python.org/3/library/csv.html#csv.DictReader

  > If a row has more fields than fieldnames, the remaining data is put in a list and stored with the fieldname specified by *restkey* (which defaults to `None`). If a non-blank row has fewer fields than fieldnames, the missing values are filled-in with the value of *restval* (which defaults to `None`).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154456183
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/440
node_id: IC_kwDOCGYnMM5Ez5p3
created_at: 2022-06-13T21:26:55Z
updated_at: 2022-06-13T21:26:55Z
author_association: OWNER
body:
  So I need to make a design decision here: what should `sqlite-utils` do with CSV files that have rows with more values than there are headings?

  Some options:

  - Ignore those extra fields entirely - silently drop that data. I'm not keen on this.
  - Throw an error. The library does this already, but the error is incomprehensible - it could turn into a useful, human-readable error instead.
  - Put the data in a JSON list in a column with a known name (`None` is not a valid column name, so not that). This could be something like `_restkey` or `_values_with_no_heading`. This feels like a better option, but I'd need to carefully pick a name for it - and come up with an answer for the question of what to do if the CSV file being important already uses that heading name for something else.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154457028
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/440
node_id: IC_kwDOCGYnMM5Ez53E
created_at: 2022-06-13T21:28:03Z
updated_at: 2022-06-13T21:28:03Z
author_association: OWNER
body:
  Whatever I decide, I can implement it in `rows_from_file()`, maybe as an optional parameter - then decide how to call it from the `sqlite-utils insert` CLI (perhaps with a new option there too).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154457893
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/440
node_id: IC_kwDOCGYnMM5Ez6El
created_at: 2022-06-13T21:29:02Z
updated_at: 2022-06-13T21:29:02Z
author_association: OWNER
body:
  Here's the current function signature for `rows_from_file()`:

  https://github.com/simonw/sqlite-utils/blob/26e6d2622c57460a24ffdd0128bbaac051d51a5f/sqlite_utils/utils.py#L174-L179

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154474482
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/440
node_id: IC_kwDOCGYnMM5Ez-Hy
created_at: 2022-06-13T21:50:59Z
updated_at: 2022-06-13T21:51:24Z
author_association: OWNER
body:
  Decision: I'm going to default to raising an exception if a row has too many values in it.

  You'll be able to pass `ignore_extras=True` to ignore those extra values, or pass `restkey="the_rest"` to stick them in a list in the `restkey` column.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155310521
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/440
node_id: IC_kwDOCGYnMM5E3KO5
created_at: 2022-06-14T14:58:50Z
updated_at: 2022-06-14T14:58:50Z
author_association: OWNER
body:
  Interesting challenge in writing tests for this: if you give `csv.Sniffer` a short example with an invalid row in it sometimes it picks the wrong delimiter!

      id,name\r\n1,Cleo,oops

  It decided the delimiter there was `e`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155317293
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/440
node_id: IC_kwDOCGYnMM5E3L4t
created_at: 2022-06-14T15:04:01Z
updated_at: 2022-06-14T15:04:01Z
author_association: OWNER
body:
  I think that's unavoidable: it looks like `csv.Sniffer` only works if you feed it a CSV file with an equal number of values in each row, which is understandable.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155350755
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/440
node_id: IC_kwDOCGYnMM5E3UDj
created_at: 2022-06-14T15:25:18Z
updated_at: 2022-06-14T15:25:18Z
author_association: OWNER
body:
  That broke `mypy`:

  `sqlite_utils/utils.py:229: error: Incompatible types in assignment (expression has type "Iterable[Dict[Any, Any]]", variable has type "DictReader[str]")`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155358637
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/440
node_id: IC_kwDOCGYnMM5E3V-t
created_at: 2022-06-14T15:31:34Z
updated_at: 2022-06-14T15:31:34Z
author_association: OWNER
body:
  Getting this past `mypy` is really hard!

  ```
  % mypy sqlite_utils
  sqlite_utils/utils.py:189: error: No overload variant of "pop" of "MutableMapping" matches argument type "None"
  sqlite_utils/utils.py:189: note: Possible overload variants:
  sqlite_utils/utils.py:189: note:     def pop(self, key: str) -> str
  sqlite_utils/utils.py:189: note:     def [_T] pop(self, key: str, default: Union[str, _T] = ...) -> Union[str, _T]
  ```
  That's because of this line:

      row.pop(key=None)

  Which is legit here - we have a dictionary where one of the keys is `None` and we want to remove that key. But the baked in type is apparently `def pop(self, key: str) -> str`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1155364367
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/412
node_id: IC_kwDOCGYnMM5E3XYP
created_at: 2022-06-14T15:36:28Z
updated_at: 2022-06-14T15:36:28Z
author_association: OWNER
body:
  Here's as far as I got with my initial prototype, in `sqlite_utils/pandas.py`:

  ```python
  from .db import Database as _Database, Table as _Table, View as _View
  import pandas as pd
  from typing import (
      Iterable,
      Union,
      Optional,
  )


  class Database(_Database):
      def query(
          self, sql: str, params: Optional[Union[Iterable, dict]] = None
      ) -> pd.DataFrame:
          return pd.DataFrame(super().query(sql, params))

      def table(self, table_name: str, **kwargs) -> Union["Table", "View"]:
          "Return a table object, optionally configured with default options."
          klass = View if table_name in self.view_names() else Table
          return klass(self, table_name, **kwargs)


  class PandasQueryable:
      def rows_where(
          self,
          where: str = None,
          where_args: Optional[Union[Iterable, dict]] = None,
          order_by: str = None,
          select: str = "*",
          limit: int = None,
          offset: int = None,
      ) -> pd.DataFrame:
          return pd.DataFrame(
              super().rows_where(
                  where,
                  where_args,
                  order_by=order_by,
                  select=select,
                  limit=limit,
                  offset=offset,
              )
          )


  class Table(PandasQueryable, _Table):
      pass


  class View(PandasQueryable, _View):
      pass
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/441#issuecomment-1155421299
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/441
node_id: IC_kwDOCGYnMM5E3lRz
created_at: 2022-06-14T16:23:52Z
updated_at: 2022-06-14T16:23:52Z
author_association: OWNER
body:
  Actually I have a thought for something that could help here: I could add a mechanism for inserting additional where filters and parameters into that `.search()` method.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/441#issuecomment-1155515426
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/441
node_id: IC_kwDOCGYnMM5E38Qi
created_at: 2022-06-14T17:53:43Z
updated_at: 2022-06-14T17:53:43Z
author_association: NONE
body:
  That would be handy (additional where filters) but I think the trick with the `with` statement is already an order of magnitude better than what I had thought of, so my problem is solved by it (plus I got to learn about `with` today!)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/443#issuecomment-1155672522
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/443
node_id: IC_kwDOCGYnMM5E4inK
created_at: 2022-06-14T20:18:58Z
updated_at: 2022-06-14T20:18:58Z
author_association: OWNER
body:
  New documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#reading-rows-from-a-file

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/442#issuecomment-1155714131
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/442
node_id: IC_kwDOCGYnMM5E4sxT
created_at: 2022-06-14T21:07:50Z
updated_at: 2022-06-14T21:07:50Z
author_association: OWNER
body:
  Here's the commit where I added that originally, including a test: https://github.com/simonw/sqlite-utils/commit/1a93b72ba710ea2271eaabc204685a27d2469374

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/442#issuecomment-1155748444
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/442
node_id: IC_kwDOCGYnMM5E41Jc
created_at: 2022-06-14T21:55:15Z
updated_at: 2022-06-14T21:55:15Z
author_association: OWNER
body:
  Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#setting-the-maximum-csv-field-size-limit

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/441#issuecomment-1155750270
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/441
node_id: IC_kwDOCGYnMM5E41l-
created_at: 2022-06-14T21:57:57Z
updated_at: 2022-06-14T21:57:57Z
author_association: OWNER
body:
  I added `where=` and `where_args=` parameters to that `.search()` method - updated documentation is here: https://sqlite-utils.datasette.io/en/latest/python-api.html#searching-with-table-search

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/431#issuecomment-1155753397
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/431
node_id: IC_kwDOCGYnMM5E42W1
created_at: 2022-06-14T22:01:38Z
updated_at: 2022-06-14T22:01:38Z
author_association: OWNER
body:
  Yeah, I think it would be neat if the library could support self-referential many-to-many in a nice way.

  I'm not sure about the `left_name/right_name` design though. Would it be possible to have this work as the user intends, by spotting that the other table name `"people"` matches the name of the current table?

  ```python
  db["people"].insert({"name": "Mary"}, pk="name").m2m(
      "people", [{"name": "Michael"}, {"name": "Suzy"}], m2m_table="parent_child", pk="name"
  )
  ```
  The created table could look like this:
  ```sql
  CREATE TABLE [parent_child] (
     [people_id_1] TEXT REFERENCES [people]([name]),
     [people_id_2] TEXT REFERENCES [people]([name]),
     PRIMARY KEY ([people_id_1], [people_id_2])
  )
  ```
  I've not thought very hard about this, so the design I'm proposing here might not work.

  Are there other reasons people might wan the `left_name=` and `right_name=` parameters? If so then I'm much happier with those.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/432#issuecomment-1155756742
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/432
node_id: IC_kwDOCGYnMM5E43LG
created_at: 2022-06-14T22:05:38Z
updated_at: 2022-06-14T22:05:49Z
author_association: OWNER
body:
  I don't like the idea of `table_names()` returning names of tables from connected databases as well, because it feels like it could lead to surprising behaviour - especially if those connected databases turn to have table names that are duplicated in the main connected database.

  It would be neat if functions like `.rows_where()` worked though.

  One thought would be to support something like this:
  ```python
  rows = db["otherdb.tablename"].rows_where()
  ```
  But... `.` is a valid character in a SQLite table name. So `"otherdb.tablename"` might ambiguously refer to a table called `tablename` in a connected database with the alias `otherdb`, OR a table in the current database with the name `otherdb.tablename`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/432#issuecomment-1155758664
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/432
node_id: IC_kwDOCGYnMM5E43pI
created_at: 2022-06-14T22:07:50Z
updated_at: 2022-06-14T22:07:50Z
author_association: OWNER
body:
  Another potential fix: add a `alias=` parameter to `rows_where()` and other similar methods. Then you could do this:

  ```python
  rows = db["tablename"].rows_where(alias="otherdb")
  ```
  This feels wrong to me: `db["tablename"]` is the bit that is supposed to return a table object. Having part of what that table object is exist as a parameter to other methods is confusing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/432#issuecomment-1155759857
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/432
node_id: IC_kwDOCGYnMM5E437x
created_at: 2022-06-14T22:09:07Z
updated_at: 2022-06-14T22:09:07Z
author_association: OWNER
body:
  Third option, and I think the one I like the best:
  ```python
  rows = db.table("tablename", alias="otherdb").rows_where(alias="otherdb")
  ```
  The `db.table(tablename)` method already exists as an alternative to `db[tablename]`: https://sqlite-utils.datasette.io/en/stable/python-api.html#python-api-table-configuration

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/432#issuecomment-1155764064
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/432
node_id: IC_kwDOCGYnMM5E449g
created_at: 2022-06-14T22:15:44Z
updated_at: 2022-06-14T22:15:44Z
author_association: OWNER
body:
  Implementing this would be a pretty big change - initial instinct is that I'd need to introduce a `self.alias` property to `Queryable` (the subclass of `Table` and `View`) and a new `self.name_with_alias` getter which returns `alias.tablename` if `alias` is set to a not-None value. Then I'd need to rewrite every piece of code like this:

  https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/db.py#L1161

  To look like this instead:
  ```python
          sql = "select {} from [{}]".format(select, self.name_with_alias)
  ```
  But some parts would be harder - for example:

  https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/db.py#L1227-L1231

  Would have to know to query  `alias.sqlite_master` instead.

  The cached table counts logic like this would need a bunch of changes too:

  https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/db.py#L644-L657

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/432#issuecomment-1155764428
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/432
node_id: IC_kwDOCGYnMM5E45DM
created_at: 2022-06-14T22:16:21Z
updated_at: 2022-06-14T22:16:21Z
author_association: OWNER
body:
  Initial idea of how the `.table()` method would change:
  ```diff
  diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
  index 7a06304..3ecb40b 100644
  --- a/sqlite_utils/db.py
  +++ b/sqlite_utils/db.py
  @@ -474,11 +474,12 @@ class Database:
               self._tracer(sql, None)
           return self.conn.executescript(sql)
   
  -    def table(self, table_name: str, **kwargs) -> Union["Table", "View"]:
  +    def table(self, table_name: str, alias: Optional[str] = None, **kwargs) -> Union["Table", "View"]:
           """
           Return a table object, optionally configured with default options.
   
           :param table_name: Name of the table
  +        :param alias: The database alias to use, if referring to a table in another connected database
           """
           klass = View if table_name in self.view_names() else Table
           return klass(self, table_name, **kwargs)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155767202
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/439
node_id: IC_kwDOCGYnMM5E45ui
created_at: 2022-06-14T22:21:10Z
updated_at: 2022-06-14T22:21:10Z
author_association: OWNER
body:
  I can't figure out why that error is being swallowed like that. The most likely culprit was this code: 

  https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/cli.py#L1021-L1043

  But I tried changing it like this:

  ```diff
  diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py
  index 86eddfb..ed26fdd 100644
  --- a/sqlite_utils/cli.py
  +++ b/sqlite_utils/cli.py
  @@ -1023,6 +1023,7 @@ def insert_upsert_implementation(
               docs, pk=pk, batch_size=batch_size, alter=alter, **extra_kwargs
           )
       except Exception as e:
  +        raise
           if (
               isinstance(e, sqlite3.OperationalError)
               and e.args
  ```
  And your steps to reproduce still got to 49% and then failed silently.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155767915
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/440
node_id: IC_kwDOCGYnMM5E455r
created_at: 2022-06-14T22:22:27Z
updated_at: 2022-06-14T22:22:27Z
author_association: OWNER
body:
  I forgot to add equivalents of `extras_key=` and `ignore_extras=` to the CLI tool - will do that in a separate issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155769216
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/439
node_id: IC_kwDOCGYnMM5E46OA
created_at: 2022-06-14T22:24:49Z
updated_at: 2022-06-14T22:25:06Z
author_association: OWNER
body:
  I have a hunch that this crash may be caused by a CSV value which is too long, as addressed at the library level in:
  - #440

  But not yet addressed in the CLI tool, see:

  - #444

  Either way though, I really don't like that errors like this are swallowed!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155771462
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/439
node_id: IC_kwDOCGYnMM5E46xG
created_at: 2022-06-14T22:28:38Z
updated_at: 2022-06-14T22:28:38Z
author_association: OWNER
body:
  Maybe this isn't a CSV field value problem - I tried this patch and didn't seem to hit the new breakpoints:
  ```diff
  diff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py
  index d2ccc5f..f1b823a 100644
  --- a/sqlite_utils/utils.py
  +++ b/sqlite_utils/utils.py
  @@ -204,13 +204,17 @@ def _extra_key_strategy(
           # DictReader adds a 'None' key with extra row values
           if None not in row:
               yield row
  -        elif ignore_extras:
  +            continue
  +        else:
  +            breakpoint()
  +        if ignore_extras:
               # ignoring row.pop(none) because of this issue:
               # https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155358637
               row.pop(None)  # type: ignore
               yield row
           elif not extras_key:
               extras = row.pop(None)  # type: ignore
  +            breakpoint()
               raise RowError(
                   "Row {} contained these extra values: {}".format(row, extras)
               )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155772244
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/439
node_id: IC_kwDOCGYnMM5E469U
created_at: 2022-06-14T22:30:03Z
updated_at: 2022-06-14T22:30:03Z
author_association: OWNER
body:
  Tried this:
  ```
  % python -i $(which sqlite-utils) insert --csv --delimiter ";" --encoding "utf-16-le" test test.db csv
    [------------------------------------]    0%
    [#################-------------------]   49%  00:00:01Traceback (most recent call last):
    File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 1072, in main
      ctx.exit()
    File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 692, in exit
      raise Exit(code)
  click.exceptions.Exit: 0

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils", line 33, in <module>
      sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')())
    File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
      return self.main(*args, **kwargs)
    File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 1090, in main
      sys.exit(e.exit_code)
  SystemExit: 0
  >>> 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155776023
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/439
node_id: IC_kwDOCGYnMM5E474X
created_at: 2022-06-14T22:36:07Z
updated_at: 2022-06-14T22:36:07Z
author_association: OWNER
body:
  Wait! The arguments in that are the wrong way round. This is correct:

      sqlite-utils insert --csv --delimiter ";" --encoding "utf-16-le" test.db test csv

  It still outputs the following:

    [------------------------------------]    0%
    [#################-------------------]   49%  00:00:02%

  But it creates a `test.db` file that is 6.2MB.

  That database has 3141 rows in it:

  ```
  % sqlite-utils tables test.db --counts -t
  table      count
  -------  -------
  test        3142
  ```
  I converted that `csv` file to utf-8 like so:

      iconv -f UTF-16LE -t UTF-8 csv > utf8.csv

  And it contains 3142 lines:
  ```
  % wc -l utf8.csv 
      3142 utf8.csv
  ```
  So my hunch here is that the problem is actually that the progress bar doesn't know how to correctly measure files in `utf-16-le` encoding!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155782835
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/439
node_id: IC_kwDOCGYnMM5E49iz
created_at: 2022-06-14T22:48:22Z
updated_at: 2022-06-14T22:49:53Z
author_association: OWNER
body:
  Here's the code that implements the progress bar in question: https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/cli.py#L918-L932

  It calls `file_progress()` which looks like this:

  https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/utils.py#L159-L175

  Which uses this:

  https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/utils.py#L148-L156

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155784284
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/439
node_id: IC_kwDOCGYnMM5E495c
created_at: 2022-06-14T22:51:03Z
updated_at: 2022-06-14T22:52:13Z
author_association: OWNER
body:
  Yes, this is the problem. The progress bar length is set to the length in bytes of the file - `os.path.getsize(file.name)` - but it's then incremented by the length of each DECODED line in turn.

  So if the file is in `utf-16-le` (twice the size of `utf-8`) the progress bar will finish at 50%!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155788944
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/439
node_id: IC_kwDOCGYnMM5E4_CQ
created_at: 2022-06-14T23:00:24Z
updated_at: 2022-06-14T23:00:24Z
author_association: OWNER
body:
  The progress bar only works if the file-like object passed to it has a `fp.fileno()` that isn't 0 (for stdin) - that's how it detects that the file is something which it can measure the size of in order to show progress.

  If we know the file size in bytes AND we know the character encoding, can we change `UpdateWrapper` to update the number of bytes-per-character instead?

  I don't think so: I can't see a way of definitively saying "for this encoding the number of bytes per character is X" - and in fact I'm pretty sure that question doesn't even make sense since variable-length encodings exist.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155789101
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/439
node_id: IC_kwDOCGYnMM5E4_Et
created_at: 2022-06-14T23:00:45Z
updated_at: 2022-06-14T23:00:45Z
author_association: OWNER
body:
  I'm going to mark this as "help wanted" and leave it open. I'm glad that it's not actually a bug where errors get swallowed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/434#issuecomment-1155791109
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/434
node_id: IC_kwDOCGYnMM5E4_kF
created_at: 2022-06-14T23:04:40Z
updated_at: 2022-06-14T23:04:40Z
author_association: OWNER
body:
  Definitely a bug - thanks for the detailed write-up!

  You're right, the code at fault is here:

  https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/db.py#L2213-L2231

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/434#issuecomment-1155794149
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/434
node_id: IC_kwDOCGYnMM5E5ATl
created_at: 2022-06-14T23:09:54Z
updated_at: 2022-06-14T23:09:54Z
author_association: OWNER
body:
  A test that demonstrates the problem:
  ```python
  @pytest.mark.parametrize("reverse_order", (True, False))
  def test_detect_fts_similar_tables(fresh_db, reverse_order):
      # https://github.com/simonw/sqlite-utils/issues/434
      table1, table2 = ("demo", "demo2")
      if reverse_order:
          table1, table2 = table2, table1

      fresh_db[table1].insert({"title": "Hello"}).enable_fts(
          ["title"], fts_version="FTS4"
      )
      fresh_db[table2].insert({"title": "Hello"}).enable_fts(
          ["title"], fts_version="FTS4"
      )
      assert fresh_db[table1].detect_fts() == "{}_fts".format(table1)
      assert fresh_db[table2].detect_fts() == "{}_fts".format(table2)
  ```
  The order matters - so this test currently passes in one direction and fails in the other:
  ```
  >       assert fresh_db[table2].detect_fts() == "{}_fts".format(table2)
  E       AssertionError: assert 'demo2_fts' == 'demo_fts'
  E         - demo_fts
  E         + demo2_fts
  E         ?     +

  tests/test_introspect.py:53: AssertionError
  ========================================================================================= short test summary info =========================================================================================
  FAILED tests/test_introspect.py::test_detect_fts_similar_tables[True] - AssertionError: assert 'demo2_fts' == 'demo_fts'
  =============================================================================== 1 failed, 1 passed, 855 deselected in 1.00s ===============================================================================
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/434#issuecomment-1155801812
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/434
node_id: IC_kwDOCGYnMM5E5CLU
created_at: 2022-06-14T23:23:32Z
updated_at: 2022-06-14T23:23:32Z
author_association: OWNER
body:
  Since table names can be quoted like this:
  ```sql
  CREATE VIRTUAL TABLE "searchable_fts"
      USING FTS4 (text1, text2, [name with . and spaces], content="searchable")
  ```
  OR like this:
  ```sql
  CREATE VIRTUAL TABLE "searchable_fts"
      USING FTS4 (text1, text2, [name with . and spaces], content=[searchable])
  ```
  This fix looks to be correct to me (copying from the updated `test_with_trace()` test):

  ```python
              (
                  "SELECT name FROM sqlite_master\n"
                  "    WHERE rootpage = 0\n"
                  "    AND (\n"
                  "        sql LIKE :like\n"
                  "        OR sql LIKE :like2\n"
                  "        OR (\n"
                  "            tbl_name = :table\n"
                  "            AND sql LIKE '%VIRTUAL TABLE%USING FTS%'\n"
                  "        )\n"
                  "    )",
                  {
                      "like": "%VIRTUAL TABLE%USING FTS%content=[dogs]%",
                      "like2": '%VIRTUAL TABLE%USING FTS%content="dogs"%',
                      "table": "dogs",
                  },
              )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/430#issuecomment-1155803262
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/430
node_id: IC_kwDOCGYnMM5E5Ch-
created_at: 2022-06-14T23:26:11Z
updated_at: 2022-06-14T23:26:11Z
author_association: OWNER
body:
  It looks like `PRAGMA temp_store` was the right option to use here: https://www.sqlite.org/pragma.html#pragma_temp_store

  `temp_store_directory` is listed as deprecated here: https://www.sqlite.org/pragma.html#pragma_temp_store_directory

  I'm going to turn this into a help-wanted documentation issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/444#issuecomment-1155804459
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/444
node_id: IC_kwDOCGYnMM5E5C0r
created_at: 2022-06-14T23:28:18Z
updated_at: 2022-06-14T23:28:18Z
author_association: OWNER
body:
  I think these become part of the `_import_options` list which is used in a few places:

  https://github.com/simonw/sqlite-utils/blob/b8af3b96f5c72317cc8783dc296a94f6719987d9/sqlite_utils/cli.py#L765-L800

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/444#issuecomment-1155815186
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/444
node_id: IC_kwDOCGYnMM5E5FcS
created_at: 2022-06-14T23:48:16Z
updated_at: 2022-06-14T23:48:16Z
author_association: OWNER
body:
  This is tricky to implement because of this code: https://github.com/simonw/sqlite-utils/blob/b8af3b96f5c72317cc8783dc296a94f6719987d9/sqlite_utils/cli.py#L938-L945

  It's reconstructing each document using the known headers here:

  `docs = (dict(zip(headers, row)) for row in reader)`

  So my first attempt at this - the diff here - did not have the desired result:

  ```diff
  diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py
  index 86eddfb..00b920b 100644
  --- a/sqlite_utils/cli.py
  +++ b/sqlite_utils/cli.py
  @@ -6,7 +6,7 @@ import hashlib
   import pathlib
   import sqlite_utils
   from sqlite_utils.db import AlterError, BadMultiValues, DescIndex
  -from sqlite_utils.utils import maximize_csv_field_size_limit
  +from sqlite_utils.utils import maximize_csv_field_size_limit, _extra_key_strategy
   from sqlite_utils import recipes
   import textwrap
   import inspect
  @@ -797,6 +797,15 @@ _import_options = (
           "--encoding",
           help="Character encoding for input, defaults to utf-8",
       ),
  +    click.option(
  +        "--ignore-extras",
  +        is_flag=True,
  +        help="If a CSV line has more than the expected number of values, ignore the extras",
  +    ),
  +    click.option(
  +        "--extras-key",
  +        help="If a CSV line has more than the expected number of values put them in a list in this column",
  +    ),
   )
   
   
  @@ -885,6 +894,8 @@ def insert_upsert_implementation(
       sniff,
       no_headers,
       encoding,
  +    ignore_extras,
  +    extras_key,
       batch_size,
       alter,
       upsert,
  @@ -909,6 +920,10 @@ def insert_upsert_implementation(
           raise click.ClickException("--flatten cannot be used with --csv or --tsv")
       if encoding and not (csv or tsv):
           raise click.ClickException("--encoding must be used with --csv or --tsv")
  +    if ignore_extras and extras_key:
  +        raise click.ClickException(
  +            "--ignore-extras and --extras-key cannot be used together"
  +        )
       if pk and len(pk) == 1:
           pk = pk[0]
       encoding = encoding or "utf-8-sig"
  @@ -935,7 +950,9 @@ def insert_upsert_implementation(
                   csv_reader_args["delimiter"] = delimiter
               if quotechar:
                   csv_reader_args["quotechar"] = quotechar
  -            reader = csv_std.reader(decoded, **csv_reader_args)
  +            reader = _extra_key_strategy(
  +                csv_std.reader(decoded, **csv_reader_args), ignore_extras, extras_key
  +            )
               first_row = next(reader)
               if no_headers:
                   headers = ["untitled_{}".format(i + 1) for i in range(len(first_row))]
  @@ -1101,6 +1118,8 @@ def insert(
       sniff,
       no_headers,
       encoding,
  +    ignore_extras,
  +    extras_key,
       batch_size,
       alter,
       detect_types,
  @@ -1176,6 +1195,8 @@ def insert(
               sniff,
               no_headers,
               encoding,
  +            ignore_extras,
  +            extras_key,
               batch_size,
               alter=alter,
               upsert=False,
  @@ -1214,6 +1235,8 @@ def upsert(
       sniff,
       no_headers,
       encoding,
  +    ignore_extras,
  +    extras_key,
       alter,
       not_null,
       default,
  @@ -1254,6 +1277,8 @@ def upsert(
               sniff,
               no_headers,
               encoding,
  +            ignore_extras,
  +            extras_key,
               batch_size,
               alter=alter,
               upsert=True,
  @@ -1297,6 +1322,8 @@ def bulk(
       sniff,
       no_headers,
       encoding,
  +    ignore_extras,
  +    extras_key,
       load_extension,
   ):
       """
  @@ -1331,6 +1358,8 @@ def bulk(
               sniff=sniff,
               no_headers=no_headers,
               encoding=encoding,
  +            ignore_extras=ignore_extras,
  +            extras_key=extras_key,
               batch_size=batch_size,
               alter=False,
               upsert=False,

  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/444#issuecomment-1155815956
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/444
node_id: IC_kwDOCGYnMM5E5FoU
created_at: 2022-06-14T23:49:56Z
updated_at: 2022-07-07T16:39:18Z
author_association: OWNER
body:
  Yeah my initial implementation there makes no sense:
  ```python
              csv_reader_args = {"dialect": dialect}
              if delimiter:
                  csv_reader_args["delimiter"] = delimiter
              if quotechar:
                  csv_reader_args["quotechar"] = quotechar
              reader = _extra_key_strategy(
                  csv_std.reader(decoded, **csv_reader_args), ignore_extras, extras_key
              )
              first_row = next(reader)
              if no_headers:
                  headers = ["untitled_{}".format(i + 1) for i in range(len(first_row))]
                  reader = itertools.chain([first_row], reader)
              else:
                  headers = first_row
              docs = (dict(zip(headers, row)) for row in reader)
  ```
  Because my `_extra_key_strategy()` helper function is designed to work against `csv.DictReader` - not against `csv.reader()` which returns a sequence of lists, not a sequence of dictionaries.

  In fact, what's happening here is that `dict(zip(headers, row))` is ignoring anything in the row that doesn't correspond to a header:

  ```pycon
  >>> list(zip(["a", "b"], [1, 2, 3]))
  [('a', 1), ('b', 2)]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155953345
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/439
node_id: IC_kwDOCGYnMM5E5nLB
created_at: 2022-06-15T03:53:43Z
updated_at: 2022-06-15T03:53:43Z
author_association: OWNER
body:
  I tried fixing this by using `.tell()` to read the file position as I was iterating through it:
  ```diff
  diff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py
  index d2ccc5f..29ad12e 100644
  --- a/sqlite_utils/utils.py
  +++ b/sqlite_utils/utils.py
  @@ -149,10 +149,13 @@ class UpdateWrapper:
       def __init__(self, wrapped, update):
           self._wrapped = wrapped
           self._update = update
  +        self._tell = wrapped.tell()
   
       def __iter__(self):
           for line in self._wrapped:
  -            self._update(len(line))
  +            tell = self._wrapped.tell()
  +            self._update(self._tell - tell)
  +            self._tell = tell
               yield line
   ```
  This did not work - I get this error:

  ```
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py", line 206, in _extra_key_strategy
      for row in reader:
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py", line 156, in __iter__
      tell = self._wrapped.tell()
  OSError: telling position disabled by next() call
  ```
  It looks like you can't use `.tell()` during iteration: https://stackoverflow.com/questions/29618936/how-to-solve-oserror-telling-position-disabled-by-next-call

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1759#issuecomment-1160717735
issue_url: https://api.github.com/repos/simonw/datasette/issues/1759
node_id: IC_kwDOBm6k_c5FLyWn
created_at: 2022-06-20T18:04:41Z
updated_at: 2022-06-20T18:04:41Z
author_association: OWNER
body:
  I don't think this change needs any changes to the documentation: https://docs.datasette.io/en/stable/custom_templates.html#custom-templates

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1759#issuecomment-1160717784
issue_url: https://api.github.com/repos/simonw/datasette/issues/1759
node_id: IC_kwDOBm6k_c5FLyXY
created_at: 2022-06-20T18:04:46Z
updated_at: 2022-06-20T18:04:46Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1759?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1759](https://codecov.io/gh/simonw/datasette/pull/1759?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b901bb0) into [main](https://codecov.io/gh/simonw/datasette/commit/2e9751672d4fe329b3c359d5b7b1992283185820?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2e97516) will **not change** coverage.
  > The diff coverage is `n/a`.

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1759   +/-   ##
  =======================================
    Coverage   91.67%   91.67%           
  =======================================
    Files          36       36           
    Lines        4658     4658           
  =======================================
    Hits         4270     4270           
    Misses        388      388           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1759?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1759?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [2e97516...b901bb0](https://codecov.io/gh/simonw/datasette/pull/1759?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/445#issuecomment-1160763268
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/445
node_id: IC_kwDOCGYnMM5FL9eE
created_at: 2022-06-20T19:09:21Z
updated_at: 2022-06-20T19:09:21Z
author_association: OWNER
body:
  Code to document: https://github.com/simonw/sqlite-utils/blob/3fbe8a784cc2f3fa0bfa8612fec9752ff9068a2b/sqlite_utils/utils.py#L318-L331

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/445#issuecomment-1160793114
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/445
node_id: IC_kwDOCGYnMM5FMEwa
created_at: 2022-06-20T19:47:36Z
updated_at: 2022-06-20T19:47:36Z
author_association: OWNER
body:
  I also added inline documentation and types: https://github.com/simonw/sqlite-utils/blob/773f2b6b20622bb986984a1c3161d5b3aaa1046b/sqlite_utils/utils.py#L318-L360

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/445#issuecomment-1160794175
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/445
node_id: IC_kwDOCGYnMM5FMFA_
created_at: 2022-06-20T19:49:02Z
updated_at: 2022-06-20T19:49:02Z
author_association: OWNER
body:
  New documentation:

  - https://sqlite-utils.datasette.io/en/latest/python-api.html#detecting-column-types-using-typetracker
  - https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite-utils-utils-typetracker

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/443#issuecomment-1160794604
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/443
node_id: IC_kwDOCGYnMM5FMFHs
created_at: 2022-06-20T19:49:37Z
updated_at: 2022-06-20T19:49:37Z
author_association: OWNER
body:
  Also now shows up here: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite-utils-utils-rows-from-file

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/446#issuecomment-1160798645
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/446
node_id: IC_kwDOCGYnMM5FMGG1
created_at: 2022-06-20T19:55:34Z
updated_at: 2022-06-20T19:55:34Z
author_association: OWNER
body:
  `just` now defaults to running the tests and linters.

  `just test` runs the tests - it can take arguments, e.g. `just test -k transform`

  `just lint` runs all of the linters.

  `just black` applies Black.

  In all case it assumes you are using `pipenv`, at least for the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1161849874
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/297
node_id: IC_kwDOCGYnMM5FQGwS
created_at: 2022-06-21T14:49:12Z
updated_at: 2022-06-21T14:49:12Z
author_association: OWNER
body:
  Since there are all sorts of existing options for `sqlite-utils insert` that won't work with this, maybe it would be better to have an entirely separate command - this for example:

      sqlite-utils fast-insert data.db mytable data.csv 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/447#issuecomment-1161857806
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/447
node_id: IC_kwDOCGYnMM5FQIsO
created_at: 2022-06-21T14:55:51Z
updated_at: 2022-06-21T14:58:14Z
author_association: OWNER
body:
  https://stackoverflow.com/a/44379513 suggests that the fix is:

      .. code-block:: text

  Or set this in `conf.py`:

      highlight_language = "none"

  I like that better - I don't like that all `::` blocks default to being treated as Python code.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1162179354
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/297
node_id: IC_kwDOCGYnMM5FRXMa
created_at: 2022-06-21T18:44:03Z
updated_at: 2022-06-21T18:44:03Z
author_association: OWNER
body:
  The thing I like about that `--fast` option is that it could selectively use this alternative mechanism just for the files for which it can work (CSV and TSV files). I could also add a `--fast` option to `sqlite-utils memory` which could then kick in only for operations that involve just TSV and CSV files.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/447#issuecomment-1162186856
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/447
node_id: IC_kwDOCGYnMM5FRZBo
created_at: 2022-06-21T18:48:46Z
updated_at: 2022-06-21T18:48:46Z
author_association: OWNER
body:
  That fixed it:

  <img width="782" alt="image" src="https://user-images.githubusercontent.com/9599/174875556-3a569c90-5c92-48eb-935c-470638deb335.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1162223668
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/297
node_id: IC_kwDOCGYnMM5FRiA0
created_at: 2022-06-21T19:19:22Z
updated_at: 2022-06-21T19:22:15Z
author_association: OWNER
body:
  Built a prototype of `--fast` for the `sqlite-utils memory` command:

  ```
  % time sqlite-utils memory taxi.csv 'SELECT passenger_count, COUNT(*), AVG(total_amount) FROM taxi GROUP BY passenger_count' --fast
  passenger_count  COUNT(*)  AVG(total_amount)
  ---------------  --------  -----------------
                   128020    32.2371511482553 
  0                42228     17.0214016766151 
  1                1533197   17.6418833067999 
  2                286461    18.0975870711456 
  3                72852     17.9153958710923 
  4                25510     18.452774990196  
  5                50291     17.2709248175672 
  6                32623     17.6002964166367 
  7                2         87.17            
  8                2         95.705           
  9                1         113.6            
  sqlite-utils memory taxi.csv  --fast  12.71s user 0.48s system 104% cpu 12.627 total
  ```
  Takes 13s - about the same time as calling `sqlite3 :memory: ...` directly as seen in https://til.simonwillison.net/sqlite/one-line-csv-operations

  Without the `--fast` option that takes several minutes (262s = 4m20s)!

  Here's the prototype so far:

  ```diff
  diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py
  index 86eddfb..1c83ef6 100644
  --- a/sqlite_utils/cli.py
  +++ b/sqlite_utils/cli.py
  @@ -14,6 +14,8 @@ import io
   import itertools
   import json
   import os
  +import shutil
  +import subprocess
   import sys
   import csv as csv_std
   import tabulate
  @@ -1669,6 +1671,7 @@ def query(
       is_flag=True,
       help="Analyze resulting tables and output results",
   )
  +@click.option("--fast", is_flag=True, help="Fast mode, only works with CSV and TSV")
   @load_extension_option
   def memory(
       paths,
  @@ -1692,6 +1695,7 @@ def memory(
       save,
       analyze,
       load_extension,
  +    fast,
   ):
       """Execute SQL query against an in-memory database, optionally populated by imported data
   
  @@ -1719,6 +1723,22 @@ def memory(
       \b
           sqlite-utils memory animals.csv --schema
       """
  +    if fast:
  +        if (
  +            attach
  +            or flatten
  +            or param
  +            or encoding
  +            or no_detect_types
  +            or analyze
  +            or load_extension
  +        ):
  +            raise click.ClickException(
  +                "--fast mode does not support any of the following options: --attach, --flatten, --param, --encoding, --no-detect-types, --analyze, --load-extension"
  +            )
  +        # TODO: Figure out and pass other supported options
  +        memory_fast(paths, sql)
  +        return
       db = sqlite_utils.Database(memory=True)
       # If --dump or --save or --analyze used but no paths detected, assume SQL query is a path:
       if (dump or save or schema or analyze) and not paths:
  @@ -1791,6 +1811,33 @@ def memory(
       )
   
   
  +def memory_fast(paths, sql):
  +    if not shutil.which("sqlite3"):
  +        raise click.ClickException("sqlite3 not found in PATH")
  +    args = ["sqlite3", ":memory:", "-cmd", ".mode csv"]
  +    table_names = []
  +
  +    def name(path):
  +        base_name = pathlib.Path(path).stem or "t"
  +        table_name = base_name
  +        prefix = 1
  +        while table_name in table_names:
  +            prefix += 1
  +            table_name = "{}_{}".format(base_name, prefix)
  +        return table_name
  +
  +    for path in paths:
  +        table_name = name(path)
  +        table_names.append(table_name)
  +        args.extend(
  +            ["-cmd", ".import {} {}".format(pathlib.Path(path).resolve(), table_name)]
  +        )
  +
  +    args.extend(["-cmd", ".mode column"])
  +    args.append(sql)
  +    subprocess.run(args)
  +
  +
   def _execute_query(
       db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols
   ):
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/446#issuecomment-1162234441
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/446
node_id: IC_kwDOCGYnMM5FRkpJ
created_at: 2022-06-21T19:28:35Z
updated_at: 2022-06-21T19:28:35Z
author_association: OWNER
body:
  `just -l` now does this:

  ```
  % just -l
  Available recipes:
      black         # Apply Black
      cog           # Rebuild docs with cog
      default       # Run tests and linters
      lint          # Run linters: black, flake8, mypy, cog
      test *options # Run pytest with supplied options
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/448#issuecomment-1162498734
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/448
node_id: IC_kwDOCGYnMM5FSlKu
created_at: 2022-06-22T00:43:45Z
updated_at: 2022-06-22T00:43:45Z
author_association: NONE
body:
  Attempted to test on a machine with a new version of Python, but install failed with an error message for the 'click' package.

  ```
  C:\WINDOWS\system32>"c:\Program Files\Python310\python.exe"
  Python 3.10.2 (tags/v3.10.2:a58ebcc, Jan 17 2022, 14:12:15) [MSC v.1929 64 bit (AMD64)] on win32
  Type "help", "copyright", "credits" or "license" for more information.
  >>> quit()

  C:\WINDOWS\system32>cd C:\Users\swood\Downloads\sqlite-utils-main-20220621\sqlite-utils-main

  C:\Users\swood\Downloads\sqlite-utils-main-20220621\sqlite-utils-main>"c:\Program Files\Python310\python.exe" setup.py install
  running install
  running bdist_egg
  running egg_info

  ...

  Installed c:\program files\python310\lib\site-packages\click_default_group_wheel-1.2.2-py3.10.egg
  Searching for click
  Downloading https://files.pythonhosted.org/packages/3d/da/f3bbf30f7e71d881585d598f67f4424b2cc4c68f39849542e81183218017/click-default-group-wheel-1.2.2.tar.gz#sha256=e90da42d92c03e88a12ed0c0b69c8a29afb5d36e3dc8d29c423ba4219e6d7747
  Best match: click default-group-wheel-1.2.2
  Processing click-default-group-wheel-1.2.2.tar.gz
  Writing C:\Users\swood\AppData\Local\Temp\easy_install-aiaj0_eh\click-default-group-wheel-1.2.2\setup.cfg
  Running click-default-group-wheel-1.2.2\setup.py -q bdist_egg --dist-dir C:\Users\swood\AppData\Local\Temp\easy_install-aiaj0_eh\click-default-group-wheel-1.2.2\egg-dist-tmp-z61a4h8n
  zip_safe flag not set; analyzing archive contents...
  removing 'c:\program files\python310\lib\site-packages\click_default_group_wheel-1.2.2-py3.10.egg' (and everything under it)
  Copying click_default_group_wheel-1.2.2-py3.10.egg to c:\program files\python310\lib\site-packages
  click-default-group-wheel 1.2.2 is already the active version in easy-install.pth

  Installed c:\program files\python310\lib\site-packages\click_default_group_wheel-1.2.2-py3.10.egg
  error: The 'click' distribution was not found and is required by click-default-group-wheel, sqlite-utils
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1760#issuecomment-1163097455
issue_url: https://api.github.com/repos/simonw/datasette/issues/1760
node_id: IC_kwDOBm6k_c5FU3Vv
created_at: 2022-06-22T13:27:08Z
updated_at: 2022-06-22T13:27:08Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1760?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1760](https://codecov.io/gh/simonw/datasette/pull/1760?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (69951ee) into [main](https://codecov.io/gh/simonw/datasette/commit/00e59ec461dc0150772b999c7cc15fcb9b507d58?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (00e59ec) will **not change** coverage.
  > The diff coverage is `n/a`.

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1760   +/-   ##
  =======================================
    Coverage   91.67%   91.67%           
  =======================================
    Files          36       36           
    Lines        4658     4658           
  =======================================
    Hits         4270     4270           
    Misses        388      388           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1760?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1760?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [00e59ec...69951ee](https://codecov.io/gh/simonw/datasette/pull/1760?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/431#issuecomment-1164460052
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/431
node_id: IC_kwDOCGYnMM5FaEAU
created_at: 2022-06-23T14:12:51Z
updated_at: 2022-06-23T14:12:51Z
author_association: NONE
body:
  Yeah, I think I prefer your suggestion: it seems cleaner than my initial `left_name=`/`right_name=` idea. Perhaps one downside is that it's less obvious what the role of each field is: in this example, is `people_id_1` a reference to parent or child?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1763#issuecomment-1168715058
issue_url: https://api.github.com/repos/simonw/datasette/issues/1763
node_id: IC_kwDOBm6k_c5FqS0y
created_at: 2022-06-28T13:19:28Z
updated_at: 2022-06-28T13:19:28Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1763?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1763](https://codecov.io/gh/simonw/datasette/pull/1763?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (fd6a817) into [main](https://codecov.io/gh/simonw/datasette/commit/00e59ec461dc0150772b999c7cc15fcb9b507d58?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (00e59ec) will **not change** coverage.
  > The diff coverage is `n/a`.

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1763   +/-   ##
  =======================================
    Coverage   91.67%   91.67%           
  =======================================
    Files          36       36           
    Lines        4658     4658           
  =======================================
    Hits         4270     4270           
    Misses        388      388           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1763?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1763?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [00e59ec...fd6a817](https://codecov.io/gh/simonw/datasette/pull/1763?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/26#issuecomment-1170595021
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/26
node_id: IC_kwDOCGYnMM5FxdzN
created_at: 2022-06-29T23:35:29Z
updated_at: 2022-06-29T23:35:29Z
author_association: NONE
body:
  Have you seen [MakeTypes](https://github.com/jvilk/MakeTypes)? Not the exact same thing but it may be relevant.

  And it's inspired by the paper ["Types from Data: Making Structured Data First-Class Citizens in F#"](https://dl.acm.org/citation.cfm?id=2908115).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1172697090
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/235
node_id: IC_kwDOCGYnMM5F5fAC
created_at: 2022-07-01T20:37:40Z
updated_at: 2022-07-01T20:37:55Z
author_association: NONE
body:
  I just ran into what appears to be the same issue on a MacBook Pro, M1 Pro. 

  Environment:
  ```
  markd@Marks-MacBook-Pro metabase % python --version                                                             
  Python 3.8.9
  markd@Marks-MacBook-Pro metabase % sqlite3 --version                                                            
  3.37.0 2021-12-09 01:34:53 9ff244ce0739f8ee52a3e9671adb4ee54c83c640b02e3f9d185fd2f9a179aapl
  markd@Marks-MacBook-Pro metabase % sqlite-utils --version                                                        
  sqlite-utils, version 3.27
  markd@Marks-MacBook-Pro metabase % sqlite3 gh.db 
  SQLite version 3.37.0 2021-12-09 01:34:53
  Enter ".help" for usage hints.
  sqlite> .dbconfig
            defensive off
              dqs_ddl on
              dqs_dml on
          enable_fkey off
          enable_qpsg off
       enable_trigger on
          enable_view on
       fts3_tokenizer off
   legacy_alter_table on
   legacy_file_format off
       load_extension off
     no_ckpt_on_close off
       reset_database off
          trigger_eqp off
       trusted_schema on
      writable_schema off
  ``` 

  Error
  ```
  markd@Marks-MacBook-Pro metabase % github-to-sqlite repos gh.db a8cteam51                                        
  Traceback (most recent call last):
    File "/Users/markd/Library/Python/3.8/bin/github-to-sqlite", line 8, in <module>
      sys.exit(cli())
    File "/Users/markd/Library/Python/3.8/lib/python/site-packages/click/core.py", line 1130, in __call__
      return self.main(*args, **kwargs)
    File "/Users/markd/Library/Python/3.8/lib/python/site-packages/click/core.py", line 1055, in main
      rv = self.invoke(ctx)
    File "/Users/markd/Library/Python/3.8/lib/python/site-packages/click/core.py", line 1657, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/Users/markd/Library/Python/3.8/lib/python/site-packages/click/core.py", line 1404, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/markd/Library/Python/3.8/lib/python/site-packages/click/core.py", line 760, in invoke
      return __callback(*args, **kwargs)
    File "/Users/markd/Library/Python/3.8/lib/python/site-packages/github_to_sqlite/cli.py", line 268, in repos
      utils.ensure_db_shape(db)
    File "/Users/markd/Library/Python/3.8/lib/python/site-packages/github_to_sqlite/utils.py", line 688, in ensure_db_shape
      ensure_foreign_keys(db)
    File "/Users/markd/Library/Python/3.8/lib/python/site-packages/github_to_sqlite/utils.py", line 682, in ensure_foreign_keys
      db[table].add_foreign_key(column, table2, column2)
    File "/Users/markd/Library/Python/3.8/lib/python/site-packages/sqlite_utils/db.py", line 2004, in add_foreign_key
      self.db.add_foreign_keys([(self.name, column, other_table, other_column)])
    File "/Users/markd/Library/Python/3.8/lib/python/site-packages/sqlite_utils/db.py", line 1019, in add_foreign_keys
      cursor.execute(
  sqlite3.OperationalError: table sqlite_master may not be modified
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1172766270
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/235
node_id: IC_kwDOCGYnMM5F5v4-
created_at: 2022-07-01T22:40:26Z
updated_at: 2022-07-01T22:40:26Z
author_association: NONE
body:
  Note, I do not get this issue using my Intel MacBook Pro =/

  Environment
  ```
  markd@Marks-MBP metabase % python3 --version
  Python 3.9.13
  markd@Marks-MBP metabase % sqlite3 --version
  3.37.0 2021-12-09 01:34:53 9ff244ce0739f8ee52a3e9671adb4ee54c83c640b02e3f9d185fd2f9a179aapl
  markd@Marks-MBP metabase % sqlite-utils --version
  sqlite-utils, version 3.27
  markd@Marks-MBP metabase % sqlite3 github.db 
  SQLite version 3.37.0 2021-12-09 01:34:53
  Enter ".help" for usage hints.
  sqlite> .dbconfig
            defensive off
              dqs_ddl on
              dqs_dml on
          enable_fkey off
          enable_qpsg off
       enable_trigger on
          enable_view on
       fts3_tokenizer off
   legacy_alter_table on
   legacy_file_format off
       load_extension off
     no_ckpt_on_close off
       reset_database off
          trigger_eqp off
       trusted_schema on
      writable_schema off
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/449#issuecomment-1172902594
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/449
node_id: IC_kwDOCGYnMM5F6RLC
created_at: 2022-07-02T13:55:51Z
updated_at: 2022-07-02T13:55:51Z
author_association: OWNER
body:
  I like the idea of this as a feature - design could look like this:

      db["my_table"].duplicate("new_table")

  Then for the CLI tool:

      sqlite-utils duplicate data.db my_table new_table

  Additional features that might be useful:

  - Duplicate across connection aliases - so you can copy a table to another database using the `duplicate` method or command
  - Ability to execute an arbitrary SQL query and save the results to a new table - basically a wrapper around `CREATE TABLE t AS SELECT ...`, again with alias support for copying to another database

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/427#issuecomment-1172903355
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/427
node_id: IC_kwDOCGYnMM5F6RW7
created_at: 2022-07-02T13:59:30Z
updated_at: 2022-07-02T13:59:30Z
author_association: OWNER
body:
  This should work to avoid that:

      sqlite-utils convert \
        ${dbfile} details dob \
        'r.parsedate(value)' \
        --where "dob != '*'"

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1646#issuecomment-1172930092
issue_url: https://api.github.com/repos/simonw/datasette/issues/1646
node_id: IC_kwDOBm6k_c5F6X4s
created_at: 2022-07-02T17:12:18Z
updated_at: 2022-07-02T17:12:18Z
author_association: NONE
body:
  I'm affected by this as well. Would be nice to be able to pass in an extension, eg `--extension=sqlite3`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1646#issuecomment-1172970910
issue_url: https://api.github.com/repos/simonw/datasette/issues/1646
node_id: IC_kwDOBm6k_c5F6h2e
created_at: 2022-07-02T22:13:13Z
updated_at: 2022-07-02T22:17:16Z
author_association: OWNER
body:
  Adding `.sqlite3` to this feature makes sense to me.

  I don't think it's worth having a `--extensión` option - it's possible right now to write a Datasette plugin that loads additional databases on startup, there's an example of how to do that in the source code for https://github.com/simonw/datasette-upload-dbs

  Here: https://github.com/simonw/datasette-upload-dbs/blob/81510ef4f0004371a3adda18100936b430535cc1/datasette_upload_dbs/__init__.py#L41-L50

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1766#issuecomment-1173142896
issue_url: https://api.github.com/repos/simonw/datasette/issues/1766
node_id: IC_kwDOBm6k_c5F7L1w
created_at: 2022-07-03T17:43:51Z
updated_at: 2022-07-03T17:43:51Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1766?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1766](https://codecov.io/gh/simonw/datasette/pull/1766?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (257fd24) into [main](https://codecov.io/gh/simonw/datasette/commit/9f1eb0d4eac483b953392157bd9fd6cc4df37de7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (9f1eb0d) will **increase** coverage by `0.00%`.
  > The diff coverage is `100.00%`.

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1766   +/-   ##
  =======================================
    Coverage   91.67%   91.67%           
  =======================================
    Files          36       36           
    Lines        4658     4659    +1     
  =======================================
  + Hits         4270     4271    +1     
    Misses        388      388           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1766?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1766/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.27% <100.00%> (+<0.01%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1766?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1766?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [9f1eb0d...257fd24](https://codecov.io/gh/simonw/datasette/pull/1766?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1713#issuecomment-1173358747
issue_url: https://api.github.com/repos/simonw/datasette/issues/1713
node_id: IC_kwDOBm6k_c5F8Aib
created_at: 2022-07-04T05:16:35Z
updated_at: 2022-07-04T05:16:35Z
author_association: CONTRIBUTOR
body:
  This feature is pretty important and would be nice if it would be all within Datasette (no separate CLI/deploy required). My workflow now is to basically just copy the result and paste into a Google Sheet, which works, but then it's not discoverable to other journalists browsing the Datasette instance. I started building a plugin similar to [datasette-saved-queries](https://datasette.io/plugins/datasette-saved-queries) but one that maintains its own DB (required if you're working with all immutable DBs), but got bogged down in details.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/449#issuecomment-1174027079
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/449
node_id: IC_kwDOCGYnMM5F-jtH
created_at: 2022-07-04T17:33:04Z
updated_at: 2022-07-04T17:48:43Z
author_association: CONTRIBUTOR
body:
  I've written the code and test. Would you be able to advise how to compare table columns in a pytest function properly? Experiencing a challenge when comparing columns.

  Test:
  ```python
  def test_duplicate(fresh_db):
      table = fresh_db.create_table(
          "table1",
          {
              "text_col": str,
              "float_col": float,
              "int_col": int,
              "bool_col": bool,
              "bytes_col": bytes,
              "datetime_col": datetime.datetime,
          },
      )
      dt = datetime.datetime.now()
      b = bytes('hello world', 'utf-8')
      data = {"text_col": "Cleo", 
              "float_col": 3.14,
              "int_col": -2,
              "bool_col": True,
              "bytes_col": b,
              "datetime_col": str(dt)}
      table1 = fresh_db["table1"]
      row_id = table1.insert(data).last_rowid
      table1.duplicate('table2')
      table2 = fresh_db["table2"]
      assert data == table2.get(row_id)
      assert table1.columns == table2.columns    # FAILS HERE
  ```

  Result:
  ![Screenshot 2022-07-05 at 1 31 55 AM](https://user-images.githubusercontent.com/1690072/177198814-daac48c9-5746-49d0-a14a-14fe181c5a2f.png)

  Failure is due to column types being named differently -- e.g. 'FLOAT' vs 'REAL', 'INTEGER' vs 'INT'. How should I go about comparing columns while accounting for equivalent types?

  Or did I miss out something in my duplication code correctly? Here's how I did it: in `db.py`, I've added the following code:
  ```python
  class Table(Queryable):
      [...]
      def duplicate(
          self, 
          name_new: str
      ) -> "Table":
          """
          Duplicate this table in this database.

          :param name_new: Name of new table.
          """
          assert self.exists()
          with self.db.conn:
              sql = "CREATE TABLE [{new_table}] AS SELECT * FROM [{table}];".format(
                  new_table = name_new,
                  table = self.name,
              )
              self.db.execute(sql)
          return self.db[name_new]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1767#issuecomment-1175376866
issue_url: https://api.github.com/repos/simonw/datasette/issues/1767
node_id: IC_kwDOBm6k_c5GDtPi
created_at: 2022-07-05T18:43:12Z
updated_at: 2022-07-05T18:43:12Z
author_association: OWNER
body:
  Some options:

  - Allow `favicon` path to be set in `metadata.yml` somehow, or via a `--setting`
  - Teach Datasette to check for `templates/favicon.ico` and fall back to the default `static/favicon.png` if that file does not exist
  - Document how to over-ride the favicon using the existing `--static` mechanism (if that's even possible)
  - Outsource this to a new plugin

  Note that I need a way to set the `content_type` correctly too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1767#issuecomment-1175377618
issue_url: https://api.github.com/repos/simonw/datasette/issues/1767
node_id: IC_kwDOBm6k_c5GDtbS
created_at: 2022-07-05T18:44:08Z
updated_at: 2022-07-05T18:44:08Z
author_association: OWNER
body:
  Loading it using the existing templating system would be better I think, since that way both custom installations AND plugins could influence the favicon in the same way that they influence the templates.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1767#issuecomment-1175378678
issue_url: https://api.github.com/repos/simonw/datasette/issues/1767
node_id: IC_kwDOBm6k_c5GDtr2
created_at: 2022-07-05T18:45:23Z
updated_at: 2022-07-05T18:45:23Z
author_association: OWNER
body:
  The content type thing is a bit weird because usually I'd base that on the file extension, but here the `favicon.ico` file extension doesn't necessarily reflect if the image itself is a PNG or some other thing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1767#issuecomment-1175381896
issue_url: https://api.github.com/repos/simonw/datasette/issues/1767
node_id: IC_kwDOBm6k_c5GDueI
created_at: 2022-07-05T18:48:14Z
updated_at: 2022-07-05T18:48:50Z
author_association: OWNER
body:
  Annoyingly it looks like the standard library `mimetypes` module only uses filenames as the clue, it doesn't look at the bytes themselves.

  I'm using that here: https://github.com/simonw/datasette/blob/9f1eb0d4eac483b953392157bd9fd6cc4df37de7/datasette/utils/asgi.py#L261-L277

  https://pypi.org/project/python-magic/ can inspect files, but I don't want to add a whole new dependency just for this one feature.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1767#issuecomment-1175385249
issue_url: https://api.github.com/repos/simonw/datasette/issues/1767
node_id: IC_kwDOBm6k_c5GDvSh
created_at: 2022-07-05T18:49:52Z
updated_at: 2022-07-05T18:49:52Z
author_association: OWNER
body:
  I could use the `imghdr` standard library module, but frustratingly it's marked as deprecated in Python 3.11! https://docs.python.org/3/library/imghdr.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1767#issuecomment-1175388492
issue_url: https://api.github.com/repos/simonw/datasette/issues/1767
node_id: IC_kwDOBm6k_c5GDwFM
created_at: 2022-07-05T18:51:26Z
updated_at: 2022-07-05T18:51:26Z
author_association: OWNER
body:
  I could vendor the necessary parts of `imghdr` - it's pretty tiny: https://github.com/python/cpython/blob/3.11/Lib/imghdr.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1767#issuecomment-1175391899
issue_url: https://api.github.com/repos/simonw/datasette/issues/1767
node_id: IC_kwDOBm6k_c5GDw6b
created_at: 2022-07-05T18:53:07Z
updated_at: 2022-07-05T18:53:07Z
author_association: OWNER
body:
  https://www.w3.org/2005/10/howto-favicon suggests that it only needs to be able to identify PNG, GIF or ICO.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1767#issuecomment-1175396774
issue_url: https://api.github.com/repos/simonw/datasette/issues/1767
node_id: IC_kwDOBm6k_c5GDyGm
created_at: 2022-07-05T18:56:43Z
updated_at: 2022-07-05T18:56:43Z
author_association: OWNER
body:
  Based on https://github.com/python/cpython/blob/3.11/Lib/imghdr.py I'm tempted to say that if the file starts with `b'\211PNG\r\n\032\n'` then it's a PNG, if it starts with `b'GIF8` then it's a GIF, anything else I assume an ICO.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1768#issuecomment-1176550192
issue_url: https://api.github.com/repos/simonw/datasette/issues/1768
node_id: IC_kwDOBm6k_c5GILsw
created_at: 2022-07-06T18:39:11Z
updated_at: 2022-07-06T18:39:11Z
author_association: OWNER
body:
  Relevant Twitter thread: https://twitter.com/simonw/status/1544710944462254080

  Looks like `python:3.10-slim-bullseye` is a good option, see https://pythonspeed.com/articles/base-image-python-docker-images/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1300#issuecomment-1177901762
issue_url: https://api.github.com/repos/simonw/datasette/issues/1300
node_id: IC_kwDOBm6k_c5GNVrC
created_at: 2022-07-07T16:34:05Z
updated_at: 2022-07-07T16:34:05Z
author_association: OWNER
body:
  Updated documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#render-cell-row-value-column-table-database-datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1768#issuecomment-1177966193
issue_url: https://api.github.com/repos/simonw/datasette/issues/1768
node_id: IC_kwDOBm6k_c5GNlZx
created_at: 2022-07-07T17:25:22Z
updated_at: 2022-07-07T17:25:22Z
author_association: OWNER
body:
  I tried this on Datasette Cloud and swapping `datasetteproject/datasette` to `python:3.10-slim-bullseye` as the base image dropped the overall image size from ~232MB to ~166MB - but note that I don't have SpatiaLite in that second image.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/449#issuecomment-1179579878
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/449
node_id: IC_kwDOCGYnMM5GTvXm
created_at: 2022-07-09T17:41:32Z
updated_at: 2022-07-09T17:41:50Z
author_association: CONTRIBUTOR
body:
  Learnt that the types in Sqlite-utils differ somewhat from those in Sqlite. I've changed my test to account for this difference and the test has passed successfully. I will submit a PR.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1048#issuecomment-1179756391
issue_url: https://api.github.com/repos/simonw/datasette/issues/1048
node_id: IC_kwDOBm6k_c5GUadn
created_at: 2022-07-10T16:12:23Z
updated_at: 2022-07-10T16:12:23Z
author_association: OWNER
body:
  Also related: `row` is now available to `render_cell()` hook as-of this issue:
  - https://github.com/simonw/datasette/issues/1300

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1048#issuecomment-1179756689
issue_url: https://api.github.com/repos/simonw/datasette/issues/1048
node_id: IC_kwDOBm6k_c5GUaiR
created_at: 2022-07-10T16:14:03Z
updated_at: 2022-07-10T16:14:03Z
author_association: OWNER
body:
  I'm considering changing these functions to accept the `row` object itself:

  ```python
   def row(self, database, table, row): 
       ...
    
   def row_blob(self, database, table, row, column): 
       ... 
  ```
  Just one catch: in order to generate the correct `row_path` we need to know the primary keys for that table. We can look those up based on having access to `database` and `table`, but doing so requires an `await ...` operation - and these functions are not `async`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1048#issuecomment-1179757067
issue_url: https://api.github.com/repos/simonw/datasette/issues/1048
node_id: IC_kwDOBm6k_c5GUaoL
created_at: 2022-07-10T16:16:19Z
updated_at: 2022-07-10T16:16:19Z
author_association: OWNER
body:
  None of the potential solutions for that problem are particularly appealing:

  - Make these URL generation methods `async` - should the other ones be `async` too for consistency?
  - Have some kind of mechanism that calculates and caches the `pks` for each table somewhere which can then be accessed here without an `async` call
  - Require `pks` is passed to these methods, having been looked up elsewhere. This is a bit gross but may end up being the best solution

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1048#issuecomment-1179757605
issue_url: https://api.github.com/repos/simonw/datasette/issues/1048
node_id: IC_kwDOBm6k_c5GUawl
created_at: 2022-07-10T16:19:42Z
updated_at: 2022-07-10T16:21:03Z
author_association: OWNER
body:
  If I do require `pks` to be passed here, maybe I could make those available to the `render_cell()` plugin hook to at least make this a bit more pleasant for plugin authors to use?

  Current hook: https://docs.datasette.io/en/latest/plugin_hooks.html#render-cell-row-value-column-table-database-datasette

  https://github.com/simonw/datasette/blob/035dc5e7b95142d4a700819a8cc4ff64aefe4efe/datasette/hookspecs.py#L62-L64

  The hook is called in two places in the codebase - when rendering a table (`pks` variable is already in scope here):

  https://github.com/simonw/datasette/blob/6373bb341457e5becfd5b67792ac2c8b9ed7c384/datasette/views/table.py#L897-L904

  And when rendering an arbitrary query:

  https://github.com/simonw/datasette/blob/6373bb341457e5becfd5b67792ac2c8b9ed7c384/datasette/views/database.py#L377-L384

  Note that in that second one `table` is `None` (which is also called out in the documentation) - `pks` would be `None` here too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1048#issuecomment-1179757912
issue_url: https://api.github.com/repos/simonw/datasette/issues/1048
node_id: IC_kwDOBm6k_c5GUa1Y
created_at: 2022-07-10T16:21:38Z
updated_at: 2022-07-10T16:21:38Z
author_association: OWNER
body:
  I think the best way to do this is to change the method signatures to:
  ```python
   def row(self, database, table, row, pks): 
       ...
    
   def row_blob(self, database, table, row, pks, column): 
       ... 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1048#issuecomment-1179758180
issue_url: https://api.github.com/repos/simonw/datasette/issues/1048
node_id: IC_kwDOBm6k_c5GUa5k
created_at: 2022-07-10T16:23:34Z
updated_at: 2022-07-10T16:23:57Z
author_association: OWNER
body:
  But do I need to pass the `use_rowid` boolean here as well, as used by `def path_from_row_pks(row, pks, use_rowid, quote=True)`? Or can I derive that from the fact that `pks` is an empty tuple?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1769#issuecomment-1185536255
issue_url: https://api.github.com/repos/simonw/datasette/issues/1769
node_id: IC_kwDOBm6k_c5Gqdj_
created_at: 2022-07-15T13:16:42Z
updated_at: 2022-07-15T13:16:42Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1769?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1769](https://codecov.io/gh/simonw/datasette/pull/1769?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (bb3c610) into [main](https://codecov.io/gh/simonw/datasette/commit/950cc7677f65aa2543067b3bbfc2b6acb98b62c8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (950cc76) will **not change** coverage.
  > The diff coverage is `n/a`.

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1769   +/-   ##
  =======================================
    Coverage   91.67%   91.67%           
  =======================================
    Files          36       36           
    Lines        4658     4658           
  =======================================
    Hits         4270     4270           
    Misses        388      388           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1769?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1769?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [950cc76...bb3c610](https://codecov.io/gh/simonw/datasette/pull/1769?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1770#issuecomment-1185925081
issue_url: https://api.github.com/repos/simonw/datasette/issues/1770
node_id: IC_kwDOBm6k_c5Gr8fZ
created_at: 2022-07-15T20:55:41Z
updated_at: 2022-07-15T20:56:16Z
author_association: OWNER
body:
  I think the hook gets called any time any exception makes it to this function: https://github.com/simonw/datasette/blob/950cc7677f65aa2543067b3bbfc2b6acb98b62c8/datasette/app.py#L1374-L1440

  Multiple plugins can register for the hook. If they return a `Response` then that's returned to the user - if they return `None` then they can quietly do something with the error (log it to Sentry for example) and let some other handler return a response.

  I think Datasette should have a default plugin hook implementation which returns the 500 error page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1770#issuecomment-1185929360
issue_url: https://api.github.com/repos/simonw/datasette/issues/1770
node_id: IC_kwDOBm6k_c5Gr9iQ
created_at: 2022-07-15T20:58:11Z
updated_at: 2022-07-15T20:58:11Z
author_association: OWNER
body:
  Proposed hook design:
  ```python
  @hookspec
  def handle_exception(datasette, request, exception):
      """Handle an uncaught exception"""
  ```
  It takes `request` in case it needs to render a template and pass the request to it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1770#issuecomment-1185931417
issue_url: https://api.github.com/repos/simonw/datasette/issues/1770
node_id: IC_kwDOBm6k_c5Gr-CZ
created_at: 2022-07-15T20:59:25Z
updated_at: 2022-07-15T20:59:25Z
author_association: OWNER
body:
  ... maybe it should take `send`? But then how would plugins know that another plugin hadn't already used `send` to send a response, and avoid two trying to send at the same time?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1770#issuecomment-1185933972
issue_url: https://api.github.com/repos/simonw/datasette/issues/1770
node_id: IC_kwDOBm6k_c5Gr-qU
created_at: 2022-07-15T21:00:56Z
updated_at: 2022-07-15T21:00:56Z
author_association: OWNER
body:
  It's weird to return a `Response` though because the code in question lives in `DatasetteRouter` which currently works outside of the layer where responses happen - everything in that class at the moment works using `send` directly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1770#issuecomment-1185935764
issue_url: https://api.github.com/repos/simonw/datasette/issues/1770
node_id: IC_kwDOBm6k_c5Gr_GU
created_at: 2022-07-15T21:02:00Z
updated_at: 2022-07-15T21:02:12Z
author_association: OWNER
body:
  Returning a `Response` is by far the most intuitive way to design this though. Plugin authors shouldn't care that `DatasetteRouter` (an undocumented internal API of Datasette) doesn't currently handle responses.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1770#issuecomment-1185937664
issue_url: https://api.github.com/repos/simonw/datasette/issues/1770
node_id: IC_kwDOBm6k_c5Gr_kA
created_at: 2022-07-15T21:03:09Z
updated_at: 2022-07-17T22:25:44Z
author_association: OWNER
body:
  Design decision:

  ```python
  @hookspec
  def handle_exception(datasette, request, exception):
      """Handle an uncaught exception. Can return a Response or None."""
  ```
  It will also support returning an awaitable, using the `await_me_maybe` pattern.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1770#issuecomment-1185943887
issue_url: https://api.github.com/repos/simonw/datasette/issues/1770
node_id: IC_kwDOBm6k_c5GsBFP
created_at: 2022-07-15T21:07:53Z
updated_at: 2022-07-15T21:07:53Z
author_association: OWNER
body:
  In `DatasetteRouter` I'm going to rename `handle_500` to `handle_exception` because it already deals with error codes other than 500 (`Forbidden` exceptions become 403 and `Base400` become 400).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1770#issuecomment-1185944799
issue_url: https://api.github.com/repos/simonw/datasette/issues/1770
node_id: IC_kwDOBm6k_c5GsBTf
created_at: 2022-07-15T21:09:08Z
updated_at: 2022-07-15T21:09:08Z
author_association: OWNER
body:
  Had to lookup that `Base400` thing: https://github.com/simonw/datasette/blob/950cc7677f65aa2543067b3bbfc2b6acb98b62c8/datasette/utils/asgi.py#L16-L29

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/452#issuecomment-1185949850
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/452
node_id: IC_kwDOCGYnMM5GsCia
created_at: 2022-07-15T21:15:45Z
updated_at: 2022-07-15T21:18:34Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/452?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#452](https://codecov.io/gh/simonw/sqlite-utils/pull/452?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (eef350f) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/42440d6345c242ee39778045e29143fb550bd2c2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (42440d6) will **increase** coverage by `0.00%`.
  > The diff coverage is `100.00%`.

  ```diff
  @@           Coverage Diff           @@
  ##             main     #452   +/-   ##
  =======================================
    Coverage   96.60%   96.61%           
  =======================================
    Files           6        6           
    Lines        2534     2540    +6     
  =======================================
  + Hits         2448     2454    +6     
    Misses         86       86           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/452?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/452/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.53% <100.00%> (+0.01%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/452?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/452?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [42440d6...eef350f](https://codecov.io/gh/simonw/sqlite-utils/pull/452?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/449#issuecomment-1185970114
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/449
node_id: IC_kwDOCGYnMM5GsHfC
created_at: 2022-07-15T21:46:13Z
updated_at: 2022-07-15T21:46:13Z
author_association: OWNER
body:
  Documentation:
  - https://sqlite-utils.datasette.io/en/latest/python-api.html#duplicating-tables
  - https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.duplicate

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/454#issuecomment-1185971600
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/454
node_id: IC_kwDOCGYnMM5GsH2Q
created_at: 2022-07-15T21:48:23Z
updated_at: 2022-07-15T21:48:23Z
author_association: OWNER
body:
  Documentation:
  - https://sqlite-utils.datasette.io/en/latest/cli.html#duplicating-tables
  - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#duplicate

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/453#issuecomment-1185974145
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/453
node_id: IC_kwDOCGYnMM5GsIeB
created_at: 2022-07-15T21:52:18Z
updated_at: 2022-07-15T21:52:18Z
author_association: OWNER
body:
  I should warn you that this isn't a supported API - I reserve the right to change how it works between release without a major version bump, because it's not part of the documented API surface.

  You'll be fine if you pin to exact versions of the library though!

  You may find this recently-documented function useful though: https://sqlite-utils.datasette.io/en/latest/python-api.html#reading-rows-from-a-file

  See:
  - #443

  I'm going to close this issue for the moment, but if anyone wants to submit a PR that cleans up this I'll happily review it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/451#issuecomment-1185975132
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/451
node_id: IC_kwDOCGYnMM5GsItc
created_at: 2022-07-15T21:53:42Z
updated_at: 2022-07-15T21:53:42Z
author_association: OWNER
body:
  Documentation can go here: https://sqlite-utils.datasette.io/en/stable/reference.html#sqlite-utils-utils

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1185982012
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/450
node_id: IC_kwDOCGYnMM5GsKY8
created_at: 2022-07-15T22:02:59Z
updated_at: 2022-07-15T22:03:49Z
author_association: OWNER
body:
  From https://sqlite-utils.datasette.io/en/latest/cli-reference.html

  Commands that currently have an `--ignore` flag, and their descriptions:

  - `insert`: Ignore records if pk already exists
  - `create-table`: If table already exists, do nothing
  - `add-foreign-key`: If foreign key already exists, do nothing
  - `drop-table`: *description is missing*
  - `create-view`: If view already exists, do nothing
  - `drop-view`: *description is missing*

  And `create-index` has:

  - `--if-not-exists`: Ignore if index already exists

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1185983894
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/450
node_id: IC_kwDOCGYnMM5GsK2W
created_at: 2022-07-15T22:06:29Z
updated_at: 2022-07-15T22:37:20Z
author_association: OWNER
body:
  Commands that could have `--ignore` added to them:

  - [x] `enable-fts`
  - [x] `disable-fts` (not doing)
  - [x] `add-column`
  - [x] `add-foreign-keys` (decided not to do this)
  - [x] `duplicate`
  - [x] `add-geometry-column` (decided not to do this)
  - [x] `create-spatial-index` (decided not to do this)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1185988277
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/450
node_id: IC_kwDOCGYnMM5GsL61
created_at: 2022-07-15T22:11:52Z
updated_at: 2022-07-15T22:11:52Z
author_association: OWNER
body:
  For `enable-fts` I should use `--replace` instead, since that matches how the Python API works: https://github.com/simonw/sqlite-utils/blob/9dd4cf891d9f4565019e030ddc712507ac87b998/sqlite_utils/db.py#L2088-L2106

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1185993791
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/450
node_id: IC_kwDOCGYnMM5GsNQ_
created_at: 2022-07-15T22:23:02Z
updated_at: 2022-07-15T22:23:02Z
author_association: OWNER
body:
  `create-spatial-index` is tricky. For consistency with `create-index` it should really use `--if-not-exists` - but under the hood it's not using `IF NOT EXISTS` because those indexes are created using `select CreateSpatialIndex(...)`.

  Really this highlights that `--if-not-exists` is a bad name for that option on `create-index`.

  I'm tempted to add `--ignore` to `create-index` and mark `--if-not-exists` as deprecated - though it will still work because I don't want to release a major version bump.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1185994282
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/450
node_id: IC_kwDOCGYnMM5GsNYq
created_at: 2022-07-15T22:24:12Z
updated_at: 2022-07-15T22:24:12Z
author_association: OWNER
body:
  I can do this for `--create-index` instead:
  ```
  Options:
    --name TEXT                Explicit name for the new index
    --unique                   Make this a unique index
    --if-not-exists, --ignore  Ignore if index already exists
    --analyze                  Run ANALYZE after creating the index
    --load-extension TEXT      SQLite extensions to load
    -h, --help                 Show this message and exit.
  ```
  That supports both names for the option.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1186002019
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/450
node_id: IC_kwDOCGYnMM5GsPRj
created_at: 2022-07-15T22:39:09Z
updated_at: 2022-07-15T22:39:09Z
author_association: OWNER
body:
  Here are all of the changes I made in this issue: https://github.com/simonw/sqlite-utils/compare/e10536c7f59abbb785f092bf83c4ab94c00e31a3...b9a89a0f2c3559989efe65f25a6e1f8fa76fe8b0

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/448#issuecomment-1186002560
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/448
node_id: IC_kwDOCGYnMM5GsPaA
created_at: 2022-07-15T22:40:40Z
updated_at: 2022-07-15T22:40:40Z
author_association: OWNER
body:
  This is very strange. GitHub Actions CI here runs against Windows and installs OK.

  Marking this as "Help wanted" to see if anyone can figure out what's going on here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1770#issuecomment-1186620168
issue_url: https://api.github.com/repos/simonw/datasette/issues/1770
node_id: IC_kwDOBm6k_c5GumMI
created_at: 2022-07-17T22:37:01Z
updated_at: 2022-07-17T22:37:01Z
author_association: OWNER
body:
  I need to refactor this code so that `Forbidden` exceptions are handled separately.

  Reason is that those already have a plugin hook of their own:

  https://github.com/simonw/datasette/blob/8188f55efc0fcca1be692b0d0c875f2d1ee99f17/datasette/app.py#L1384-L1395

  My first attempt at this refactored that entire `handle_exception` method to call the new plugin hook, moving its current implementation into a `datasette/handle_exception.py` default plugin implementation - but it felt weird having that plugin implementation then itself call the `pm.hook.forbidden()` hook.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1770#issuecomment-1186620367
issue_url: https://api.github.com/repos/simonw/datasette/issues/1770
node_id: IC_kwDOBm6k_c5GumPP
created_at: 2022-07-17T22:38:25Z
updated_at: 2022-07-17T22:38:25Z
author_association: OWNER
body:
  I think this is the right place to move the code to catch `Forbidden` exceptions (and forwards them to that plugin hook): https://github.com/simonw/datasette/blob/8188f55efc0fcca1be692b0d0c875f2d1ee99f17/datasette/app.py#L1269-L1278

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1770#issuecomment-1186629556
issue_url: https://api.github.com/repos/simonw/datasette/issues/1770
node_id: IC_kwDOBm6k_c5Guoe0
created_at: 2022-07-17T23:25:08Z
updated_at: 2022-07-17T23:25:08Z
author_association: OWNER
body:
  Keeping this issue open until I've proven the new plugin hook works by releasing a plugin that uses it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1770#issuecomment-1186630298
issue_url: https://api.github.com/repos/simonw/datasette/issues/1770
node_id: IC_kwDOBm6k_c5Guoqa
created_at: 2022-07-17T23:29:37Z
updated_at: 2022-07-17T23:29:37Z
author_association: OWNER
body:
  Documentation for the new hook: https://docs.datasette.io/en/latest/plugin_hooks.html#handle-exception-datasette-request-exception

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188223933
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74
node_id: IC_kwDODFdgUs5G0tu9
created_at: 2022-07-18T19:40:50Z
updated_at: 2022-07-18T19:42:41Z
author_association: MEMBER
body:
  Here's how the demo is deployed: https://github.com/dogsheep/github-to-sqlite/blob/dbac2e5dd8a562b45d8255a265859cf8020ca22a/.github/workflows/deploy-demo.yml#L103-L119

  I'm suspicious of `py-gfm`, which is used like this:

  https://github.com/dogsheep/github-to-sqlite/blob/dbac2e5dd8a562b45d8255a265859cf8020ca22a/demo-metadata.json#L49-L51

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188225625
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74
node_id: IC_kwDODFdgUs5G0uJZ
created_at: 2022-07-18T19:41:52Z
updated_at: 2022-07-18T19:41:52Z
author_association: MEMBER
body:
  https://github.com/Zopieux/py-gfm says that library is no longer maintained, and suggests https://github.com/Zopieux/pycmarkgfm as an alternative.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/74#issuecomment-1188228964
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/74
node_id: IC_kwDODFdgUs5G0u9k
created_at: 2022-07-18T19:45:30Z
updated_at: 2022-07-18T19:47:35Z
author_association: MEMBER
body:
  `pycmarkgfm` doesn't implement the Markdown plugin extension I was using.

  I'm going to drop the GFM rendering from the demo, and just treat it as regular markdown.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/903#issuecomment-1188338136
issue_url: https://api.github.com/repos/simonw/datasette/issues/903
node_id: IC_kwDOBm6k_c5G1JnY
created_at: 2022-07-18T21:34:33Z
updated_at: 2022-07-18T21:34:33Z
author_association: OWNER
body:
  Documented here: https://docs.datasette.io/en/latest/testing_plugins.html#registering-a-plugin-for-the-duration-of-a-test

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/455#issuecomment-1189130490
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/455
node_id: IC_kwDOCGYnMM5G4LD6
created_at: 2022-07-19T14:32:10Z
updated_at: 2022-07-19T14:32:10Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/455?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#455](https://codecov.io/gh/simonw/sqlite-utils/pull/455?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1b35a92) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/855bce8c3823718def13e0b8928c58bf857e41b2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (855bce8) will **not change** coverage.
  > The diff coverage is `n/a`.

  ```diff
  @@           Coverage Diff           @@
  ##             main     #455   +/-   ##
  =======================================
    Coverage   96.64%   96.64%           
  =======================================
    Files           6        6           
    Lines        2566     2566           
  =======================================
    Hits         2480     2480           
    Misses         86       86           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/455?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/455/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.54% <ø> (ø)` | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/455?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/455?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [855bce8...1b35a92](https://codecov.io/gh/simonw/sqlite-utils/pull/455?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/456#issuecomment-1189698289
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/456
node_id: IC_kwDOCGYnMM5G6Vrx
created_at: 2022-07-20T01:13:47Z
updated_at: 2022-07-20T01:13:47Z
author_association: OWNER
body:
  Really interesting idea! Can you flesh out what this might look like, maybe with a before and after example schema?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/456#issuecomment-1190272780
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/456
node_id: IC_kwDOCGYnMM5G8h8M
created_at: 2022-07-20T13:14:54Z
updated_at: 2022-07-20T13:14:54Z
author_association: CONTRIBUTOR
body:
  for example, i have data on votes that look like this:

  | ballot_id | option_id | choice |
  |-|-|-|
  | 1 | 1 | 0 | 
  | 1 | 2 | 1 |
  | 1 | 3 | 0 |
  | 1 | 4 | 1 |
  | 2 | 1 | 1 |
  | 2 | 2 | 0 |
  | 2 | 3 | 1 |
  | 2 | 4 | 0 |

  and i want to reshape from this long form to this wide form:

  | ballot_id | option_id_1 | option_id_2 | option_id_3 | option_id_ 4|
  |-|-|-|-| -|
  | 1 | 0 | 1 | 0 | 1 | 
  | 2 | 1 | 0 | 1| 0 | 

  i could do such a think like this.

  ```sql
  select ballot_id, 
  sum(choice) filter (where option_id = 1) as option_id_1,
  sum(choice) filter (where option_id = 2) as option_id_2,
  sum(choice) filter (where option_id = 3) as option_id_3,
  sum(choice) filter (where option_id = 4) as option_id_4
  from vote
  group by ballot_id
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/456#issuecomment-1190449764
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/456
node_id: IC_kwDOCGYnMM5G9NJk
created_at: 2022-07-20T15:45:54Z
updated_at: 2022-07-20T15:45:54Z
author_association: NONE
body:
  > hadley wickham's melt and reshape could be good inspo: http://had.co.nz/reshape/introduction.pdf

  Note that Hadley has since implemented `pivot_longer` and `pivot_wider` instead of the previous verbs/functions that he used. Those can be found in the tidyr package and are probably the best reference which includes all of the learnings from years of user feedback. https://tidyr.tidyverse.org/articles/pivot.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-1190995982
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31
node_id: IC_kwDOD079W85G_SgO
created_at: 2022-07-21T03:26:38Z
updated_at: 2023-04-14T22:41:31Z
author_association: NONE
body:
  👋 Any update on getting this merged?

  Alternatively, is there a work around for this issue to unblock myself?

  edit to add: huge fan of both this project and `osxphotos`, thanks so much for your work here 🙏  If I had any experience with Python I would offer to help but somehow I've managed to not write any Python in 10+ years of programming 😅 

  Edit again to add:

  > Alternatively, is there a work around for this issue to unblock myself?

  Yes, there is. I was able to apply the patch of this PR and it applies (mostly) cleanly and works.

  - verified I have a high enough version of `osxphotos`
  - downloaded the .patch of this (by appending `.patch` to the URL)
  - edited the patch to remove the `setup.py` changes
  - `cd` to the directory containing `dogsheep-photos` and `git apply 31.patch`

reactions:
  {"total_count": 4, "+1": 4, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/1636#issuecomment-1192862243
issue_url: https://api.github.com/repos/simonw/datasette/issues/1636
node_id: IC_kwDOBm6k_c5HGaIj
created_at: 2022-07-22T19:10:48Z
updated_at: 2022-07-22T19:11:01Z
author_association: OWNER
body:
  I keep running into a need for this. Every time I create a new plugin that defines a new permission I wish there was a clean way to grant that permission to new users without installing some other permissions plugin.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1175#issuecomment-1195442266
issue_url: https://api.github.com/repos/simonw/datasette/issues/1175
node_id: IC_kwDOBm6k_c5HQQBa
created_at: 2022-07-26T12:52:10Z
updated_at: 2022-07-26T12:52:10Z
author_association: NONE
body:
  I'm using this in a separate FastAPI app, worked perfectly when I changed the AsyncBoundLogger to BoundLogger only.

  Also for some reason, I'm now getting some logs surfacing from internal packages, like Elasticsearch. But don't have time to deal with that now.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1773#issuecomment-1196165351
issue_url: https://api.github.com/repos/simonw/datasette/issues/1773
node_id: IC_kwDOBm6k_c5HTAjn
created_at: 2022-07-27T01:24:13Z
updated_at: 2022-07-27T01:24:13Z
author_association: OWNER
body:
  Traceback:
  ```
  Traceback (most recent call last):
    File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 1264, in route_path
      response = await view(request, send)
    File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 134, in view
      return await self.dispatch_request(request)
    File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 91, in dispatch_request
      return await handler(request)
    File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 361, in get
      response_or_template_contexts = await self.data(request, **data_kwargs)
    File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 157, in data
      return await self._data_traced(request, default_labels, _next, _size)
    File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 633, in _data_traced
      prefix = rows[-2][sort or sort_desc]
  IndexError: No item with that key
  ```
  That's this code here: https://github.com/simonw/datasette/blob/7af67b54b7d9bca43e948510fc62f6db2b748fa8/datasette/views/table.py#L631-L645

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1773#issuecomment-1196166269
issue_url: https://api.github.com/repos/simonw/datasette/issues/1773
node_id: IC_kwDOBm6k_c5HTAx9
created_at: 2022-07-27T01:26:09Z
updated_at: 2022-07-27T01:26:09Z
author_association: OWNER
body:
  So the problem here is that in generating the `?_next=` next page link we need the value from the specified sort column - but we're not selecting it any more.

  Possible fixes:

  - Always include the sort column in the list of columns that are selected, then filter that out before they are displayed
  - Use a second query to figure out the `_sort` or `_sort_desc` value for that last row, since we know its primary key (we always select primary keys)

  Not sure which solution is more elegant. I think it might be the second one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1773#issuecomment-1196167102
issue_url: https://api.github.com/repos/simonw/datasette/issues/1773
node_id: IC_kwDOBm6k_c5HTA--
created_at: 2022-07-27T01:28:02Z
updated_at: 2022-07-27T01:28:02Z
author_association: OWNER
body:
  So code would look something like this:

  ```python
  try:
      prefix = rows[-2][sort or sort_desc]
  except KeyError:
      # Didn't select sort/sort_desc column - look up value by primary key instead
      primary_key = rows[-2]["pk"] # But more complex than this
      prefix = (await db.execute("select * from {table} where pk = ?", [primary_key])).first_value()
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/457#issuecomment-1197090890
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/457
node_id: IC_kwDOCGYnMM5HWihK
created_at: 2022-07-27T17:40:14Z
updated_at: 2022-07-27T17:40:40Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/457?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#457](https://codecov.io/gh/simonw/sqlite-utils/pull/457?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (80b3907) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/9e6cceac1c0e086429e2d308b700e59cc53a1991?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (9e6ccea) will **not change** coverage.
  > The diff coverage is `n/a`.

  ```diff
  @@           Coverage Diff           @@
  ##             main     #457   +/-   ##
  =======================================
    Coverage   96.64%   96.64%           
  =======================================
    Files           6        6           
    Lines        2566     2566           
  =======================================
    Hits         2480     2480           
    Misses         86       86           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/457?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/457?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [9e6ccea...80b3907](https://codecov.io/gh/simonw/sqlite-utils/pull/457?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/458#issuecomment-1197509096
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/458
node_id: IC_kwDOCGYnMM5HYIno
created_at: 2022-07-28T00:14:22Z
updated_at: 2022-07-28T00:14:22Z
author_association: OWNER
body:
  Updated documentation: https://github.com/simonw/sqlite-utils/blob/1491b66dd7439dd87cd5cd4c4684f46eb3c5751b/docs/python-api.rst#registering-custom-sql-functions

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1198414383
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/235
node_id: IC_kwDOCGYnMM5Hblov
created_at: 2022-07-28T17:10:06Z
updated_at: 2022-07-28T17:10:06Z
author_association: NONE
body:
  I was able to fight through this by capturing the SQL commands from the `add_foreign_keys()` function in sqlite-utils and then executing them manually via the sqlite3 client, first setting PRAGMA writable_schema on and then updating the sqlite_master table. Still no clue why they were failing when run in context...

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1272#issuecomment-1199115002
issue_url: https://api.github.com/repos/simonw/datasette/issues/1272
node_id: IC_kwDOBm6k_c5HeQr6
created_at: 2022-07-29T10:22:58Z
updated_at: 2022-07-29T10:22:58Z
author_association: NONE
body:
  > test_dockerfile.py



  > I'm skipping this for the moment because the new Dockerfile shape introduced in [#1249 (comment)](https://github.com/simonw/datasette/issues/1249#issuecomment-804404544) isn't compatible with this technique, since it installs Datasette from PyPI rather than directly from the repo.
  > 
  > Will need to change that if I want to do this unit tests thing.

  Hello,
  can't you copy in a later step directly in the output directory, e.g. COPY test_dockerfile.py /usr/lib/python*/.. or something like that ? 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1191#issuecomment-1200732975
issue_url: https://api.github.com/repos/simonw/datasette/issues/1191
node_id: IC_kwDOBm6k_c5Hkbsv
created_at: 2022-08-01T05:39:27Z
updated_at: 2022-08-01T05:39:27Z
author_association: CONTRIBUTOR
body:
  I've got a URL shortening plugin that I would like to embed on the query page but I'd like avoid capturing the entire `query.html` template. A feature like this would solve it. Where's this at and how can I help?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/460#issuecomment-1202947160
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/460
node_id: IC_kwDOCGYnMM5Hs4RY
created_at: 2022-08-02T16:20:42Z
updated_at: 2022-08-02T20:53:19Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/460?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#460](https://codecov.io/gh/simonw/sqlite-utils/pull/460?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (f21f4cc) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/1856002e3c0fcc9f09f72ab7d97ad8c75f6de7df?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1856002) will **not change** coverage.
  > The diff coverage is `n/a`.

  ```diff
  @@           Coverage Diff           @@
  ##             main     #460   +/-   ##
  =======================================
    Coverage   96.64%   96.64%           
  =======================================
    Files           6        6           
    Lines        2566     2566           
  =======================================
    Hits         2480     2480           
    Misses         86       86           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/460?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/460?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [1856002...f21f4cc](https://codecov.io/gh/simonw/sqlite-utils/pull/460?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/460#issuecomment-1202947222
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/460
node_id: IC_kwDOCGYnMM5Hs4SW
created_at: 2022-08-02T16:20:47Z
updated_at: 2022-08-02T16:20:47Z
author_association: OWNER
body:
  That preview link it added didn't work, maybe because I have a custom domain setup?

  It linked to: https://readthedocs-preview--460.org.readthedocs.build/en/460/ - which 404s

  Here's the correct link: https://sqlite-utils--460.org.readthedocs.build/en/460/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/461#issuecomment-1203168339
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/461
node_id: IC_kwDOCGYnMM5HtuRT
created_at: 2022-08-02T20:12:14Z
updated_at: 2022-08-02T20:12:14Z
author_association: OWNER
body:
  The neat thing about these is that they can be embedded directly in a GitHub README. Not sure if they show up on PyPI project pages, though I expect that they would.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/460#issuecomment-1203190312
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/460
node_id: IC_kwDOCGYnMM5Htzoo
created_at: 2022-08-02T20:36:58Z
updated_at: 2022-08-02T20:36:58Z
author_association: OWNER
body:
  > That preview link it added didn't work, maybe because I have a custom domain setup?

  Entirely my fault, fixed here :https://github.com/simonw/sqlite-utils/commit/98a28cbfe6cea67f6334b42b74f35b0ddd309561

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/462#issuecomment-1203207043
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/462
node_id: IC_kwDOCGYnMM5Ht3uD
created_at: 2022-08-02T20:57:30Z
updated_at: 2022-08-02T20:57:30Z
author_association: OWNER
body:
  I created a new link for this - https://discord.gg/Ass7bCAMDw - which should drop people straight into the `#sqlite-utils` channel in the Datasette Discord.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/462#issuecomment-1203207940
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/462
node_id: IC_kwDOCGYnMM5Ht38E
created_at: 2022-08-02T20:58:38Z
updated_at: 2022-08-02T21:05:58Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/462?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#462](https://codecov.io/gh/simonw/sqlite-utils/pull/462?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0a14a03) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/1856002e3c0fcc9f09f72ab7d97ad8c75f6de7df?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1856002) will **not change** coverage.
  > The diff coverage is `n/a`.

  ```diff
  @@           Coverage Diff           @@
  ##             main     #462   +/-   ##
  =======================================
    Coverage   96.64%   96.64%           
  =======================================
    Files           6        6           
    Lines        2566     2566           
  =======================================
    Hits         2480     2480           
    Misses         86       86           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/462?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/462?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [1856002...0a14a03](https://codecov.io/gh/simonw/sqlite-utils/pull/462?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1206241356
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/235
node_id: IC_kwDOCGYnMM5H5chM
created_at: 2022-08-05T09:26:15Z
updated_at: 2022-08-05T09:29:42Z
author_association: NONE
body:
  I am getting the same error when using github-to-sqlite (which uses sqlite-utils internally). I am also using an M1 MacBook Pro, with macOS Monterey 12.5, with Python 3.10.6 for arm64 installed using pyenv. I have sqlite-utils 3.28 installed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-1208757153
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51
node_id: IC_kwDODFdgUs5IDCuh
created_at: 2022-08-09T00:29:44Z
updated_at: 2022-08-09T00:29:44Z
author_association: NONE
body:
  I've been looking into how to to get this data out of Github (especially now there are "secondary rate limits" without an advertised allowance separate from the regular rate limits. 

  I've had decent success with the Airbyte github extractor (aside from one data quality issue https://github.com/airbytehq/airbyte/pull/15420 ). Airbyte splits data extraction between the GraphQL and REST endpoints depending on the resource type, but they're very comprehensive. 

  https://github.com/airbytehq/airbyte/blob/306a75ef5370728e0912cf52a1a898a530db0c90/airbyte-integrations/connectors/source-github/source_github/streams.py#L22-L122

  Before this, I tried a few solutions in my own custom wrapper mentioned in this thread + its children https://github.com/PyGithub/PyGithub/issues/1989 , but they weren't working as expected.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1779#issuecomment-1210675046
issue_url: https://api.github.com/repos/simonw/datasette/issues/1779
node_id: IC_kwDOBm6k_c5IKW9m
created_at: 2022-08-10T13:28:37Z
updated_at: 2022-08-10T13:28:37Z
author_association: CONTRIBUTOR
body:
  maybe a simpler solution is to set the maxscale to like 2? since datasette is not set up to make use of container scaling anyway?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/464#issuecomment-1214184512
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/464
node_id: IC_kwDOCGYnMM5IXvxA
created_at: 2022-08-13T16:20:41Z
updated_at: 2022-08-13T16:20:41Z
author_association: OWNER
body:
  https://github.com/beanbaginc/beanbag-docutils/blob/master/beanbag_docutils/sphinx/ext/github.py via https://twitter.com/chipx86/status/1558271211901050880 looks like a good option.

  It works using this Sphinx mechanism: https://www.sphinx-doc.org/en/master/usage/extensions/linkcode.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/464#issuecomment-1214184631
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/464
node_id: IC_kwDOCGYnMM5IXvy3
created_at: 2022-08-13T16:21:40Z
updated_at: 2022-08-13T16:21:40Z
author_association: OWNER
body:
  I tried it out and ran into a bug, which I've reported and submitted a PR for:

  - https://github.com/beanbaginc/beanbag-docutils/pull/1
  - https://github.com/beanbaginc/beanbag-docutils/pull/2

  I'm going to ship a version that uses my fork of the repo here: https://github.com/simonw/beanbag-docutils/tree/bytes-in-url

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/464#issuecomment-1214187919
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/464
node_id: IC_kwDOCGYnMM5IXwmP
created_at: 2022-08-13T16:46:01Z
updated_at: 2022-08-13T16:49:36Z
author_association: OWNER
body:
  Source links are now live here: https://sqlite-utils.datasette.io/en/latest/reference.html

  ![CleanShot 2022-08-13 at 09 49 02@2x](https://user-images.githubusercontent.com/9599/184503393-6559ab2b-cd02-4c85-98b9-62d10b3bdab8.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/464#issuecomment-1214199975
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/464
node_id: IC_kwDOCGYnMM5IXzin
created_at: 2022-08-13T18:09:02Z
updated_at: 2022-08-13T18:09:02Z
author_association: OWNER
body:
  I may be able to do this without that extra dependency: https://www.sphinx-doc.org/en/master/usage/extensions/linkcode.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1768#issuecomment-1214403875
issue_url: https://api.github.com/repos/simonw/datasette/issues/1768
node_id: IC_kwDOBm6k_c5IYlUj
created_at: 2022-08-14T15:46:49Z
updated_at: 2022-08-14T15:46:49Z
author_association: OWNER
body:
  https://hub.docker.com/layers/python/library/python/3.10.6-slim-bullseye/images/sha256-cf85cd32e60184a94d88a0103c289d09024abffaa77680d116d7cc837668ea15?context=explore is the most recent - `3.10.6-slim-bullseye`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1768#issuecomment-1214404690
issue_url: https://api.github.com/repos/simonw/datasette/issues/1768
node_id: IC_kwDOBm6k_c5IYlhS
created_at: 2022-08-14T15:52:02Z
updated_at: 2022-08-14T15:52:02Z
author_association: OWNER
body:
  Tested locally with:

      datasette package fixtures.db --extra-options="--setting sql_time_limit_ms 2500" \
        -t datasette-package-python-upgrade

  And then:

  ```
  docker run -p 8081:8001 datasette-package-python-upgrade
  INFO:     Started server process [8]
  INFO:     Waiting for application startup.
  INFO:     Application startup complete.
  INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)
  ```
  <img width="832" alt="image" src="https://user-images.githubusercontent.com/9599/184544853-bc2d06cc-ff9a-47f3-9744-e8916e531929.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1768#issuecomment-1214405023
issue_url: https://api.github.com/repos/simonw/datasette/issues/1768
node_id: IC_kwDOBm6k_c5IYlmf
created_at: 2022-08-14T15:54:11Z
updated_at: 2022-08-14T15:54:11Z
author_association: OWNER
body:
  Also trying this locally:

      datasette publish cloudrun fixtures.db --service issue-1768

  https://issue-1768-j7hipcg4aq-uc.a.run.app/-/versions

  <img width="775" alt="image" src="https://user-images.githubusercontent.com/9599/184544946-7cbce9c6-4cb3-4efa-96d8-1741afd1d6f4.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1773#issuecomment-1214407104
issue_url: https://api.github.com/repos/simonw/datasette/issues/1773
node_id: IC_kwDOBm6k_c5IYmHA
created_at: 2022-08-14T16:06:25Z
updated_at: 2022-08-14T16:06:25Z
author_association: OWNER
body:
  https://latest.datasette.io/fixtures/sortable?_sort_desc=sortable&_col=sortable_with_nulls works now.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1738#issuecomment-1214407462
issue_url: https://api.github.com/repos/simonw/datasette/issues/1738
node_id: IC_kwDOBm6k_c5IYmMm
created_at: 2022-08-14T16:08:23Z
updated_at: 2022-08-14T16:09:49Z
author_association: OWNER
body:
  This isn't just a mobile display issue.

  https://latest.datasette.io/fixtures/compound_three_primary_keys?_sort_desc=pk1 - click "Apply" and each time it adds another `?_sort_desc=pk1` to the URL.

  This doesn't happen for `?_sort=pk` - only for `?_sort_desc=pk1`: https://latest.datasette.io/fixtures/compound_three_primary_keys?_sort=pk1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1728#issuecomment-1214408558
issue_url: https://api.github.com/repos/simonw/datasette/issues/1728
node_id: IC_kwDOBm6k_c5IYmdu
created_at: 2022-08-14T16:14:36Z
updated_at: 2022-08-14T16:14:36Z
author_association: OWNER
body:
  Decision: I'm going to implement this:

  > Don't allow the user to even submit the form - show a message saying that this canned query is unavailable because the database cannot be written to

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1728#issuecomment-1214411623
issue_url: https://api.github.com/repos/simonw/datasette/issues/1728
node_id: IC_kwDOBm6k_c5IYnNn
created_at: 2022-08-14T16:34:20Z
updated_at: 2022-08-14T16:34:20Z
author_association: OWNER
body:
  Here's what it looks like:

  <img width="717" alt="image" src="https://user-images.githubusercontent.com/9599/184546269-53fc42bc-dce9-4345-a20b-b00a3c0046d7.png">

  I put the `disabled` attribute on the form submission button too. There's no visible difference in how that button is displayed but since this is only a state that should be seen by the developer I think that's OK, especially combined with the warning at the top of the page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1779#issuecomment-1214411951
issue_url: https://api.github.com/repos/simonw/datasette/issues/1779
node_id: IC_kwDOBm6k_c5IYnSv
created_at: 2022-08-14T16:36:55Z
updated_at: 2022-08-14T16:36:55Z
author_association: OWNER
body:
  I'm going to default maxscale to 2 but provide an extra command line option for `datasette publish cloudrun` which lets people set it to something higher if they want to.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1779#issuecomment-1214412135
issue_url: https://api.github.com/repos/simonw/datasette/issues/1779
node_id: IC_kwDOBm6k_c5IYnVn
created_at: 2022-08-14T16:38:08Z
updated_at: 2022-08-14T16:38:08Z
author_association: OWNER
body:
  Actually I disagree that Datasette isn't setup to make use of container scaling: part of the idea behind the [Baked Data](https://simonwillison.net/2021/Jul/28/baked-data/) pattern is that you can scale to handle effectively unlimited traffic by running multiple copies of your application, each with their own duplicate copy of the database.

  So I'm going to default maxScale to 10 and still let people customize it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1779#issuecomment-1214412382
issue_url: https://api.github.com/repos/simonw/datasette/issues/1779
node_id: IC_kwDOBm6k_c5IYnZe
created_at: 2022-08-14T16:40:02Z
updated_at: 2022-08-14T16:40:02Z
author_association: OWNER
body:
  Here's the relevant part of the `datasette publish` command from that failed Actions workflow:
  ```
    datasette publish cloudrun f7.db nlrb.db opdr.db old_nlrb.db voluntary_recognitions.db work_stoppages.db lm20.db chips.db \
      --memory 8Gi \
      --cpu 2
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1779#issuecomment-1214413161
issue_url: https://api.github.com/repos/simonw/datasette/issues/1779
node_id: IC_kwDOBm6k_c5IYnlp
created_at: 2022-08-14T16:45:36Z
updated_at: 2022-08-14T16:45:36Z
author_association: OWNER
body:
  Tried duplicating this error locally but the following command succeeded when I expected it to fail:

      datasette publish cloudrun fixtures.db --memory 8Gi --cpu 2 --service issue-1779

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1779#issuecomment-1214413419
issue_url: https://api.github.com/repos/simonw/datasette/issues/1779
node_id: IC_kwDOBm6k_c5IYnpr
created_at: 2022-08-14T16:47:27Z
updated_at: 2022-08-14T16:47:27Z
author_association: OWNER
body:
  Maybe I need to upgrade:
  ```
  % gcloud --version
  Google Cloud SDK 378.0.0
  alpha 2022.03.18
  bq 2.0.74
  core 2022.03.18
  gsutil 5.8
  Updates are available for some Google Cloud CLI components.  To install them,
  please run:
    $ gcloud components update
  ```
  ```
  % gcloud components update
  Beginning update. This process may take several minutes.


  Your current Google Cloud CLI version is: 378.0.0
  You will be upgraded to version: 397.0.0

  ┌─────────────────────────────────────────────────────────────────────────────┐
  │                      These components will be updated.                      │
  ├─────────────────────────────────────────────────────┬────────────┬──────────┤
  │                         Name                        │  Version   │   Size   │
  ├─────────────────────────────────────────────────────┼────────────┼──────────┤
  │ BigQuery Command Line Tool                          │     2.0.75 │  1.6 MiB │
  │ BigQuery Command Line Tool (Platform Specific)      │     2.0.75 │  < 1 MiB │
  │ Cloud Storage Command Line Tool                     │       5.11 │ 15.5 MiB │
  │ Cloud Storage Command Line Tool (Platform Specific) │       5.11 │  < 1 MiB │
  │ Google Cloud CLI Core Libraries                     │ 2022.08.05 │ 24.3 MiB │
  │ Google Cloud CLI Core Libraries (Platform Specific) │ 2022.08.05 │  < 1 MiB │
  │ anthoscli                                           │     0.2.28 │ 48.0 MiB │
  │ gcloud Alpha Commands                               │ 2022.08.05 │  < 1 MiB │
  │ gcloud cli dependencies                             │ 2022.07.29 │ 11.2 MiB │
  └─────────────────────────────────────────────────────┴────────────┴──────────┘

  A lot has changed since your last upgrade.  For the latest full release notes,
  please visit:
    https://cloud.google.com/sdk/release_notes
  ```
  ```
  % gcloud --version        
  Google Cloud SDK 397.0.0
  alpha 2022.08.05
  bq 2.0.75
  core 2022.08.05
  gsutil 5.11
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1779#issuecomment-1214413750
issue_url: https://api.github.com/repos/simonw/datasette/issues/1779
node_id: IC_kwDOBm6k_c5IYnu2
created_at: 2022-08-14T16:50:08Z
updated_at: 2022-08-14T16:50:08Z
author_association: OWNER
body:
  Just spotted this in the failing Actions workflow:

      gcloud config set run/region us-central1

  I tried that locally too but the deploy still succeeds.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1779#issuecomment-1214414013
issue_url: https://api.github.com/repos/simonw/datasette/issues/1779
node_id: IC_kwDOBm6k_c5IYny9
created_at: 2022-08-14T16:52:14Z
updated_at: 2022-08-14T16:52:14Z
author_association: OWNER
body:
  Just tried this instead, and it still worked and deployed OK:

      datasette publish cloudrun fixtures.db --memory 16Gi --cpu 4 --service issue-1779

  @fgregg I'm not able to replicate your deployment failure I'm afraid.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1779#issuecomment-1214414320
issue_url: https://api.github.com/repos/simonw/datasette/issues/1779
node_id: IC_kwDOBm6k_c5IYn3w
created_at: 2022-08-14T16:54:32Z
updated_at: 2022-08-14T16:54:32Z
author_association: OWNER
body:
  (I deleted my `issue-1779` project using the UI at https://console.cloud.google.com/run?project=datasette-222320)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1779#issuecomment-1214414471
issue_url: https://api.github.com/repos/simonw/datasette/issues/1779
node_id: IC_kwDOBm6k_c5IYn6H
created_at: 2022-08-14T16:55:40Z
updated_at: 2022-08-14T16:55:40Z
author_association: OWNER
body:
  Here's the start of the man page for `gcloud run deploy`:
  ```
  NAME
      gcloud run deploy - deploy a container to Cloud Run

  SYNOPSIS
      gcloud run deploy [[SERVICE] --namespace=NAMESPACE] [--args=[ARG,...]]
          [--async] [--command=[COMMAND,...]] [--concurrency=CONCURRENCY]
          [--cpu=CPU] [--ingress=INGRESS; default="all"]
          [--max-instances=MAX_INSTANCES] [--memory=MEMORY]
          [--min-instances=MIN_INSTANCES]
  ```
  I'm going to expose `--max-instances` and `--min-instances` as extra option to `datasette publish cloudrun`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1779#issuecomment-1214416491
issue_url: https://api.github.com/repos/simonw/datasette/issues/1779
node_id: IC_kwDOBm6k_c5IYoZr
created_at: 2022-08-14T17:07:34Z
updated_at: 2022-08-14T17:07:34Z
author_association: OWNER
body:
  Tested that with:

      datasette publish cloudrun fixtures.db --service issue-1779 --min-instances 2 --max-instances 4

  <img width="1303" alt="image" src="https://user-images.githubusercontent.com/9599/184547516-5882f41c-0952-4cac-ae33-c79db7ceada9.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1782#issuecomment-1214420523
issue_url: https://api.github.com/repos/simonw/datasette/issues/1782
node_id: IC_kwDOBm6k_c5IYpYr
created_at: 2022-08-14T17:37:39Z
updated_at: 2022-08-14T17:37:39Z
author_association: OWNER
body:
  I'm not going to write annotated release notes for this one. I think the existing release notes are detailed enough, and the most exciting parts of this release relate to Datasette Lite which I have already written about extensively (and linked from the new release notes).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1783#issuecomment-1215741101
issue_url: https://api.github.com/repos/simonw/datasette/issues/1783
node_id: IC_kwDOBm6k_c5Idryt
created_at: 2022-08-15T20:12:04Z
updated_at: 2022-08-15T20:12:26Z
author_association: OWNER
body:
  This will get a whole lot easier if I implement the ideas in here:

  - #1163

  I won't need to talk people through installing `sqlite-utils` too in that case.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1783#issuecomment-1215742398
issue_url: https://api.github.com/repos/simonw/datasette/issues/1783
node_id: IC_kwDOBm6k_c5IdsG-
created_at: 2022-08-15T20:13:29Z
updated_at: 2022-08-15T20:13:29Z
author_association: OWNER
body:
  This tutorial is relevant: https://datasette.io/tutorials/clean-data - maybe for the moment I can link to that from the documentation in more places?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1780#issuecomment-1215992640
issue_url: https://api.github.com/repos/simonw/datasette/issues/1780
node_id: IC_kwDOBm6k_c5IepNA
created_at: 2022-08-15T23:50:02Z
updated_at: 2022-08-15T23:50:02Z
author_association: OWNER
body:
  This is deliberate, but maybe it's a bad design decision?

  Right now Datasette assumes you made a mistake if you set `facet_time_limit_ms` higher than `sql_time_limit_ms` and ignores you.

  Maybe it would be better if it refused to start the server at all and showed you an error message?

  ```
  % datasette . --setting facet_time_limit_ms 30000
  Error: facet_time_limit_ms greater than sql_time_limit_ms. Try this instead:

      datasette . --setting facet_time_limit_ms 30000 --setting sql_time_limit_ms 30000
  ```

  Or perhaps running this should set both of the time limits to 30s:

      datasette . --setting facet_time_limit_ms 30000

  I'm nervous about doing that though as it feels like it may surprise people who only wanted to increase one of the limits.

  Option 3: make it so it's possible for facets to have a greater time limit than SQL queries generally.

  Maybe that's the best option? It's not surprising to people, and I think it's reasonable to implement.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1784#issuecomment-1216035677
issue_url: https://api.github.com/repos/simonw/datasette/issues/1784
node_id: IC_kwDOBm6k_c5Ieztd
created_at: 2022-08-16T01:13:18Z
updated_at: 2022-08-16T01:13:32Z
author_association: OWNER
body:
  I didn't know SQLite could do this.

  I agree with you - I like the `:` option you described there.

  I'd like to stick with `--load-extension` - `--load` is general enough that I might have it do something else in the future, and this is a rare enough advanced feature that I'm OK with it being verbose.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1784#issuecomment-1216042605
issue_url: https://api.github.com/repos/simonw/datasette/issues/1784
node_id: IC_kwDOBm6k_c5Ie1Zt
created_at: 2022-08-16T01:28:46Z
updated_at: 2022-08-16T01:28:46Z
author_association: OWNER
body:
  Do you want to write a PR for this?

  Ideally I'd want this to be covered by a test, but it looks like `--load-extension` isn't covered by tests at all yet. It's a tricky thing to add to the unit tests because there are plenty of Python installs out there (including the one on my Mac) that don't support `.enable_load_extension()` at all.

  So maybe a test which uses `@pytest.mark.skipIf()` to skip itself if the environment it is running in doesn't support that feature?

  But we'd also need to bundle a compiled SQLite extension to use in the test, which would be a slightly weird thing to have in the repo.

  We do have one previous example of something like that though: https://github.com/simonw/datasette/blob/main/tests/spatialite.db and the script that builds it https://github.com/simonw/datasette/blob/main/tests/build_small_spatialite_db.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/465#issuecomment-1218565147
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/465
node_id: IC_kwDOCGYnMM5IodQb
created_at: 2022-08-17T22:44:40Z
updated_at: 2022-08-17T23:21:22Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/465?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#465](https://codecov.io/gh/simonw/sqlite-utils/pull/465?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (bb90d33) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/45e24deffea042b5db7ab84cd1eb63b3ed9bb9da?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (45e24de) will **not change** coverage.
  > The diff coverage is `n/a`.

  ```diff
  @@           Coverage Diff           @@
  ##             main     #465   +/-   ##
  =======================================
    Coverage   96.64%   96.64%           
  =======================================
    Files           6        6           
    Lines        2566     2566           
  =======================================
    Hits         2480     2480           
    Misses         86       86           
  ```



  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/465#issuecomment-1218663096
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/465
node_id: IC_kwDOCGYnMM5Io1K4
created_at: 2022-08-17T23:37:51Z
updated_at: 2022-08-17T23:37:51Z
author_association: OWNER
body:
  Source links work on the preview: https://sqlite-utils--465.org.readthedocs.build/en/465/reference.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1786#issuecomment-1218849845
issue_url: https://api.github.com/repos/simonw/datasette/issues/1786
node_id: IC_kwDOBm6k_c5Ipiw1
created_at: 2022-08-18T01:16:02Z
updated_at: 2022-08-18T01:16:02Z
author_association: OWNER
body:
  Would be neat if it could adjust the non-JS height of the test area to fit the query displayed when the page is loaded.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1786#issuecomment-1219657981
issue_url: https://api.github.com/repos/simonw/datasette/issues/1786
node_id: IC_kwDOBm6k_c5IsoD9
created_at: 2022-08-18T15:50:53Z
updated_at: 2022-08-18T15:50:53Z
author_association: OWNER
body:
  I went with number of lines + 2 ems in height:

  <img width="482" alt="image" src="https://user-images.githubusercontent.com/9599/185438949-c8b8bb7f-413d-4557-ab81-c318d1beabd5.png">

  It gets confused by wrapping text but I think that's OK:

  <img width="495" alt="image" src="https://user-images.githubusercontent.com/9599/185439072-35e2a2af-d11d-44ee-afeb-095d0d2d928f.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1786#issuecomment-1219683223
issue_url: https://api.github.com/repos/simonw/datasette/issues/1786
node_id: IC_kwDOBm6k_c5IsuOX
created_at: 2022-08-18T16:15:13Z
updated_at: 2022-08-18T16:15:13Z
author_association: OWNER
body:
  Hard to demo this because the JavaScript kicks in, but if you view source on this page:

  https://latest.datasette.io/_memory?sql=select+id%2C+name%2C+ingredients%2C+%28%0D%0A++select+json_group_array%28value%29+from+json_each%28ingredients%29%0D%0A++where+value+in+%28select+value+from+json_each%28%3Ap0%29%29%0D%0A%29+as+matching_ingredients%0D%0Afrom+recipes%0D%0Awhere+json_array_length%28matching_ingredients%29+%3E+0%0D%0Aorder+by+json_array_length%28matching_ingredients%29+desc

  If you view source on that page:

  ```html
              <p><textarea id="sql-editor" name="sql" style="height: 9em"
              >select id, name, ingredients, (
    select json_group_array(value) from json_each(ingredients)
    where value in (select value from json_each(:p0))
  ) as matching_ingredients
  from recipes
  where json_array_length(matching_ingredients) &gt; 0
  order by json_array_length(matching_ingredients) desc</textarea></p>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1787#issuecomment-1219773265
issue_url: https://api.github.com/repos/simonw/datasette/issues/1787
node_id: IC_kwDOBm6k_c5ItENR
created_at: 2022-08-18T17:54:37Z
updated_at: 2022-08-18T17:54:37Z
author_association: OWNER
body:
  It's mentioned in the CLI reference here: https://docs.datasette.io/en/0.62/cli-reference.html#datasette-serve-help - but that's missing some important details such as:

  > The exit code will be 0 if the request succeeds and 1 if the request produced an HTTP status code other than 200 - e.g. a 404 or 500 error. This means you can use `datasette --get /` to run tests against a Datasette application in a continuous integration environment such as GitHub Actions.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1787#issuecomment-1219777448
issue_url: https://api.github.com/repos/simonw/datasette/issues/1787
node_id: IC_kwDOBm6k_c5ItFOo
created_at: 2022-08-18T17:59:06Z
updated_at: 2022-08-18T17:59:06Z
author_association: OWNER
body:
  I'm going to expand this page: https://docs.datasette.io/en/0.62/cli-reference.html - at the moment it just shows `--help` output, but I'm going to turn it into a page with additional written prose for each command plus sections on  things like `--get`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1787#issuecomment-1219778295
issue_url: https://api.github.com/repos/simonw/datasette/issues/1787
node_id: IC_kwDOBm6k_c5ItFb3
created_at: 2022-08-18T18:00:04Z
updated_at: 2022-08-18T18:00:04Z
author_association: OWNER
body:
  I'm also going to move it to an earlier point in the menu - it's currently right at the very end, but if it has more than just the `--help` content it deserves to be listed earlier.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1787#issuecomment-1219782423
issue_url: https://api.github.com/repos/simonw/datasette/issues/1787
node_id: IC_kwDOBm6k_c5ItGcX
created_at: 2022-08-18T18:04:28Z
updated_at: 2022-08-18T18:04:28Z
author_association: OWNER
body:
  Aside: analytics show that the "getting started" page is the highest traffic page in all of the docs, so it deserves extra attention to make it as good as it can possibly be.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1787#issuecomment-1220008317
issue_url: https://api.github.com/repos/simonw/datasette/issues/1787
node_id: IC_kwDOBm6k_c5It9l9
created_at: 2022-08-18T21:56:21Z
updated_at: 2022-08-18T21:57:09Z
author_association: OWNER
body:
  Newly updated CLI reference page: https://docs.datasette.io/en/latest/cli-reference.html

  And the `datasette --get` section is gone from https://docs.datasette.io/en/latest/getting_started.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1789#issuecomment-1221179830
issue_url: https://api.github.com/repos/simonw/datasette/issues/1789
node_id: IC_kwDOBm6k_c5Iybm2
created_at: 2022-08-20T00:03:23Z
updated_at: 2022-08-20T00:03:23Z
author_association: OWNER
body:
  This looks great!

  You need to run `cog -r docs/*.rst` to update the docs, then add those to the PR.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1789#issuecomment-1221180941
issue_url: https://api.github.com/repos/simonw/datasette/issues/1789
node_id: IC_kwDOBm6k_c5Iyb4N
created_at: 2022-08-20T00:07:00Z
updated_at: 2022-08-20T00:07:00Z
author_association: OWNER
body:
  If you wanted to be really ambitious about documentation a page that explained what it takes to actually run SQLite extensions in Python would be great. I'm still trying to figure out the best way to do that in macOS myself.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1789#issuecomment-1221576460
issue_url: https://api.github.com/repos/simonw/datasette/issues/1789
node_id: IC_kwDOBm6k_c5Iz8cM
created_at: 2022-08-21T16:16:42Z
updated_at: 2022-08-21T16:16:42Z
author_association: CONTRIBUTOR
body:
  Rebased, Read the docs failure should now now fixed

  Re docs - ya that's a pretty ambitious page, I'm still not 100% sure what the best practices are/should be... Would be happy to make that page in a future PR

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1789#issuecomment-1221621084
issue_url: https://api.github.com/repos/simonw/datasette/issues/1789
node_id: IC_kwDOBm6k_c5I0HVc
created_at: 2022-08-21T21:07:25Z
updated_at: 2022-08-21T21:07:25Z
author_association: OWNER
body:
  Any tips on how I can get the GitHub Action `test.yml` workflow to compile this so that the test runs there?

  I imagine I need an extra step before https://github.com/simonw/datasette/blob/663ac431fe7202c85967568d82b2034f92b9aa43/.github/workflows/test.yml#L31

  Neatest thing would be to cache the compiled file between actions runs (in the Actions cache) so we don't have to add the SQLite build dependencies on most runs.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/pocket-to-sqlite/issues/11#issuecomment-1221621466
issue_url: https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/11
node_id: IC_kwDODLZ_YM5I0Hba
created_at: 2022-08-21T21:09:47Z
updated_at: 2022-08-21T21:09:47Z
author_association: MEMBER
body:
  Great catch, thanks.

  I'm going to use it to mean `--auth` - since other tools in the Dogsheep family have the same convention.

   `--all` will be the only way to specify all.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/pocket-to-sqlite/pull/7#issuecomment-1221621700
issue_url: https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/7
node_id: IC_kwDODLZ_YM5I0HfE
created_at: 2022-08-21T21:11:12Z
updated_at: 2022-08-21T21:11:12Z
author_association: MEMBER
body:
  I thought this might need a documentation update but `--all` is already covered: https://github.com/dogsheep/pocket-to-sqlite/blob/0.2.1/README.md

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1789#issuecomment-1223347322
issue_url: https://api.github.com/repos/simonw/datasette/issues/1789
node_id: IC_kwDOBm6k_c5I6sx6
created_at: 2022-08-23T00:03:20Z
updated_at: 2022-08-23T00:03:20Z
author_association: CONTRIBUTOR
body:
  @simonw to build the extension on ubuntu, you can run:

  ```
  apt-get update && apt-get install libsqlite3-dev gcc
  gcc ext.c -fPIC -shared -o ext.so
  ```

  I'm not the best with Actions, but if you set the cache key to `ext.c`, run those two commands to download dependencies + compile to `ext.so`, then the unit test should pick it up and run it correctly. Let me know if you want me to update the PR with that added

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1223378004
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/235
node_id: IC_kwDOCGYnMM5I60RU
created_at: 2022-08-23T00:44:11Z
updated_at: 2022-08-23T00:44:11Z
author_association: NONE
body:
  This bug affects me as well. Env:

  ```
  Python 3.8.12
  sqlite-utils, version 3.28
  sqlite3 3.32.3
  MacOS Big Sur 11.6.7
  Intel
  ```

  Similar to @mdrovdahl, I was able to work around this bug by piping the SQL string constructed in `add_foreign_keys` to the `sqlite3` command itself. Specifically, if you're trying to patch this yourself, replace [lines 1026-1039 of db.py in your site packages](https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L1026-L1039) with something similar to the following:

  ```
  print("PRAGMA writable_schema = 1;")
  for table_name, new_sql in table_sql.items():
      print("UPDATE sqlite_master SET sql = '{}' WHERE name = '{}';".format(
           new_sql, table_name)
      )
  print("PRAGMA writable_schema = 0;")
  print("VACUUM;")
  ```
   
  Then from your terminal:
  `db-to-sqlite "<connection string>"  your.db --all > output.sql && sqlite3 your.db < output.sql`

  If you want to run this with `-p`, you'll have to actually open a file in code to write to instead of redirecting the output.

reactions:
  {"total_count": 3, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 3, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/1791#issuecomment-1224080105
issue_url: https://api.github.com/repos/simonw/datasette/issues/1791
node_id: IC_kwDOBm6k_c5I9frp
created_at: 2022-08-23T13:29:51Z
updated_at: 2022-08-23T13:29:51Z
author_association: NONE
body:
  For now I've installed datasette via homebrew.

  I then converted my CSV to SQLite via sqlite-utils as per the article [Cleaning data with sqlite](https://datasette.io/tutorials/clean-data).

  Now I can use my `metadata.json` file as follows:

  `datasette mydb.db -m mymetadata.json` 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224264744
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/467
node_id: IC_kwDOCGYnMM5I-Mwo
created_at: 2022-08-23T15:51:29Z
updated_at: 2022-08-23T15:53:29Z
author_association: OWNER
body:
  Jeff suggested `db[table].(..., alter=True)` for this.

  ```python
      db["urls"].create(
          {
              "url": str,
              "crawled": bool,
              "body": str,
              "headers": dict,
              "status": int,
              "status_text": str,
          },
          pk="url",
          defaults={"crawled": False},
          if_not_exists=True,
          alter=True,
      )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224268273
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/467
node_id: IC_kwDOCGYnMM5I-Nnx
created_at: 2022-08-23T15:54:24Z
updated_at: 2022-08-23T15:54:24Z
author_association: OWNER
body:
  I'm not crazy about having to pass both `alter=True` and `if_not_exists=True` - maybe `alter` should imply `if_not_exists`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224271324
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/467
node_id: IC_kwDOCGYnMM5I-OXc
created_at: 2022-08-23T15:56:58Z
updated_at: 2022-08-23T15:56:58Z
author_association: OWNER
body:
  Should passing `alter=True` also drop any columns that aren't included in the new table structure?

  It could even spot column types that aren't correct and fix those.

  Is that consistent with the expectations set by how `alter=True` works elsewhere?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224272854
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/467
node_id: IC_kwDOCGYnMM5I-OvW
created_at: 2022-08-23T15:58:14Z
updated_at: 2022-08-23T15:58:14Z
author_association: OWNER
body:
  Could call it `ensure=True` here if it works differently enough from `alter=True` that the behavior could be confusing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224278280
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/467
node_id: IC_kwDOCGYnMM5I-QEI
created_at: 2022-08-23T16:02:07Z
updated_at: 2022-08-23T16:02:07Z
author_association: OWNER
body:
  Thinking about this more, I think `alter=True` is a good name for this option even if it does more than the same option on `.insert()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224280225
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/467
node_id: IC_kwDOCGYnMM5I-Qih
created_at: 2022-08-23T16:03:33Z
updated_at: 2022-08-23T16:03:33Z
author_association: OWNER
body:
  Maybe there should be a separate `table.alter(...)` method that does the actual work here, with `.create(..., alter=True)` as syntactic sugar for triggering that if the table exists already.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224283367
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/467
node_id: IC_kwDOCGYnMM5I-RTn
created_at: 2022-08-23T16:05:55Z
updated_at: 2022-08-23T16:05:55Z
author_association: OWNER
body:
  ... but that's what the `table.transform(...)` method does already!

  So maybe this is actually a `transform=True` parameter to `create()` that triggers `table.transform(...)` if necessary.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224382336
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/467
node_id: IC_kwDOCGYnMM5I-peA
created_at: 2022-08-23T17:16:13Z
updated_at: 2022-08-23T17:16:13Z
author_association: CONTRIBUTOR
body:
  > Should passing `alter=True` also drop any columns that aren't included in the new table structure?
  > 
  > It could even spot column types that aren't correct and fix those.
  > 
  > Is that consistent with the expectations set by how `alter=True` works elsewhere?

  I would lean towards not dropping them (or making a `drop=True` or `drop_columns=True`or `drop_missing_columns=True`) to work with existing tables easier. 

  I do like that sqlite-utils mostly just works with existing tables but it's also nice to add to existing fields in a few cases. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224385575
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/467
node_id: IC_kwDOCGYnMM5I-qQn
created_at: 2022-08-23T17:19:00Z
updated_at: 2022-08-23T17:19:00Z
author_association: OWNER
body:
  Initial prototype:
  ```diff
  diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
  index 18a442a..03fd345 100644
  --- a/sqlite_utils/db.py
  +++ b/sqlite_utils/db.py
  @@ -875,6 +875,7 @@ class Database:
           hash_id_columns: Optional[Iterable[str]] = None,
           extracts: Optional[Union[Dict[str, str], List[str]]] = None,
           if_not_exists: bool = False,
  +        transform: bool = False,
       ) -> "Table":
           """
           Create a table with the specified name and the specified ``{column_name: type}`` columns.
  @@ -892,7 +893,39 @@ class Database:
           :param hash_id_columns: List of columns to be used when calculating the hash ID for a row
           :param extracts: List or dictionary of columns to be extracted during inserts, see :ref:`python_api_extracts`
           :param if_not_exists: Use ``CREATE TABLE IF NOT EXISTS``
  -        """
  +        :param transform: If table already exists, transform it to fit the specified schema
  +        """
  +        # Transform table to match the new definition if table already exists:
  +        if transform and self[name].exists():
  +            # First add missing columns and columns to drop
  +            existing_columns = self[name].columns_dict
  +            missing_columns = dict(
  +                (col_name, col_type)
  +                for col_name, col_type in columns.items()
  +                if col_name not in existing_columns
  +            )
  +            columns_to_drop = [
  +                column for column in existing_columns if column not in columns
  +            ]
  +            if missing_columns:
  +                for col_name, col_type in missing_columns.items():
  +                    self[name].add_column(col_name, col_type)
  +            # Do we need to reset the column order?
  +            column_order = None
  +            if list(existing_columns) != list(columns):
  +                column_order = list(columns)
  +            # Only run .transform() if there is something to do
  +            # TODO: this misses changes like pk= without also column changes
  +            if columns_to_drop or missing_columns or column_order:
  +                self[name].transform(
  +                    types=columns,
  +                    drop=columns_to_drop,
  +                    column_order=column_order,
  +                    not_null=not_null,
  +                    defaults=defaults,
  +                    pk=pk,
  +                )
  +            return cast(Table, self[name])
           sql = self.create_table_sql(
               name=name,
               columns=columns,
  @@ -1477,6 +1510,7 @@ class Table(Queryable):
           hash_id_columns: Optional[Iterable[str]] = None,
           extracts: Optional[Union[Dict[str, str], List[str]]] = None,
           if_not_exists: bool = False,
  +        transform: bool = False,
       ) -> "Table":
           """
           Create a table with the specified columns.
  @@ -1508,6 +1542,7 @@ class Table(Queryable):
                   hash_id_columns=hash_id_columns,
                   extracts=extracts,
                   if_not_exists=if_not_exists,
  +                transform=transform,
               )
           return self
   ```
  Needs more thought about how things like just a change to `pk=` should work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224386951
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/467
node_id: IC_kwDOCGYnMM5I-qmH
created_at: 2022-08-23T17:20:07Z
updated_at: 2022-08-23T17:20:07Z
author_association: OWNER
body:
  Example of that prototype working:
  ```pycon
  >>> from sqlite_utils import Database
  >>> db = Database(memory=True)
  >>> db["dogs"].create({"id": int, "name": str}, pk="id")
  <Table dogs (id, name)>
  >>> db["dogs"].create({"id": int, "name": str, "age": int}, pk="id", transform=True)
  <Table dogs (id, name, age)>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1224412060
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/468
node_id: IC_kwDOCGYnMM5I-wuc
created_at: 2022-08-23T17:30:24Z
updated_at: 2022-08-27T23:11:44Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/468?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#468](https://codecov.io/gh/simonw/sqlite-utils/pull/468?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (fc38480) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/c5f8a2eb1a81a18b52825cc649112f71fe419b12?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c5f8a2e) will **decrease** coverage by `0.12%`.
  > The diff coverage is `88.88%`.

  > :exclamation: Current head fc38480 differs from pull request most recent head 2f6a64f. Consider uploading reports for the commit 2f6a64f to get more accurate results

  ```diff
  @@            Coverage Diff             @@
  ##             main     #468      +/-   ##
  ==========================================
  - Coverage   96.60%   96.47%   -0.13%     
  ==========================================
    Files           6        6              
    Lines        2590     2640      +50     
  ==========================================
  + Hits         2502     2547      +45     
  - Misses         88       93       +5     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/468?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/468/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.86% <66.66%> (-0.09%)` | :arrow_down: |
  | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/468/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.31% <90.19%> (-0.23%)` | :arrow_down: |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1789#issuecomment-1224592097
issue_url: https://api.github.com/repos/simonw/datasette/issues/1789
node_id: IC_kwDOBm6k_c5I_crh
created_at: 2022-08-23T18:42:51Z
updated_at: 2022-08-23T18:42:51Z
author_association: OWNER
body:
  Looks like it's not actually necessary to `apt-get install` anything extra in order to compile that test extension:

  <img width="595" alt="image" src="https://user-images.githubusercontent.com/9599/186238465-e1b8dba8-c9f6-4b92-bc5c-b979dcf8fc74.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1792#issuecomment-1224606178
issue_url: https://api.github.com/repos/simonw/datasette/issues/1792
node_id: IC_kwDOBm6k_c5I_gHi
created_at: 2022-08-23T18:49:07Z
updated_at: 2022-08-24T00:11:37Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1792?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1792](https://codecov.io/gh/simonw/datasette/pull/1792?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (f61cfb7) into [main](https://codecov.io/gh/simonw/datasette/commit/fd1086c6867f3e3582b1eca456e4ea95f6cecf8b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (fd1086c) will **increase** coverage by `0.04%`.
  > The diff coverage is `n/a`.

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1792      +/-   ##
  ==========================================
  + Coverage   91.66%   91.71%   +0.04%     
  ==========================================
    Files          38       38              
    Lines        4705     4705              
  ==========================================
  + Hits         4313     4315       +2     
  + Misses        392      390       -2     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1792?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1792/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `94.13% <0.00%> (+0.65%)` | :arrow_up: |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1792#issuecomment-1224869578
issue_url: https://api.github.com/repos/simonw/datasette/issues/1792
node_id: IC_kwDOBm6k_c5JAgbK
created_at: 2022-08-23T20:47:59Z
updated_at: 2022-08-23T20:47:59Z
author_association: OWNER
body:
  Really weird: the tests are failing on Python 3.7 only for some reason, in a test that should be completely unrelated to this extension feature.

  <img width="1022" alt="image" src="https://user-images.githubusercontent.com/9599/186262505-9e849125-6bb0-42fd-905c-23e879c5e508.png">

  https://github.com/simonw/datasette/runs/7982525580?check_suite_focus=true

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229115501
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/470
node_id: IC_kwDOCGYnMM5JQtBt
created_at: 2022-08-27T03:54:26Z
updated_at: 2022-08-27T03:54:26Z
author_association: OWNER
body:
  I can borrow Alex's example extension for the tests too: https://github.com/simonw/datasette/blob/ba35105eee2d3ba620e4f230028a02b2e2571df2/tests/ext.c

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229116423
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/471
node_id: IC_kwDOCGYnMM5JQtQH
created_at: 2022-08-27T04:00:52Z
updated_at: 2022-08-27T04:00:52Z
author_association: OWNER
body:
  Alternative design would be `--function name definition` - like this:

  ```
  sqlite-utils data.db 'update images set domain = extract_domain(url)' --function extract_domain '
      from urllib.parse import urlparse
      return urlparse(url).netloc
  '
  ```
  I like the `--functions` design better because it leaves space for import statements at the top of the code block, and allows more than one function to be defined in a single go.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229118619
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/471
node_id: IC_kwDOCGYnMM5JQtyb
created_at: 2022-08-27T04:14:52Z
updated_at: 2022-08-27T04:14:52Z
author_association: OWNER
body:
  Quick prototype:
  ```diff
  diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py
  index 43e76fa..5dee4f6 100644
  --- a/sqlite_utils/cli.py
  +++ b/sqlite_utils/cli.py
  @@ -1633,6 +1633,9 @@ def drop_view(path, view, ignore, load_extension):
       type=(str, str),
       help="Named :parameters for SQL query",
   )
  +@click.option(
  +    "--functions", help="Python code defining one or more custom SQL functions"
  +)
   @load_extension_option
   def query(
       path,
  @@ -1649,6 +1652,7 @@ def query(
       raw,
       param,
       load_extension,
  +    functions,
   ):
       """Execute SQL query and return the results as JSON
   
  @@ -1665,6 +1669,16 @@ def query(
       _load_extensions(db, load_extension)
       db.register_fts4_bm25()
   
  +    # Register any Python functions as SQL functions:
  +    if functions:
  +        locals = {}
  +        globals = {}
  +        exec(functions, globals, locals)
  +        # Register all callables in the locals dict:
  +        for name, value in locals.items():
  +            if callable(value):
  +                db.register_function(value, name=name)
  +
       _execute_query(
           db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols
       )
  ```
  Demo:
  ```bash
   % sqlite-utils :memory: 'select 1 + dog()' --functions '
  quote> def dog():
  quote>   return 2
  quote> '
  [{"1 + dog()": 3}]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229119171
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/471
node_id: IC_kwDOCGYnMM5JQt7D
created_at: 2022-08-27T04:18:28Z
updated_at: 2022-08-27T04:18:28Z
author_association: OWNER
body:
  I tried this:
  ```
  sqlite-utils :memory: 'select extract_domain("https://www.google.com/blah")' --functions '
  from urllib.parse import urlparse

  def extract_domain(url):
      from urllib.parse import urlparse
      return urlparse(url).netloc
  '
  ```
  And got:
  ```
  NameError: name 'urlparse' is not defined
  Error: user-defined function raised exception
  ```
  But this worked OK:
  ```
  % sqlite-utils :memory: 'select extract_domain("https://www.google.com/blah")' --functions '
  def extract_domain(url):
      from urllib.parse import urlparse
      return urlparse(url).netloc
  '
  [{"extract_domain(\"https://www.google.com/blah\")": "www.google.com"}]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229119999
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/471
node_id: IC_kwDOCGYnMM5JQuH_
created_at: 2022-08-27T04:24:58Z
updated_at: 2022-08-27T04:24:58Z
author_association: OWNER
body:
  I've encountered this problem before: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-convert-complex

  > ```
  > $ sqlite-utils convert content.db articles score '
  > import random
  > random.seed(10)
  > 
  > def convert(value):
  >     global random
  >     return random.random()
  > '
  > ```
  > Note the `global random` line here. Due to the way the tool compiles Python code, this is necessary to ensure the `random` module is available within the `convert()` function. If you were to omit this you would see a `NameError: name 'random' is not defined` error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229120104
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/471
node_id: IC_kwDOCGYnMM5JQuJo
created_at: 2022-08-27T04:25:39Z
updated_at: 2022-08-27T04:25:39Z
author_association: OWNER
body:
  This works:
  ```
  sqlite-utils :memory: 'select extract_domain("https://www.google.com/blah")' --functions '
  from urllib.parse import urlparse

  def extract_domain(url):
      global urlparse
      return urlparse(url).netloc
  '
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229120653
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/471
node_id: IC_kwDOCGYnMM5JQuSN
created_at: 2022-08-27T04:29:49Z
updated_at: 2022-08-27T04:30:03Z
author_association: OWNER
body:
  Found a fix for that!

  I replaced this:

  ```python
          locals = {}
          globals = {}
          exec(functions, globals, locals)
          # Register all callables in the locals dict:
          for name, value in locals.items():
              if callable(value):
                  db.register_function(value, name=name)
  ```
  With this:
  ```python
          globals = {}
          exec(functions, globals)
          # Register all callables in the globals dict:
          for name, value in globals.items():
              if callable(value):
                  db.register_function(value, name=name)
  ```
  Because https://docs.python.org/3/library/functions.html#exec says:

  > If only *globals* is provided, it must be a dictionary (and not a subclass of dictionary), which will be used for both the global and the local variables. If *globals* and *locals* are given, they are used for the global and local variables, respectively.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229124549
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/471
node_id: IC_kwDOCGYnMM5JQvPF
created_at: 2022-08-27T05:03:39Z
updated_at: 2022-08-27T05:03:39Z
author_association: OWNER
body:
  I don't think I need separate documentation for `sqlite-utils memory` and `sqlite-tils bulk` since they work the same, and the `--help` text provides the necessary hints.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229125114
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/471
node_id: IC_kwDOCGYnMM5JQvX6
created_at: 2022-08-27T05:08:58Z
updated_at: 2022-08-27T05:08:58Z
author_association: OWNER
body:
  Testing `bulk --functions`:

  ```
  % sqlite-utils create-table chickens.db chickens id integer name text name_upper text 
  % echo 'id,name
  1,Blue
  2,Snowy
  3,Azi
  4,Lila
  5,Suna
  6,Cardi' | sqlite-utils bulk chickens.db '
  insert into chickens (id, name, name_upper) values (:id, :name, myupper(:name))
  ' - --functions '
  def myupper(s):
      return s.upper()
  ' --csv
  % sqlite-utils rows chickens.db chickens
  [{"id": 1, "name": "Blue", "name_upper": "BLUE"},
   {"id": 2, "name": "Snowy", "name_upper": "SNOWY"},
   {"id": 3, "name": "Azi", "name_upper": "AZI"},
   {"id": 4, "name": "Lila", "name_upper": "LILA"},
   {"id": 5, "name": "Suna", "name_upper": "SUNA"},
   {"id": 6, "name": "Cardi", "name_upper": "CARDI"}]
  ```

reactions:
  {"total_count": 2, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/sqlite-utils/issues/472#issuecomment-1229125614
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/472
node_id: IC_kwDOCGYnMM5JQvfu
created_at: 2022-08-27T05:12:59Z
updated_at: 2022-08-27T05:12:59Z
author_association: OWNER
body:
  Annoyingly this means the `--import` option is no longer required, but removing it would be a backwards incompatible breakage so I need to leave it in. I can minimize it in the documentation though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/472#issuecomment-1229125890
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/472
node_id: IC_kwDOCGYnMM5JQvkC
created_at: 2022-08-27T05:15:26Z
updated_at: 2022-08-27T05:15:26Z
author_association: OWNER
body:
  Here's the docs I get to update to remove the documented `global` hack: https://sqlite-utils.datasette.io/en/3.28/cli.html#cli-convert-complex

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229128617
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/470
node_id: IC_kwDOCGYnMM5JQwOp
created_at: 2022-08-27T05:39:06Z
updated_at: 2022-08-27T05:39:06Z
author_association: OWNER
body:
  That didn't run the tests as expected on Ubuntu: https://github.com/simonw/sqlite-utils/runs/8048229213?check_suite_focus=true

  ```
  tests/test_cli.py::test_load_extension[None-should_pass0-should_fail0] SKIPPED [ 19%]
  tests/test_cli.py::test_load_extension[sqlite3_ext_b_init-b-should_fail1] SKIPPED [ 19%]
  tests/test_cli.py::test_load_extension[sqlite3_ext_c_init-c-should_fail2] SKIPPED [ 19%]
  ```

  That should have worked because of this: https://github.com/simonw/sqlite-utils/blob/3f694e51a0212a297f663e2fc7f9cdad8775a550/.github/workflows/test.yml#L38-L41

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229128950
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/470
node_id: IC_kwDOCGYnMM5JQwT2
created_at: 2022-08-27T05:42:04Z
updated_at: 2022-08-27T05:42:04Z
author_association: OWNER
body:
  Extension looks to be correctly compiled:

  ```
  Run (cd tests && gcc ext.c -fPIC -shared -o ext.so && ls -lah)
  total 412K
  drwxr-xr-x 3 runner docker 4.0K Aug 27 05:40 .
  drwxr-xr-x 8 runner docker 4.0K Aug 27 05:40 ..
  -rw-r--r-- 1 runner docker    0 Aug 27 05:40 __init__.py
  -rw-r--r-- 1 runner docker  760 Aug 27 05:40 conftest.py
  -rw-r--r-- 1 runner docker 1.6K Aug 27 05:40 ext.c
  -rwxr-xr-x 1 runner docker  16K Aug 27 05:40 ext.so
  drwxr-xr-x 2 runner docker 4.0K Aug 27 05:40 sniff
  ```
  So why doesn't this check pass?

  https://github.com/simonw/sqlite-utils/blob/3f694e51a0212a297f663e2fc7f9cdad8775a550/tests/test_cli.py#L25-L30

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229130158
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/470
node_id: IC_kwDOCGYnMM5JQwmu
created_at: 2022-08-27T05:52:46Z
updated_at: 2022-08-27T05:52:46Z
author_association: OWNER
body:
  To run the tests locally on my Mac I needed to run Ubuntu in Docker (for the compilation step).

  This worked:

      docker run -it -v "$(pwd):/app" ubuntu:latest /bin/bash

  Then:

  ```
  apt-get update && apt-get install python3-pip gcc libsqlite3-dev -y
  cd /app/tests
  gcc ext.c -fPIC -shared -o ext.so
  cd /app
  pip3 install -e '.[test]'
  pytest -k test_load_extension
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229130242
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/470
node_id: IC_kwDOCGYnMM5JQwoC
created_at: 2022-08-27T05:53:31Z
updated_at: 2022-08-27T05:53:31Z
author_association: OWNER
body:
  Tests passed in CI against Ubuntu:
  ```
  tests/test_cli.py::test_load_extension[None-should_pass0-should_fail0] PASSED [ 19%]
  tests/test_cli.py::test_load_extension[sqlite3_ext_b_init-b-should_fail1] PASSED [ 19%]
  tests/test_cli.py::test_load_extension[sqlite3_ext_c_init-c-should_fail2] PASSED [ 19%]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/473#issuecomment-1229130453
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/473
node_id: IC_kwDOCGYnMM5JQwrV
created_at: 2022-08-27T05:55:12Z
updated_at: 2022-08-27T05:55:18Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/473?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#473](https://codecov.io/gh/simonw/sqlite-utils/pull/473?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (72db599) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/a46a5e3a9e03dcdd8c84a92e4a5dbfa02ba461fa?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a46a5e3) will **decrease** coverage by `0.07%`.
  > The diff coverage is `50.00%`.

  ```diff
  @@            Coverage Diff             @@
  ##             main     #473      +/-   ##
  ==========================================
  - Coverage   96.67%   96.60%   -0.08%     
  ==========================================
    Files           6        6              
    Lines        2587     2590       +3     
  ==========================================
  + Hits         2501     2502       +1     
  - Misses         86       88       +2     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/473?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/473/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.94% <50.00%> (-0.18%)` | :arrow_down: |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/455#issuecomment-1229205990
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/455
node_id: IC_kwDOCGYnMM5JRDHm
created_at: 2022-08-27T14:44:45Z
updated_at: 2022-08-27T14:44:45Z
author_association: OWNER
body:
  I did not know `is` worked against strings and integers and booleans!

  https://latest.datasette.io/_memory?sql=select+1+is+1%2C+2.3+is+2.3%2C+true+is+true%2C+true+is+false%2C+%27dog%27+is+%27dog%27%2C+%27dog%27+is+%27cat%27%2C+null+is+null%2C+null+is+not+null

  ```sql
  select 1 is 1, 2.3 is 2.3, true is true, true is false, 'dog' is 'dog', 'dog' is 'cat', null is null, null is not null
  ```

  |   1 is 1 |   2.3 is 2.3 |   true is true |   true is false |   'dog' is 'dog' |   'dog' is 'cat' |   null is null |   null is not null |
  |----------|--------------|----------------|-----------------|------------------|------------------|----------------|--------------------|
  |        1 |            1 |              1 |               0 |                1 |                0 |              1 |                  0 |

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229207725
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/468
node_id: IC_kwDOCGYnMM5JRDit
created_at: 2022-08-27T14:55:32Z
updated_at: 2022-08-27T22:03:50Z
author_association: OWNER
body:
  The main challenge here is coming up with comprehensive tests. The cases I need to cover are from this block of code:

  https://github.com/simonw/sqlite-utils/blob/c5f8a2eb1a81a18b52825cc649112f71fe419b12/sqlite_utils/db.py#L1468-L1480

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/474#issuecomment-1229265285
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/474
node_id: IC_kwDOCGYnMM5JRRmF
created_at: 2022-08-27T20:52:53Z
updated_at: 2022-08-27T20:52:53Z
author_association: OWNER
body:
  A couple of tricks I use here. Firstly, I often create the table before the import using the `sqlite-utils create-table` command: https://sqlite-utils.datasette.io/en/stable/cli.html#creating-tables

  The other current option is to use the `bulk` command, which lets you construct a custom SQL query to execute against every row from a CSV file: https://sqlite-utils.datasette.io/en/stable/cli.html#executing-sql-in-bulk

  Do either of those options work here or is there a useful new feature that would work better?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229276554
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/468
node_id: IC_kwDOCGYnMM5JRUWK
created_at: 2022-08-27T22:05:40Z
updated_at: 2022-08-27T22:05:51Z
author_association: OWNER
body:
  For the moment I'm not going to pay attention to `foreign_keys` changes - I will note that these are not modified in the documentation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229279539
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/468
node_id: IC_kwDOCGYnMM5JRVEz
created_at: 2022-08-27T22:24:33Z
updated_at: 2022-08-27T22:24:33Z
author_association: OWNER
body:
  Interesting challenge with `default_value`: I need to be able to tell if the default values passed to `.create()` differ from those in the database already.

  Introspecting that is a bit tricky:

  ```pycon
  >>> import sqlite_utils
  >>> db = sqlite_utils.Database(memory=True)
  >>> db["blah"].create({"id": int, "name": str}, not_null=("name",), defaults={"name": "bob"})
  <Table blah (id, name)>
  >>> db["blah"].columns
  [Column(cid=0, name='id', type='INTEGER', notnull=0, default_value=None, is_pk=0), Column(cid=1, name='name', type='TEXT', notnull=1, default_value="'bob'", is_pk=0)]
  ```
  Note how a default value of the Python string `bob` is represented in the results of `PRAGMA table_info()` as `default_value="'bob'"` - it's got single quotes added to it!

  So comparing default values from introspecting the database needs me to first parse that syntax. This may require a new table introspection method.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/475#issuecomment-1229283063
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/475
node_id: IC_kwDOCGYnMM5JRV73
created_at: 2022-08-27T22:44:45Z
updated_at: 2022-08-27T22:44:45Z
author_association: OWNER
body:
  Documentation:
  - https://sqlite-utils.datasette.io/en/latest/python-api.html#default-values
  - https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.default_values

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229284539
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/468
node_id: IC_kwDOCGYnMM5JRWS7
created_at: 2022-08-27T22:54:07Z
updated_at: 2022-08-27T22:55:01Z
author_association: OWNER
body:
  Just needs documentation now, which can go here: https://sqlite-utils--468.org.readthedocs.build/en/468/python-api.html#explicitly-creating-a-table

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229303691
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/468
node_id: IC_kwDOCGYnMM5JRa-L
created_at: 2022-08-27T23:13:06Z
updated_at: 2022-08-27T23:13:06Z
author_association: OWNER
body:
  Documentation:
  - https://sqlite-utils--468.org.readthedocs.build/en/468/python-api.html#explicitly-creating-a-table
  - https://sqlite-utils--468.org.readthedocs.build/en/468/cli.html#creating-tables

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1229311612
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/467
node_id: IC_kwDOCGYnMM5JRc58
created_at: 2022-08-27T23:19:41Z
updated_at: 2022-08-27T23:19:41Z
author_association: OWNER
body:
  Documentation:

  - https://sqlite-utils.datasette.io/en/latest/python-api.html#explicitly-creating-a-table
  - https://sqlite-utils.datasette.io/en/latest/cli.html#creating-tables

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/476#issuecomment-1229320114
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/476
node_id: IC_kwDOCGYnMM5JRe-y
created_at: 2022-08-27T23:26:48Z
updated_at: 2022-08-27T23:26:48Z
author_association: OWNER
body:
  - #458 - the `register_function(name=...)` argument
  - New tutorial:  [Cleaning data with sqlite-utils and Datasette](https://datasette.io/tutorials/clean-data) provides a tutorial introduction (and accompanying ten minute video) about using this tool.
  - New Discord community, https://discord.gg/Ass7bCAMDw
  - #469 `sqlite-utils rows --order` option
  - #471 `sqlite-utils query --functions` option
  - #472 Improved code compilation pattern
  - #473 Support entrypoints for `--load-extension`
  - #455
  - #475 `table.default_values` property
  - #467

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/474#issuecomment-1229449018
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/474
node_id: IC_kwDOCGYnMM5JR-c6
created_at: 2022-08-28T12:40:13Z
updated_at: 2022-08-28T12:40:13Z
author_association: NONE
body:
  Creating the table before inserting is a useful workaround, thanks. It does require figuring out the `create table` syntax and listing all the fields manually, though, which loses some of the magic of sqlite-utils.

  I was expecting to find an option like `--headers=foo,bar` (or `--header-row='foo\tbar'`, if that would be easier) - not necessarily that exact syntax, but something that would essentially be treated the same as having a header row in the file.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/478#issuecomment-1229563718
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/478
node_id: IC_kwDOCGYnMM5JSadG
created_at: 2022-08-28T22:07:03Z
updated_at: 2022-08-28T22:07:03Z
author_association: OWNER
body:
  What should happen if you run this:

      sqlite-utils tables data.db bad_table

  Where `bad_table` is a table that does not exist?

  I think it should error rather than failing silently. That way you can use the error code to confirm that a list of expected tables exist.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1796#issuecomment-1231126795
issue_url: https://api.github.com/repos/simonw/datasette/issues/1796
node_id: IC_kwDOBm6k_c5JYYEL
created_at: 2022-08-30T04:29:35Z
updated_at: 2022-08-30T04:29:35Z
author_association: OWNER
body:
  I wrote upgrade instructions for CodeMirror here: https://docs.datasette.io/en/0.62/contributing.html#upgrading-codemirror

  See also:
  - #948

  But... those instructions don't work any more, because they start with downloading the CodeMirror zip - but according to https://codemirror.net/docs/migration/

  > The 5.x library was distributed as UMD modules, which could either be loaded directly with <script> tags or bundled as CommonJS modules. The new version exposes only modules, which will require some kind of build step before you can use them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1796#issuecomment-1231127022
issue_url: https://api.github.com/repos/simonw/datasette/issues/1796
node_id: IC_kwDOBm6k_c5JYYHu
created_at: 2022-08-30T04:30:07Z
updated_at: 2022-08-30T04:30:07Z
author_association: OWNER
body:
  Also relevant: Wikimedia are upgrading to CodeMirror 6 partly because of improved mobile support: https://phabricator.wikimedia.org/T259059

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/480#issuecomment-1232089808
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/480
node_id: IC_kwDOCGYnMM5JcDLQ
created_at: 2022-08-30T19:43:02Z
updated_at: 2022-08-30T19:43:02Z
author_association: OWNER
body:
  Yeah this seems like a reasonable addition to me. 

  Needs a test, which can go next to this one: https://github.com/simonw/sqlite-utils/blob/087753cd42c406f1e060c1822dcd9b5fda3d60f4/tests/test_fts.py#L561

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/480#issuecomment-1232091957
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/480
node_id: IC_kwDOCGYnMM5JcDs1
created_at: 2022-08-30T19:45:23Z
updated_at: 2022-08-31T03:34:37Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/480?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **96.47**% // Head: **96.47**% // Increases project coverage by **`+0.00%`** :tada:
  > Coverage data is based on head [(`3dbfca6`)](https://codecov.io/gh/simonw/sqlite-utils/pull/480?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`087753c`)](https://codecov.io/gh/simonw/sqlite-utils/commit/087753cd42c406f1e060c1822dcd9b5fda3d60f4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch coverage: 100.00% of modified lines in pull request are covered.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main     #480   +/-   ##
  =======================================
    Coverage   96.47%   96.47%           
  =======================================
    Files           6        6           
    Lines        2640     2642    +2     
  =======================================
  + Hits         2547     2549    +2     
    Misses         93       93           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/480?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/480/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.32% <100.00%> (+<0.01%)` | :arrow_up: |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/480?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/474#issuecomment-1232424050
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/474
node_id: IC_kwDOCGYnMM5JdUxy
created_at: 2022-08-31T03:42:28Z
updated_at: 2022-08-31T03:42:28Z
author_association: OWNER
body:
  Yeah, having a way of inserting a CSV file but saying "only I want the `title` column to come first" does make sense to me as a feature.

  It can use `-o` for consistency with https://sqlite-utils.datasette.io/en/stable/cli-reference.html#search and https://sqlite-utils.datasette.io/en/stable/cli-reference.html#rows and https://sqlite-utils.datasette.io/en/stable/cli-reference.html#transform

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1775#issuecomment-1233298125
issue_url: https://api.github.com/repos/simonw/datasette/issues/1775
node_id: IC_kwDOBm6k_c5JgqLN
created_at: 2022-08-31T18:53:49Z
updated_at: 2022-08-31T18:53:49Z
author_association: NONE
body:
  I also would like a French version UI and am prepared to contribute to this translation.

  Having worked an i18n project in the past, there can be complexities with the strings and the various grammars and syntaxes, depending on how things are set up.
  It's a pretty big job to retroactively i18n-ify things, but for us at least we have requirements to provide French-language interfaces at our University, so I would be willing to participate in this initiative.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1775#issuecomment-1233680261
issue_url: https://api.github.com/repos/simonw/datasette/issues/1775
node_id: IC_kwDOBm6k_c5JiHeF
created_at: 2022-09-01T03:05:57Z
updated_at: 2022-09-01T03:05:57Z
author_association: OWNER
body:
  OK, I'm convinced that it's time to start figuring this out.

  I've done a little bit of this with Django in the past, but Datasette isn't built on Django.

  It looks to me like the key library for implementing this is Babel: https://babel.pocoo.org/en/latest/

  It's been around since 2007 and is very widely used: https://github.com/python-babel/babel/network/dependents?package_id=UGFja2FnZS01MDM0NTU3NQ%3D%3D

  Also found these hints on getting it to work with Jinja: https://stackoverflow.com/questions/12046998/babel-doesnt-recognize-jinja2-extraction-method-for-language-support

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1775#issuecomment-1233680769
issue_url: https://api.github.com/repos/simonw/datasette/issues/1775
node_id: IC_kwDOBm6k_c5JiHmB
created_at: 2022-09-01T03:06:44Z
updated_at: 2022-09-01T03:06:44Z
author_association: OWNER
body:
  My hope is that Datasette actually has a relatively small i18n footprint - there really aren't that many screens with strings that will need translating.

  Translating the documentation would be a _much_ bigger task, but that's not a blocker on getting the interface itself internationalized.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1775#issuecomment-1233682708
issue_url: https://api.github.com/repos/simonw/datasette/issues/1775
node_id: IC_kwDOBm6k_c5JiIEU
created_at: 2022-09-01T03:09:36Z
updated_at: 2022-09-01T03:09:36Z
author_association: OWNER
body:
  Some tutorials that look useful:

  - https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xiii-i18n-and-l10n
  - https://phrase.com/blog/posts/i18n-advantages-babel-python/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1775#issuecomment-1233684765
issue_url: https://api.github.com/repos/simonw/datasette/issues/1775
node_id: IC_kwDOBm6k_c5JiIkd
created_at: 2022-09-01T03:12:50Z
updated_at: 2022-09-01T03:12:50Z
author_association: NONE
body:
  I want to begin translation to es and it documentation, if u like i would do PR asap.
  But user interface, frontend and backend is a good feature be i18n support.
  When you have may be .pot file I can translate to, or Json file with all strings for internationalized

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1775#issuecomment-1233697166
issue_url: https://api.github.com/repos/simonw/datasette/issues/1775
node_id: IC_kwDOBm6k_c5JiLmO
created_at: 2022-09-01T03:36:44Z
updated_at: 2022-09-01T03:36:44Z
author_association: OWNER
body:
  I don't want to start any efforts around documentation translation until after the Datasette 1.0 release, because I'd like to be confident that we're not translating documentation that may have some big changes before Datasette is fully stable!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1775#issuecomment-1233702481
issue_url: https://api.github.com/repos/simonw/datasette/issues/1775
node_id: IC_kwDOBm6k_c5JiM5R
created_at: 2022-09-01T03:48:27Z
updated_at: 2022-09-01T03:48:27Z
author_association: NONE
body:
  I want to do may be 3 or 4 PR for your evaluation, and when you allowed
  then I want supporting that effort

  El mié, 31 de ago. de 2022, 10:36 p. m., Simon Willison <
  ***@***.***> escribió:

  > I don't want to start any efforts around documentation translation until
  > after the Datasette 1.0 release, because I'd like to be confident that
  > we're not translating documentation that may have some big changes before
  > Datasette is fully stable!
  >
  > —
  > Reply to this email directly, view it on GitHub
  > <https://github.com/simonw/datasette/issues/1775#issuecomment-1233697166>,
  > or unsubscribe
  > <https://github.com/notifications/unsubscribe-auth/AADIWFDUWQ22P3KVJVMRC7TV4AQFNANCNFSM55EFFS5Q>
  > .
  > You are receiving this because you authored the thread.Message ID:
  > ***@***.***>
  >

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1798#issuecomment-1234925616
issue_url: https://api.github.com/repos/simonw/datasette/issues/1798
node_id: IC_kwDOBm6k_c5Jm3gw
created_at: 2022-09-01T23:56:33Z
updated_at: 2022-09-01T23:56:33Z
author_association: OWNER
body:
  Oh that's a classic YAML problem, I hadn't realized it would affect Datasette like this!

  You should be able to fix it by quoting the `off` key like this:

  ```yaml
  title: Some title
  description_html: |-
    <p>This is an experiment.</p>
  databases:
    "off":
      tables:
        products_from_owners:
          title: products_from_owners*
          description_html: |-
            <p>Description</p>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1798#issuecomment-1234925618
issue_url: https://api.github.com/repos/simonw/datasette/issues/1798
node_id: IC_kwDOBm6k_c5Jm3gy
created_at: 2022-09-02T00:01:00Z
updated_at: 2022-09-02T00:01:00Z
author_association: OWNER
body:
  There's a thread about this here, but it doesn't have any workaround that look to me like they would be worth applying: https://stackoverflow.com/questions/36463531/pyyaml-automatically-converting-certain-keys-to-boolean-values

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1798#issuecomment-1234925621
issue_url: https://api.github.com/repos/simonw/datasette/issues/1798
node_id: IC_kwDOBm6k_c5Jm3g1
created_at: 2022-09-02T00:01:47Z
updated_at: 2022-09-02T00:01:47Z
author_association: OWNER
body:
  It's a nasty gotcha though. I'm inclined to add validation to Datasette that fails to start the server with an error if you attempt to start it with metadata that accidentally has a boolean where a database name is expected.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1798#issuecomment-1234925975
issue_url: https://api.github.com/repos/simonw/datasette/issues/1798
node_id: IC_kwDOBm6k_c5Jm3mX
created_at: 2022-09-02T00:02:53Z
updated_at: 2022-09-02T00:02:53Z
author_association: OWNER
body:
  Here's a very old issue that suggests validating metadata:
  - https://github.com/simonw/datasette/issues/260

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/260#issuecomment-1234926923
issue_url: https://api.github.com/repos/simonw/datasette/issues/260
node_id: IC_kwDOBm6k_c5Jm31L
created_at: 2022-09-02T00:04:26Z
updated_at: 2022-09-02T00:04:45Z
author_association: OWNER
body:
  Interesting example of why this would be valuable here:
  - https://github.com/simonw/datasette/issues/1798

  This YAML file:
  ```yaml
  title: Some title
  description_html: |-
    <p>This is an experiment.</p>
  databases:
    off:
      tables:
        products_from_owners:
          title: products_from_owners*
  ```
  Was loaded as equivalent to this JSON:
  ```json
  {
      "title": "Some title",
      "description_html": "<p>This is an experiment.</p>",
      "databases": {
          "false": {
              "tables": {
                  "products_from_owners": {
                      "title": "products_from_owners*"
                  }
               }
          }
      }
  }
  ```
  Validation that caught this would have been useful.

reactions:
[cell repeats an earlier one in this file: 112 characters]


