The GitHub JSON APIs have a really interesting convention with respect to related objects.

Consider https://api.github.com/repos/simonw/sqlite-utils/issues - here's a truncated subset:

  {
    "id": 449818897,
    "node_id": "MDU6SXNzdWU0NDk4MTg4OTc=",
    "number": 24,
    "title": "Additional Column Constraints?",
    "user": {
      "login": "IgnoredAmbience",
      "id": 98555,
      "node_id": "MDQ6VXNlcjk4NTU1",
      "avatar_url": "https://avatars0.githubusercontent.com/u/98555?v=4",
      "gravatar_id": ""
    },
    "labels": [
      {
        "id": 993377884,
        "node_id": "MDU6TGFiZWw5OTMzNzc4ODQ=",
        "url": "https://api.github.com/repos/simonw/sqlite-utils/labels/enhancement",
        "name": "enhancement",
        "color": "a2eeef",
        "default": true
      }
    ],
    "state": "open"
  }

The user column lists a complete user. The labels column has a list of labels.

Since both user and label have populated id field this is actually enough information for us to create records for them AND set up the corresponding foreign key (for user) and m2m relationships (for labels).

It would be really neat if sqlite-utils had some kind of mechanism for correctly processing these kind of patterns.

Thanks to jq there's not much need for extra customization of the shape here - if we support a narrowly defined structure users can use jq to reshape arbitrary JSON to match.

Updates the requirements on jinja2 to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps black from 22.1.0 to 22.6.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps black from 22.1.0 to 22.3.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

is there a duplicate table functionality? Otherwise, I'd be happy to submit a PR.

In sqlite3 it would look like:

import sqlite3 as sl

con = sl.connect('prompt-tune.db')

def db_duplicate_table(table_name, table_name_new, con=con):
    # Duplicates table `table_name` to a new table `table_name_new`.
    try:
        cur = con.cursor()
        cur.execute(f"""CREATE TABLE {table_name_new} AS SELECT * FROM {table_name}""")
    except Exception as e:
        print(e)
    finally:
        cur.close()

db_duplicate_table('orig_table', 'new_table')

I am building a database, in which one of the tables has a many-to-many relationship to itself. As far as I can see, this is not (yet) possible using .m2m() in sqlite-utils. This may be a bit of a niche use case, so feel free to close this issue if you feel it would introduce too much complexity compared to the benefits.

Example: suppose I have a table of people, and I want to store the information that John and Mary have two children, Michael and Suzy. It would be neat if I could do something like this:

from sqlite_utils import Database

db = Database(memory=True)
db["people"].insert({"name": "John"}, pk="name").m2m(
    "people", [{"name": "Michael"}, {"name": "Suzy"}], m2m_table="parent_child", pk="name"
)
db["people"].insert({"name": "Mary"}, pk="name").m2m(
    "people", [{"name": "Michael"}, {"name": "Suzy"}], m2m_table="parent_child", pk="name"
)

But if I do that, the many-to-many table parent_child has only one column:

CREATE TABLE [parent_child] (
   [people_id] TEXT REFERENCES [people]([name]),
   PRIMARY KEY ([people_id], [people_id])
)

This could be solved by adding one or two keyword_arguments to .m2m(), e.g. .m2m(..., left_name=None, right_name=None) or .m2m(..., names=(None, None)).

Columns duration, totalDistance and totalEnergyBurned should be converted to float.

https://github.com/dogsheep/healthkit-to-sqlite/blob/71e36e1cf034b96de2a8e6652265d782d3fdf63b/healthkit_to_sqlite/utils.py#L50-L57

Hi, thanks for the project!

For the JSON output, I would consider defaulting to ensure_ascii=False (UTF-8 seems pretty universal) or making it an option. When dealing with non-Latin text, ensure_ascii=True (the default) can triple the size of the output.

Bumps furo from 2022.4.7 to 2022.6.21.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps furo from 2022.4.7 to 2022.6.4.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Attempting to run the example given here (without extra bracket ;-):
https://sqlite-utils.datasette.io/en/stable/python-api.html#reading-rows-from-a-file

from sqlite_utils.utils import rows_from_file
import io

rows, format = rows_from_file(io.StringIO("id,name\n1,Cleo"))
print(list(rows), format)
# Outputs [{'id': '1', 'name': 'Cleo'}] Format.CSV

Gives error

>"c:\Program Files\Python37\python.exe" test2.py
Traceback (most recent call last):
  File "test2.py", line 4, in <module>
    rows, format = rows_from_file(io.StringIO("id,name\n1,Cleo"))
  File "C:\Users\swood\Downloads\sqlite-utils-main-20220621\sqlite-utils-main\sqlite_utils\utils.py", line 300, in rows_from_file
    first_bytes = buffered.peek(2048).strip()
AttributeError: '_io.StringIO' object has no attribute 'readinto'

I am running Python on Windows.

>"c:\Program Files\Python37\python.exe"
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

I keep committing code that fails additional tests like mypy and flake8 and black. Automate those using Just.

As seen in https://til.simonwillison.net/sqlite/import-csv - .mode csv and then .import school.csv schools is hugely faster than importing via sqlite-utils insert and doing the work in Python - but it can only be implemented by shelling out to the sqlite3 CLI tool, it's not functionality that is exposed to the Python sqlite3 module.

An option to use this would be useful - maybe something like this:

sqlite-utils insert blah.db blah blah.csv --fast

https://sqlite-utils.datasette.io/en/stable/cli-reference.html#insert

It looks like Python keywords are being incorrectly highlighted here.

rows_from_file() isn't part of the documented API but maybe it should be!

_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154385916_

I've used it in a couple of external places now:

• https://github.com/simonw/datasette-socrata/blob/32fb256a461bf0e790eca10bdc7dd9d96c20f7c4/datasette_socrata/__init__.py#L264-L280
• https://github.com/simonw/datasette-lite/blob/caa8eade10f0321c64f9f65c4561186f02d57c5b/webworker.js#L55-L64

Refs:
- https://github.com/simonw/datasette-lite/issues/32

To allow users customise the facet content as they would prefer such as sorting of facet results. ordering of suggested facets etc.

I forgot to add equivalents of extras_key= and ignore_extras= to the CLI tool - will do that in a separate issue.

_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155767915_

The program crashes without any error.

wget "https://artsdatabanken.no/Fab2018/api/export/csv"
sqlite-utils create-database test.db
sqlite-utils insert --csv --delimiter ";" --encoding "utf-16-le" test test.db csv 
  [------------------------------------]    0%
  [#################-------------------]   49%  00:00:01

I would like to highlight various issues:
1. sqlite-utils catches exceptions without printing the stacktrace and/or reraising the exception, so there is no easy way to use pdb or similar to debug the program, solution: add a debug option
2. Silent crash: this is related to (1.), and it happens when there is a catch-all mechanism; solution: let the program fail.

I'm trying to figure out a way to get the table.extract() method to complete successfully -- I'm not sure if maybe the cause (and a possible solution) of this on Ubuntu Server 22.04 is to adjust some of the PRAGMA values within SQLite itself ... on another Linux system (PopOS), using this method on this same database appears to work just fine.

Here's the bit that's causing the error, and the resulting error output:

# combine these columns into 1 table "bib_properties" :
# best_title
# bib_level_code
# mat_type
# material_code
# best_author
db["circ_trans"].extract(
    ["best_title", "bib_level_code", "mat_type", "material_code", "best_author"], 
    table="bib_properties", 
    fk_column="bib_properties_id"
)

db["circ_trans"].extract(
    ["call_number"], 
    table="call_number", 
    fk_column="call_number_id",
    rename={"call_number": "value"}
)

---------------------------------------------------------------------------
OperationalError                          Traceback (most recent call last)
Input In [17], in <cell line: 7>()
      1 # combine these columns into 1 table "bib_properties" :
      2 # best_title
      3 # bib_level_code
      4 # mat_type
      5 # material_code
      6 # best_author
----> 7 db["circ_trans"].extract(
      8     ["best_title", "bib_level_code", "mat_type", "material_code", "best_author"], 
      9     table="bib_properties", 
     10     fk_column="bib_properties_id"
     11 )
     13 db["circ_trans"].extract(
     14     ["call_number"], 
     15     table="call_number", 
     16     fk_column="call_number_id",
     17     rename={"call_number": "value"}
     18 )

File ~/jupyter/venv/lib/python3.10/site-packages/sqlite_utils/db.py:1764, in Table.extract(self, columns, table, fk_column, rename)
   1761         column_order.append(c.name)
   1763 # Drop the unnecessary columns and rename lookup column
-> 1764 self.transform(
   1765     drop=set(columns),
   1766     rename={magic_lookup_column: fk_column},
   1767     column_order=column_order,
   1768 )
   1770 # And add the foreign key constraint
   1771 self.add_foreign_key(fk_column, table, "id")

File ~/jupyter/venv/lib/python3.10/site-packages/sqlite_utils/db.py:1526, in Table.transform(self, types, rename, drop, pk, not_null, defaults, drop_foreign_keys, column_order)
   1524 with self.db.conn:
   1525     for sql in sqls:
-> 1526         self.db.execute(sql)
   1527     # Run the foreign_key_check before we commit
   1528     if pragma_foreign_keys_was_on:

File ~/jupyter/venv/lib/python3.10/site-packages/sqlite_utils/db.py:465, in Database.execute(self, sql, parameters)
    463     return self.conn.execute(sql, parameters)
    464 else:
--> 465     return self.conn.execute(sql)

OperationalError: database or disk is full

This database is about 17G in total size, so I'm assuming the error is coming from the vacuum ... where i'm assuming it's maybe trying to do the temp storage in a location that doesn't have sufficient room. The disk space is more than ample on the host in question (1.8T is free in the directory where the sqlite db resides) The /tmp directory however is limited on a smaller disk associated with the OS

I'm trying to think if there's a way to set the PRAGMA temp_store or maybe if it's temp_store_directory that I'm after ... to use the same local directory of where the file is located (maybe this is a property of the version of sqlite on the system?)

# SET the temp file store to be a file ...
print(db.execute('PRAGMA temp_store').fetchall())
print(db.execute('PRAGMA temp_store=FILE').fetchall())

print(db.execute('PRAGMA temp_store').fetchall())

# the users home directory ...
print(db.execute("PRAGMA temp_store_directory='/home/plchuser/'").fetchall())
print(db.execute("PRAGMA sqlite3_temp_directory='/home/plchuser/'").fetchall())

print(db.execute("PRAGMA temp_store_directory").fetchall())
print(db.execute("PRAGMA sqlite3_temp_directory").fetchall())

[(1,)]
[]
[(1,)]
[]
[]
[('/home/plchuser/',)]
[]

Here's the docs on the Temporary File Storage Locations
https://www.sqlite.org/tempfiles.html

Windows 10
Python 3.9.6

When I was running a full text search through the Python library, I noticed that the query was being run on a different full text search table than the one I was trying to search.

I took a look at the following function

https://github.com/simonw/sqlite-utils/blob/841ad44bacaff05ec79ef78166d12e80c82ba6d7/sqlite_utils/db.py#L2213

and noticed:

sql LIKE '%VIRTUAL TABLE%USING FTS%content=%{table}%'

My database contains tables with similar names and %{table}% was matching another table that ended differently in its name.
I have included a sample test that shows this occurring:

I search for Marsupials in db["books"] and The Clue of the Broken Blade is returned.

This occurs since the search for Marsupials was "successfully" done against db["booksb"] and rowid 1 is returned. "The Clue of the Broken Blade" has a rowid of 1 in db["books"] and this is what is returned from the search.

def test_fts_search_with_similar_table_names(fresh_db):
    db = Database(memory=True)
    db["books"].insert_all(
        [
            {
                "title": "The Clue of the Broken Blade",
                "author": "Franklin W. Dixon",
            },
            {
                "title": "Habits of Australian Marsupials",
                "author": "Marlee Hawkins",
            },
        ]
    )
    db["booksb"].insert(
        {
            "title": "Habits of Australian Marsupials",
            "author": "Marlee Hawkins",
        }
    )

    db["booksb"].enable_fts(["title", "author"])
    db["books"].enable_fts(["title", "author"])


    query = "Marsupials"

    assert [
            {   "rowid": 1,
                "title": "Habits of Australian Marsupials",
                "author": "Marlee Hawkins",
            },
        ] == list(db["books"].search(query))

Original title: csv.DictReader can have None as key

In some cases, csv.DictReader can have None as key for unnamed columns, and a list of values as value.
sqlite_utils.utils.rows_from_file cannot handle that:

url="https://artsdatabanken.no/Fab2018/api/export/csv"
db = sqlite_utils.Database(":memory")

with urlopen(url) as fab:
    reader, _ = sqlite_utils.utils.rows_from_file(fab, encoding="utf-16le")   
    db["fab2018"].insert_all(reader, pk="Id")

Result:

Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 2924, in insert_all
    chunk = list(chunk)
  File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 3454, in fix_square_braces
    if any("[" in key or "]" in key for key in record.keys()):
  File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 3454, in <genexpr>
    if any("[" in key or "]" in key for key in record.keys()):
TypeError: argument of type 'NoneType' is not iterable

Code:
https://github.com/simonw/sqlite-utils/blob/59be60c471fd7a2c4be7f75e8911163e618ff5ca/sqlite_utils/db.py#L3454

sqlite-utils insert from command line is not affected by this issue.

Hi,

I noticed rows_where() doesn't return any rows from tables which are from attached databases. The exists() function returns false. As far as I can see this is because the table_names() function only looks for table names in the current database and not in attached (or temp) databases.

Besides, rows_where(), also insert_all() and delete_where() didn't do what I was expecting because of this. For the moment I've patched table_names() for myself, see below but I'm not sure what the total impact is on the other functions like lookup truncate etc which all use exists(). Also view_names() doesn't look for views in attached or temp databases.

 def table_names(self, fts4: bool = False, fts5: bool = False) -> List[str]:
        "A list of string table names in this database."
        where = ["type = 'table'"]
        if fts4:
            where.append("sql like '%USING FTS4%'")
        if fts5:
            where.append("sql like '%USING FTS5%'")
        dbs = [x[1] for x in self.execute('pragma database_list').fetchall()]    
        lst=[]
        for db in dbs:    
            sql = "select name from {} where {}".format(db+".sqlite_master"," AND ".join(where))
            lst.extend(r[0] for r in self.execute(sql).fetchall())
        return lst

What is the right way to limit a full text search query to some rows of a table?

For example, I have a table that contains the following columns: title, content, owner (each row represents a document). The owner column is a username. It feels right to store all documents in one table, instead of having one table per owner. In particular because I'd like to full text search all documents, only documents owned by one user and documents owned by a set of users.

I tried to combine .rows_where("owner = ?", "1234") and .search() from the Table class but I don't think that is meant to work. I discovered .search_sql() as a way to generate the FTS SQL statement. By hand I can edit it to add a AND [original].[owner] = :owner to the where clause. This seems to do what I want.

My two questions:
1. is adding a AND ... to the where clause actually the right thing to do or should I be doing something else (my SQL skills are low)?
2. is there a built-in to sqlite-utils way to achieve this?

Right now I am thinking I will make my own version of search_sql() that generates a query that contains an additional owner = :owner for my particular use-case.

Bonus question: is this generally useful/something to add to sqlite-utils or too niche?

I'm not sure why this happens but sqlite-utils makes my terminal cursor disappear after running commands like sqlite-utils insert. I've only noticed this behavior in sqlite-utils. Not sure if it is a bug in kitty, fish or sqlite-utils

I can still type commands after it runs but the text cursor is invisible

This code here runs only if cli.py is imported: https://github.com/simonw/sqlite-utils/blob/7ddf5300886a32d6daf60cf1d71efe492b65c87e/sqlite_utils/cli.py#L50-L59

I found myself needing the same fix in another library:

• https://github.com/simonw/datasette-socrata/issues/13

It should be a documented utility function.

It would be neat if there was a way to use this more seamlessly with Pandas, in particular Pandas dataframes - but without making Pandas a required dependency.

This allows _json to accept a wildcard for when there are many JSON columns that the user wants to convert. I hope this is useful. I've tested it on our datasette and haven't ran into any issues. I imagine on a large set of results, there could be some performance issues, but it will probably be negligible for most use cases.

On a side note, I ran into an issue where I had to upgrade black on my system beyond the pinned version in setup.py. Here is the upstream issue <https://github.com/psf/black/issues/2964 . I didn't include this in the PR yet since I didn't look into the issue too far, but I can if you would like.

Currently for canned queries, you have to inline SQL in your metadata.yaml like so:

databases:
  fixtures:
    queries:
      neighborhood_search:
        sql: |-
          select neighborhood, facet_cities.name, state
          from facetable
            join facet_cities on facetable.city_id = facet_cities.id
          where neighborhood like '%' || :text || '%'
          order by neighborhood
        title: Search neighborhoods

This works fine, but for a few reasons, I usually have my canned queries already written in separate .sql files. I'd like to instead re-use those instead of re-writing it.

So, I'd like to see a new "sql_file" key that works like so:

metadata.yaml:

databases:
  fixtures:
    queries:
      neighborhood_search:
        sql_file: neighborhood_search.sql
        title: Search neighborhoods

neighborhood_search.sql:

select neighborhood, facet_cities.name, state
from facetable
join facet_cities on facetable.city_id = facet_cities.id
where neighborhood like '%' || :text || '%'
order by neighborhood

Both of these would work in the exact same way, where Datasette would instead open + include neighborhood_search.sql on startup.

A few reasons why I'd like to keep my canned queries SQL separate from metadata.yaml:

• Keeping SQL in standalone SQL files means syntax highlighting and other text editor integrations in my code
• Multiline strings in yaml, while functional, are a tad cumbersome and are hard to edit
• Works well with other tools (can pipe .sql files into the sqlite3 CLI, or use with other SQLite clients easier)
• Typically my canned queries are quite long compared to everything else in my metadata.yaml, so I'd love to separate it where possible

Let me know if this is a feature you'd like to see, I can try to send up a PR if this sounds right!

The --create option currently creates databases if they are missing, but does not enable WAL mode for them.

It turns out WAL mode is useful for databases that are accepting writes!

I think a --create-wal option that both creates them AND sets WAL mode on any that are created would be a good idea.

I've read issue #514 which resulted in running Datasette via systemd as recommended approach. We've also adopted this (for now), but I notice that Uvicorn says the following:

Uvicorn includes a Gunicorn worker class allowing you to run ASGI applications, with all of Uvicorn's performance benefits, while also giving you Gunicorn's fully-featured process management.

This allows you to increase or decrease the number of worker processes on the fly, restart worker processes gracefully, or perform server upgrades without downtime.

For production deployments we recommend using gunicorn with the uvicorn worker class.

We usually deploy Python applications via Gunicorn for these process management features (e.g. --daemon and --pid). Is this something that would/could work with Datasette as well?

Under some conditions, GitHub caches removed starred repositories and ends up leaving dangling None user references.

Traceback (most recent call last):
  File "/home/dogsheep/dogsheep/github-to-sqlite/bin/github-to-sqlite", line 8, in <module>
    sys.exit(cli())
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/github_to_sqlite/cli.py", line 181, in starred
    utils.save_stars(db, user, stars)
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/github_to_sqlite/utils.py", line 494, in save_stars
    repo_id = save_repo(db, repo)
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/github_to_sqlite/utils.py", line 308, in save_repo
    to_save["owner"] = save_user(db, to_save["owner"])
  File "/home/dogsheep/dogsheep/github-to-sqlite/lib64/python3.10/site-packages/github_to_sqlite/utils.py", line 229, in save_user
    for key, value in user.items()
AttributeError: 'NoneType' object has no attribute 'items'

Restrict the GitHub token permissions only to the required ones; this way, even if the attackers will succeed in compromising your workflow, they won’t be able to do much.

• Included permissions for the action. https://github.com/ossf/scorecard/blob/main/docs/checks.md#token-permissions

https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#permissions

https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs

Keeping your GitHub Actions and workflows secure Part 1: Preventing pwn requests

Signed-off-by: naveen 172697+naveensrinivasan@users.noreply.github.com

Fixes a typo.

It seems to me Janus dependency is not necessary, async with app.database_write_mutex(): out = await app.transaction(func) may be enough.

Comment here: https://lobste.rs/s/fki4tj/architecture_notes_datasette#c_a2ihon

(As you will be able to tell from the terminology I use, I am not a frontend guy, but I hope you will understand.)

When a table is wide and needs horizontal scrolling to see the columns towards the end, the user needs to scroll horizontally. However, since the container for the HTML table (div with class table-wrapper) isn't limited by the window size, I first need to vertically scroll near to the bottom of the page in order to scroll horizontally. Then I can scroll back up again. This isn't very user friendly. Instead, I think it would make sense to constrain the table's size (when necessary), so that the vertical and horizontal scrollbars either always are visible or at least not far out of reach.

I understand that I could provide my own template and / or CSS, but I think it would probably make sense to adjust the default in this regard.

I think it would be great if the Datasette metadata would allow the label_column table key to list multiple columns. Something like:

    "tables": {
        "person": {
            "label_column": ["first_name", "last_name"]
        },

It would even be interesting with a "label expression" similar to a Python f-string. E.g. {row.last_name}, {row.first_name}.

I am trying to insert https://artsdatabanken.no/Fab2018/api/export/csv into a SQLite database, but I have an error when using sqlite-utils:

sqlite-utils insert --csv --delimiter ";" --encoding="utf-16-le" --pk "Id" csv fremmedart test.db
  [------------------------------------]    0%
Error: 'utf-16-le' codec can't decode bytes in position 98-99: illegal UTF-16 surrogate

The input you provided uses a character encoding other than utf-8.

You can fix this by passing the --encoding= option with the encoding of the file.

If you do not know the encoding, running 'file filename.csv' may tell you.

It's often worth trying: --encoding=latin-1

I tried to convert the file using iconv -f "utf-16le" -t "utf-8", but I still get a similar error (slightly different position):

sqlite-utils insert --csv --delimiter ";" --encoding=utf-8 --pk "Id" csv_utf8 fremmedart test.db
  [------------------------------------]    0%
Error: 'utf-8' codec can't decode byte 0xd9 in position 99: invalid continuation byte

The input you provided uses a character encoding other than utf-8.

You can fix this by passing the --encoding= option with the encoding of the file.

If you do not know the encoding, running 'file filename.csv' may tell you.

It's often worth trying: --encoding=latin-1

I have no issues reading such file using this Python code:

content = open('csv', encoding='utf-16-le').read())

in2csv works too.

Fix typo

A search of the plugins directory doesn't turn up anything, but is is possible to set up a Datasette app which uses my organisation's LDAP for auth?

If not, how much work would it be to write one (I may have some spare cycles on my team to do this, but we haven't written a datasette plugin before).

Ran the auth command in the same directory I'd previously set up an auth.json file for twitter-to-sqlite and it was completely overwritten. Not the biggest issue, but still unexpected. Ideally, for me, the keys would just be added to the existing file, but getting a warning and a chance to back out would be a good solution as well.

I'm not sure if this is due to a new format or just because my ageinfo file is blank, but trying to import an archive would crash when it got to that file. This PR adds a guard clause in the ageinfo transformer and sets a default value that doesn't throw an exception. Seems likely to be the same issue mentioned by danp in https://github.com/dogsheep/twitter-to-sqlite/issues/54, my ageinfo file looks the same. Added that same ageinfo file to the test archive as well to help confirm my workaround didn't break anything.

Let me know if you want any changes!

Follows:
- #435

Imitates:
- https://github.com/simonw/datasette/issues/1748

I'll use https://github.com/executablebooks/sphinx-copybutton - here's the Datasette commit: https://github.com/simonw/datasette/commit/1465fea4798599eccfe7e8f012bd8d9adfac3039

As seen in:
- https://github.com/simonw/datasette/issues/1746
- https://github.com/simonw/shot-scraper/issues/77

https://github.com/pradyunsg/furo

I just did this for shot-scraper and I really like it: https://shot-scraper.datasette.io/en/latest/

• https://github.com/simonw/shot-scraper/issues/77

YAML configuration is much better for multi-line strings, and I'm increasingly adding configuration options to Datasette that benefit from that - fragments of HTML in description_html or SQL queries used to configure things like https://github.com/simonw/datasette-atom for example.

Rather than confusing things by showing both in the documentation, I should switch all of the default examples to use YAML instead.

Similar to the ones in datasette-copyable which are implemented here: https://github.com/executablebooks/sphinx-copybutton/tree/f84c001a0507f8ec46779d0701b079a265564583

On https://docs.datasette.io/en/stable/getting_started.html

Got this error while investigating:
- #421

Even though I was using the LD_PRELOAD trick from https://til.simonwillison.net/sqlite/ld-preload to use a newer version of SQLite.

_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098531354_

The getting started docs currently suggest you try this to browse your Chrome history:

datasette ~/Library/Application\ Support/Google/Chrome/Default/History

But if Chrome is running you will likely get this error:

sqlite3.OperationalError: database is locked

Turns out there's a workaround for this which I just spotted on the SQLite forum:

You can do this using a URI filename:
sqlite3 'file:places.sqlite?mode=ro&nolock=1'
That opens the file places.sqlite in read-only mode with locking disabled. This isn't safe, in that changes to the database made by other corrections are likely to cause this connection to return incorrect results or crash. Read-only mode should at least mean that you don't corrupt the database in the process.

Noticed that cog -r docs/*.rst isn't documented in https://docs.datasette.io/en/latest/contributing.html#editing-and-building-the-documentation

Because I'm using it in this plugin:
- https://github.com/simonw/datasette-upload-dbs/issues/1

view-source:https://calands.datasettes.com/calands/CPAD_2020a_SuperUnits.geojson?_sort=id&id__exact=4&_labels=on&_trace=1 is showing me a blank page.

See https://github.com/simonw/datasette-publish-now/issues/9#issuecomment-618155764 - the --extra-options mechanism is in practice just used to set --config options in data that you publish, but that means you end up with pretty messy looking commands:

datasette publish my.db --extra-options="--config default_page_size:50 --config sql_time_limit_ms:3500"

A neater design would be to support --config as an option for datasette publish directly:

datasette publish my.db --config default_page_size:50 --config sql_time_limit_ms:3500

Wide tables aren't a huge problem if you know to click and drag right. But it's not at all obvious to do that. (it also tends to blue-select any content as it's dragging.) Depending on column widths, public users might entirely miss all the columns to the right.

There is a scrollbar at the bottom of the table, but I'm displaying ALL my records because it's the only way for datasette-vega to make accurate charts. So that bottom scrollbar is likely to be missed. I wonder if some sort of javascript-y mouseover to an arrow might help, similar to those seen in image carousels. Ah: here's a perfect example:

1. Visit http://google.com
2. Search for: animals endangered
3. Note the 'g-right-button' (in the code) that looks like a right-facing caret in a circle.
4. Click on that and the carousel scrolls right (and 'g-left-button' appears on the left).

Might be tricky to do that on a table, rather than a one-row carousel, but it's worth experimenting with.

Another option is just to put the scrollbars at the top of the table, too.

Meantime, I'm trying to build a button like the "View/hide all columns on https://salaries.news.baltimoresun.com/salaries-be494cf/2019+Maryland+state+salaries
Might be nice to have that available by default, with settings in the metadata showing which are on by default.

(I saw some other closed issues related to horizontal scrolling, and admit I don't entirely understand them. For instance, the animated gif at https://github.com/simonw/datasette/issues/998#issuecomment-714117534 confuses me. )

I noticed that my Pyodide Datasette prototype is downloading the same database file every single time rather than browser caching it:

I've already got this working as a prototype. Here are the changes I had to make:

• Replace the two dependencies that don't publish pure Python wheels to PyPI: click-default-group and python-baseconv
• Get Datasette to work without threading - which it turns out is exclusively used for database connections
• Make the uvicorn dependency optional (only needed when Datasette runs in the CLI)

TODO:

• Switch to click-default-group-wheel
• https://github.com/simonw/datasette/issues/1734
• Work around uvicorn import error
• https://github.com/simonw/datasette/issues/1735
• #1737

Goal is to be able to do the following directly in https://pyodide.org/en/stable/console.html

import micropip
await micropip.install("datasette")
from datasette.app import Datasette
ds = Datasette()
await ds.client.get("/.json")

Exceptions are normal expected behavior when typecasting an invalid format. However, r.parsedate() is really just re-formatting strings and keeping the type as text. So it may be better to print-and-pass on exception so the user can see a complete list of invalid values -- while also allowing for the parser to reformat the remaining valid values.

sqlite-utils convert idfpr.db license "Expiration Date" "r.parsedate(value)"
  [#######-----------------------------]   21%  00:01:57Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/sqlite_utils/db.py", line 2336, in convert_value
    return fn(v)
  File "<string>", line 2, in fn
  File "/usr/local/lib/python3.9/dist-packages/sqlite_utils/recipes.py", line 8, in parsedate
    parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat()
  File "/usr/lib/python3/dist-packages/dateutil/parser/_parser.py", line 1374, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/usr/lib/python3/dist-packages/dateutil/parser/_parser.py", line 652, in parse
    raise ParserError("String does not contain a date: %s", timestr)
dateutil.parser._parser.ParserError: String does not contain a date:   /  /    

In this case, I had just one variation of an invalid date: ' / / '. But theoretically there could be many values that would have to be fixed one at a time with the current exception handling.

I have a page templates/filer/{filer_id}.html. It uses filer_id in a sql() call to fetch data. With 0.61.1 this no longer works because the spaces in IDs isn't preserved. Instead, the escaped version is passed into the template and the id isn't present in my db.

Datasette should unescape the url component before passing them into the template.

Triggered this error while playing with the sort desc checkbox and the apply button that are only visible on this page at mobile screen width:

https://latest.datasette.io/fixtures/compound_three_primary_keys?_sort_desc=pk1

Navigate to that page (with the browser narrow enough to show the box), un-check the box and click Apply:

Also notable: I managed to get to a page with ?_sort_desk=pk1 in the URL three times by clicking around with that button.

Refs:
- #1733

Need something in the test suite such that if Datasette breaks against Pyodide in the future we hear about it.

I'm thinking this is an opportunity to use shot-scraper javascript.

I was going to vendor baseconv.py, but then I reconsidered - what if there are plugins out there that expect import baseconv to work because they have depended on Datasette?

I used https://cs.github.com/ and as far as I can tell there aren't any!

So I'm going to remove that dependency and work out a smarter way to do this - probably by providing a utility function within Datasette itself.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1733#issuecomment-1115258737_

I'm going to add a Datasette setting to disable threading entirely, designed for usage in this particular case.

I thought about adding a new setting, then I noticed this:

datasette mydatabase.db --setting num_sql_threads 10

I'm going to let users set that to 0 to disable threaded execution of SQL queries.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1733#issuecomment-1115278325_

Closes #1735, closes #1733

Trying to get this to work with Pyodide.

Refs: https://github.com/simonw/click-default-group-wheel/issues/3

I unfortunately was not successful to deploy to fly.io. Please see the details above of the three scenarios that I took. I am also new to datasette.

Failed to deploy. Attaching logs:
1. Tried with an app created via flyctl apps create frosty-fog-8565 and the ran datasette publish fly covid.db --app frosty-fog-8565

Deploying frosty-fog-8565
==> Validating app configuration
--> Validating app configuration done
Services
TCP 80/443 ⇢ 8080

Error error connecting to docker: An unknown error occured.

Traceback (most recent call last):
  File "c:\users\grott\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\grott\anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\grott\Anaconda3\Scripts\datasette.exe\__main__.py", line 7, in <module>
  File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\grott\anaconda3\lib\site-packages\datasette_publish_fly\__init__.py", line 156, in fly
    "--remote-only",
  File "c:\users\grott\anaconda3\lib\contextlib.py", line 119, in __exit__
    next(self.gen)
  File "c:\users\grott\anaconda3\lib\site-packages\datasette\utils\__init__.py", line 451, in temporary_docker_directory
    tmp.cleanup()
  File "c:\users\grott\anaconda3\lib\tempfile.py", line 811, in cleanup
    _shutil.rmtree(self.name)
  File "c:\users\grott\anaconda3\lib\shutil.py", line 516, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "c:\users\grott\anaconda3\lib\shutil.py", line 395, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\users\grott\anaconda3\lib\shutil.py", line 404, in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
  File "c:\users\grott\anaconda3\lib\shutil.py", line 402, in _rmtree_unsafe
    os.rmdir(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\grott\\AppData\\Local\\Temp\\tmpgcm8cz66\\frosty-fog-8565'

1. Tried also with an app that gets autogenerate when running flyctl launch. This also generates the .toml file. Ran then datasette publish fly covid.db --app dark-feather-168 but different error now
   ```Deploying dark-feather-168
   ==> Validating app configuration

Error not possible to validate configuration: server returned Post "https://api.fly.io/graphql": unexpected EOF

Traceback (most recent call last):
File "c:\users\grott\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
exec(code, run_globals)
File "C:\Users\grott\Anaconda3\Scripts\datasette.exe__main__.py", line 7, in <module>
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 829, in call
return self.main(args, kwargs)
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, ctx.params)
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 610, in invoke
return callback(args, **kwargs)
File "c:\users\grott\anaconda3\lib\site-packages\datasette_publish_fly__init__.py", line 156, in fly
"--remote-only",
File "c:\users\grott\anaconda3\lib\contextlib.py", line 119, in exit
next(self.gen)
File "c:\users\grott\anaconda3\lib\site-packages\datasette\utils__init__.py", line 451, in temporary_docker_directory
tmp.cleanup()
File "c:\users\grott\anaconda3\lib\tempfile.py", line 811, in cleanup
_shutil.rmtree(self.name)
File "c:\users\grott\anaconda3\lib\shutil.py", line 516, in rmtree
return _rmtree_unsafe(path, onerror)
File "c:\users\grott\anaconda3\lib\shutil.py", line 395, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "c:\users\grott\anaconda3\lib\shutil.py", line 404, in _rmtree_unsafe
onerror(os.rmdir, path, sys.exc_info())
File "c:\users\grott\anaconda3\lib\shutil.py", line 402, in _rmtree_unsafe
os.rmdir(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\Users\grott\AppData\Local\Temp\tmpnoyewcre\dark-feather-168'

These are also the contents of the generated **.toml file** in 2 scenario:

fly.toml file generated for dark-feather-168 on 2021-09-28T20:35:44+02:00

app = "dark-feather-168"

kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]

[experimental]
allowed_public_ports = []
auto_rollback = true

[[services]]
http_checks = []
internal_port = 8080
processes = ["app"]
protocol = "tcp"
script_checks = []

[services.concurrency]
hard_limit = 25
soft_limit = 20
type = "connections"

[[services.ports]]
handlers = ["http"]
port = 80

[[services.ports]]
handlers = ["tls", "http"]
port = 443

[[services.tcp_checks]]
grace_period = "1s"
interval = "15s"
restart_limit = 6
timeout = "2s"
```

1. But also trying datasette package covid.db to create a local DOCKERFILE to later try to push it via flyctl deploy fails as well.

```[+] Building 147.3s (11/11) FINISHED
=> [internal] load build definition from Dockerfile 0.2s
=> => transferring dockerfile: 396B 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/python:3.8 4.7s
=> [auth] library/python:pull token for registry-1.docker.io 0.0s
=> [internal] load build context 0.1s
=> => transferring context: 82.37kB 0.0s
=> [1/5] FROM docker.io/library/python:3.8@sha256:530de807b46a11734e2587a784573c12c5034f2f14025f838589e6c0e3 108.3s
=> => resolve docker.io/library/python:3.8@sha256:530de807b46a11734e2587a784573c12c5034f2f14025f838589e6c0e3b5 0.0s
=> => sha256:56182bcdf4d4283aa1f46944b4ef7ac881e28b4d5526720a4e9ba03a4730846a 2.22kB / 2.22kB 0.0s
=> => sha256:955615a668ce169f8a1443fc6b6e6215f43fe0babfb4790712a2d3171f34d366 54.93MB / 54.93MB 21.6s
=> => sha256:911ea9f2bd51e53a455297e0631e18a72a86d7e2c8e1807176e80f991bde5d64 10.87MB / 10.87MB 15.5s
=> => sha256:530de807b46a11734e2587a784573c12c5034f2f14025f838589e6c0e3b5c5b6 1.86kB / 1.86kB 0.0s
=> => sha256:ff08f08727e50193dcf499afc30594c47e70cc96f6fcfd1a01240524624264d0 8.65kB / 8.65kB 0.0s
=> => sha256:2756ef5f69a5190f4308619e0f446d95f5515eef4a814dbad0bcebbbbc7b25a8 5.15MB / 5.15MB 6.4s
=> => sha256:27b0a22ee906271a6ce9ddd1754fdd7d3b59078e0b57b6cc054c7ed7ac301587 54.57MB / 54.57MB 37.7s
=> => sha256:8584d51a9262f9a3a436dea09ba40fa50f85802018f9bd299eee1bf538481077 196.45MB / 196.45MB 82.3s
=> => sha256:524774b7d3638702fe9ae0ea3fcfb81b027dfd75cc2fc14f0119e764b9543d58 6.29MB / 6.29MB 26.6s
=> => extracting sha256:955615a668ce169f8a1443fc6b6e6215f43fe0babfb4790712a2d3171f34d366 5.4s
=> => sha256:9460f6b75036e38367e2f27bb15e85777c5d6cd52ad168741c9566186415aa26 16.81MB / 16.81MB 40.5s
=> => extracting sha256:2756ef5f69a5190f4308619e0f446d95f5515eef4a814dbad0bcebbbbc7b25a8 0.6s
=> => extracting sha256:911ea9f2bd51e53a455297e0631e18a72a86d7e2c8e1807176e80f991bde5d64 0.6s
=> => sha256:9bc548096c181514aa1253966a330134d939496027f92f57ab376cd236eb280b 232B / 232B 40.1s
=> => extracting sha256:27b0a22ee906271a6ce9ddd1754fdd7d3b59078e0b57b6cc054c7ed7ac301587 5.8s
=> => sha256:1d87379b86b89fd3b8bb1621128f00c8f962756e6aaaed264ec38db733273543 2.35MB / 2.35MB 41.8s
=> => extracting sha256:8584d51a9262f9a3a436dea09ba40fa50f85802018f9bd299eee1bf538481077 18.8s
=> => extracting sha256:524774b7d3638702fe9ae0ea3fcfb81b027dfd75cc2fc14f0119e764b9543d58 1.2s
=> => extracting sha256:9460f6b75036e38367e2f27bb15e85777c5d6cd52ad168741c9566186415aa26 2.9s
=> => extracting sha256:9bc548096c181514aa1253966a330134d939496027f92f57ab376cd236eb280b 0.0s
=> => extracting sha256:1d87379b86b89fd3b8bb1621128f00c8f962756e6aaaed264ec38db733273543 0.8s
=> [2/5] COPY . /app 2.3s
=> [3/5] WORKDIR /app 0.2s
=> [4/5] RUN pip install -U datasette 26.9s
=> [5/5] RUN datasette inspect covid.db --inspect-file inspect-data.json 3.1s
=> exporting to image 1.2s
=> => exporting layers 1.2s
=> => writing image sha256:b5db0c205cd3454c21fbb00ecf6043f261540bcf91c2dfc36d418f1a23a75d7a 0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Traceback (most recent call last):
"main", mod_spec)
File "c:\users\grott\anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\grott\Anaconda3\Scripts\datasette.exe__main__.py", line 7, in <module>
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 829, in call
return self.main(args, kwargs)
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, ctx.params)
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 610, in invoke
return callback(args, **kwargs)
File "c:\users\grott\anaconda3\lib\site-packages\datasette\cli.py", line 283, in package
call(args)
File "c:\users\grott\anaconda3\lib\contextlib.py", line 119, in exit
next(self.gen)
File "c:\users\grott\anaconda3\lib\site-packages\datasette\utils__init__.py", line 451, in temporary_docker_directory
tmp.cleanup()
File "c:\users\grott\anaconda3\lib\tempfile.py", line 811, in cleanup
_shutil.rmtree(self.name)
File "c:\users\grott\anaconda3\lib\shutil.py", line 516, in rmtree
return _rmtree_unsafe(path, onerror)
File "c:\users\grott\anaconda3\lib\shutil.py", line 395, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "c:\users\grott\anaconda3\lib\shutil.py", line 404, in _rmtree_unsafe
onerror(os.rmdir, path, sys.exc_info())
File "c:\users\grott\anaconda3\lib\shutil.py", line 402, in _rmtree_unsafe
os.rmdir(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\Users\grott\AppData\Local\Temp\tmpkb27qid3\datasette'```

I added parallel SQL query execution here:
- https://github.com/simonw/datasette/issues/1723

My hunch is that this will take advantage of multiple cores, since Python's sqlite3 module releases the GIL once a query is passed to SQLite.

I'd really like to prove this is the case though. Just not sure how to do it!

Larger question: is this performance optimization actually improving performance at all? Under what circumstances is it worthwhile?

Part of:
- #262
- #1518

In #1727 I realized that the SQL tracing was measuring a whole bunch of stuff outside of the SQL query itself.

I started experimenting with this fix for that but it didn't work - I got back an empty JSON array of traces for some reason:

diff --git a/datasette/database.py b/datasette/database.py
index ba594a8..d7f9172 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -7,7 +7,7 @@ import sys
 import threading
 import uuid

-from .tracer import trace
+from .tracer import trace, trace_child_tasks
 from .utils import (
     detect_fts,
     detect_primary_keys,
@@ -207,30 +207,31 @@ class Database:
                 time_limit_ms = custom_time_limit

             with sqlite_timelimit(conn, time_limit_ms):
-                try:
-                    cursor = conn.cursor()
-                    cursor.execute(sql, params if params is not None else {})
-                    max_returned_rows = self.ds.max_returned_rows
-                    if max_returned_rows == page_size:
-                        max_returned_rows += 1
-                    if max_returned_rows and truncate:
-                        rows = cursor.fetchmany(max_returned_rows + 1)
-                        truncated = len(rows) > max_returned_rows
-                        rows = rows[:max_returned_rows]
-                    else:
-                        rows = cursor.fetchall()
-                        truncated = False
-                except (sqlite3.OperationalError, sqlite3.DatabaseError) as e:
-                    if e.args == ("interrupted",):
-                        raise QueryInterrupted(e, sql, params)
-                    if log_sql_errors:
-                        sys.stderr.write(
-                            "ERROR: conn={}, sql = {}, params = {}: {}\n".format(
-                                conn, repr(sql), params, e
+                with trace("sql", database=self.name, sql=sql.strip(), params=params):
+                    try:
+                        cursor = conn.cursor()
+                        cursor.execute(sql, params if params is not None else {})
+                        max_returned_rows = self.ds.max_returned_rows
+                        if max_returned_rows == page_size:
+                            max_returned_rows += 1
+                        if max_returned_rows and truncate:
+                            rows = cursor.fetchmany(max_returned_rows + 1)
+                            truncated = len(rows) > max_returned_rows
+                            rows = rows[:max_returned_rows]
+                        else:
+                            rows = cursor.fetchall()
+                            truncated = False
+                    except (sqlite3.OperationalError, sqlite3.DatabaseError) as e:
+                        if e.args == ("interrupted",):
+                            raise QueryInterrupted(e, sql, params)
+                        if log_sql_errors:
+                            sys.stderr.write(
+                                "ERROR: conn={}, sql = {}, params = {}: {}\n".format(
+                                    conn, repr(sql), params, e
+                                )
                             )
-                        )
-                        sys.stderr.flush()
-                    raise
+                            sys.stderr.flush()
+                        raise

             if truncate:
                 return Results(rows, truncated, cursor.description)
@@ -238,9 +239,8 @@ class Database:
             else:
                 return Results(rows, False, cursor.description)

-        with trace("sql", database=self.name, sql=sql.strip(), params=params):
-            results = await self.execute_fn(sql_operation_in_thread)
-        return results
+        with trace_child_tasks():
+            return await self.execute_fn(sql_operation_in_thread)

     @property
     def size(self):

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1727#issuecomment-1111602802_

I've been working on a dependency injection mechanism in a separate library:

• https://github.com/simonw/asyncinject

I think it's ready to try out with Datasette to see if it's a pattern that will work here.

I'm going to attempt to refactor TableView to use it. There are two overall goals here:

• Use asyncinject to add parallel execution of some aspects of the table page - most notably I want to be able to execute the count(*) query, the select ... query, the various faceting queries and the facet suggestion queries in parallel - and measure if doing so is good for performance.
• Use it to execute different output formats (possibly with some changes to the existing register_output_renderer() plugin hook). I want CSV and JSON to use the same mechanism that plugins use.

Stretch goal is to get this working with streaming data too, see:

• 1101

When i hit "Apply" button to search with "_exact" for a column syntax the URL prefix is removed from the url.

And the result is:

If I add the marked row to url_builder.py it seams to work:

I've been banging my head against a wall for a while and would appreciate any pointers...

• I have a writeable canned query to update rows in the db.
• I'm using the github-oauth plugin for authentication.
• I have allow set on the query to accept my GitHub id and a GH organisation.
• Authentication seems to work as expected both locally and on Cloudrun -- viewing /-/actor gives the same result in both environments
• I can access the 'padlocked' canned query in both environments.

Everything seems to be the same, but the canned query works perfectly when run locally, and fails when I try it on Cloudrun. I'm redirected back to the canned query page and the db is not changed. There's nothing in the Cloudstor logs to indicate an error.

Any clues as to where I should be looking for the problem?

Relates to #1533 - and to the work I've been doing on the https://github.com/simonw/datasette-table Web Component.

It would be useful to support users pasting in a URL to a Datasette table or query without first having to add the .json extension themselves - since then other systems could hit that URL with Accept: application/json to get back the JSON representation without first needing to read the Link: header from #1533 to figure out what the URL to that JSON is.

(There is weird logic deep in Datasette that says that you add .json to the path UNLESS the table name itself ends with .json, in which case you add ?_format=json - this is super-confusing).

[Update: I removed that confusing feature here: https://simonwillison.net/2022/Mar/19/weeknotes/]

When you use the delete_where() function on a table, it never commits....

Is that intentional?

Spun off from:
- #1715

A page talking about how to run Datasette securely, and security concerns to take into account.

https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_trace=1 is not showing the trace JSON at the bottom of the page.

Confirmed that trace_debug is true on https://global-power-plants.datasettes.com/-/settings

Possibly related:

• https://github.com/simonw/datasette-total-page-time/issues/1

Trace from https://latest-with-plugins.datasette.io/github/commits?_facet=repo&_trace=1&_facet=committer

What's going on in that gap? Can I improve the tracing output to show some non-SQL queries to figure that out?

Noticed this while working on:
- #1715

This code here isn't showing up in traces: https://github.com/simonw/datasette/blob/579f59dcec43a91dd7d404e00b87a00afd8515f2/datasette/views/table.py#L218-L220

Because those functions don't use the regular trace-instrumented db.execute() code path - they work directly against a connection instead: https://github.com/simonw/datasette/blob/579f59dcec43a91dd7d404e00b87a00afd8515f2/datasette/utils/__init__.py#L610-L626

Designed in:
- #1720

Part of:
- #262
- #1709

Refs:
- #262
- #1709

I realized that this is a really natural plugin hook - and if I design it as a hook I can implement Datasette's core extras as default plugins.

https://www.sqlite.org/lang_savepoint.html

Savepoints are like regular transactions except they have names and can be nested.

Would there be any value in adding support to them to sqlite-utils, potentially as some kind of context manager? Something like this:

with db.savepoint("name"):
    # do stuff
    with db.savepoint("name2"):
        # do more stuff
        raise Release # Rolls back to before "name2" savepoint

I've never used this feature so I'm not comfortable adding anything like this without a bunch of extra research.

The RowTableShared class is making this a whole lot more complicated.

I'm going to split the RowView view out into an entirely separate views/row.py module.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1715#issuecomment-1108875068_

Datasette views currently work by creating a set of data that should be returned as JSON, then defining an additional, optional template_data() function which is called if the view is being rendered as HTML.

This template_data() function calculates extra template context variables which are necessary for the HTML view but should not be included in the JSON.

Example of how that is used today: https://github.com/simonw/datasette/blob/2b79f2bdeb1efa86e0756e741292d625f91cb93d/datasette/views/table.py#L672-L704

With features like Facets in #255 I'm beginning to want to move more items into the template_data() - in the case of facets it's the suggested_facets array. This saves that feature from being calculated (involving several SQL queries) for the JSON case where it is unlikely to be used.

But... as an API user, I want to still optionally be able to access that information.

Solution: Add a ?_extra=suggested_facets&_extra=table_metadata argument which can be used to optionally request additional blocks to be added to the JSON API.

Then redefine as many of the current template_data() features as extra arguments instead, and teach Datasette to return certain extras by default when rendering templates.

This could allow the JSON representation to be slimmed down further (removing e.g. the table_definition and view_definition keys) while still making that information available to API users who need it.

For example on this page: https://docs.datasette.io/en/stable/writing_plugins.html#packaging-a-plugin

I wonder if there's an easy way for me to enforce this for Sphinx documentation?

I've found that the Cloudrun build phase often hits a timeout limit with large databases. I believe the default timeout is 10 minutes. This pull request just adds a --timeout option to the cloudrun publish command and passes the value on to the build step.

GitHub can support this in blame views now too:

https://docs.github.com/en/repositories/working-with-files/using-files/viewing-a-file#ignore-commits-in-the-blame-view

Missing values in my dataset are denoted by a single asterisk. I am trying to parse string dates into dates. This works fine for columns without missing values, but, when the column contains "*", I get the following:

$ sqlite-utils convert ${dbfile} details dob 'r.parsedate(value)' 
  [------------------------------------]    0%Traceback (most recent call last):
  File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/db.py", line 2508, in convert_value
    return fn(v)
  File "<string>", line 2, in fn
  File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/recipes.py", line 8, in parsedate
    parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat()
  File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 1368, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 643, in parse
    raise ParserError("Unknown string format: %s", timestr)
dateutil.parser._parser.ParserError: Unknown string format: *

Traceback (most recent call last):
  File "/usr/local/bin/sqlite-utils", line 33, in <module>
    sys.exit(load_entry_point('sqlite-utils==3.25.1', 'console_scripts', 'sqlite-utils')())
  File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/cli.py", line 2698, in convert
    db[table].convert(
  File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/db.py", line 2524, in convert
    self.db.execute(sql, where_args or [])
  File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/db.py", line 458, in execute
    return self.conn.execute(sql, parameters)
sqlite3.OperationalError: user-defined function raised exception

fixes #1224

This PR ensures all databases found in a configuration directory that match the files in inspect-data.json will be set to immutable as outlined in https://docs.datasette.io/en/latest/settings.html#configuration-directory-mode

specifically on building the datasette instance it checks:
- if immutables is an empty tuple - as passed by the cli code
- if immutables is the default function value None - when it's not explicitly set

And correctly builds the immutable database list from the inspect-data[file] keys.

Note for this to work the inspect-data.json file must contain file paths which are relative to the configuration directory otherwise the file paths won't match and the dbs won't be set to immutable.

I couldn't find an easy way to test this due to the way make_app_client works, happy to take directions on adding a test for this.

I've updated the relevant docs as well, i.e. use the inspect cli cmd from the config directory path to create the relevant file

cd $config_dir
datasette inspect *.db --inspect-file=inspect-data.json

https://docs.datasette.io/en/latest/performance.html#using-datasette-inspect

Hi @simonw 👋

Currently, stored commit data doesn't quite give me the information I'm needing...

Committer date and author date are not 100% reliable for dividing a commit history up by release or branch.
A PR created before a release but merged after can have earlier dates… — this can be quite frustrating if you're trying to pin down commits for a release: It should be there!, but then isn't. (This gets worse using release branches.)

Would you be open to adding the sha of a parent of a commit to the commit table? (As an FK? 🤔 — likely not feasible.)

It's part of the response body:

  "parents": [
    {
      "url": "https://api.github.com/repos/octocat/Hello-World/commits/6dcb09b5b57875f334f61aebed695e2e4193db5e",
      "sha": "6dcb09b5b57875f334f61aebed695e2e4193db5e"
    }
  ],

I think this list should only have a single entry. (🤔 — not sure why it's a list then...)

With this it would be possible to build/reconstruct a chain of commits from the history, that I don't think is available as yet (unless you know a better way).

It is certainly possible to get sequential lists of commits out of git directly, so the same would be possible combining tools, but wondering if a single tool could do it.

What do you think?
Thanks! 🏅

I'd like to implement this by first extending the register_output_renderer() hook to support streaming huge responses, then switching CSV to use the plugin hook in addition to TSV using it.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1096#issuecomment-732542285_

Motivation

For a long running command (longer than 1 minute) for a big table (like pull requests or commits), it can be tricky to know if the script is still running, or if a rate limit/error was encountered

We know how many pages there are, so it may be possible to indicate how many remain.

https://github.com/dogsheep/github-to-sqlite/blob/a6e237f75a4b86963d91dcb5c9582e3a1b3349d6/github_to_sqlite/utils.py#L367

Resources

• Using the existing Click API:
• https://click.palletsprojects.com/en/5.x/utils/#showing-progress-bars
• Loading spinner: https://github.com/pavdmyt/yaspin
• Progress bar: https://github.com/tqdm/tqdm

The official python images on docker hub come with a slim variant that is significantly smaller than the default. The diff does not change the default, but allows to switch to the slim variant with commandline switch (--slim-base-image)

Size comparison:

$ datasette package some.db -t fat --install "datasette-basemap datasette-cluster-map"

$ datasette  package some.db -t slim --slim-base-image --install "datasette-basemap datasette-cluster-map"

$ docker images
REPOSITORY   TAG       IMAGE ID       CREATED          SIZE
fat          latest    807b393ace0d   9 seconds ago    978MB
slim         latest    31bc5e63505c   8 minutes ago    191MB

https://twitter.com/simonw/status/1514392335718645760

Maybe @datasetteproj should grow a feature that lets you cache the results of a query and give that snapshot a stable permalink

A plugin that publishes the JSON output of a query to an S3 bucket would be pretty neat... especially if it could also be configured to re-publish the results on a schedule

A lot of people said they would find this useful.

Probably going to build this as a plugin.

I think that each comment inserted into the database should include a column onstory that contains the ID of the story on which the comment was made. This is exactly equivalent to the link after "on:" at the top of an HN comment page (example). We could do this either by directly retrieving the HTML page and using Beautiful Soup to find that link, or alternatively recurse up the tree in the Firebase API using the parent field (probably using functools.lru_cache in case a person has commented a bunch of times on the same story).

This PR changes the display of facets to hopefully make them more readable.

For every command/API method there should be a link to the equivalent in the other form factor.

Maybe also link to the API and CLI reference pages too.

This bug relates to https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1066139147

New error when using CLI: "sqlite-utils indexes global.db --table"

(app-root) sqlite-utils indexes global.db --table
Error: near "(": syntax error
(app-root) sqlite-utils --version
sqlite-utils, version 3.25.1
(app-root) sqlite3 --version
3.36.0 2021-06-18 18:36:39
(app-root) python --version
Python 3.8.11

Dockerfile

FROM centos/python-38-centos7

USER root

RUN yum update -y
RUN yum upgrade -y


# epel
RUN yum -y install epel-release && yum clean all

# SQLite
RUN yum -y install zlib-devel geos geos-devel proj proj-devel freexl freexl-devel libxml2-devel 

WORKDIR /build/
COPY sqlite-autoconf-3360000.tar.gz ./
RUN tar -zxf sqlite-autoconf-3360000.tar.gz
WORKDIR /build/sqlite-autoconf-3360000
RUN ./configure
RUN make
RUN make install

# 
RUN /opt/app-root/bin/python3.8 -m pip install --upgrade pip
RUN pip install sqlite-utils

Seen here:

• https://github.com/simonw/geojson-to-sqlite/issues/30

Attempting to create a table with no columns produced this confusing error:

File "/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/geojson_to_sqlite/utils.py", line 69, in import_features
    db[table].create(column_types, pk=pk)
  File "/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/sqlite_utils/db.py", line 863, in create
    self.db.create_table(
  File "/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/sqlite_utils/db.py", line 517, in create_table
    self.execute(sql)
  File "/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/sqlite_utils/db.py", line 236, in execute
    return self.conn.execute(sql)
sqlite3.OperationalError: near ")": syntax error

Binary: 2,427,344 bytes would be nicer - even better, include a tooltip showing that size translated using this function: https://github.com/simonw/datasette/blob/138e4d9a53e3982137294ba383303c3a848cfca4/datasette/utils/__init__.py#L837-L846