I'm currently working on a new feature for Hypothesis, where we can dump a tidy jsonlines table of all the test cases we tried - including arguments, outcomes, timings, coverage, etc. Exploring this seems like a perfect cases for sqlite-utils and datasette, but I pretty quickly ran into an integer overflow problem and don't want to recommend that experience to my users.

I originally went to report this as a bug... and then found https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895581038 almost exactly matched my repro 😅

https://github.com/simonw/sqlite-utils/issues/110#issuecomment-626391063 suggests that using NUMERIC would avoid this overflow error, although "If the TEXT value is a well-formed integer literal that is too large to fit in a 64-bit signed integer, it is converted to REAL." suggests that this would come at the cost of rounding to the nearest float value. Maybe I should just convert large integers to float before writing out my json?

After a bit more hacking, "manually cast large integers to float" seems like a decent solution for my particular case, but having written it up I thought I might as well post this issue anyway - I hope it's useful feedback, and won't mind at all if you close as wontfix if it's not.

Initially, I found an issue in datasette where it wouldn’t find spatialite when running on my Radxa Rock 5B - an RK3588 powered SBC, running the arm64 build of Debian Bullseye. I confirmed the same behaviour on my Raspberry Pi 4 - a BCM2711 powered SBC, running the arm64 build of Debian Bookworm.

$ datasette --load-extension=spatialite example.db Error: Could not find SpatiaLite extension

I did some digging and realised the issue originates in this project. Even with the libsqlite3-mod-spatialite package installed, pytest skips all of the GIS tests in the project.

``` $ apt list --installed | grep spatial […] libsqlite3-mod-spatialite/stable,now 5.0.1-3 arm64 [installed]

$ ls -l /usr/lib//spatial* lrwxrwxrwx 1 root root 23 Dec 1 2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so -> mod_spatialite.so.7.1.0 lrwxrwxrwx 1 root root 23 Dec 1 2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so.7 -> mod_spatialite.so.7.1.0 -rw-r--r-- 1 root root 7348584 Dec 1 2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so.7.1.0 ```

$ pytest tests/test_get.py ...... [ 73%] tests/test_gis.py ssssssssssss [ 75%] tests/test_hypothesis.py .... [ 75%]

I tracked the issue down to the find_sqlite() function in the utils.py file. The SPATIALITE_PATHS array doesn’t have an entry for the location of this module on arm64 linux.

Hi ! I noticed that when I am trying to use the delete method of the Table object, the record get properly deleted from the table, but the cascading delete triggers on foreign keys do not activate.

self.db["contact"].delete(contact_id)

I tried querying the database directly via DB Browser and the triggers work without any issue. Looked up the source code and behind the scene this method is just querying the database normally so I'm not exactly sure where this behavior comes from.

Thank you in advance for your time !

I'm going to try and get this working through an environment variable, so that you can start Datasette and it will only load a subset of plugins including those that use the register_commands() hook.

Initial research on this: - https://github.com/pytest-dev/pluggy/issues/422

I've used this pattern twice now: https://til.simonwillison.net/datasette/register-new-plugin-hooks - in datasette-graphql and datasette-low-disk-space-hook. I should describe the pattern on https://docs.datasette.io/en/stable/writing_plugins.html

The BaseView class has a method for cascading permission checks, but it's not easily accessible to plugins.

https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/views/base.py#L75-L99

This leaves plugins like datasette-graphql having to implement their own versions of this logic, which is bad: https://github.com/simonw/datasette-graphql/issues/65

First check view-database - if that says False then disallow access, if it says True then allow access. If it says None check view-instance.

This should become a supported API that plugins are encouraged to use.

Discussed with Alex this morning. We think the American spelling is fine here (it's shorter than catalogue) and that it's a slightly less lazy name than core_.

Follows: - https://github.com/simonw/datasette/issues/2157

I have a neater pattern for this now - instead of: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_create.py#L472-L475

I now prefer:

https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_create.py#L1163-L1171

For example this: https://github.com/simonw/datasette/blob/4535568f2ce907af646304d0ebce2500ebd55677/docs/authentication.rst?plain=1#L33-L35

The problem with that $ prefix is that it prevents users from copying and pasting the raw command.

https://docs.datasette.io/en/stable/authentication.html#using-the-root-actor

From Discord: https://discordapp.com/channels/823971286308356157/996877076982415491/1068227071156965486

Caused by: - #2112

https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%22pk1%22+%3D+%3Ap0+order+by+pk1%2C+pk2%2C+pk3+limit+101&p0=b

The :p0 form field is missing. Submitting the form results in this error:

In the old code:

https://github.com/simonw/datasette/blob/08181823990a71ffa5a1b57b37259198eaa43e06/datasette/views/database.py#L34-L35

This name class attribute was later used by some of the plugin hooks, passed as view_name: https://github.com/simonw/datasette/blob/18dd88ee4d78fe9d760e9da96028ae06d938a85c/datasette/hookspecs.py#L50-L54

Figure out how that should work once I've refactored those classes to view functions instead.

Refs: - #2109

While using sqlite-utils convert I realized it would be handy if you could pass --pdb to have it open the debugger at the first instance of a failed conversion.

Refs: - #2109

The idea here is that hitting /content without a ?sql= will show an empty result set AND default to including a bunch of extras about the list of tables in the database.

Then I won't have to think about /content and /content?sql= as separate pages any more.

Hi, I'm trying to have a display function in Dogsheep's config.yml that includes something like this:

```

{{ display.title }} (source)

{{ display.snippet|safe }}

```

Unfortunately, rendering fails with a message 'urls is undefined'.

The same happens if I'm trying to build a row URL manually, using filters like quote_plus (as my keys are URLs).

Any hints?

Thanks!

Also need a design for an option for the .transform() method to indicate that the new table should be created with a new name without dropping the old one.

I think keep_table="name_of_table" is good for this.

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646657324

For every command/API method there should be a link to the equivalent in the other form factor.

Maybe also link to the API and CLI reference pages too.

CSV files with empty cells in (which come through as the empty string) are common and a bit gross.

Having an option that means "and if it's an empty string store null instead) would be cool.

I brainstormed name options here https://chat.openai.com/share/c947b738-ee7d-419c-af90-bc84e90987da

I found myself wanting to insert rows from a 849MB CSV file without processing the whole thing: https://huggingface.co/datasets/jerpint-org/HackAPrompt-Playground-Submissions/tree/main

sqlite-utils prints \r\n but it should probably print \n (unless the platform is detected as Windows?)

It has tripped me up a few times when piping the output of sqlite-utils to other programs:

$ sqlite-utils --no-headers --csv ~/lb/fs/d.db 'select path from media limit 1' | cat -A /mnt/d7/file^M$ $ sqlite-utils --no-headers --csv ~/lb/fs/d.db 'select path from media limit 1' | tr -d '\r' | cat -A /mnt/d7/file$

https://click.palletsprojects.com/en/8.1.x/shell-completion/

This works for zsh:

eval "$(_SQLITE_UTILS_COMPLETE=zsh_source sqlite-utils)"

This will probably work for bash:

eval "$(_SQLITE_UTILS_COMPLETE=bash_source sqlite-utils)"

Need to add this to the installation docs here: https://sqlite-utils.datasette.io/en/stable/installation.html - along with the pattern for adding that to .zshrc or whatever.

Noticed this while testing: - #547

If you pass a non-existent column to -c/--column you don't get an error message.

Noticed this:

``` % sqlite-utils analyze-tables content.db repos -c delete_branch_on_merge --common-limit 20 --no-least repos.delete_branch_on_merge: (1/1)

Total rows: 158 Null rows: 158 Blank rows: 0

Distinct values: 0

Most common: 158: None ```

The 158: None there is duplicate information considering we already know there are 158/158 null rows.

Reproducible code below:

```bash

echo 'create table bar (baz text, created_at timestamp default CURRENT_TIMESTAMP)' | sqlite3 foo.db sqlite3 foo.db SQLite version 3.36.0 2021-06-18 18:36:39 Enter ".help" for usage hints. sqlite> .schema bar CREATE TABLE bar (baz text, created_at timestamp default CURRENT_TIMESTAMP); sqlite> .exit sqlite-utils transform foo.db bar --column-order baz sqlite3 foo.db SQLite version 3.36.0 2021-06-18 18:36:39 Enter ".help" for usage hints. sqlite> .schema bar CREATE TABLE IF NOT EXISTS "bar" ( [baz] TEXT, [created_at] FLOAT DEFAULT 'CURRENT_TIMESTAMP' ); sqlite> .exit sqlite-utils transform foo.db bar --column-order baz sqlite3 foo.db SQLite version 3.36.0 2021-06-18 18:36:39 Enter ".help" for usage hints. sqlite> .schema bar CREATE TABLE IF NOT EXISTS "bar" ( [baz] TEXT, [created_at] FLOAT DEFAULT '''CURRENT_TIMESTAMP''' ); ```

Issuing either

py -Wdefault -m sqlite_utils insert dogs.db dogs dogs0.csv --csv [#############-----------------------] 36% [####################################] 100%C:\Users\Doug\AppData\Local\Programs\Python\Python311\Lib\site-packages\sqlite_utils\cli.py:1187: ResourceWarning: unclosed file <_io.TextIOWrapper name='dogs0.csv' encoding='utf-8-sig'> insert_upsert_implementation( ResourceWarning: Enable tracemalloc to get the object allocation traceback or set pythonwarnings=default sqlite-utils insert dogs.db dogs dogs0.csv --csv [#############-----------------------] 36% [####################################] 100%C:\Users\Doug\AppData\Local\Programs\Python\Python311\Lib\site-packages\sqlite_utils\cli.py:1187: ResourceWarning: unclosed file <_io.TextIOWrapper name='dogs0.csv' encoding='utf-8-sig'> insert_upsert_implementation( ResourceWarning: Enable tracemalloc to get the object allocation traceback

exhibits a ResourceWarning indicating that the CSV file being loaded is not closed.

sqlite-utils --version sqlite-utils, version 3.30 py --version Python 3.11.2 Windows Version 10.0.19045 Build 19045 SQLite version 3.41.0 2023-02-21 18:09:37

Neat project! You can probably increase performance using my python interface to Notes, macnotesapp, which uses Scripting Bridge and bulk queries for much better performance than AppleScript.

Another related project is PyXA which uses Scripting Bridge to access Notes (and many other apps) and can return all the notes at once as opposed to calling AppleScript for each note.

macnotesapp allows you to access multiple accounts and folders as well.

```python from macnotesapp import NotesApp

NotesApp() provides interface to Notes.app

notesapp = NotesApp()

Get list of notes (Note objects for each note)

notes = notesapp.notes() note = notes[0] print( note.id, note.account, note.folder, note.name, note.body, note.plaintext, note.password_protected, )

print(note.asdict())

```

On the database that I have sent via linkedIn, datasette works great, but the following URL gives a 500 error.

http://127.0.0.1:8001/literature/authors_papers?authorId=100550354

The cause of the error is not apparent.

Is this expected behaviour?

David

I got a confusing sqlite3.OperationalError: disk I/O error error in my tests, it turned out it was because this: python ds = Datasette(path) Should have been this: python ds = Datasette([path])

Originally posted by @simonw in https://github.com/simonw/datasette-faiss/issues/1#issuecomment-1378252673

The link in Play with a live demo in Getting started to https://fivethirtyeight.datasettes.com/fivethirtyeight is broken and the datasette is no longer working (maybe due to the end of the free tier).

CI has started failing for Python 3.6: https://github.com/simonw/sqlite-utils/actions/runs/3576322798

It's fixable by swiching away from ubuntu-latest according to:

• https://github.com/actions/setup-python/issues/355#issuecomment-1335042510

But https://endoflife.date/python says that 3.6 end of life was almost 6 years ago, and end of security support nearly 1 year ago.

So I'm OK dropping support entirely - Python 3.6 users will still be able to install version 3.30, just not any releases that come next.

Mainly CORS support and a few small bug fixes.

Changes: https://github.com/simonw/datasette/compare/1.0a0...99da46f7258225fc6fd8e94ddc20859ccccc4109

https://latest.datasette.io/-/api hides ephemeral down at the bottom, would be more interesting if it was at the top.

Related: - #1915

• views/table.py has got a bit too big - I think the write classes should be pulled out into a separate module.
• [x] There's duplicate logic for deciding if the table and database exist and checking permissions

Refs: - #1871

I noticed the API explorer doesn't show any links on https://latest-1-0-dev.datasette.io/-/api because the fixtures database is immutable.

It should still show read examples there.

It strikes me that users should NOT be able to use a token to create additional tokens.

The current design actually does allow that, since the dstok_ Bearer token can be used to authenticate calls to the /-/create-token page.

So I think I need a mechanism whereby that page can only allow access to users authenticated by cookie.

Not obvious how to do that though, since Datasette's authentication actor system is designed to abstract that detail away!

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1850#issuecomment-1291417100

I'm not really sure what caused this and it happened in the middle of my program (after running for 35775 seconds).

Extracting metadata 49.9% (chunk 9893 of 19831) ... File "/home/xk/.local/lib/python3.10/site-packages/xklb/fs_extract.py", line 90, in extract_chunk args.db["media"].insert_all(utils.list_dict_filter_bool(media), pk="path", alter=True, replace=True) File "/home/xk/.local/lib/python3.10/site-packages/sqlite_utils/db.py", line 3107, in insert_all self.insert_chunk( File "/home/xk/.local/lib/python3.10/site-packages/sqlite_utils/db.py", line 2872, in insert_chunk result = self.db.execute(query, params) File "/home/xk/.local/lib/python3.10/site-packages/sqlite_utils/db.py", line 483, in execute return self.conn.execute(sql, parameters) UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc3' in position 62: surrogates not allowed

This might be relevant: https://stackoverflow.com/questions/31898353/python-cant-encode-with-surrogateescape

I'm going to try re-running with

`py def execute( self, sql: str, parameters: Optional[Union[Iterable, dict]] = None ) -> sqlite3.Cursor: """ Execute SQL query and return asqlite3.Cursor``.

    :param sql: SQL query to execute
    :param parameters: Parameters to use in that query - an iterable for ``where id = ?``
      parameters, or a dictionary for ``where id = :id``
    """
    try:
        if self._tracer:
            self._tracer(sql, parameters)
        if parameters is not None:
            return self.conn.execute(sql, parameters)
        else:
            return self.conn.execute(sql)
    except UnicodeEncodeError:
        sql = sql.encode('utf-8', 'surrogatepass').decode('utf-8')
        if parameters is not None:
            parameters = parameters.encode('utf-8', 'surrogatepass').decode('utf-8')
        return self.execute(sql, parameters)

```

I think it's been long enough now that I can drop the code that syncs the main branch to master.

I originally added this for people who might be using datasette publish ... --branch master - which might only have been me anyway!

Needed for the Datasette 1.0 release, so template authors can trust that Datasette is unlikely to break their templates.

It would be nice to have a column_names. Similar to table_names.

Or if you could get one or all of the following syntax to work for both Database and Table that might be even better:

Style 1 - if 'table1' in db - if 'col1' in db['table1']

Style 2 - if 'table1' in db.tables - if 'col1' in db['table1'].columns

maybe the table ones actually work but I'm too lazy to check. I just know that I have to do:

[c.name for c in db['table1'].columns]

Edit: This is possible with columns_dict. I have actually used that before but I forgot about it. Feel free to close, but I do think accessing this data could be more consistent and intuitive.

I ended up needing to use db.conn.close() to fix this issue: - #503

I think .close() should be a method on Database itself.

The way errors are represented has changed: https://github.com/simonw/sqlite-utils/actions/runs/3323588047/jobs/5494127154 ``` ____ testquery_invalid_function ______

db_path = '/tmp/pytest-of-runner/pytest-0/test_query_invalid_function0/test.db'

def test_query_invalid_function(db_path):
    result = CliRunner().invoke(
        cli.cli, [db_path, "select bad()", "--functions", "def invalid_python"]
    )
    assert result.exit_code == 1

  assert (
        result.output.strip()
        == "Error: Error in functions definition: invalid syntax (<string>, line 1)"
    )

E AssertionError: assert 'Error: Error...ing>, line 1)' == 'Error: Error...ing>, line 1)' E - Error: Error in functions definition: invalid syntax (<string>, line 1) E ? ^^^^^^ ^^^^^^ E + Error: Error in functions definition: expected '(' (<string>, line 1) E ? ^^^^^^^ ^^^ ```

Is there any performance or size difference with column order in SQLITE ? similar to this https://www.cybertec-postgresql.com/en/column-order-in-postgresql-does-matter/

It might be interesting to have an option to create with an optimized column order. I'm assuming this would look something like INTEGER columns, REAL columns, BLOB columns, TEXT columns, NULL columns. NULL columns at the end because they are more likely to be TEXT and it is impossible to know if they will become INTEGER

(Of course, any schema evolution would reduce optimization but maybe column order could also be re-evaluated when schema changes)

edit:

this is easy to accomplish with the existing transform method:

int_columns = [k for k, v in table_columns.items() if v == int] db[table].transform(column_order=[*int_columns])

• The prepare_jinja2_environment(env, datasette) plugin hook now accepts an optional datasette argument. Hook implementations can also now return an async function which will be awaited automatically. (#1809)
• --load-extension option now supports entrypoints. Thanks, Alex Garcia. (#1789)
• New tutorial: Cleaning data with sqlite-utils and Datasette.
• Facet size can now be set per-table with the new facet_size table metadata option. (#1804)
• truncate_cells_html setting now also affects long URLs in columns. (#1805)
• Database(is_mutable=) now defaults to True. (#1808)
• Non-JavaScript textarea now increases height to fit the SQL query. (#1786)
• More detailed command descriptions on the CLI reference page. (#1787)
• Datasette no longer enforces upper bounds on its depenedencies. (#1800)
• Facets are now displayed with better line-breaks in long values. Thanks, Daniel Rech. (#1794)
• The settings.json file used in Configuration directory mode is now validated on startup. (#1816)

👋 thank you for an incredibly useful project!

I have noticed that my deployed instance on Fly does not include the next and next_url keys even for a truncated response :

This is publically accessible here: https://collectif-objets-datasette.fly.dev/collectif-objets.json?sql=select+*+from+mairies

However when I run the dataset server locally with the same data I get these next keys for the exact same query:

I am wondering if I've missed some config or something specific to deployments on Fly.io?

I am running datasette v0.62, without any specific config :

• locally poetry run datasette data/collectif-objets.sqlite
• for the deploy : poetry run datasette publish fly data/collectif-objets.sqlite

as visible in the Makefile. The very limited codebase is public but the sqlite db is not versioned yet because it is too large.

When inserting or upserting a jsonlines file, no progressbar is shown. Expected behavior is that, just like with .csv/.tsv files, also for a jsonlines file (--nl), unless --silent is provided, a progressbar is shown.

bash sql-utils upsert mydb.db posts posts.jl --nl --pk post_id (silence)

Currently file_progress is only called within the tsv/csv logic, however I think it can be safely wrapped around all the all the input formats that use decoded: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/cli.py#L963

Interesting challenge with default_value: I need to be able to tell if the default values passed to .create() differ from those in the database already.

Introspecting that is a bit tricky:

```pycon

import sqlite_utils db = sqlite_utils.Database(memory=True) db["blah"].create({"id": int, "name": str}, not_null=("name",), defaults={"name": "bob"}) <Table blah (id, name)> db["blah"].columns [Column(cid=0, name='id', type='INTEGER', notnull=0, default_value=None, is_pk=0), Column(cid=1, name='name', type='TEXT', notnull=1, default_value="'bob'", is_pk=0)] `` Note how a default value of the Python stringbobis represented in the results ofPRAGMA table_info()asdefault_value="'bob'"` - it's got single quotes added to it!

So comparing default values from introspecting the database needs me to first parse that syntax. This may require a new table introspection method.

Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/468#issuecomment-1229279539

I've run into an issue with extract and I don't believe this is the intended behaviour.

I'm working with a database with music listening information. Currently it has one large table listens that contains all information. I'm trying to normalize the database by extracting relevant columns to separate tables (artists, tracks, albums). Not every track has an album.

A simplified demonstration with just track_title and album_title columns: ```ipython In [1]: import sqlite_utils

In [2]: db = sqlite_utils.Database(memory=True)

In [3]: db["listens"].insert_all([ ...: {"id": 1, "track_title": "foo", "album_title": "bar"}, ...: {"id": 2, "track_title": "baz", "album_title": None} ...: ], pk="id") Out[3]: <Table listens (id, track_title, album_title)> ```

The track in the first row has an album, the second track doesn't. Now I extract album information into a separate column: ```ipython In [4]: db["listens"].extract(columns=["album_title"], table="albums", fk_column="album_id") Out[4]: <Table listens (id, track_title, album_id)>

In [5]: list(db["albums"].rows) Out[5]: [{'id': 1, 'album_title': 'bar'}, {'id': 2, 'album_title': None}]

In [6]: list(db["listens"].rows) Out[6]: [{'id': 1, 'track_title': 'foo', 'album_id': 1}, {'id': 2, 'track_title': 'baz', 'album_id': None}] ```

This behaves as expected -- the album table contains entries for both the existing album and the NULL album. The listens table has a foreign key only for the first row (since the album in the second row was empty).

Now I want to extract the track information as well. Album information belongs to the track so I want to extract both columns to a new table. ```ipython In [7]: db["listens"].extract(columns=["track_title", "album_id"], table="tracks", fk_column="track_id") Out[7]: <Table listens (id, track_id)>

In [8]: list(db["tracks"].rows) Out[8]: [{'id': 1, 'track_title': 'foo', 'album_id': 1}, {'id': 2, 'track_title': 'baz', 'album_id': None}]

In [9]: list(db["listens"].rows) Out[9]: [{'id': 1, 'track_id': 1}, {'id': 2, 'track_id': None}] ```

Extracting to the tracks table worked fine (both tracks are present with correct columns). However, the listens table only has a foreign key to the newly created tracks for the first row, the foreign key in the second row is NULL.

Changing the order of extracts doesn't help.

I poked around in the source a bit and I believe this line (essentially comparing NULL = NULL) is the problem, but I don't know enough about SQL to create a reliable fix myself.

For consistency with search: https://sqlite-utils.datasette.io/en/stable/cli-reference.html#search

-o, --order TEXT Order by ('column' or 'column desc')

I wanted to run sqlite-utils rows db.db mytable --order 'rowid desc' to see the most recently imported rows.

In this example:

```python @db.register_function def reverse_string(s): return "".join(reversed(list(s)))

print(db.execute('select reverse_string("hello")').fetchone()[0])

``` There's currently no way to over-ride the automatically selected name for the SQL function.

As of 0.62 https://lite.datasette.io is a supported piece of the overall Datasette ecosystem.

https://til.simonwillison.net/pytest/registering-plugins-in-tests

Would be useful to include this pattern on https://datasette.readthedocs.io/en/stable/testing_plugins.html

I'm using the [insert_upsert_implementation](https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/cli.py) function directly in my Python code to import a csv file with all the bells and whistles sqlite-utils provides, but I'm getting a resource warning that a io.TextWrapper object is not closed.

The warning goes away when wrapping the code from this line in a try/finally block like:

try: ... ... finally: decoded.close() (might be that sniff_buffer must also be closed if non null, but I might be wrong)

I suspect Python closes the reference automatically when the sqlite-utils cli run is done, but since my code doesn't exit, I'm getting the warning.

Alternatively, it'd be cool if the 'import csv/tsv' functionality could be added directly to the Database class.

CLI equivalent of: - #449

Missing values in my dataset are denoted by a single asterisk. I am trying to parse string dates into dates. This works fine for columns without missing values, but, when the column contains "*", I get the following:

``` $ sqlite-utils convert ${dbfile} details dob 'r.parsedate(value)' [------------------------------------] 0%Traceback (most recent call last): File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/db.py", line 2508, in convert_value return fn(v) File "<string>", line 2, in fn File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/recipes.py", line 8, in parsedate parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat() File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 1368, in parse return DEFAULTPARSER.parse(timestr, **kwargs) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 643, in parse raise ParserError("Unknown string format: %s", timestr) dateutil.parser._parser.ParserError: Unknown string format: *

Traceback (most recent call last): File "/usr/local/bin/sqlite-utils", line 33, in <module> sys.exit(load_entry_point('sqlite-utils==3.25.1', 'console_scripts', 'sqlite-utils')()) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1128, in call return self.main(args, kwargs) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 754, in invoke return __callback(args, **kwargs) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/cli.py", line 2698, in convert db[table].convert( File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/db.py", line 2524, in convert self.db.execute(sql, where_args or []) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/db.py", line 458, in execute return self.conn.execute(sql, parameters) sqlite3.OperationalError: user-defined function raised exception ```

(As you will be able to tell from the terminology I use, I am not a frontend guy, but I hope you will understand.)

When a table is wide and needs horizontal scrolling to see the columns towards the end, the user needs to scroll horizontally. However, since the container for the HTML table (div with class table-wrapper) isn't limited by the window size, I first need to vertically scroll near to the bottom of the page in order to scroll horizontally. Then I can scroll back up again. This isn't very user friendly. Instead, I think it would make sense to constrain the table's size (when necessary), so that the vertical and horizontal scrollbars either always are visible or at least not far out of reach.

I understand that I could provide my own template and / or CSS, but I think it would probably make sense to adjust the default in this regard.

On https://docs.datasette.io/en/stable/getting_started.html

Noticed that cog -r docs/*.rst isn't documented in https://docs.datasette.io/en/latest/contributing.html#editing-and-building-the-documentation

I'm going to add a Datasette setting to disable threading entirely, designed for usage in this particular case.

I thought about adding a new setting, then I noticed this:

datasette mydatabase.db --setting num_sql_threads 10

I'm going to let users set that to 0 to disable threaded execution of SQL queries.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1733#issuecomment-1115278325

GitHub can support this in blame views now too:

https://docs.github.com/en/repositories/working-with-files/using-files/viewing-a-file#ignore-commits-in-the-blame-view

Seen here:

• https://github.com/simonw/geojson-to-sqlite/issues/30

Attempting to create a table with no columns produced this confusing error:

File "/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/geojson_to_sqlite/utils.py", line 69, in import_features db[table].create(column_types, pk=pk) File "/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/sqlite_utils/db.py", line 863, in create self.db.create_table( File "/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/sqlite_utils/db.py", line 517, in create_table self.execute(sql) File "/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/sqlite_utils/db.py", line 236, in execute return self.conn.execute(sql) sqlite3.OperationalError: near ")": syntax error

I'll also encourage testing against both Datasette 0.x and Datasette 1.0 using a GitHub Actions matrix.

Recommend the https://github.com/simonw/datasette-block-robots plugin if you are going to run a large database in Cloud Run (one with a lot of rows).

I just created an alternative .fake() method because I wanted to fake the url_vars captured in the route as well: ```python from datasette.utils.asgi import Request class Request(Request):

@classmethod
def fake(cls, path_with_query_string, method="GET", scheme="http", url_vars=None):
    """Useful for constructing Request objects for tests"""
    path, _, query_string = path_with_query_string.partition("?")
    scope = {
        "http_version": "1.1",
        "method": method,
        "path": path,
        "raw_path": path_with_query_string.encode("latin-1"),
        "query_string": query_string.encode("latin-1"),
        "scheme": scheme,
        "type": "http",
    }
    if url_vars:
        scope["url_route"] = {
            "kwargs": url_vars
        }
    return cls(scope, None)

```

https://docs.datasette.io/en/0.61.1/internals.html#add-message-request-message-message-type-datasette-info says:

.add_message(request, message, message_type=datasette.INFO)

But in the code it's:

https://github.com/simonw/datasette/blob/6b99e4a66ba0ed8fca8ee41ceb7206928b60d5d1/datasette/app.py#L582

if hashed_urls are on, then a -000 suffix is added to the _memory database, and the cache settings are set just as if it was a normal hashed database.

in particular, this header is set:

cache-control: max-age=31536000

this is not appropriate because the _memory-000 database isn't really hashed based on the contents of the databases (see #1561).

Either the cache-control header should be changed, or the _memory db should have a hash suffix that does depend on the contents of the databases.

500 error on https://datasette-hashed-urls-preview.vercel.app/fixtures-09f8f95?sql=select+*+from+facetable

Here's the trace: File "/Users/simon/.local/share/virtualenvs/datasette-hashed-urls-ssI2fO50/lib/python3.10/site-packages/datasette/views/database.py", line 54, in data return await QueryView(self.ds).data( File "/Users/simon/.local/share/virtualenvs/datasette-hashed-urls-ssI2fO50/lib/python3.10/site-packages/datasette/views/database.py", line 232, in data self.ds.get_database(database), sql File "/Users/simon/.local/share/virtualenvs/datasette-hashed-urls-ssI2fO50/lib/python3.10/site-packages/datasette/app.py", line 401, in get_database return self.databases[name] KeyError: 'fixtures-aa7318b' It looks like this is a Datasette bug, which is frustrating because I just shipped Datasette 0.61 five minutes ago!

Originally posted by @simonw in https://github.com/simonw/datasette-hashed-urls/issues/13#issuecomment-1076693667

Spotted this while working on: - #1677

https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/utils/init.py#L1005-L1021

Follow-on from: - #1675

Refs: - #1660

HI, when dealing with large json file (~170GB) i got the following error Traceback (most recent call last): File "/usr/local/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/usr/lib/python3/dist-packages/click/core.py", line 1126, in __call__ return self.main(*args, **kwargs) File "/usr/lib/python3/dist-packages/click/core.py", line 1051, in main rv = self.invoke(ctx) File "/usr/lib/python3/dist-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3/dist-packages/click/core.py", line 1393, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/lib/python3/dist-packages/click/core.py", line 752, in invoke return __callback(*args, **kwargs) File "/usr/local/lib/python3.9/dist-packages/sqlite_utils/cli.py", line 1300, in memory rows, format_used = rows_from_file(csv_fp, format=format, encoding=encoding) File "/usr/local/lib/python3.9/dist-packages/sqlite_utils/utils.py", line 185, in rows_from_file return rows_from_file(buffered, format=Format.JSON) File "/usr/local/lib/python3.9/dist-packages/sqlite_utils/utils.py", line 156, in rows_from_file decoded = json.load(fp) File "/usr/lib/python3.9/json/__init__.py", line 293, in load return loads(fp.read(), MemoryError

I've made a real mess of this. I'm going to revert Datasettemain back to the last commit that passed the tests and try this again in a branch.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1657#issuecomment-1068125636

It seems currently all datasette publish requires a real backend server which is able to query the database and send results back to the frontend. There are a few projects to on-demand download a portion of data from the database from a sqlite lite database url, and present it directly to the user. These methods leverages web assembly under the hood. I think datasette is a perfect use case for this technology. Below are a few examples of querying sqlite database from frontend directly.

• Using sqlite3 as a notekeeping document graph with automatic reference indexing
• Hosting SQLite databases on Github Pages - (or any static file hoster) - phiresky's blog
• Static torrent website with peer-to-peer queries over BitTorrent on 2M records

``` ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

datasette 0.60.2 requires asgiref<3.5.0,>=3.2.10, but you'll have asgiref 3.5.0 which is incompatible. That's after I forced an upgrade of `uvicorn` due to this warning: ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

uvicorn 0.13.1 requires click==7.*, but you'll have click 8.0.4 which is incompatible. ```

Inspired by: - #397

To match the option on create-index: https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-index

--if-not-exists Ignore if index already exists

Hi @simonw , If I run

sqlite3 .load /usr/local/lib/mod_spatialite.so select spatialite_version();

I have 5.0.0.

If I run

sqlite-utils :memory: "select spatialite_version()" --load-extension=spatialite

I have

Traceback (most recent call last): File "/home/aborruso/.local/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/home/aborruso/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/home/aborruso/.local/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/home/aborruso/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/aborruso/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/aborruso/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/aborruso/.local/lib/python3.8/site-packages/sqlite_utils/cli.py", line 936, in query _load_extensions(db, load_extension) File "/home/aborruso/.local/lib/python3.8/site-packages/sqlite_utils/cli.py", line 1326, in _load_extensions db.conn.load_extension(ext) TypeError: argument 1 must be str, not None

How to load properly spatialite extension in sqlite-utils?

Thank you very muc

Yeah, apt-get isn't a thing on macOS so 4a2a3e2fd0d5534f446b3f1fee34cb165e4d86d2 (to test #79 against real SpatiaLite) broke.

Currently: https://sqlite-utils.datasette.io/en/stable/python-api.html#insert-replacing-data

If you want to insert a record or replace an existing record with the same primary key, using the replace=True argument to .insert() or .insert_all():

Should describe the exception you get first, then how to use replace to avoid it.

Currently throws an ugly traceback: % echo '[ {"id": 1, "name": "Lila"}, {"id": 1, "name": "Lila"} ]' | sqlite-utils upsert data.db chickens - Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils", line 33, in <module> sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')()) File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 1137, in __call__ return self.main(*args, **kwargs) File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 1062, in main rv = self.invoke(ctx) File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 1668, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py", line 763, in invoke return __callback(*args, **kwargs) File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py", line 1104, in upsert insert_upsert_implementation( File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py", line 906, in insert_upsert_implementation db[table].insert_all( File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 2615, in insert_all raise PrimaryKeyRequired("upsert() requires a pk") sqlite_utils.db.PrimaryKeyRequired: upsert() requires a pk

You have to read a lot of stuff in a lot of different places to get started with the Python library. Add a getting started introduction to https://sqlite-utils.datasette.io/en/stable/python-api.html

I was trying to initalize SpatiaLite in a write connection: ```pycon

from datasette.app import Datasette ds = Datasette(memory=True, files=[], sqlite_extensions=["spatialite"]) db = ds.add_memory_database('geo') await db.execute_write("select InitSpatialMetadata(1)") UUID('3f143baa-4e3d-5842-a36f-4fa2f683b72f') no such function: InitSpatialMetadata ``` It looks like the code that loads additional modules only works on read-only connections, not on write connections:

https://github.com/simonw/datasette/blob/92a5280d2e75c39424a75ad6226fc74400ae984f/datasette/database.py#L146-L153

Compared to:

https://github.com/simonw/datasette/blob/92a5280d2e75c39424a75ad6226fc74400ae984f/datasette/database.py#L124-L132

https://github.com/simonw/datasette/blob/242bc89fdf2e775e340d69a4e851b3a9accb31c6/datasette/cli.py#L533-L548

This could use the find_spatialite() function and, if it finds something, suggest the user use --load-extension=spatialite

https://github.com/simonw/datasette/blob/242bc89fdf2e775e340d69a4e851b3a9accb31c6/datasette/utils/init.py#L1015-L1019

I added this to sqlite-utils in https://github.com/simonw/sqlite-utils/issues/136 and I really like it: pass a special value of spatialite and Datasette should attempt to load it from known likely installation locations.

The example output for datasette plugins --allon this page has got out of date: https://docs.datasette.io/en/stable/plugins.html#seeing-what-plugins-are-installed

I introduced cog in #1594 - I can use this to replace the older update-docs-help.py mechanism:

https://github.com/simonw/datasette/blob/76d66d5b2bf10249c0beaac0999b93ac8d757f48/tests/test_docs.py#L36-L53

Made obvious by the new CLI reference page added in #1594. https://docs.datasette.io/en/latest/cli-reference.html#datasette-inspect-help Commands: serve* Serve up specified SQLite database files with a web UI inspect install Install Python packages - e.g. ``` Usage: datasette inspect [OPTIONS] [FILES]...

Options: --inspect-file TEXT --load-extension TEXT Path to a SQLite extension to load --help Show this message and exit. ```

Found I wanted this while I was debugging #625 just to clean up the debug traces, but it makes sense as a partner to ?_nofacet=1 and ?_nocount=1 from #1350 and #1353.

You've caused me to rethink this feature - I no longer think there's value in only showing these numbers if ?_facet_size=max as opposed to all of the time.

Originally posted by @simonw in https://github.com/simonw/datasette/issues/1423#issuecomment-995023410

CLI equivalent of table.rows_where() - should accept parameters too. Work on this at the same time as #381.

For these commits: https://github.com/simonw/sqlite-utils/compare/3.20...129141572f249ea290e2a075437e2ebaad215859

For https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-data-from-files

Given a file called dogs.jpg stem would be dogs and ext would be jpg.

Need to decide what happens for dogs.and.cats.jpg.gz.

I've installed the alpha version but get an error when starting up Datasette:

Traceback (most recent call last): File "/Users/tim/.pyenv/versions/stock-exchange/bin/datasette", line 5, in <module> from datasette.cli import cli File "/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/cli.py", line 15, in <module> from .app import Datasette, DEFAULT_SETTINGS, SETTINGS, SQLITE_LIMIT_ATTACHED, pm File "/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/app.py", line 31, in <module> from .views.database import DatabaseDownload, DatabaseView File "/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/views/database.py", line 25, in <module> from datasette.plugins import pm File "/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/plugins.py", line 29, in <module> mod = importlib.import_module(plugin) File "/Users/tim/.pyenv/versions/3.8.5/lib/python3.8/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/Users/tim/.pyenv/versions/3.8.5/envs/stock-exchange/lib/python3.8/site-packages/datasette/filters.py", line 9, in <module> @hookimpl(specname="filters_from_request") TypeError: __call__() got an unexpected keyword argument 'specname'

Originally posted by @wragge in https://github.com/simonw/datasette/issues/1547#issuecomment-997511968

Could someone tell me that part of the code is responsible for validating the sql that guarantees that only a table can be read

Deprecated pytest-runner is not necessary for running the testsuite.

It's not possible to use a module that requires a nested import, such as xml.etree.ElementTree, at the moment. I found and fixed this bug in git-history, I should replicate that fix (and accompanying documentation) here: https://github.com/simonw/git-history/issues/39

currently --flatten option works only for insert command, it would be cool if it worked for memory as well to query nested json

e.g. on https://latest.datasette.io/fixtures/sortable?_nocol=sortable

This causes weird behaviour when you e.g. facet by a hidden column, since selecting facets and then re-submitting the form will clear the selected filter.

Datasette treats columns that start with an underscore as querystring parameters it should ignore!

Discovered in https://github.com/simonw/git-history/issues/14#issuecomment-968192464

https://docs.datasette.io/en/stable/sql_queries.html#named-parameters

It's not obvious that the values from parameters are always SQLite strings, which means that you can't do e.g. integer comparisons on them without casting them first. The documentation here should include an example of this.

👋 trying to upgrade the formula to use the latest release, but runs into some regression test issue with --get command.

My QQ is does this datasette --get '/:memory:.json?sql=select+3*5' supposed to return 15? Thanks!

relates to https://github.com/Homebrew/homebrew-core/pull/87369

Hi Simon,

It appears the fix for #57 doesn't fix things for me:

$ twitter-to-sqlite --version twitter-to-sqlite, version 0.21.4 $ python --version Python 3.9.6

$ twitter-to-sqlite home-timeline -a twitter-auth.json twitter/timeline.db --since Importing tweets Error: Use either --since or --since_id, not both

Is there any way I can help debug this?