github
id | node_id | number | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at | author_association | pull_request | body | repo | type | active_lock_reason | performed_via_github_app | reactions | draft | state_reason |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
771608692 | MDU6SXNzdWU3NzE2MDg2OTI= | 14 | UNIQUE constraint failed: workouts.id | 1234956 | open | 0 | 5 | 2020-12-20T15:11:20Z | 2023-07-10T14:46:52Z | NONE | I'm getting an error on my initial attempt to import data: ```console $ healthkit-to-sqlite 20201119\ healthkit\ export.zip healthkit.db Importing from HealthKit [###################################-] 98% 00:00:01 Traceback (most recent call last): File "venv/bin/healthkit-to-sqlite", line 8, in <module> sys.exit(cli()) File "venv/lib/python3.9/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "venv/lib/python3.9/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "venv/lib/python3.9/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "venv/lib/python3.9/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "venv/lib/python3.9/site-packages/healthkit_to_sqlite/cli.py", line 57, in cli convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf) File "venv/lib/python3.9/site-packages/healthkit_to_sqlite/utils.py", line 34, in convert_xml_to_sqlite workout_to_db(el, db, zipfile) File "venv/lib/python3.9/site-packages/healthkit_to_sqlite/utils.py", line 57, in workout_to_db pk = db["workouts"].insert(record, alter=True, hash_id="id").last_pk File "venv/lib/python3.9/site-packages/sqlite_utils/db.py", line 1660, in insert return self.insert_all( File "venv/lib/python3.9/site-packages/sqlite_utils/db.py", line 1778, in insert_all self.insert_chunk( File "venv/lib/python3.9/site-packages/sqlite_utils/db.py", line 1588, in insert_chunk result = self.db.execute(query, params) File "venv/lib/python3.9/site-packages/sqlite_utils/db.py", line 213, in execute return self.conn.execute(sql, parameters) sqlite3.IntegrityError: UNIQUE constraint failed: workouts.id ``` | 197882382 | issue | { "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1279144769 | I_kwDOCGYnMM5MPjNB | 448 | Reading rows from a file => AttributeError: '_io.StringIO' object has no attribute 'readinto' | 236907 | closed | 0 | 5 | 2022-06-21T21:48:27Z | 2023-05-08T22:01:00Z | 2023-05-08T22:01:00Z | NONE | Attempting to run the example given here (without extra bracket ;-): https://sqlite-utils.datasette.io/en/stable/python-api.html#reading-rows-from-a-file ``` from sqlite_utils.utils import rows_from_file import io rows, format = rows_from_file(io.StringIO("id,name\n1,Cleo")) print(list(rows), format) # Outputs [{'id': '1', 'name': 'Cleo'}] Format.CSV ``` Gives error ``` >"c:\Program Files\Python37\python.exe" test2.py Traceback (most recent call last): File "test2.py", line 4, in <module> rows, format = rows_from_file(io.StringIO("id,name\n1,Cleo")) File "C:\Users\swood\Downloads\sqlite-utils-main-20220621\sqlite-utils-main\sqlite_utils\utils.py", line 300, in rows_from_file first_bytes = buffered.peek(2048).strip() AttributeError: '_io.StringIO' object has no attribute 'readinto' ``` I am running Python on Windows. ``` >"c:\Program Files\Python37\python.exe" Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/448/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1465194249 | I_kwDOCGYnMM5XVRcJ | 514 | upsert of new row with check constraints fails | 193185 | closed | 0 | 5 | 2022-11-26T16:12:23Z | 2023-05-08T21:50:52Z | 2023-05-08T21:50:51Z | NONE | (I originally opened this in https://github.com/simonw/datasette-insert/issues/20, but I see that that library depends on sqlite-utils) In the case of a new row, upsert first adds the row, specifying only its pkeys: https://github.com/simonw/sqlite-utils/blob/965ca0d5f5bffe06cc02cd7741344d1ddddf9d56/sqlite_utils/db.py#L2783-L2787 This means that a table with NON NULL (or other constraint) columns that aren't part of the pkey can't have new rows upserted. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/514/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
828858421 | MDU6SXNzdWU4Mjg4NTg0MjE= | 1258 | Allow canned query params to specify default values | 1385831 | open | 0 | 5 | 2021-03-11T07:19:02Z | 2023-02-20T23:39:58Z | NONE | If I call a canned query that includes named parameters, without passing any parameters, datasette runs the query anyway, resulting in an HTTP status code 400, and a visible error in the browser, with only a link back to home. This means that one of the default links on https://site/database/ will lead to a broken page with no apparent way out. ![image](https://user-images.githubusercontent.com/1385831/110748683-13e72300-820e-11eb-855c-32e03dfef5bf.png) Is there any way to skip performing the query when parameters aren't supplied, but otherwise render the usual canned query page? Alternatively, can I supply default values for my parameters, either when defining my canned queries or when linking to the canned query page from the default database template. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1258/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1306984363 | I_kwDOBm6k_c5N5v-r | 1771 | minor a11y: <select> has no visual indicator when tabbed to | 1473102 | closed | 0 | 5 | 2022-07-17T04:30:14Z | 2022-12-18T06:34:20Z | 2022-12-18T06:28:12Z | NONE | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1771/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||||
1128466114 | I_kwDOCGYnMM5DQwbC | 406 | Creating tables with custom datatypes | 82988 | open | 0 | 5 | 2022-02-09T12:16:31Z | 2022-09-15T18:13:50Z | NONE | Via https://stackoverflow.com/a/18622264/454773 I note the ability to register custom handlers for novel datatypes that can map into and out of things like sqlite `BLOB`s. From a quick look and a quick play, I didn't spot a way to do this in `sqlite_utils`? For example: ```python # Via https://stackoverflow.com/a/18622264/454773 import sqlite3 import numpy as np import io def adapt_array(arr): """ http://stackoverflow.com/a/31312102/190597 (SoulNibbler) """ out = io.BytesIO() np.save(out, arr) out.seek(0) return sqlite3.Binary(out.read()) def convert_array(text): out = io.BytesIO(text) out.seek(0) return np.load(out) # Converts np.array to TEXT when inserting sqlite3.register_adapter(np.ndarray, adapt_array) # Converts TEXT to np.array when selecting sqlite3.register_converter("array", convert_array) ``` ```python from sqlite_utils import Database db = Database('test.db') # Reset the database connection to used the parsed datatype # sqlite_utils doesn't seem to support eg: # Database('test.db', detect_types=sqlite3.PARSE_DECLTYPES) db.conn = sqlite3.connect(db_name, detect_types=sqlite3.PARSE_DECLTYPES) # Create a table the old fashioned way # but using the new custom data type vector_table_create = """ CREATE TABLE dummy (title TEXT, vector array ); """ cur = db.conn.cursor() cur.execute(vector_table_create) # sqlite_utils doesn't appear to support custom types (yet?!) # The following errors on the "array" datatype """ db["dummy"].create({ "title": str, "vector": "array", }) """ ``` We can then add / retrieve records from the database where the datatype of the `vector` field is a custom registered `array` type (which is to say, a `numpy` array): ```python import numpy as np db["dummy"].insert({'title':"test1", 'vector':np.array([1,2,3])}) for row in db.query("SELECT * FROM dummy"): print(row['title'], row['vector'], type(row['vector'])) """ test1 [1 2 3] <class '… | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/406/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1236693079 | I_kwDOCGYnMM5JtnBX | 432 | Support `rows_where()`, `delete_where()` etc for attached alias databases | 11597658 | open | 0 | 5 | 2022-05-16T06:38:58Z | 2022-06-14T22:16:48Z | NONE | Hi, I noticed `rows_where()` doesn't return any rows from tables which are from attached databases. The `exists()` function returns false. As far as I can see this is because the `table_names()` function only looks for table names in the current database and not in attached (or temp) databases. Besides, `rows_where()`, also `insert_all()` and `delete_where()` didn't do what I was expecting because of this. For the moment I've patched `table_names()` for myself, see below but I'm not sure what the total impact is on the other functions like lookup truncate etc which all use `exists()`. Also `view_names()` doesn't look for views in attached or temp databases. ```python def table_names(self, fts4: bool = False, fts5: bool = False) -> List[str]: "A list of string table names in this database." where = ["type = 'table'"] if fts4: where.append("sql like '%USING FTS4%'") if fts5: where.append("sql like '%USING FTS5%'") dbs = [x[1] for x in self.execute('pragma database_list').fetchall()] lst=[] for db in dbs: sql = "select name from {} where {}".format(db+".sqlite_master"," AND ".join(where)) lst.extend(r[0] for r in self.execute(sql).fetchall()) return lst ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/432/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
810397025 | MDU6SXNzdWU4MTAzOTcwMjU= | 1228 | 500 error caused by faceting if a column called `n` exists | 7107523 | closed | 0 | 5 | 2021-02-17T17:41:20Z | 2022-03-19T06:44:40Z | 2022-03-19T01:38:04Z | NONE | I recently discovered `datasette` thanks to your great talk at FOSDEM and would like to use it for some projects. However, when trying to use it on databases created from some csv ot tsv files, I am sometimes getting this issue when going to http://127.0.0.1:8001/databasetest/databasetest and I don't exactly understand what it refers to. So far, I couldn't find anything relevant when reviewing the raw text files that could explain this issue, nor could I find something obvious between the files that generate this issue and those that don't. Does the error ring a bell and, if so, could you please point me to the right direction? ``` $ datasette databasetest.db INFO: Started server process [1408482] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit) INFO: 127.0.0.1:56394 - "GET / HTTP/1.1" 200 OK INFO: 127.0.0.1:56394 - "GET /-/static/app.css?4e362c HTTP/1.1" 200 OK INFO: 127.0.0.1:56396 - "GET /-/static-plugins/datasette_vega/main.2acbb312.css HTTP/1.1" 200 OK INFO: 127.0.0.1:56398 - "GET /-/static-plugins/datasette_vega/main.08f5d3d8.js HTTP/1.1" 200 OK Traceback (most recent call last): File "/home/kabouik/.local/lib/python3.7/site-packages/datasette/app.py", line 1099, in route_path response = await view(request, send) File "/home/kabouik/.local/lib/python3.7/site-packages/datasette/views/base.py", line 147, in view request, **request.scope["url_route"]["kwargs"] File "/home/kabouik/.local/lib/python3.7/site-packages/datasette/views/base.py", line 121, in dispatch_request return await handler(request, *args, **kwargs) File "/home/kabouik/.local/lib/python3.7/site-packages/datasette/views/base.py", line 260, in get request, database, hash, correct_hash_provided, **kwargs File "/home/kabouik/.local/lib/python3.7/site-packages/datasette/views/base.py", line 434, in view_get request, database, hash, **kwargs File "/home/kabouik/.loc… | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1228/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
924748955 | MDU6SXNzdWU5MjQ3NDg5NTU= | 1380 | Serve all db files in a folder | 193463 | open | 0 | 5 | 2021-06-18T10:03:32Z | 2021-11-13T08:09:11Z | NONE | I tried to get the `serve` command to serve all the .db files in the `/mnt` folder but is seems that the server does not refresh the list of files. In more detail: * Starting datasette as a docker container with: ``` docker run -p 8001:8001 -v `pwd`:/mnt \ datasetteproject/datasette \ datasette -p 8001 -h 0.0.0.0 /mnt ``` * Datasette correctly serves all the *.db files found in the /mnt folder * When the server is running, if I copy a new file in the $PWD folder, Datasette does not seem to see the new files, forcing me to restart Docker. Is there an option/setting that I overlooked, or is this something missing? BTW, the `--reload` setting, although at first glance is what you think you need, does not seem to do anything in regards of seeing all *.db files. Thanks! | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/1380/reactions", "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
920636216 | MDU6SXNzdWU5MjA2MzYyMTY= | 64 | feature: support "events" | 231498 | open | 0 | 5 | 2021-06-14T17:42:49Z | 2021-06-15T00:48:37Z | NONE | the GitHub API provides the ability to fetch all events for a given user, organization, or repository: https://docs.github.com/en/rest/reference/activity#list-events-for-the-authenticated-user this would allow users to export all of the issue comments, new issues, etc. that they created. something which is currently missing from the GitHub takeout exports. | 207052882 | issue | { "url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
778380836 | MDU6SXNzdWU3NzgzODA4MzY= | 4 | Feature Request: Gmail | 203343 | open | 0 | 5 | 2021-01-04T21:31:09Z | 2021-03-04T20:54:44Z | NONE | From takeout, I only exported my Gmail account. Ideally I could parse this into sqlite via this tool. | 206649770 | issue | { "url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
711649325 | MDU6SXNzdWU3MTE2NDkzMjU= | 182 | Better handling of encodings other than utf-8 for "sqlite-utils insert" | 765871 | closed | 0 | 5 | 2020-09-30T05:43:48Z | 2020-10-16T17:20:41Z | 2020-10-16T17:18:52Z | NONE | Makefile: ``` data.db: curl -O http://maps.natalian.org/data.txt go run csv-write.go > data.csv sqlite-utils insert data.db travels data.csv --csv clean: rm data* ``` [csv-write.go](https://gist.github.com/kaihendry/dff2442de20d73f900026d13bf7a11d9) Error message is: ``` sqlite-utils insert data.db travels data.csv --csv Traceback (most recent call last): File "/home/hendry/.local/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/home/hendry/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/home/hendry/.local/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/home/hendry/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/hendry/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/hendry/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/hendry/.local/lib/python3.8/site-packages/sqlite_utils/cli.py", line 614, in insert insert_upsert_implementation( File "/home/hendry/.local/lib/python3.8/site-packages/sqlite_utils/cli.py", line 553, in insert_upsert_implementation headers = next(reader) File "/usr/lib/python3.8/codecs.py", line 322, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe3 in position 1234: invalid continuation byte make: *** [Makefile:4: data.db] Error 1 [hendry@t14s datasette-map]$ sqlite-utils --version sqlite-utils, version 2.19 ``` Little bit surprised if Go is spewing out bad Unicode, but I'm not sure how to grok `position 1234`.. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/182/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
556814876 | MDU6SXNzdWU1NTY4MTQ4NzY= | 662 | Escape_fts5_query-hookimplementation does not work with queries to standard tables | 2181410 | closed | 0 | 5 | 2020-01-29T11:56:03Z | 2020-01-30T00:30:20Z | 2020-01-30T00:30:19Z | NONE | Hi Simon Thank you for adding the escape_function, but it does not work on my datasette-installation (0.33). I've added the following file to my datasette-dir: /plugins/sql_functions.py: `from datasette import hookimpl def escape_fts_query(query): bits = query.split() return ' '.join('"{}"'.format(bit.replace('"', '')) for bit in bits) @hookimpl def prepare_connection(conn): conn.create_function("escape_fts_query", 1, escape_fts_query)` It has no effect on the standard queries to the tables though, as they still produce errors when including any characters like '-', '/', '+' or '?' Does the function only work when using costum queries, where I can include the escape_fts-function explicitly in the sql-query? PS. I'm calling datasette with --plugins=plugins, and my other plugins work just fine. PPS. The fts5 virtual table is created with 'sqlite3' like so: `CREATE VIRTUAL TABLE "cases_fts" USING FTS5( title, subtitle, resume, suggestion, presentation, detail = full, content_rowid = 'id', content = 'cases', tokenize='unicode61', 'remove_diacritics 2', 'tokenchars "-_"' );` Thanks! _Originally posted by @clausjuhl in https://github.com/simonw/datasette/issues/651#issuecomment-579675357_ | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/662/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
500783373 | MDU6SXNzdWU1MDA3ODMzNzM= | 62 | [enhancement] Method to delete a row in python | 4454869 | closed | 0 | 5 | 2019-10-01T09:45:47Z | 2019-11-04T16:30:34Z | 2019-11-04T16:18:18Z | NONE | Hi ! Thanks for the lib ! Obviously, every possible sql queries won't have a dedicated method. But I was thinking : a method to delete a row (I'm terrible with names, maybe `delete_where()` or something, would be useful. I have a Database, with primary key. For the moment, I use : ```Python3 db.conn.execute(f"DELETE FROM table WHERE key = {key_id}") db.conn.commit() ``` to delete a row I don't need anymore, giving his primary key. Works like a charm. Just an idea : ```Python3 table.delete_where_pkey({'key': key_id}) ``` or something (I know, I'm terrible at naming methods...). Pros : well, no need to write SQL query. Cons : WHERE normally allows to do many more things (operators =, <>, >, <, BETWEEN), not to mention AND, OR, etc... Method is maybe to specific, and/or a pain to render more flexible. Again, just a thought. Writing his own sql works too, so... Thanks again. See yah. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/62/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
480961330 | MDU6SXNzdWU0ODA5NjEzMzA= | 54 | Ability to list views, and to access db["view_name"].rows / rows_where / etc | 20264 | closed | 0 | 5 | 2019-08-15T02:00:28Z | 2019-08-23T12:41:09Z | 2019-08-23T12:20:15Z | NONE | The docs show me how to create a view via `db.create_view()` but I can't seem to get back to that view post-creation; if I query it as a table it returns `None`, and it doesn't appear in the table listing, even though querying the view works fine from inside the sqlite3 command-line. It'd be great to have the view as a pseudo-table, or if the python/sqlite3 module makes that hard to pull off (I couldn't figure it out), to have that edge-case documented next to the `db.create_view()` docs. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/54/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
333238932 | MDU6SXNzdWUzMzMyMzg5MzI= | 316 | datasette inspect takes a very long time on large dbs | 132230 | closed | 0 | 5 | 2018-06-18T11:56:27Z | 2019-05-11T18:26:25Z | 2019-05-11T18:26:25Z | NONE | Hi, I want to expose data in a very large sqlite database (~600Gb) to the web. I have used datasette with success on smaller test databases with the same schema - it works very well (thanks!). However, using the full db, both `datasette inspect` and `datasette serve` seem to hang or pause for a very long time (tens of minutes) on startup. Is this expected behaviour? (I noticed that the output of `datasette inspect` includes row counts for each table. Simply counting the rows in this db will take a long time (tens of millions of rows across each of ~10 tables), so I wondered if this is the source of the problem.) Any help on a workaround would be appreciated. | 107914493 | issue | { "url": "https://api.github.com/repos/simonw/datasette/issues/316/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed |