github
id | node_id | number | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at | author_association | pull_request | body | repo | type | active_lock_reason | performed_via_github_app | reactions | draft | state_reason |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1114543475 | I_kwDOCGYnMM5CbpVz | 388 | Link to stable docs from older versions | 9599 | closed | 0 | 7 | 2022-01-26T01:55:46Z | 2023-03-26T23:43:12Z | 2022-01-26T02:00:22Z | OWNER | https://sqlite-utils.datasette.io/en/2.14.1/ isn't showing a link to the stable release right now. I should also apply the same fix I used for Datasette in: - https://github.com/simonw/datasette/issues/1608 TIL: https://til.simonwillison.net/readthedocs/link-from-latest-to-stable | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/388/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1620516340 | I_kwDOCGYnMM5glx30 | 533 | ReadTheDocs error: not all arguments converted during string formatting | 9599 | closed | 0 | 2 | 2023-03-12T21:21:05Z | 2023-03-12T21:25:33Z | 2023-03-12T21:25:33Z | OWNER | This came up as a failure running tests for: - #531 Traceback on https://readthedocs.org/projects/sqlite-utils/builds/19749348/ ``` File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/531/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 889, in interpreted nodes, messages2 = role_fn(role, rawsource, text, lineno, self) File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/531/lib/python3.8/site-packages/sphinx/ext/extlinks.py", line 103, in role title = caption % part TypeError: not all arguments converted during string formatting Exception occurred: File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/531/lib/python3.8/site-packages/sphinx/ext/extlinks.py", line 103, in role title = caption % part TypeError: not all arguments converted during string formatting ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/533/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1572766460 | I_kwDOCGYnMM5dvoL8 | 524 | Transformation type `--type DATETIME` | 21095447 | closed | 0 | 15 | 2023-02-06T15:18:42Z | 2023-02-15T12:10:54Z | 2023-02-15T12:10:54Z | NONE | Hey. Currently i do transformation with the type `--type TEXT`, but i noticed using the sqlalchemy based library [dataset](https://github.com/pudo/dataset) that is reading and writing differ depending on the column types `TEXT`, `DATETIME`. Is it possible to alter a column type to `DATETIME` somehow using Sqlite-Utils? | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/524/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1000275035 | PR_kwDOCGYnMM4r7n-9 | 327 | Extract expand: Support JSON Arrays | 101753 | closed | 0 | 0 | 2021-09-19T10:34:30Z | 2022-12-29T09:05:36Z | 2022-12-29T09:05:36Z | NONE | simonw/sqlite-utils/pulls/327 | Hi, I needed to extract data in JSON Arrays to normalize data imports. I've quickly hacked the following together based on #241 which refers to #239 where you, @simonw, wrote: > Could this handle lists of objects too? That would be pretty amazing - if the column has a [{...}, {...}] list in it could turn that into a many-to-many. They way this works in my work is that many-to-many relationships are created for anything that maps to an dictionary in a list, and many-to-one relations for everything else (assumed to be scalar values). Not sure what the best approach here would be? Are many-to-one relationships are at all useful here? What do you think about this approach? I could try to add it to the cli interface and documentation if wanted. Thanks for this awesome piece of software in any case! :sun_with_face: | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/327/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1487764628 | I_kwDOCGYnMM5YrXyU | 518 | flake8 ValueError: Error code '#' supplied to 'extend-ignore' option... | 9599 | closed | 0 | 0 | 2022-12-10T01:30:24Z | 2022-12-10T01:36:46Z | 2022-12-10T01:36:46Z | OWNER | > `Error code '#' supplied to 'extend-ignore' option does not match '^[A-Z]{1,3}[0-9]{0,3}$'` https://github.com/simonw/sqlite-utils/actions/runs/3662011265/jobs/6190770361 I think from this: https://github.com/simonw/sqlite-utils/blob/e660635cea6c32f4022818380b1e1ee88e7c93a6/setup.cfg#L1-L3 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/518/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1487757143 | I_kwDOCGYnMM5YrV9X | 517 | Drop support for Python 3.6 | 9599 | closed | 0 | 1 | 2022-12-10T01:23:31Z | 2022-12-10T01:36:36Z | 2022-12-10T01:36:36Z | OWNER | CI has started failing for Python 3.6: https://github.com/simonw/sqlite-utils/actions/runs/3576322798 <img width="945" alt="image" src="https://user-images.githubusercontent.com/9599/206821688-e4176cb3-ea82-460d-8e93-626fb4fa0269.png"> It's fixable by swiching away from `ubuntu-latest` according to: - https://github.com/actions/setup-python/issues/355#issuecomment-1335042510 But https://endoflife.date/python says that 3.6 end of life was almost 6 years ago, and end of security support nearly 1 year ago. So I'm OK dropping support entirely - Python 3.6 users will still be able to install version 3.30, just not any releases that come next. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/517/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1434911255 | I_kwDOCGYnMM5VhwIX | 510 | Cannot enable FTS5 despite it being available | 1176293 | closed | 0 | 3 | 2022-11-03T16:03:49Z | 2022-11-18T18:37:52Z | 2022-11-17T10:36:28Z | NONE | When I do `sqlite-utils enable-fts my.db table_name column_name` (with or without `--fts5`), I get an FTS4 virtual table instead of the expected FTS5. FTS5 is however available and Python/SQLite versions do not seem to be the issue. I can manually create the FTS5 virtual table, and then Datasette also works with it from this same Python environment. `>>> sqlite3.version` `2.6.0` `>>> sqlite3.sqlite_version` `3.39.4` `PRAGMA compile_options;` includes `ENABLE_FTS5`. `sqlite-utils, version 3.30`. Any ideas what's happening and how to fix? | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/510/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1430563092 | PR_kwDOCGYnMM5B6_6K | 508 | Allow surrogates in parameters | 7908073 | closed | 0 | 2 | 2022-10-31T22:11:49Z | 2022-11-17T15:11:16Z | 2022-10-31T22:55:36Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/508 | closes #507 https://dwheeler.com/essays/fixing-unix-linux-filenames.html <!-- readthedocs-preview sqlite-utils start --> ---- :books: Documentation preview :books:: https://sqlite-utils--508.org.readthedocs.build/en/508/ <!-- readthedocs-preview sqlite-utils end --> | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/508/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1450952393 | I_kwDOCGYnMM5We8bJ | 512 | mypy failures in CI | 9599 | closed | 0 | 3 | 2022-11-16T06:22:48Z | 2022-11-16T07:49:51Z | 2022-11-16T07:49:50Z | OWNER | https://github.com/simonw/sqlite-utils/actions/runs/3472012235 failed on Python 3.11: Truncated output: ``` sqlite_utils/db.py:2467: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True sqlite_utils/db.py:2467: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase sqlite_utils/db.py:2530: error: Incompatible default for argument "where" (default has type "None", argument has type "str") [assignment] sqlite_utils/db.py:2530: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True sqlite_utils/db.py:2530: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase sqlite_utils/db.py:2658: error: Argument 1 to "count_where" of "Queryable" has incompatible type "Optional[str]"; expected "str" [arg-type] Found 23 errors in 1 file (checked 51 source files) ``` Best look at https://github.com/hauntsaninja/no_implicit_optional | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/512/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1436539554 | I_kwDOCGYnMM5Vn9qi | 511 | [insert_all, upsert_all] IntegrityError: constraint failed | 7908073 | closed | 0 | 2 | 2022-11-04T19:21:48Z | 2022-11-04T22:59:54Z | 2022-11-04T22:54:09Z | CONTRIBUTOR | My understand is that `INSERT OR IGNORE` will ignore when inserts would cause duplicate keys so I'm not sure exactly why the error is raised from `sqlite3`. ``` import argparse from pathlib import Path from xklb import db, utils from xklb.utils import log def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser() parser.add_argument("database") parser.add_argument("dbs", nargs="*") parser.add_argument("--upsert") parser.add_argument("--db", "-db", help=argparse.SUPPRESS) parser.add_argument("--verbose", "-v", action="count", default=0) args = parser.parse_args() if args.db: args.database = args.db Path(args.database).touch() args.db = db.connect(args) log.info(utils.dict_filter_bool(args.__dict__)) return args def merge_db(args, source_db): source_db = str(Path(source_db).resolve()) s_db = db.connect(argparse.Namespace(database=source_db, verbose=args.verbose)) for table in [s for s in s_db.table_names() if not "_fts" in s and not s.startswith("sqlite_")]: log.info("[%s]: %s", source_db, table) with s_db.conn: data = s_db[table].rows with args.db.conn: if args.upsert: args.db[table].upsert_all(data, pk=args.upsert.split(","), alter=True) else: args.db[table].insert_all(data, alter=True, replace=True) def merge_dbs(): args = parse_args() for s_db in args.dbs: merge_db(args, s_db) if __name__ == "__main__": merge_dbs() ``` ``` $ lb-dev merge video.db tube_71.db --upsert path -vv SQL: INSERT OR IGNORE INTO [media]([path]) VALUES(?); - params: ['https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz'] ... File ~/.local/lib/python3.10/site-packages/sqlite_utils/db.py:3122, in Table.insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, hash_id_columns, alter, ignore, re… | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/511/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
473083260 | MDU6SXNzdWU0NzMwODMyNjA= | 50 | "Too many SQL variables" on large inserts | 9599 | closed | 0 | 4 | 2019-07-25T21:43:31Z | 2022-11-04T14:38:36Z | 2019-07-28T11:59:33Z | OWNER | Reported here: https://github.com/dogsheep/healthkit-to-sqlite/issues/9 It looks like there's a default limit of 999 variables - we need to be smart about that, maybe dynamically lower the batch size based on the number of columns. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/50/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1429029604 | I_kwDOCGYnMM5VLULk | 506 | Make `cursor.rowcount` accessible (wontfix) | 9599 | closed | 0 | 3 | 2022-10-30T21:51:55Z | 2022-11-01T17:37:47Z | 2022-11-01T17:37:13Z | OWNER | In building this Datasette feature on top of `sqlite-utils` I thought it might be useful to expose the number of rows that had been affected by a bulk insert or update - the `cursor.rowcount`: - https://github.com/simonw/datasette/issues/1866 This isn't currently exposed by `sqlite-utils`. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/506/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1430325103 | I_kwDOCGYnMM5VQQdv | 507 | conn.execute: UnicodeEncodeError: 'utf-8' codec can't encode character | 7908073 | closed | 0 | 1 | 2022-10-31T18:49:51Z | 2022-11-01T00:40:17Z | 2022-11-01T00:40:16Z | CONTRIBUTOR | I'm not really sure what caused this and it happened in the middle of my program (after running for 35775 seconds). ``` Extracting metadata 49.9% (chunk 9893 of 19831) ... File "/home/xk/.local/lib/python3.10/site-packages/xklb/fs_extract.py", line 90, in extract_chunk args.db["media"].insert_all(utils.list_dict_filter_bool(media), pk="path", alter=True, replace=True) File "/home/xk/.local/lib/python3.10/site-packages/sqlite_utils/db.py", line 3107, in insert_all self.insert_chunk( File "/home/xk/.local/lib/python3.10/site-packages/sqlite_utils/db.py", line 2872, in insert_chunk result = self.db.execute(query, params) File "/home/xk/.local/lib/python3.10/site-packages/sqlite_utils/db.py", line 483, in execute return self.conn.execute(sql, parameters) UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc3' in position 62: surrogates not allowed ``` This might be relevant: https://stackoverflow.com/questions/31898353/python-cant-encode-with-surrogateescape I'm going to try re-running with ```py def execute( self, sql: str, parameters: Optional[Union[Iterable, dict]] = None ) -> sqlite3.Cursor: """ Execute SQL query and return a ``sqlite3.Cursor``. :param sql: SQL query to execute :param parameters: Parameters to use in that query - an iterable for ``where id = ?`` parameters, or a dictionary for ``where id = :id`` """ try: if self._tracer: self._tracer(sql, parameters) if parameters is not None: return self.conn.execute(sql, parameters) else: return self.conn.execute(sql) except UnicodeEncodeError: sql = sql.encode('utf-8', 'surrogatepass').decode('utf-8') if parameters is not None: parameters = parameters.encode('utf-8', 'surrogatepass').decode('utf-8') return self.execute(sql, parameters) ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/507/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1423182778 | I_kwDOCGYnMM5U1Au6 | 505 | Release sqlite-utils 3.30 | 9599 | closed | 0 | 2 | 2022-10-25T22:20:05Z | 2022-10-25T22:41:26Z | 2022-10-25T22:41:16Z | OWNER | https://github.com/simonw/sqlite-utils/compare/3.29...defa2974c6d3abc19be28d6b319649b8028dc966 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/505/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1392690202 | I_kwDOCGYnMM5TAsQa | 495 | Support JSON values returned from .convert() functions | 649467 | closed | 0 | 3 | 2022-09-30T16:33:49Z | 2022-10-25T21:23:37Z | 2022-10-25T21:23:28Z | NONE | When using the convert function on a JSON column, the result of the conversion function must be a string. If the return value is either a dict (object) or a list (array), the convert call will error out with an unhelpful user defined function exception. It makes sense that since the original column value was a string and required conversion to data structures, the result should be converted back into a JSON string as well. However, other functions auto-convert to JSON string representation, so the fact that convert doesn't could be surprising. At least the documentation should note this requirement, because the sqlite error messages won't readily reveal the issue. Jf only sqlite's JSON column type meant something :) | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/495/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1393212964 | I_kwDOCGYnMM5TCr4k | 497 | column_names | 7908073 | closed | 0 | 1 | 2022-10-01T03:34:21Z | 2022-10-25T21:09:28Z | 2022-10-25T21:09:28Z | CONTRIBUTOR | It would be nice to have a `column_names`. Similar to `table_names`. Or if you could get one or all of the following syntax to work for both Database and Table that might be even better: Style 1 - `if 'table1' in db` - `if 'col1' in db['table1']` Style 2 - `if 'table1' in db.tables` - `if 'col1' in db['table1'].columns` maybe the table ones actually work but I'm too lazy to check. I just know that I have to do: `[c.name for c in db['table1'].columns]` Edit: This is possible with `columns_dict`. I have actually used that before but I forgot about it. Feel free to close, but I do think accessing this data could be more consistent and intuitive. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/497/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1423069384 | I_kwDOCGYnMM5U0lDI | 504 | db.close() method, calling db.conn.close() | 9599 | closed | 0 | 1 | 2022-10-25T20:50:50Z | 2022-10-25T21:00:29Z | 2022-10-25T20:57:47Z | OWNER | I ended up needing to use `db.conn.close()` to fix this issue: - #503 I think `.close()` should be a method on `Database` itself. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/504/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1423000702 | I_kwDOCGYnMM5U0UR- | 503 | test_recreate failing on Windows Python 3.11 | 9599 | closed | 0 | 10 | 2022-10-25T20:01:41Z | 2022-10-25T20:47:34Z | 2022-10-25T20:45:43Z | OWNER | https://github.com/simonw/sqlite-utils/actions/runs/3323672128/jobs/5494726927 Related: - #502 ``` FAILED tests/test_recreate.py::test_recreate[True-True] - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_recreate_True_True_0\\data.db' FAILED tests/test_recreate.py::test_recreate[False-True] - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_recreate_False_True_0\\data.db' ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/503/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1422954582 | I_kwDOCGYnMM5U0JBW | 502 | Fix tests for Python 3.11 | 9599 | closed | 0 | 1 | 2022-10-25T19:20:31Z | 2022-10-25T19:23:47Z | 2022-10-25T19:23:47Z | OWNER | The way errors are represented has changed: https://github.com/simonw/sqlite-utils/actions/runs/3323588047/jobs/5494127154 ``` _________________________ test_query_invalid_function __________________________ db_path = '/tmp/pytest-of-runner/pytest-0/test_query_invalid_function0/test.db' def test_query_invalid_function(db_path): result = CliRunner().invoke( cli.cli, [db_path, "select bad()", "--functions", "def invalid_python"] ) assert result.exit_code == 1 > assert ( result.output.strip() == "Error: Error in functions definition: invalid syntax (<string>, line 1)" ) E AssertionError: assert 'Error: Error...ing>, line 1)' == 'Error: Error...ing>, line 1)' E - Error: Error in functions definition: invalid syntax (<string>, line 1) E ? ^^^^^^ ^^^^^^ E + Error: Error in functions definition: expected '(' (<string>, line 1) E ? ^^^^^^^ ^^^ ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/502/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1413641049 | I_kwDOCGYnMM5UQnNZ | 501 | Tests failing due to updated tabulate library | 9599 | closed | 0 | 4 | 2022-10-18T18:07:52Z | 2022-10-18T18:23:40Z | 2022-10-18T18:23:40Z | OWNER | Failure here: https://github.com/simonw/sqlite-utils/actions/runs/3275786702/jobs/5391063221 I figured out the problem: ```diff diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst index b88e38a..82b4b6c 100644 --- a/docs/cli-reference.rst +++ b/docs/cli-reference.rst @@ -112,11 +112,15 @@ See :ref:`cli_query`. --tsv Output TSV --no-headers Omit CSV headers -t, --table Output as a formatted table - --fmt TEXT Table format - one of fancy_grid, fancy_outline, - github, grid, html, jira, latex, latex_booktabs, - latex_longtable, latex_raw, mediawiki, moinmoin, - orgtbl, pipe, plain, presto, pretty, psql, rst, - simple, textile, tsv, unsafehtml, youtrack + --fmt TEXT Table format - one of asciidoc, double_grid, + double_outline, fancy_grid, fancy_outline, github, + grid, heavy_grid, heavy_outline, html, jira, + latex, latex_booktabs, latex_longtable, latex_raw, + mediawiki, mixed_grid, mixed_outline, moinmoin, + orgtbl, outline, pipe, plain, presto, pretty, + psql, rounded_grid, rounded_outline, rst, simple, + simple_grid, simple_outline, textile, tsv, + unsafehtml, youtrack --json-cols Detect JSON cols and output them as JSON, not escaped strings -r, --raw Raw output, first column of first row @@ -176,11 +180,15 @@ See :ref:`cli_memory`. --tsv Output TSV --no-headers Omit CSV headers -t, --table Output as a formatte… | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/501/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1413610718 | I_kwDOCGYnMM5UQfze | 500 | Turn --flatten into a documented utility function | 9599 | closed | 0 | 4 | 2022-10-18T17:43:36Z | 2022-10-18T18:02:10Z | 2022-10-18T18:00:40Z | OWNER | The `--flatten` implementation isn't currently available to Python code - people have to roll their own implementation. Feedback from a conversation at DjangoCon. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/500/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1404013495 | PR_kwDOCGYnMM5AicIh | 498 | fix: enable-fts permanently save triggers | 7908073 | closed | 0 | 2 | 2022-10-11T05:10:51Z | 2022-10-15T04:33:08Z | 2022-10-11T06:34:31Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/498 | I was wondering why my all my databases were giving wild search results. Turns out create_trigger was not sticking! Running `sqlite-utils triggers x.db` shows `[]` after running `enable-fts` using the python api. Looking at the counts trigger it seems that is the right way to save triggers. triggers show up now <!-- readthedocs-preview sqlite-utils start --> ---- :books: Documentation preview :books:: https://sqlite-utils--498.org.readthedocs.build/en/498/ <!-- readthedocs-preview sqlite-utils end --> | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/498/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1361355564 | I_kwDOCGYnMM5RJKMs | 482 | balanced table default column_order | 7908073 | closed | 0 | 1 | 2022-09-05T03:00:18Z | 2022-10-10T17:43:02Z | 2022-09-06T20:17:27Z | CONTRIBUTOR | Is there any performance or size difference with column order in SQLITE ? similar to this https://www.cybertec-postgresql.com/en/column-order-in-postgresql-does-matter/ It might be interesting to have an option to create with an optimized column order. I'm assuming this would look something like INTEGER columns, REAL columns, BLOB columns, TEXT columns, NULL columns. NULL columns at the end because they are more likely to be TEXT and it is impossible to know if they will become INTEGER (Of course, any schema evolution would reduce optimization but maybe column order could also be re-evaluated when schema changes) edit: this is easy to accomplish with the existing `transform` method: ``` int_columns = [k for k, v in table_columns.items() if v == int] db[table].transform(column_order=[*int_columns]) ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/482/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1386593843 | I_kwDOCGYnMM5Spb4z | 494 | Document how to use Just | 9599 | closed | 0 | 2 | 2022-09-26T19:25:12Z | 2022-09-26T19:32:36Z | 2022-09-26T19:26:39Z | OWNER | I'm using `just` a lot know, based on this file - I should add that to https://sqlite-utils.datasette.io/en/latest/contributing.html https://github.com/simonw/sqlite-utils/blob/afbd2b2cba45cccb305c3d4638d18db4dd3d4bbd/Justfile#L1-L24 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/494/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1363765916 | I_kwDOCGYnMM5RSWqc | 483 | `sqlite-utils install` command | 9599 | closed | 0 | 2 | 2022-09-06T20:13:55Z | 2022-09-26T19:04:43Z | 2022-09-26T18:57:15Z | OWNER | With the addition of `--functions` in: - #471 In addition to the existing `convert` command, there are now very good reasons to want to install additional packages into the same virtual environment as `sqlite-utils` itself, to allow them to be used with those features. This isn't easy if you installed the tool with `pipx` or `brew install sqlite-utils`. Datasette solved this problem with the `datasette install` command: - https://github.com/simonw/datasette/issues/925 `sqlite-utils` could benefit from the same idea. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/483/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1382457780 | I_kwDOCGYnMM5SZqG0 | 490 | Ability to insert multi-line files | 6180701 | closed | 0 | 4 | 2022-09-22T13:29:22Z | 2022-09-26T18:24:44Z | 2022-09-23T16:37:58Z | NONE | I was looking into how to parse application log files that contain multiline text (e.g. Java stack traces) into sqlite. I can see that at the moment `--lines` helps, but falls short when processing multi-line texts. I wonder if this functionality would be useful for sqlite-utils. A similar approach to Elastic logstash/filebeat can be adopted: https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html Potential changes: - add a `--multiline` option - additional properties for - multiline-pattern (regex expression) - multiline-negate: true/false - multiline-what: previous or next Or if this is achievable in a different way, please share. Thanks! | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/490/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1366512990 | PR_kwDOCGYnMM4-nBs9 | 486 | progressbar for inserts/upserts of all fileformats, closes #485 | 99098079 | closed | 0 | 7 | 2022-09-08T14:58:02Z | 2022-09-15T20:40:03Z | 2022-09-15T20:37:51Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/486 | <!-- readthedocs-preview sqlite-utils start --> ---- :books: Documentation preview :books:: https://sqlite-utils--486.org.readthedocs.build/en/486/ <!-- readthedocs-preview sqlite-utils end --> | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1366423176 | I_kwDOCGYnMM5RcfaI | 485 | Progressbar not shown when inserting/upserting jsonlines file | 99098079 | closed | 0 | 1 | 2022-09-08T14:13:18Z | 2022-09-15T20:39:52Z | 2022-09-15T20:37:52Z | CONTRIBUTOR | When inserting or upserting a jsonlines file, no progressbar is shown. Expected behavior is that, just like with .csv/.tsv files, also for a jsonlines file (--nl), unless --silent is provided, a progressbar is shown. ```bash sql-utils upsert mydb.db posts posts.jl --nl --pk post_id (silence) ``` Currently `file_progress` is only called within the tsv/csv logic, however I think it can be safely wrapped around all the all the input formats that use `decoded`: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/cli.py#L963 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/485/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1367835380 | I_kwDOCGYnMM5Rh4L0 | 487 | Specify foreign key against compound key in other table | 540968 | closed | 0 | 2 | 2022-09-09T13:32:09Z | 2022-09-11T04:00:44Z | 2022-09-11T04:00:44Z | NONE | When inserting rows via the library, is it possible to specify a foreign key to a compound primary key? For example, suppose I create a table: ``` db = Database('events.db') db['events'].insert_all([ {'venue': 'Times Square', 'date': '2022-12-31', 'title': 'Rockin New Year Eve'}, {'venue': 'Wembley Stadium', 'date': '2022-06-05', 'title': 'FA Cup'}, {'venue': 'Times Square', 'date': '2021-12-31', 'title': 'Rockin New Year Eve'}, ], pk=('date', 'venue')) ``` And I want to add related data in another table: ``` act = {'name': 'Rick Astley', 'venue': 'Times Square', 'date': '2021-12-31' } db['performers'].insert(act, pk=<???>) ``` Is it possible to specify a value for `pk` that will point to the compound primary key in `events`? SQLite does support it: https://www.sqlite.org/foreignkeys.html#fk_composite | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/487/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1353441389 | I_kwDOCGYnMM5Qq-Bt | 477 | Conda Forge | 49702524 | closed | 0 | 2 | 2022-08-28T19:03:08Z | 2022-09-07T03:46:55Z | 2022-09-07T03:46:55Z | NONE | Hello! I have successfully put this package on to Conda Forge, and I have extending the invitation for the owner/maintainers of this package to be maintainers on Conda Forge as well. Let me know if you are interested! Thanks. https://github.com/conda-forge/sqlite-utils-feedstock | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/477/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1352932716 | I_kwDOCGYnMM5QpB1s | 471 | sqlite-utils query --functions mechanism for registering extra functions | 9599 | closed | 0 | 8355157 | 12 | 2022-08-27T03:57:53Z | 2022-09-07T03:46:26Z | 2022-08-27T05:10:57Z | OWNER | It would be really cool if you could register additional custom SQL functions for use with the `sqlite-utils query` command - something like this: ``` sqlite-utils data.db 'update images set domain = extract_domain(url)' --functions ' from urllib.parse import urlparse def extract_domain(url): return urlparse(url).netloc ' ``` Every function defined in that code block would be registered with the connection, unless the name began with an underscore. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1355433619 | PR_kwDOCGYnMM4-B7Mc | 480 | search_sql add include_rank option | 7908073 | closed | 0 | 4 | 2022-08-30T09:10:29Z | 2022-08-31T03:40:35Z | 2022-08-31T03:40:35Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/480 | I haven't tested this yet but wanted to get a heads-up whether this kind of change would be useful or if I should just duplicate the function and tweak it within my code <!-- readthedocs-preview sqlite-utils start --> ---- :books: Documentation preview :books:: https://sqlite-utils--480.org.readthedocs.build/en/480/ <!-- readthedocs-preview sqlite-utils end --> | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/480/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1178546862 | I_kwDOCGYnMM5GPzKu | 420 | Document how to use a `--convert` function that runs initialization code first | 770231 | closed | 0 | 12 | 2022-03-23T19:07:36Z | 2022-08-28T11:34:37Z | 2022-03-25T20:07:33Z | NONE | When I have an insert command with transform like this: ``` cat items.json | jq '.data' | sqlite-utils insert listings.db listings - --convert ' d = enchant.Dict("en_US") row["is_dictionary_word"] = d.check(row["name"]) ' --import=enchant --ignore ``` I noticed as the number of rows increases the operation becomes quite slow, likely due to the creation of the `d = enchant.Dict("en_US")` object for each row. Is there a way to share that instance `d` between transform function calls, like a shared context? | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/420/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1353196970 | I_kwDOCGYnMM5QqCWq | 476 | Release notes for 3.29 | 9599 | closed | 0 | 8355157 | 2 | 2022-08-27T23:21:21Z | 2022-08-28T04:07:15Z | 2022-08-28T04:07:03Z | OWNER | https://github.com/simonw/sqlite-utils/compare/3.28...104f37fa4d2e7e5999c1d829267b62c737f74d3e | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/476/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1348169997 | I_kwDOCGYnMM5QW3EN | 467 | Mechanism for ensuring a table has all the columns | 9599 | closed | 0 | 8355157 | 13 | 2022-08-23T15:50:23Z | 2022-08-27T23:19:41Z | 2022-08-27T23:17:56Z | OWNER | Suggested by @jefftriplett on Discord: https://discord.com/channels/823971286308356157/997738192360964156/1011655389063958600 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/467/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1348294436 | PR_kwDOCGYnMM49qP2V | 468 | db[table].create(..., transform=True) and create-table --transform | 9599 | closed | 0 | 8355157 | 6 | 2022-08-23T17:27:58Z | 2022-08-27T23:17:55Z | 2022-08-27T23:17:55Z | OWNER | simonw/sqlite-utils/pulls/468 | Work in progress. Still needs documentation and tests (and to cover more cases of things that might have changed). Refs: - #467 <!-- readthedocs-preview sqlite-utils start --> ---- :books: Documentation preview :books:: https://sqlite-utils--468.org.readthedocs.build/en/468/ <!-- readthedocs-preview sqlite-utils end --> | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/468/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | ||||
1353189941 | I_kwDOCGYnMM5QqAo1 | 475 | table.default_values introspection property | 9599 | closed | 0 | 8355157 | 1 | 2022-08-27T22:33:31Z | 2022-08-27T22:44:46Z | 2022-08-27T22:43:02Z | OWNER | > Interesting challenge with `default_value`: I need to be able to tell if the default values passed to `.create()` differ from those in the database already. > > Introspecting that is a bit tricky: > > ```pycon > >>> import sqlite_utils > >>> db = sqlite_utils.Database(memory=True) > >>> db["blah"].create({"id": int, "name": str}, not_null=("name",), defaults={"name": "bob"}) > <Table blah (id, name)> > >>> db["blah"].columns > [Column(cid=0, name='id', type='INTEGER', notnull=0, default_value=None, is_pk=0), Column(cid=1, name='name', type='TEXT', notnull=1, default_value="'bob'", is_pk=0)] > ``` > Note how a default value of the Python string `bob` is represented in the results of `PRAGMA table_info()` as `default_value="'bob'"` - it's got single quotes added to it! > > So comparing default values from introspecting the database needs me to first parse that syntax. This may require a new table introspection method. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/468#issuecomment-1229279539_ | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/475/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1199158210 | I_kwDOCGYnMM5HebPC | 423 | .extract() doesn't set foreign key when extracted columns contain NULL value | 37447552 | closed | 0 | 1 | 2022-04-10T20:05:30Z | 2022-08-27T14:45:04Z | 2022-08-27T14:45:04Z | NONE | I've run into an issue with `extract` and I don't believe this is the intended behaviour. I'm working with a database with music listening information. Currently it has one large table `listens` that contains all information. I'm trying to normalize the database by extracting relevant columns to separate tables (`artists`, `tracks`, `albums`). Not every track has an album. A simplified demonstration with just `track_title` and `album_title` columns: ```ipython In [1]: import sqlite_utils In [2]: db = sqlite_utils.Database(memory=True) In [3]: db["listens"].insert_all([ ...: {"id": 1, "track_title": "foo", "album_title": "bar"}, ...: {"id": 2, "track_title": "baz", "album_title": None} ...: ], pk="id") Out[3]: <Table listens (id, track_title, album_title)> ``` The track in the first row has an album, the second track doesn't. Now I extract album information into a separate column: ```ipython In [4]: db["listens"].extract(columns=["album_title"], table="albums", fk_column="album_id") Out[4]: <Table listens (id, track_title, album_id)> In [5]: list(db["albums"].rows) Out[5]: [{'id': 1, 'album_title': 'bar'}, {'id': 2, 'album_title': None}] In [6]: list(db["listens"].rows) Out[6]: [{'id': 1, 'track_title': 'foo', 'album_id': 1}, {'id': 2, 'track_title': 'baz', 'album_id': None}] ``` This behaves as expected -- the `album` table contains entries for both the existing album and the NULL album. The `listens` table has a foreign key only for the first row (since the album in the second row was empty). Now I want to extract the track information as well. Album information belongs to the track so I want to extract both columns to a new table. ```ipython In [7]: db["listens"].extract(columns=["track_title", "album_id"], table="tracks", fk_column="track_id") Out[7]: <Table listens (id, track_id)> In [8]: list(db["tracks"].rows) Out[8]: [{'id': 1, 'track_title': 'foo', 'album_id': 1}, {'id': 2, 'track_title': 'baz', 'album_id': None}] In [9]: list(db["… | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/423/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1309542173 | PR_kwDOCGYnMM47pwAb | 455 | in extract code, check equality with IS instead of = for nulls | 536941 | closed | 0 | 3 | 2022-07-19T13:40:25Z | 2022-08-27T14:45:03Z | 2022-08-27T14:45:03Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/455 | sqlite "IS" is equivalent to SQL "IS NOT DISTINCT FROM" closes #423 | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/455/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1352953535 | PR_kwDOCGYnMM4950Az | 473 | Support entrypoints for `--load-extension` | 9599 | closed | 0 | 1 | 2022-08-27T05:53:59Z | 2022-08-27T05:55:52Z | 2022-08-27T05:55:47Z | OWNER | simonw/sqlite-utils/pulls/473 | Refs #470 <!-- readthedocs-preview sqlite-utils start --> ---- :books: Documentation preview :books:: https://sqlite-utils--473.org.readthedocs.build/en/473/ <!-- readthedocs-preview sqlite-utils end --> | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/473/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1352932038 | I_kwDOCGYnMM5QpBrG | 470 | Upgrade `--load-extension` to accept entrypoints like Datasette | 9599 | closed | 0 | 8355157 | 6 | 2022-08-27T03:53:20Z | 2022-08-27T05:55:49Z | 2022-08-27T05:55:48Z | OWNER | Imitate: - https://github.com/simonw/datasette/pull/1789 ``` # would load default entrypoint like before datasette data.db --load-extension ext # loads the extensions with the "sqlite3_foo_init" entrpoint datasette data.db --load-extension ext:sqlite3_foo_init # loads the extensions with the "sqlite3_bar_init" entrpoint datasette data.db --load-extension ext:sqlite3_bar_init ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/470/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1352946135 | I_kwDOCGYnMM5QpFHX | 472 | Reuse the locals/globals fix from --functions for other code accepting options | 9599 | closed | 0 | 8355157 | 2 | 2022-08-27T05:12:05Z | 2022-08-27T05:20:12Z | 2022-08-27T05:20:12Z | OWNER | I figured out a workaround for the ugly `global x` hack here: - https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229120653 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/472/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1352931464 | I_kwDOCGYnMM5QpBiI | 469 | sqlite-utils rows --order option | 9599 | closed | 0 | 8355157 | 1 | 2022-08-27T03:49:51Z | 2022-08-27T04:30:49Z | 2022-08-27T04:10:32Z | OWNER | For consistency with `search`: https://sqlite-utils.datasette.io/en/stable/cli-reference.html#search ``` -o, --order TEXT Order by ('column' or 'column desc') ``` I wanted to run `sqlite-utils rows db.db mytable --order 'rowid desc'` to see the most recently imported rows. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/469/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1320243134 | I_kwDOCGYnMM5OsU-- | 458 | Support custom names for registered functions | 9599 | closed | 0 | 8355157 | 1 | 2022-07-28T00:13:00Z | 2022-08-27T03:56:01Z | 2022-07-28T00:13:57Z | OWNER | In this example: ```python @db.register_function def reverse_string(s): return "".join(reversed(list(s))) print(db.execute('select reverse_string("hello")').fetchone()[0]) ``` There's currently no way to over-ride the automatically selected name for the SQL function. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/458/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1319881016 | PR_kwDOCGYnMM48Mmde | 457 | Link to installation instructions | 9599 | closed | 0 | 8355157 | 2 | 2022-07-27T17:38:36Z | 2022-08-27T03:55:52Z | 2022-07-27T17:57:50Z | OWNER | simonw/sqlite-utils/pulls/457 | Also testing https://docs.readthedocs.io/en/stable/pull-requests.html | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/457/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | ||||
1326087800 | PR_kwDOCGYnMM48hI-_ | 460 | Cross-link CLI to Python docs | 9599 | closed | 0 | 4 | 2022-08-02T16:18:28Z | 2022-08-18T21:58:10Z | 2022-08-18T21:58:07Z | OWNER | simonw/sqlite-utils/pulls/460 | Work in progress, partly to test the ReadTheDocs preview link action. Refs: - #426 <!-- readthedocs-preview readthedocs-preview start --> ---- :books: Documentation preview :books:: https://readthedocs-preview--460.org.readthedocs.build/en/460/ <!-- readthedocs-preview readthedocs-preview end --> | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/460/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1334416486 | PR_kwDOCGYnMM488n6D | 463 | Use Read the Docs action v1 | 244656 | closed | 0 | 1 | 2022-08-10T10:31:47Z | 2022-08-18T08:30:14Z | 2022-08-17T23:11:16Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/463 | Read the Docs repository was renamed from `readthedocs/readthedocs-preview` to `readthedocs/actions/`. Now, the `preview` action is under `readthedocs/actions/preview` and is tagged as `v1` <!-- readthedocs-preview sqlite-utils start --> ---- :books: Documentation preview :books:: https://sqlite-utils--463.org.readthedocs.build/en/463/ <!-- readthedocs-preview sqlite-utils end --> | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/463/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1342357149 | PR_kwDOCGYnMM49Wsnq | 465 | beanbag-docutils>=2.0 | 9599 | closed | 0 | 2 | 2022-08-17T22:41:39Z | 2022-08-17T23:38:07Z | 2022-08-17T23:38:02Z | OWNER | simonw/sqlite-utils/pulls/465 | Refs #464 | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/465/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1338001039 | I_kwDOCGYnMM5PwEaP | 464 | Link from documentation to source code | 9599 | closed | 0 | 5 | 2022-08-13T16:19:57Z | 2022-08-17T23:38:03Z | 2022-08-17T23:38:03Z | OWNER | Twitter conversation asking for ways to automate this here: https://twitter.com/simonw/status/1558260492015046656 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/464/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1342374388 | PR_kwDOCGYnMM49Wv9T | 466 | Use Read the Docs action v1 (#463) | 9599 | closed | 0 | 0 | 2022-08-17T23:11:50Z | 2022-08-17T23:11:54Z | 2022-08-17T23:11:54Z | OWNER | simonw/sqlite-utils/pulls/466 | Read the Docs repository was renamed from `readthedocs/readthedocs-preview` to `readthedocs/actions/`. Now, the `preview` action is under `readthedocs/actions/preview` and is tagged as `v1` | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/466/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1326391841 | PR_kwDOCGYnMM48iLGF | 462 | Discord badge | 9599 | closed | 0 | 2 | 2022-08-02T20:56:04Z | 2022-08-02T21:15:57Z | 2022-08-02T21:15:52Z | OWNER | simonw/sqlite-utils/pulls/462 | Also testing fix for: - https://github.com/readthedocs/readthedocs-preview/issues/10 <!-- readthedocs-preview sqlite-utils start --> ---- :books: Documentation preview :books:: https://sqlite-utils--462.org.readthedocs.build/en/462/ <!-- readthedocs-preview sqlite-utils end --> | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/462/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1292060682 | I_kwDOCGYnMM5NA0gK | 450 | Add --ignore option to more commands | 9599 | closed | 0 | 9 | 2022-07-02T13:52:02Z | 2022-07-15T22:39:09Z | 2022-07-15T22:37:45Z | OWNER | As seen in https://sqlite-utils.datasette.io/en/stable/cli-reference.html#add-foreign-key Could make this TIL trick unnecessary: https://til.simonwillison.net/bash/ignore-errors | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/450/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1298531653 | I_kwDOCGYnMM5NZgVF | 451 | Make sqlite_utils.utils.chunks a documented function | 9599 | closed | 0 | 2 | 2022-07-08T06:01:04Z | 2022-07-15T22:09:34Z | 2022-07-15T21:59:33Z | OWNER | I want to use it in another project: https://github.com/simonw/sqlite-utils/blob/8a9fe6498faf783a1fdeb1793e661ad194a05267/sqlite_utils/utils.py#L471-L474 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/451/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1303169663 | I_kwDOCGYnMM5NrMp_ | 453 | 'unclosed file' warning when using insert_upsert_implementation from Python | 311257 | closed | 0 | 1 | 2022-07-13T09:34:35Z | 2022-07-15T21:52:25Z | 2022-07-15T21:52:21Z | NONE | I'm using the `[insert_upsert_implementation](https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/cli.py)` function directly in my Python code to import a csv file with all the bells and whistles `sqlite-utils` provides, but I'm getting a resource warning that a io.TextWrapper object is not closed. The warning goes away when wrapping the code from [this line](https://github.com/simonw/sqlite-utils/blob/42440d6345c242ee39778045e29143fb550bd2c2/sqlite_utils/cli.py#L924) in a try/finally block like: ``` try: ... ... finally: decoded.close() ``` (might be that `sniff_buffer` must also be closed if non null, but I might be wrong) I suspect Python closes the reference automatically when the sqlite-utils cli run is done, but since my code doesn't exit, I'm getting the warning. Alternatively, it'd be cool if the 'import csv/tsv' functionality could be added directly to the Database class. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/453/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1306548397 | I_kwDOCGYnMM5N4Fit | 454 | CLI command for duplicating tables | 9599 | closed | 0 | 1 | 2022-07-15T21:31:27Z | 2022-07-15T21:48:23Z | 2022-07-15T21:45:51Z | OWNER | CLI equivalent of: - #449 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/454/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1279863844 | I_kwDOCGYnMM5MSSwk | 449 | Utilities for duplicating tables and creating a table with the results of a query | 1690072 | closed | 0 | 4 | 2022-06-22T09:41:43Z | 2022-07-15T21:46:13Z | 2022-07-15T21:21:36Z | CONTRIBUTOR | is there a duplicate table functionality? Otherwise, I'd be happy to submit a PR. In sqlite3 it would look like: ```python import sqlite3 as sl con = sl.connect('prompt-tune.db') def db_duplicate_table(table_name, table_name_new, con=con): # Duplicates table `table_name` to a new table `table_name_new`. try: cur = con.cursor() cur.execute(f"""CREATE TABLE {table_name_new} AS SELECT * FROM {table_name}""") except Exception as e: print(e) finally: cur.close() db_duplicate_table('orig_table', 'new_table') ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/449/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1299760627 | PR_kwDOCGYnMM47JUun | 452 | Add duplicate table feature | 1690072 | closed | 0 | 1 | 2022-07-09T20:24:31Z | 2022-07-15T21:21:37Z | 2022-07-15T21:21:36Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/452 | This PR addresses a feature request raised in issue #449. Specifically this PR adds a functionality that lets users duplicate a table via: ```python table_new = db["my_table"].duplicate("new_table") ``` Test added in file `tests/test_duplicate.py`. Happy to make changes to meet maintainers' feedback, if any. | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/452/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1212701569 | I_kwDOCGYnMM5ISFuB | 427 | sqlite-utils convert date parsing recipe complains about trying to parse "*" | 1385831 | closed | 0 | 1 | 2022-04-22T19:27:10Z | 2022-07-02T13:59:59Z | 2022-07-02T13:59:32Z | NONE | Missing values in my dataset are denoted by a single asterisk. I am trying to parse string dates into dates. This works fine for columns without missing values, but, when the column contains "*", I get the following: ``` $ sqlite-utils convert ${dbfile} details dob 'r.parsedate(value)' [------------------------------------] 0%Traceback (most recent call last): File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/db.py", line 2508, in convert_value return fn(v) File "<string>", line 2, in fn File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/recipes.py", line 8, in parsedate parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat() File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 1368, in parse return DEFAULTPARSER.parse(timestr, **kwargs) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 643, in parse raise ParserError("Unknown string format: %s", timestr) dateutil.parser._parser.ParserError: Unknown string format: * Traceback (most recent call last): File "/usr/local/bin/sqlite-utils", line 33, in <module> sys.exit(load_entry_point('sqlite-utils==3.25.1', 'console_scripts', 'sqlite-utils')()) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1128, in __call__ return self.main(*args, **kwargs) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, … | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/427/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1277328147 | I_kwDOCGYnMM5MInsT | 446 | Use Just to automate running tests and linters locally | 9599 | closed | 0 | 2 | 2022-06-20T19:51:09Z | 2022-06-21T19:28:35Z | 2022-06-20T19:54:50Z | OWNER | I keep committing code that fails additional tests like `mypy` and `flake8` and `black`. Automate those using Just. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/446/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1278571700 | I_kwDOCGYnMM5MNXS0 | 447 | Incorrect syntax highlighting in docs CLI reference | 9599 | closed | 0 | 3 | 2022-06-21T14:53:10Z | 2022-06-21T18:48:47Z | 2022-06-21T18:48:46Z | OWNER | https://sqlite-utils.datasette.io/en/stable/cli-reference.html#insert ![CE020DDA-27FB-49C3-9EA6-37457DC4C321](https://user-images.githubusercontent.com/9599/174830380-06530537-b870-41c0-a8af-03c7fa720c6f.jpeg) It looks like Python keywords are being incorrectly highlighted here. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/447/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1269998342 | I_kwDOCGYnMM5LsqMG | 443 | Make `utils.rows_from_file()` a documented API | 9599 | closed | 0 | 2 | 2022-06-13T21:53:24Z | 2022-06-20T19:49:37Z | 2022-06-14T20:12:46Z | OWNER | > `rows_from_file()` isn't part of the documented API but maybe it should be! _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154385916_ | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/443/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1277295119 | I_kwDOCGYnMM5MIfoP | 445 | `sqlite_utils.utils.TypeTracker` should be a documented API | 9599 | closed | 0 | 3 | 2022-06-20T19:08:28Z | 2022-06-20T19:49:02Z | 2022-06-20T19:46:58Z | OWNER | I've used it in a couple of external places now: - https://github.com/simonw/datasette-socrata/blob/32fb256a461bf0e790eca10bdc7dd9d96c20f7c4/datasette_socrata/__init__.py#L264-L280 - https://github.com/simonw/datasette-lite/blob/caa8eade10f0321c64f9f65c4561186f02d57c5b/webworker.js#L55-L64 Refs: - https://github.com/simonw/datasette-lite/issues/32 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/445/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1243151184 | I_kwDOCGYnMM5KGPtQ | 434 | `detect_fts()` identifies the wrong table if tables have names that are subsets of each other | 559711 | closed | 0 | 3 | 2022-05-20T13:28:31Z | 2022-06-14T23:24:09Z | 2022-06-14T23:24:09Z | NONE | Windows 10 Python 3.9.6 When I was running a full text search through the Python library, I noticed that the query was being run on a different full text search table than the one I was trying to search. I took a look at the following function https://github.com/simonw/sqlite-utils/blob/841ad44bacaff05ec79ef78166d12e80c82ba6d7/sqlite_utils/db.py#L2213 and noticed: ```python sql LIKE '%VIRTUAL TABLE%USING FTS%content=%{table}%' ``` My database contains tables with similar names and %{table}% was matching another table that ended differently in its name. I have included a sample test that shows this occurring: I search for Marsupials in db["books"] and The Clue of the Broken Blade is returned. This occurs since the search for Marsupials was "successfully" done against db["booksb"] and rowid 1 is returned. "The Clue of the Broken Blade" has a rowid of 1 in db["books"] and this is what is returned from the search. ```python def test_fts_search_with_similar_table_names(fresh_db): db = Database(memory=True) db["books"].insert_all( [ { "title": "The Clue of the Broken Blade", "author": "Franklin W. Dixon", }, { "title": "Habits of Australian Marsupials", "author": "Marlee Hawkins", }, ] ) db["booksb"].insert( { "title": "Habits of Australian Marsupials", "author": "Marlee Hawkins", } ) db["booksb"].enable_fts(["title", "author"]) db["books"].enable_fts(["title", "author"]) query = "Marsupials" assert [ { "rowid": 1, "title": "Habits of Australian Marsupials", "author": "Marlee Hawkins", }, ] == list(db["books"].search(query)) ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/434/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1250629388 | I_kwDOCGYnMM5KixcM | 440 | CSV files with too many values in a row cause errors | 4068 | closed | 0 | 20 | 2022-05-27T10:54:44Z | 2022-06-14T22:23:01Z | 2022-06-14T20:12:46Z | NONE | *Original title: csv.DictReader can have None as key* In some cases, `csv.DictReader` can have `None` as key for unnamed columns, and a list of values as value. `sqlite_utils.utils.rows_from_file` cannot handle that: ```python url="https://artsdatabanken.no/Fab2018/api/export/csv" db = sqlite_utils.Database(":memory") with urlopen(url) as fab: reader, _ = sqlite_utils.utils.rows_from_file(fab, encoding="utf-16le") db["fab2018"].insert_all(reader, pk="Id") ``` Result: ``` Traceback (most recent call last): File "<stdin>", line 3, in <module> File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 2924, in insert_all chunk = list(chunk) File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 3454, in fix_square_braces if any("[" in key or "]" in key for key in record.keys()): File "/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py", line 3454, in <genexpr> if any("[" in key or "]" in key for key in record.keys()): TypeError: argument of type 'NoneType' is not iterable ``` Code: https://github.com/simonw/sqlite-utils/blob/59be60c471fd7a2c4be7f75e8911163e618ff5ca/sqlite_utils/db.py#L3454 `sqlite-utils insert` from command line is not affected by this issue. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1257724585 | I_kwDOCGYnMM5K91qp | 441 | Combining `rows_where()` and `search()` to limit which rows are searched | 1448859 | closed | 0 | 4 | 2022-06-02T06:01:55Z | 2022-06-14T21:57:57Z | 2022-06-14T21:54:38Z | NONE | What is the right way to limit a full text search query to some rows of a table? For example, I have a table that contains the following columns: `title`, `content`, `owner` (each row represents a document). The `owner` column is a username. It feels right to store all documents in one table, instead of having one table per owner. In particular because I'd like to full text search all documents, only documents owned by one user and documents owned by a set of users. I tried to combine `.rows_where("owner = ?", "1234")` and `.search()` from the `Table` class but I don't think that is meant to work. I discovered `.search_sql()` as a way to generate the FTS SQL statement. By hand I can edit it to add a `AND [original].[owner] = :owner` to the `where` clause. This seems to do what I want. My two questions: 1. is adding a `AND ...` to the `where` clause actually the right thing to do or should I be doing something else (my SQL skills are low)? 2. is there a built-in to sqlite-utils way to achieve this? Right now I am thinking I will make my own version of `search_sql()` that generates a query that contains an additional `owner = :owner` for my particular use-case. Bonus question: is this generally useful/something to add to sqlite-utils or too niche? | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/441/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1269886084 | I_kwDOCGYnMM5LsOyE | 442 | `maximize_csv_field_size_limit()` utility function | 9599 | closed | 0 | 2 | 2022-06-13T19:54:54Z | 2022-06-14T21:55:15Z | 2022-06-14T21:31:49Z | OWNER | This code here runs only if `cli.py` is imported: https://github.com/simonw/sqlite-utils/blob/7ddf5300886a32d6daf60cf1d71efe492b65c87e/sqlite_utils/cli.py#L50-L59 I found myself needing the same fix in another library: - https://github.com/simonw/datasette-socrata/issues/13 It should be a documented utility function. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/442/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1244294227 | PR_kwDOCGYnMM44P4GG | 437 | docs to dogs | 114388 | closed | 0 | 1 | 2022-05-22T15:50:33Z | 2022-05-30T21:32:41Z | 2022-05-30T21:32:41Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/437 | Fixes a typo. | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/437/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1250161887 | I_kwDOCGYnMM5Kg_Tf | 438 | illegal UTF-16 surrogate | 4068 | closed | 0 | 2 | 2022-05-26T22:49:52Z | 2022-05-27T08:21:53Z | 2022-05-27T08:21:53Z | NONE | I am trying to insert `https://artsdatabanken.no/Fab2018/api/export/csv` into a SQLite database, but I have an error when using `sqlite-utils`: ``` sqlite-utils insert --csv --delimiter ";" --encoding="utf-16-le" --pk "Id" csv fremmedart test.db [------------------------------------] 0% Error: 'utf-16-le' codec can't decode bytes in position 98-99: illegal UTF-16 surrogate The input you provided uses a character encoding other than utf-8. You can fix this by passing the --encoding= option with the encoding of the file. If you do not know the encoding, running 'file filename.csv' may tell you. It's often worth trying: --encoding=latin-1 ``` I tried to convert the file using `iconv -f "utf-16le" -t "utf-8"`, but I still get a similar error (slightly different position): ``` sqlite-utils insert --csv --delimiter ";" --encoding=utf-8 --pk "Id" csv_utf8 fremmedart test.db [------------------------------------] 0% Error: 'utf-8' codec can't decode byte 0xd9 in position 99: invalid continuation byte The input you provided uses a character encoding other than utf-8. You can fix this by passing the --encoding= option with the encoding of the file. If you do not know the encoding, running 'file filename.csv' may tell you. It's often worth trying: --encoding=latin-1 ``` I have no issues reading such file using this Python code: ```python content = open('csv', encoding='utf-16-le').read()) ``` `in2csv` works too. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/438/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1243715381 | I_kwDOCGYnMM5KIZc1 | 436 | Add "copy to clipboard" button to code examples in documentation | 9599 | closed | 0 | 0 | 2022-05-20T21:53:23Z | 2022-05-20T21:57:53Z | 2022-05-20T21:57:53Z | OWNER | Follows: - #435 Imitates: - https://github.com/simonw/datasette/issues/1748 I'll use https://github.com/executablebooks/sphinx-copybutton - here's the Datasette commit: https://github.com/simonw/datasette/commit/1465fea4798599eccfe7e8f012bd8d9adfac3039 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/436/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1243704847 | I_kwDOCGYnMM5KIW4P | 435 | Switch to Furo documentation theme | 9599 | closed | 0 | 2 | 2022-05-20T21:46:39Z | 2022-05-20T21:56:10Z | 2022-05-20T21:54:43Z | OWNER | As seen in: - https://github.com/simonw/datasette/issues/1746 - https://github.com/simonw/shot-scraper/issues/77 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/435/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1173023272 | I_kwDOCGYnMM5F6uoo | 416 | Options for how `r.parsedate()` should handle invalid dates | 638427 | closed | 0 | 11 | 2022-03-17T23:29:55Z | 2022-05-03T21:36:49Z | 2022-03-21T04:01:39Z | NONE | Exceptions are normal expected behavior when typecasting an invalid format. However, r.parsedate() is really just re-formatting strings and keeping the type as text. So it may be better to print-and-pass on exception so the user can see a complete list of invalid values -- while also allowing for the parser to reformat the remaining valid values. ``` sqlite-utils convert idfpr.db license "Expiration Date" "r.parsedate(value)" [#######-----------------------------] 21% 00:01:57Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/sqlite_utils/db.py", line 2336, in convert_value return fn(v) File "<string>", line 2, in fn File "/usr/local/lib/python3.9/dist-packages/sqlite_utils/recipes.py", line 8, in parsedate parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat() File "/usr/lib/python3/dist-packages/dateutil/parser/_parser.py", line 1374, in parse return DEFAULTPARSER.parse(timestr, **kwargs) File "/usr/lib/python3/dist-packages/dateutil/parser/_parser.py", line 652, in parse raise ParserError("String does not contain a date: %s", timestr) dateutil.parser._parser.ParserError: String does not contain a date: / / ``` In this case, I had just one variation of an invalid date: ' / / '. But theoretically there could be many values that would have to be fixed one at a time with the current exception handling. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/416/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1223177069 | PR_kwDOCGYnMM43LrKB | 429 | Depend on click-default-group-wheel | 9599 | closed | 0 | 2 | 2022-05-02T18:03:10Z | 2022-05-02T18:52:42Z | 2022-05-02T18:05:00Z | OWNER | simonw/sqlite-utils/pulls/429 | Trying to get this to work with Pyodide. Refs: https://github.com/simonw/click-default-group-wheel/issues/3 | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/429/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1180427792 | I_kwDOCGYnMM5GW-YQ | 421 | "Error: near "(": syntax error" when using sqlite-utils indexes CLI | 24938923 | closed | 0 | 8 | 2022-03-25T07:12:51Z | 2022-04-13T22:41:59Z | 2022-04-13T22:41:59Z | NONE | This bug relates to https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1066139147 **New error when using CLI: "sqlite-utils indexes global.db --table"** ``` (app-root) sqlite-utils indexes global.db --table Error: near "(": syntax error (app-root) sqlite-utils --version sqlite-utils, version 3.25.1 (app-root) sqlite3 --version 3.36.0 2021-06-18 18:36:39 (app-root) python --version Python 3.8.11 ``` Dockerfile ``` FROM centos/python-38-centos7 USER root RUN yum update -y RUN yum upgrade -y # epel RUN yum -y install epel-release && yum clean all # SQLite RUN yum -y install zlib-devel geos geos-devel proj proj-devel freexl freexl-devel libxml2-devel WORKDIR /build/ COPY sqlite-autoconf-3360000.tar.gz ./ RUN tar -zxf sqlite-autoconf-3360000.tar.gz WORKDIR /build/sqlite-autoconf-3360000 RUN ./configure RUN make RUN make install # RUN /opt/app-root/bin/python3.8 -m pip install --upgrade pip RUN pip install sqlite-utils ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/421/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1200866134 | I_kwDOCGYnMM5Hk8NW | 424 | Better error message if you try to create a table with no columns | 9599 | closed | 0 | 1 | 2022-04-12T02:43:20Z | 2022-04-13T22:40:15Z | 2022-04-13T22:40:10Z | OWNER | Seen here: - https://github.com/simonw/geojson-to-sqlite/issues/30 Attempting to create a table with no columns produced this confusing error: ``` File "/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/geojson_to_sqlite/utils.py", line 69, in import_features db[table].create(column_types, pk=pk) File "/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/sqlite_utils/db.py", line 863, in create self.db.create_table( File "/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/sqlite_utils/db.py", line 517, in create_table self.execute(sql) File "/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/sqlite_utils/db.py", line 236, in execute return self.conn.execute(sql) sqlite3.OperationalError: near ")": syntax error ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/424/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1175744654 | I_kwDOCGYnMM5GFHCO | 417 | insert fails on JSONL with whitespace | 9954 | closed | 0 | 3 | 2022-03-21T17:58:14Z | 2022-03-25T21:19:06Z | 2022-03-25T21:17:13Z | NONE | Any JSON that is newline-delimited and has whitespace (newlines) between the start of a JSON object and an attribute fails due to a parse error. e.g. given the valid JSONL: ```{ "attribute": "value" } { "attribute": "value2" } ``` I would expect that `sqlite-utils insert --nl my.db mytable file.jsonl` would properly import the data into `mytable`. However, the following error is thrown instead: `json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 1 (char 2)` It makes sense that since the file is intended to be newline separated, the thing being parsed is "{" (which obviously fails), however the default newline-separated output of `jq` isn't compact. Using `jq -c` avoids this problem, but the fix is unintuitive and undocumented. Proposed solutions: 1. Default to a "loose" newline-separated parse; this could be implemented internally as [the equivalent of] a `jq -c` filter ahead of the insert step. 2. Catch the JSONDecodeError (or pre-empt it in the case of a record === "{\n") and give the user a "it looks like your json isn't _actually_ newline-delimited; try running it through `jq -c` instead" error message. It might just have been too early in the morning when I was playing with this, but running pipes of data through sqlite-utils without the 'knack' of it led to some false starts. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/417/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1178456794 | I_kwDOCGYnMM5GPdLa | 418 | Add generated files to .gitignore | 25778 | closed | 0 | 0 | 2022-03-23T17:48:12Z | 2022-03-24T21:01:44Z | 2022-03-24T21:01:44Z | CONTRIBUTOR | I end up with these in my local directory: .hypothesis/ Pipfile Pipfile.lock pyproject.toml Might as well gitignore them. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/418/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1178484369 | PR_kwDOCGYnMM405rPe | 419 | Ignore common generated files | 25778 | closed | 0 | 1 | 2022-03-23T18:06:22Z | 2022-03-24T21:01:44Z | 2022-03-24T21:01:44Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/419 | Closes #418 This adds four files to `.gitignore`: .hypothesis/ Pipfile Pipfile.lock pyproject.toml Those are all generated in the course of development and testing. | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/419/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1091819089 | I_kwDOCGYnMM5BE9ZR | 360 | MemoryError | 559453 | closed | 0 | 1 | 2022-01-01T13:39:17Z | 2022-03-21T04:22:46Z | 2022-03-21T04:22:46Z | NONE | HI, when dealing with large json file (~170GB) i got the following error ``` Traceback (most recent call last): File "/usr/local/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/usr/lib/python3/dist-packages/click/core.py", line 1126, in __call__ return self.main(*args, **kwargs) File "/usr/lib/python3/dist-packages/click/core.py", line 1051, in main rv = self.invoke(ctx) File "/usr/lib/python3/dist-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3/dist-packages/click/core.py", line 1393, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/lib/python3/dist-packages/click/core.py", line 752, in invoke return __callback(*args, **kwargs) File "/usr/local/lib/python3.9/dist-packages/sqlite_utils/cli.py", line 1300, in memory rows, format_used = rows_from_file(csv_fp, format=format, encoding=encoding) File "/usr/local/lib/python3.9/dist-packages/sqlite_utils/utils.py", line 185, in rows_from_file return rows_from_file(buffered, format=Format.JSON) File "/usr/local/lib/python3.9/dist-packages/sqlite_utils/utils.py", line 156, in rows_from_file decoded = json.load(fp) File "/usr/lib/python3.9/json/__init__.py", line 293, in load return loads(fp.read(), MemoryError ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/360/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1171599874 | I_kwDOCGYnMM5F1TIC | 415 | Convert with `--multi` and `--dry-run` flag does not work | 3976183 | closed | 0 | 2 | 2022-03-16T21:59:46Z | 2022-03-21T04:18:24Z | 2022-03-21T04:18:24Z | NONE | It's not possible to combine `--multi` and `--dry-run` flag in the `convert` command. Let's first create a simple database from JSON string ```console $ echo '[{"foo": "abc"}]' | sqlite-utils insert demo.db demo - $ sqlite-utils query demo.db "SELECT * FROM demo" [{"foo": "abc"}] ``` and then try to convert the "foo" column with a static value "bar" (see docs [Converting a column into multiple columns](https://sqlite-utils.datasette.io/en/stable/cli.html#converting-a-column-into-multiple-columns)) ```console $ sqlite-utils convert demo.db demo foo '{"foo": "bar"}' --multi --dry-run Traceback (most recent call last): File "/home/dotcs/anaconda3/envs/tools/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py", line 1128, in __call__ return self.main(*args, **kwargs) File "/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/sqlite_utils/cli.py", line 2686, in convert for row in db.conn.execute(sql, where_args).fetchall(): sqlite3.OperationalError: user-defined function raised exception ``` But without the `--dry-run` flag it does work as expected: ```console $ sqlite-utils convert demo.db demo foo '{"foo": "bar"}' --multi $ sqlite-utils query demo.db "SELECT * FROM demo" [{"foo": "bar"}] ``` ```console $ sqlite-utils --version sqlite-utils, versio… | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/415/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1145882578 | I_kwDOCGYnMM5ETMfS | 408 | `deterministic=True` fails on versions of SQLite prior to 3.8.3 | 24938923 | closed | 0 | 6 | 2022-02-21T14:36:43Z | 2022-03-13T16:54:09Z | 2022-03-02T00:38:11Z | NONE | Hi, love your work. I am unable to lookup indexes in a database using sqlite-utils: ` sqlite-utils indexes city_spec.db --table` or `sqlite-utils indexes city_spec.db MyTable ` **Software** sqlite-utils, version 3.24 sqlite3 --version: 3.36.0 **Output:** Traceback (most recent call last): File "/opt/app-root/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/opt/app-root/lib64/python3.8/site-packages/click/core.py", line 1128, in __call__ return self.main(*args, **kwargs) File "/opt/app-root/lib64/python3.8/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/opt/app-root/lib64/python3.8/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/app-root/lib64/python3.8/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/app-root/lib64/python3.8/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/opt/app-root/lib64/python3.8/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, **kwargs) File "/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/cli.py", line 2123, in indexes ctx.invoke( File "/opt/app-root/lib64/python3.8/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/cli.py", line 1624, in query db.register_fts4_bm25() File "/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py", line 403, in register_fts4_bm25 self.register_function(rank_bm25, deterministic=True) File "/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py", line 399, in register_function register(fn) File "/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py", line 392, in register self.conn.create_function(name, arity, fn, **kwargs) sqlite3.NotSupportedE… | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/408/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1166731361 | I_kwDOCGYnMM5Fiuhh | 414 | I forgot to include the changelog in the 3.25.1 release | 9599 | closed | 0 | 7 | 2022-03-11T18:32:36Z | 2022-03-11T18:40:39Z | 2022-03-11T18:40:39Z | OWNER | I pushed a release for https://github.com/simonw/sqlite-utils/releases/tag/3.25.1 but forgot to include the release notes in `docs/changelog.rst` This means https://sqlite-utils.datasette.io/en/stable/changelog.html isn't showing them. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/414/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1166587040 | I_kwDOCGYnMM5FiLSg | 413 | Display autodoc type information more legibly | 9599 | closed | 0 | 5 | 2022-03-11T15:58:20Z | 2022-03-11T18:07:10Z | 2022-03-11T18:07:10Z | OWNER | https://sqlite-utils.datasette.io/en/3.25/reference.html#sqlite_utils.db.Table.insert looks like this at the moment: <img width="703" alt="image" src="https://user-images.githubusercontent.com/9599/157902622-368935a8-93f2-42e9-98ad-94a45c818e80.png"> | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/413/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1149729902 | PR_kwDOCGYnMM4zbaJy | 410 | Correct spelling mistakes (found with codespell) | 3818 | closed | 0 | 1 | 2022-02-24T20:44:18Z | 2022-03-06T08:48:29Z | 2022-03-01T21:05:29Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/410 | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/410/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | ||||||
1063388037 | I_kwDOCGYnMM4_YgOF | 343 | Provide function to generate hash_id from specified columns | 82988 | closed | 0 | 4 | 2021-11-25T10:12:12Z | 2022-03-02T04:25:25Z | 2022-03-02T04:25:25Z | NONE | Hi I note that you define `_hash()` to create a `hash_id` from non-id column values in a table [here](https://github.com/simonw/sqlite-utils/blob/8f386a0d300d1b1c76132bb75972b755049fb742/sqlite_utils/db.py#L2996). It would be useful to be able to call a complementary function to generate a corresponding `_id` from a subset of specified columns when adding items to another table, eg to support the creation of foreign keys. Or is there a better pattern for doing that? | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/343/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1138948786 | PR_kwDOCGYnMM4y3yW0 | 407 | Add SpatiaLite helpers to CLI | 25778 | closed | 0 | 7 | 2022-02-15T16:50:17Z | 2022-02-16T01:49:40Z | 2022-02-16T00:58:08Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/407 | Closes #398 This adds SpatiaLite helpers to the CLI. ```sh # init spatialite when creating a database sqlite-utils create database.db --enable-wal --init-spatialite # add geometry columns # needs a database, table, geometry column name, type, with optional SRID and not-null # this will throw an error if the table doesn't already exist sqlite-utils add-geometry-column database.db table-name geometry --srid 4326 --not-null # spatial index an existing table/column # this will throw an error it the table and column don't exist sqlite-utils create-spatial-index database.db table-name geometry ``` Docs and tests are included. | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/407/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1128139375 | I_kwDOCGYnMM5DPgpv | 405 | `Database(memory_name="name")` constructor argument | 9599 | closed | 0 | 2 | 2022-02-09T07:15:03Z | 2022-02-16T01:23:16Z | 2022-02-16T01:23:16Z | OWNER | SQLite in-memory databases can be named, in which case multiple connections can be opened to a shared in-memory database running within the same process. Datasette supports this - SQLite could support it too. https://docs.datasette.io/en/0.60.2/internals.html#database-ds-path-none-is-mutable-false-is-memory-false-memory-name-none | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/405/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1124237013 | I_kwDOCGYnMM5DAn7V | 398 | Add SpatiaLite helpers to CLI | 25778 | closed | 0 | 9 | 2022-02-04T14:01:28Z | 2022-02-16T01:02:29Z | 2022-02-16T00:58:07Z | CONTRIBUTOR | Now that #385 is merged, add CLI versions of those methods. ```sh # init spatialite sqlite-utils init-spatialite database.db # or maybe/also sqlite-utils create database.db --enable-wal --spatialite # add geometry columns # needs a database, table, geometry column name, type, with optional SRID and not-null # this needs to create a table if it doesn't already exist sqlite-utils add-geometry-column database.db table-name geometry --srid 4326 --not-null # spatial index an existing table/column sqlite-utils create-spatial-index database.db table-name geometry ``` Should be mostly straightforward. The one thing worth highlighting in docs is that geometry columns can only be added to existing tables. Trying to add a geometry column to a table that doesn't exist yet might mean you have a schema like `{"rowid": int, "geometry": bytes}`. Might be worth nudging people to explicitly create a table first, then add geometry columns. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/398/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1128120451 | I_kwDOCGYnMM5DPcCD | 404 | Add example of `--convert` to the help for `sqlite-utils insert` | 9599 | closed | 0 | 2 | 2022-02-09T06:49:09Z | 2022-02-09T06:56:35Z | 2022-02-09T06:55:16Z | OWNER | https://sqlite-utils.datasette.io/en/3.23/cli-reference.html#insert would be more useful if it included an example of `--convert` in action. I can maybe use an example from https://simonwillison.net/2022/Jan/11/sqlite-utils/ | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/404/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1126692066 | I_kwDOCGYnMM5DJ_Ti | 403 | Document how to add a primary key to a rowid table using `sqlite-utils transform --pk` | 536941 | closed | 0 | 4 | 2022-02-08T01:39:40Z | 2022-02-09T04:22:43Z | 2022-02-08T19:33:59Z | CONTRIBUTOR | *Original title: Add option for adding a new, serial, primary key* sometimes we have tables that don't have primary keys, but ought to have them. we *can* use rowid for that, but it would often be nicer to have an explicit primary key. using the current value of rowid would be fine. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/403/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1125081640 | I_kwDOCGYnMM5DD2Io | 401 | Update SpatiaLite example in the documentation | 9599 | closed | 0 | 2 | 2022-02-06T02:02:07Z | 2022-02-06T02:05:03Z | 2022-02-06T02:03:24Z | OWNER | This one here: https://sqlite-utils.datasette.io/en/3.23/python-api.html#converting-column-values-using-sql-functions It should take advantage of the new methods from: - #79 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/401/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1125077063 | I_kwDOCGYnMM5DD1BH | 400 | `sqlite-utils create-table` ... `--if-not-exists` | 9599 | closed | 0 | 1 | 2022-02-06T01:32:53Z | 2022-02-06T01:34:53Z | 2022-02-06T01:34:46Z | OWNER | Inspired by: - #397 To match the option on `create-index`: https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-index ``` --if-not-exists Ignore if index already exists ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/400/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1123903919 | I_kwDOCGYnMM5C_Wmv | 397 | Support IF NOT EXISTS for table creation | 738408 | closed | 0 | 3 | 2022-02-04T07:41:15Z | 2022-02-06T01:30:46Z | 2022-02-06T01:29:01Z | NONE | Currently, I have a bunch of code that looks like this: ```python subjects = db["subjects"] if db["subjects"].exists() else db["subjects"].create({ ... }) ``` It would be neat if sqlite-utils could simplify that by supporting `CREATE TABLE IF NOT EXISTS`, so that I'd be able to write, e.g. ```python subjects = db["subjects"].create({...}, if_not_exists=True) ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/397/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
683805434 | MDU6SXNzdWU2ODM4MDU0MzQ= | 135 | Code for finding SpatiaLite in the usual locations | 9599 | closed | 0 | 3 | 2020-08-21T20:15:34Z | 2022-02-05T00:04:26Z | 2020-08-21T20:30:13Z | OWNER | I built this for `shapefile-to-sqlite` but it would be useful in `sqlite-utils` too: https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L16-L19 ```python SPATIALITE_PATHS = ( "/usr/lib/x86_64-linux-gnu/mod_spatialite.so", "/usr/local/lib/mod_spatialite.dylib", ) ``` https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L105-L109 ```python def find_spatialite(): for path in SPATIALITE_PATHS: if os.path.exists(path): return path return None ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/135/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
683812642 | MDU6SXNzdWU2ODM4MTI2NDI= | 136 | --load-extension=spatialite shortcut option | 9599 | closed | 0 | 3 | 2020-08-21T20:31:25Z | 2022-02-05T00:04:26Z | 2020-10-16T19:14:32Z | OWNER | In conjunction with #135 - this would do the same thing as `--load-extension=path-to-spatialite` (see #134) | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/136/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
723460107 | MDU6SXNzdWU3MjM0NjAxMDc= | 187 | Maybe: Utility method / CLI tool for initializing SpatiaLite | 9599 | closed | 0 | 2 | 2020-10-16T19:04:03Z | 2022-02-05T00:04:26Z | 2020-10-16T19:15:13Z | OWNER | > I think this should initialize SpatiaLite against the current database if it has not been initialized already. > > Relevant code: https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L112-L126 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/187/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
723708310 | MDU6SXNzdWU3MjM3MDgzMTA= | 188 | About loading spatialite | 30607 | closed | 0 | 1 | 2020-10-17T08:47:02Z | 2022-02-05T00:04:26Z | 2020-10-17T08:52:58Z | NONE | Hi @simonw , If I run ``` sqlite3 .load /usr/local/lib/mod_spatialite.so select spatialite_version(); ``` I have `5.0.0`. ![image](https://user-images.githubusercontent.com/30607/96332706-d8cd3300-1065-11eb-906b-daf99963198e.png) If I run ``` sqlite-utils :memory: "select spatialite_version()" --load-extension=spatialite ``` I have ``` Traceback (most recent call last): File "/home/aborruso/.local/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/home/aborruso/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/home/aborruso/.local/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/home/aborruso/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/aborruso/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/aborruso/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/aborruso/.local/lib/python3.8/site-packages/sqlite_utils/cli.py", line 936, in query _load_extensions(db, load_extension) File "/home/aborruso/.local/lib/python3.8/site-packages/sqlite_utils/cli.py", line 1326, in _load_extensions db.conn.load_extension(ext) TypeError: argument 1 must be str, not None ``` How to load properly spatialite extension in sqlite-utils? Thank you very muc | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/188/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1102899312 | PR_kwDOCGYnMM4w_p22 | 385 | Add new spatialite helper methods | 25778 | closed | 0 | 16 | 2022-01-14T03:57:30Z | 2022-02-05T00:04:26Z | 2022-02-04T05:55:10Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/385 | Refs #79 This PR adds three new Spatialite-related methods to Database and Table: - `Database.init_spatialite` loads the Spatialite extension and initializes it - `Table.add_geometry_column` adds a geometry column - `Table.create_spatial_index` creates a spatial index Has tests and documentation. Feedback very welcome. | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/385/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
534507142 | MDU6SXNzdWU1MzQ1MDcxNDI= | 69 | Feature request: enable extensions loading | 30607 | closed | 0 | 3 | 2019-12-08T08:06:25Z | 2022-02-05T00:04:25Z | 2020-10-16T18:42:49Z | NONE | Hi, it would be great to add a parameter that enables the load of a sqlite extension you need. Something like "-ext modspatialite". In this way your great tool would be even more comfortable and powerful. Thank you very much | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/69/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
557825032 | MDU6SXNzdWU1NTc4MjUwMzI= | 77 | Ability to insert data that is transformed by a SQL function | 9599 | closed | 0 | 2 | 2020-01-30T23:45:55Z | 2022-02-05T00:04:25Z | 2020-01-31T00:24:32Z | OWNER | I want to be able to run the equivalent of this SQL insert: ```python # Convert to "Well Known Text" format wkt = shape(geojson['geometry']).wkt # Insert and commit the record conn.execute("INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))", ( "Wales", wkt )) conn.commit() ``` From the Datasette SpatiaLite docs: https://datasette.readthedocs.io/en/stable/spatialite.html To do this, I need a way of telling `sqlite-utils` that a specific column should be wrapped in `GeomFromText(?, 4326)`. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/77/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
557842245 | MDU6SXNzdWU1NTc4NDIyNDU= | 79 | Helper methods for working with SpatiaLite | 9599 | closed | 0 | 8 | 2020-01-31T00:39:19Z | 2022-02-05T00:04:25Z | 2022-02-04T05:55:11Z | OWNER | As demonstrated by this piece of documentation, using SpatiaLite with sqlite-utils requires a fair bit of boilerplate: https://github.com/simonw/sqlite-utils/blob/f7289174e66ae4d91d57de94bbd9d09fabf7aff4/docs/python-api.rst#L880-L909 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/79/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1123849278 | I_kwDOCGYnMM5C_JQ- | 395 | "apt-get: command not found" error on macOS | 9599 | closed | 0 | 1 | 2022-02-04T06:03:42Z | 2022-02-04T06:10:58Z | 2022-02-04T06:10:58Z | OWNER | Yeah, `apt-get` isn't a thing on macOS so 4a2a3e2fd0d5534f446b3f1fee34cb165e4d86d2 (to test #79 against real SpatiaLite) broke. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/395/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed |