Getting syntax error, while running python wheel taskSyntaxError: invalid syntax (command--1-3037714297, line 20)File ~/.ipykernel/5648/command--1-3037714297:20 import python-json-logger ^ SyntaxError: invalid syntax
Hi all, "Driver is up but is not responsive, likely due to GC." This is the message in cluster event logs. Can anyone help me with this. What does GC means? Garbage collection? Can we control it externally?
9/10 times GC is due to out of memory exceptions.@Jaron spark.catalog.clearCache() is not a configurable option, but rather a command to submit.
I have created a dlt pipeline based four sql notebooks, each containing between 1 and 3 queries. Each query begins with "create or refresh live table ..." yet each one outputs a materialized view. I have tried deleting the materialized views and ru...
Hello guysI am trying to read an excel file and even using PERMISSIVE mode, its truncating the records that contains #REF in any column Can anyone please help me on that? schema = StructType([\ StructField('Col1', DateType(), True), \ <-------...
HiI have a databricks table that has a column with a string datatypeWhen loading data, certain rows are prefixed by ' > 'Does anybody know what would cause this?It seems to be when the string is above a certain number of characters (around 200)Thanks
It looks like default string data length exceeded and due to that its showing this kind of symbol and extra new line in the column.
I'm using the following to call a Databricks REST API. When I use a PAT for access_token, everything works fine. When I use a Microsoft Entra ID access token, the response returns 400. The service principal has access to the workspace and is part of ...
hello, Try to print the repose and see are you table to see the access_token in the payload else looks like access issue.Try to go to the workspace setting and grant token access permission to the service principle.Workspace > Setting
How to use a SQL IDE with Databricks SQLDatabricks provides SQL IDE support using DataGrip and DBeaver with Databricks SQL.Let us know in the comments if you've used DataDrip or DBeaver with Databricks! Let us know if there are any other SQL IDEs you...
dbeaver is perfectly working fine but I fount one issue it wont show the correct error for query.
Hi all,I have been having some trouble running a workflow that consists of 3 tasks that run sequentially. Task1 runs on an all-purpose cluster and kicks off Task2 that needs to run on a job cluster. Task2 kicks off Task3 which also uses a job cluster...
I don't know if you've cut off your yaml snippet, but your snippet doesn't show your job cluster with key job-cluster. Just to validate, your job cluster is also defined in your workflow yaml?Edit: Looking it it again and knowing the defaults, it loo...
Could you please provide guidance on the correct way to dynamically import a Python module from a user-specific path in Databricks Repos? Any advice on resolving the ModuleNotFoundError would be greatly appreciated.udf_check_table_exists notebook:fro...
I am currently trying to deploy external parquet tables to the Databricks UC using terraform. However, for some tables I get the following error:Error: cannot create sql table: Post "https://[MASKED]/api/2.0/sql/statements/": context deadline exceede...
Hey @Retired_mod,thanks for your reply and sorry for the late reply from my side. I couldn't fix the problem with the databricks terraform provider unfortunately. I now switched to using liquibase to deploy tables to databricks.
I'm able to create and use widgets using the UI in my SQL notebooks, but they get lost quite frequently when the notebook is reset.There is documentation suggesting we can create widgets in code in SQL: https://learn.microsoft.com/en-us/azure/databri...
Hi @JeremyH - can you please try adding like the below in your query and see if widgets are getting populated? {{parameter_name }}
Hi Community members, I tried creating a Delta UniForm table using databricks notebook. I created a database without providing location. It took the dbfs default storage location. On top of that I was able to create a Delta UniForm table. Then I trie...
HiI have started using serverless compute but have come across the limitation that I cannot use the local filesystem for temporarily storing the files and directories before moving them to where they need to be in adls.Does anyone have a way of unzip...
Do you have the ability to make a Unity Catalog volume? You could use it as temporary storage before migrating the files to adls.
I'm trying to run a notebook that has 14 commands from adf notebook activity, I'm writing the transformed data to a delta table in the last command. I'm handling the last command in try except block to handle errors, and raising error on exception.wh...
Hi Community! I’m working in a Fatabricks notebook and using the Oracle JDBC Thin Client connector to query and Oracle table, merge together and select specific rows from my dataframe and write those rows to a table back in Oracle. All of this works...
This stack overflow post ran into the same issue, would be worth trying How to delete a record from Oracle Table in Python SQLAlchemy - Stack Overflow
| User | Count |
|---|---|
| 1624 | |
| 790 | |
| 504 | |
| 349 | |
| 287 |
